'The Signal and the Noise': Secrets of a master numbers cruncher

Nate Silver interviewed more than 100 forecasters in a dozen fields and tells many of the stories.
Nate Silver interviewed more than 100 forecasters in a dozen fields and tells many of the stories. (ROBERT GAULDIN / Penguin Press)
Posted: October 23, 2012

Why So Many Predictions Fail - But Some Don't

By Nate Silver

Penguin. 523 pp. $27.95.

Reviewed by Sharon Bertsch McGrayne
I don't play chess. I don't gamble. I don't play poker. But I loved this book.

Trust Nate Silver to write a wise but entertaining book in which the heroes are baseball coaches, poker players, gamblers, weather forecasters, and chess-playing computer scientists - and the villains are economists and political scientists, especially those who offer themselves as pundits.

Nate Silver fell in love with statistics - namely, baseball statistics - at age 6. Explaining his "unhealthy obsession" with baseball stats, Silver writes,

Baseball offers perhaps the world's richest data set: pretty much everything that has happened on a major-league playing field in the past 140 years has been dutifully and accurately recorded, and hundreds of players play in the big leagues every year.

So in his early 20s, bored by his day job, he started an Internet site that made remarkably correct predictions about the performance of each Major League Baseball player.

At the same time, Silver moonlighted as a professional poker player online. When Congress shut down his favorite Internet poker site in 2006, he got interested in politics.

Soon his website, FiveThirtyEight.com, was averaging the forecasts of many political pollsters but, in an important innovation, weighting them according to the accuracy of their previous predictions. During the 2008 election, Silver forecast not only the winner in 49 of the 50 states, but also the outcome of 35 U.S. Senate races.

As Silver explains in The Signal and the Noise, his secret ingredient was Bayes' theorem. Poker, for example, is "fundamentally and profoundly Bayesian," with players unconsciously using this 18th-century mathematical approach by refining their initial assessment of their opponents' cards after every bet, check, and call. Says Silver, "If you doubt the practical uses of Bayes's theorem, you have probably never witnessed a poker game."

Silver conceived his book as "an investigation of data-driven predictions in fields ranging from baseball to finance to national security." He interviewed more than 100 forecasters in a dozen fields and tells many of their stories.

Among my favorite data-using experts were:

Sports bettor Haralabos "Bob" Voulgaris, who can earn $1 million in an off year by applying Bayes' approach more expertly than many economists or scientists.

Murray Campbell, one of the chief engineers of IBM's Deep Blue computer program, which defeated Garry Kasparov, the world's top chess player. "There are more possible chess games than the number of atoms in the universe," Silver tells us.

Earthquake scientists, who still can't predict the big ones. Weather forecasters can check their forecasts by gazing skyward, but seismologists study rocks 10 miles underground.

Silver concludes that "prediction in the era of Big Data" is not going very well. The exponential growth in information over the Web is not a cure-all. Data-driven predictions can succeed, or fail, in spectacular ways.

Silver's hero is the National Weather Service, which has become 350 percent more accurate in the last 25 years, especially in forecasting hurricanes. Science explains how weather systems behave, so Weather Service forecasters make a model simulating the atmosphere. Then, because they cannot quantify the position of each molecule in clouds, rainstorms, and hurricanes, they make probabilistic predictions.

Today, the Weather Service can predict within 100 miles where a hurricane is most likely to make landfall three days later, providing time to evacuate an area, provided that people listen.

In contrast, mainstream forecasters in economics, political science, and domestic and international affairs failed to predict the fall of the Soviet empire; the Sept. 11, 2001 attacks; and the recessions of 1990, 2001, and 2007, "even once they were already under way." And Standard & Poor's rated trillions of dollars in investments as almost completely safe although they turned out to be almost completely unsafe. To make matters worse, the more media interviews economists gave, the worse their predictions tended to be. Yet they remained supremely confident about the precision and accuracy of their forecasts.

Still, Silver is optimistic that "Big Data will produce progress - eventually." But too much data is junk, and it's hard to separate signals from noise, wheat from chaff.

Bayesian statisticians, including Silver, say we can learn from uncertainty by refining our initial opinion of the situation every time more evidence appears. If we admit we might be wrong, Bayes will edge us closer and closer to the truth, albeit a little bit at a time.

Above all, we need an attitude change, Silver says. He wants us to look

beyond our personal biases and prejudices and toward the truth of a problem. . . . You'll need to adopt some different habits from the pundits you see on TV. You will need to learn how to express - and quantify - the uncertainty in your predictions. You will need to update your forecast as facts and circumstances change. You will need to recognize that there is wisdom in seeing the world from a different viewpoint. The more you are willing to do these things, the more capable you will be of evaluating a wide variety of information without abusing it.

Although statistics is a mathematical science, Silver manages to tell his story very well without much math. He has filled his entertaining and highly readable book with graphs and tables explaining his conclusions. But the reader looking for Bayes' theorem won't find it in the form statisticians use. Instead, Silver tucks the essential part of it away in the bottom right-hand corners of three tables on Pages 245, 247, and 248.

Sharon Bertsch McGrayne is the author of "The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy." Her website is McGrayne.com.

comments powered by Disqus