I followed Nate Silver’s blog (FiveThirtyEight) closely during the run-up to election day 2012. His premise was simple: grab every public poll possible, attempt to correct for pollsters’ known biases, and produce a forecast based on the result. Somehow no one had thought to do this before. Silver simply crunched the numbers and nailed the outcomes in every state. Meanwhile, pundits, bloggers, and assorted blowhards made predictions based on nothing but gut feeling and partisan hackery, and they mostly missed the mark (often by a wide margin).
I was looking forward to reading more about his methodology in this book, as well as his take on the principles involved in making predictions from noisy data. In this regard, I wasn’t disappointed. Silver does a good job of laying out the rules of the road:
- It’s easy to mistake essentially random fluctuations for a meaningful pattern, and in some contexts (say, earthquake predictions), this can have devastating results.
- Having a well-formed, testable theory is better than just looking for any correlations you can find in your data set.
- Always make predictions and update your probability estimates like a good Bayesian. Your predictions should approach reality as you continually refine them.
- Watch out for biases in yourself and in your data set.
- Often overlooked: make sure incentives are aligned with the results you would like to achieve.
Also, some specific interesting facts:
- Making a living at poker is really hard. Without any really bad players at the table, it’s nearly impossible for anyone but the top players to turn a profit.
- The efficient market hypothesis doesn’t hold up to scrutiny; however, even though the stock market has discernible patterns, it may not be possible to exploit the patterns and consistently beat the market.
- Weather prediction has gotten a lot better in the last couple decades, even though most people think it hasn’t.
- Both earthquakes and terrorist attacks follow a power law distribution.
If you’re a stock trader, scientist, gambler, or simply someone who wants to form an accurate picture in a noisy environment, there’s something in this book for you. It’s nice to see this kind of clear-headed, rational thinking becoming sexier recently. The book is also well cited, which helps give weight to some of the more counterintuitive claims.
There was a missed opportunity to spend some time on results from the medical research industry. It’s well known that publication bias and other factors result in misleadingly positive results for new treatments, which ultimately go away after independent researchers attempt (unsuccessfully) to reproduce the results. It seems like a pertinent, prototypical case of finding patterns in noise, one which could have been instructive.
A final note: Silver is not the best writer; his prose is uneven and occasionally downright awkward. His casual style works fine for a blog, but here it diminishes the impact the book could otherwise have had. This is his first published book, and it shows. There are also a couple glaring mistakes that make me think he needed a better editor.