Nate Silver's Finest Hour (Part 1 of 2)
A story told through our most hallowed of traditions: Twitter spats
I. 2012
This post isn’t about 2012. But it does start there.
2012 is a big year for Nate Silver. He publishes his first book, The Signal and the Noise, and it does great. Amazon ranks it the Best Non-fiction Book of 2012! The Wall Street Journal ranks it one of the top ten books of 2012! It reaches #4 on the New York Times best sellers list! If there’s a ranking of top nonfiction books to get for your dad for Christmas, it’s definitely number one.
Better yet, his blog, 538 - which partnered with the NY Times starting in 2010 - becomes the talk of the town.
In the lead up to Election Night 2012, the pundit word du jour was “tossup”. The national polls were tight, so pundits liked to talk about how close the race between Obama and Romney was.1 See the words of talking head Joe Scarborough:
Anybody that thinks that this race is anything but a tossup right now is such an ideologue, they should be kept away from typewriters, computers, laptops and microphones for the next 10 days, because they’re jokes.
Nate, with his fancy ‘models’ and ‘polls’, thought otherwise. At the time, 538 gave a Obama an 80% chance of winning the electoral college. So naturally, Nate does what any stats nerd would do: he challenges Joe Scarborough to a bet on Twitter.
Ever the politician, Joe replies with:
It goes back and forth like this, with Nate trying to get Joe to bet and Joe saying something to the effect of “I’d rather bet… on the America people 🇺🇸 🦅 ” *star-spangled banner starts playing*.
Of course, Election Day rolls around and Obama wins handily. 538 correctly predicts all 50 states in the 2012 presidential election, and therefore the overall Obama-victory outcome as well. Liberals rejoice and vault Nate into the pantheon of election heroes. The public views this achievement as a victory for his stat-nerd approach over that of traditional political pundits. Nate is vindicated, Joe Scarborough bet or not.
II. Poker
In a twist fit for a mediocre Twilight Zone episode, Nate’s success also spells his undoing. How the public sees him, and how he sees himself, diverge.
Silver sees himself as a risk assessor. He sees himself as the ESPN feature that calculates the odds each player has of winning a poker hand.
It would be weird to watch the poker game above, see the 72%, and claim “ESPN is calling this hand for Faraz.” No. Given the visible cards, they’re assessing who’s more likely to win and who’s at risk of losing. Even with high probabilities, it’s not a guarantee of anything - if you have a 95% chance of winning, that means definitively that there is a set of cards that will result in your loss.
In his book and his blog, Nate references poker constantly. (He made a bunch of money in online poker, so, you know, figures.) In his final post before the 2012 election, he explicitly compares his election forecast to poker:
I hope you’ll excuse the cliché, but it’s appropriate here: in poker, making an inside straight requires you to catch one of 4 cards out of 48 remaining in the deck, the chances of which are about 8 percent. Those are now about Mr. Romney’s chances of winning the Electoral College, according to the FiveThirtyEight forecast.
As any poker player knows, those 8 percent chances do come up once in a while. If it happens this year, then a lot of polling firms will have to re-examine their assumptions — and we will have to re-examine ours about how trustworthy the polls are. But the odds are that Mr. Obama will win another term.
Even when he calculates Obama has 92% chance of winning, even when Joe Scarborough calls him a joke and says the election is a toss-up, Nate doesn’t say “Fuck you Joe, Obama is totally, absolutely going to win.” He makes a drawn out comparison to poker to illustrate that his role is not to predict the future. His role is not to tell you what cards will be dealt next. His role is to tell you what to make of your current hand.
After 2012, though, this is not how the public sees him. After calling 49 states right in 2008 and 50 states right in 2012, the general population isn’t thinking in probabilities. They’re thinking in magic. Nate is not a risk assessor; he’s a goddamn fortune teller.
III. Days Before the 2016 Election
Thanks to 538’s phenomenal success in 2012, election forecasting is a burgeoning field in 2016. Everyone and their roommate has started a statistical model predicting the 2016 presidential election. We’ve got:
538, not-all-that-amicably divorced from the New York Times, is now its own ESPN-funded website.
The New York Times’ Upshot, which was created in part to fill in for 538.
The Huffington Post
Daily Kos
Slate
Some dummies at a school called ‘Princeton’
It’s worth pointing out that, like 538, none of these organizations are pollsters. They are election forecasters. They grade, aggregate, and layer other information on top of polls to create their forecasts, but they do not run polls themselves. Because these election forecasters rely on polls, they often get lumped in with pollsters, but they’re separate.
In case you somehow missed the emotional turmoil of the 2016 election, almost all the polls showed Clinton leading. Naturally, all the election forecasters, who all rely on the polls, show Clinton as more likely to win than not.
But there’s a big range in their assessments. The day of the election, the NYT Upshot gives Trump a 15% chance of winning the electoral college. Slate gives him a 10%; Daily Kos a 8%; Huffington Post a 1.7%. That bunch of dummies at Princeton? They give Trump a 1% chance.
And then there’s 538, which gives Trump a 29% chance of winning. This is actually down from his peak a day or two earlier, when they gave him a 35% chance.
Silver gives some thoughtful explanations for why they’re way more bullish on Trump than others, most clearly laid out here. I don’t want to jump the gun, but I also feel obligated to draw attention to one reason he gives for 538’s relative bullishness on Trump. So, pre-election Nate, why might Trump have a chance?
State outcomes are highly correlated with one another, so polling errors in one state are likely to be replicated in other, similar states.
… Basically, this means that you shouldn’t count on states to behave independently of one another, especially if they’re demographically similar. If Clinton loses Pennsylvania despite having a big lead in the polls there, for instance, she might also have problems in Michigan, North Carolina and other swing states. What seems like an impregnable firewall in the Electoral College may begin to collapse.
Silver wrote this 2 weeks before the election, and - spoilers for the election outcome - it is so spot-on I had to double-check that it wasn’t written after the election. Clinton lost almost entirely because a polling error with one demographic group - non-college-educated white people - meant that she lost at least three states with a lot of them.
Whatever Silver’s reasons, the Twitterati are, at best, uncomfortable with his conclusions. The pundit response is to gently cast doubt on 538’s model. Paul Krugman links to a now-defunct discussion saying that 538 is probably wrong. Matt Yglesias, who I regret to lump in with other pundits, says Nate Silver underrates Clinton’s odds. Wired has a piece touting Sam Wang (the Princeton guy who gave Trump a 1%) as the new king of election forecasting, hilariously and sadly, the day before the election. There’s a kind of shocking hubris throughout that acts as if Clinton has already won.2 Who needs to see results?
But nobody, nobody is as ready to dunk on 538 as the Huffington Post. I don’t know what kind of editorial beef they have, but hoo boy do they not like Nate. The opening salvo comes from Ryan Grim, with his article “Nate Silver Is Unskewing Polls -- All Of Them -- In Trump's Direction”. He accuses Silver of “just guessing”, “monkeying around with the numbers”, and “making a mockery of the very forecasting industry that he popularized”.3
In response, Nate goes on a bender.
The Twitter thread is worth reading in all its nerd rage, but in short, he says: everything we do is empirically verified; we make changes because they work, not because I randomly want to do them; if you haven’t accounted for polling errors correlated between states (*cough* maybe in the Blue Wall *cough*), your model will be fucked up. Also, the HuffPo model is fucked up. (I’m paraphrasing.)
For good measure, he ends the rant with:
Maybe because Nate went for the balls, the Huffington Post writes another article the a few days later titled “What’s Wrong With 538?” It’s the same attack as the other Huffington Post articles. Nate’s model is too complicated; he’s becoming like the pundits he used to criticize; he’s hedging because he lacks confidence or because he’s trying to drum up interest in 538. The critics condescendingly shake their heads and say, “Don’t listen to him - Hillary’s got this.”
Narrator: On the next Good Reason… she doesn’t.
In a weird twist of fate, Obama and the Dems had a big Electoral College advantage that protected him against (what appeared to be) a close national race.
This hubris is particularly strong in the Wired article, which features this quote:
When the smoke clears on Tuesday---and it will clear---what will emerge is Wang and his Princeton Election Consortium website and calculations. … What will be vindicated is precisely the sort of math approach that Silver once rode to fame and fortune.
And ends with this r/agedlikemilk quote:
… when Clinton crosses the finish line with something close to 300 Electoral College votes and a popular vote victory somewhere between two and five percentage points; and Nate Silver is telling his 1.7 million Twitter followers that he'd been right all along this election, Sam Wang will be standing tall above the fray, draped in his "median-based probability election" cloak.
Long live the new election data king.
In my favorite snippet of this, Grim claims he’s doing the hard, brave thing by telling his audience exactly what it wants to hear. “It’s not easy to sit here and tell you that Clinton has a 98 percent chance of winning,” he says. “Everything inside us screams out that life is too full of uncertainty, that being so sure is just a fantasy. But that’s what the numbers say.”
Reading this is like watching the Fyre festival documentary