A Forecast Is Not a Vote Count
What polling can tell us, and why we should stop asking it to tell us the ending.
By Wayne Ince | Breaking Ranks Blog
A map fills with red and blue. A percentage appears beside a party's name. Within minutes, a cable chyron or a partisan account declares the election over.
The ballots have other plans.
With the November 3 midterms here, anyone following Nate Silver and the other forecasters should keep four jobs straight. Polls estimate what voters prefer. Models calculate probabilities. Maps illustrate scenarios. Commentary argues about what all of it means.
None of them is an election result.
The forecasters know this. Silver has been warning about uncertainty for years, and the good models print their caveats right next to their numbers. The trouble starts when someone screenshots the number and leaves the caveat behind.
A lead is an estimate
A poll tells us what a sample of people said, during a set window, under a set of survey methods. It is evidence about the electorate. It does not count the electorate's votes.
Who answers the phone matters. Who actually shows up matters. The assumptions a pollster uses to pick out likely voters matter.
A narrow lead is worth watching. The better question is what the evidence supports and how much doubt is still on the table.
A probability is not a promise
On the eve of the election, Silver Bulletin's FLIPR model gave Democrats an 80.1 percent chance of winning the Senate. Here are its race-level numbers:
Race Democratic win probability
Texas, Ohio, Maine and KansasAbout 75% each
Alaska63%
Iowa56%
Michigan (Democratic hold)85%
Source: Silver Bulletin FLIPR forecast, natesilver.net, as of [DATE AND TIME].
A 75 percent chance of winning does not mean a candidate gets 75 percent of the vote, or wins comfortably. It means the model gives that outcome about three chances in four, based on its evidence and its assumptions.
The fourth chance belongs in the headline too.
At 56 percent, Iowa is close to a coin flip. At 85 percent, Michigan is a strong Democratic hold, and a loss is still on the board.
Four races at 75 percent each do not add up to a 75 percent chance of sweeping all four. If the races moved independently, the odds of a sweep would be about one in three. They don't move independently. The same polling miss or national swing can drag several races in the same direction at once.
An underdog win does not prove a forecast was junk. You judge a forecaster across many calls, and you check whether its confidence held up. That cuts both ways: a forecaster owes readers a public record of how past odds performed, and readers should go look for it.
"Favored" is useful information. Read it as "certain" and you've changed what it says.
The rule holds for whichever side ends up on the wrong end of these numbers. It holds for me too. I call races on this blog, and those calls are estimates.
A scenario map shows one possibility
Election maps make uncertainty look solid. They let us picture which combinations of states or districts could add up to a majority.
They also make a hypothetical easy to remember as a prediction. That is the danger.
A scenario can show one path to control without proving that path is likely. A handful of maps picked to show different outcomes is not a fair sample of everything a model ran.
Treat those maps as a set of questions: which path, which obstacles, which assumptions. They are not an advance copy of the final count.
Interpretation needs its own label
There is one more step between the numbers and the story we tell about them.
A polling edge may point to an opening. A forecast may show one party with more routes to a majority. Neither one tells you why voters are choosing what they choose.
Claims about anger, enthusiasm, persuasion or a mandate need their own evidence. A number cannot carry every explanation we want to hang on it.
For writers, that means picking the right verb. A poll estimates. A model projects. A scenario illustrates. An analyst argues.
Get the verb right and readers can see where the evidence stops and our judgment starts.
What to do tomorrow night
When a race gets called early, check who called it and how much of the expected vote is counted. When the early count and the forecast disagree, look at what's still out: mail ballots, late-reporting counties, provisional ballots. Early returns lean one way or the other depending on what gets counted first. Wait for the count.
Readers deserve coverage that shows confidence without faking certainty. We can name the favorite, describe a plausible upset and lay out the stakes while admitting nobody knows the outcome yet.
Read the polls. Examine the forecasts. Question the assumptions.
Then let the votes settle what only votes can settle.
When you see an election forecast, what do you notice first: the favorite's odds, or the chance the favorite loses?
Comments