Oops, they did it again. Big misses in the recent Michigan and Wisconsin Democratic primaries left some proclaiming (again) that no one should trust the polls. The better approach is to follow President Ronald Reagan’s dictum and “trust, but verify.”
Political polling is clearly going through a rough patch. The consistent underestimation of President Donald Trump’s level of general-election support is the most widely known example of reduced reliability. But the problem extends at least as far back as 2012, when many pollsters thought Republican nominee Mitt Romney would beat President Barack Obama. The night came to a surprisingly (for conservatives, at least) quick end as Obama’s nearly four-point victory meant he easily carried almost every swing state.
It’s getting harder to produce accurate results, and professional pollsters know why: Almost no one answers calls from people they don’t already know. This means that the bedrock statistical fact undergirding modern surveys — that a small random sample of any population can accurately measure the opinions of the whole population within a small margin of error — can no longer be relied upon.
Pollsters have adapted by adopting new survey techniques and using demographic benchmarks to adjust the samples they obtain — a process known as “weighting.” Trouble is, their results are only as good as those benchmarks. And given today’s severe polarization, even a 1 or 2 percent error in measuring the underlying partisanship of the electorate can be the difference between predicting Vice President Kamala Harris would win most swing states and the reality of Trump’s sweep.
Polling partisan primaries is even more difficult than general elections. In those races, pollsters simply don’t have the demographic benchmarks they need to weight. Predicting primary turnout levels, which vary wildly between cycles, is devilishly hard. States that allow anyone to cast ballots in a party primary, as Michigan and Wisconsin do, make it harder still. How many independents or Republicans could decide to vote in a Democratic primary? Impossible to know ahead of time — especially with the swirling crosscurrents and incentives created by this year’s spate of high-stakes left vs. establishment contests.
Both of these issues were probably at the root of the industry’s recent colossal failure. Political data analyst G. Elliott Morris did a deep dive into one Wisconsin poll of the Democratic gubernatorial race that had showed democratic socialist Francesca Hong ahead by 22 points in August (she lost to establishment candidate David Crowley by half a point) and found that the sample was both too young and too left leaning. That’s not surprising, since the primary drew more than 792,000 votes, over 250,000 votes more than the Democrats’ previous competitive gubernatorial contest.
Record turnout was probably an issue for Michigan’s pollsters, too. More than 1.5 million people voted in that state’s Democratic Senate primary, topping the prior high-water mark of about 1.2 million. Turnout nearly reached the record for a Democratic presidential primary in Michigan, 2020’s 1.6 million. Most pollsters would be afraid to release a survey that assumed record-high turnout, even if their data suggested that might happen.
I encountered that while covering Hungary’s recent national election. A leading pollster told me his data suggested record turnout, which favored his party, the opposition Tisza. But his public release adjusted the turnout level downward to meet the prior record, which correspondingly decreased the margin by which Tisza led. But he was right the first time. On election night, there was indeed a record turnout, and Tisza’s winning margin was even a bit higher than what his unreleased data had predicted.
There’s a clear lesson in all this: Always look under the hood. When reporting on polling results, analysts need to examine how the pollster weighted the survey and compare those to benchmarks used by others. They need to assess whether the assumptions embedded within those weights seem reasonable. And they need to look carefully at a survey’s trend data to see whether a post-release shift in voter opinion is likely.
Analysts also need to read a poll in the light of well-known political phenomena. For example, a third-party candidate will almost always get between one-third and one-half fewer votes than suggested by any “final” pre-election survey that doesn’t poll through the final weekend. That’s because many partisans unhappy with their party’s nominee reconsider casting a protest vote as the possibility that the other party might win becomes real. I used this technique in 2016 to predict Hillary Clinton’s popular vote margin nearly exactly, even though almost all the final polls had her leading by three or more percentage points.
Analysts should also be wary of assumptions that self-professed voter enthusiasm is an accurate predictor of turnout. In 2012, for example, Gallup estimated that Black voters would turn out at a slightly lower rate than their then-record 2008 performance. That’s not surprising given that polls found Romney supporters were much more enthusiastic about voting than Obama backers. Yet on Election Day, Black people voted at higher rates than White people for the first time, and even surpassed their rate from four years earlier.
So let the poll reader beware. Political polling, for all its faults, remains the best source of available information for assessing where a race stands at any point in time. Just treat it the way you would a house you were interested in buying: Have a look at that foundation before signing the deal.
The post Polls are getting worse. They’re still the best gauge we have. appeared first on Washington Post.
