If my inbox is any indication, we’ve reached the point when everyone just wants to know whether the polls will be right.
Fair enough, so in this newsletter, I’ll respond to a few of your questions about the specific risks facing pollsters today.
But first I wanted to report one rare opportunity to put our New York Times/Siena University polls to an early test: Alaska’s unusual top-four Senate primary, held in August. It’s a plausible test because the primary ballot is a lot like a general election: Everyone was eligible to vote, and candidates from all parties appeared on the same ballot, with the top four vote-getters advancing to the general election.
If the polls were badly off, we might expect to see it by comparing the results of the primary with the polling results among those who participated in the primary.
The results among polling respondents who voted in the primary look pretty good. Among those who voted in the primary, the Democrat Mary Peltola had a six-point lead over the Republican incumbent, Dan Sullivan. That’s quite similar to the actual Alaska primary result, when the Democratic candidates outgained the Republicans by five points (or seven points if we exclude the vote for the “other” Dan Sullivan, who clearly won the support of Democratic voters who tried to advance him to play a spoiler role).
So it doesn’t seem our Alaska poll overestimated how well the Democrats would do, at least among this segment of the electorate. That’s certainly more reassuring than if the poll had found, say, Ms. Peltola ahead by 14 points among primary voters. But it won’t let pollsters sleep easy at night either. Pollsters face all kinds of risks, and you asked about many of them.
Nonresponse bias
The most common reason to think polls will overestimate Democrats’ strength is nonresponse bias — the possibility that polls will be biased because President Trump’s supporters don’t respond to surveys.
Is it still a problem?
“In the past, the issue of nonresponse bias and weighting techniques have come under discussion. Did the recent polls have evidence that these issues persist?” — Juan Regalon
The short answer is, yes: Democrats continue to be likelier to respond to surveys than Republicans (among white voters).
In our final Times/Siena national poll, white Democrats were about 30 percent likelier to respond than white Republicans. That’s a higher gap than in 2024 and pretty similar to what we saw in our final polls in 2020, when the polls underestimated Mr. Trump’s support by about four points on average.
The persistence of nonresponse bias absolutely introduces the risk that the polls will underestimate Republicans, but it doesn’t doom the polls either. Back in 2022, white Democrats were likelier to pick up the phone at a pretty similar clip, but the polls wound up being solid in the end.
How is that possible? Mostly because of a technique called “weighting,” in which pollsters give more weight to underrepresented groups. So if a survey doesn’t get enough registered Republicans, pollsters will give more weight to the registered Republicans who did take the poll. Weighting doesn’t guarantee that the polls will be accurate, but it also means that the polls aren’t guaranteed to be off.
Recall vote
One common way to weight a poll is by using “recall vote” — how people say they voted in the previous presidential election.
The idea is that a pollster will weight the poll such that the proportion of people who recall voting for Mr. Trump or Kamala Harris matches the actual result of the last election.
This is an enormous change in the polling industry, and it’s one of the biggest reasons we can’t assume polls will simply err as they did in the past. Indeed, the geographic pattern of polling error was quite different in 2022 and 2024 compared with 2016 and 2020, when polls tended to err most in predominantly white working-class states like Wisconsin. Instead, polls in recent cycles haven’t been so bad in Wisconsin, even as pollsters have struggled in swingy states like Florida and New Jersey, where it’s possible that weighting by recall vote has helped nudge numbers toward the prior election result.
As the New Jersey and Florida examples suggest, weighting by recall vote is a little fraught. It’s quite clear that some people don’t give accurate answers to the question. Many people who didn’t vote claim to have done so. Many people who did vote refuse to offer an answer. And historically, people seem likelier to report voting for the winner than they actually did (though this did not seem to be an issue in 2024).
This year, there’s another concern about recall-vote weighting: the possibility that Mr. Trump’s voters won’t divulge that they supported him. If so, it could mean the polls actually underestimate Democrats.
“I was wondering if you could analyze the idea that 2024 Trump voters, due to disapproval of the president, are now saying they didn’t vote for him in 2024. If pollsters are still trying to anchor to a 2024 recall vote, this effect could erroneously make polling redder than it ends up being by missing voters who are switching preferences.” — Mario
The only thing I know for sure about recall vote is that it’s prone to error, so I am open to the theory. There’s also some evidence to support it: Pollsters who recontacted previously interviewed voters found that Trump ’24 supporters were less likely than Harris ’24 supporters to report that they backed him in 2024 when asked about it years later, especially if they disapproved of his performance.
That said, I would not take this theory to the bank. These analyses are based on people whom pollsters can successfully recontact, and historically the recall vote results look quite different among recontacted respondents than other polls taken at the same time.
One factor may be attrition bias, where pollsters can succeed only at recontacting the most politically engaged voters, who in recent years tend to be Democrats. If pollsters succeed at recontacting a relatively Democratic-leaning group of voters, the Trump voters they reach may be relatively likely to defect, making it seem he’s weaker than he is.
Turnout
The polls can also miss because of turnout, and there’s every indication that Democrats will benefit from some kind of strong turnout. How strong? That’s a tough question for pollsters to answer, but their results might depend on it.
There’s no way to know until Election Day whether the polls got it right, but many of you are still interested in whether the polls assume a strong turnout or not for Democrats.
In particular, many polls have shown Democrats doing even better among likely voters than among all registered voters, though there have been some exceptions:
“I would be curious to see the registered voter numbers in your polls if that’s possible, and if so, whether Dems are doing better with RVs, as Fox showed in their recent polls.” — Paul Frucht
If we take all of the Times/Siena polls from September and October, they show essentially zero gap between how registered and likely voters say they would vote for Congress.
That might be a bit surprising in this national political environment, when Democrats seem more energized to vote. What’s going on? The most important factor is that Democrats have made many of their gains among young, nonwhite and disengaged voters, who turn out at lower rates in midterm elections. As a consequence, Republicans have an advantage among regularly high-turnout voters relative to lower-turnout voters. That’s a reversal from 2024, when Mr. Trump and Republicans seemed to excel among disengaged voters.
Or put another way: The poll doesn’t really show any problem for Democrats among highly engaged voters; it’s just outweighed by the party’s colossal strength among rare voters or nonvoters.
That said, it’s entirely possible that Democrats enjoy a stronger turnout than the one supposed by our polling. The Times/Siena likely voter model blends people’s record of past voting with whether they say they will vote in the coming election. This year, Democrats have routinely done even better among people who say they’ll vote. In our last national poll, Democrats led by 14 points among voters who said they were at least very likely to vote or had already done so.
What else keeps us up at night?
We’ve already touched on nonresponse bias, recall vote and turnout as potential sources of error. But honestly, that’s not the biggest thing that concerns me right now.
“I am wondering what sorts of “fears” you may have in terms of polling misses.” — Cordelia
I think my single biggest “fear” is that the key Senate races are in solidly Republican states. As a consequence, the election will be decided by an asymmetric group of conflicted voters: self-identified Republican-leaning voters who voted for Mr. Trump but now disapprove of his performance and, at least as of our last poll, prefer a Democratic Senate candidate.
In our last Times/Siena polls of five red-state battlegrounds, Trump ’24 voters were more than twice as likely to be undecided as Harris voters. For good measure, about 9 percent of Mr. Trump’s supporters in the poll backed Democrats; by comparison, less than 3 percent of Ms. Harris’s supporters backed Republicans.
This gives Republicans an asymmetric opportunity to improve down the stretch, merely by bringing undecided or conflicted Trump voters back to their side. There’s absolutely no guarantee that they pull it off in this political environment and against these Democratic candidates. But it makes it easy to imagine how Republicans might outperform their polling numbers, even if today’s polls are perfectly representative.
The post Questions for a Pollster: What Are Your ‘Fears’ for the Midterms? appeared first on New York Times.
