This week’s Pollwatcher answers a reader’s question about how polls select respondents and highlights how partisans in one state have flipped on a Democratic senator. But first…
Are people honest when taking polls? I often see claims that people regularly lie to pollsters, a claim that’s difficult to prove or disprove. Still, the idea that people are willing and able to report their actual opinions is a core assumption for public opinion polls to work.
There are examples of “social desirability” bias in surveys on certain topics: People overreport voting in elections and underreport drug use. The “Bradley Effect” theorized that White voters overreported support for Black candidates in 1980s elections, although this pattern didn’t repeat in either of Barack Obama’s presidential runs.
But what about standard opinion questions like “what is the most important issue facing the country?” Do people report which issues really matter to them?
Want this in your inbox?
Sign up for The Pollwatcher newsletter here.
The answer may be in our fingertips and eyeballs, according to a recent study published in Public Opinion Quarterly, a top journal. A trio of researchers at Université Laval in Quebec explained that when people experience strong emotions, they can also have physical reactions. Sweating is one indicator of emotional response, as is how much our pupils dilate when reacting to something we see.
How they did it
In a dimly lit room, researchers showed 116 Quebec adults a variety of words associated with five issues, including the economy, environment, health, immigration and culture. As they answered short questions about each word, a sensor measured how quickly electricity flowed through their index and middle fingers — a proxy for sweat — while another sensor tracked changes in the diameter of their pupils. Participants also completed a typical survey asking about the importance of the same issues.
Co-author of the study Camille Tremblay-Antoine described the study’s core question in an email:
When we ask people questions and collect their answers, are those answers actually valid? Would we get the same results if those same people came into a laboratory and we measured their responses with a lie detector?
‘The short answer is: yes.’
That’s the basic conclusion from Tremblay-Antoine. The study found strong positive correlations, about 0.7 out of a maximum of 1, between how important people rated issues and their both sweat rates and pupil sizes. The results were statistically significant despite the study’s relative small sample.
Not all survey questions performed equally, the study found. Questions that forced respondents to choose their most important issue were more closely tied to physical reactions than those that allowed respondents to rate multiple issues as important.
But overall, the results were encouraging, Tremblay-Antoine wrote. “When we ask people what matters to them or what they prefer, we can have real confidence that their answers reflect their genuine priorities. It may seem obvious, but it had never actually been tested this way.”
Top-issue polls face other challenges
While the novel Canadian study suggests people can tell pollsters which issues they feel passionately about, measuring and interpreting the public’s biggest concerns can be difficult.
Top-issue polls sometimes show differing results depending on which issues are listed. A May Quinnipiac University poll found 26 percent of voters chose “preserving democracy” as the nation’s most urgent issue, but a Gallup poll asking an open-ended question found 2 percent of adults naming elections or democracy as the country’s top problem. Open-ended questions rely mainly on issues at the top of voters’ minds, but can miss other issues that they have strong feelings about.
In addition, the connection to how people vote is shaky. One study by Thomas J. Leeper and Joshua Robison analyzed 48 issues across presidential elections from 1980 to 2008 to see whether voters who said an issue was important based their vote on those issues. The results were inconsistent: issue importance mattered in some cases, but in many other cases it did not.
Your questions!
Thanks to the more than 50 readers who sent in questions or ideas about polling! I look forward to writing about all of them in the coming months, but Cynthia’s query seems perfect to start with:
Q: I have often wondered about the legitimacy of polls because I don’t know who or how people are selected to participate. So
1. How are people selected?
2. How can I participate?
A: There are two main sampling approaches. Random or probability-based samples start with a “sampling frame” that includes everyone in the target population, such as the U.S. Postal Service’s address file or a state’s list of registered voters. A poll contacts a random sample from that frame using one or more methods, such as by phone, text, email or phone. Actual samples are more complex to overcome the fact that the frame is usually missing contact information for many people, or that certain types of people are more likely to respond.
Non-probability or opt-in online samples are the second main way polls recruit respondents, using online ads that offer rewards or payment for taking surveys.
While some polls recruit respondents for a single interview, survey panels that recruit respondents for multiple surveys are increasingly common. Some panels recruit panel members using non-probability methods, while others use random sampling.
2. How can I participate?
While anyone can sign up to take online surveys through an opt-in survey company, there’s no way to volunteer for surveys conducted using random sampling. Survey methods have changed dramatically since the early 2000s, with a massive growth in opt-in online surveys and decline in live-caller phone polls, as documented by the Pew Research Center.
In a future newsletter, I’ll look at how different methods compare in accuracy.
Have a question about polling or public opinion? Send me an email at [email protected]
Worth your attention
A Democratic senator more popular with Republicans — Pennsylvania Sen. John Fetterman is a Democrat, but you wouldn’t know it from a New York Times/Philadelphia Inquirer/Siena College poll released this week.
Just 22 percent of Democratic registered voters rated Fetterman favorably, while 73 percent of Republicans rated him favorably. Those results mark a turnabout since 2024, when 74 percent of Democrats had a positive view of Fetterman compared with 25 percent of Democrats.
The Times noted that Fetterman has frequently appeared in conservative news media criticizing his fellow Democrats as too liberal, “turning himself into something of a modern-day political wrestling heel.”
The latest poll was conducted Aug. 17-21 among a random sample of 760 Pennsylvania registered voters sampled from a statewide voter list, with live interviewers calling cellphones and landlines. The margin of sampling error was plus/minus-4.5 percentage points.
What else do you want to know?
Send your questions, comments and ideas to [email protected].
The post We know you care about the environment because you’re sweating appeared first on Washington Post.
