When preparing for the first New York Times/Siena poll of this year’s U.S. Senate race in Maine, we were faced with a common challenge in survey design: How to ask about sensitive topics without biasing the results of the poll.
Asking voters about controversies is a delicate task. Our goal is to measure the voters’ opinions as they existed before we called them. For that reason, we try to limit the amount of new information we expose voters to during a survey. But it’s equally important that voters understand exactly what they are being asked about.
An indelicate question can carry even bigger risks in a charged political climate. Such a question may even cause respondents to hang up, which ends their interviews and potentially biases the overall results from the poll.
In the Maine Senate race, there were a number of sensitive topics that we thought might have the power to affect voters’ decisions. Graham Platner, the Democratic nominee, has been the subject of numerous controversies. Among them are questions about his past treatment of women, a tattoo that resembled a Nazi symbol, which he has had covered, and social media posts written years ago that many found offensive.
We first asked how aware people were of the stories. In the case of Mr. Platner, nearly 80 percent of voters in Maine had heard “a lot” about the various controversies.
That is a larger share than is typical for most news events. For example, in a 2024 national Times/Siena survey, only 26 percent of registered voters said they were paying “a lot” of attention to news stories about Donald J. Trump’s numerous legal cases.
Choosing the exact language for the Maine race was a balancing act between specificity and the desire not to bias respondents. One option we considered was describing the text messages sent by Mr. Platner as “sexual text messages sent to multiple women while married,” but we worried that could introduce bias to respondents by creating an environment where they felt pressured to respond negatively. Another version described the communications only as “text messages,” but we worried that was too vague and could be interpreted too broadly.
We concluded the best option was a middle ground that provided clarity with minimal opportunities for introducing bias, describing the communications as “text messages he exchanged with women while married.”
After establishing whether respondents had heard about those stories, we asked those who had whether the stories affected their support for him.
This language went through several revisions to ensure we were avoiding a phenomenon in survey research known as “expressive responding,” in which voters may give a response that aligns with their identity — for example, their political party — even if it isn’t what they truly believe.
We also considered asking separate questions about each of the individual controversies. We decided to ask one question that combined all three because we worried it might be difficult for respondents to disentangle concerns about one controversy from another. Put another way, if you are questioning your support for Mr. Platner, it may become difficult to separate whether that is because of his social media posts or his tattoo.
In the end, we asked respondents to tell us what impact the controversies had cumulatively: “They mean that I cannot support him, they make me question whether I can support him, or they have not impacted my support for him one way or another.”
In the poll, nearly 30 percent of Mr. Platner’s current backers reported that the stories made them question their support. At the same time, they still plan to vote for him. Just how deeply they are questioning their support goes beyond what we can answer in this poll.
Independent pollsters like The Times/Siena Poll approach these situations very differently from campaign pollsters, who use their polls to track how voters’ views shift when presented with new information while being interviewed for the poll — an approach sometimes known as message testing.
After asking respondents who they plan to support in the upcoming election, campaign pollsters often provide respondents with both positive and negative messages.
The pollsters will then ask a second time who voters plan to support. This technique allows the campaigns to assess the best way to present policy ideas or attack their opponent.
The revised results are sometimes called the “informed ballot.” These kinds of results are not eligible to be included in The Times’s polling trackers because they do not reflect bona fide public opinion.
Alternatively, partisan pollsters may ask questions that subtly encourage respondents to answer in a particular way, yielding results that may be more favorable to their cause. This is sometimes called push polling. These polls can be selectively released in order to feed a narrative that is favorable or hurtful to a campaign.
For independent pollsters like The Times/Siena Poll, these ideas go directly against our goal of measuring public opinion, not shaping it.
Our approach, in which we ask those who are already aware of information how it affected their support, has challenges as well. People are not always good at articulating how their opinion might change, particularly after they have already voiced their support or opposition to a candidate.
And even with all our efforts to be precise in our language, once we start interviewing voters, we still occasionally find cases where the question is misinterpreted. In the Maine survey, we asked whether voters thought Senator Susan Collins, the Republican nominee, was “too old to be an effective senator” and whether Mr. Platner was “too inexperienced to be an effective senator.”
We wanted to avoid a well-known problem in survey research in which questions about whether a respondent agrees or disagrees with a statement can cause people to reflexively say they agree. So we made the initial versions of those questions part of a series of similar questions asking whether they thought traits described each candidate “very well, somewhat well, not too well or not at all well.”
While the poll was being conducted, a review of recordings of the interviews highlighted that respondents were not always responding to the full wording of the question. Instead, based on recordings of their interviews, it was clear that many respondents were reacting to only a part of it, such as whether Ms. Collins was simply “old.”
So we changed the wording to ask whether respondents simply agreed or disagreed, which despite its drawbacks is easier to comprehend. While the number of respondents who completed the survey after that change was small, people were slightly less likely to say that Ms. Collins was too old or that Mr. Platner was too inexperienced — although significant shares of voters, including some of their own supporters, still agreed.
The post How Our Polling Team Writes Sensitive Questions on Controversies appeared first on New York Times.
