A recent Axios story on maternal health policy referenced “findings” that a majority of people trusted their doctors and nurses. On the surface, there’s nothing unusual about that. What wasn’t originally mentioned, however, was that these findings were made up.
Clicking through the links revealed (as did a subsequent editor’s note and clarification by Axios) that the public opinion poll was a computer simulation run by the artificial intelligence start-up Aaru. No people were involved in the creation of these opinions.
The practice Aaru used is called silicon sampling, and it’s suddenly everywhere. The idea behind silicon sampling is simple and tantalizing. Because large language models can generate responses that emulate human answers, polling companies see an opportunity to use A.I. agents to simulate survey responses at a small fraction of the cost and time required for traditional polling.
Phone polling has become exponentially harder. Web polling is too uncertain. Silicon sampling removes the messy, costly part of asking people what they think.
But this undermines the very idea of the opinion poll. Public opinion is used to guide policy, politics and social science, and it has value only insofar as it summarizes the beliefs and opinions of actual humans. Using simulations of human opinions in place of the real thing will only worsen our broken information ecosystem, and sow distrust. We should not turn to an artificial society to try to understand our real one.
The journalist Walter Lippmann, in his influential 1922 book “Public Opinion,” wrote that humans form “pictures in their heads” of the societies they live in. He called these pictures “fictions” and “pseudo-environments,” arguing that a democracy needed tools to fix those pictures, and that opinion polling could serve that role. Surveys would never be perfect, but Mr. Lippmann thought they were critical for getting us closer to an accurate sense of the will of the people.
But polling implementation has proved daunting over the years. To have a small margin of error, polling requires gathering responses from a large and accurate sample of the population. But pollsters have a hard time reaching people. Some people might be too busy to talk on the phone or fill out internet surveys. To make up for these issues, pollsters lean on statistical models to account for variables that can skew results.
That process is imperfect and messy. Let’s say a pollster wants to learn how many people in the United States are in favor of a certain policy measure, but the pollster ends up with a survey that includes 80 percent Republicans and only 20 percent Democrats. The pollster may think that in reality the country is closer to a 50-50 split, so the results are rebalanced to reflect that perceived reality. This means that the percentages you read as the results of polling are the output of the model, not numbers from the actual survey data.
The problem is that every model is designed with its own biases, because pollsters disagree about which variables deserve more weight. In 2016, The New York Times’s chief political analyst, Nate Cohn, ran an experiment in which he gave five pollsters the same election poll data. (That included Siena College, which conducts opinion polls for The Times and first acquired the data.)
Mr. Cohn found a 5 percent range of difference among what the five pollsters’ models returned. That range was larger than the margin of error typically associated with random sampling, meaning that the modeling assumptions were meaningfully skewing the results. This is alarming, because it suggests that pollsters can use modeling to nudge polls in a certain direction and influence public opinion itself, rather than merely to report what the public thinks.
Silicon sampling makes these problems worse. The computational whiz kids behind silicon sampling are so excited about A.I. that they will insist that their complex predictive computer simulations are accurate because they are trained on what’s been observed in the past — therefore, they excel at simulating human behavior in the present and predicting what’s next. However, prediction is not the point of polling. The point is gathering current opinion.
This method might sound absurd; we certainly think it is. And to make matters worse, there’s plenty of evidence that it doesn’t produce particularly reliable results. A recent study (that hasn’t been peer-reviewed yet) suggests that the biases that skew polls skew silicon sampling numbers even more strongly. The further from people we get, the more the simulation becomes a mirror of the pollster’s beliefs.
Nonetheless, the A.I. modelers are pushing ahead, and there is a lot of money behind them. Ipsos is working with Stanford University, it says, to “pioneer the use of A.I. and synthetic data in market and public opinion research” by creating digital twins — “virtual representations of real-world survey respondents.” Gallup has partnered with the silicon sampler Simile to create 1,000 A.I.-generated digital twins for clients. CVS (whose venture capital arm has invested in Simile) has also partnered with the start-up to “answer questions about its customers.”
The companies that offer this service are proliferating, with hundreds of millions of dollars in funding from some of the biggest firms in Silicon Valley. They promise “believable proxies of human behavior” for anyone who needs to check what people might think before acting. Market research is perhaps the largest sector in which silicon sampling will be used, since it will make starting a business that much cheaper.
If we do not slam the brakes on silicon sampling, we could see a significant undermining of trust in public opinion work and social science research more broadly. The results of such studies — like the fully simulated Aaru poll — are muddled opinions packaged as objective facts. It is unequivocally false to say that a majority of people trust their own doctors and nurses on the basis of an A.I. survey.
What happens when these surveys tell us relative levels of support in the open Democratic nomination field for 2028? To wit, Aaru ran a full simulation of the U.S. 2024 presidential election on the eve of Election Day; Kamala Harris won, narrowly.
Pure fictions are on the brink of being treated as scientific and political knowledge. If we do not pull back, our understanding of society might become artificial, too.
Leif Weatherby (@leifweatherby) is the author of “Language Machines.” Benjamin Recht regularly blogs at argmin.net and is the author of “The Irrational Decision.”
The Times is committed to publishing a diversity of letters to the editor. We’d like to hear what you think about this or any of our articles. Here are some tips. And here’s our email: [email protected].
Follow the New York Times Opinion section on Facebook, Instagram, TikTok, Bluesky, WhatsApp and Threads.
The post It’s Called Silicon Sampling, and It’s Going to Ruin Public Opinion Polling appeared first on New York Times.
