During an interview in late April with Dr. Phil, Robert F. Kennedy Jr. reiterated his appeal to parents on vaccine safety: “We live in a democracy, and part of the responsibility of being a parent is to do your own research.”
The U.S. health secretary has also announced his own investigation, pledging to find an answer to the autism “epidemic” by September. It’s an ambitious goal. It’s also a realistic one but only if he already has an answer in mind.
To tell the story you want with statistics, you don’t have to lie or fabricate data — though that happens, too. More often, statistics are manipulated, figures massaged and results skewed through subtler means. Sometimes, it’s sloppiness or unconscious bias at work. Other times, the distortion is deliberate.
Whether the numbers attempt to tell a story about the economy, immigration, education or public health, we should empower ourselves to recognize the deception.
Vaccine data are far from immune to statistical trickery and its consequences.
Not only might individuals skip a vaccine and get unnecessarily sick, but the viral spread of misinformation can poke holes in the herd immunity needed to protect a population. One new, untampered statistic tells a chilling story: A meager 10% drop from today’s already dangerously low measles vaccination rates could spark an estimated 13-fold increase in annual cases.
Statistics wield incredible power. I developed a deep respect for them during my first career as a biostatistician. Today, as a journalist, I see numbers leveraged for good and for bad. I’ve seen them help the public and policymakers interpret complex data, detect patterns and make better decisions — evidenced in my reporting on data dashboards during the COVID-19 pandemic. I’ve also seen data withheld and statistics doctored for less-than-noble aims by chemical companies, the gun industry, police departments, the U.S. military, climate change deniers and vaccine skeptics, to name a few.
If left unaware of the deceit, the public can’t hold these groups accountable. And if citizens base their votes and other decisions — like whether to vaccinate their child — on distorted or false information, our democracy and our health lose again.
Fortunately, inoculation against misinformation is available. As Kennedy and his collaborators dig into vaccine and autism data, as measles cases mount, and as you “do your own research” or simply digest your news and social feeds, here are five red flags to watch for.
Chance
The infamous paper that launched the vaccine-autism controversy was based on just 12 children. Its author claimed that eight showed signs of developmental regression after receiving the measles-mumps-rubella vaccine. The study was later retracted for scientific misconduct. But even without fraud, the sample size should raise alarm. Chance alone could explain such a small cluster of cases. Contrast that with rigorous studies — like one in Denmark with more than 650,000 participants — that consistently find no relationship between the MMR vaccine and autism.
We should be just as wary when studies test a grab bag of possible outcomes. Suppose researchers ask whether a vaccine causes heart disease, diabetes, any of a dozen types of cancer or any of five neurodevelopmental disorders. Even if the vaccine is in reality not affecting any of those 20 outcomes, when researchers try to study so many things all at once, statistical noise can mean one may erroneously appear “significant” just by chance. A more rigorous and targeted study would be far less likely to give that false positive.
Count quality
Big numbers can impress. But quality counts. In 2021, the Delphi-Facebook survey estimated near real-time COVID-19 vaccine uptake using weekly responses from around 250,000 people. On paper, the large sample size conveyed statistical confidence. But in practice, the data missed the mark. The sample was biased and unrepresentative of the overall population. By late May, the study had overestimated vaccine uptake by a wide margin — 70% compared with the true rate of 53%. That inflated figure may have lulled the public and policymakers into a false sense of security.
Beware, too, of the misuse of raw data. Figures from the Vaccine Adverse Event Reporting System appear in many papers and posts asserting vaccine harms. But this system was set up only as an early warning system. Anyone can submit a report on a suspected reaction. If a hint of a pattern emerges, then researchers will investigate to determine if the signal represents an actual risk. As its own website warns, the initial reports may be “incomplete, inaccurate, coincidental, or unverifiable.” People may be apt to connect an event that occurs shortly after vaccination with the shot itself, for example, especially if they personally fear the safety of vaccines. To demonstrate the system’s fallibility, a doctor filed a report saying he turned into the Incredible Hulk after receiving a flu vaccine. The entry was initially accepted into the database.
Cherry-picking
One study circulating in the anti-vax community was led by David Geier, the same figure tapped by Kennedy to head his federal autism and vaccine investigation. The study found a connection between autism and vaccines containing the preservative thimerosal. But it hinges on a critical flaw: Cases of autism and the comparison group came from different time periods. Because vaccination rates changed dramatically over time, the design introduced a spurious association.
Among myriad ways to manufacture a desired conclusion is the strategic choice of time frame, analysis method or how the data are presented. By plotting only convenient variables or truncating inconvenient values, for example, you can tell the story of your choosing. One COVID-era graph appeared to show that vaccines did not prevent deaths. The trick? It compared vaccine uptake with cumulative deaths — a number that can only rise over time, and so of course would broadly move in the same direction as the uptake rate of a desperately needed new vaccine that the public is clamoring for.
Another sleight of hand to play down the size of a problem: Acknowledge a not-so-unusual number of outbreaks while ignoring how large or how deadly those outbreaks were, just as Kennedy did in February with measles.
Correlation vs. causation
A widely shared study recently referenced by Kennedy reports a link between vaccination and neurodevelopmental disorders among 9-year-olds in Florida. This one, too, is riddled with problems — namely, its failure to account for other factors that could explain the results. Children whose parents more regularly use the healthcare system, for example, are more likely to get both vaccinated and diagnosed. Healthcare engagement confounds the relationship. So, we can’t say the vaccine caused neurodevelopmental disorders any more than we could say that increased consumption of margarine resulted in a higher divorce rate in Maine. These are cases of correlation, not causation.
Something similar and even more interesting cropped up when people compared death rates by COVID-19 vaccination status. At first glance, an unexpected pattern emerged: The vaccinated were dying at about twice the rate of the unvaccinated. The catch here? The analysis didn’t account for age. Older people were more likely both to die and to get vaccinated. Once researchers broke the data down into age groups, a more accurate — and reverse — picture emerged: The unvaccinated were dying at higher rates.
Context and conflicts
Talk of an uptick in autism diagnoses often skips crucial context: expanded awareness, broader diagnostic criteria and financial incentives for diagnosis. There could well be a surge in the number of cases without any surge in the true incidence of the disorder.
Also, discussions motivated by a desire to explain autism or to oppose vaccines tend to omit the robust studies that have debunked any link between vaccines and autism — because those would be unhelpful to the agendas. Vaccine opponents may further ignore the glaring conflicts of interest behind many of the studies still pushing that autism narrative. Geier had a study retracted, in part, for not disclosing his involvement in vaccine-related litigation.
Conflicts of interest surround Kennedy as well. He has spent years pushing anti-vaccine claims despite overwhelming evidence of vaccine safety and despite not being a doctor or a scientist. Now that he is in a position of authority over public health, he should at least be held to the same ethical standards as a scientist. Modern scientific practice calls for statisticians to specify their hypotheses and analysis plans before data are collected. This ensures transparency and objectivity, and reduces the risk of data dredging and misleading results. Statisticians follow where the data lead rather than mold or seek out data to fit a predetermined narrative.
Kennedy’s team appears to be following a different playbook. According to a former top vaccine official, Kennedy’s team requested a wish list of data seemingly to justify their autism theory: The team asked for cases of brain swelling and deaths caused by the measles vaccine. The official said there are no such cases. Someone who keeps hunting for evidence to back up his discredited theory is not conducting science.
Our stories should be malleable. Our statistics should not.
Lynne Peeples, a science writer, is the author of “The Inner Clock: Living in Sync With Our Circadian Rhythms.”
The post Contributor: On autism and vaccines, there are lies, damned lies and statistics appeared first on Los Angeles Times.
