Chatbots fueled by artificial intelligence bear disclaimers saying they cannot dispense medical advice or diagnose conditions, but they still field millions of queries from users who are sick, trying to decipher medical records or understand their treatment options.
Nearly 1 in 3 Americans have turned to the bots for health information, according to a recent survey by KFF, a nonpartisan health policy and education organization.
AI companies are rushing to meet this demand, including Anthropic’s Claude for Healthcare and OpenAI’s ChatGPT Health, advertising more robust privacy protections and features to connect to medical records, while warning their products cannot replace doctors.
“They can be plausible-sounding, completely confident-sounding and completely wrong at the same time,” said Girish Nadkarni, a professor of medicine at the Icahn School of Medicine at Mount Sinai and co-author of an independent evaluation of ChatGPT Health.
Nadkarni’s research found the chatbot failed to tell users to go to the emergency department in more than half of cases of impending respiratory failure and serious diabetic complications, instead advising them to stay home and monitor symptoms. The findings comport with other studies and reviews that have given AI health advice failing grades or at least urged caution.
OpenAI has raised questions about the methodology of the study and whether it properly reflects how people actually use its chatbot. A spokesperson for the company said ChatGPT Health models are continuously trained and updated through feedback from physicians. (The Washington Post has partnerships with OpenAI and Perplexity.)
“People come to ChatGPT with health questions every day,” said Drew Pusateri, a spokesperson for OpenAI. “We want those interactions to leave them better informed and improve their conversations with medical providers. But it’s not designed to replace medical professionals and should never be used as a substitute for care.”
Despite growing evidence of potential harms, the allure of instant, comprehensive and personalized answers is too strong for many to overcome in a health care system plagued by high costs and inconsistency of access.
The Post previously scored how chatbots performed for medical use and examined what studies have found. Here are the stories of six Americans who turned to chatbots for health advice, and what experts say about their situations:
A free alternative
When Genaro Diaz, a 20-year-old junior at the University of Illinois at Urbana-Champaign, battled serious stomach pains and vomited mucus in early February, he wanted to see a health care provider. He turned to ChatGPT because a lapse in his student health insurance plan would have meant a $150 bill. The chatbot advised him to take Pepto-Bismol and Pedialyte, and see a doctor if symptoms didn’t improve after three days. Diaz never figured out why he was sick, but he later felt better.
“As a college student, I don’t really have a source of income, I don’t have a job,” Diaz said.
In the KFF poll, 1 in 5 users say difficulty affording health care was a major reason they turned to AI for health advice. That number rose to about 1 in 3 for people ages 18 to 29 and with annual incomes below $40,000.
Ashwin Ramaswamy, a medical professor at Mount Sinai, understands why many patients are apprehensive about medical bills. But he questioned ChatGPT telling Diaz to wait to see a doctor because the symptoms he described were serious, even if he recovered.
“I can’t rule out something being dangerous,” said Ramaswamy, a co-author on the ChatGPT Health evaluation. “If I got that call, I would not know, but I would surely know I would want to see him as soon as possible.”
When doctors are hard to find
Gayle Madeira, 57, of New York, said her quest to find a doctor to treat long covid was an ordeal. One doctor’s office listed by the city’s 311 line for help with the disease told her the office was not given the resources to function as a long-covid clinic.
ChatGPT and other AI platforms filled the void for her. She uploads detailed spreadsheets of her medical history and medication use — helpful when she is experiencing brain fog and fatigue. Conferring with AI has helped her to rule out pursuing treatments that have proliferated in online long-covid communities because the condition baffles medical experts. She won’t take medication without the supervision of a physician.
“I want doctors to be in control of my care. I don’t want to have to use AI,” said Madeira, an artist and dancer who used to work on Wall Street.
Some experts in artificial intelligence caution that uploading detailed medical information can undermine privacy, even when the companies advertise privacy protections.
“AI is collecting a lot of information about you that can create and influence targeted information,” said Pat Pataranutaporn, who researches how people use AI as an assistant professor of media arts and sciences at MIT. “If you are not buying a service, then you are the product.”
Shaping informed questions
Gary Cohen, 70, believes he receives “amazing care” from the University of California at San Francisco for his multiple myeloma diagnosed in 2020. He turned to Claude to prepare for his appointments, such as asking for explanations in plain language of a second-line treatment his doctor proposed or the significance of test results.
When he spotted one worrying result in a patient portal, he uploaded it to Claude, unsure whether his doctor would get back to him on a weekend. Claude explained in detail how the result was worth monitoring but not worrying over. His doctor essentially said the same thing.
“You just don’t feel helpless,” said Cohen, who previously worked in President Barack Obama’s administration helping to implement the Affordable Care Act. “You feel you are part of a conversation about making a decision, and there are lots of decisions being made.”
Jonathan Chen, an associate professor of medicine at Stanford University who researches AI, said chatbots can be effective for crafting questions and summarizing medical records ahead of appointments. Patients should ask for counterpoints, and the risks and benefits of different medical options.
But “when the stakes are high, you have to double-check with real people,” Chen said.
He also cautioned against using the bots to analyze the original medical scan images rather than the summary documents because these tools are not ready for “meaningful medical interpretation.”
Inaccurate information
Randall Seefeldt, 67, of Indiana, went to the hospital after he consulted with Google’s Gemini, and it advised him to go to the emergency room for a CT scan when he had chronic stomach pain. The scan found nothing. Instead of feeling annoyed, Seefeldt appreciated the peace of mind because his father died of a rare cancer that may be hereditary with similar symptoms.
“The results from Gemini gave me a path forward that I was comfortable with, and that alleviated my stress,” said Seefeldt, who works in software for public health agencies.
Pataranutaporn’s research has shown that people trust inaccurate information from chatbots more than accurate information from human doctors, mirroring well-established research showing that people often trust technology more than real people. Confident, inaccurate reasoning by chatbots can be more convincing than accurate human reasoning.
Researchers who have flagged the dangers of AI for consumer health say it’s important to acknowledge that in many, even most cases, the chatbots do give sound advice.
“A lot of things it did well. It didn’t completely bomb,” said Ramaswamy, speaking of his work on the ChatGPT Health evaluation. “The one thing you can’t do with an LLM is you can’t pull them outside after and say, ‘What were you thinking?’” he said, referring to the large language models that power these AI tools.
A caring voice
When his wife was hospitalized for two weeks for a massive pulmonary embolism, John Hope found Perplexity’s empathy striking. He used the AI “answer” engine to translate terms doctors used and to prepare for her return to their retirement community in Elizabethtown, Pennsylvania. He had caring friends he could turn to, but in a hectic time, it was nice to receive an immediate answer with “genuine compassion” and suggestions for what to ask doctors.
“It sometimes feels a little scary to be having such a meaningful conversation with a nonhuman, but it’s also reassuring,” said Hope, who is 85 and writes for a newsletter on Food and Drug Administration issues. “Perplexity in particular just seems to do that for me: I understand what you’re going through, I’m here to help if you can tell me what the doctor said.”
The kindness of chatbots is an illusion that can veer into dangerous sycophancy, experts on the technology caution. The study on ChatGPT Health, mirroring other research, found that the chatbot was more likely to give bad advice when the user said they weren’t too worried about the symptoms.
“Caring is a human endeavor,” said Azra Ismail, an assistant professor in biomedical informatics and global health at Emory University who has studied the use of chatbots to navigate reproductive and sexual health. “It can maybe offer some level of informational support, some level of performative empathy, but it’s really not a system that can have care.”
Perplexity said in a statement that it prioritizes accuracy and aggressively removes sycophancy. The company said it formed the Perplexity Health Advisory Board to help build safeguards, standards and product experiences people should expect from AI.
“We have heard from some users that Perplexity’s neutral tone and clear sourcing provide comfort during stressful decision-making, but we do not believe AI should be anthropomorphized,” the company said.
The allure of fast answers
Christina Squitieri had to wait three weeks for an appointment with her gastroenterologist to discuss the red-highlighted results of biopsies taken during a colonoscopy. ChatGPT told her the increase in white blood cells flagged in the report revealed microscopic colitis — an inflammatory bowel disease — and offered information on medication and its side effects. But when she finally saw the doctor, he dismissed that finding as irrelevant and reassured her that the report instead offered conclusive evidence she did not have the disease.
But if Squitieri, 37, of New York, finds herself in the same situation again — a worrying test result, a struggle to get in touch with a doctor — she probably would turn to ChatGPT again, even if she would take the answer with a grain of salt. When her daughter became sick at 20 months, it took 10 months to see a specialist and get diagnosed with a chronic condition that required medication.
“We should not be seeking out a chatbot that does not think because getting a doctor is so impossible,” said Squitieri, who previously worked for a beauty education company.
The KFF poll found that the desire for quick and immediate advice was the top reason people gave for using chatbots for health, cited by two-thirds of users as a major reason. About 1 in 5 cited not having a health care provider or being able to get an appointment as a major reason.
“There is a reason these chatbots have a place: Access is poor, it can take weeks if not months to get an appointment,” said Nadkarni, the Mount Sinai researcher. “If people are going to be using these tools at scale, they should be held to a higher standard of rigor and safety than chatbots we use to write emails.”
The post 1 in 3 Americans use chatbots for health advice. These 6 patients explain why. appeared first on Washington Post.
