Four major generative AI chatbots – OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini and Perplexity AI – did a shoddy job of summarizing the news, a study by the BBC found.
The organization fed content from its website to the services and then asked them questions about the news. More than half of the answers contained “significant inaccuracies” and distortions, the BBC said.
Examples of errors included Gemini incorrectly saying the UK’s National Health Service did not recommend vaping as an aid to quit smoking and ChatGPT and Copilot both wrongly indicating that politicians Rishi Sunak and Nicola Sturgeon were still in office. Perplexity also misquoted a BBC News report about the Middle East, saying Iran showed “restraint” and calling Israel’s actions “aggressive” – adjectives that did not appear in the initial news report.
“The price of AI’s extraordinary benefits must not be a world where people searching for answers are served distorted, defective content that presents itself as fact,” BBC News and Current Affairs CEO Deborah Turness wrote in a blog post. “In what can feel like a chaotic world, it surely cannot be right that consumers seeking clarity are met with yet more confusion.”
Watch on Deadline
Turness, who was president of NBC News from 2013 to 2017, noted that the BBC is pursuing a number of AI initiatives and holding talks with tech companies to develop new automated tools. But she called on tech giants to pull back on alerts, posing a stark question: “We live in troubled times, and how long will it be before an AI-distorted headline causes significant real world harm?”
The findings come nearly a month after Apple stopped sending news alerts powered by AI after several of them proved to be wildly inaccurate.
Journalists and news organizations told the tech giant about a number of gaffes. One alert generated by Apple from BBC News falsely told readers that Luigi Mangione, the man accused of fatally shooting the CEO of United Healthcare, had shot himself. (He was arrested and is in jail awaiting trial.) Another falsely told some BBC Sport app users that tennis star Rafael Nadal had come out as gay.
Turness saluted Apple’s “bold and responsible decision” to suspend the alerts given the “high stakes” of the effort to get things right. “The companies developing Gen AI tools are playing with fire,” she declared.
In the study, the BBC asked journalists and category experts to evaluate the answers from the chatbots. They determined that 51% of all AI-generated answers had flaws, with 19% of responses citing BBC content actually introducing errors including incorrect statements of facts, numbers and dates.
Max Goldbart contributed to this report.
The post AI Chatbots Have “Significant Inaccuracies” When Summarizing News, BBC Says; Top Exec Deborah Turness Says Tech Firms Are “Playing With Fire” appeared first on Deadline.
