AI can produce convincing answers, but a professor says the greater risk is when people stop questioning them.
Lucy Gill-Simmen, associate dean for Education and Student Experience at Royal Holloway, University of London, said the technology could make people less willing to question, verify, and investigate information on their own.
“Hallucinations are a visible problem because we notice them when an answer is wrong,” she told Business Insider of AI’s well-documented tendency to invent false information. “The deeper risk is that people stop asking how they know whether an answer is right.”
Gill-Simmen said knowledge has traditionally required effort — comparing sources, testing assumptions, and working through uncertainty. AI, by contrast, can produce a convincing answer without requiring users to go through that process.
“My concern is not simply misinformation,” she said. “It is that people become less inclined to verify, question, and investigate for themselves. In education, the struggle is often where learning happens. AI removes much of that struggle.”
AI as a substitute for thinking
Gill-Simmen’s concern echoes a growing body of research into what Wharton researchers have called “cognitive surrender” — a tendency to accept AI-generated answers rather than actively evaluate them.
Steven Shaw, a postdoctoral researcher in marketing at Wharton, warned about it in a report earlier this year, telling Business Insider that people risk becoming “passive followers of unthought thoughts” by adopting AI-generated ideas without fully processing them.
In three experiments involving 1,372 participants completing 9,593 reasoning tasks, he found that participants chose to consult an AI assistant on more than half of the tasks and, once they did, accepted its recommendations about 92.7% of the time when it was correct and 79.8% of the time even when it was intentionally wrong.
Similar concerns have emerged in education. Kimberley Hardcastle, a business and marketing professor at the UK’s Northumbria University, said AI could erode people’s ability to independently verify, challenge, and construct knowledge without algorithms.
Gill-Simmen described the phenomenon as “epistemic atrophy,” or “the gradual weakening of the habits through which knowledge is acquired.”
“When people work through a problem themselves, they develop mental models, strengthen reasoning skills, and build confidence in their own thinking,” she said.
“The issue isn’t that AI becomes a substitute for memory. It’s that it risks becoming a substitute for thinking.”
‘Illusion of understanding’
Gill-Simmen sees this firsthand among her students, who she said sometimes experience an “illusion of understanding.”
An AI-generated explanation can sound convincing, but when students are asked to explain the reasoning themselves or apply an idea in a new context, the depth of understanding is often much weaker, she said.
Wharton professor Ethan Mollick said that AI works best as a form of “co-intelligence” that helps people explore ideas and challenge assumptions, provided humans remain responsible for evaluating its output.
In a paper he coauthored with learning scientist and wife Lilach Mollick, he wrote that students should “critically assess and interrogate AI outputs, rather than passively accept them,” allowing AI to “serve as a supportive tool for their work, not a replacement.”
“Ironically, in a way, the more capable AI becomes, the more important these human capacities become,” Gill-Simmen said.
“In a world filled with fluent AI-generated outputs, success will depend less on producing information and more on evaluating it wisely,” she added.
Read the original article on Business Insider
The post The real danger of AI isn’t that it’s wrong — it’s that it could make us stop thinking for ourselves, a professor says appeared first on Business Insider.
