AI Advice Kills Critical Thinking: Study Finds People More Confident but Less Accurate

Academics from French and Italian universities found that AI advice suppresses critical thinking, leading people to confidently repeat incorrect information provided by AI systems. The research was led by Valerio Capraro, associate professor at the University of Milano-Bicocca, alongside co-authors Chiara Marcoccia of École Normale Supérieure and Walter Quattrociocchi of Sapienza University of Rome.

The study's title encapsulates its findings: "AI advice suppresses people's willingness to say 'I don't know', even when the advice is wrong and accuracy is incentivized." The researchers designed questions around visual details in films — such as the colour of a team's uniform in Bend It Like Beckham or the vehicle a character drives in Like a Cat on a Highway — areas where large language models typically fail.

The experiment used Step 3.5 Flash, a model that was usually wrong, so that any reduction in judgment could not be attributed to sensible reliance on a reliable tool. Participants were split into two groups: one answering without AI assistance, the other with access to AI advice.

In the baseline group, 44% of participants admitted they did not know the answer. With AI advice available, that figure collapsed to just 3%. Accuracy also fell sharply — from 27% in the baseline to only 9% when AI was used.

Despite the drop in accuracy, confidence surged. The baseline confidence level stood at 30%, but with AI assistance it rose to 76%. "Basically people became much worse — the accuracy was only one third — but they were twice as confident," Capraro said.

When monetary incentives were introduced, results improved marginally. Willingness to admit ignorance rose from 3% to 8%, and accuracy climbed from 9% to 16%, though both figures remained well below the baselines of 44% and 27% respectively.

"For humans, the capacity to say, 'I don't know,' is very important because it represents the recognition of the limits of our own knowledge," Capraro said. "But now with AI, we can get an easy answer to virtually every question, so we wondered whether this would interfere with human capacity to say, 'I don't know,' to suspend judgment."

Capraro said the issue must be addressed at a societal level through AI literacy and education policy. "A much more promising approach would be at the educational level," he said, adding that while model providers should try to help, their incentives may not align with solving the problem.

"I'm very much concerned for children, because adults have learned critical thinking. But for children who basically are born with these systems, the risk is that they don't even learn the basic critical skills," Capraro said. The researchers contend their findings can be generalised beyond film trivia to other domains.