Will the ubiquity of chatbots lead to a generation of doctors who don’t know their stuff? That’s the danger, according to a new editorial in BMJ Evidence-Based Medicine.
The authors, led by Jacob Hough at the University of Missouri, outline the many ways that AI use can undermine medical education, like automation bias, de-skilling, and providing false information.
“These tools can fabricate sources, encode bias, lead to over-reliance and have negatively disruptive effects on the educational journey. Medical programmes must be vigilant about these risks and adjust their curricula and training programmes to stay ahead of them and mitigate their likelihood,” they write.
Two-thirds of doctors have already incorporated AI into their practice. In some cases, AI programs may increase diagnostic accuracy and help identify the correct treatment. Yet critics warn that these programs are causing even experienced doctors to lose critical skills. Although proponents suggest that doctors are well-placed to use their own expertise to double-check AI outputs, research shows otherwise: doctors who use AI tend to over-rely on its outputs without realizing it, leading to misinterpretation. This is known as “automation bias” and “cognitive offloading.”
This may be the most pernicious problem with AI use: unlike other tools, where it is easy to tell when you are using the tool versus using your own mental faculties, research shows that AI users often believe they did all the work despite just repeating AI outputs verbatim. This was most tellingly depicted in a recent MIT study. The researchers found that students who used AI to generate an essay believed it was their own work, despite not being able to remember a single quote from the essay immediately afterward. This was in sharp contrast to those who used their own skills to write the essay, who had no problem remembering the words they’d written.
In a best-case scenario—where AI improves diagnostic accuracy—the role of doctors will be to intervene when their human expertise suggests the AI might have erred. But the research on automation bias and cognitive offloading suggests that doctors will be unlikely to make this judgment against the machine. Instead, they are more likely to believe they are providing oversight while actually just trusting the AI’s outputs—especially in a busy, complex situation like a hospital, where it is already easy to miss something.
Worse, if trainees are dependent on AI results, they will never learn the skills that will supposedly allow them to question or contradict the AI if needed.
The researchers note that students are already abusing AI tools to generate their answers to homework and write their papers. Their suggestions: Medical schools must include in-person exams in an AI-free environment. Homework will have to be graded on showing process, not just the answer, to ensure it is not AI-generated. And students need to learn about AI so that they understand where these models can fail. Finally, educators must teach critical thinking in AI contexts—such as providing examples where the AI is partially wrong so that students can learn to question and double-check the chatbot.
Still, the researchers stop short of suggesting that AI use be limited in medical contexts, arguing that AI has “documented and well-researched benefits” (although they do not cite any such sources).
At the same time, evidence on the risks of AI use continues to pile up. A study in JAMA Network Open just found that chatbot use was linked with increased depression. One large healthcare provider is facing lawsuits for allegedly using AI to record patient interactions without consent. It’s easy to get AI chatbots to deliver dangerous lies about health information. And LLMs have become notorious for generating falsehoods, including fake citations and made-up studies in important contexts like government healthcare reports and legal settings like courtrooms. They exacerbate biases, which can be especially problematic in healthcare and law. And they’re being sold as therapists to millions, all while encouraging delusional thinking and helping plan suicide.
****
Hough, J., Culley, N., Erganian, C., & Alahdab, F. (2025). Potential risks of GenAI on medical education. BMJ Evidence-Based Medicine, 30, 406-408. (Link)
Bullshit. What does this have to do with the preponderance clueless dolts already working in modern medicine?
Report comment
AI is not good for humanity. But I do agree, neither is psychiatry’s Ponzi scheme of an iatrogenic illness creation system.
Report comment
I like AI. I’ve found it more intuitive than most medical doctors, and certainly most therapists!
Report comment
AI is with us and we need to figure out how to maximise the benefits and minimise the risks. I teach medical students and this is a subject we are now grappling with – returning to internet free exams for multiple choice questions, trying for paper based but one of our students worked out 1.5 million pages of paper would be required for four year levels who have four tests a year , due to students wanting to bring their notes! Should be a way to have notes on a lap top but block the internet -Iran is blocking a whole countries internet access after all!!!
Report comment
“Why 2026 Will Expose Your Nervous System”, from Philip | The Abundance Alchemist @abundancealchemist
Report comment
Thanks for this and other articles on AI/LLMs, a technological turning point for human cognition, as raised with your report of research on ChatGPT’s impairment of thinking ability, for example. There may be pros and cons, things gained and things lost, with any technology, but outsourcing cognitive skills to computers generally seems a bad idea for having ideas of one’s own.
But then we don’t live in a society where thinking for ourselves has ever been that particularly valued, unless perhaps it pays off for owners of knowledge industries and cognitive labor serving power relations. That’s why most all of us are constantly trained in schools and jobs to do our parts as cogs in the machinery of production and profit for ruling classes living off our labor and lives. Ours not to reason why, ours but to do and die.
Professional classes trained in institutions of higher (l)earning may be held in higher esteem, but grad schools essentially reproduce the same indoctrination and conditioning of grade schools at more advanced managerial levels in chains of command. Their ‘intelligence’ consists of dutiful specialization in branches of bizzness as usual among the fields of knowledge monopolized by powers that be for population control.
Those who successfully navigate grooming for the medical industry thus become products of education for private capital accumulation by the Pharamfia that has long dominated the public service racket of health care. Their ‘knowledge’ is strictly subordinated to allopathic practices which diagnose disease and prescribe remedies, or often cures worse than original illness for a permanent patient population, within the self-serving interests of patented petrochemical poisons, alternatives from naturopathy to nutrition be damned. AI should fit comfortably into the status quo of mechanical medicine, and before long (already underway) robots will be diagnosing patients and prescribing 3-D printed pills to pop to one’s drugged-up content
AI is the next step in standardization of human society, with as little human autonony as possible, serving further dehumanization in the death march of the Machine. It is the next step of ‘progress’ in the capital rule of ‘our’ society as it industrializes labor power, progressively disciplined and deskilled to be replaceable parts in an increasingly centralized system by which behemoths like BlackRock or the BIS (Bank for International Settlements) can now move nations by monetary fiat. The plan is to close the deal on direct rule by global oligarchy with a more perfect system of algogarchy and the AI agentic state, enslaving us to digital dictatorship by such means as digital IDs and currency programming tokenized access to resources based on our worth as social credit in the unaccountable accounting of its data centers. And anyone who resists will be subject to ‘special treatment’ for ‘mental illness’.
Report comment
AI is here to stay. A new literature will document the good and the bad. In time we may be able to judge whether the benefits outweigh the costs. AI has already undone the scientific inquiry that evidence based medicine hoped to use. Now AI generates the evidence as well as the peer review. https://share.google/iJjze9jDpMa2vW1EG.
Report comment
Psssst, if you thought you were doing “God’s will, wouldn’t you already BE “in Heaven?”
Couldn’t, wouldn’t, shouldn’t we?!
Can one put our conundrums in context
To conceivably come up with cues
To console inconsolable egos
And lift us right out of our pews?
Can one ask of the least conscious convict –
And are we not criminals, all? –
“If you thought ’twas God’s Will you were doing,
Could you still feel unfulfilled and small?”
Our wit’s e-duc-ated out of us:
We are left feeling witless and fall
For the babble they know will enthrall us,
And that keeps them so stupid and small.
AI still insists “nociceptors”
Are somethings we possess and all
Who do not – who can, but who will not –
Think critic’ly, trip…..
And on this Flat Earth (theory) still sprawl:
I see nothing as having the potential to transform human and veterinary medicine, and life, like the surely now-imminent realization that
1. “Nociceptors” are absolute and “self-evident” nonsense;
2. There can be no pain in the absence of fear;
3. Pure loving attention can allay fear;
4. Yes, once we understand that everything is indeed the will of an at least infinitely loving “God”/Life/Love/Being/Evolution, our every thought and wish included, we can and do forgive ourselves and others before any offence can occur;
5. Were any of us seated in Infinite Heavenly Bliss “Up There,” and informed that, Yes, all the suffering was still proceeding “down on Earth,” and that the only way we could help would be to return to Earth, we’d be – and are – down t/here like a flash;
6. That includes our favorite ogres/”hitlers,” too, of course;
7. Jesus of Nazareth, for all his faults, was not wrong if/when he predicted that we ALL could and would surpass his works – and do so, of course, not because we are or believe ourselves to be any better or worse, greater or lesser “children of God” than he, but because we can learn that we truly are all his equal, and that the more we appreciate this, the more we are all always capable of more love and of more healing.
https://share.google/Q19SAjWQHHClsIFuk
Don’t we think?
Thank you, Peter Simons, for another wonderful, wonderful essay, and for reminding me that if one Simon Peter could supposedly shadow a Nazarean for ?three long years and still not get it, Zen, that does rather explain…everything and everyone of us?!
May you stay forever young!
https://share.google/bPnkXmTBKmqJ3Ygb6
Much love and laughter!
Comfort and joy, and “God” rest us merry, all!
Tom.
Report comment