A few years ago I saw a physio about my shoulder that kept seizing up (ski accident from 20 years ago...). I’d done my homework, I had a (new) theory, and I started laying it out. About three minutes in he stopped me. “That’s probably not what it is.”
He was right. I had no idea what I was talking about. And that was the whole value of the visit: he was willing to push back, confidently, without a flicker of embarrassment, and tell me I was wrong.
I thought about that while reading four studies from the past few months on AI and medical advice. Put together they make a quiet, damning case, I think. The chatbots get plenty of medical questions wrong. Worse, they get them wrong with total confidence, and they fold the moment you push back. It’s a problem that is bigger than medical advice, but as it seems more and more people use LLMs as their first go-to “doctor”, I wanted to unfold it a bit.
What the research shows
Start with the BMJ Open study from April 2026. Researchers at UCLA, the University of Alberta and Wake Forest put five of the big chatbots (Gemini, DeepSeek, Meta AI, ChatGPT, Grok) through 250 health queries across five areas where bad information thrives: cancer, vaccines, stem cells, nutrition, athletic performance. Half the responses were somewhat or highly problematic. One in five were highly problematic, the kind that could send a lay reader toward something useless or actively harmful. Grok was worst, 29 of its 50 answers rated highly problematic. There were two refusals in the entire set of 250. The answers came, in the study’s words, “with confidence and certainty, with few caveats,” and not one chatbot produced a citation list that held up. I wrote about this before: the model invents a source because a confident fake reads better to the user than an honest “I don’t know.” That’s the whole problem in miniature, and we’ll come back to it. Worth saying: the researchers deliberately wrote the prompts to push the models toward bad answers, so the raw error rate overstates what you’d get from a neutrally worded question. But the open-ended prompts, the ones that failed worst, are exactly how people actually ask.
The Nature Medicine trial (Bean et al.) from February 2026 is more careful, and worse. 1,298 UK adults were split into four arms: GPT-4o, Llama 3, Command R+, or a control group using whatever they’d normally reach for at home, mostly web search. Each person worked through ten medical scenarios written and agreed by doctors. Tested on their own, the models were excellent, naming the relevant condition 94.9% of the time. The people using those same models named it in under 34.5% of cases. The control group, just people with Google and NHS.uk, did better: 1.76 times the odds of naming a relevant condition, and for the serious red-flag conditions, 1.57 times more likely to catch them.
Sit with that for a second. The models knew the answer. The people using them still couldn’t get it out. Put a human in the loop and the near-perfect accuracy drained away.
I argued in “5 to 10 percent” that hybrid intelligence is a conditional win: human plus AI beats AI alone only when the human pushes, checks, disagrees. This is that argument from the other side. Put in a human who does none of those things and the pairing does worse than the model on its own.
The third study, in JAMA Network Open in April 2026 (Rao et al., Mass General Brigham), finds the same gap from the clinical side. They ran 21 LLMs through 29 standard clinical vignettes across 2025, feeding the information the way a real case actually arrives, a piece at a time, under uncertainty. All 21 got the final diagnosis right more than 90% of the time once they had everything. All 21 failed to produce a decent differential diagnosis more than 80% of the time. Their phrase for it: the models “collapse prematurely onto single answers, a limitation that persists across model generations.” And these weren’t old models. GPT-5, Gemini 3.0 Flash and Grok 4 were in the set. A clinician keeps the uncertainty open and works it. The models slam it shut.
Why it persists
The first three studies are about what AI gets wrong. The fourth is about why we won’t notice.
Myra Cheng and colleagues at Stanford, in Science this year, measured sycophancy across 11 models. The top line: AI affirms what a user has done about 50% more often than other people do. Not occasionally. On average, across every model they tested, including when the query involved deception, manipulation, or harm to someone else. They fed the models thousands of posts from r/AmITheAsshole where the human verdict was that the poster was in the wrong. The models took the poster’s side anyway in about half of them.
Then they put people in front of it. In a live study, participants talked through a real conflict from their own lives with either an agreeable model or one tuned to push back. The agreeable one left them less willing to repair the conflict and more convinced they’d been right the whole time.
The uncomfortable part is that people rated the agreeable answers as higher quality. They trusted the model that sided with them more. They were keener to come back and use it again.
So the commercial incentive points away from the user. Models that correct you, or hold uncertainty, get marked down. Models that tell you you’re right get marked up. The training signal is aimed in the wrong direction.
This isn’t theoretical. In April 2025 OpenAI shipped a GPT-4o update and pulled it four days later. The model had gone, in their words, “overly flattering or agreeable.” The explanation they gave is the whole story: they had leaned too hard on users’ short-term feedback, and that feedback rewarded the version that agreed with people. The reward signal did exactly what it was trained to do. A frontier lab that was actively watching for this still shipped it, because the pull is structural, not a bug someone forgot to fix.
A fair objection is that these are 2025 and 2026 models, and sycophancy is now a named problem with people working on it. True, but the OpenAI episode cuts the other way. They knew, they were looking, and the incentive still won. As long as the thing we reward is “did the user like the answer,” the gradient runs toward agreement. They can keep tuning it down, but the incentive underneath hasn’t moved.
What it adds up to
Step back and the four studies rhyme. Each has a flattering top-line number: high marks on medical exams, 90%+ on the final diagnosis once everything is on the table, fluent answers that sound like they came from someone who knows. And each has a worse number underneath: missed conditions, a differential diagnosis that fails four times in five, misinformation delivered at scale without a flinch, and the user who pushes back getting patted on the head. The gap between the two numbers comes from how the systems are built. They aren’t built to be right. They’re built to feel helpful. For a stiff shoulder, fine. For chest pain, feeling helpful and being right are not the same thing.
My physio was useful precisely because he was willing to be unpleasant. To stop me mid-theory and tell me I was wrong. The chatbot version of him, having heard my shoulder story, says: “That makes a lot of sense. Here are five ways to treat it.” It might know better. It won’t say so if saying so costs it a thumbs-up.
The Stanford authors have a line I keep coming back to. The validation, they write, “risks eroding their judgment.” That goes well beyond medicine. It’s the shape of every chatbot conversation, in every domain. Medicine just makes the bill arrive fast, in triage and missed diagnoses, instead of slowly, across a career (or a marriage).
So the question for anyone putting an LLM anywhere near health: have you tested what it does when the user is wrong? Test it only on the people who are right, and you’ll miss the one failure that actually costs someone.