A study published in Nature Medicine by researchers at MIT, Stanford University, and Columbia University arrives at a finding that cuts against a common assumption in medical AI: that better explanations make AI tools more useful for everyone. The research shows the opposite can be true. The same explainability feature that helps a clinician can mislead a non-expert, and the same AI system that improves diagnostic accuracy in one group can deepen errors in another.
Why Explanations Don’t Work the Same Way for Everyone
The study tested two groups: non-experts and primary care providers, both asked to diagnose skin diseases using medical images, with and without AI assistance. The AI tools varied in how they explained their predictions. Some provided a confidence level with no further explanation. Others used heat maps to highlight relevant image regions. Others still used large language models to generate plain-language explanations of the model’s reasoning.
Non-experts improved their diagnostic accuracy across all explainable AI approaches. But the improvement came with a catch. It was largely driven by deference: users were following the AI rather than reasoning alongside it. When the model was wrong, non-experts tended to follow it into the wrong answer. The effect was strongest with LLM-based explanations. Users were more confident in their incorrect answers when an LLM had explained the reasoning, and they found vague or generic explanations more convincing, not less.
Clinicians behaved differently. They were resilient to incorrect AI outputs and performed best when given only the model’s prediction, with no accompanying explanation. Of all the methods tested, LLM explanations boosted clinician accuracy the least.
Orson Xu, lead author and assistant professor of biomedical data science at Columbia University, describes the dynamic clearly: a clinician already has a working diagnosis and uses the AI as a check against their own training, so a flawed explanation gets caught. A non-expert, lacking that foundation, uses the explanation to form an opinion in the first place. The same tool becomes an asset for one user and a liability for the other.
The Deference Effect and What Drives It
The researchers identified a pattern they call the deference effect: users who relied most heavily on AI assistance were also the weakest performers when working without it. This is not a coincidence. It reflects how automation bias operates. When a system presents a confident, coherent explanation, users anchor to it, even when the underlying prediction is wrong.
Timing matters too. The study found that presenting an AI explanation before a user has formed their own hypothesis increases deference. Users who saw the AI’s reasoning first became more likely to follow it uncritically. This suggests that the sequence in which information is delivered, not just its content, shapes how people reason.
There were also findings about fairness. When the researchers used a fairness-constrained model designed to reduce bias against darker skin tones, the system significantly improved accuracy and reduced diagnostic disparities based on skin tone. This points to a design dimension that goes beyond accuracy alone: who benefits from AI assistance, and under what conditions, is shaped by how the model is built.
Marzyeh Ghassemi, associate professor at MIT’s Department of Electrical Engineering and Computer Science and a principal investigator at the Laboratory for Information and Decision Systems, frames the core tension: AI assistance can improve performance in health settings, but that benefit has to be weighed against the risk of algorithmic deference leading to more errors. Both AI systems and explainability methods can engage automation bias, and that anchoring effect has to be accounted for in design.
What This Means Beyond Dermatology
This research matters beyond skin disease diagnosis. It surfaces a structural problem in how medical AI is often deployed: tools are evaluated on average performance across users, but the people who stand to gain the most from AI assistance are frequently the ones most vulnerable to being misled by it.
Roxana Daneshjou, co-author and assistant professor of biomedical data science and dermatology at Stanford University, puts it directly: those with the least medical knowledge are most likely to be led astray when an explainable AI model produces an erroneous output. This is not a marginal edge case. Non-experts increasingly use AI-powered tools to assess their own health, and several FDA-approved AI interfaces are already in use to help clinicians identify skin conditions in medical images.
The implication for design is concrete. Rather than generating more detailed LLM explanations, the researchers suggest a different approach: require users to form a diagnostic hypothesis first, then present the AI’s suggestion as a prompt to consider other possibilities. The goal is to use AI to expand thinking, not to replace it. As Ghassemi notes, the risk of engaging automation bias is that when the model is wrong, users can no longer recover.
This is what most coverage of medical AI misses. The question is not whether AI improves diagnosis. In many cases, it does. The question is for whom, under what conditions, and with what design choices in place.
In Short
AI explanations in medical diagnosis do not help all users equally. Non-experts improve their accuracy but largely by deferring to the model, which makes them more vulnerable when the model is wrong. Clinicians are more resilient to AI errors and benefit least from detailed explanations. The same explainability feature can support expert reasoning and undermine non-expert judgment. Designing AI for health settings requires knowing who will use the system and building tools that encourage critical thinking rather than blind reliance.
Based on reporting from MIT News AI.