A foundational assumption has quietly governed the use of patient data in artificial intelligence research for years. The logic runs as follows: if names and identifying details are stripped from medical records before those records are used to train an AI model, the individuals behind the data are effectively protected. A study published in Nature by researchers including Haoran Zhang and Marzyeh Ghassemi, both at the Department of Electrical Engineering and Computer Science at MIT, challenges that assumption directly. Their analysis of the work by Knolle et al., also published in Nature, makes a pointed argument: in the era of powerful machine-learning models, de-identification is not the privacy guarantee it was assumed to be.
The Implicit Bargain That May No Longer Hold
The relationship between patients and AI researchers has always rested on an exchange. Patients, and the health systems that hold their records, allow sensitive data to be used for research purposes. In return, that research is expected to produce tools that improve care, whether through earlier diagnoses, better treatment options, or more efficient clinical workflows. De-identification sits at the center of this arrangement. It is the mechanism that is supposed to make the exchange ethically acceptable.
The problem identified by Knolle et al. is not that de-identification fails in some technical or procedural sense. The problem is more fundamental. Once a machine-learning model has been trained on data, the model itself can become a source of information about the individuals whose records shaped it. Someone with access to the trained model, even without access to the original dataset, may be able to infer whether a specific person’s data was used in training. This class of vulnerability is known in the research literature as a membership inference attack, and it has been documented in prior work cited in the Nature piece, including research presented at major machine-learning and security conferences.
Why Powerful Models Make the Problem Worse
This is where the analysis becomes particularly important for anyone thinking about the trajectory of medical AI. The concern is not limited to small or poorly designed systems. The Nature commentary is explicit: the assumption that training data cannot be reconstructed from a model is “often untrue” specifically in the context of powerful machine-learning models. The more capable the model, the more it tends to memorize details from its training data, and the more susceptible it becomes to this kind of privacy leakage.
This creates a tension that sits at the heart of modern AI development. The same properties that make a model clinically useful, its ability to learn fine-grained patterns from large and complex datasets, are the properties that increase privacy risk. Optimizing for performance and optimizing for privacy protection can pull in opposite directions.
The research community has proposed technical countermeasures. One approach, differential privacy, involves adding carefully calibrated noise to the training process to limit how much any individual data point can influence the final model. Work on this approach was published as early as 2016, and it is cited in the Nature piece. But differential privacy involves tradeoffs: the stronger the privacy guarantee, the more model performance tends to degrade. That tradeoff is not merely a technical inconvenience. In a medical context, where model accuracy can affect diagnostic outcomes, it carries real consequences.
Unequal Exposure: Who Bears the Risk
Here is what most coverage of this topic misses. The title of the related paper cited in the Nature commentary is telling: “Disparate privacy risks from medical AI.” The risk is not distributed evenly across patient populations. Research cited in the piece, including work published in The Lancet Digital Health, has shown that AI systems trained on medical data can encode and reflect demographic disparities. The privacy dimension adds another layer to this concern.
If certain groups of patients, defined by demographic characteristics, health conditions, or the volume of data they contribute to training sets, face higher exposure to membership inference or data reconstruction, then the implicit bargain described above is not just broken in general. It is broken more severely for some people than for others. Patients who may already face inequities in how AI systems treat them clinically could also face greater privacy vulnerabilities from the same systems.
This is a structural issue, not an accident of implementation. It points toward the need for privacy analysis to be disaggregated, meaning researchers and developers should examine not just whether a model leaks information on average, but which individuals or groups are most exposed.
In Short
De-identification of medical records does not guarantee that individuals are protected once their data has been used to train an AI model. Research published in Nature by MIT-affiliated researchers, drawing on the findings of Knolle et al., demonstrates that powerful machine-learning models can be queried in ways that reveal information about the people whose data shaped them. The privacy risks are not uniform: some patients may be significantly more exposed than others. The implicit agreement that has made patient data available for AI research, the promise that de-identification is sufficient protection, needs to be revisited. Technical solutions exist, but they involve tradeoffs that the field has not yet resolved.
Based on reporting from Nature: Machine Learning.