Frontier AI models keep their reasoning secret. Or so the companies building them believed. A team of computer scientists has now demonstrated that this secrecy is more fragile than assumed, with implications that stretch from personal data security to the geopolitics of AI development.
The Vulnerability Hidden in Plain Sight
When advanced AI models tackle complex problems, they do not simply produce an answer. They work through a chain of thought, breaking the problem into steps and analyzing each one in sequence. Companies treat these reasoning traces as proprietary. Revealing them would allow competitors to use them for training new models, a process known as distillation. To prevent this, companies encrypt the reasoning and send it to a user’s machine in a form that is not supposed to be readable.
The key word is “supposed.” Researchers from the University of Tübingen, the Max Planck Institute, the AI safety institute MATS Research, and the security company Snyk identified a structural weakness in how this encryption works in practice. Most major AI providers offer their models in multiple sizes. Larger models are more capable and more expensive to run; smaller versions exist for users who want lower costs. Crucially, the smaller models share the same decryption infrastructure as their larger counterparts, but they have received less alignment training. That means they are less likely to refuse to reveal what they are processing.
By feeding encrypted reasoning traces to a smaller version of the same model family, the researchers found they could recover the hidden reasoning that the larger model had produced. The attack does not require breaking any encryption key. It exploits a gap in how consistently alignment is applied across a model family. As Alexander Panfilov, a computer scientist at the University of Tübingen who was involved in the work, put it: “All major frontier model providers we tested share this vulnerability.”
The same method also surfaced something more immediately alarming: personal information, including passwords and API keys, embedded in reasoning traces captured from a user’s machine. OpenAI, Anthropic, and Google were alerted to the vulnerability, and each has since adjusted its API to prevent private information from being extracted this way. The distillation risk, however, has not been fully resolved. Panfilov notes that fixing it entirely would require a fundamental overhaul of how these companies’ APIs are structured.
The Distillation Question and What the Evidence Actually Shows
The research opens a second, more contested front: the question of whether some AI models have already been built by distilling reasoning from competitors’ closed systems.
To investigate this, the researchers fed 90 questions to a set of models, then gave some open-weight models the first few words of reasoning traces captured from proprietary systems. In several cases, the open models generated outputs that closely matched the hidden reasoning of the closed ones. This was particularly pronounced with Kimi K3, an open-weight model from the Chinese company Moonshot AI, which produced outputs strikingly similar to the hidden reasoning traces of Claude Opus 4.8 and GPT 5.6 Sol for certain prompts. Two other open-weight models, DeepSeek from China and Inkling from the US company Thinking Machines, did not show the same pattern.
The researchers are careful about what this means. Their paper explicitly states that the findings “cannot causally establish distillation.” Similarity in reasoning patterns is suggestive, not conclusive. Moonshot AI did not respond to requests for comment before publication.
This matters because distillation has become a politically charged topic. OpenAI told US lawmakers that DeepSeek appeared to have copied one of its models to build its reasoning model R1. Anthropic told lawmakers that Alibaba had systematically distilled its models to build Qwen. These are serious claims, and the new research adds a technical dimension to the debate without settling it. Kyle Miller, a researcher at the Center for Security and Emerging Technology, a tech policy think tank, has noted that it remains unclear how much distillation actually benefits Chinese AI development, and that removing the ability to distill would not, in his view, dramatically change the competitive landscape.
Why This Goes Beyond a Security Patch
Here is what most coverage of this story misses. The vulnerability itself has been partially fixed. The deeper issue is structural, and it points to a tension that will not be resolved by an API update.
AI companies have built business models around keeping reasoning proprietary. That secrecy is both a competitive moat and, they argue, a safety measure. But the architecture required to deliver AI capabilities at scale, with models of different sizes sharing infrastructure, creates seams. Those seams can be found and exploited. The researchers demonstrated this with a method that Florian Tramer, a computer scientist at ETH Zürich who specializes in computer security, described as “very cool” in its elegance: swap messages to a weaker model variant that shares the decryption key but has weaker alignment.
The broader implication is that AI systems reveal their inner workings in ways their designers do not fully anticipate. Distillation is already widely used across the industry. Mark Zuckerberg, CEO of Meta, described it in a blog post as “an important principle of how the open source ecosystem works.” Yarin Gal, a computer scientist at Oxford University, has noted that distillation has helped AI advance more rapidly, and that universal restrictions on the practice would slow the rate of progress for everyone.
The question is not whether distillation happens. It does, openly and legitimately, across the industry. The question is whether the boundaries companies draw around their proprietary reasoning are as solid as they believe. This research suggests they are not.
In Short
A research team has shown that the hidden reasoning of major AI models can be extracted by exploiting a gap between larger and smaller versions of the same model family. The method also exposed personal data embedded in reasoning traces, a vulnerability that has since been patched. The same technique raises questions about whether some open-weight models were built using distilled reasoning from closed competitors, though the evidence is suggestive rather than conclusive. The deeper lesson is that the architecture of AI delivery creates structural vulnerabilities that cannot always be fixed with a simple update.
Based on reporting from Wired.