AI regulation debates tend to focus on a simple question: which company should be held responsible when something goes wrong? A recent analysis published in the Proceedings of the National Academy of Sciences suggests that framing is the wrong one entirely. The real question is how a rule aimed at one company changes the behavior of every other company in the chain. The answer, it turns out, has consequences that most policy discussions are not yet equipped to handle.
The Supply Chain That AI Regulation Ignores
Most AI products are not built by a single company. They are assembled across layers. A general-purpose model, produced by a large AI developer such as OpenAI or Anthropic, serves as the foundation. A second company then adapts that model for a specific use, such as summarizing physicians’ clinical notes or powering a customer service chatbot. Each layer carries its own safety responsibilities: the model maker handles training, evaluation, and documentation; the downstream company handles clinical validation, fine-tuning, and error monitoring.
This layered structure is precisely what most regulation fails to account for. The EU AI Act, which the European Commission is preparing to enforce, imposes different safety and transparency requirements on general-purpose model providers and on companies building applications on top of them. That distinction is a step forward. But the critical question is not just who gets regulated. It is how a requirement targeting one layer reshapes the incentives of everyone else.
How Weak Rules Can Make Products Less Safe
Benjamin Laufer, a senior applied researcher at Microsoft and incoming assistant professor at the University of Washington Information School, explored this mechanism alongside coauthors Jon Kleinberg and Hoda Heidari. Their game-theoretic model examined what happens when a regulator sets a minimum safety requirement for a downstream application company, while leaving the upstream model maker unconstrained.
The finding is counterintuitive. A weak safety floor, defined as a requirement at or below the level firms would have reached on their own, can actually reduce overall product safety. The downstream company complies, increasing its safety investment. But the model maker, knowing that a safety check exists further down the chain, reduces its own investment by more than the downstream company adds. The net result is a less safe product, despite the presence of a regulation.
The analogy Laufer uses is instructive. Imagine a food safety regime that requires restaurants to inspect every ingredient they serve, while placing no obligations on large suppliers. Restaurants bear real responsibility, and the rule is not unreasonable on its face. But suppliers, anticipating that restaurants will catch any problems, have less reason to maintain their own quality controls. The same logic applies to AI development chains.
The more constructive finding is the flip side. When appropriately calibrated requirements apply to both the upstream model maker and the downstream application company, regulation can function as a coordination mechanism. Each firm invests in safety with the assurance that the other must do the same. In some scenarios the researchers modeled, this dual accountability led to greater safety and better performance for consumers, while leaving both firms financially better off.
Why This Matters Beyond the Regulatory Debate
This is what most coverage of AI regulation misses. The conversation tends to treat safety as a fixed quantity that gets assigned to whoever is closest to the end user. Laufer’s framework reveals that safety is not a fixed quantity at all. It is a strategic variable, and the rules governing one actor change how every other actor behaves.
The implications extend beyond any single law or jurisdiction. In June, the U.S. government required Anthropic to restrict access to its two newest models for foreign nationals, citing national security concerns. Unable to verify nationality in real time, Anthropic temporarily withdrew access for all users. Shortly after, the Chinese company Moonshot AI released its Kimi K3 model and published its full model weights. These episodes illustrate the political pressures that shape AI regulation, but they also underscore the stakes. When governments intervene at the model level, the downstream companies building on those models face disruptions they did not cause and cannot fully control.
As AI models become more capable and more widely deployed, the number of such interventions is likely to grow. Each one will test whether regulatory frameworks are designed to distribute accountability across the full development chain, or whether they allow safety to be quietly passed down until someone else picks up the bill.
The question regulators need to ask is not simply who is best positioned to prevent harm. It is who must contribute, at every layer, to make the final product genuinely safer. A rule that answers only the first question may end up answering neither.
In Short
Regulating only the downstream layer of AI development can backfire. When model makers know that application companies must meet safety standards, they may reduce their own safety investments by more than the downstream companies add, leaving the final product less safe overall. Research published in the Proceedings of the National Academy of Sciences by Benjamin Laufer, Jon Kleinberg, and Hoda Heidari shows that well-calibrated rules applying to both upstream and downstream actors can instead create shared accountability and stronger safety outcomes. The lesson for policymakers is structural: AI safety cannot be outsourced to the layer closest to the user.
Based on reporting from Fast Company - Tech.