Teaching a robot to place a cup in a sink or move a soda can from a shelf to a table sounds simple. It is not. Every action requires practice across dozens of different settings, and physically staging all of those environments takes enormous time and labor. This is the core bottleneck slowing down the development of capable robots, and it is the problem a new system called SceneSmith, developed by researchers at MIT CSAIL and Toyota Research Institute, is designed to address.
The Data Problem That Keeps Robots in the Lab
Robots, much like humans, learn best through experience. The challenge is that real-world training is slow, expensive, and difficult to scale. Simulation offers an obvious alternative: build virtual environments where robots can practice, fail, and try again without any physical consequences. The problem has never been the physics engines that power these simulators. Those have improved considerably. The gap has been in the quality and variety of the virtual worlds themselves. Most generated environments were too sparse, too generic, and too unlike the cluttered, unpredictable spaces robots actually need to navigate.
Russ Tedrake, the Toyota Professor of Electrical Engineering and Computer Science, Aeronautics and Astronautics, and Mechanical Engineering at MIT and a principal investigator at CSAIL, frames the challenge directly: creating sufficiently rich and diverse simulation content to capture the complexity of the real world has remained one of the hardest unsolved problems in robotics training.
SceneSmith is a direct attempt to close that gap.
Three Agents, One Virtual Room
The system works by deploying three AI agents, each with a distinct role, to collaboratively construct a 3D scene from a simple text prompt. A user might type “generate a garage with a car, a workbench, tires stacked in the corner, and a ladder against the wall,” and the system builds it.
Each agent draws on a vision-language model, specifically GPT-5.2, which has been trained on large volumes of text and images and carries what the researchers describe as a kind of spatial knowledge about how everyday places are supposed to look. The first agent, the “designer,” generates the initial layout and populates it with objects. The second, the “critic,” evaluates whether the scene looks realistic and flags problems, such as a bathtub placed in a living room. The third, the “orchestrator,” manages the back-and-forth between the other two and decides when the scene is finished, even rewinding the process if the output does not meet quality standards.
The result is environments that contain up to six times more objects per scene than prior methods. The researchers generated over 1,300 scenes during development. Nicholas Pfaff, an MIT EECS PhD student and lead author on the paper, notes that the system produced creative and diverse arrangements that were not explicitly programmed into the prompts. The agents improvised, drawing on the internet-scale knowledge embedded in the underlying model.
SceneSmith also goes beyond visual realism. Objects generated by the system carry physical properties including mass, friction, and inertia, making them usable inside physics simulation software rather than just visually plausible. This includes articulated objects like cabinets that robots can open and close, a category that earlier baseline systems rarely handled.
Why Realism in Simulation Actually Matters
The most rigorous test of whether a simulated environment is genuinely useful is not visual inspection. It is whether a robot trained on real-world data can operate inside the virtual space without any additional training. The researchers ran exactly this test: they placed a pretrained robot policy, one that had never encountered a SceneSmith scene, into the generated environments. The robot successfully completed tasks like taking an apple from a bowl and placing it on a cutting board. If the simulated spaces had not closely resembled the real settings the robot had learned from, the task would simply have failed.
In a separate evaluation, over 200 users compared SceneSmith’s output against other scene-generation systems, including HSM and Holodeck. They found SceneSmith’s visuals to be more realistic over 90 percent of the time and judged it to be more faithful to the original prompts. The system also passed a policy evaluation test in which a VLM agent assessed 100 unique robot action plans across generated environments. Humans agreed with the model’s verdicts over 99 percent of the time, suggesting that simulation-based evaluation could reliably catch flawed robot behaviors before any real-world deployment.
Jeremy Binagia, an applied scientist at Amazon Robotics who was not involved in the research, describes SceneSmith as advancing the state of the art in several concrete ways: higher object density, physical accuracy rather than just visual realism, and the ability to generate assets that are not constrained to a fixed library.
There is a trade-off. Generating a single scene can take multiple hours because each object is created and scrutinized individually. The researchers acknowledge that greater computing power would substantially reduce this cost.
In Short
SceneSmith shows that AI agents can do more than answer questions or generate text. They can build the training environments that other AI systems, embedded in physical robots, need to develop reliable skills. The bottleneck in robotics has never been the robots themselves. It has been the data. Systems like this one shift the economics of that problem, making it possible to generate rich, physically accurate virtual worlds at a scale that manual construction cannot match. The engineers are still in the loop, defining the prompts and evaluating the outputs. The agents handle the volume.
Based on reporting from MIT News AI.