The most shared chart in AI measures something narrow and easy to misread: how long a task, as timed by humans, a model can successfully complete. The evaluation nonprofit METR has now updated that chart, and the revision matters less for the trend line it extends than for the weaknesses it finally exposes. Models keep improving at a rapid clip. On tasks that require judgment, they are not keeping pace.
What the original graph actually measured
METR’s method was simple in concept and easy to get wrong in the retelling. Models attempted a large suite of tasks, each of which humans had also completed. Every task was scored by how long the human testers needed to finish it. Models were then rated by the length of the tasks they could complete.
The headline finding was striking: every four months, the length of tasks that frontier models could successfully complete doubled. But the measure was subtle, and the plot was widely misinterpreted. A line on a chart about “task length” is not a line about general intelligence, and many people read it as one.
There was also a quieter limitation. The original suite leaned on verifiable tasks, ones where software can check automatically whether the AI succeeded or failed. That is not an accident. It is the kind of task we know how to train for.
Why verifiable tasks are the easy part
The training recipe is well understood. Researchers have a system attempt a task many times, strengthen the neural pathways that contribute to success, and weaken those that contribute to failure. This is reinforcement learning, and it works best when “success” can be checked by a machine. That is why coding and math have become such strong suits for AI: a program either runs or it doesn’t, a proof either holds or it doesn’t.
Now consider choosing a new research direction. Success is fuzzy, and it may take a very long time to find out whether the choice was good. The same property that makes these tasks hard to train for makes them hard to evaluate, which is why METR’s first plot left them out entirely.
Yet this is exactly the category that matters if AI systems are ever to automate research and development. The updated plot adds tasks that are estimated to take humans far longer, plus more open-ended ones.
Where the models fall short
In one new task, models were asked to design new AI evaluations that METR could use in its own research, with human METR employees scoring the results. The most advanced models are now better than almost every human at software engineering, but here the outcome was clear: not one AI-designed evaluation was good enough for METR staff to consider useful.
In another task, models scored philosophical arguments, a stand-in for exercising good judgment. Models again fell short of human performance. METR itself notes that its employees disagree about how relevant this task is to automating AI R&D, which is a useful reminder of how unsettled the measurement question still is.
There is a hopeful detail, though. Some of the newest models, such as Opus 5.5, tend to reach what METR calls “borderline above the bar” on the evaluation design tasks. That is not yet usable for METR’s research, but a year ago every model scored “clearly below the bar.”
The caveats are substantial. Completion times in the new report are mostly estimates, since paying people for tasks that could take hundreds of hours, or that only a few experts can do, is prohibitively expensive. On the philosophical task, one person rated most of the arguments, so models were judged by how well they agreed with that individual. And METR includes few of these tasks overall, because they are so hard to design and score.
Why the debate is not settled
This is the part most coverage misses: the report does not resolve the big question, and the experts quoted in it say so. Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, welcomes the push toward more creative evaluations. He was not surprised by the shortfall on open-ended work, pointing to an August preprint in which he and collaborators found that AI models cannot yet reproduce human computer science research on their own. He suspects research taste, the ability to choose promising questions, hypotheses, and experiments, will prove a major bottleneck.
Others argue the opposite: that models trained on easy-to-verify tasks will eventually handle open-ended ones too. Progress on verifiable tasks does not appear to be slowing, which keeps that possibility alive. Najoung Kim of Boston University says the observed improvement tells us something about the current moment, but she is unsure how to extrapolate it. Matt Putz of METR adds that trends for judgment-laden tasks are unlikely to ever be as clean and robust as those for easy-to-specify ones.
The stakes explain the attention. A separate, heavily redacted METR report earlier this month claimed AI might be speeding up AI progress at Anthropic by a factor of two, which fueled fears about recursive self-improvement, where AI systems design their own successors with no human involvement. The new findings sit awkwardly against that fear: the very skills that would make such a loop possible, taste and judgment, are where models still trail.
What this means beyond the chart
A single trend line cannot describe a technology with uneven strengths. The practical lesson is to ask what kind of task a claim is about. Where success can be checked automatically, AI capability is advancing quickly. Where success depends on judgment, context, and long feedback loops, the picture is murkier, and the measurement tools are themselves immature.
That also reframes the human role. The tasks where models still struggle, choosing what is worth investigating and deciding what counts as good, are the ones where human judgment is most visibly in the loop. Tools that handle volume and verifiable work well can free people to spend more time on exactly that.
In short
METR’s updated time horizon plot confirms that AI keeps improving quickly on verifiable tasks, while showing that models still fall short on judgment-heavy work like designing useful evaluations or scoring philosophical arguments. Newer models are edging closer, but the evidence is thin because such tasks are hard to design, score, and time. Experts disagree on whether skill from easy-to-check tasks will eventually transfer, so the honest reading is that the question is open, not answered. Treat any single chart about AI progress as a measurement of one narrow thing, and ask what it leaves out.
Based on reporting from MIT Technology Review.
Read the original study.