Research nonprofit METR released an update to its METR time horizon plot on November 20, 2026, finding that frontier AI systems keep making fast gains in software code but still lag on tasks that need human judgment. While earlier tests showed that the length of jobs models could finish doubled every four months, METR found that raw coding skill does not translate into broader research work [1].
Early Benchmarks on Code
METR built its time horizon plot to track how long AI tools can run on practical jobs without breaking down. To establish a baseline, METR gave human programmers a set of software puzzles and tracked how many hours each task took to finish. Evaluators then gave those exact jobs to AI models, ranking each system by the human time of the longest task it solved. In those first tests, top models showed rapid exponential growth, doubling the length of completed work roughly every four months. Frontier labs noticed the pattern [1].
That quick progress happened mostly on testable code and math (such as coding and math), where simple scripts can check whether an answer works. AI labs train neural networks on these tasks with reinforcement learning, a method where models try a problem many times, building up paths that lead to success and trimming paths that fail. Because compilers give instant yes-or-no feedback, models from OpenAI and Anthropic quickly learned to solve hard software bugs [1].
Automatic checks fail once a job moves away from math into open-ended thinking. Choosing a promising hypothesis or picking a new research path can’t be scored by running a test suite. Clear scores make training easy. Since reinforcement learning needs a clear score to update weights, training models on fuzzy goals remains tough. METR researchers realized that testing only coding gives a skewed picture of AI skills [1].
Why the METR Time Horizon Matters
Interest in automated research surged this month after a redacted METR report claimed AI tools might double the pace of AI work at Anthropic. That finding fueled talk that labs could reach recursive self-improvement, where AI systems build their own successors [1].
Anthropic recently completed a blockbuster IPO. The public debut brought intense scrutiny to claims that software agents might speed up AI research. But writing simple code differs from leading a lab project. To automate genuine research, a model must pick good questions, drop bad leads, and set up solid tests. Without good taste, fast coding just yields piles of useless code [1].
To test these deeper skills, METR added open-ended tasks that reflect real scientific choices. The updated METR time horizon plot now includes jobs estimated to take human experts dozens or hundreds of hours, testing whether models can maintain focus across long projects. Rather than checking code alone, these new tests measure whether systems can evaluate hypotheses and make sound intellectual choices over extended horizons [1].
Failures on Open Tasks
The gap between writing code and exercising judgment showed up clearly when METR tested models on evaluation design. METR asked frontier systems to create new AI benchmarks for its own lab work, which METR staff then graded. While top models beat almost all humans at writing code, not one AI-designed benchmark was good enough for METR staff to use in real research. No AI test met that bar [1].
Newer frontier models did show slight progress over older versions from last year. Anthropic model Opus 5.5 earned a score METR called borderline above the bar, up from twelve months ago when every tested model scored clearly below the bar. Still, falling short of real usefulness shows how hard it is to automate scientific judgment in practical laboratory work. In another trial, METR had models score philosophical arguments, but systems again fell behind human testers. The test also stirred debate inside METR, as employees argued over whether rating philosophy arguments reflects the judgment needed to run AI labs. Because one person graded most arguments, model scores depended on the views of a single individual [1].

Outside experts welcomed the shift toward open-ended tests despite the low scores. UC Berkeley researcher Sayash Kapoor said current models can’t conduct science without human guidance. “We’re at the point where AI evaluations need to get more creative,” Kapoor said, adding that “trying to push the envelope of the space of evaluations is a welcome change” [1].
METR Time Horizon Test Limits
Measuring qualitative judgment brings practical hurdles that do not arise in software tests, as reported by MIT Technology Review. When building the first time horizon plot, METR timed human testers directly as they worked through each coding problem. For the new suite, METR staff relied mostly on estimates of human completion times because hiring qualified professionals to work through tasks requiring hundreds of hours would be prohibitively slow and expensive for any evaluation group [1].
Subjective grading also adds noise to model scores across open-ended benchmarks. In the philosophy argument test, a single human rater scored most entries, meaning AI systems were judged by how closely they matched that one person’s opinions. Polling a larger panel of academic thinkers could change those results, but recruiting specialists across different fields remains difficult. Matt Putz, a member of the technical staff at METR, pointed to the trouble with measuring open tasks. “By their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify,” Putz said in an email, adding that “ultimately, I don’t think we’ll be able to establish trends as clean and robust as for the easy-to-specify” tasks [1].
Because open-ended tests are tough to design and score, METR included only a few of them in the latest report. That small sample makes it hard to predict whether models will continue to improve. “The improvement we observe, I think that does tell us something for the current moment,” said Boston University researcher Najoung Kim, who added that she was unsure how those gains might extrapolate over time [1].
The Bottleneck in Automated Research
AI scientists remain divided on whether coding gains will carry over into research that needs judgment. Some engineers at OpenAI and Anthropic argue that models trained on code will slowly pick up general problem-solving habits. But other researchers believe that scientific discovery requires creative thinking that code compilers can’t teach [1].
Industry data from frontier labs matches the findings in the updated METR time horizon plot. In a blog post published on Wednesday, OpenAI reported that its models perform worse on real-world tasks assigned by staff than on easy coding benchmarks from METR’s original suite. Frontier models can pass hard software tests, but they still stumble on everyday office jobs that lack automated scoring [1].
For now, developing research taste remains the main hurdle on the road to automated science. Sayash Kapoor at UC Berkeley said that choosing promising questions and strong hypotheses will be the primary bottleneck in automating research and development, even if raw coding speed accelerates. While METR shows that AI software tools keep improving, researchers agree that automated discovery will require deeper gains in human judgment before autonomous systems can lead scientific inquiry [1].
- ONLINE NEWS Huckins, G. (2026, November 20). The most misunderstood graph in AI just got a major update. MIT Technology Review. [Article Link]