METR adds judgment-heavy AI tasks—and finds models still fall short
Newer models show early gains at designing AI evaluations, but none produced work METR judged useful. Measuring that progress is itself a challenge.
Loading page…
Newer models show early gains at designing AI evaluations, but none produced work METR judged useful. Measuring that progress is itself a challenge.
Listen to this story
METR’s expanded benchmark probes work that is harder to score, including designing AI evaluations and judging philosophical arguments. In the evaluation-design task, no model produced work employees considered useful, although newer systems such as Opus 5.5 reached a “borderline above the bar” rating. The results suggest that progress on measurable tasks does not yet establish readiness for research work; moreover, many task durations are now estimated, and limited expert ratings make the new scores less reliable for forecasting.
METR’s original time-horizon plot showed the length of tasks frontier models could complete doubling roughly every four months.
The philosophical-argument task scored models below humans, but METR employees disagree on how relevant it is to automating AI research and development.
Most updated task durations are estimates rather than timed human attempts, partly because some assignments take hundreds of hours or require rare expertise.
METR has widened its influential measure of AI progress to include longer, more open-ended work—and found a gap between improving scores and usable research. In one new task, no model designed an AI evaluation that METR employees considered useful, according to MIT Technology Review’s November 20 coverage of the updated report.
The AI evaluation nonprofit’s original “time horizon” plot measured task difficulty using the time people needed to finish each assignment. Models were then rated by the length of tasks they could complete. It showed a striking pattern: the length of tasks frontier models could successfully finish doubled about every four months.
But that original suite left out tasks whose success is difficult to judge. Coding and math often offer answers that can be checked automatically. Choosing a promising research direction does not. The distinction matters for both training models and assessing whether they can take over research work.
When success can be checked, researchers can have a model attempt a task repeatedly and reinforce the approaches that work. This training method, reinforcement learning, is harder to apply when a good outcome is less clearly defined or takes longer to establish. METR’s update starts testing more of that harder-to-measure territory.
The updated suite adds tasks estimated to take people much longer, alongside assignments that require more judgment. Two examples show what METR is trying to capture—and why the results need careful interpretation:
There is movement within those disappointing results. Some newer models, including Opus 5.5, reached what METR calls “borderline above the bar” on evaluation design. That category still did not mean their proposals were usable for METR’s research. A year earlier, every model had scored “clearly below the bar.”
Experts differ on what separates those gains from autonomous research. Computer scientist Sayash Kapoor told MIT Technology Review that “research taste”—choosing promising questions, hypotheses and experiments—could be a major bottleneck. Others expect training on tasks with checkable answers eventually to carry over to more open-ended work.
By their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify
Matt Putz, METR technical staff member, in an email to MIT Technology Review
The new suite also changes how much direct human measurement sits behind the time estimates. For the original plot, people actually attempted each task. In the update, completion times are mostly estimates: paying qualified people to tackle assignments lasting hundreds of hours, or requiring rare expertise, is expensive and slow.
The philosophical-argument test has another constraint. One person rated most of the arguments, so model scores largely measure agreement with that individual’s judgments. Surveying more experts could change the results, but would also require more time. That makes the choice of evaluator consequential when the test itself concerns judgment.
Because these assignments are difficult to design and score, METR includes relatively few of them. That small base makes it hard to predict whether, or how quickly, the observed gains will continue. Boston University researcher Najoung Kim welcomed what the improvement reveals about current models while cautioning against extrapolating from it.
Putz expects that difficulty to persist: trends for judgment-heavy work may never be as clean or robust as those for tasks with clear specifications. METR’s broader testing reaches closer to the work involved in research, but offers a less tidy basis for forecasting its automation.
Loading discussion...
Join the conversation
Explain when agreement is meaningful, and when it might be misleading.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.