AI models may be making progress on open-ended tasks, although they still trail human performance and remain difficult to evaluate reliably. A new report from METR found that some recent systems, including Opus 5.5, reached what the organization calls “borderline above-the-baseline” performance on evaluation-design tasks. That marks a change from a year ago, when every model was classified as “clearly substandard,” but researchers caution that the limited number of such evaluations makes future progress difficult to predict.
Open-ended AI tasks remain difficult to measure
METR reports include relatively few open-ended and judgment-based tasks because they are challenging to design and score. This makes it difficult to determine whether models will continue improving in these areas, or how quickly that improvement might occur.
“I think the improvements we’ve observed tell us something at this point,” says Najung Kim, a linguist and computer scientist at Boston University. “I still don’t know how to think about how that would be estimated.”
Matt Putz, a member of METR’s technical staff, also said that progress on these tasks is likely to remain difficult to predict.
“Tasks that are difficult to verify or require judgment by their very nature are much more difficult to study, and especially difficult to quantify,” he said in an email. “Ultimately, I don’t think you can establish trends as clean and robust as those that are easy to identify.”
Estimating the time required for AI research tasks
METR faces practical challenges when testing AI performance on difficult and poorly specified tasks—the kinds of tasks most relevant to the potential automation of AI research and development. METR’s time horizon plot measures task difficulty by estimating how long a human would need to complete each task. In the original plot, researchers obtained these estimates by having people attempt the tasks themselves.
The new report provides most of the time estimates instead. Paying people to complete tasks that can take hundreds of hours, or that only a small number of experts are qualified to perform, is both expensive and time-consuming.
Some tasks also have specific limitations. In one philosophical discussion task, for example, a single person evaluates most of the arguments. AI agents are then judged partly by how much their conclusions agree with that individual. The results could change if more experts assessed the arguments, but conducting a broader evaluation would require additional time.
Researchers urge caution about automated AI R&D forecasts
It is still early to assess models on tasks that are difficult to verify or depend on human judgment. Experts consulted by the report view the inclusion of these tasks in METR’s latest timeline plan as a meaningful step forward, but they agree that the evidence is not yet sufficient to determine whether automated AI research and development will become a reality—or how quickly it could happen.
“This move towards open-ended challenges is promising,” says Kapur. However, he added, “there’s still a lot of work to do to really get a more complete understanding.”
Source: www.technologyreview.com


