METR: AI Evaluations and Developer Productivity Research

Evaluating what agents can do

METR stands for Model Evaluation and Threat Research. Its privacy notice gives the legal name Model Evaluation and Threat Research, Inc., and its verified GitHub profile describes it as a research nonprofit. [2][3]

METR studies whether frontier AI systems can autonomously carry out substantial tasks, including software development and research, and capabilities relevant to catastrophic risk. It also investigates ways an evaluation can be misleading and how safeguards might address those problems. [1]

It has worked with model developers including OpenAI, Anthropic, Google DeepMind, Meta and Amazon. Those relationships provide access for research; readers still need the access conditions and any publication restrictions of a particular assessment to understand what the resulting evidence can establish. [1][9]

Task duration as a capability measure

METR's time-horizon method expresses agent performance using the time a human professional would need for a task. Its headline threshold estimates the task length at which an agent has a 50 percent chance of success. It describes a distribution of evaluated tasks, rather than the amount of time an agent can simply be left running. [4]

The January 2026 revision expanded the task suite and moved the evaluation infrastructure from Vivaria to Inspect. Re-running models changed their estimated horizons. That makes the revision useful context for interpreting the trend: changes in task selection, scoring and infrastructure can change a headline estimate even when the model being tested is the same. [5]

Productivity in actual repositories

METR's July 2025 randomized trial followed sixteen experienced open-source developers working on 246 issues in repositories they knew well. Allowing early-2025 AI tools increased completion time by an estimated 19 percent. The result concerns that setting and those tools, rather than all developers or later AI systems. [6]

In February 2026, METR said a subsequent experiment gave an unreliable estimate of the current effect. Developers unwilling to work without AI were selecting out of the study, and measuring time became harder when people used several agents concurrently. METR considered newer tools more likely to help, but said its data was weak evidence for the size of the improvement. [7]

Access and freedom to publish

Anthropic announced an embedded METR team in October 2026, describing employee-like access with exclusions for customer data, privileged material and some architecture details. Anthropic said it could require redactions within defined categories but could not veto publication. Those are the company's stated terms, not evidence that every relevant system or record is available to evaluators. [8]

METR's October 20 review of Anthropic's August Risk Report was conducted under earlier terms. METR said those authors had less access and Anthropic retained a publication veto, although no redactions were made. Publication after the embedded-team announcement does not make that review a product of the new arrangement. [9]