The workflow moved beyond the editor
GitHub introduced Copilot in June 2021 as an editor assistant that could suggest lines and functions. Claude Code arrived in research preview in February 2025 with the ability to inspect repositories, edit files, run tests, and work with command-line tools. These are different responsibilities, even when both products are described as coding assistance.
Why the productivity findings differ
A 2023 controlled Copilot experiment reported 55.8% less time to complete a JavaScript HTTP-server task among participants who finished. That is a different test from experienced maintainers changing familiar repositories. The result supports a speedup in that experimental setting; it does not supply a universal multiplier for software teams.
Capability and productivity are different claims
METR’s early-2025 experiment found that 16 experienced open-source developers took 19% longer on their own project tasks when AI tools were allowed. That result was specific to its developers, tasks, and tools. In February 2026, METR reported that selection effects made its follow-up an unreliable estimate of the current productivity impact: developers and tasks benefiting most could be missing from the experiment.
Learning is an outcome of its own
In Anthropic’s January 2026 experiment, 52 developers learned the unfamiliar Trio library. The AI group averaged 50% on an immediate quiz versus 67% without AI: a 17 percentage-point gap. The time difference was not statistically significant. This tests learning with a chat assistant, not the long-term effects of delegating to an autonomous agent.
What real agent sessions can tell us
Anthropic’s June 2026 report analyzed roughly 400,000 Claude Code sessions from October 2025 through April 2026. It associated greater inferred task expertise with more successful sessions. Both expertise and success were derived from transcripts, and the researchers did not observe real-world outcomes directly. This describes usage patterns; it does not establish how much faster the same work would have been without AI.
What this means for a team
Evaluate a coding agent on completed, reviewed changes in your own repositories. Count review time, failed attempts, and rework alongside generation time. As the agent takes on execution, the engineering work shifts toward task boundaries, permissions, tests, and a reviewer’s ability to understand what happened. A launch announcement establishes availability; it does not establish a productivity gain.
Compare the evidence
Different settings and outcomes. Read the study design before comparing the headline numbers.
Copilot experiment · 2023
- Setting
- Implement a JavaScript HTTP server in a controlled task.
- Finding
- 55.8% less completion time with Copilot among finishers.
- Limit
- One task; excludes non-completers from the time comparison and does not measure maintenance.
METR experiment · 2025
- Setting
- Experienced maintainers working in familiar repositories.
- Finding
- 19% longer with AI allowed.
- Limit
- Specific early-2025 tools and population; later follow-up has selection bias.
Skill formation · 2026
- Setting
- Learn an unfamiliar library, then take a quiz.
- Finding
- 50% versus 67% on the immediate quiz.
- Limit
- Small tutorial study; long-term retention was not measured.
Claude Code usage · 2026
- Setting
- Observe interactive sessions from existing users.
- Finding
- Inferred expertise correlates with session success.
- Limit
- Classified transcripts, not a causal productivity experiment.
How the story developed
Code suggestions enter the editor
Copilot’s technical preview makes generated code part of the editing loop.
Open event record →A controlled task shows a speedup
The Copilot study reports faster completion of an HTTP-server implementation.
The terminal becomes an agent workspace
Claude Code’s preview combines repository context with editing and execution.
Open event record →A field experiment challenges assumed speedups
METR measures experienced developers doing real work in familiar repositories.
A learning experiment measures comprehension
Anthropic separates immediate mastery from task completion speed.
Open event record →The measurement problem changes
METR describes selection bias and concurrent agent use as obstacles to its follow-up experiment.
Usage research examines expertise
An observational study examines collaboration and inferred success in Claude Code sessions.
What remains open
- Which tasks save time after review and rework are counted?
- How should productivity be measured when a developer supervises several agents?
- Which permissions can a team delegate while keeping failures recoverable?
Newer developments to investigate
Unreviewed research leads after this brief’s evidence cutoff.
Matched “claude code” in the title.
Matched “coding agent” in the title.
Source notebook
What each source supports, and how to read it.
Introducing GitHub Copilot: your AI pair programmer ↗
Establishes the original editor workflow; vendor claims are not independent evaluation.
Claude 3.7 Sonnet and Claude Code ↗
Documents the research preview and its repository tools.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity ↗
A randomized study of 16 developers and 246 tasks; its population and tool period limit generalization.
We are Changing our Developer Productivity Experiment Design ↗
Explains why follow-up results do not provide a reliable current effect size.
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot ↗
One JavaScript HTTP-server task; includes authors affiliated with GitHub and Microsoft.
How AI assistance impacts the formation of coding skills ↗
Randomized tutorial experiment; immediate comprehension is not long-term retention.
Agentic coding and persistent returns to expertise ↗
Transcript classifiers infer expertise and success; no randomized no-AI comparison.
Brief history
2026-09-05 · Initial brief connects the change in coding workflow to both METR studies and their measurement limits.
2026-09-05 · Expanded the evidence through June 2026 with a controlled Copilot experiment, a learning study, and observational agent-usage research. Added a comparison of methods and limitations.