Anthropic Study: How AI Assistance Affected Coding Skills

The test came after the code

Anthropic studied 52 developers who knew Python but had not used Trio, an asynchronous programming library. Participants completed two coding tasks, with or without an AI assistant, then took a quiz on the concepts they had just used. [1]

The tasks resembled learning from a tutorial: participants received starter code and explanations, then implemented features. Assessment emphasized reading code, finding errors and understanding concepts. Finishing the implementation and understanding it were separate outcomes. [3]

The AI group averaged 50% on the immediate quiz, compared with 67% for the group without AI: a 17-percentage-point gap. The difference was statistically significant. AI participants finished about two minutes faster on average, but that time difference was not statistically significant. [1]

The way participants asked for help mattered

Researchers examined screen recordings and grouped AI use by behavior. Some participants delegated the implementation and moved on. Others asked questions about generated code or requested conceptual explanations while doing the coding themselves. The latter patterns were associated with stronger quiz performance. [3]

Those groups were small and were identified from observed behavior, not randomly assigned learning strategies. The analysis suggests useful habits to investigate; it does not prove that a particular prompting technique caused better learning. [1]

Atlas interpretation: For someone learning an unfamiliar library, the practical distinction is whether the assistant removes typing or removes the need to think through the program. Asking why a task waits at a particular point, then predicting what changes if that wait is removed, creates a check on understanding that a finished implementation cannot supply. [3]

A question for coding agents, not a verdict on them

The experiment used a sidebar chat assistant, not Claude Code. It did not measure the effects of delegating an entire repository task to an agent, and the immediate quiz cannot establish what participants remembered months later. [1]

Atlas interpretation: Teams adopting coding agents need both completed work and people able to diagnose it. This study makes a useful case for checking those outcomes separately during training: ask someone to explain a change or repair a related failure, rather than treating successful task completion as evidence that they learned the underlying system. [3]