Does AI make developers 55% faster or 19% slower? Reading both studies closely

"AI makes developers twice as fast" or "AI slows developers down"? Both headlines have research behind them. If you're a Tech Lead or PM deciding whether to invest in AI tools for your team, read the conditions of each study rather than just the number. This post compares the two most-cited controlled experiments and draws out how to measure productivity in your own team.

Study 1: Copilot makes developers 55% faster (2022–2023)

• Peng et al. (with GitHub and Microsoft) recruited programmers, randomly split them into two groups and gave both the same task: write an HTTP server in JavaScript.
• The group using GitHub Copilot finished 55.8% faster.
• GitHub's accompanying survey found most users felt less frustrated and better able to stay focused on repetitive work.

Study 2: Experienced developers 19% slower (METR, 2025)

• METR ran a randomised trial with 16 experienced developers on the large open-source repos they had contributed to for years, covering 246 real issues.
• Each issue was randomly assigned: AI allowed (mostly Cursor with Claude 3.5/3.7 Sonnet) or not.
• With AI allowed, they took 19% longer.
• Most striking: beforehand they predicted AI would speed them up by 24%; afterwards they still believed it had sped them up by about 20%.

Why the opposite results?

METR itself stresses that the result doesn't mean AI is useless for all developers or tasks; it shows that benefits depend heavily on context, and tools change very quickly.

The most important lesson: perception isn't measurement

The gap between "believed 20% faster" and "actually 19% slower" is why you can't base investment decisions on perception surveys. Time spent prompting, waiting, reading and fixing AI output is easy to underestimate.

Measuring productivity in your team

1. Use real work, randomise: same kind of task, half with AI, half without.
2. Measure outcomes, not activity: time from pickup to merge, post-release bugs, review rounds; not lines of code.
3. Split by task type: new code, bug fixes, refactors and test writing give very different results.
4. Track long enough: the first few weeks are a learning period.

Risks and caveats

• Quoting research out of context: using the 55% figure to promise leadership results on legacy maintenance work sets the wrong expectations.
• Ignoring quality: faster with more bugs isn't productivity.
• Results age fast: both studies used the tools of their time; re-measure periodically.

How to apply it

• Dev: Log your actual time on a few tasks with and without AI; you may be surprised.
• Tech Lead: Run a small 4–6 week trial tracking lead time and defect rate before buying tools for the whole department.
• PM: Report AI benefits by task type with conditions, instead of one overall "% productivity" figure.