Dev Workflow Enhancers · Testing Methodology
Human vs AI Coding: The Real Productivity Test Results
We built a repeatable test protocol, ran it, and then cross-checked our numbers against every major published study — including the trial that found AI made experienced developers 19% slower. Here’s the full methodology and what it actually means for your workflow.
Ask ten developers whether AI made them faster this week and you’ll get ten confident, contradictory answers. Ask the research and the picture gets even messier: one rigorous trial says AI made professional coders 55% faster on a bounded task. Another, run two years later on real production codebases, found the opposite — developers were 19% slower with AI tools, despite walking away convinced they’d saved time. Both studies used real developers, real code, and sound experimental design. So which one is lying? Neither. They tested different things, and that’s exactly the trap most “AI vs human” comparisons fall into.
This report is our attempt to fix that. We’re not here to add a tenth hot take about whether AI coding assistants are overhyped or underrated. We built a test protocol you can actually replicate on your own team, ran it against a batch of real tasks, and laid it next to the most credible published research we could find — including the METR randomized controlled trial, the Microsoft/GitHub Copilot experiment, and Google’s 2025 DORA report. What follows is the methodology, the numbers, and the framework we now use every time we evaluate a tool for the AI-Support Tools section of this site.
The problem: every AI productivity claim contradicts another
Start with the two data points that anchor this whole debate. In 2023, Microsoft and GitHub researchers ran a controlled experiment where 95 professional developers were asked to build an HTTP server in JavaScript as fast as possible. The group using GitHub Copilot finished the task 55.8% faster than the control group, a gap wide enough to be both statistically and practically significant1.
Then, in July 2025, the nonprofit AI-evaluation lab METR published a very different kind of study. Instead of a single bounded task, they recruited 16 experienced open-source developers and let them work on 246 real issues — bug fixes, features, and refactors — inside codebases they had personally maintained for an average of five years. Each task was randomly assigned to an “AI allowed” or “AI disallowed” condition. The result: developers using AI tools took 19% longer to finish their tasks, not less2. Before the study, those same developers had predicted AI would cut their time by 24%. Afterward, they still believed it had made them roughly 20% faster. The stopwatch disagreed with both estimates.
Neither study is wrong. They simply measured different conditions: novel task versus familiar codebase, greenfield code versus a million-line legacy repository, a JavaScript beginner-friendly server versus production bug fixes with real stakes. That gap is the entire reason “how we tested the tools” matters more than the headline number. A test protocol that ignores task type will always produce a number that’s technically true and practically misleading.
The Perception-Performance Gap, explained
The most useful finding to come out of the METR trial wasn’t the 19% slowdown itself — it was the three-way mismatch between forecast, feeling, and fact. We call this the Perception-Performance Gap, and it’s the single biggest reason self-reported productivity surveys are unreliable evidence on their own.
Original framework: the Perception-Performance Gap
Three numbers, one task. Forecast — what a developer predicts before starting (in METR’s trial, a 24% expected speedup). Feeling — what they report immediately after finishing (a 20% felt speedup). Fact — what a stopwatch or version-control timestamp actually measured (a 19% measured slowdown). When you only collect forecast or feeling data — which is what most vendor case studies and developer surveys do — you’re measuring how AI changes the experience of coding, not how it changes the outcome. Both are worth knowing. Only one tells you whether to trust the tool with your deadline.
Why does the gap exist at all? A few mechanisms show up consistently across the research we reviewed:
- Typing feels like the bottleneck, but it usually isn’t. Watching code appear instantly creates a strong sense of progress, even when the actual time sink is reviewing, correcting, or re-prompting that code.
- Review time is invisible in the moment. Verifying that AI-generated code is correct, secure, and idiomatic is real work, but it doesn’t feel like “coding,” so developers under-count it when estimating their own speed.
- Novelty inflates satisfaction. Anthropic’s own internal research on AI-assisted work found that heavy AI use could measurably reduce conceptual understanding, code-reading skill, and debugging ability, even when developers reported feeling more capable.
- Familiar codebases punish shortcuts. The METR developers had years of tacit knowledge about their own repositories. An AI assistant with no memory of that history sometimes proposed changes that looked plausible but required extra correction — a hidden tax that a stopwatch catches and a survey does not.
The gap between how fast AI feels and how fast it actually is isn’t a rounding error — in the METR data it was a 39-percentage-point swing between forecast and measured result. That’s the size of gap that determines whether a sprint deadline gets hit.
Our testing protocol: the Matched-Task Method
Once we understood the Perception-Performance Gap, it was obvious that a simple “try the tool for a week and see how it feels” test was going to produce a biased answer before we even started. So we built a protocol designed to strip out the two biggest sources of noise: task difficulty variance and self-report bias. We’re calling it the Matched-Task Method, and it borrows directly from the randomized-controlled-trial structure that made the METR and GitHub Copilot studies credible in the first place.
Step 1 — Pull tasks from a real backlog, not a benchmark
Synthetic coding benchmarks are convenient but they don’t capture the messiness of production work: undocumented edge cases, inconsistent naming, half-finished migrations. We sourced tasks from an active backlog spanning four categories — boilerplate/scaffolding, bug fixes, small feature additions, and documentation — because those categories show the most divergent results across the published literature.
Step 2 — Match tasks in pairs by estimated difficulty
For every task worked with AI assistance, we paired it with a same-category task of comparable estimated size, then randomly assigned which one got AI support. This mirrors METR’s randomization design and avoids the classic mistake of comparing an easy AI-assisted task against a hard unassisted one.
Step 3 — Measure wall-clock time, not developer estimates
Every task was timed from first commit to merged, reviewed pull request — not from a self-reported “how long did that take you” survey. This single change is what separates a rigorous test from a vendor testimonial.
Step 4 — Run a blind second-pass code review
A separate reviewer, unaware of which tasks used AI assistance, scored each pull request for correctness, test coverage, and rework needed after initial review. Speed without a quality check just measures how fast bugs get shipped.
Step 5 — Separate forecast, feeling, and fact
Before each task, the developer estimated expected completion time. After, they rated perceived difficulty and speed. We then compared all three against the measured result — the same three-way comparison that exposed the Perception-Performance Gap in the METR data.
Why this protocol, and not a simpler one
A before/after comparison on the same developer doing the same task twice is contaminated by learning effects — the second attempt is faster regardless of the tool. A team-wide survey is contaminated by the Perception-Performance Gap. Randomized, matched-pair task assignment with blind review is the closest a small-scale test can get to the design that made the GitHub Copilot and METR trials credible enough to publish.
What the data shows, task by task
The clearest signal — from our own matched-pair testing and echoed across the wider literature — isn’t “AI is faster” or “AI is slower.” It’s that the effect flips sign depending on task type. Here’s the pattern that shows up consistently:
Where AI assistance wins clearly
- Boilerplate and scaffolding. A separate study of 15 real development tasks found roughly 50% time savings on documentation and autocompletion work specifically, the category where AI has the least room to introduce subtle logic errors.
- Unfamiliar languages or frameworks. An earlier controlled study found that AI assistance helped non-programmers reach performance levels close to experienced programmers, and gave a meaningful speed boost to programmers working outside their usual stack.
- Bounded, well-specified greenfield tasks. This is exactly the condition in the GitHub Copilot HTTP-server experiment — a fresh file, a clear spec, no legacy context to reconcile.
Where AI assistance stalls or reverses
- Large, mature, unfamiliar-to-the-AI codebases. The METR trial’s 19% slowdown occurred specifically in repositories averaging over a million lines of code, where developers already held deep tacit knowledge the AI had no access to.
- Complex refactors and architectural decisions. The same research found that large functions and complex, multi-file logic caused projected time savings to drop sharply once real-world tasks replaced simple ones.
- Anything requiring careful verification. Every study we reviewed that measured code quality alongside speed found a “verification tax” — time spent checking AI output for correctness, security issues, or subtle bugs — that eats into the raw speed gain, sometimes enough to erase it entirely.
A pattern worth remembering
Across every credible study we could find, AI’s advantage shrinks as task familiarity and codebase complexity rise, and grows as task novelty and boilerplate share rise. That single relationship explains almost every “contradictory” headline in this space.
The Task-Complexity Curve framework
To make that relationship usable — not just descriptive — we sketch it as what we call the Task-Complexity Curve. It’s a planning heuristic, not a measured regression line: a way to predict, before you start a task, whether AI assistance is likely to help or hurt, based on two variables — how well-specified the task is, and how much tacit context about your specific codebase it requires.
Use the curve as a pre-task gut check, not a formula: the further left a task sits — small, well-specified, low tacit-knowledge requirement — the more confidently you can lean on AI assistance and trust your speed instinct. The further right it sits, the more that same instinct is exactly what the Perception-Performance Gap will distort, and the more a second human reviewer earns their keep.
How the major published studies compare
Here’s the full picture side by side. We include the study design because, as this whole report argues, the design determines the answer more than the tool does.
| Study | Design | Task type | Result |
|---|---|---|---|
| Peng et al., Microsoft Research (2023) | Randomized controlled trial, 95 developers | Bounded greenfield task — HTTP server in JavaScript | +55.8% faster with GitHub Copilot |
| Chatterjee et al., ANZ Bank field study | Six-week workplace experiment | Real banking software tasks, mixed seniority | +42.4% average faster with Copilot |
| METR RCT (2025) | Randomized controlled trial, 16 experienced developers, 246 tasks | Real issues in mature open-source repos (5+ yrs familiarity) | −19% slower with AI tools |
| Google DORA (2025) | Global survey, ~5,000 professionals | Mixed — organizational throughput, not individual time | 90% daily AI adoption; gains concentrated in high-maturity teams |
| Pandey et al., real-codebase study | 15 development tasks, live codebase | Documentation & autocompletion vs. complex functions | ~50% savings on docs; dropped to 33–36% on complex tasks |
Read across the table and the disagreement mostly dissolves. The studies showing large gains tested bounded, well-specified, or low-context tasks. The study showing a loss tested real production work inside codebases the developers had spent years internalizing. Our own matched-task results land exactly where that pattern predicts: solid gains on scaffolding and unfamiliar-stack work, a wash or a mild loss on complex refactors inside our own long-running projects.
How to run this test on your own team
You don’t need a research grant to get a credible answer for your own codebase — you need discipline about what you measure. Here’s the condensed checklist version of our protocol:
- Pull 10–20 real backlog tickets, spread across boilerplate, bug-fix, feature, and documentation categories.
- Pair tickets by estimated size within each category, then randomly assign AI-allowed vs. AI-disallowed.
- Track wall-clock time from first commit to merged PR — not developer self-estimates.
- Have a second reviewer, blind to which condition each PR came from, score correctness and rework needed.
- Separately record each developer’s pre-task forecast and post-task felt-speed rating, so you can see your own Perception-Performance Gap.
- Segment your results by task type before drawing any conclusion — an aggregate number across boilerplate and legacy refactors will hide the real signal.
One more thing worth building into any internal test: revisit it. The tools move fast enough that even METR found its own methodology needed reworking within a year, after newer agentic tools like Claude Code and Codex changed how developers actually used AI assistance day to day. A testing protocol is not a one-time verdict — it’s a habit. For a deeper look at where teams tend to blur “using a tool well” with “automating a process,” our piece on workflow enhancement versus automation is a useful companion read before you scale any of this beyond a single test.
If your test is specifically about editor-level assistance rather than end-to-end task completion, it’s also worth separating “does this tool speed up typing” from “does this tool speed up shipping.” Our roundup of open-source workflow tools and our list of JavaScript snippets that actually saved real project time are both good low-context, high-boilerplate categories to start with if you want an early, encouraging result before tackling harder tasks.
Frequently asked questions
Do AI coding tools actually make developers faster?
It depends heavily on the task. Controlled trials on bounded, well-specified tasks like building a small HTTP server show large speed gains from tools like GitHub Copilot. But the 2025 METR randomized trial found that experienced developers working in large, familiar codebases were about 19% slower with AI tools, even though they believed AI had sped them up. AI helps most on unfamiliar syntax and boilerplate, and helps least on complex, high-context work.
Why do developers feel faster with AI even when they are not?
Researchers call this the Perception-Performance Gap. Typing less and watching code appear feels like progress, even when time is lost reviewing, correcting, or re-prompting the AI’s output. In the METR trial, developers predicted a 24% speedup before starting, reported feeling roughly 20% faster after finishing, and were measured at 19% slower overall.
What is the best way to test an AI coding tool for yourself?
Run a matched-pairs test: pick real, similar-sized tasks from your own backlog, randomly assign half to be done with AI assistance and half without, and track wall-clock time plus a separate blind code-review pass. Self-reported speed estimates alone are not reliable evidence.
Which coding tasks benefit most from AI assistance?
Boilerplate generation, test scaffolding, documentation, and working in an unfamiliar language or framework tend to show the clearest gains. Tasks requiring deep familiarity with a large, mature codebase, subtle debugging, or architectural judgment tend to show smaller gains or net slowdowns.
The bottom line
“Human vs AI” was always the wrong framing for this test. The real variable isn’t which side of the matchup wins — it’s which task you hand to which side. Well-specified, low-context, boilerplate-heavy work is where AI assistance earns its keep, often dramatically. Deep, familiar, high-stakes codework is where your own judgment, plus a healthy dose of skepticism toward how fast the tool feels, still wins. Build a test that separates those two conditions, measure wall-clock time instead of vibes, and the contradictory headlines stop looking contradictory at all.
Sources
- Peng, S. et al. “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot.” Microsoft Research, 2023. microsoft.com/research
- Becker, J., Rush, N., Barnes, E., Rein, D. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” METR, July 2025. metr.org · arXiv:2507.09089
- METR. “We Are Changing Our Developer Productivity Experiment Design.” February 2026. metr.org
- DORA / Google Cloud. “2025 State of AI-assisted Software Development Report.” research.google