As AI tools continue to infiltrate software development workflows, a new study from Model Evaluation and Transparency Research (METR) offers a sobering insight: experienced developers may actually be slower when using generative AI tools.
A new METR study reveals that, contrary to popular belief—and even developers’ own perceptions—AI didn’t boost productivity. In real-world programming tasks, it actually caused delays. This has major implications for engineering leaders relying on AI as a force multiplier.
The METR Experiment: Putting AI to the Test in Real Conditions
To evaluate the actual impact of generative AI tools in professional software environments, METR conducted a randomized controlled trial (RCT) in the first half of 2025. Unlike standardized benchmarks like SWE-Bench or RE-Bench, which often isolate tasks from real development contexts, METR embedded their experiment in authentic open-source repositories with over 22,000 GitHub stars and millions of lines of code.
Who Participated?
- 16 experienced developers
- Each with hundreds of hours using AI language models
- Average of 5 years contributing to mature open-source projects
What Was Measured?
- Resolution time across 246 real-world coding tasks (bugs, refactoring, enhancements)
- Performance with and without the use of AI (mainly Cursor Pro using Claude 3.5/3.7)
- Pull request quality and screen-recorded workflows
- Self-reported and observed completion time
Key Finding: AI Slowed Down Performance by 19%
Despite their confidence in the tools, developers working with AI took 19% longer to complete tasks. The result is not just statistically significant — it directly contradicts both user perception and external expert forecasts.
Perceived vs. Actual Performance
- Expected Time Saved with AI (Pre-trial): +24%
- Perceived Time Saved (Post-trial): +20%
- Actual Time Impact: –19% (slower with AI)
- Predicted Time Gains by Experts: Economists: –39%, ML experts: –38%
This disconnect highlights a growing concern in tech: AI tools feel helpful, even when they aren’t.
Why Did Developers Perform Worse with AI? METR’s Five Key Explanations
- Prompting Habits Were Too Basic
Developers often relied on minimal or vague prompts, limiting the usefulness of AI suggestions. - Limited Familiarity with Interfaces
Despite being power users, many participants lacked fluency with Cursor’s deeper features, leading to friction and hesitation. - Code Quality Standards Were High
The studied repositories demanded production-level output, which AI-generated code couldn’t always meet. - AI Models Struggled with Edge Cases
Claude 3.5/3.7 performed well on general tasks but underperformed on complex logic, forcing developers to correct or rewrite code. - Cognitive Distraction and Experimentation
Constantly evaluating AI output may have disrupted focus, delaying progress.
Importantly, METR ruled out issues like data collection bias or measurement error. The result? A robust and transparent insight into the friction between AI tools and developer productivity.
Not All Doom and Gloom: Context Matters
The study doesn’t argue that AI is always detrimental. Instead, it calls for nuanced integration.
“Productivity gains are neither immediate nor automatic,” the study concludes. “They depend on a fine adjustment between tool, task, and professional context.”
In lower-stakes environments or for routine tasks like boilerplate generation or documentation, AI may still deliver value. But for high-standard, real-world programming environments, expert developers may need to recalibrate how, when, and why they use AI.
Lessons for Engineering Teams and AI Tool Builders
What Engineering Managers Should Consider
- Don’t assume AI is a universal booster — tailor its use to task complexity.
- Prioritize training on effective prompting and tool fluency.
- Encourage developers to log and reflect on AI-assisted vs. manual work.
- Use real-time performance metrics to evaluate tool effectiveness over perceived speed.
For AI Tool Developers
- Build better context awareness across large codebases.
- Improve error detection and trust calibration.
- Help users understand where AI adds value—and where it doesn’t.
Conclusion: A Wake-Up Call for AI Enthusiasts in Development
The METR study is a crucial reminder that technology’s perceived value doesn’t always match its actual utility. While AI coding tools like Cursor and Claude are powerful, they require the right context, skill, and expectations.
For senior developers and engineering leaders, this is a cue to pause and re-evaluate: is AI saving us time—or costing us clarity?
Until AI tools mature to meet the full complexity of human-led software development, human judgment still leads the way.
