What We Know We Don't Know: Empirical Software Engineering
Empirical Software Engineering is the study of what actually works in programming. Instead of trusting our instincts we collect data, run studies, and peer-review our results. This talk is all about how we empirically find the facts in software and some of the challenges we face, with a particular focus on software defects and productivity. Talk doesn’t seem to be online yet; in the meantime, you can see a recording of an older version of the talk here.
Remote Work Productivity Study: Surprising Findings From a 4-Year Analysis
Explore key findings from a longitudinal remote work productivity study. Learn how working from home impacts performance and why productivity remains strong.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
We conduct a randomized controlled trial to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower.
AI Code Is Producing a Quality Crisis Nobody Wants to Talk About
The productivity numbers look great. AI coding tools are everywhere.
We are Changing our Developer Productivity Experiment Design
Our second developer productivity study faces selection effects from wider AI adoption, prompting us to redesign our approach.
Does AI Actually Boost Developer Productivity? (100k Devs Study) - Yegor Denisov-Blanch, Stanford
The Next Two Years of Software Engineering
Exploring five critical questions shaping software engineering through 2026, with contrasting scenarios for each. These lenses help prepare for the evolving ...
R. S. Doiel, Software Engineer/Analyst — Robert's ramblings
By R. S. Doiel, 2026-02-21 (revised: 2026-03-03, epilogue added 2026-03-27)
A Guide to Measuring Engineering Team Performance
“You can’t manage what you can’t measure.” While software development practices constantly change, there will always be a tier of truly top engineering teams who stand above…
Working in Public: The Making and Maintenance of Open S…
An inside look at modern open source software developer…
Measuring the Impact of Early-2025 AI on Experienced Open-Source...
Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the...
Software Acceleration and Desynchronization
A look at the ever-present drive to make software delivery faster and how it might break down various activity loops in organizations.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers. 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience. Each task is randomly assigned to allow or disallow usage of early 2025 AI tools. When AI tools are allowed, developers primarily use Cursor Pro, a popular code editor, and Claude 3.5/3.7 Sonnet. Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down. This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). To understand this result, we collect and evaluate evidence for 20 properties of our setting that a priori could contribute to the observed slowdown effect--for example, the size and quality standards of projects, or prior developer experience with AI tooling. Although the influence of experimental artifacts cannot be entirely ruled out, the robustness of the slowdown effect across our analyses suggests it is unlikely to primarily be a function of our experimental design.
How Cursor is building the future of AI coding with Claude
Study finds AI tools made open source software developers 19 percent slower
Coders spent more time prompting and reviewing AI generations than they saved on coding.
