Empirical Software Engineering is the study of what actually works in programming. Instead of trusting our instincts we collect data, run studies, and peer-review our results. This talk is all about how we empirically find the facts in software and some of the challenges we face, with a particular focus on software defects and productivity. Talk doesn’t seem to be online yet; in the meantime, you can see a recording of an older version of the talk here.
Software Engineering Productivity Research - Home
Laws of Software Engineering
A collection of principles and patterns that shape software systems, teams, and decisions.
The Next Two Years of Software Engineering
Exploring five critical questions shaping software engineering through 2026, with contrasting scenarios for each. These lenses help prepare for the evolving ...
How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests
Recent advances in large language models and their rapid adoption across software engineering tasks have made Artificial Intelligence (AI) coding agents an integral component of modern software development workflows. While developers increasingly benefit from these coding agents, their impact on software quality remains insufficiently understood. In particular, how agentic contributions evolve across the software development lifecycle has not been thoroughly investigated. This study aims to characterize agentic pull requests (PR) in comparison to human generated PRs and to examine how their properties change across different stages of the development lifecycle. Using the AIDev dataset, we first analyze how differences in merge rates between agentic and human generated PRs vary over time. We then identify the types of development tasks where AI coding agents are predominantly applied and investigate how these task distributions evolve across development quarters. Finally, we compare a set of key characteristics of agentic and human generated PRs, focusing on their implications for software quality and their temporal dynamics. Overall, our findings provide an empirical and longitudinal perspective on the role of AI coding agents in software development, offering a more nuanced understanding of their benefits and limitations in real-world practices.
How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests
Recent advances in large language models and their rapid adoption across software engineering tasks have made Artificial Intelligence (AI) coding agents an integral component of modern software development workflows. While developers increasingly benefit from these coding agents, their impact on software quality remains insufficiently understood. In particular, how agentic contributions evolve across the software development lifecycle has not been thoroughly investigated. This study aims to characterize agentic pull requests (PR) in comparison to human generated PRs and to examine how their properties change across different stages of the development lifecycle. Using the AIDev dataset, we first analyze how differences in merge rates between agentic and human generated PRs vary over time. We then identify the types of development tasks where AI coding agents are predominantly applied and investigate how these task distributions evolve across development quarters. Finally, we compare a set of key characteristics of agentic and human generated PRs, focusing on their implications for software quality and their temporal dynamics. Overall, our findings provide an empirical and longitudinal perspective on the role of AI coding agents in software development, offering a more nuanced understanding of their benefits and limitations in real-world practices.
Measuring the Impact of Early-2025 AI on Experienced Open-Source...
Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the...
Seeing Like a Programmer (LambdaConf 2024) — Sympolymathesy, by Chris Krycho
How do we make good software, and indeed, what makes software good: both as software, and in terms of its place in the world?
How to create software quality.
I’ve been reading Steven Sinofsky’s Hardcore Software, and particularly enjoyed this quote from a memo discussed in the Zero Defects chapter: You can improve the quality of your code, and if you do, the rewards for yourself and for Microsoft will be immense. The hardest part is to decide that you want to write perfect code. If I wrote that in an internal memo, I imagine the engineering team would mutiny, but software quality is certainly an interesting topic where I continue to refine my thinking. There are so many software quality playbooks out there, and I increasingly believe that all these playbooks work in their intended context, but are often misapplied.
The Evolutionary Ecology of Software: Constraints, Innovation, and the AI Disruption
This chapter investigates the evolutionary ecology of software, focusing on the symbiotic relationship between software and innovation. An interplay between constraints, tinkering, and frequency-dependent selection drives the complex evolutionary trajectories of these socio-technological systems. Our approach integrates agent-based modeling and case studies, drawing on complex network analysis and evolutionary theory to explore how software evolves under the competing forces of novelty generation and imitation. By examining the evolution of programming languages and their impact on developer practices, we illustrate how technological artifacts co-evolve with and shape societal norms, cultural dynamics, and human interactions. This ecological perspective also informs our analysis of the emerging role of AI-driven development tools in software evolution. While large language models (LLMs) provide unprecedented access to information, their widespread adoption introduces new evolutionary pressures that may contribute to cultural stagnation, much like the decline of diversity in past software ecosystems. Understanding the evolutionary pressures introduced by AI-mediated software production is critical for anticipating broader patterns of cultural change, technological adaptation, and the future of software innovation.
Has This Report EXPOSED THE TRUTH About AI Assisted Software Development?
Getting Started with ML and AI in Research Software | Software Sustainability Institute
Getting started with ML in research software means embracing a shift in how results are produced and reproduced. Instead of a fixed execution path, research software teams work with systems whose behaviour emerges from data, configuration, and training dynamics. Reproducibility becomes a matter of capturing the process rather than relying solely on the code. The tools and techniques outlined here can be adopted incrementally into existing projects, and together they provide a practical foundation for reproducible ML research.
The Economics of Software Teams: Why Most Engineering Organizations Are Flying Blind
A breakdown of what software development teams actually cost, what they need to generate to be financially viable, and why most organizations have no visibility into either number.
Notes on software quality
“The absence of problems” is the best definition I can come up with for quality.
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Large Language Model (LLM) agents have been widely adopted in modern software development workflows. SWE-bench [13] and related works [23, 24, 22, 25, 15] establish the task of issue resolution as a de-facto standard for assessing their capability and usefulness. In this setting, an agent is given an entire codebase, a task description (e.g., a bug report or feature request) in natural language and is instructed to produce a code patch that resolves the issue and passes the repository’s test suite. These benchmarks have been instrumental in demonstrating both the substantial potential and the persistent limitations of current models as SWE agents.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
We conduct a randomized controlled trial to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower.
Understanding Spec-Driven-Development: Kiro, spec-kit, and Tessl
Notes from my Thoughtworks colleagues on AI-assisted software delivery
