My most recent paper submission (preprint available) is about improving the verifiability of computer-aided research, and contains many references to the related subject of reproducibility. A reviewer asked the same question about all these references: isn't this the same as for experiments done with lab equipment? Is software worse? I think the answers are of general interest, so here they are.
Konrad Hinsen's blog
At the recent SciCodes Symposium, I brought up the question of reviewing research software during the panel discussion. One panelist then raised the question of why we should review research software. I found this question surprising at first, but I do agree that it deserves an answer. Here is mine.
Science is open software
TL;DR I claim that modern science is synonymous with open source software. This post explains why, why it matters, and what you can (and should) do next. Why do you care about (open source) software? - Everyone I spend a lot of my time working on software. I have been asked why software matters more times than I can remember. Software is, people say, not science. It’s a time sink, something to rush past in the pursuit of what really matters: results (and papers if you’re in academia).
Linking the world's research to the code it runs on - OpenAlex blog
Research relies on software. Software written by scientists, for science, runs through the entire modern research stack: NumPy and SciPy, R and ggplot2, Jupyter, BLAST, ImageJ, AlphaFold. Yet in the scholarly record, that software is nearly invisible. Software is not usually cited formally in publications and is usually just mentioned in the text, which means […]
Konrad Hinsen's blog
Thirty years after my first contact with computational (ir)reproducibility, I am happy to note that many things have improved. Reproducibility, computational and otherwise, is increasingly recognized as an important aspect of scientific quality control, and mostly considered worth striving for. However, I also note that more and more people, including reproducibility activists, have lost contact with the day-to-day reality in which reproducibility matters. Reproducibility is becoming an item on a checklist, and its precise incarnation the subject of political bickering aimed at making it easy to check off that item. So let's take a look at why computational reproducibility matters for researchers.
Ten quick tips to SNIFF out sustainable and secure scientific software
Modern computational biology depends heavily on open-source software tools, analysis pipelines, and containerized workflows developed and shared by the research community. While there is extensive guidance (including Quick Tips and Simple Rules articles) on how to build robust and sustainable scientific software, far less has been written for researchers in the role of software users evaluating whether an existing tool is reliable, secure, and sustainable enough for their work. Here we present ten quick tips to help researchers critically assess the tools they adopt. Our tips are organized around a framework that centers on key evaluation features: source, network, interaction, fit, and fragility (SNIFF). These dimensions prompt researchers to consider who maintains a tool and why, whether it is embedded in a broader ecosystem, how actively its developers and users engage, whether it matches the intended use case and licensing requirements, and how robust its dependencies and security practices are. By applying these tips, researchers can make more informed decisions, reduce the risk of relying on abandoned or insecure software, and contribute to a more sustainable scientific software ecosystem.
PRISM: Capturing the Invisible Art of Scientific Practice
A New Tool for Recording and Scaling Laboratory Expertise
Measuring the Impact of Early-2025 AI on Experienced Open-Source...
Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the...
Konrad Hinsen's blog
How can we document software and computational analyses in such a way that others can convince themselves of their validity, and build on them for their own work? The question has been around for many years, and a number of attempts have been made to provide partial answers. This post provides a brief review and describes my own tentative answer, inviting you to play with it.
How to SNIFF out Good Scientific Software
New PLOS Computational Biology paper: "Ten Quick Tips to SNIFF Out Sustainable and Secure Scientific Software"
Improving Science That Uses Code
Abstract. As code is now an inextricable part of science it should be supported by competent Software Engineering, analogously to statistical claims being
The AI Chemist: To be trustworthy, LLMs need to show their work
Good scientists reveal how they do their experiments and report their results; so should any machine-driven research
Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers
The concept of reproducibility can have different interpretations across various research fields and even within the same field [39]. To avoid confusion, we first specify our terms, broadly defining reproducibility and then further categorizing it into various types and degrees. The first distinction comes from Goodman et al. [42], who specify a fundamental division between whether we (i) mean reproducible in principle (termed “methods” reproducibility) due to sufficient description/sharing of methodologies, materials, etc., or (ii) whether results/conclusions actually prove to be reproducible when experiments or analyses are re-done. In the second category, they distinguish “results” and “inferential” reproducibility, depending on whether the analyses or inferences to broader conclusions are reproduced.
Tests Are The New Moat | Daniel Saewitz
As AI becomes better at cloning people's open source work, what ends up becoming most valuable are software contracts, tests, and API surface area. This clashes the incentives of clearly defining your commercialized open source software with protecting it.
Fabricated citations: an audit across 2·5 million biomedical papers
Scientific literature depends on the integrity of its references. Each reference implicitly asserts that a verifiable source exists and supports the claims being made. When references point to non-existent studies, readers, reviewers, and policy makers are unable to evaluate the evidence.
Fabricated citations: an audit across 2·5 million biomedical papers
Scientific literature depends on the integrity of its references. Each reference implicitly asserts that a verifiable source exists and supports the claims being made. When references point to non-existent studies, readers, reviewers, and policy makers are unable to evaluate the evidence.
Computational reproducibility
The why and how of making scientific computations reproducible
Konrad Hinsen's blog
Home Page - Software Heritage
GNU Guix transactional package manager and distribution — GNU Guix
Keynote: Reproducibility and replicability of computer simulations | Canal U
Reproducible research: methodological principles for transparent…
Reproducible Research II: Practices and tools for managing compu…