ReproAI: the plugin for reproducible replication packages

Training AI Scientists to Replicate Research

The replicability of papers is a cornerstone of scientific knowledge, ensuring the reliability of existing results and providing a base for further experiments. The act of replication typically illuminates details that were previously underspecified, and thus requires similar hypothesis-driven exploration to open-ended research. In this work, we develop Replica, a scalable task space for paper replication. To provide reward signal, we introduce an auto-generated rubric-based judge that has low noise and agrees with human assessment of replication quality. We post-train Faraday, a 27B-parameter "AI Scientist" agent that leverages coding agents as tools, surpassing the performance of Claude Opus 4.8 and GPT-5.5 on held-out replication tasks. Qualitative analysis of individual rollouts reveals that Faraday adopts a more scientifically-principled approach. We believe that our results provide a stepping stone towards AI agents capable of long-horizon scientific innovation without requiring complex harnesses.

Keynote: Reproducibility and replicability of computer simulations | Canal U

Since the early days of the reproducibility crisis, much progress has been made in understanding and improving computational reproducibility and replicability (R and R)...

Getting Started with ML and AI in Research Software | Software Sustainability Institute

Getting started with ML in research software means embracing a shift in how results are produced and reproduced. Instead of a fixed execution path, research software teams work with systems whose behaviour emerges from data, configuration, and training dynamics. Reproducibility becomes a matter of capturing the process rather than relying solely on the code. The tools and techniques outlined here can be adopted incrementally into existing projects, and together they provide a practical foundation for reproducible ML research.

Git AI - Track AI Code all the way to production

Cross-agent observability from prompt to production. Track AI-generated code from Cursor, Claude Code, GitHub Copilot, Gemini, and more through the entire SDLC.

Towards End-to-End Automation of AI Research

The automation of science is a long-standing ambition in the field of AI. While the community has made significant progress in automating individual components of the scientific process, a system that autonomously navigates the entire research lifecycle -- from conception to publication -- has remained out of reach. Here, we present the strongest demonstration to date toward automating the entire process end-to-end. We present The AI Scientist, which creates research ideas, writes code, runs experiments, plots and analyzes data, writes the entire scientific manuscript and performs its own peer review. Its ideas, execution, and presentation are of sufficient quality to produce a manuscript generated by an AI system that passes the first round of peer review at a major machine learning conference workshop. The workshop has an acceptance rate of 70 percent. Our system leverages modern foundation models within a complex agentic system. We evaluate The AI Scientist in two settings: a focused mode using human-provided code templates as an initial scaffold to conduct research on a specific topic, and a template-free, open-ended mode that leverages agentic search for wider scientific exploration. Both settings produce diverse ideas and automatically test, report on, and evaluate them. This achievement demonstrates AI's growing capacity for scientific contribution and signifies a potential paradigm shift in how research is conducted. As with any impactful new technology, there could be significant risks, including taxing overwhelmed review systems and adding noise to scientific literature. However, if developed responsibly, such autonomous systems could greatly accelerate scientific discovery.

Agent Plugins

A portable package format for reusable components that extend AI agents.

GitHub - Responsible-Dataset-Sharing/easy-dataset-share: A CLI tool that helps AI researchers share datasets responsibly.

A CLI tool that helps AI researchers share datasets responsibly. - Responsible-Dataset-Sharing/easy-dataset-share

Managing Changes from Multiple AI Agents

In this video, I walk you through managing changes from multiple AI agents in our to-do app, specifically focusing on the Expert to CSV button. We explore design options using ground agents, where one agent proposes a green design and the other a blue one. After some time, both agents complete their tasks, and their changes are synchronized to GitHub and our local machine. I demonstrate the final designs and decide to ship the green version as the main one. I encourage you to apply this approach in your own projects and consider using rebase in GitHub for similar tasks.

The Last Human-Written Paper: Agent-Native Research Artifacts

Scientific publication compresses a branching, iterative research process into a linear narrative, discarding the majority of what was discovered along the way. This compilation imposes two structural costs: a Storytelling Tax, where failed experiments, rejected hypotheses, and the branching exploration process are discarded to fit a linear narrative; and an Engineering Tax, where the gap between reviewer-sufficient prose and agent-sufficient specification leaves critical implementation details unwritten. Tolerable for human readers, these costs become critical when AI agents must understand, reproduce, and extend published work. We introduce the Agent-Native Research Artifact (ARA), a protocol that replaces the narrative paper with a machine-executable research package structured around four layers: scientific logic, executable code with full specifications, an exploration graph that preserves the failures compilation discards, and evidence grounding every claim in raw outputs. Three mechanisms support the ecosystem: a Live Research Manager that captures decisions and dead ends during ordinary development; an ARA Compiler that translates legacy PDFs and repos into ARAs; and an ARA-native review system that automates objective checks so human reviewers can focus on significance, novelty, and taste. On PaperBench and RE-Bench, ARA raises question-answering accuracy from 72.4% to 93.7% and reproduction success from 57.4% to 64.4%. On RE-Bench's five open-ended extension tasks, preserved failure traces in ARA accelerate progress, but can also constrain a capable agent from stepping outside the prior-run box depending on the agent's capabilities. Our code is open-sourced at https://github.com/Orchestra-Research/Agent-Native-Research-Artifact.

NBER on Twitter / X

An LLM workflow that reproduces, improves, and extends economics research has been applied to over 4,000 published replication packages, from Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro https://t.co/jrWNP9wOlf pic.twitter.com/ZX6Eem16vo— NBER (@nberpubs) September 28, 2026

Antithesis: autonomous software testing

Try the Antithesis autonomous testing platform and find bugs in your software with perfect reproducibility

GRN · German Reproducibility Network

Working together for trustworthy and useful research

Software Is Made Between Commits

From the Zed Blog: Agents turned the conversation into the real source of our software. DeltaDB is the version control built for it.

PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing

Synthesizing unstructured research materials into manuscripts is an essential yet under-explored challenge in AI-driven scientific discovery. Existing autonomous writers are rigidly coupled to specific experimental pipelines, and produce superficial literature reviews. We introduce PaperOrchestra, a multi-agent framework for automated AI research paper writing. It flexibly transforms unconstrained pre-writing materials into submission-ready LaTeX manuscripts, including comprehensive literature synthesis and generated visuals, such as plots and conceptual diagrams. To evaluate performance, we present PaperWritingBench, the first standardized benchmark of reverse-engineered raw materials from 200 top-tier AI conference papers, alongside a comprehensive suite of automated evaluators. In side-by-side human evaluations, PaperOrchestra significantly outperforms autonomous baselines, achieving an absolute win rate margin of 50%-68% in literature review quality, and 14%-38% in overall manuscript quality.

AI-Assisted Tools in the CHI 2027 Papers Review Process: What They Do, What They Do Not Do, and How Humans Remain Responsible - ACM CHI 2027

AI-Assisted Tools in the CHI 2027 Papers Review Process: What They Do, What They Do Not Do, and How Humans Remain Responsible Authors: Anna Cox, Tony Tang, Thomas Kosch, Erin Solovey, Petra Isenberg, Regan MandrykDate: 2026-08-29 Introduction In our recent post explaining the CHI 2027 Papers review process, we said we would pilot several AI-assisted […]