Collection
AI and/for science
Updated 4d
Collection
Updated 4d
Simon on Twitter / X
Can AI help connect theorems humans write in papers to proofs computers can check?We just released TheoremGraph (https://t.co/PQ8FcFQGat), and I made a 3Blue1Brown style video overview of the idea.This project was my first real research experience, and it meant a lot. Start… pic.twitter.com/yMUA0QziXM— Simon (@waskaja) June 29, 2026
Przemek Chojecki | PC on Twitter / X
The Growing Map of Open Mathematical Problems.We mapped 15,000+ conjectures from UnsolvedMath to show potential links between concepts.It also shows how under formalized the frontier is (less than 10%). pic.twitter.com/nm0PCpXVPf— Przemek Chojecki | PC (@prz_chojecki) August 28, 2026
AI Simulation Platform for Human Behavior | Simile
Simulate how real customers respond to a launch, price change, or campaign — before you ship. Built by the Stanford researchers behind generative agents.
AI for science: What can it do? Can it do things? Let’s find out!
A what we’re reading spotlight
OSF
Previewing the Model Hardware Standard
Anthropic is opening a research preview of the Model Hardware Standard (MHS), a shared specification for AI agents to safely operate physical devices, to a first group of scientific research labs and advanced manufacturers.
On genAI: Was prototyping really a bottleneck?
I keep hearing folks claim that the fact we can ‘prototype’ so quickly now is a good thing (thanks to modern genAI). But what if the slow parts about prototyping are actually what make it worth doing?
Aniket Panjwani on Twitter / X
Here's something I don't understand about Refine.1 review costs $50 and 10 reviews cost $300. So, unless they're taking a loss, average cost of inference on a Refine review is no more than $30.On a ChatGPT Pro plan, for $200/mo, you're getting the equivalent of ~$14k/mo in…— Aniket Panjwani (@aniketapanjwani) August 20, 2026
Q&A from the slop trenches – GeoSpatial ML
We reviewed 22 ML conference submissions this summer. Fifteen had fabricated citations, hallucinated authors, or clear LLM slop, so we complain about that a bit, and release the paper references audit we now run.
Can AI agents conduct open-ended AI research? Early evidence from two case studies
Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R\&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper's original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.
Coordination Tech in Science: Letters, Journals, and Whatever Comes Next | shishyko!
To modernize our scientific infrastructure, we need new contextualization and coordination technologies that decouple trust from legacy branding — shifting from gatekeeping on write to algorithmic contextualization on read.
The fall of the theorem economy
How AI could destroy mathematics and barely touch it
A-MEM: Agentic Memory for LLM Agents
While large language model (LLM) agents can effectively use external tools for complex real-world tasks, they require memory systems to leverage historical experiences. Current memory systems enable basic storage and retrieval but lack sophisticated memory organization, despite recent attempts to incorporate graph databases. Moreover, these systems' fixed operations and structures limit their adaptability across diverse tasks. To address this limitation, this paper proposes a novel agentic memory system for LLM agents that can dynamically organize memories in an agentic way. Following the basic principles of the Zettelkasten method, we designed our memory system to create interconnected knowledge networks through dynamic indexing and linking. When a new memory is added, we generate a comprehensive note containing multiple structured attributes, including contextual descriptions, keywords, and tags. The system then analyzes historical memories to identify relevant connections, establishing links where meaningful similarities exist. Additionally, this process enables memory evolution - as new memories are integrated, they can trigger updates to the contextual representations and attributes of existing historical memories, allowing the memory network to continuously refine its understanding. Our approach combines the structured organization principles of Zettelkasten with the flexibility of agent-driven decision making, allowing for more adaptive and context-aware memory management. Empirical experiments on six foundation models show superior improvement against existing SOTA baselines. The source code for evaluating performance is available at https://github.com/WujiangXu/A-mem, while the source code of the agentic memory system is available at https://github.com/WujiangXu/A-mem-sys.
Can AI Agents Synthesize Scientific Conclusions?
Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions. Yet, their ability to do so in high-stakes domains such as health remains unclear. We introduce SciConBench, a large-scale live benchmark of 9.11K questions and expert-written conclusions from systematic reviews to evaluate open-domain scientific conclusion synthesis. The benchmark draws on an expert-validated automated evaluation pipeline that decomposes conclusions into atomic facts and measures correctness and comprehensiveness via factual precision and recall. To mitigate data leakage, we further introduce SciConHarness, a clean-room evaluation harness that equips agents with controlled web interaction to ensure valid measurement. Evaluating 8 frontier models and deep research agents, we find that factual quality remains low: under clean-room settings, the best agent achieves only a factual F1 of 0.337. Our clean-room setting consistently reduces performance relative to unconstrained evaluation, suggesting that leakage inflates estimates of models' true synthesis capabilities. Finally, we audit consumer-facing agents (e.g., Google AI Overview, OpenEvidence) and find they frequently generate incomplete and sometimes contradictory conclusions, even when the ground-truth answer is available. Overall, our results show that reliable synthesis of scientific conclusions remains an open challenge, and that clean-room evaluation is essential for assessing open-domain AI agents.
