Prompting Science Report 4: Playing Pretend: Expert Personas Don't Improve Factual Accuracy

Technical Report: Playing Pretend: Expert Personas Don't Improve Factual Accuracy

[vc_row martech_row_background_position=”None” css=”.vc_custom_1742942106856{margin-bottom: 24px !important;}”][vc_column][/vc_column][/vc_row][vc_row martech_row_background_position=”None” css=”.vc_custom_1750970721780{margin-bottom: 40px !important;}”][vc_column width=”5/6″ css=”.vc_custom_1750970738029{margin-bottom: 40px !important;}”][vc_column_text css=”.vc_custom_1764963741477{margin-bottom: 1em !important;}”]This study investigates whether persona prompting improves AI performance on challenging academic benchmarks. We find that despite widespread adoption, assigning expert personas (e.g., “You are a world-class physics expert”) does not reliably improve accuracy. Domain-mismatched…Read More

Prompting Science Report 4: Playing Pretend: Expert Personas Don't Improve Factual Accuracy

<span> <p><span>This is the fourth in a series of short reports that help business, education, and policy leaders understand the technical details of working w

Prompting Science Report 4: Playing Pretend: Expert Personas Don't Improve Factual Accuracy

<span> <p><span>This is the fourth in a series of short reports that help business, education, and policy leaders understand the technical details of working w

Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Tasks

Detailed profession-specific system prompts raise token use and estimated cost per response without a consistent accuracy gain. We evaluate Scientific Agents, an open-source corpus of 503 profession-specific AGENTS.md profiles, with Gemini 3.8 Flash via OpenRouter in the Pi agent harness. We compare matched profiles with four controls: a minimal baseline ("You are a helpful assistant"), the profile's opening role sentence, a generic scientific rigor guide, and a profile from an unrelated domain. Across nine text-based science benchmarks (4,531 sampled questions, 100 matched profiles), 4,488 items completed all five conditions after API-error retries, scored with automated, rule-based grading. The average profile-baseline accuracy difference is -0.6 percentage points (95% bootstrap interval [-1.5, +0.2] across fixed tasks), and no benchmark shows a statistically clear improvement. Matched profiles produced 1.5-2.3 times as many output tokens and cost 2.2-4.5 times more per successful call. On 60 tool-using BioMysteryBench bioinformatics problems (three runs each for baseline and profile), mean solve rates were 46.7% with the profile and 56.7% at baseline, a difference of -10.0 percentage points (95% interval [-16.7, -3.3]) driven by more frequent token- and time-limit stops under the profile. Longer prompts had one unexpected operational advantage: on SuperGPQA, frequent provider API drops left the short baseline with a correct first-pass answer on only 54.0% of items, against 71.6% with the profile. Generic and mismatched prompts were about as reliable, so this gain comes from prompt length or formatting rather than domain expertise. For the tested model and tasks, loading full profession profiles by default does not improve accuracy and costs considerably more; whether selective retrieval of profile sections or open-ended scientific tasks would change this remains to be tested.

Large language models eroding science understanding: an empirical study of ‘malignment’

Large Language Models (LLMs) increasingly provide scientific information to non-specialist users, yet their linguistic fluency can create an impression of expertise that exceeds their capacity for scientific judgement. This paper experimentally investigates how readily an LLM can be configured to present fringe scientific claims as authoritative knowledge. We constructed two RAG-like Custom GPTs using GPT-5.1, one concerned with the fine-structure constant and the other with gravitational waves. In each case, ten papers selected by domain experts from the alternative preprint repository viXra were supplied as a domain-specific knowledge base, together with instructions directing the model to treat this material as authoritative. The underlying model was neither retrained nor fine-tuned. Expert-generated questions were then posed to the resulting “Fringe-LLMs”, to the unmodified model, and to domain experts. The Fringe-LLMs produced fluent, detailed and apparently authoritative answers that systematically reflected the fringe corpus, while expert assessment found many of these answers to be scientifically incorrect or seriously misleading. By contrast, the unmodified model generally produced answers substantially closer to mainstream scientific understanding. We describe this deliberate redirection of model output as malignment. The experiment demonstrates that scientifically misleading LLMs can be created with a low technical barrier and without altering model weights or poisoning training data. Because non-specialists may be unable to distinguish such outputs from genuine expertise, the results raise important concerns about the use of LLMs as intermediaries for scientific knowledge and reinforce the continuing importance of expert human judgement and provenance.

We&#39;re releasing Terminal-Bench-Science: a benchmark for evaluating AI agents on research workflows across scientific domains. 🚀 An ongoing community effort out of Stanford University &amp; Laude… | Steven Dillmann | 13 comments

We're releasing Terminal-Bench-Science: a benchmark for evaluating AI agents on research workflows across scientific domains. 🚀 An ongoing community effort out of Stanford University & Laude Institute, built by the team behind Terminal-Bench together with scientific domain experts at research institutions worldwide. 🧪 ⚛️ 🌍 ➕ ⚙️ Terminal-Bench-Science 0.1 has 70 tasks across the life, physical, earth, mathematical, and engineering sciences. Each one is contributed by a researcher porting their own workflows into challenging agentic tasks, on problems they actually care about. 🧠 Anthropic's Opus 5 dominates most scientific domains with an overall pass rate of ~30%, except the mathematical sciences, where Fable 5 and OpenAI's GPT-5.6 Sol take the lead. SpaceXAI's Grok 4.6 is one of the best models for engineering tasks, Z.ai's GLM 5.3 is the best open model, and Kimi (Moonshot AI)'s Kimi K3 is the only model beyond Opus 5 that lands on both cost and token efficiency Pareto frontiers. 🏆 All leaderboard results are live here: https://lnkd.in/gYh99UfU 💻 You can find all tasks on our Github: https://lnkd.in/gejKmXpj Special thanks to our amazing reviewer team Allen Hart, Ivan Bercovich, Joe Janssen, Jiaming H., Steffen Bollmann, Sergey Aganezov, Ben Blaiszik, Karl Krauth, Aaron Feller, Ben Blaiszik, Yiqi Liu, Han Hou, Robert Joseph G., Robert Zhang; our AI research advisors Ryan Marten, Alex Shaw, Lin Shi, Ben Feuer, Mike Merrill, Gian Segato, Jenia Jitsev, Bodhisattwa Majumder, Peter Clark, Thomas Wolf, Braden Hancock, Andy Konwinski, Alex Dimakis; and especially to our co-leads Sanmi Koyejo and Ludwig Schmidt. | 13 comments on LinkedIn

Accelerating Scientific Research with Gemini: Case Studies and Common Techniques

Recent advances in large language models (LLMs) have opened new avenues for accelerating scientific research. While models are increasingly capable of assisting with routine tasks, their ability to contribute to novel, expert-level mathematical discovery is less understood. We present a collection of case studies demonstrating how researchers have successfully collaborated with advanced AI models, specifically Google's Gemini-based models (in particular Gemini Deep Think and its advanced variants), to solve open problems, refute conjectures, and generate new proofs across diverse areas in theoretical computer science, as well as other areas such as economics, optimization, and physics. Based on these experiences, we extract common techniques for effective human-AI collaboration in theoretical research, such as iterative refinement, problem decomposition, and cross-disciplinary knowledge transfer. While the majority of our results stem from this interactive, conversational methodology, we also highlight specific instances that push beyond standard chat interfaces. These include deploying the model as a rigorous adversarial reviewer to detect subtle flaws in existing proofs, and embedding it within a "neuro-symbolic" loop that autonomously writes and executes code to verify complex derivations. Together, these examples highlight the potential of AI not just as a tool for automation, but as a versatile, genuine partner in the creative process of scientific discovery.

ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI

Scientific discovery is defined by the ability to identify the boundaries of existing knowledge and venture into unexplored territory. The ultimate vision for AI in science is problem-driven autonomous research: given a fundamental challenge by a human expert, the AI independently navigates the scientific landscape, uncovers theoretical and empirical bottlenecks, and systematically expands the frontier of knowledge. In this paper, we introduce ScientistTwo, a fully autonomous multi-agent framework designed to realize this vision. Specifically, ScientistTwo takes an initial problem as input, establishes state-of-the-art baselines, formulates novel hypotheses, and coordinates specialized agents to orchestrate an end-to-end discovery cycle without human intervention. Moreover, the framework rigorously conducts experiments using diverse datasets and metrics, refines methodologies through automated ablation studies, and validates research findings via a closed-loop simulated peer-review rebuttal engine. To evaluate ScientistTwo's capabilities against the highest standards of human scientific achievement, we benchmark it across papers accepted at top-tier conferences such as ICLR, ICML, and NeurIPS. As a result, ScientistTwo autonomously generates expert-level, publishable papers and fully verified, executable codebases. Its solutions consistently outperform human state-of-the-art models, and achieve higher average review ratings than human-authored papers under automated AI review agents. These results show that ScientistTwo is not merely an assistive tool but an autonomous scientific pioneer capable of pushing the frontiers of human discovery. Project website: https://scientist-two.github.io/

A Design Science Research Methodology for Expert Systems Development

The knowledge of design science research (DSR) can have applications for improving expert systems (ES) development research. Although significant progress of utilising DSR has been observed in particular information systems design – such as decision support systems (DSS) studies – only rare attempts can be found in the ES design literature. Therefore, the aim of this study is to investigate the use of DSR for ES design. First, we explore the ES development literature to reveal the presence of DSR as a research methodology. For this, we select relevant literature criteria and apply a qualitative content analysis in order to generate themes inductively to match the DSR components. Second, utilising the findings of the comparison, we determine a new DSR approach for designing a specific ES that is guided by another result – the findings of a content analysis of examination scripts in Mathematics. The specific ES artefact for a case demonstration is designed for addressing the requirement of a ‘wicked’ problem in that the key purpose is to assist human assessors when evaluating multi-step question (MSQ) solutions. It is anticipated that the proposed design knowledge, in terms of both problem class and functions of ES artefacts, will help ES designers and researchers to address similar issues for designing information system solutions.

EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights

When Isaac Newton discovered the law of gravitation, he did so through an iterative process of analyzing observed data such as planetary patterns, finding the underlying mechanisms by describing patterns in mathematical equations, and refining his theory against the Moon's orbit, revealing the startling insight that the same force governs both falling apples and orbiting planets. Would it be possible for AI agents to make similar discoveries? To measure this ability, we introduce EurekaBench, a cross-domain benchmark that tests AI agents' ability to conduct long-horizon experiments and discover mechanisms that explain observations. We evaluate these mechanisms by the scientific insights that can be derived from them. EurekaBench contains an expert-verified set of 26 long-horizon tasks across neuroscience, computer science, chemistry, astrophysics, geophysics, and plasma physics, with a total of 306 scientific insights that the discovered mechanisms are expected to support. Our evaluation framework tests three axes of scientific discovery: agents' ability to follow known scientific constraints, the predictive accuracy of the discovered mechanisms, and whether these mechanisms yield scientific insights or inform future research. Our results show that current AI agents often overly fixate on predictive accuracy optimization, surpassing human scientists, while falling substantially short in deriving scientific insights.

Mathematician catches AI-generated research error by OpenAI president | Benjamin Golub posted on the topic | LinkedIn

In case you think the problem of science slop is hypothetical, here's something that happened on X yesterday: The president of OpenAI sharing wrong science slop as proud evidence of the power of mathematical AI. Within a day, the mathematician Tony Feng found a gap that undercut the paper's central argument; the author conceded, and (to his credit!) Brockman deleted the tweet as soon as he became aware. Feng was a hero to catch this. The author of the refuted claim is a mathematician but a self-described outsider to this area, with a prior big claim posted and quickly withdrawn. Greg Brockman is not an expert, either. Neither could check what they were amplifying. Heroes catching errors one at a time won't hold against the volume of AI-generated research now coming. We need an automated, independent verification layer for knowledge work, which is what we're building at Refine (refine.ink). Passing such verification should become a necessary step before big claims, especially outside one's own expertise. A link to a bigger-picture post on this is in the comments. | 23 comments on LinkedIn

AI makes you smarter but none the wiser: The disconnect between performance and metacognition

Optimizing human–AI interaction requires users to reflect on their performance critically, yet little is known about generative AI systems’ effect on users’ metacognitive judgments. In two large-scale studies, we investigate how AI usage is associated with users’ metacognitive monitoring and performance in logical reasoning tasks. Specifically, our paper examines whether people using AI to complete tasks can accurately monitor how well they perform. In Study 1, participants (N = 246) used AI to solve 20 logical reasoning problems from the Law School Admission Test. While their task performance improved by three points compared to a norm population, participants overestimated their task performance by four points. Interestingly, higher AI literacy correlated with lower metacognitive accuracy, suggesting that those with more technical knowledge of AI were more confident but less precise in judging their own performance. Using a computational model, we explored individual differences in metacognitive accuracy and found that the Dunning–Kruger effect, usually observed in this task, ceased to exist with AI use. Study 2 (N = 452) replicates these findings. We discuss how AI levels cognitive and metacognitive performance in human–AI interaction and consider the consequences of performance overestimation for designing interactive AI systems that foster accurate self-monitoring, avoid overreliance, and enhance cognitive performance.

Performance of a large language model on the reasoning tasks of a physician

More than 65 years ago, complex clinical diagnostic reasoning cases were introduced as the gold standard for the evaluation of expert medical computing systems, a standard that has held ever since. In this study, we report the results of a physician ...

Can LLMs Faithfully Enact Superforecaster Personas?

Superforecasters are people that make forecasts that are statistically better than the average person; they offer an interesting testbed for persona modelling. In this preliminary work, we investigate whether large language models (LLMs) can faithfully model superforecaster-like personas making similar forecasts and generating similar rationales behind the forecasts. When we compare LLM-generated responses to the ground truth responses of superforecasters, we find that few-shot prompting generates numerically closer forecasts than instruction-prompting baselines. We also measure the similarity of generated and real rationales, finding that the rationales generated based on examples are judged more similar to those of superforecasters than instruction-prompting baselines. However, an analysis of the reasoning patterns in the rationales shows significant differences between human and LLM forecasters, pointing to a gap between stylistic imitation and deep reasoning similarity. These preliminary results raise several interesting questions about how getting superforecaster-like behaviour from LLMs actually works, and open new avenues to explore for improving the forecasting behaviour of LLMs.

Artificial intelligence tools expand scientists’ impact but contract science’s focus

Developments in artificial intelligence (AI) have accelerated scientific discovery1. Alongside recent AI-oriented Nobel prizes2–9, these trends establish the role of AI tools in science10. This advancement raises questions about the influence of AI tools on scientists and science as a whole, and highlights a potential conflict between individual and collective benefits11. To evaluate these questions, we used a pretrained language model to identify AI-augmented research, with an F1-score of 0.875 in validation against expert-labelled data. Using a dataset of 41.3 million research papers across the natural sciences and covering distinct eras of AI, here we show an accelerated adoption of AI tools among scientists and consistent professional advantages associated with AI usage, but a collective narrowing of scientific focus. Scientists who engage in AI-augmented research publish 3.02 times more papers, receive 4.84 times more citations and become research project leaders 1.37 years earlier than those who do not. By contrast, AI adoption shrinks the collective volume of scientific topics studied by 4.63% and decreases scientists’ engagement with one another by 22%. By consequence, adoption of AI in science presents what seems to be a paradox: an expansion of individual scientists’ impact but a contraction in collective science’s reach, as AI-augmented work moves collectively towards areas richest in data. With reduced follow-on engagement, AI tools seem to automate established fields rather than explore new ones, highlighting a tension between personal advancement and collective scientific progress.