Expert opinion and judgment enter into the practice of statistical inference and decision-making in numerous ways. Indeed, there is essentially no aspect of scientific investigation in which judgme...
Uncertain Judgements: Eliciting Experts' Probabilities
Elicitation is the process of extracting expert knowledge about some unknown quantity or quantities, and formulating that information as a probability distribution. Elicitation is important in situations, such as modelling the safety of nuclear installations or assessing the risk of terrorist attacks, where expert knowledge is essentially the only source of good information. It also plays a major role in other contexts by augmenting scarce observational data, through the use of Bayesian statistical methods. However, elicitation is not a simple task, and practitioners need to be aware of a wide range of research findings in order to elicit expert judgements accurately and reliably. Uncertain Judgements introduces the area, before guiding the reader through the study of appropriate elicitation methods, illustrated by a variety of multi-disciplinary examples. This is achieved by: Presenting a methodological framework for the elicitation of expert knowledge incorporating findings from both statistical and psychological research. Detailing techniques for the elicitation of a wide range of standard distributions, appropriate to the most common types of quantities. Providing a comprehensive review of the available literature and pointing to the best practice methods and future research needs. Using examples from many disciplines, including statistics, psychology, engineering and health sciences. Including an extensive glossary of statistical and psychological terms. An ideal source and guide for statisticians and psychologists with interests in expert judgement or practical applications of Bayesian analysis, Uncertain Judgements will also benefit decision-makers, risk analysts, engineers and researchers in the medical and social sciences.
AI, peer review and the human activity of science
When researchers cede their scientific judgement to machines, we lose something important.
AI Sycophancy and Decisions
We examine whether sycophantic AI advice distorts decisions. Our experiment involves 1,500 participants in 30 decision environments spanning core domains in eco
AI for Research | Scite
Scite searches 280M+ scholarly articles to give you citation-backed answers and show you how every claim is supported or disputed.
A Bayesian Truth Serum for Subjective Data
Subjective judgments, an essential information source for science and policy, are problematic because there are no public criteria for assessing judgmental truthfulness. I present a scoring method for eliciting truthful subjective data in situations ...
Elicit: AI for research & decision-making
Elicit helps you synthesize evidence, understand the landscape, design programs and experiments, draft proposals, and weigh tradeoffs. Trusted by over 7 million researchers and 8 of the top 20 pharmaceutical companies
Copying Better: How To Acquire The Tacit Knowledge of Experts
Much of expertise is tacit: that is, it cannot be captured through words alone. We look at techniques, drawn from the field of Naturalistic Decision Making, designed to acquire the tacit knowledge of experts.
Artificial intelligence and illusions of understanding in scientific research
Scientists are enthusiastically imagining ways in which artificial intelligence (AI) tools might improve research. Why are AI tools so attractive and what are the risks of implementing them across the research pipeline? Here we develop a taxonomy of scientists’ visions for AI, observing that their appeal comes from promises to improve productivity and objectivity by overcoming human shortcomings. But proposed AI solutions can also exploit our cognitive limitations, making us vulnerable to illusions of understanding in which we believe we understand more about the world than we actually do. Such illusions obscure the scientific community’s ability to see the formation of scientific monocultures, in which some types of methods, questions and viewpoints come to dominate alternative approaches, making science less innovative and more vulnerable to errors. The proliferation of AI tools in science risks introducing a phase of scientific enquiry in which we produce more but understand less. By analysing the appeal of these tools, we provide a framework for advancing discussions of responsible knowledge production in the age of AI.
Establishing trust in automated reasoning - MetaROR
Since its beginnings in the 1940s, automated reasoning by computers has become a tool of ever growing importance in scientific research. So far, the rules underlying automated reasoning have mainly been formulated by humans, in the form of program source code. Rules derived from large amounts of data, via machine learning techniques, are a complementary approach currently under intense development. The question of why we should trust these systems, and the results obtained with their help, has been discussed by early practitioners of computational science, but was later forgotten. The present work focuses on independent reviewing, an important source of trust in science, and identifies the characteristics of automated reasoning systems that affect their reviewability. It also discusses possible steps towards increasing reviewability and trustworthiness via a combination of technical and social measures.
AI co-scientists are revolutionizing how research is done
Artificial-intelligence systems can generate hypotheses, design experiments and analyse data — but humans still need to decide what makes sense.
AI’s Growing Role as Scientific Peer Reviewer | Stanford HAI
Stanford computer scientist James Zou is exploring how AI can accelerate scientific research and peer review. His finding: AI excels at spotting gaps, but judgment calls still need humans.
Rethinking bioinformatics expertise in the era of artificial intelligence
Artificial intelligence (AI) is often framed as replacing scientific expertise, especially in bioinformatics. We argue instead that AI is a powerful accelerant whose value depends on expert guidance in design, data curation, interpretation, and governance. Because AI cannot judge biological meaning or verify scientific validity, bioinformaticians remain essential. Their role is shifting from workflow execution toward AI design, complex discovery, and responsible institutional leadership across research, translation, and clinical practice.
"One of the important skills of the professional scientists... is to know what to read and what not to read. This is narrow community knowledge built up through extended face-to-face socialisation in expert groups."
Large language models eroding science understanding: an empirical study of ‘malignment’ - AI and Ethics
link.springer.com
