The High Cost of Not Doing Experiments - Behavioral Scientist

How Field Experiments in Economics Can Complement Psychological Research on Judgment Biases

This review summarizes results of field experiments examining individual behaviors across several market settings—from open-air markets to rideshare markets to tax-compliance markets—where people sort themselves into market roles wherein they make consequential decisions. Using three distinct examples from my own research on the endowment effect, left-digit bias, and omission bias, I showcase how field experiments can help researchers understand mediators, heterogeneity, and causal moderation involved in judgment biases in the field. In this manner, the review highlights that economic field experiments can serve an invaluable intellectual role alongside traditional laboratory research.

The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework

We review 74 experiments with no, low, or high performance-based financial incentives. The modal result has no effect on mean performance (though variance is usually reduced by higher payment). Higher incentive does improve performance often, typically judgment tasks that are responsive to better effort. Incentives also reduce “presentation” effects (e.g., generosity and risk-seeking). Incentive effects are comparable to effects of other variables, particularly “cognitive capital” and task “production” demands, and interact with those variables, so a narrow-minded focus on incentives alone is misguided. We also note that no replicated study has made rationality violations disappear purely by raising incentives.

Against theory-motivated experimentation: Can random experimental choice lead to better theories?

Scientists must choose which among many experiments to perform. We study the epistemic success of experimental choice strategies proposed by philosophers of science or executed by scientists themselves. We develop a multi-agent model of the scientific process that jointly formalizes its core aspects: active experimentation, theorizing, and social learning. We find that agents who choose new experiments at random develop the most informative and predictive theories of the world. The agents aiming to confirm, falsify theories, or resolve theoretical disagreements end up with an illusion of epistemic success: they develop promising accounts for the data they collected, while misrepresenting the ground truth that they intended to learn about. Agents experimenting in these theory-motivated ways acquire less diverse or less representative samples from the ground truth that also turn out to be easier to account for. Random data collection, on the other hand, combines virtues of diverse and representative sampling from a target scientific domain which enables cumulative development of the successful theoretical accounts of it. We suggest that randomization, already a gold standard within experiments, is also beneficial at the level of experiments themselves.

Monetary incentives, what are they good for?

This paper is a critical reflection on the use of monetary incentives in economic experiments. The argument is that incentives have their effect through their influence on one or more of three fact...

How tiny experiments can set you free | Anne-Laure Le Cunff | TEDxNashville | Transcript

When Dr Anne-Laure Le Cunff was told she might die, her first instinct was to check her calendar. In this talk, she reveals how our desperate need for control keeps us stuck and how tiny experiments can set us free. Dr Anne-Laure Le Cunff is a neuroscientist, author, and entrepreneur who studies curiosity – both in the lab and in life. Her research at King’s College London explores how our brains seek, learn, and adapt, spanning areas from ADHD to AI and mental health. She founded Ness Labs, a science-based learning platform helping people live more experimental lives. Her bestselling book Tiny Experiments offers a practical guide to transforming uncertainty into curiosity, creativity, and self-discovery. Through her research and writing, she bridges science and everyday life, showing how curiosity can help us loosen control, embrace change, and uncover who we truly are.

Research Debt

Science is a human activity. When we fail to distill and explain research, we accumulate a kind of debt...

Integrative experiments identify how punishment affects welfare in public goods games

Despite decades of research, the conditions under which punishment promotes cooperation remain unclear. Through an integrative experiment varying 14 design parameters of public goods games across 360 experimental conditions (147,618 decisions from 7100 participants), we reveal substantial heterogeneity in punishment effectiveness: Its impact on welfare ranges from 43% improvement to 44% reduction depending on the game parameters. To characterize these patterns, we developed models that outperformed human forecasters in predicting punishment effectiveness in new experiments. Communication emerges as the most important factor, followed by contribution framing (opt out versus opt in), contribution type (variable versus all-or-nothing), game length, and outcome visibility, though these factors often interact. The results reframe the debate from whether punishment works to when it does, demonstrating how integrative experiments enable discovery of generalizable patterns in social phenomena. , Editor’s summary People face conflicts between maximizing personal gain versus supporting collective interests. If we cooperatively recycle or donate to charities, it benefits society, but it also costs us time and resources that could be selfishly preserved for ourselves. We impose penalties to deter those undesirable or selfish behaviors, but under what conditions do punishments or penalties effectively modify behavior to benefit group welfare? Alsobay et al . systematically and simultaneously varied 14 factors together instead of in isolation. Punishment was unequivocally most effective when paired with consistent communication, particularly over time. Another effective factor was “opting out” or withdrawing some, but not all, endowments already in the public fund. These methodological advances revealed when, rather than whether, punishment works. —Ekeoma Uzogara , INTRODUCTION Human societies face many situations where individual and collective interests conflict, often referred to as social dilemmas. Costly peer punishment has been studied for more than 25 years in public goods games (stylized behavioral experiments in which individuals decide how much to contribute to a shared pool that benefits everyone) as a mechanism to promote cooperation. Prior research has identified many contextual factors that moderate punishment’s effectiveness, including game length, communication, group size, punishment cost, and so on. However, the specific conditions under which punishment improves group welfare remain unclear. RATIONALE We argue that this lack of clarity derives from the dominant experimental paradigm, in which any given study manipulates only one or a few theoretically informed factors. Because such studies differ in many ways (different experimental procedures, populations), their results are often difficult to compare or integrate. Consequently, one can list many factors that have some effect, but cannot say how much each matters relative to the others, or how they work together, and as a result, cannot predict when punishment will help or harm welfare in new settings. To address this fundamental knowledge gap, we use an integrative experimental design and systematically vary 14 parameters across 360 conditions (147,618 decisions from 7100 participants) to elucidate when punishment improves versus undermines welfare in public goods games, which factors matter most, and how they interact. RESULTS The effect of punishment on welfare ranged from 43% improvement to 44% reduction depending on the specific combination of game parameters. To characterize this heterogeneity, we trained a model that outperformed all 553 human forecasters (laypeople and experts) in predicting whether punishment would help or harm welfare in new experiments. Communication emerged as roughly three times more important than any other factor, followed by contribution framing (opt in versus opt out), contribution type (variable versus all-or-nothing), game length, and peer outcome visibility (whether participants can see others’ earnings). These factors often interact. For example, longer games enhance punishment’s effectiveness only when communication is available, and contribution framing effects depend on both contribution type and outcome visibility. CONCLUSION Many phenomena in social science are shaped by many factors whose interactions are consequential, yet the dominant experimental paradigm often limits its inquiry to “does a given effect exist?” and examines hypothesized factors in isolation. As a result, research programs can accumulate many partial explanations without a clear picture of how they combine to determine outcomes across settings. Knowing that factors matter individually is fundamentally different from knowing how much each matters and how they interact. The integrative approach implemented here offers one way forward. It varies many factors simultaneously within a shared design space, evaluates models by their predictive accuracy on new experiments, and probes those models to constrain and develop theory. Our hope is that integrative experiment designs, combined with models that integrate prediction and explanation, represent a path toward more cumulative social science. Integrative experiment reveals when punishment helps versus harms. We systematically varied 14 design parameters across 360 experimental conditions. The effect of punishment on cooperation efficiency ranged from −44% to +43% depending on the specific game parameters. Communication emerged as three times more important than any other factor, followed by contribution framing, contribution type, and game length.

Unwillingness to pay for privacy: A field experiment

We measure willingness to pay for privacy in a field experiment. Participants bought at most one DVD from one of two competing online stores. One store consistently required more sensitive personal data than the other, but otherwise the stores were identical. In one treatment, DVDs were one Euro cheaper at the store requesting more personal information, and almost all buyers chose the cheaper store. Surprisingly, in the second treatment when prices were identical, participants bought from both shops equally often.

Claims about scientific rigour require rigour

Nature Human Behaviour - Claims about scientific rigour require rigour

Misbehaving: the making of behavioral economics

Nobel laureate Richard H. Thaler has spent his career s…

Philosophy Experiments — Interactive Thought Experiments

Explore 27 interactive philosophy experiments that challenge your moral intuitions, test your logical reasoning, and probe the boundaries of belief.

Pay Enough or Don't Pay at All*

Abstract. Economists usually assume that monetary incentives improve performance, and psychologists claim that the opposite may happen. We present and disc

Why I don’t trust most human-AI interaction experimental research – Jason Collins blog

Behavioural economics, data science and artificial intelligence.

Pain of Paying? — A Metaphor Gone Literal: Evidence from Neural and Behavioral Science

How do individuals consider the price of a good when making purchase decisions? Standard economic theories assume an analytical process: Individuals consider th

On Taste, Effort & Curiosity - again

When AI collapses how long it takes to ship, what’s left is judgment, experimentation, and knowing what not to build.