re: serial experiments

Keynote: Reproducibility and replicability of computer simulations | Canal U

Since the early days of the reproducibility crisis, much progress has been made in understanding and improving computational reproducibility and replicability (R and R)...

Retraction Watch

Tracking retractions as a window into the scientific process

Interface Experimentation | Projects

This page indexes the files currently present in interface-experiments.

Experiments — ux-phi

Playable versions of canonical experiments in philosophy: the Socratic-questionnaire thought experiments — the trolley problem, Gettier cases, the side-effect effect, Mary the colour scientist.

The High Cost of Not Doing Experiments - Behavioral Scientist

We can pay dearly, in blood, treasure, and well-being, for experiments that aren’t done. -Richard Nisbett, Mindware

Philosophy Experiments — Interactive Thought Experiments

Explore 27 interactive philosophy experiments that challenge your moral intuitions, test your logical reasoning, and probe the boundaries of belief.

classic Trolley Experiment through the ERM v2

The classic Trolley Experiment through the ERM v2.0 protocol moves it from a philosophical parlor trick to a high-stakes stress test of systemic stability. Unlike traditional ethics (utilitarianism vs. deontology), ERM evaluates the Lever Pull as a hypothesis of system persistence. Stage 1 – Hyp...

ReScience organization

Reproducible Science is good. Replicated Science is better. - ReScience organization

Dispatch

An occasional update on the lab’s latest findings, appearances, and happenings.

Small Telescopes - Uri Simonsohn, 2015

This article introduces a new approach for evaluating replication results. It combines effect-size estimation with hypothesis testing, assessing the extent to w...

Mechanism Experiments and Policy Evaluations

Randomized controlled trials are increasingly used to evaluate policies. How can we make these experiments as useful as possible for policy purposes? We argue greater use should be made of experiments that identify the behavioral mechanisms that are central to clearly specified policy questions, what we call "mechanism experiments." These types of experiments can be of great policy value even if the intervention that is tested (or its setting) does not correspond exactly to any realistic policy option.

Taming dynamic feeds

ideas, work and experiments

Demand characteristics in human–computer experiments

Demand characteristics refer to cues that can inform participants in experiments about the hypothesis and influence their behavior. They lead researchers to erroneously infer non-existing effects, undermining the experimental integrity of empirical studies. Despite a widespread acknowledgment of their confounding influence in experimental psychology, experiments involving humans and computers to a lesser extent consider effects of demand characteristics, as computerized protocols are thought to be immune to some experimenter biases. Furthermore, demand characteristics are considered to mainly effect subjective measures. As a result, demand characteristics often remain uncontrolled in studies involving computers, and in particular for objective measures such as performance. In this paper, we present two experiments that underline the importance of demand characteristics in human–computer interaction experiments. In a text-entry study, we made participants believe they were evaluating a research-based keyboard. This belief led to increased performance and self-reported user experience. In a second study, we conducted a thought experiment on the illusion of body ownership in virtual reality, where the experimental design indicated the study hypothesis. We found hypothesis-compliant responses from participants, even when they did not experience the illusion. We conclude that demand characteristics pose a significant challenge to the interpretation and validity of human–computer experiments, even when they are fully automated. We discuss the implications and offer guidelines to mitigate effects of demand characteristics.