Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing

The control of the false discovery rate in multiple testing under dependency

Benjamini and Hochberg suggest that the false discovery rate may be the appropriate error rate to control in many applied multiple testing problems. A simple procedure was given there as an FDR controlling procedure for independent test statistics and was shown to be much more powerful than comparable procedures which control the traditional familywise error rate. We prove that this same procedure also controls the false discovery rate when the test statistics have positive regression dependency on each of the test statistics corresponding to the true null hypotheses. This condition for positive dependency is general enough to cover many problems of practical interest, including the comparisons of many treatments with a single control, multivariate normal test statistics with positive correlation matrix and multivariate $t$. Furthermore, the test statistics may be discrete, and the tested hypotheses composite without posing special difficulties. For all other forms of dependency, a simple conservative modification of the procedure controls the false discovery rate. Thus the range of problems for which a procedure with proven FDR control can be offered is greatly increased.

Why Most Published Research Findings Are False

Summary There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

Multiple Hypothesis Testing

Sign in to access your institutional or personal subscription or get immediate access to your online copy - available in PDF and ePub formats

Assessing and mitigating batch effects in large-scale omics studies

Batch effects in omics data are notoriously common technical variations unrelated to study objectives, and may result in misleading outcomes if uncorrected, or hinder biomedical discovery if over-corrected. Assessing and mitigating batch effects is crucial for ensuring the reliability and reproducibility of omics data and minimizing the impact of technical variations on biological interpretation. In this review, we highlight the profound negative impact of batch effects and the urgent need to address this challenging problem in large-scale omics studies. We summarize potential sources of batch effects, current progress in evaluating and correcting them, and consortium efforts aiming to tackle them.

Small Telescopes - Uri Simonsohn, 2015

This article introduces a new approach for evaluating replication results. It combines effect-size estimation with hypothesis testing, assessing the extent to w...

Hypothesis: A new approach to property-based testing

The property-based testing library for Python

LLM-Assisted Reanalysis of Unsolved Rare Disease Genomes Increases Diagnostic Yield

Rare and undiagnosed genetic disorders affect millions of patients globally, and many patients endure years of inconclusive testing. Conventional genomic interpretation can be insufficiently sensit...

The natural selection of bad science

Abstract. Poor research design and data analysis encourage false-positive findings. Such poor methods persist despite perennial calls for improvement, sugg

Daniël Lakens on Twitter / X

Can they? Published in PNAS 🚩Edited by Susan Fiske🚩None of the link to preregs are public in publication🚩difference between significant and non-significant is itself not significant errors 🚩marginally significant results interpreted as support for hypothesis 🚩and then some https://t.co/6n8aJ2DSuo pic.twitter.com/c0eVKnlUfK— Daniël Lakens (@lakens) July 30, 2024

A universal approach to mocking

Defunctionalise your continuations and your tests can run any computation a step at a time.

Rethinking Data Shapley for Data Selection Tasks: Misleads and Merits

Data Shapley provides a principled approach to data valuation and plays a crucial role in data-centric machine learning (ML) research. Data selection is considered a standard application of Data Shapley. However, its data selection performance has shown to be inconsistent across settings in the literature. This study aims to deepen our understanding of this phenomenon. We introduce a hypothesis testing framework and show that Data Shapley’s performance can be no better than random selection without specific constraints on utility functions. We identify a class of utility functions, monotonically transformed modular functions, within which Data Shapley optimally selects data. Based on this insight, we propose a heuristic for predicting Data Shapley’s effectiveness in data selection tasks. Our experiments corroborate these findings, adding new insights into when Data Shapley may or may not succeed.

LLMs believe false statements even after explicit warnings that they're false

Fine-tuning tests show "bias... toward confidently representing the claims as true."

High Performance AI Lab

High Performance AI Lab builds open inference systems and publishes the conditions behind every number — device, model, quant, and rep count.

Inference characteristics of Llama · Cursor

A primer on inference math and an examination of the surprising costs of Llama.