Small Telescopes: Detectability and the Evaluation of Replication Results

Why Most Published Research Findings Are False

Summary There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

ReproAI: the plugin for reproducible replication packages

An author-facing AI plugin that helps you prepare a cleaner, reproducible replication package before submission.

Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers

The concept of reproducibility can have different interpretations across various research fields and even within the same field [39]. To avoid confusion, we first specify our terms, broadly defining reproducibility and then further categorizing it into various types and degrees. The first distinction comes from Goodman et al. [42], who specify a fundamental division between whether we (i) mean reproducible in principle (termed “methods” reproducibility) due to sufficient description/sharing of methodologies, materials, etc., or (ii) whether results/conclusions actually prove to be reproducible when experiments or analyses are re-done. In the second category, they distinguish “results” and “inferential” reproducibility, depending on whether the analyses or inferences to broader conclusions are reproduced.

Post-publication critique at top-ranked journals across scientific disciplines: a cross-sectional assessment of policies and practice

Journals exert considerable control over letters, commentaries and online comments that criticize prior research (post-publication critique). We assessed policies (Study One) and practice (Study Two) related to post-publication critique at 15 top-ranked journals in each of 22 scientific disciplines ( N = 330 journals). Two-hundred and seven (63%) journals accepted post-publication critique and often imposed limits on length (median 1000, interquartile range (IQR) 500–1200 words) and time-to-submit (median 12, IQR 4–26 weeks). The most restrictive limits were 175 words and two weeks; some policies imposed no limits. Of 2066 randomly sampled research articles published in 2018 by journals accepting post-publication critique, 39 (1.9%, 95% confidence interval [1.4, 2.6]) were linked to at least one post-publication critique (there were 58 post-publication critiques in total). Of the 58 post-publication critiques, 44 received an author reply, of which 41 asserted that original conclusions were unchanged. Clinical Medicine had the most active culture of post-publication critique: all journals accepted post-publication critique and published the most post-publication critique overall, but also imposed the strictest limits on length (median 400, IQR 400–550 words) and time-to-submit (median 4, IQR 4–6 weeks). Our findings suggest that top-ranked academic journals often pose serious barriers to the cultivation, documentation and dissemination of post-publication critique.

The strain on scientific publishing

Scientists are increasingly overwhelmed by the volume of articles being published. Total articles indexed in Scopus and Web of Science have grown exponentially in recent years; in 2022 the article total was approximately ~47% higher than in 2016, which has outpaced the limited growth - if any - in the number of practising scientists. Thus, publication workload per scientist (writing, reviewing, editing) has increased dramatically. We define this problem as the strain on scientific publishing. To analyse this strain, we present five data-driven metrics showing publisher growth, processing times, and citation behaviours. We draw these data from web scrapes, requests for data from publishers, and material that is freely available through publisher websites. Our findings are based on millions of papers produced by leading academic publishers. We find specific groups have disproportionately grown in their articles published per year, contributing to this strain. Some publishers enabled this growth by adopting a strategy of hosting special issues, which publish articles with reduced turnaround times. Given pressures on researchers to publish or perish to be competitive for funding applications, this strain was likely amplified by these offers to publish more articles. We also observed widespread year-over-year inflation of journal impact factors coinciding with this strain, which risks confusing quality signals. Such exponential growth cannot be sustained. The metrics we define here should enable this evolving conversation to reach actionable solutions to address the strain on scientific publishing.

mediation: R Package for Causal Mediation Analysis

In this paper, we describe the R package mediation for conducting causal mediation analysis in applied empirical research. In many scientific disciplines, the goal of researchers is not only estimating causal effects of a treatment but also understanding the process in which the treatment causally affects the outcome. Causal mediation analysis is frequently used to assess potential causal mechanisms. The mediation package implements a comprehensive suite of statistical tools for conducting such an analysis. The package is organized into two distinct approaches. Using the model-based approach, researchers can estimate causal mediation effects and conduct sensitivity analysis under the standard research design. Furthermore, the design-based approach provides several analysis tools that are applicable under different experimental designs. This approach requires weaker assumptions than the model-based approach. We also implement a statistical method for dealing with multiple (causally dependent) mediators, which are often encountered in practice. Finally, the package also offers a methodology for assessing causal mediation in the presence of treatment noncompliance, a common problem in randomized trials.

A protocol for structured robustness reproductions and replicability assessments

Abstract. Robustness reproductions and replicability discussions are on the rise in response to concerns about a potential credibility crisis in economics.

Reformation of science publishing: the Stockholm Declaration

Science relies on integrity and trustworthiness. But scientists under career pressure are lured to purchase fake publications from ‘paper mills’ that use AI-generated data, text and image fabrication. The number of low-quality or fraudulent publications is rising to hundreds of thousands per year, which—if unchecked—will damage the scientific and economic progress of our societies. The result is editor and reviewer fatigue, irreproducible experiments, misguided experiments, disinformation and escalating costs that devour funding from taxpayers intended for research. It is high time to reevaluate current publishing models and outline a global plan to stop this unhealthy development. A conference was therefore organized by the Royal Swedish Academy of Sciences to draft an action plan with specific recommendations, as follows. (i) Academia should resume control of publishing using non-profit publishing models (e.g. diamond open-access). (ii) Adjust incentive systems to merit quality, not quantity, in a reputation economy where the gaming of publication numbers and citation metrics distorts the perception of academic excellence. (iii) Implement mechanisms to prevent and detect fake publications and fraud which are independent of publishers. (iv) Draft and implement legislations, regulations and policies to increase publishing quality and integrity. This is a call to action for universities, academies, science organizations and funders to unite and join this effort.

Can we measure trust in scientific publications? - LSE Impact

Jonathon Alexis Coates outlines how a constellation of static and dynamic indicators could provide a means for assessing the trustworthiness of published research

datasetpapers — a public research experiment

An experimental approach to versioned, forkable, machine-readable analyses. A prototype, not a product or service.

datasetpapers — a public research experiment

An experimental approach to versioned, forkable, machine-readable analyses. A prototype, not a product or service.

Is replication <i>possible</i> in qualitative research? A response to Makel et al. (2022)

There has been much debate in recent years about how open research practices, which have been promoted in efforts to improve research robustness, may (not) be appropriate for qualitative methodolog...

Current market rates for scholarly publishing services

For decades, the supra-inflation increase of subscription prices for scholarly journals has concerned scholarly institutions. After years of fruitless efforts to solve this “serials crisis”, open access has been proposed as the latest potential solution. However, also the prices for open access publishing are high and are rising well beyond inflation. What has been missing from the public discussion so far is a quantitative approach to determine the actual costs of efficiently publishing a scholarly article using state-of-the-art technologies, such that informed decisions can be made as to appropriate price levels. Here we provide a granular, step-by-step calculation of the costs associated with publishing primary research articles, from submission, through peer-review, to publication, indexing and archiving. We find that these costs range from less than US$200 per article in modern, large scale publishing platforms using post-publication peer-review, to about US$1,000 per article in prestigious journals with rejection rates exceeding 90%. The publication costs for a representative scholarly article today come to lie at around US$400. These results appear uncontroversial as they not only match previous data using different methodologies, but also conform to the costs that many publishers have openly or privately shared. We discuss the numerous additional non-publication items that make up the difference between these publication costs and final price at the more expensive, legacy publishers.

Current market rates for scholarly publishing services

For decades, the supra-inflation increase of subscription prices for scholarly journals has concerned scholarly institutions. After years of fruitless efforts to solve this “serials crisis”, open access has been proposed as the latest potential solution. However, also the prices for open access publishing are high and are rising well beyond inflation. What has been missing from the public discussion so far is a quantitative approach to determine the actual costs of efficiently publishing a scholarly article using state-of-the-art technologies, such that informed decisions can be made as to appropriate price levels. Here we provide a granular, step-by-step calculation of the costs associated with publishing primary research articles, from submission, through peer-review, to publication, indexing and archiving. We find that these costs range from less than US$200 per article in modern, large scale publishing platforms using post-publication peer-review, to about US$1,000 per article in prestigious journals with rejection rates exceeding 90%. The publication costs for a representative scholarly article today come to lie at around US$400. These results appear uncontroversial as they not only match previous data using different methodologies, but also conform to the costs that many publishers have openly or privately shared. We discuss the numerous additional non-publication items that make up the difference between these publication costs and final price at the more expensive, legacy publishers.