Publication bias in the social sciences: Unlocking the file drawer

Half of social-science studies fail replication test in years-long project

Results from massive, ‘eagerly awaited’ initiative reinforce concerns about the credibility of science — but raise hope for solutions.

Scientific production in the era of Large Language Models

Large Language Models (LLMs) are rapidly reshaping scientific research. We analyze these changes in multiple, large-scale datasets with 2.1M preprints, 28K peer review reports, and 246M online accesses to scientific documents. We find: 1) scientists adopting LLMs to draft manuscripts demonstrate a large increase in paper production, ranging from 23.7-89.3% depending on scientific field and author background, 2) LLM use has reversed the relationship between writing complexity and paper quality, leading to an influx of manuscripts that are linguistically complex but substantively underwhelming, and 3) LLM adopters access and cite more diverse prior work, including books and younger, less-cited documents. These findings highlight a stunning shift in scientific production that will likely require a change in how journals, funding agencies, and tenure committees evaluate scientific works.

The strain on scientific publishing

Scientists are increasingly overwhelmed by the volume of articles being published. Total articles indexed in Scopus and Web of Science have grown exponentially in recent years; in 2022 the article total was approximately ~47% higher than in 2016, which has outpaced the limited growth - if any - in the number of practising scientists. Thus, publication workload per scientist (writing, reviewing, editing) has increased dramatically. We define this problem as the strain on scientific publishing. To analyse this strain, we present five data-driven metrics showing publisher growth, processing times, and citation behaviours. We draw these data from web scrapes, requests for data from publishers, and material that is freely available through publisher websites. Our findings are based on millions of papers produced by leading academic publishers. We find specific groups have disproportionately grown in their articles published per year, contributing to this strain. Some publishers enabled this growth by adopting a strategy of hosting special issues, which publish articles with reduced turnaround times. Given pressures on researchers to publish or perish to be competitive for funding applications, this strain was likely amplified by these offers to publish more articles. We also observed widespread year-over-year inflation of journal impact factors coinciding with this strain, which risks confusing quality signals. Such exponential growth cannot be sustained. The metrics we define here should enable this evolving conversation to reach actionable solutions to address the strain on scientific publishing.

Why Most Published Research Findings Are False

Summary There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

Screening, sorting, and the feedback cycles that imperil peer review

Scholarly journals rely on peer review to identify the science most worthy of publication. Yet finding willing and qualified reviewers to evaluate manuscripts has become an increasingly challenging task, possibly even threatening the long-term viability of peer review as an institution. What can or should be done to salvage it? Here, we develop mathematical models to reveal the intricate interactions among incentives faced by authors, reviewers, and readers in their endeavors to identify the best science. Two facets are particularly salient. First, peer review partially reveals authors’ private sense of their work’s quality through their decisions of where to send their manuscripts. Second, journals’ reliance on traditionally unpaid and largely unrewarded review labor deprives them of a standard market mechanism—wages—to recruit additional reviewers when review labor is in short supply. We highlight a resulting feedback loop that threatens to overwhelm the peer review system: (1) an increase in submissions overtaxes the pool of suitable peer reviewers; (2) the accuracy of review drops because journals must either solicit assistance from less qualified reviewers or ask current reviewers to do more; (3) as review accuracy drops, submissions further increase as more authors try their luck at venues that might otherwise be a stretch. We illustrate how this cycle is propelled by the increasing emphasis on high-impact publications, the proliferation of journals, and competition among these journals for peer reviews. Finally, we suggest interventions that could slow or even reverse this cycle of peer-review meltdown.

Do Science <i>Kardashians</i> Get Citation Premium? Self‐Fulfilling Effects of Social Media on Scientific Impact

ABSTRACT We analyze whether the visibility of scientists on social media affects the number of academic citations. We use the global COVID‐19 pandemic as a quasinatural experiment that exogenously increased public attention and the demand for expertise. Using publications on COVID‐related topics by social media stars and their coauthors prior to the outbreak of the pandemic, we find that social media stars' pre‐COVID‐era papers received about – more citations annually per paper after 2019. Quantitatively comparable results are obtained when we use scientists' Kardashian index (K‐index) as a benchmark for stardom, however we find no significant effects when using the intensive margin of scientists' K‐indexes. We provide a brief discussion of policy implications in light of these findings.

A New Paradigm for Scientific Publishing, Peer Review, and Impact Assessment

Scientific publishing and peer review have evolved little in three centuries, while the demands placed on them have grown profoundly. The growing role of artificial intelligence has underscored deep, systemic shortcomings of an aging system that has largely evaded innovation, a system whose origins are appallingly closer to the invention of the printing press than to the internet. We can do better – much better. This article is intended as the beginning of a communal experiment: a living document that critically reviews the modern academic publishing and peer-review system and presents a concrete framework to address what bibliometrics experts¹ have characterized as "the pervasive misapplication of indicators to the evaluation of scientific performance". Building on the Leiden Manifesto, DORA, and a body of scholarship spanning many disciplines and decades, we present a community-governed, non-profit platform organized around three trust-weighted impact factors, for articles, authors, and reviewers, with full algorithmic transparency, an open development log, and structural decoupling of credibility scoring from content moderation and from monetization. We invite the community to discuss, critique, and help shape it.

The Paper Factory

How can large language models (LLMs) contribute to social science research, and what parts of research remain stubbornly human? Building on existing LLM tools, we offer a multi-agent workflow capable of producing a full quantitative social science paper from an initial prompt. The workflow relies on researchers codifying their heuristics for doing data analysis, and we suggest some core design principles for researchers interested in building on this scaffolding. Using this case, we also examine what current LLM capabilities reveal about the organization of research. LLM agents can lower the cost of pursuing high-risk ideas, expand robustness and transparency, reduce concerns about the scientific file drawer, and force scholars to articulate the heuristics that create valuable work. But they also pose challenges, both in terms of the quality of papers and in the adequacy of scientific institutions to adapt. Meeting these challenges will require new institutional norms that make use of these tools observable, auditable, and accountable.

Reformation of science publishing: the Stockholm Declaration

Science relies on integrity and trustworthiness. But scientists under career pressure are lured to purchase fake publications from ‘paper mills’ that use AI-generated data, text and image fabrication. The number of low-quality or fraudulent publications is rising to hundreds of thousands per year, which—if unchecked—will damage the scientific and economic progress of our societies. The result is editor and reviewer fatigue, irreproducible experiments, misguided experiments, disinformation and escalating costs that devour funding from taxpayers intended for research. It is high time to reevaluate current publishing models and outline a global plan to stop this unhealthy development. A conference was therefore organized by the Royal Swedish Academy of Sciences to draft an action plan with specific recommendations, as follows. (i) Academia should resume control of publishing using non-profit publishing models (e.g. diamond open-access). (ii) Adjust incentive systems to merit quality, not quantity, in a reputation economy where the gaming of publication numbers and citation metrics distorts the perception of academic excellence. (iii) Implement mechanisms to prevent and detect fake publications and fraud which are independent of publishers. (iv) Draft and implement legislations, regulations and policies to increase publishing quality and integrity. This is a call to action for universities, academies, science organizations and funders to unite and join this effort.

Research topic choice: Motivations, strategies, and consequences

Abstract. Scientists’ choices of what research topics to pursue are highly consequential and have been the subject of many studies. However, these studies are dispersed across several fields and literatures. This paper provides a review of this body of work. It first reviews theory from economics and sociology to explain how topic choice fits into scientists’ broader competitive strategy, and how the social valuation of research topics reproduces inequalities in science. Then it examines empirical literature on how research topics are chosen in practice, first looking at observational accounts derived from large-scale secondary data sources, then looking at self-reported accounts elicited by surveys and interviews of researchers. Finally, it concludes with a synthesis of the theoretical and empirical literature and identifies research gaps. Themes in these literatures include the rewards associated with certain topics, demographic differences in topic choices, and whether choices are broadly reward maximizing or driven by social context, identities, and path dependence. We also identify several research gaps, including the types and impacts of costs impeding topic entry and switching, the causal mechanisms associating topics with rewards, and the discrepancy between scientists’ self-reported topic motivations and observed behavior.

Against theory-motivated experimentation: Can random experimental choice lead to better theories?

Scientists must choose which among many experiments to perform. We study the epistemic success of experimental choice strategies proposed by philosophers of science or executed by scientists themselves. We develop a multi-agent model of the scientific process that jointly formalizes its core aspects: active experimentation, theorizing, and social learning. We find that agents who choose new experiments at random develop the most informative and predictive theories of the world. The agents aiming to confirm, falsify theories, or resolve theoretical disagreements end up with an illusion of epistemic success: they develop promising accounts for the data they collected, while misrepresenting the ground truth that they intended to learn about. Agents experimenting in these theory-motivated ways acquire less diverse or less representative samples from the ground truth that also turn out to be easier to account for. Random data collection, on the other hand, combines virtues of diverse and representative sampling from a target scientific domain which enables cumulative development of the successful theoretical accounts of it. We suggest that randomization, already a gold standard within experiments, is also beneficial at the level of experiments themselves.

Sharing Knowledge Openly: Author Gender, Race/Ethnicity, and Feminist Science

Open science is an increasingly important movement in contemporary social science. Many funders have begun to mandate open practices, newer journals like Socius are open access, and preprint servers have increased readership of paywalled articles. Feminist scholars have written about the values and pitfalls of open research. In this paper, we put those ideas in conversation with each other and empirical data on open-access publishing to develop a feminist framework for thinking about feminist knowledge, data, publishing, and audience. Using a survey of 19,462 social scientists, linked with their entire Web of Science publication history from 2000 to 2023 and all papers posted to SocArXiv, we replicate past work showing that articles by women are less likely to be some types of open access. We extend these findings to scholars of color and feminist scholarship, while also demonstrating a few surprising null results for all three groups.

Post-publication critique at top-ranked journals across scientific disciplines: a cross-sectional assessment of policies and practice

Journals exert considerable control over letters, commentaries and online comments that criticize prior research (post-publication critique). We assessed policies (Study One) and practice (Study Two) related to post-publication critique at 15 top-ranked journals in each of 22 scientific disciplines ( N = 330 journals). Two-hundred and seven (63%) journals accepted post-publication critique and often imposed limits on length (median 1000, interquartile range (IQR) 500–1200 words) and time-to-submit (median 12, IQR 4–26 weeks). The most restrictive limits were 175 words and two weeks; some policies imposed no limits. Of 2066 randomly sampled research articles published in 2018 by journals accepting post-publication critique, 39 (1.9%, 95% confidence interval [1.4, 2.6]) were linked to at least one post-publication critique (there were 58 post-publication critiques in total). Of the 58 post-publication critiques, 44 received an author reply, of which 41 asserted that original conclusions were unchanged. Clinical Medicine had the most active culture of post-publication critique: all journals accepted post-publication critique and published the most post-publication critique overall, but also imposed the strictest limits on length (median 400, IQR 400–550 words) and time-to-submit (median 4, IQR 4–6 weeks). Our findings suggest that top-ranked academic journals often pose serious barriers to the cultivation, documentation and dissemination of post-publication critique.

Social media promotion improves job market outcomes

Social media has transformed how academics disseminate research, but its effect on academic job outcomes remains unclear. Previous research has shown correlations between social media exposure and metrics like citation counts, but these relationships may be confounded by unobserved factors such as researcher quality or access to professional networks. We examine whether social media promotion causally affects job market outcomes in economics through a field experiment on Twitter (now X). We first collect tweets about job market papers from 519 candidates and post them from a dedicated account. We then randomize half of the posts to be quote-tweeted by established economists in the candidates’ fields, and measure the effects on both online visibility and hiring outcomes. We find that posts in the treatment group receive 441% more views and 303% more likes than those in the control group. Candidates whose posts were assigned to be quote-tweeted receive one additional flyout invitation compared to the control group average of 5.4 flyouts. Furthermore, women in the treatment group receive 0.9 more job offers than women in the control group, who receive 3 offers on average. Exploring mechanisms, we find that academic reputation drives these results, with stronger effects for quote-tweets from highly cited scholars and for candidates from top institutions. Our findings suggest social media promotion causally increases research visibility and improves academic job market outcomes.

Tracing LLM Social Reasoning to Pretraining Data: A Provenance Audit for Social Simulation

LLM-based social simulations depend on social-reasoning capabilities whose pretraining evidence is usually opaque. We treat training-data attribution as a provenance audit: for an open model and corpus, which training regions are associated with social reasoning, how do those profiles differ from adjacent capabilities, and does intervention confirm that influential data is load-bearing? We compute gradient-based attribution (TrackStar via Bergson) over a working set from the de-duplicated Dolma3 mix, aggregate influence across WebOrganizer’s 24-format ×24-topic taxonomy (576 bins), and contrast SocialIQA and MMLU Social Sciences against ARC-Challenge and MMLU STEM. In OLMo3-7B, SocialIQA is the profile outlier: its positive attribution is more diffuse and shifted toward interpersonal and narrative regions rather than documentation-heavy ones. Influence-targeted unlearning validates this association most clearly for SocialIQA, while the comparison tasks show weaker, null, or reversed effects. A held-out social-probe suite tests the breadth and limits of the pattern without claiming population-level simulation fidelity.