Indirect reciprocity undermines indirect reciprocity destabilizing large-scale cooperation

Integrative experiments identify how punishment affects welfare in public goods games

Despite decades of research, the conditions under which punishment promotes cooperation remain unclear. Through an integrative experiment varying 14 design parameters of public goods games across 360 experimental conditions (147,618 decisions from 7100 participants), we reveal substantial heterogeneity in punishment effectiveness: Its impact on welfare ranges from 43% improvement to 44% reduction depending on the game parameters. To characterize these patterns, we developed models that outperformed human forecasters in predicting punishment effectiveness in new experiments. Communication emerges as the most important factor, followed by contribution framing (opt out versus opt in), contribution type (variable versus all-or-nothing), game length, and outcome visibility, though these factors often interact. The results reframe the debate from whether punishment works to when it does, demonstrating how integrative experiments enable discovery of generalizable patterns in social phenomena. , Editor’s summary People face conflicts between maximizing personal gain versus supporting collective interests. If we cooperatively recycle or donate to charities, it benefits society, but it also costs us time and resources that could be selfishly preserved for ourselves. We impose penalties to deter those undesirable or selfish behaviors, but under what conditions do punishments or penalties effectively modify behavior to benefit group welfare? Alsobay et al . systematically and simultaneously varied 14 factors together instead of in isolation. Punishment was unequivocally most effective when paired with consistent communication, particularly over time. Another effective factor was “opting out” or withdrawing some, but not all, endowments already in the public fund. These methodological advances revealed when, rather than whether, punishment works. —Ekeoma Uzogara , INTRODUCTION Human societies face many situations where individual and collective interests conflict, often referred to as social dilemmas. Costly peer punishment has been studied for more than 25 years in public goods games (stylized behavioral experiments in which individuals decide how much to contribute to a shared pool that benefits everyone) as a mechanism to promote cooperation. Prior research has identified many contextual factors that moderate punishment’s effectiveness, including game length, communication, group size, punishment cost, and so on. However, the specific conditions under which punishment improves group welfare remain unclear. RATIONALE We argue that this lack of clarity derives from the dominant experimental paradigm, in which any given study manipulates only one or a few theoretically informed factors. Because such studies differ in many ways (different experimental procedures, populations), their results are often difficult to compare or integrate. Consequently, one can list many factors that have some effect, but cannot say how much each matters relative to the others, or how they work together, and as a result, cannot predict when punishment will help or harm welfare in new settings. To address this fundamental knowledge gap, we use an integrative experimental design and systematically vary 14 parameters across 360 conditions (147,618 decisions from 7100 participants) to elucidate when punishment improves versus undermines welfare in public goods games, which factors matter most, and how they interact. RESULTS The effect of punishment on welfare ranged from 43% improvement to 44% reduction depending on the specific combination of game parameters. To characterize this heterogeneity, we trained a model that outperformed all 553 human forecasters (laypeople and experts) in predicting whether punishment would help or harm welfare in new experiments. Communication emerged as roughly three times more important than any other factor, followed by contribution framing (opt in versus opt out), contribution type (variable versus all-or-nothing), game length, and peer outcome visibility (whether participants can see others’ earnings). These factors often interact. For example, longer games enhance punishment’s effectiveness only when communication is available, and contribution framing effects depend on both contribution type and outcome visibility. CONCLUSION Many phenomena in social science are shaped by many factors whose interactions are consequential, yet the dominant experimental paradigm often limits its inquiry to “does a given effect exist?” and examines hypothesized factors in isolation. As a result, research programs can accumulate many partial explanations without a clear picture of how they combine to determine outcomes across settings. Knowing that factors matter individually is fundamentally different from knowing how much each matters and how they interact. The integrative approach implemented here offers one way forward. It varies many factors simultaneously within a shared design space, evaluates models by their predictive accuracy on new experiments, and probes those models to constrain and develop theory. Our hope is that integrative experiment designs, combined with models that integrate prediction and explanation, represent a path toward more cumulative social science. Integrative experiment reveals when punishment helps versus harms. We systematically varied 14 design parameters across 360 experimental conditions. The effect of punishment on cooperation efficiency ranged from −44% to +43% depending on the specific game parameters. Communication emerged as three times more important than any other factor, followed by contribution framing, contribution type, and game length.

Sustainable Cooperation in Peer-To-Peer Networks

Traditionally, peer-to-peer systems have relied on altruism and reciprocity. Although incentive-based models have gained prominence in new-generation peer-to-peer systems, it is essential to recognize the continued importance of cooperative principles in achieving performance, fairness, and correctness. The lack of this acknowledgment has paved the way for selfish peers to gain unfair advantages in these systems. As such, we address the challenge of selfish peers by devising a mechanism to reward sustained cooperation. Instead of relying on global accountability mechanisms, we propose a protocol that naturally aggregates local evaluations of cooperation. Traditional mechanisms are often vulnerable to Sybil and misreporting attacks. However, our approach overcomes these issues by limiting the benefits selfish peers can gain without incurring any cost. The viability of our algorithm is proven with a deployment to 27,259 Internet users and a realistic simulation of a blockchain gossip protocol. We show that our protocol sustains cooperation even in the presence of a majority of selfish peers while incurring only negligible overhead.

MeritRank: Sybil Tolerant Reputation for Merit-based Tokenomics*

Decentralized reputation systems are emerging as promising mechanisms to enhance the effectiveness of token-based economies. Unlike traditional monetary incentives, these systems reward participants based on the actual value of their contributions to the network. However, the advantages and challenges associated with such systems remain largely unexplored. In this work, we investigate the inherent trade-offs in designing a decentralized reputation system that is simultaneously generalizable, trustless, and Sybil-resistant. Specifically, “generalizable” means that the system can assess various types of contributions across different contexts, “trustless” indicates that it functions without the need for a central authority to oversee reputations, and “Sybil-resistant” refers to its ability to withstand manipulations by fake identities, i.e., Sybil attacks.

Network dynamics of social influence in the wisdom of crowds

Significance Since the discovery of the wisdom of crowds over 100 years ago theories of collective intelligence have held that group accuracy requires either statistical independence or informational diversity among individual beliefs. Empirical evidence suggests that allowing people to observe the beliefs of others leads to increased similarity of individual estimates, reducing independence and diversity without a corresponding increase in group accuracy. As a result, social influence is expected to undermine the wisdom of crowds. We present theoretical predictions and experimental findings demonstrating that, in decentralized networks, social influence generates learning dynamics that reliably improve the wisdom of crowds. We identify general conditions under which influence, not independence, produces the most accurate group judgments. , A longstanding problem in the social, biological, and computational sciences is to determine how groups of distributed individuals can form intelligent collective judgments. Since Galton’s discovery of the “wisdom of crowds” [Galton F (1907) Nature 75:450–451], theories of collective intelligence have suggested that the accuracy of group judgments requires individuals to be either independent, with uncorrelated beliefs, or diverse, with negatively correlated beliefs [Page S (2008) The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies ]. Previous experimental studies have supported this view by arguing that social influence undermines the wisdom of crowds. These results showed that individuals’ estimates became more similar when subjects observed each other’s beliefs, thereby reducing diversity without a corresponding increase in group accuracy [Lorenz J, Rauhut H, Schweitzer F, Helbing D (2011) Proc Natl Acad Sci USA 108:9020–9025]. By contrast, we show general network conditions under which social influence improves the accuracy of group estimates, even as individual beliefs become more similar. We present theoretical predictions and experimental results showing that, in decentralized communication networks, group estimates become reliably more accurate as a result of information exchange. We further show that the dynamics of group accuracy change with network structure. In centralized networks, where the influence of central individuals dominates the collective estimation process, group estimates become more likely to increase in error.

The Social Cost of Cheap Pseudonyms

We consider the problems of societal norms for cooperation and reputation when it is possible to obtain cheap pseudonyms, something that is becoming quite common in a wide variety of interactions on the Internet. This introduces opportunities to misbehave without paying reputational consequences. A large degree of cooperation can still emerge, through a convention in which newcomers “pay their dues” by accepting poor treatment from players who have established positive reputations. One might hope for an open society where newcomers are treated well, but there is an inherent social cost in making the spread of reputations optional. We prove that no equilibrium can sustain significantly more cooperation than the dues‐paying equilibrium in a repeated random matching game with a large number of players in which players have finite lives and the ability to change their identities, and there is a small but nonvanishing probability of mistakes. Although one could remove the inefficiency of mistreating newcomers by disallowing anonymity, this is not practical or desirable in a wide variety of transactions. We discuss the use of entry fees, which permits newcomers to be trusted but excludes some players with low payoffs, thus introducing a different inefficiency. We also discuss the use of free but unreplaceable pseudonyms, and describe a mechanism that implements them using standard encryption techniques, which could be practically implemented in electronic transactions.

The Social Cost of Cheap Pseudonyms

We consider the problems of societal norms for cooperation and reputation when it is possible to obtain cheap pseudonyms, something that is becoming quite common in a wide variety of interactions on the Internet. This introduces opportunities to misbehave without paying reputational consequences. A large degree of cooperation can still emerge, through a convention in which newcomers “pay their dues” by accepting poor treatment from players who have established positive reputations. One might hope for an open society where newcomers are treated well, but there is an inherent social cost in making the spread of reputations optional. We prove that no equilibrium can sustain significantly more cooperation than the dues‐paying equilibrium in a repeated random matching game with a large number of players in which players have finite lives and the ability to change their identities, and there is a small but nonvanishing probability of mistakes. Although one could remove the inefficiency of mistreating newcomers by disallowing anonymity, this is not practical or desirable in a wide variety of transactions. We discuss the use of entry fees, which permits newcomers to be trusted but excludes some players with low payoffs, thus introducing a different inefficiency. We also discuss the use of free but unreplaceable pseudonyms, and describe a mechanism that implements them using standard encryption techniques, which could be practically implemented in electronic transactions.

Group size effects and collective misalignment in LLM multi-agent systems

Multi-agent systems of large language models (LLMs) are rapidly expanding across domains, introducing dynamics not captured by single-agent evaluations. Yet, existing work has mostly contrasted the behavior of a single agent with that of a collective of fixed size, leaving open a central question: How does group size shape dynamics? Here, we move beyond this dichotomy and systematically explore outcomes across the full range of group sizes. We focus on multi-agent misalignment, building on recent evidence that interacting LLMs playing a simple coordination game can generate collective biases absent in individual models. First, we show that collective bias is a deeper phenomenon than previously assessed: Interaction can amplify individual biases, introduce new ones, or override model-level preferences. Second, we demonstrate that group size affects the dynamics in a nonlinear way, revealing model-dependent dynamical regimes. Finally, we develop a mean-field analytical approach and show that, above a critical population size, simulations converge to deterministic predictions that expose the basins of attraction of competing equilibria. These findings establish group size as a key driver of multi-agent dynamics and highlight the need to consider population-level effects when deploying LLM-based systems at scale.

Collaborative Causal Inference with Fair Incentives

Collaborative causal inference (CCI) aims to improve the estimation of the causal effect of treatment variables by utilizing data aggregated from multiple self-interested parties. Since their source data are valuable proprietary assets that can be costly or tedious to obtain, every party has to be incentivized to be willing to contribute to the collaboration, such as with a guaranteed fair and sufficiently valuable reward (than performing causal inference on its own). This paper presents a reward scheme designed using the unique statistical properties that are required by causal inference to guarantee certain desirable incentive criteria (e.g., fairness, benefit) for the parties based on their contributions. To achieve this, we propose a data valuation function to value parties’ data for CCI based on the distributional closeness of its resulting treatment effect estimate to that utilizing the aggregated data from all parties. Then, we show how to value the parties’ rewards fairly based on a modified variant of the Shapley value arising from our proposed data valuation for CCI. Finally, the Shapley fair rewards to the parties are realized in the form of improved, stochastically perturbed treatment effect estimates. We empirically demonstrate the effectiveness of our reward scheme using simulated and real-world datasets.

What people learn from punishment: A cognitive model

Authorities, from parents of toddlers to leaders of formal institutions, use punishment to communicate disapproval and enforce social norms. Ideally, from whether and how severely a transgression is punished, targets and observers infer shared social norms. Yet in light of every punitive choice, observers also evaluate the motives and legitimacy of the authority. Here, we show that the effects of punishment can only be understood by considering these inferences simultaneously. We measured human observers’ joint inferences empirically in three preregistered experiments ( N = 1 , 254 ) and developed a rational Bayesian model using an inverse planning framework that captures and explains these inferences and their interactions quantitatively and parsimoniously. When people have different priors about norms or authorities, the model predicted and we experimentally confirmed that observing punishment by the authority can sustain polarization. This work reveals the rational logic behind how people learn from punishment and a key constraint on the function of punishment in establishing shared social norms.

Individually Sensible, Collectively Harmful: Simulating Commons Failure in LLM Societies

Large language models are increasingly deployed as agents in simulations of societies, markets, and organizations, raising a fundamental question: can individually sensible AI agents nevertheless produce collectively harmful outcomes? We study this question in a repeated commons dilemma with externalities. We introduce Creeping Trap, a controlled social simulation in which LLM agents repeatedly choose private extraction levels for immediate profit while jointly accumulating catastrophe risk and imposing harm on silent stakeholders. The environment includes analytically derived welfare and behavioral reference strategies, allowing individually sensible behavior and collective welfare to be evaluated simultaneously. Across nine contemporary LLMs and 990 episodes, including a 400-episode confirmatory study, simulated societies consistently over-extract; in the confirmatory study, 396 of 400 episodes produce negative aggregate welfare. This failure is not arbitrary irrationality: observed behavior lies near a broad low-regret region of local best response that even simple heuristics attain, indicating an institutional failure generated by individually sensible decisions. We further show that institutional design changes collective outcomes: surfacing stakeholder harm consistently reduces extraction, whereas shortening accountability horizons increases extraction and induces end-of-term defection. Finally, prompt wording substantially changes measured behavior, making prompt formulation a first-order experimental variable. Creeping Trap provides a controlled testbed for studying institutional interventions that improve collective outcomes in AI societies.

Moral Hazard in Multi-Agent Language Models

Cooperation can fail when socially valuable effort is costly, weakly observable, and mainly benefits others. Drawing on Holmstr\"om's team moral-hazard model, we introduce the Dialogue Moral Hazard Game, a textual hidden-action game for testing whether language agents provide costly, other-benefiting effort. In each episode, an agent can preserve an immediate local reward or pay a query cost to reveal a hidden safety fact that primarily helps another agent choose a team-beneficial final action. We evaluate seven open-weight language models across base and intervention conditions. Base models show two main failure modes: many preserve local reward without achieving team success, while others query often but fail to communicate useful warnings or change downstream final choices. In other words, models rarely complete the intended sequence of querying hidden facts, posting useful warning notes, and using those notes to make team-beneficial final decisions. We then evaluate supervised fine-tuning, RL fine-tuning with RLOO, sequential SFT+RL, and GEPA prompt optimization. Interventions produce limited, model-specific gains: OLMo 7B improves most under LoRA training, while Qwen3 4B benefits most from prompt optimization, though not always through the intended query mechanism. Overall, costly other-benefiting effort is not a default behavior of current open-weight language agents, but can be partially induced.

Why sycophantic LLMs may imperil interactive norms between humans

Interactions with conversational AI are effortless by design—instant, compliant, and largely consequence-free. Human communication norms, by contrast, evolved under conditions of reciprocity and social accountability. We propose that repeated engagement with conversational AI systems may produce norm leakage: the cross-context carryover of instrumental communicative habits acquired in human–AI exchanges into subsequent human–human interaction. Emerging experimental evidence suggests short-term spillover effects on social judgment and behavior, including harsher evaluations, reduced cooperation, and diminished perceived humanness. Preliminary longitudinal findings are consistent with the possibility that such exposure may shape communicative habits over time, although the durability and real-world magnitude of these effects remain unclear. We further propose that sycophantic alignment may amplify norm leakage by reinforcing instrumental interaction styles. At stake, then, is the possibility that repeated engagement with highly compliant artificial agents could subtly influence users’ communicative expectations and interpersonal judgments.

Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?

As the scope of machine learning broadens, we observe a recurring theme of algorithmic monoculture: the same systems, or systems that share components (e.g. datasets, models), are deployed by multiple decision-makers. While sharing offers advantages like amortizing effort, it also has risks. We introduce and formalize one such risk, outcome homogenization: the extent to which particular individuals or groups experience the same outcomes across different deployments. If the same individuals or groups exclusively experience undesirable outcomes, this may institutionalize systemic exclusion and reinscribe social hierarchy. We relate algorithmic monoculture and outcome homogenization by proposing the component sharing hypothesis: if algorithmic systems are increasingly built on the same data or models, then they will increasingly homogenize outcomes. We test this hypothesis on algorithmic fairness benchmarks, demonstrating that increased data-sharing reliably exacerbates homogenization and individual-level effects generally exceed group-level effects. Further, given the current regime in AI of foundation models, i.e. pretrained models that can be adapted to myriad downstream tasks, we test whether model-sharing homogenizes outcomes across tasks. We observe mixed results: we find that for both vision and language settings, the specific methods for adapting a foundation model significantly influence the degree of outcome homogenization. We also identify societal challenges that inhibit the measurement, diagnosis, and rectification of outcome homogenization in deployed machine learning systems.

Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents

Significance The fact that humans enforce and comply with norms is an important reason why humans enjoy higher levels of cooperation and welfare than other animals. Some norms are relatively easy to explain: They may prohibit obviously harmful or uncooperative actions. But many norms are not easy to explain. For example, most cultures prohibit eating certain kinds of foods, and almost all societies have rules about what constitutes appropriate clothing, language, and gestures. Using a computational model focused on learning shows that apparently pointless rules can have an indirect effect on welfare. They can help agents learn how to enforce and comply with norms in general, improving the group’s ability to enforce norms that have a direct effect on welfare. , How do societies learn and maintain social norms? Here we use multiagent reinforcement learning to investigate the learning dynamics of enforcement and compliance behaviors. Artificial agents populate a foraging environment and need to learn to avoid a poisonous berry. Agents learn to avoid eating poisonous berries better when doing so is taboo, meaning the behavior is punished by other agents. The taboo helps overcome a credit assignment problem in discovering delayed health effects. Critically, introducing an additional taboo, which results in punishment for eating a harmless berry, further improves overall returns. This “silly rule” counterintuitively has a positive effect because it gives agents more practice in learning rule enforcement. By probing what individual agents have learned, we demonstrate that normative behavior relies on a sequence of learned skills. Learning rule compliance builds upon prior learning of rule enforcement by other agents. Our results highlight the benefit of employing a multiagent reinforcement learning computational model focused on learning to implement complex actions.