Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents

Learning how to behave: cognitive learning processes account for asymmetries in adaptation to social norms

Changes to social settings caused by migration, cultural change or pandemics force us to adapt to new social norms. Social norms provide groups of individuals with behavioural prescriptions and therefore can be inferred by observing their behaviour. This work aims to examine how cognitive learning processes affect adaptation and learning of new social norms. Using a multiplayer game, I found that participants initially complied with various social norms exhibited by the behaviour of bot-players. After gaining experience with one norm, adaptation to a new norm was observed in all cases but one, where an active-harm norm was resistant to adaptation. Using computational learning models, I found that active behaviours were learned faster than omissions, and harmful behaviours were more readily attributed to all group members than beneficial behaviours. These results provide a cognitive foundation for learning and adaptation to descriptive norms and can inform future investigations of group-level learning and cross-cultural adaptation.

Social learning rules and the effectiveness of behavioural policy: an agent-based model

Behaviour-change interventions unfold in social systems where people learn from others. We develop a stylised agent-based model to examine how four canonical social learning rules – conformist transmission, informational prestige-biased copying, payoff-biased copying and random copying – shape the impact of a simple seeding intervention. Two arms evolve under identical conditions and learning rules, differing only in initial adoption: both start with exactly 5% baseline adopters and the treatment arm additionally seeds 20% of the remaining non-adopters, yielding an exact 25% vs 5% contrast at $t = 0$. Across homogeneous populations, 70/30 mixed ecologies and sweeps over the share of payoff-biased learners, we track adoption trajectories and treatment–control lift; we also vary payoff parameters, prestige informativeness and conformist thresholds in robustness analyses. We find that the same seeding intervention can stall, drift or cascade depending on the learning ecology. In the baseline specification, conformist dynamics exhibit threshold effects that erase treatment gains, prestige-biased and random copying can preserve positive final lift when diffusion remains incomplete and payoff-biased copying mainly changes the diffusion regime rather than preserving large end-point gaps. Robustness checks show that negative payoff premia suppress diffusion, weak or noisy payoff signals can generate treatment advantages, prestige effects depend on how informative prestige is and conformist treatment effects are concentrated in narrow threshold-boundary regions. These results motivate policy heuristics that evaluate interventions relative to local diffusion potential, make successful outcomes visible when payoff cues matter and tailor seeding to the prevailing mix of learning rules.

The Psychology of Normative Cognition

From an early age, humans exhibit a tendency to identify, adopt, andenforce the norms of their local communities. Norms are the socialrules that mark out what is appropriate, allowed, required, orforbidden in different situations for various community members. Theserules are informal in the sense that although they are sometimesrepresented in formal laws, like the rule governing which side of theroad to drive on, they need not be explicitly codified to effectivelyinfluence behavior. There are rules that forbid theft or the breakingof promises, but also rules that govern how close it is appropriate tostand to someone while talking to them, or how loud one should talkduring the conversation. Thus understood, norms regulate a wide rangeof activity. They exhibit cultural variability in their prescriptionsand proscriptions, but the presence of norms in general appears to beculturally universal. Some norms exhibit characteristics that areoften associated with morality, such as a rule that applies toeveryone and prohibits causing unnecessary harm. Other norms applyonly to certain people, such as those that delimit appropriateclothing for members of different genders, or those concerning theexpectations and responsibilities ascribed to individuals who occupypositions of leadership. The norms that prevail in a community can bemore or less fair, reasonable, or impartial, and can be subject tocritique and change.

Rethinking Norm Psychology

Norms permeate human life. Most of people’s activities can be characterized by rules about what is appropriate, allowed, required, or forbidden—rules that are crucial in making people hyper-cooperative animals. In this article, I examine the current cognitive-evolutionary account of “norm psychology” and propose an alternative that is better supported by evidence and better placed to promote interdisciplinary dialogue. The incumbent theory focuses on rules and claims that humans genetically inherit cognitive and motivational mechanisms specialized for processing these rules. The cultural-evolutionary alternative defines normativity in relation to behavior—compliance, enforcement, and commentary—and suggests that it depends on implicit and explicit processes. The implicit processes are genetically inherited and domain-general; rather than being specialized for normativity, they do many jobs in many species. The explicit processes are culturally inherited and domain-specific; they are constructed from mentalizing and reasoning by social interaction in childhood. The cultural-evolutionary, or “cognitive gadget,” perspective suggests that people alive today—parents, educators, elders, politicians, lawyers—have more responsibility for sustaining normativity than the nativist view implies. People’s actions not only shape and transmit the rules, but they also create in each new generation mental processes that can grasp the rules and put them into action.

Integrative experiments identify how punishment affects welfare in public goods games

Despite decades of research, the conditions under which punishment promotes cooperation remain unclear. Through an integrative experiment varying 14 design parameters of public goods games across 360 experimental conditions (147,618 decisions from 7100 participants), we reveal substantial heterogeneity in punishment effectiveness: Its impact on welfare ranges from 43% improvement to 44% reduction depending on the game parameters. To characterize these patterns, we developed models that outperformed human forecasters in predicting punishment effectiveness in new experiments. Communication emerges as the most important factor, followed by contribution framing (opt out versus opt in), contribution type (variable versus all-or-nothing), game length, and outcome visibility, though these factors often interact. The results reframe the debate from whether punishment works to when it does, demonstrating how integrative experiments enable discovery of generalizable patterns in social phenomena. , Editor’s summary People face conflicts between maximizing personal gain versus supporting collective interests. If we cooperatively recycle or donate to charities, it benefits society, but it also costs us time and resources that could be selfishly preserved for ourselves. We impose penalties to deter those undesirable or selfish behaviors, but under what conditions do punishments or penalties effectively modify behavior to benefit group welfare? Alsobay et al . systematically and simultaneously varied 14 factors together instead of in isolation. Punishment was unequivocally most effective when paired with consistent communication, particularly over time. Another effective factor was “opting out” or withdrawing some, but not all, endowments already in the public fund. These methodological advances revealed when, rather than whether, punishment works. —Ekeoma Uzogara , INTRODUCTION Human societies face many situations where individual and collective interests conflict, often referred to as social dilemmas. Costly peer punishment has been studied for more than 25 years in public goods games (stylized behavioral experiments in which individuals decide how much to contribute to a shared pool that benefits everyone) as a mechanism to promote cooperation. Prior research has identified many contextual factors that moderate punishment’s effectiveness, including game length, communication, group size, punishment cost, and so on. However, the specific conditions under which punishment improves group welfare remain unclear. RATIONALE We argue that this lack of clarity derives from the dominant experimental paradigm, in which any given study manipulates only one or a few theoretically informed factors. Because such studies differ in many ways (different experimental procedures, populations), their results are often difficult to compare or integrate. Consequently, one can list many factors that have some effect, but cannot say how much each matters relative to the others, or how they work together, and as a result, cannot predict when punishment will help or harm welfare in new settings. To address this fundamental knowledge gap, we use an integrative experimental design and systematically vary 14 parameters across 360 conditions (147,618 decisions from 7100 participants) to elucidate when punishment improves versus undermines welfare in public goods games, which factors matter most, and how they interact. RESULTS The effect of punishment on welfare ranged from 43% improvement to 44% reduction depending on the specific combination of game parameters. To characterize this heterogeneity, we trained a model that outperformed all 553 human forecasters (laypeople and experts) in predicting whether punishment would help or harm welfare in new experiments. Communication emerged as roughly three times more important than any other factor, followed by contribution framing (opt in versus opt out), contribution type (variable versus all-or-nothing), game length, and peer outcome visibility (whether participants can see others’ earnings). These factors often interact. For example, longer games enhance punishment’s effectiveness only when communication is available, and contribution framing effects depend on both contribution type and outcome visibility. CONCLUSION Many phenomena in social science are shaped by many factors whose interactions are consequential, yet the dominant experimental paradigm often limits its inquiry to “does a given effect exist?” and examines hypothesized factors in isolation. As a result, research programs can accumulate many partial explanations without a clear picture of how they combine to determine outcomes across settings. Knowing that factors matter individually is fundamentally different from knowing how much each matters and how they interact. The integrative approach implemented here offers one way forward. It varies many factors simultaneously within a shared design space, evaluates models by their predictive accuracy on new experiments, and probes those models to constrain and develop theory. Our hope is that integrative experiment designs, combined with models that integrate prediction and explanation, represent a path toward more cumulative social science. Integrative experiment reveals when punishment helps versus harms. We systematically varied 14 design parameters across 360 experimental conditions. The effect of punishment on cooperation efficiency ranged from −44% to +43% depending on the specific game parameters. Communication emerged as three times more important than any other factor, followed by contribution framing, contribution type, and game length.

How we learn social norms: a three-stage model for social norm learning

As social animals, humans are unique to make the world function well by developing, maintaining, and enforcing social norms. As a prerequisite among these norm-related processes, learning social norms can act as a basis that helps us quickly coordinate with others, which is beneficial to social inclusion when people enter into a new environment or experience certain sociocultural changes. Given the positive effects of learning social norms on social order and sociocultural adaptability in daily life, there is an urgent need to understand the underlying mechanisms of social norm learning. In this article, we review a set of works regarding social norms and highlight the specificity of social norm learning. We then propose an integrated model of social norm learning containing three stages, i.e., pre-learning, reinforcement learning, and internalization, map a potential brain network in processing social norm learning, and further discuss the potential influencing factors that modulate social norm learning. Finally, we outline a couple of future directions along this line, including theoretical (i.e., societal and individual differences in social norm learning), methodological (i.e., longitudinal research, experimental methods, neuroimaging studies), and practical issues.

What people learn from punishment: A cognitive model

Authorities, from parents of toddlers to leaders of formal institutions, use punishment to communicate disapproval and enforce social norms. Ideally, from whether and how severely a transgression is punished, targets and observers infer shared social norms. Yet in light of every punitive choice, observers also evaluate the motives and legitimacy of the authority. Here, we show that the effects of punishment can only be understood by considering these inferences simultaneously. We measured human observers’ joint inferences empirically in three preregistered experiments ( N = 1 , 254 ) and developed a rational Bayesian model using an inverse planning framework that captures and explains these inferences and their interactions quantitatively and parsimoniously. When people have different priors about norms or authorities, the model predicted and we experimentally confirmed that observing punishment by the authority can sustain polarization. This work reveals the rational logic behind how people learn from punishment and a key constraint on the function of punishment in establishing shared social norms.

Bayesian Inference of Social Norms as Shared Constraints on Behavior

People act upon their desires, but often, also act in adherence to implicit social norms. How do people infer these unstated social norms from others' behavior, especially in novel social contexts? We propose that laypeople have intuitive theories of social norms as behavioral constraints shared across different agents in the same social context. We formalize inference of norms using a Bayesian Theory of Mind approach, and show that this computational approach provides excellent predictions of how people infer norms in two scenarios. Our results suggest that people separate the influence of norms and individual desires on others' actions, and have implications for modelling generalizations of hidden causes of behavior.

Mechanisms of social cognition

Social animals including humans share a range of social mechanisms that are automatic and implicit and enable learning by observation. Learning from others includes imitation of actions and mirroring of emotions. Learning about others, such as their group membership and reputation, is crucial for social interactions that depend on trust. For accurate prediction of others' changeable dispositions, mentalizing is required, i.e., tracking of intentions, desires, and beliefs. Implicit mentalizing is present in infants less than one year old as well as in some nonhuman species. Explicit mentalizing is a meta-cognitive process and enhances the ability to learn about the world through self-monitoring and reflection, and may be uniquely human. Meta-cognitive processes can also exert control over automatic behavior, for instance, when short-term gains oppose long-term aims or when selfish and prosocial interests collide. We suggest that they also underlie the ability to explicitly share experiences with other agents, as in reflective discussion and teaching. These are key in increasing the accuracy of the models of the world that we construct.

Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences

In social settings, much of human behavior is governed by unspoken rules of conduct rooted in societal norms. For artificial systems to be fully integrated into social environments, adherence to such norms is a central prerequisite. To investigate whether language generation models can serve as behavioral priors for systems deployed in social settings, we evaluate their ability to generate action descriptions that achieve predefined goals under normative constraints. Moreover, we examine if models can anticipate likely consequences of actions that either observe or violate known norms, or explain why certain actions are preferable by generating relevant norm hypotheses. For this purpose, we introduce Moral Stories, a crowd-sourced dataset of structured, branching narratives for the study of grounded, goal-oriented social reasoning. Finally, we propose decoding strategies that combine multiple expert models to significantly improve the quality of generated actions, consequences, and norms compared to strong baselines.

The Emergence of Social Norms and Conventions

The utility of our actions frequently depends upon the beliefs and behavior of other agents. Thankfully, through experience, we learn norms and conventions that provide stable expectations for navigating our social world. Here, we review several distinct influences on their content and distribution. At the level of individuals locally interacting in dyads, success depends on rapidly adapting pre-existing norms to the local context. Hence, norms are shaped by complex cognitive processes involved in learning and social reasoning.

AI for identifying social norm violation

Identifying social norms and their violation is a challenge facing several projects in computational science. This paper presents a novel approach to identifying social norm violations. We used GPT-3, zero-shot classification, and automatic rule discovery to develop simple predictive models grounded in psychological knowledge. Tested on two massive datasets, the models present significant predictive performance and show that even complex social situations can be functionally analyzed through modern computational tools.

Why sycophantic LLMs may imperil interactive norms between humans

Interactions with conversational AI are effortless by design—instant, compliant, and largely consequence-free. Human communication norms, by contrast, evolved under conditions of reciprocity and social accountability. We propose that repeated engagement with conversational AI systems may produce norm leakage: the cross-context carryover of instrumental communicative habits acquired in human–AI exchanges into subsequent human–human interaction. Emerging experimental evidence suggests short-term spillover effects on social judgment and behavior, including harsher evaluations, reduced cooperation, and diminished perceived humanness. Preliminary longitudinal findings are consistent with the possibility that such exposure may shape communicative habits over time, although the durability and real-world magnitude of these effects remain unclear. We further propose that sycophantic alignment may amplify norm leakage by reinforcing instrumental interaction styles. At stake, then, is the possibility that repeated engagement with highly compliant artificial agents could subtly influence users’ communicative expectations and interpersonal judgments.

The evolution of cultural adaptations: Fijian food taboos protect against dangerous marine toxins

The application of evolutionary theory to understanding the origins of our species' capacities for social learning has generated key insights into cultural evolution. By focusing on how our psychology has evolved to adaptively extract beliefs and practices by observing others, theorists have hypothesized how social learning can, over generations, give rise to culturally evolved adaptations. While much field research documents the subtle ways in which culturally transmitted beliefs and practices adapt people to their local environments, and much experimental work reveals the predicted patterns of social learning, little research connects real-world adaptive cultural traits to the patterns of transmission predicted by these theories. Addressing this gap, we show how food taboos for pregnant and lactating women in Fiji selectively target the most toxic marine species, effectively reducing a woman's chances of fish poisoning by 30 per cent during pregnancy and 60 per cent during breastfeeding. We further analyse how these taboos are transmitted, showing support for cultural evolutionary models that combine familial transmission with selective learning from locally prestigious individuals. In addition, we explore how particular aspects of human cognitive processes increase the frequency of some non-adaptive taboos. This case demonstrates how evolutionary theory can be deployed to explain both adaptive and non-adaptive behavioural patterns.

Moral Hazard in Multi-Agent Language Models

Cooperation can fail when socially valuable effort is costly, weakly observable, and mainly benefits others. Drawing on Holmstr\"om's team moral-hazard model, we introduce the Dialogue Moral Hazard Game, a textual hidden-action game for testing whether language agents provide costly, other-benefiting effort. In each episode, an agent can preserve an immediate local reward or pay a query cost to reveal a hidden safety fact that primarily helps another agent choose a team-beneficial final action. We evaluate seven open-weight language models across base and intervention conditions. Base models show two main failure modes: many preserve local reward without achieving team success, while others query often but fail to communicate useful warnings or change downstream final choices. In other words, models rarely complete the intended sequence of querying hidden facts, posting useful warning notes, and using those notes to make team-beneficial final decisions. We then evaluate supervised fine-tuning, RL fine-tuning with RLOO, sequential SFT+RL, and GEPA prompt optimization. Interventions produce limited, model-specific gains: OLMo 7B improves most under LoRA training, while Qwen3 4B benefits most from prompt optimization, though not always through the intended query mechanism. Overall, costly other-benefiting effort is not a default behavior of current open-weight language agents, but can be partially induced.