Social Sim @ COLM 2026 — Accepted Papers

When the Generator Grades Itself: A Self-Judge Under-Flags Unsupported Outputs at Matched Scale in LLM Social Simulation

A researcher who validates an LLM social simulation with an automated fidelity check rarely sees which model ran that check. The cheapest and most common choice is the same model that produced the simulation: the model that generated the behavior also certifies that the behavior is faithful. We measure what that choice misses, and we test whether capability rather than family explains it. On n=107 frozen generated responses scored under a support-in-context rubric, we run three judges under identical rubric, prompts, context, and decoding: a same-family self-judge (the 72B generator scoring itself), a matched-scale (~70B) judge from a distinct provider family, and a stronger frontier cross-family judge. The self-judge flags 6/107 unsupported outputs (5.6%); the matched-scale distinct-family judge, in the same capability class but a different family, flags 57/107 (53.3%). The disagreement is perfectly one-directional: 51 items the matched judge flags and the self-judge passes, 0 in reverse (McNemar p<1e-12). This asymmetry is consistent with a self-/family effect and rules out the reading that the self-judge is simply a weaker judge detecting less; one matched-scale judge cannot, on its own, separate that family component from a generic strictness difference between two models. Capability still adds on top: the frontier judge flags more again (92.5%). The cheapest default judge passes nearly every output a same-scale distinct-family judge reads as unsupported, exactly the model-bias artifacts a fidelity check exists to catch. We give a minimal audit protocol and argue that one-directional disagreement should function as a validation rule, not a tuning knob. We test one generator on a context-grounding proxy; transfer to live persona transcripts is the open boundary.

Tracing LLM Social Reasoning to Pretraining Data: A Provenance Audit for Social Simulation

LLM-based social simulations depend on social-reasoning capabilities whose pretraining evidence is usually opaque. We treat training-data attribution as a provenance audit: for an open model and corpus, which training regions are associated with social reasoning, how do those profiles differ from adjacent capabilities, and does intervention confirm that influential data is load-bearing? We compute gradient-based attribution (TrackStar via Bergson) over a working set from the de-duplicated Dolma3 mix, aggregate influence across WebOrganizer’s 24-format ×24-topic taxonomy (576 bins), and contrast SocialIQA and MMLU Social Sciences against ARC-Challenge and MMLU STEM. In OLMo3-7B, SocialIQA is the profile outlier: its positive attribution is more diffuse and shifted toward interpersonal and narrative regions rather than documentation-heavy ones. Influence-targeted unlearning validates this association most clearly for SocialIQA, while the comparison tasks show weaker, null, or reversed effects. A held-out social-probe suite tests the breadth and limits of the pattern without claiming population-level simulation fidelity.

The Effect Of Warning Message Length on Language Model Simulated Response in Emergency Situations

The following study introduces a new agentic AI social simulation program (the Hurricane Evacuation Response Optimizer [HERO] program) designed to simulate the responses of advantaged and disadvantaged populations to hurricane relevant alerts and warnings used by the United States National Weather Service (NWS) and the Federal Emergency Management Agency (FEMA). In this project, we evaluate the responses of simulated public civilians to Flash Flood warnings and WEA alerts, assessing if there is a statistical difference in simulation response to longer and shorter emergency messages. We also assess how agent responses to LLM generated, real-world, and nonsensical alerts and warnings compare to one another through cosine similarity and summary statistics. The study concludes that our social simulation responds to prompts with varying character lengths in a statistically different manner, with higher metrics of understanding and greater variance in responses towards WEA alerts when compared to Flash Flood warnings. We also find that LLM-generated alerts were more similar through use of cosine similarity to real-world alerts than nonsensical ones.

Synonymix: Unified Group Personas for Generative Simulations

Generative agent simulations operate at two scales: individual personas for character interaction, and population models for collective behavior analysis and intervention testing. We propose a third scale: meso-level simulation - interaction with group-level representations that retain grounding in rich individual experience. To enable this, we present Synonymix, a pipeline that constructs a "unigraph" from multiple life story personas via graph-based abstraction and merging, producing a queryable collective representation that can be explored for sensemaking or sampled for synthetic persona generation. Evaluating synthetic agents on General Social Survey items, we demonstrate behavioral signal preservation beyond demographic baselines (p<0.001, r=0.59) with demonstrable privacy guarantee (max source contribution <13%). We invite discussion on interaction modalities enabled by meso-level simulations, and whether "high-fidelity" personas can ever capture the texture of lived experience.

SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery

Scaling qualitative data collection requires automating semi-structured interviews. However, existing LLM-based systems largely focus on covering predefined topics and struggle to follow up on emergent themes that arise organically in participant responses. This limitation stems from the absence of an explicit objective that formalizes the trade-off between covering a predefined topic guide and adaptive exploration. In this work, we frame adaptive semi-structured interviewing as an optimization problem and define interview utility as balancing coverage of a predefined topic guide, discovery of emergent themes, and interview cost. Based on this formulation, we introduce SparkMe, a multi-agent LLM interviewer that performs proactive planning via simulated conversation rollouts to select high-utility questions. SparkMe achieves higher topic guide coverage ($+4.7$% over the best baseline) and elicits richer emergent insights while using fewer conversational turns. We further validate SparkMe in a user study with $70$ participants across $7$ professions, where domain experts rate it as producing high-quality adaptive interviews that surface profession-specific insights not captured by baseline systems. The code and datasets for SparkMe will be released upon acceptance.

Social OMNI-EPIC: Learning-Progress-Driven Curriculum Generation for Social Interaction

To address the rapid saturation of fixed social benchmarks by increasingly capable models, we introduce Social OMNI-EPIC, an open-ended curriculum generation framework that adapts learning-progress-driven environment design to LLM-based social simulation. Starting from a seed archive, it continuously generates scenarios targeting a learner's evolving capability frontier using intrinsic signals of learning progress and interestingness. Across 90 generated scenarios, Social OMNI-EPIC shifts the distribution from predominantly easy scenarios toward frontier-level challenges and becomes increasingly calibrated over time.

Social Choice Foundations for Simulation-Augmented Generation

Users increasingly turn to AI systems for normative assistance—guidance on what one ought to do or think—yet models are often opaque about whose viewpoints they represent. A promising approach is simulation-augmented generation (SAGE), which involves querying generative simulations of individuals in a target population at inference time, soliciting their open-ended judgments, and synthesizing them into a response while transparently reporting whose viewpoints are reflected. However, inference-time simulation raises acute scalability constraints. Since the key benefit of simulation is improved representativeness, the core challenge is scaling simulation without sacrificing representation. We introduce the first formalization of this problem, grounded in proportional clustering concepts from social choice theory. We prove that to represent a population of m humans, we need only create n << m simulations of them, and need only dynamically query k << n of those simulations at inference time, while still maintaining approximate proportional representation guarantees for the full population. We empirically investigate how the guarantees achieved by our method are affected by the fidelity between the simulation and the human.

Simulating Aggregate Electoral Opinion Trajectories with LLM Personas and Media Exposure

LLM-based election simulation has largely been evaluated against a single final outcome, in many cases neglecting how much it can recover the trajectory shaped by media exposure and shifting candidate support. We propose evaluating election simulations in terms of trajectory fidelity: whether persona-conditioned LLM agents can update their beliefs in response to media to reproduce the temporal movement of public opinion, both in aggregate and across demographic subgroups. We design a framework in which each LLM agent carries its belief across time and updates it as new campaign information arrives. We further investigate how performance can vary across simulation design choices such as media exposure model, election context, and survey questionnaire. In a study on the 2026 Seoul mayoral and local elections, we validate simulated trajectories against weekly public-opinion polls and find that simulations that carry prior beliefs across time—rather than simulating each poll date independently—substantially improves alignment with observed polling movement, while the way media exposure is personalized leaves aggregate movement largely intact but reshapes subgroup-level dynamics. We propose a framework for validating simulated opinion trajectories against real-world traces, and discuss the simulation design choices that can influence the performance.

Shall We Play a Game? Language Models for Open-ended Wargames

LLM-based social simulations can make a generated transcript look like a single behavioral signal, but the model behind that transcript may be doing several different jobs: choosing what an actor says or does, deciding what happens after an action, or both. The difference matters especially in open-ended wargames, where models are prized for handling unusual actions and ambiguous consequences. We report a scoping review of 223 de-duplicated AI-in-wargames and strategic-simulation papers retrieved through May 1, 2026, describing each simulation by its model-control profile: whether the language model has open-ended control over player actions, adjudication, or both. Only 20 of 223 studies (≈9%) give language models both roles. Before treating LM outputs as social simulations, researchers need to know how much creative control the model has over actions and consequences. For open-ended simulations, fidelity depends not only on whether agents behave plausibly, but also on whether language models can reliably act as adjudicators or world models.

Role Steering of Language Models for Social Simulations

Social simulations built from language-model agents need role-conditioned behavior that can be checked before agents are placed into a simulated population. We introduce an activation-steering screening workflow for role-conditioned agents: define a role profile, extract a role-specific direction, sweep four steering coefficients, evaluate role-profile alignment, and pass or flag each candidate configuration. On OLMo-3-7B-Instruct, we apply the workflow to a mixed 275-role inventory with 228 role-agnostic questions, GPT-4.1-mini prompted role references, and GPT-4.1-mini judges. We also introduce CastVectors, which receive higher judged role-profile alignment than an assistant-axis directional control from prior persona-vector work, with mean overall scores of 63.2 versus 41.1 across the tested grid. They also preserve high lexical diversity, while the control drops sharply at larger coefficients. The role-level screen is the main contribution: most roles improve as steering increases, but 38 roles decline across all six measured dimensions, showing why simulation builders should choose coefficients per role rather than deploy a uniform high-strength setting. We make our code and evaluation artifacts available at https://github.com/eilab-gt/casting-call-vectors

Reason-Mediated Behavioral Models for Auditing LLM Social Simulators

Large language models are increasingly used as social simulators, including as synthetic survey respondents. Most evaluations ask whether simulated outcomes resemble human outcomes. We argue that this is necessary but too weak: a simulator can match the final answer while using the wrong behavioral mechanism. We study this problem through a 94-person sunscreen concept test in which each respondent evaluated three product concepts and wrote open-ended rationales. We map those rationales into signed reason states $Z$, where positive signs support adoption and negative signs block it. This gives a practical audit: holding respondent descriptors $D$, category context $K$, and concept treatment $X$ fixed, do human-stated reasons help predict behavior $Y$, and can an LLM simulate the same reason state without seeing the human rationale or outcome? Human-stated reasons substantially improve held-out prediction of purchase intent. LLM-simulated reasons are more brittle: they often sound plausible, but frequently echo the concept board rather than recover the respondent's acceptance or rejection path. The paper contributes an evaluation framework for social simulators. Reason states do not identify natural causal effects by themselves, but they provide an interpretable test of whether a simulator's stated reasons align with human evidence.

Psychometric Framework for Comparing Alignment- Conditioned Behavior in Large Language Models

This paper introduces an interpretive framework for comparing the behavioral tendencies that post-training alignment induces in large language models (LLMs). Post-training alignment techniques including Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and constitutional or rule-based prompting strategies steer LLMs toward human-preferred behavior. Using a scenario-based cross-model psychometric framework grounded in the Big Five personality traits (OCEAN: Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism), we infer personality-like profiles for four prominent LLMs. We then map these profiles onto the responsiveness and demandingness dimensions from developmental psychology's parenting-styles theory, positioning each model in a two-dimensional responsiveness–demandingness space. Treating this mapping as an interpretive lens rather than a definitive measure of alignment quality, we examine how models distribute across the four parenting archetypes and discuss the assumptions required to read these positions normatively. This framing offers an exploratory and interpretable notion of "parenting LLMs" for reasoning about alignment regimes, while we explicitly delineate the limits of the analogy. We validate the framework with two independent human annotators on 440 paired ratings; both annotators yield moderate agreement with the LLM-based scoring and the relative ranking of models is preserved across all three rater sources.

PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior

Large language models (LLMs) are increasingly used to simulate human behavior, but their ability to simulate individual privacy decisions is not well understood. In this paper, we address the problem of evaluating whether a core set of user persona attributes can drive LLMs to simulate individual-level privacy behavior. We introduce PrivacySIM, an evaluation suite that benchmarks LLM simulation of user privacy behavior against the ground-truth responses of $1{,}000$ users. These users are drawn from five published user studies on privacy spanning LLM healthcare consultations, conversational agents, and chatbots. Drawing on these user studies, we hypothesize three persona facets as plausible predictors of privacy decision-making: demographics, previous experiences, and stated privacy attitudes. We condition nine frontier LLMs on subsets of these three facets and measure how often each model's response to a data-sharing scenario matches the user's actual response. Our findings show that (1) privacy persona conditioning consistently improves simulation quality over no-persona conditioning, but even the strongest model (40.4% accuracy) remains far from faithfully simulating individual privacy decisions. (2) A user's stated privacy attitudes alone may not be the best predictor because they often diverge from the user's actual privacy behavior. (3) Users with high AI/chatbot experience but low stated privacy attitudes are the most challenging to simulate. PrivacySIM is a first step toward understanding and improving the capabilities of LLMs to simulate user privacy decisions. We release PrivacySIM to enable further evaluation of LLM privacy simulation.

Position: Synthetic Persona Needs Explicit Grounding and Standardized Reporting

This position paper argues that synthetic personas used in Large Language Models (LLM) social simulations need explicit grounding. By grounding, we mean that researchers should state where persona information comes from, what social actor or population it is meant to represent, what evidence supports it, and what kinds of simulation claims it can justify. Without such grounding, persona-based simulations risk treating appearance of plausible descriptions or vivid narratives as evidence of realism. We identify several recurring failure modes that explicit grounding can help expose, and argue that these failures should not be diagnosed only after simulation results are produced; they should be addressed at the level of persona construction and reporting. We conclude by proposing lightweight reporting standards for persona provenance, construction process, selection or sampling logic, internal consistency checks, model enactment checks, and intended inference.

Position: AI Is Not Ready for Strategic Conflicts

Open-ended strategic wargames are high-stakes LM-based social simulations: they model adversaries, institutions, escalation, plan brittleness, doctrine, and crisis response. Language models (LMs) are attractive in these settings because they can play agents, generate scenario branches, adjudicate ambiguous actions, and summarize lessons. The same affordances make open-ended player and adjudicator roles dangerous: model language can determine both what an actor attempts and what subsequently becomes simulated reality. This position paper argues that no LM-enabled wargame should inform high-stakes planning, doctrine, policy, or crisis response without an auditable safety case, and that the proper use of openended wargames today is to stress-test decision-influencing LM agents. We identify five failure modes: decision laundering, adjudication opacity, role collapse, escalation-through-adjudication, and failure of strategic imagination. Ordinary benchmarks cannot establish safety for these settings. Wargames can expose failures as stress tests; they are not themselves safety cases for consequential use. Until such safety cases exist, AI is not ready for strategic conflicts.

PersonaEval: Persona-Based User Simulation for Evaluating Interactive Applications

Real user studies are important for understanding how people interact with systems under test or already deployed. In practice, however, they are often costly, time-consuming, and difficult to scale. To address these challenges, we introduce PersonaEval, a persona-based user simulation framework that approximates real-user behavior across diverse interactive settings. PersonaEval connects simulated users drawn from existing persona datasets to task-specific application interfaces and collects the interaction trajectories and outcomes. PersonaEval provides a plug-and-play evaluation workflow in which the application being evaluated can be easily changed. In this demo, we present PersonaEval on three forms of interactive applications: surveys, chatbots, and web applications. Together, these examples show that PersonaEval can support repeatable, parallelizable, and scalable evaluation across different interaction settings, while producing user-oriented feedback and task-specific behavior.

PSI-Bench: Towards Clinically Grounded and Interpretable Evaluation of Depression Patient Simulators

Patient simulators are gaining traction in mental health training by providing scalable exposure to complex and sensitive patient interactions. Simulating depressed patients is particularly challenging, as safety constraints and high patient variability complicate simulations and underscore the need for simulators that capture diverse and realistic patient behaviors. However, existing evaluations heavily rely on LLM-judges with poorly specified prompts and do not assess behavioral diversity. We introduce PSI-Bench, an automatic evaluation framework that provides interpretable, clinically grounded diagnostics of depression patient simulator behavior across turn-, dialogue-, and population-level dimensions. Using PSI-Bench, we benchmark seven LLMs across two simulator frameworks and find that simulators produce overly long, lexically diverse responses, show reduced variability, resolve emotions too quickly, and follow a uniform negative-to-positive trajectory. We also show that the simulation framework has a larger impact on fidelity than the model scale. Results from a human study demonstrate that our benchmark is strongly aligned with mental health professionals' judgments. Our work reveals key limitations of current depression patient simulators and provides an interpretable, extensible benchmark to guide future simulator design and evaluation.

PB-GRPO: Learning Socially Adaptive LLM Agents from Persona-Driven Simulation with Preference-Batched GRPO

Building LLMs that behave well socially, not merely correctly, requires more than producing locally helpful responses. A socially competent agent must infer users’ unstated goals, respect their preferences, adapt as the conversation unfolds, and know when to ask rather than act. These behaviors are inherently multi-turn and social, making them hard to optimize: real interaction data is scarce, and user preferences are typically latent rather than directly observable. To address these challenges, we build on a persona-driven social simulation environment (consisting of a persona library, LLM-based user simulators, and a user-satisfaction scoring system ranging from [0, 1]), to introduce preference-batched GRPO (PB-GRPO), a post-training algorithm that learns socially adaptive policies from conversation-level feedback. Compared to vanilla GRPO, PB-GRPO computes advantages using a normalization estimated across a bucket of users with similar preferences, stabilizing training across a diverse social population. Empirical evidence shows that PB-GRPO improves models’ social behavior over strong reinforcement-learning baselines in our simulated environment.

Optimizing Behavioral Experiment Design with In-Silico Simulations

Running behavioral experiments is expensive, slow, and vulnerable to design flaws detectable only after data collection. We introduce a preliminary multi-agent framework that uses simulations to iteratively refine behavioral study designs before recruiting human participants. With personas instantiated through target demographic attributes, we elicit explanatory reasoning traces from LLM simulations that serve as diagnostic evidence of structural flaws. In two survey case studies, simulation-rich Critic outperforms a design-only baseline on the precision of high-severity structural errors, while providing little value for identifying missed opportunities. These early results suggest reasoning traces can expose a qualitatively distinct class of design flaws inaccessible to the design-only baseline.

One Crowd, Three Cascades: Information Diffusion in LLM-Agent Simulations Is Governed by Serialization, Not Society

LLM-agent simulations are increasingly used to study how information spreads through a population, and are validated by matching macroscopic cascade statistics—final reach, structural virality, cascade-size shape—to real diffusion data. We argue this cannot certify fidelity, and show it empirically. We measure an LLM agent's exposure-response curve—its probability of adopting given $k$ of $d$ active neighbors—then hold that social fact fixed and change only its serialization: a count ("3 of 10"), a fraction ("30%"), or an enumeration of names. The encodings are semantically identical, so a faithful mechanism must be invariant to them. It is not. Across five scenarios on a local model, the inferred contagion class (simple vs. complex) flips with serialization in four of five, and the final epidemic size on a fixed network and seed changes by a median factor of $1.9\times$ (over $100\times$ when an enumerated list collapses adoption); aggregate encodings even produce a non-monotonic reversal at saturation that no contagion model (SIR, independent cascade, linear threshold) admits. Since one can reach almost any target cascade by choosing the encoding, matching a macroscopic observable does not identify a social mechanism. We turn semantic invariance into a cheap, API-free serialization-invariance diagnostic: rerun under equivalent serializations of neighbor state; if the cascade diverges, the result is a typographical artifact, not a social dynamic. All experiments run on small local models on a laptop.