Surveys are an essential approach for eliciting otherwise invisible factors such as perceptions, knowledge and beliefs, attitudes, and reasoning. These factors are critical determinants of social, economic, and political outcomes. Surveys are not merely a research tool. They are also not only a way of collecting data. Instead, they involve creating the process that will generate the data. This allows the researcher to create their own identifying and controlled variation. Thanks to the rise of mobile technologies and platforms, surveys offer valuable opportunities either to study broadly representative samples or to focus on specific groups. This article offers guidance on the complete survey process, from the design of the questions and experiments to the recruitment of respondents and the collection of data to the analysis of survey responses. It covers issues related to the sampling process, selection and attrition, attention and carelessness, survey question design and measurement, response biases, and survey experiments.
Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization
The growing popularity of generative artificial intelligence (AI) tools presents new challenges for data quality in online surveys and experiments. This study examines participants’ use of large language models to answer open-ended survey questions and describes empirical tendencies in human versus large language model (LLM)-generated text responses. In an original survey of research participants recruited from a popular online platform for sourcing social science research subjects, 34 percent reported using LLMs to help them answer open-ended survey questions. Simulations comparing human-written responses from three pre-ChatGPT studies with LLM-generated text reveal that LLM responses are more homogeneous and positive, particularly when they describe social groups in sensitive questions. These homogenization patterns may mask important underlying social variation in attitudes and beliefs among human subjects, raising concerns about data validity. Our findings shed light on the scope and potential consequences of participants’ LLM use in online research.
Silicon sampling answers with country-level assumptions, not individual attitudes: Cross-national evidence from the European Social Survey
Silicon sampling uses large language models (LLMs) to simulate survey respondents. Whether it recovers cross-national variation, and why, remains unresolved. This study evaluates it against European Social Survey Round 11 (30 countries, 42 items) with two open-weight LLMs under first- and third-person prompts, plus backstory and response-format experiments. Aggregate recovery is moderate and uneven across items. Adding the country name to a three-variable demographic backstory raises the median per-item correlation between simulated and observed country means from -0.03 to 0.52, and the richer profiles tested add no consistent gain. The respondent's country label acts as a country-level assumption that respondent detail does not revise. Naming the response-scale endpoints in words stops the model from ranking countries backwards, so the answer format sets the direction of the ranking. Individual-level recovery remains negligible in every condition and does not track aggregate recovery across countries. An average of neighboring countries, using no LLM, recovers country levels more accurately than every model condition and ranks them about as well. Silicon sampling can thus support exploratory country-ranking comparison after item-level validation and with the response format reported. It does not support individual or distributional inference.
The Internet Doesn’t Have to Be Bad for Democracy
A new form of online survey uses crowdsourcing and data visualization to reveal the hidden nuances in partisan debates.
Opinion | This Is What Will Ruin Public Opinion Polling for Good
Instead of navigating the obstacles to conduct polls with human respondents, pollsters are running A.I. simulations instead. Why?
Do LLMs Exhibit Human-like Response Biases? A Case Study in Survey Design
Abstract One widely cited barrier to the adoption of LLMs as proxies for humans in subjective tasks is their sensitivity to prompt wording—but interestingly, humans also display sensitivities to instruction changes in the form of response biases. We investigate the extent to which LLMs reflect human response biases, if at all. We look to survey design, where human response biases caused by changes in the wordings of “prompts” have been extensively explored in social psychology literature. Drawing from these works, we design a dataset and framework to evaluate whether LLMs exhibit human-like response biases in survey questionnaires. Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior, particularly in models that have undergone RLHF. Furthermore, even if a model shows a significant change in the same direction as humans, we find that they are sensitive to perturbations that do not elicit significant changes in humans. These results highlight the pitfalls of using LLMs as human proxies, and underscore the need for finer-grained characterizations of model behavior.1
Designing Information Provision Experiments
Information provision experiments allow researchers to test economic theories and answer policy-relevant questions by varying the information set available to respondents. We survey the emerging literature using information provision experiments in economics and discuss applications in macroeconomics, finance, political economy, public economics, labor economics, and health economics. We also discuss design considerations and provide best-practice recommendations on how to (i) measure beliefs; (ii) design the information intervention; (iii) measure belief updating; (iv) deal with potential confounds, such as experimenter demand effects; and (v) recruit respondents using online panels. We finally discuss typical effect sizes and provide sample size recommendations.
Why I Slack
Earlier this year, I ran a survey to get ideas about how leaders could mobilize. 1311 of you filled out the survey which mean I’m certain the results are full o
The thing it turns out lots of us want
Last month I posted a questionnaire. Here’s what you said
How WEIRD is CHI?
Computer technology is often designed in technology hubs in Western countries, invariably making it “WEIRD”, because it is based on the intuition, knowledge, and values of people who are Western, Educated, Industrialized, Rich, and Democratic. Developing technology that is universally useful and engaging requires knowledge about members of WEIRD and non-WEIRD societies alike. In other words, it requires us, the CHI community, to generate this knowledge by studying representative participant samples. To find out to what extent CHI participant samples are from Western societies, we analyzed papers published in the CHI proceedings between 2016-2020. Our findings show that 73% of CHI study findings are based on Western participant samples, representing less than 12% of the world’s population. Furthermore, we show that most participant samples at CHI tend to come from industrialized, rich, and democratic countries with generally highly educated populations. Encouragingly, recent years have seen a slight increase in non-Western samples and those that include several countries. We discuss suggestions for further broadening the international representation of CHI participant samples.
A Theory of Response Sampling in LLMs: Part Descriptive and Part Prescriptive
Large Language Models (LLMs) are increasingly utilized in autonomous decision-making, where they sample options from vast action spaces. However, the heuristics that guide this sampling process remain under-explored. We study this sampling behavior and show that this underlying heuristics resembles that of human decision-making: comprising a descriptive component (reflecting statistical norm) and a prescriptive component (implicit ideal encoded in the LLM) of a concept. We show that this deviation of a sample from the statistical norm towards a prescriptive component consistently appears in concepts across diverse real-world domains like public health, and economic trends. To further illustrate the theory, we demonstrate that concept prototypes in LLMs are affected by prescriptive norms, similar to the concept of normality in humans. Through case studies and comparison with human studies, we illustrate that in real-world applications, the shift of samples toward an ideal value in LLMs' outputs can result in significantly biased decision-making, raising ethical concerns.
Do AI and bogus respondents threaten polling’s future?
Courtney Kennedy, vice president of methods and innovation, answers some common questions about the current polling landscape in the U.S.
The Preference Survey Module: A Validated Instrument for Measuring Risk, Time, and Social Preferences
Incentivized choice experiments are a key approach to measuring preferences in economics but are also costly. Survey measures are a low-cost alternative but can suffer from additional forms of measurement error due to their hypothetical nature. This paper seeks to leverage the strengths of both approaches by proposing a new survey module on risk aversion, time discounting, trust, altruism, positive and negative reciprocity, in which survey items are selected based on ability to predict choices in corresponding, incentivized experiments. The methodology and results provided in the paper can also potentially provide a model for researchers who have specific requirements and want to design their own modules. This paper was accepted by Yan Chen, behavioral economics and decision analysis. Funding: The project received funding from the European Research Council under the European Union’s Seventh Framework Programme (FP7-2007-2013) [Grant 209214]. A. Falk and T. Dohmen acknowledge funding from the Deutsche Forschungsgemeinschaft (German Research Foundation) [Grant CRC TR 224 (Project A01)] and Germany’s Excellence Strategy [Grant EXC 2126/1-390838866]. Supplemental Material: Data and the online appendices are available at https://doi.org/10.1287/mnsc.2022.4455 .
Silicon Samples and Synthetic Surveys: Can AI Stand In for Human Respondents?
Synthetic surveys using digital twins do a poor job estimating the public mood and are especially error-prone for some subgroups, including Republicans and Black adults.
Social Sampling Explains Apparent Biases in Judgments of Social Environments
How people assess their social environments plays a central role in how they evaluate their life circumstances. Using a large probabilistic national sample, we investigated how accurately people estimate characteristics of the general population. For most characteristics, people seemed to underestimate the quality of others’ lives and showed apparent self-enhancement, but for some characteristics, they seemed to overestimate the quality of others’ lives and showed apparent self-depreciation. In addition, people who were worse off appeared to enhance their social position more than those who were better off. We demonstrated that these effects can be explained by a simple social-sampling model. According to the model, people infer how others are doing by sampling from their own immediate social environments. Interplay of these sampling processes and the specific structure of social environments leads to the apparent biases. The model predicts the empirical results better than alternative accounts and highlights the importance of considering environmental structure when studying human cognition.
OpenMeet Survey - Create and Share Surveys with ATProto
Create and share surveys with your community using the ATProto ecosystem. Free, open-source, and privacy-focused.
