Significance Algorithmic monoculture is a growing concern in the use of algorithms for high-stakes screening decisions in areas such as employment and lending. If many firms use the same algorithm, even if it is more accurate than the alternatives, the resulting “monoculture” may be susceptible to correlated failures, much as a monocultural system is in biological settings. To investigate this concern, we develop a model of selection under monoculture. We find that even without any assumption of shocks or correlated failures—i.e., under “normal operations”—the quality of decisions may decrease when multiple firms use the same algorithm. Thus, the introduction of a more accurate algorithm may decrease social welfare—a kind of “Braess’ paradox” for algorithmic decision-making. , As algorithms are increasingly applied to screen applicants for high-stakes decisions in employment, lending, and other domains, concerns have been raised about the effects of algorithmic monoculture, in which many decision-makers all rely on the same algorithm. This concern invokes analogies to agriculture, where a monocultural system runs the risk of severe harm from unexpected shocks. Here, we show that the dangers of algorithmic monoculture run much deeper, in that monocultural convergence on a single algorithm by a group of decision-making agents, even when the algorithm is more accurate for any one agent in isolation, can reduce the overall quality of the decisions being made by the full collection of agents. Unexpected shocks are therefore not needed to expose the risks of monoculture; it can hurt accuracy even under “normal” operations and even for algorithms that are more accurate when used by only a single decision-maker. Our results rely on minimal assumptions and involve the development of a probabilistic framework for analyzing systems that use multiple noisy estimates of a set of alternatives.
Algorithmic monoculture and social welfare
Significance Algorithmic monoculture is a growing concern in the use of algorithms for high-stakes screening decisions in areas such as employment and lending. If many firms use the same algorithm, even if it is more accurate than the alternatives, the resulting “monoculture” may be susceptible to correlated failures, much as a monocultural system is in biological settings. To investigate this concern, we develop a model of selection under monoculture. We find that even without any assumption of shocks or correlated failures—i.e., under “normal operations”—the quality of decisions may decrease when multiple firms use the same algorithm. Thus, the introduction of a more accurate algorithm may decrease social welfare—a kind of “Braess’ paradox” for algorithmic decision-making. , As algorithms are increasingly applied to screen applicants for high-stakes decisions in employment, lending, and other domains, concerns have been raised about the effects of algorithmic monoculture, in which many decision-makers all rely on the same algorithm. This concern invokes analogies to agriculture, where a monocultural system runs the risk of severe harm from unexpected shocks. Here, we show that the dangers of algorithmic monoculture run much deeper, in that monocultural convergence on a single algorithm by a group of decision-making agents, even when the algorithm is more accurate for any one agent in isolation, can reduce the overall quality of the decisions being made by the full collection of agents. Unexpected shocks are therefore not needed to expose the risks of monoculture; it can hurt accuracy even under “normal” operations and even for algorithms that are more accurate when used by only a single decision-maker. Our results rely on minimal assumptions and involve the development of a probabilistic framework for analyzing systems that use multiple noisy estimates of a set of alternatives.
Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
As the scope of machine learning broadens, we observe a recurring theme of algorithmic monoculture: the same systems, or systems that share components (e.g. datasets, models), are deployed by multiple decision-makers. While sharing offers advantages like amortizing effort, it also has risks. We introduce and formalize one such risk, outcome homogenization: the extent to which particular individuals or groups experience the same outcomes across different deployments. If the same individuals or groups exclusively experience undesirable outcomes, this may institutionalize systemic exclusion and reinscribe social hierarchy. We relate algorithmic monoculture and outcome homogenization by proposing the component sharing hypothesis: if algorithmic systems are increasingly built on the same data or models, then they will increasingly homogenize outcomes. We test this hypothesis on algorithmic fairness benchmarks, demonstrating that increased data-sharing reliably exacerbates homogenization and individual-level effects generally exceed group-level effects. Further, given the current regime in AI of foundation models, i.e. pretrained models that can be adapted to myriad downstream tasks, we test whether model-sharing homogenizes outcomes across tasks. We observe mixed results: we find that for both vision and language settings, the specific methods for adapting a foundation model significantly influence the degree of outcome homogenization. We also identify societal challenges that inhibit the measurement, diagnosis, and rectification of outcome homogenization in deployed machine learning systems.
Algorithmic Monocultures in Hiring
Many employers procure hiring algorithms from the same third-party vendors. Over 60% of the Fortune 100 use HireVue's algorithms. When hiring algorithms from a single vendor mediate hiring decisions at multiple employers, they constitute an algorithmic monoculture.
Algorithmic Monocultures in Hiring - Stanford Digital Economy Lab
Many employers screen job applicants with algorithms built by the same few algorithm vendors. We hypothesize that algorithmic monoculture leads to the same individuals and members of the same racial groups facing rejection. We acquire and analyze a novel dataset of 3 million applicants submitting 4 million applications where all the applications are screened by algorithms built by the same vendor.
monoculture
Monoculture is an agricultural term referring to the practice of growing only one species in a field at a time, but the term has also been adopted to computing. Identical computers that run identical software are an example of an algorithmic monoculture. Monocultures tend to be vulnerable because all their elements (be they plants or computers) share the same vulnerabilities and other weaknesses.
Algorithmic Bias in Lending: Evidence from a Fintech Audit
Algorithmic lending has transformed the consumer credit landscape, with machine learning models commonly facilitating underwriting decisions. To comply with fair lending laws, these algorithms exclude legally protected characteristics, such as race and gender. Yet algorithmic underwriting can still inadvertently favor certain groups, prompting concerns about whether lending algorithms exhibit discriminatory behavior. Using proprietary loan-level data from a major U.S. fintech platform, we audit lending decisions across approximately 80,000 personal loans. We find that loans made to men and Black borrowers yielded lower profits than loans to other groups, suggesting that men and Black borrowers benefited from relatively favorable pricing. We trace these disparities to miscalibration in the platform's underwriting model, which overestimates risk for women and underestimates risk for Black borrowers. We then show that one could correct this miscalibration -- and the corresponding disparities -- by including race and gender in underwriting models, illustrating a tension between competing notions of fairness.
People Reject Algorithms in Uncertain Decision Domains Because They Have Diminishing Sensitivity to Forecasting Error
Will people use self-driving cars, virtual doctors, and other algorithmic decision-makers if they outperform humans? The answer depends on the uncertainty inherent in the decision domain. We propose that people have diminishing sensitivity to forecasting error and that this preference results in people favoring riskier (and often worse-performing) decision-making methods, such as human judgment, in inherently uncertain domains. In nine studies ( N = 4,820), we found that (a) people have diminishing sensitivity to each marginal unit of error that a forecast produces, (b) people are less likely to use the best possible algorithm in decision domains that are more unpredictable, (c) people choose between decision-making methods on the basis of the perceived likelihood of those methods producing a near-perfect answer, and (d) people prefer methods that exhibit higher variance in performance (all else being equal). To the extent that investing, medical decision-making, and other domains are inherently uncertain, people may be unwilling to use even the best possible algorithm in those domains.
Putting nudges in perspective
Conventional economic policy focuses on ‘economic’ solutions (e.g. taxes, incentives, regulation) to problems caused by market-level factors such as externalities, misaligned incentives and information asymmetries. By contrast, ‘nudges’ provide behavioural solutions to problems that have generally been assumed to originate from limitations in human decision making, such as present bias. While policy-makers have good reason for exploiting the power of nudges, we argue that these extremes leave open a large space of policy options that have received less attention in the academic literature. First, there is no reason that solution and problem need have the same theoretical basis: there are promising behavioural solutions to problems that have causes that are well explained by traditional economics, and conventional economic solutions often offer the best line of attack on problems of behavioural origin. Second, there is a wide range of hybrid policy actions with both economic and behavioural components (e.g. framing a tax or incentive in a specific way), and there exist many societal problems – perhaps the majority – that arise from both economic and behavioural factors (e.g. firms’ exploitation of consumers’ behavioural biases). This paper aims to remind policy-makers that behavioural economics can influence policy in a variety of ways, of which nudges are the most prominent but not necessarily the most powerful.
The Unaccountability Machine: Why Big Systems Make Terrible Decisions—and How the World Lost Its Mind
Longlisted for the 2024 Financial Times Book of the Year. How life and the economy became a black box—a collection of systems no one understands, producing outcomes no one likes. Passengers get bumped from flights. Phone menus disconnect. Automated financial trades produce market collapse. Of all the challenges in modern life, some of the most vexing come from our relationships with automation: a large system does us wrong, and there’s nothing we can do about it. The problem, economist Dan Davies shows, is accountability sinks: systems in which decisions are delegated to a complex rule book or set of standard procedures, making it impossible to identify the source of mistakes when they happen. In our increasingly unhuman world—lives dominated by algorithms, artificial intelligence, and large organizations—these accountability sinks produce more than just aggravation. They make life and economy unknowable—a black box for no reason. In The Unaccountability Machine, Davies lays bare how markets, institutions, and even governments systematically generate outcomes that no one—not even those involved in making them—seems to want. Since the earliest days of the computer age, theorists have foreseen the dangers of complex systems without personal accountability. In response, British business scholar Stafford Beer developed an accountability-first approach to management called “cybernetics,” which might have taken off had his biggest client (the Chilean government) not fallen to a bloody coup in 1973. With his signature blend of economic and journalistic rigor, Davies examines what’s gone wrong since Beer, including what might have been had the world embraced cybernetics when it had the chance. The Unaccountability Machine is a revelatory and resonant account of how modern life became predisposed to dysfunction.
AI Has Ruined the Job Market
Maybe flawed people were better than brute algorithms.
Individually Sensible, Collectively Harmful: Simulating Commons Failure in LLM Societies
Large language models are increasingly deployed as agents in simulations of societies, markets, and organizations, raising a fundamental question: can individually sensible AI agents nevertheless produce collectively harmful outcomes? We study this question in a repeated commons dilemma with externalities. We introduce Creeping Trap, a controlled social simulation in which LLM agents repeatedly choose private extraction levels for immediate profit while jointly accumulating catastrophe risk and imposing harm on silent stakeholders. The environment includes analytically derived welfare and behavioral reference strategies, allowing individually sensible behavior and collective welfare to be evaluated simultaneously. Across nine contemporary LLMs and 990 episodes, including a 400-episode confirmatory study, simulated societies consistently over-extract; in the confirmatory study, 396 of 400 episodes produce negative aggregate welfare. This failure is not arbitrary irrationality: observed behavior lies near a broad low-regret region of local best response that even simple heuristics attain, indicating an institutional failure generated by individually sensible decisions. We further show that institutional design changes collective outcomes: surfacing stakeholder harm consistently reduces extraction, whereas shortening accountability horizons increases extraction and induces end-of-term defection. Finally, prompt wording substantially changes measured behavior, making prompt formulation a first-order experimental variable. Creeping Trap provides a controlled testbed for studying institutional interventions that improve collective outcomes in AI societies.
Choice Bracketing
When making many choices, a person can broadly bracket them by assessing the consequences of all of them taken together, or narrowly bracket them by making each choice in isolation. We integrate research conducted in a wide range of decision contexts which shows that choice bracketing is an important determinant of behavior. Because broad bracketing allows people to take into account all the consequences of their actions, it generally leads to choices that yield higher utility. The evidence that we review, however, shows that people often fail to bracket broadly when it would be feasible for them to do so. In addition to documenting the diverse effects of bracketing, we also discuss factors that determine whether people bracket narrowly or broadly. We conclude with a discussion of normative aspects of bracketing and argue that there are some situations in which narrower bracketing results in superior decision making.
Underspecified Human Decision Experiments Considered Harmful
Decision-making with information displays is a key focus of research in areas like human-AI collaboration and data visualization. However, what constitutes a decision problem, and what is required for an experiment to conclude that decisions are flawed, remain imprecise. We present a widely applicable definition of a decision problem synthesized from statistical decision theory and information economics. We claim that to attribute loss in human performance to bias, an experiment must provide the information that a rational agent would need to identify the normative decision. We evaluate whether recent empirical research on AI-assisted decisions achieves this standard. We find that only 10 (26%) of 39 studies that claim to identify biased behavior presented participants with sufficient information to make this claim in at least one treatment condition. We motivate the value of studying well-defined decision problems by describing a characterization of performance losses they allow to be conceived.
Characterizing Fairness Over the Set of Good Models Under Selective Labels
Algorithmic risk assessments are used to inform decisions in a wide variety of high-stakes settings. Often multiple predictive models deliver similar overall performance but differ markedly in their predictions for individual cases, an empirical phenomenon known as the “Rashomon Effect.” These models may have different properties over various groups, and therefore have different predictive fairness properties. We develop a framework for characterizing predictive fairness properties over the set of models that deliver similar overall performance, or “the set of good models.” Our framework addresses the empirically relevant challenge of selectively labelled data in the setting where the selection decision and outcome are unconfounded given the observed data features. Our framework can be used to 1) audit for predictive bias; or 2) replace an existing model with one that has better fairness properties. We illustrate these use cases on a recidivism prediction task and a real-world credit-scoring task.
Taking AI Welfare Seriously
In this report, we argue that there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future. That means that the prospect of AI welfare and moral patienthood, i.e. of AI systems with their own interests and moral significance, is no longer an issue only for sci-fi or the distant future. It is an issue for the near future, and AI companies and other actors have a responsibility to start taking it seriously. We also recommend three early steps that AI companies and other actors can take: They can (1) acknowledge that AI welfare is an important and difficult issue (and ensure that language model outputs do the same), (2) start assessing AI systems for evidence of consciousness and robust agency, and (3) prepare policies and procedures for treating AI systems with an appropriate level of moral concern. To be clear, our argument in this report is not that AI systems definitely are, or will be, conscious, robustly agentic, or otherwise morally significant. Instead, our argument is that there is substantial uncertainty about these possibilities, and so we need to improve our understanding of AI welfare and our ability to make wise decisions about this issue. Otherwise there is a significant risk that we will mishandle decisions about AI welfare, mistakenly harming AI systems that matter morally and/or mistakenly caring for AI systems that do not.