Good Luck Opting Out: Manipulative Design Patterns in Opt-Out Processes

Settings are not a design failure

The systematic thinking in our industry is that settings are the result of design failure. As designers, our goal is to create product experiences that don’t require any adjustment by the user. So offering customization options is often seen as a failure to make firm product decisions. I think there is a misunderstanding about what settings really are.

Task-Dependent Algorithm Aversion

Research suggests that consumers are averse to relying on algorithms to perform tasks that are typically done by humans, despite the fact that algorithms often perform better. The authors explore when and why this is true in a wide variety of domains. They find that algorithms are trusted and relied on less for tasks that seem subjective (vs. objective) in nature. However, they show that perceived task objectivity is malleable and that increasing a task’s perceived objectivity increases trust in and use of algorithms for that task. Consumers mistakenly believe that algorithms lack the abilities required to perform subjective tasks. Increasing algorithms’ perceived affective human-likeness is therefore effective at increasing the use of algorithms for subjective tasks. These findings are supported by the results of four online lab studies with over 1,400 participants and two online field studies with over 56,000 participants. The results provide insights into when and why consumers are likely to use algorithms and how marketers can increase their use when they outperform humans.

Task-Dependent Algorithm Aversion

Research suggests that consumers are averse to relying on algorithms to perform tasks that are typically done by humans, despite the fact that algorithms often perform better. The authors explore when and why this is true in a wide variety of domains. They find that algorithms are trusted and relied on less for tasks that seem subjective (vs. objective) in nature. However, they show that perceived task objectivity is malleable and that increasing a task’s perceived objectivity increases trust in and use of algorithms for that task. Consumers mistakenly believe that algorithms lack the abilities required to perform subjective tasks. Increasing algorithms’ perceived affective human-likeness is therefore effective at increasing the use of algorithms for subjective tasks. These findings are supported by the results of four online lab studies with over 1,400 participants and two online field studies with over 56,000 participants. The results provide insights into when and why consumers are likely to use algorithms and how marketers can increase their use when they outperform humans.

Data Brokers’ and AI Firms’ Opt-Out Forms Are Built to Fail, Report Finds

A new study finds AI companies, defense firms, and dating apps are among 38 data collectors allegedly using manipulative design to confuse users while collecting their data.

The Challenge of Understanding What Users Want: Inconsistent Preferences and Engagement Optimization

Online platforms have a wealth of data, run countless experiments, and use industrial-scale algorithms to optimize user experience. Despite this, many users seem to regret the time they spend on these platforms. One possible explanation is that incentives are misaligned: platforms are not optimizing for user happiness. We suggest the problem runs deeper, transcending the specific incentives of any particular platform, and instead stems from a mistaken foundational assumption. To understand what users want, platforms look at what users do. This is a kind of revealed-preference assumption that is ubiquitous in the way user models are built. Yet research has demonstrated, and personal experience affirms, that we often make choices in the moment that are inconsistent with what we actually want. The behavioral economics and psychology literatures suggest, for example, that we can choose mindlessly or that we can be too myopic in our choices, behaviors that feel entirely familiar on online platforms. In this work, we develop a model of media consumption where users have inconsistent preferences. We consider a platform which wants to maximize user utility, but only observes behavioral data in the form of the user’s engagement. We show how our model of users’ preference inconsistencies produces phenomena that are familiar from everyday experience but difficult to capture in traditional user interaction models. These phenomena include users who have long sessions on a platform but derive very little utility from it, and platform changes that steadily raise user engagement before abruptly causing users to go “cold turkey” and quit. A key ingredient in our model is a formulation for how platforms determine what to show users: they optimize over a large set of potential content (the content manifold) parametrized by underlying features of the content. Whether improving engagement improves user welfare depends on the direction of movement in the content manifold: For certain directions of change, increasing engagement makes users less happy, whereas in other directions on the same manifold, increasing engagement makes users happier. We provide a characterization of the structure of content manifolds for which increasing engagement fails to increase user utility. By linking these effects to abstractions of platform design choices, our model thus creates a theoretical framework and vocabulary in which to explore interactions between design, behavioral science, and social media. This paper was accepted by Yan Chen, behavioral economics and decision analysis. Funding: This work was supported by the Vannevar Bush Faculty Fellowship and Multidisciplinary University Research Initiative [Grant W911NF-19-0217]. Supplemental Material: The online appendices are available at https://doi.org/10.1287/mnsc.2022.03683 .

Study Finds Consumers Are Actively Turned Off by Products That Use AI

Researchers have found that including the words "artificial intelligence" in product marketing is a major turn-off for consumers.

Please Stop Making Me Opt Out of AI

I’m sick of “opt-out” toggles for automatically enabled generative AI features. It’s past time to make “opt in” the default setting for sensitive features.

Design choices: Mechanism design and platform capitalism

Mechanism design is a form of optimization developed in economic theory. It casts economists as institutional engineers, choosing an outcome and then arranging a set of market rules and conditions to achieve it. The toolkit from mechanism design is widely used in economics, policymaking, and now in building and managing online environments. Mechanism design has become one of the most pervasive yet inconspicuous influences on the digital mediation of social life. Its optimizing schemes structure online advertising markets and other multi-sided platform businesses. Whatever normative rationales mechanism design might draw on in its economic origins, as its influence has grown and its applications have become more computational, we suggest those justifications for using mechanism design to orchestrate and optimize human interaction are losing traction. In this article, we ask what ideological work mechanism design is doing in economics, computer science, and its applications to the governance of digital platforms. Observing mechanism design in action in algorithmic environments, we argue it has become a tool for producing information domination, distributing social costs in ways that benefit designers, and controlling and coordinating participants in multi-sided platforms.

Read the Principles — Design Justice Network

Design mediates so much of our realities and has tremendous impact on our lives, yet very few of us participate in design processes. In particular, the people who are most adversely affected by design decisions — about visual culture, new technologies, the planning of our communities, or the structure of our political and economic systems — tend to have the least influence on those decisions and how they are made.

Six ‘dark patterns’ used to manipulate you when shopping online

Online shopping offers unparalleled convenience, but also poses significant risks due to manipulative design practices known as ‘dark patterns’. These tactics subtly influence consumer behaviour, often leading to unintended purchases or compromising consumer privacy. Here are six dark patterns to watch out for next time you are shopping online.

Making the Case for Actionable Prosocial Design Research

Regulating harmful design is only half the equation — attention must also go toward how platforms can build prosocial design, and toward the research that shows them how.

Design From the Margins

In an age of virtual connectivity and increased reliance on the internet for daily functions, including by marginalized groups, can companies and technologists reframe their features or standards to support the most marginalized users’ needs? Can the modes of resilience within digital spaces from some of the most marginalized groups be listened to, learned from, and centered when creating technology? Design from the Margins (DFM), a design process that centers the most impacted and marginalized users from ideation to production, pushes the notion that not only is this something that can and must be done, but also that it is highly beneficial for all users and companies. For this to happen, consumer interest conversations need to be framed outside the “biggest use case” scenarios and United States and European Union-centrisms and refocused on the cases often left in the margins: the decentered cases. This report outlines how the DFM method can be used to build our most well-known and relied-upon technologies for decentered cases (often deemed “edge cases” which is atypical or less common use case for a product) from the beginning of the design process, rather than retrofitting them post-deployment to cater to communities with what are perceived to be extra needs.

Time-inconsistent Preferences and Consumer Self-Control

Abstract Why do consumers sometimes act against their own better judgment, engaging in behavior that is often regretted after the fact and that would have been rejected with adequate forethought? More generally, how do consumers attempt to maintain self-control in the face of time-inconsistent preferences? This article addresses consumer impatience by developing a decision-theoretic model based on reference points. The model explains how and why consumers experience sudden increases in desire for a product, increases that can result in the temporary overriding of long-term preferences. Tactics that consumers use to control their own behavior are also discussed. Consumer self-control is framed as a struggle between two psychological forces, desire and willpower. Finally, two general classes of self-control strategies are described: those that directly reduce desire, and those that overcome desire through will power.

About Browser Choice Allliance

The issue Microsoft’s deceptive tacticsimpede consumer choiceFor years, Microsoft has manipulated desktop browser choice, using deceptive tactics to push Edge onto users. Instead of competing on quality, it relies on forced resets, misleading prompts, and hidden settings to trap people into using its browser. Every time Microsoft forces, pushes or…

How Artificial Intelligence Constrains the Human Experience

AbstractArtificial intelligence (AI) and related technologies are transforming many consumption activities, powering breakthroughs that expand the human experience by enhancing human capabilities, performance, and creativity. While this explains the consumer enthusiasm and rapid adoption of these technologies, AI systems can also have the opposite effect: reducing and constraining the range of experiences that are available to consumers. This article examines the mechanisms through which AI can constrain the human experience, considering individual, interpersonal, and societal processes. Our analysis uncovers a complex interplay between the advantages of AI and its inadvertent negative repercussions, which potentially restrict human autonomy, self-identity, relational dynamics, and social behavior. In this article, we propose three different mechanisms at the core of these constraining forces: parametric reductionism, agency transference, and regulated expression. Our exploration of these mechanisms highlights the risks connected to system design and points to questions and implications for future researchers and policymakers.