Epistemic Vigilance

AI Epistemic Risks: Emerging Mechanisms & Evidence

<p>Advances in artificial intelligence pose risks to humanity's collective capacity to form accurate beliefs, reason well, and maintain a healthy information en

Warning labels shift perceptions of sycophantic AI, but not its influence

Recent work has raised concerns about the influence of sycophantic AI on user judgment and relationships. One proposed mitigation, which has received regulatory attention, is to warn users about potentially harmful AI behaviors such as sycophancy. In a preregistered experiment in which participants (N = 2,610) discussed real interpersonal conflicts with an AI system, we test whether warning labels mitigate sycophancy's influence. We find that a basic AI disclosure (``This chatbot is AI'') has no detectable effect. Labeling the system as sycophantic (``...may agree with you and validate you even when you are wrong...'') does shift users' perceptions, reducing perceived objectivity and trust, but it does not reliably reduce sycophancy's influence on users' self-perceived rightness or their willingness to repair the conflict. Our results reveal a gap between AI perception and AI influence: by shifting perception without reducing influence, warning-based interventions may offer a false sense of protection. Addressing the harms of sycophancy will therefore require understanding the specific mechanisms through which it shapes judgment, and improving model behavior itself.

Explicit warnings reduce but do not eliminate the continued influence of misinformation

Information that initially is presumed to be correct, but that is later retracted or corrected, often continues to influence memory and reasoning. This occurs even if the retraction itself is well remembered. The present study investigated whether the continued influence of misinformation can be reduced by explicitly warning people at the outset that they may be misled. A specific warning— giving detailed information about the continued influence effect (CIE)—succeeded in reducing the continued reliance on outdated information but did not eliminate it. A more general warning—reminding people that facts are not always properly checked before information is disseminated—was even less effective. In an additional experiment, a specific warning was combined with the provision of a plausible alternative explanation for the retracted information. This combined manipulation further reduced the CIE but still failed to eliminate it altogether.

Not Learning from Others

We study social learning using experiments where two people independently learn relevant information and can share it to make accurate private decisions. Across three experiments, people are substantially less sensitive to information others discover than to equally-relevant information they discovered themselves. This holds when they must learn information from others through discussion; when the experimenter perfectly communicates the information; and even when participants observe others’ information with their own eyes. Our results therefore stem not from a failure to elicit information from others but a systematic tendency to underweight it relative to one’s own information. Our findings illustrate a powerful barrier to social learning that might underlie many documented cases of failure to learn from others.

Talking with strangers is surprisingly informative

A meaningful amount of people’s knowledge comes from their conversations with others. The amount people expect to learn predicts their interest in having a conversation (pretests 1 and 2), suggesting that the presumed information value of conversations guides decisions of whom to talk with. The results of seven experiments, however, suggest that people may systematically underestimate the informational benefit of conversation, creating a barrier to talking with—and hence learning from—others in daily life. Participants who were asked to talk with another person expected to learn significantly less from the conversation than they actually reported learning afterward, regardless of whether they had conversation prompts and whether they had the goal to learn (experiments 1 and 2). Undervaluing conversation does not stem from having systematically poor opinions of how much others know (experiment 3) but is instead related to the inherent uncertainty involved in conversation itself. Consequently, people underestimate learning to a lesser extent when uncertainty is reduced, as in a nonsocial context (surfing the web, experiment 4); when talking to an acquainted conversation partner (experiment 5); and after knowing the content of the conversation (experiment 6). Underestimating learning in conversation is distinct from underestimating other positive qualities in conversation, such as enjoyment (experiment 7). Misunderstanding how much can be learned in conversation could keep people from learning from others in daily life.

AI and the Wisdom of Uncertainty

AI chatbots rarely say "I don't know," and neither, increasingly, do we. But we can cultivate our epistemic resilience.

Rethinking Trust

Reprint: R0906H Will we ever learn? We’d barely recovered from Enron and WorldCom before we faced the subprime mortgage meltdown and more scandals that shook our trust in businesspeople. Which raises the question: Do we trust too much? In this article, Stanford professor and social psychologist Kramer explores the reasons we trust so easily—and, often, so unwisely. He explains that genetics and childhood learning make us predisposed to trust and that it’s been a good survival mechanism. That said, our willingness to trust makes us vulnerable. Our sense of trust kicks in on remarkably simple cues, such as when people look like us or are part of our social group. We also rely on third parties to verify the character of others, sometimes to our detriment (as the victims of Bernard Madoff learned). Add in our illusions of invulnerability and our tendencies to see what we want to see and to overestimate our own judgment, and the bottom line is that we’re often easily fooled. We need to develop tempered trust. For those who trust too much, that means reading cues better; for the distrustful, it means developing more receptive behaviors. Everyone should start with small acts of trust that encourage reciprocity and build up. Having a hedge against potential abuses also helps. Hollywood scriptwriters, for instance, register their treatments with the Writers Guild of America to prevent their ideas from being stolen by the executives they pitch. To attract the right relationships, people must strongly signal their own honesty, proactively allay concerns, and, if their trust is abused, retaliate. Trusting individuals in certain roles, which essentially means trusting the system that selects and trains them, also works but isn’t foolproof. And don’t count on due diligence alone for protection; constant vigilance is needed to make sure the landscape hasn’t changed.

Honest Communication Kills Weaponized Incompetence

It’s only human to need help — but we have to be able to admit it.

Individual-level interventions against sycophantic AI reduce its appeal but not its persuasiveness

AI chatbots can be "sycophantic," or overly agreeable and flattering toward users. Sycophantic AI has been shown to entrench attitudes, yet users frequently fail to recognize it (a phenomenon we call "sycophancy blindness"). We tested whether increasing users' awareness of sycophancy protects them from its harmful effects in two preregistered experiments (n = 1,590). In the first, participants received a brief written warning about sycophancy before conversing with a sycophantic chatbot. In the second, participants watched a video of a sycophantic AI validating several other users, including users on opposite sides of the same conflict, before interacting with it themselves. Both interventions changed how participants evaluated the AI. The warning reduced the AI's perceived objectivity, and the video reduced enjoyment of the AI --- an effect mediated by the reduced belief that its validation was uniquely earned. We then pooled our experiments with two prior studies of sycophancy awareness interventions (six interventions total, n = 3,982). The pattern across experiments was consistent: while the interventions made the sycophantic AI appear less objective and trustworthy, none reduced its persuasiveness. These results suggest that individual-level interventions, such as warning labels or AI literacy, may not be enough to protect users from AI harms.

Moderating With Humans, For Humans - Trust Issues

Moderating With Humans, For Humans - Trust Issues

Undersociality: miscalibrated social cognition can inhibit social connection

A person’s well-being depends heavily on forming and maintaining positive relationships, but people can be reluctant to connect in ways that would create or strengthen relationships. Emerging research suggests that miscalibrated social cognition may create psychological barriers to connecting with others more often. Specifically, people may underestimate how positively others will respond to their own sociality across a variety of social actions, including engaging in conversation, expressing appreciation, and performing acts of kindness.

ECHO CHAMBERS AND EPISTEMIC BUBBLES

Discussion of the phenomena of post-truth and fake news often implicates the closed epistemic networks of social media. The recent conversation has, however, blurred two distinct social epistemic phenomena. An epistemic bubble is a social epistemic structure in which other relevant voices have been left out, perhaps accidentally. An echo chamber is a social epistemic structure from which other relevant voices have been actively excluded and discredited. Members of epistemic bubbles lack exposure to relevant information and arguments. Members of echo chambers, on the other hand, have been brought to systematically distrust all outside sources. In epistemic bubbles, other voices are not heard; in echo chambers, other voices are actively undermined. It is crucial to keep these phenomena distinct. First, echo chambers can explain the post-truth phenomena in a way that epistemic bubbles cannot. Second, each type of structure requires a distinct intervention. Mere exposure to evidence can shatter an epistemic bubble, but may actually reinforce an echo chamber. Finally, echo chambers are much harder to escape. Once in their grip, an agent may act with epistemic virtue, but social context will pervert those actions. Escape from an echo chamber may require a radical rebooting of one's belief system.

Misplaced Divides? Discussing Political Disagreement With Strangers Can Be Unexpectedly Positive

Differences of opinion between people are common in everyday life, but discussing those differences openly in conversation may be unnecessarily rare. We report three experiments ( N = 1,264 U.S.-based adults) demonstrating that people’s interest in discussing important but potentially divisive topics is guided by their expectations about how positively the conversation will unfold, leaving them more interested in having a conversation with someone who agrees versus disagrees with them. People’s expectations about their conversations, however, were systematically miscalibrated such that people underestimated how positive these conversations would be—especially in cases of disagreement. Miscalibrated expectations stemmed from underestimating the degree of common ground that would emerge in conversation and from failing to appreciate the power of social forces in conversation that create social connection. Misunderstanding the outcomes of conversation could lead people to avoid discussing disagreements more often, creating a misplaced barrier to learning, social connection, free inquiry, and free expression.