Epistemic Vigilance

AI Epistemic Risks: Emerging Mechanisms & Evidence

<p>Advances in artificial intelligence pose risks to humanity's collective capacity to form accurate beliefs, reason well, and maintain a healthy information en

Warning labels shift perceptions of sycophantic AI, but not its influence

Recent work has raised concerns about the influence of sycophantic AI on user judgment and relationships. One proposed mitigation, which has received regulatory attention, is to warn users about potentially harmful AI behaviors such as sycophancy. In a preregistered experiment in which participants (N = 2,610) discussed real interpersonal conflicts with an AI system, we test whether warning labels mitigate sycophancy's influence. We find that a basic AI disclosure (``This chatbot is AI'') has no detectable effect. Labeling the system as sycophantic (``...may agree with you and validate you even when you are wrong...'') does shift users' perceptions, reducing perceived objectivity and trust, but it does not reliably reduce sycophancy's influence on users' self-perceived rightness or their willingness to repair the conflict. Our results reveal a gap between AI perception and AI influence: by shifting perception without reducing influence, warning-based interventions may offer a false sense of protection. Addressing the harms of sycophancy will therefore require understanding the specific mechanisms through which it shapes judgment, and improving model behavior itself.

Not Learning from Others

We study social learning using experiments where two people independently learn relevant information and can share it to make accurate private decisions. Across three experiments, people are substantially less sensitive to information others discover than to equally-relevant information they discovered themselves. This holds when they must learn information from others through discussion; when the experimenter perfectly communicates the information; and even when participants observe others’ information with their own eyes. Our results therefore stem not from a failure to elicit information from others but a systematic tendency to underweight it relative to one’s own information. Our findings illustrate a powerful barrier to social learning that might underlie many documented cases of failure to learn from others.

Talking with strangers is surprisingly informative

A meaningful amount of people’s knowledge comes from their conversations with others. The amount people expect to learn predicts their interest in having a conversation (pretests 1 and 2), suggesting that the presumed information value of conversations guides decisions of whom to talk with. The results of seven experiments, however, suggest that people may systematically underestimate the informational benefit of conversation, creating a barrier to talking with—and hence learning from—others in daily life. Participants who were asked to talk with another person expected to learn significantly less from the conversation than they actually reported learning afterward, regardless of whether they had conversation prompts and whether they had the goal to learn (experiments 1 and 2). Undervaluing conversation does not stem from having systematically poor opinions of how much others know (experiment 3) but is instead related to the inherent uncertainty involved in conversation itself. Consequently, people underestimate learning to a lesser extent when uncertainty is reduced, as in a nonsocial context (surfing the web, experiment 4); when talking to an acquainted conversation partner (experiment 5); and after knowing the content of the conversation (experiment 6). Underestimating learning in conversation is distinct from underestimating other positive qualities in conversation, such as enjoyment (experiment 7). Misunderstanding how much can be learned in conversation could keep people from learning from others in daily life.

AI and the Wisdom of Uncertainty

AI chatbots rarely say "I don't know," and neither, increasingly, do we. But we can cultivate our epistemic resilience.

Honest Communication Kills Weaponized Incompetence

It’s only human to need help — but we have to be able to admit it.

Individual-level interventions against sycophantic AI reduce its appeal but not its persuasiveness

AI chatbots can be "sycophantic," or overly agreeable and flattering toward users. Sycophantic AI has been shown to entrench attitudes, yet users frequently fail to recognize it (a phenomenon we call "sycophancy blindness"). We tested whether increasing users' awareness of sycophancy protects them from its harmful effects in two preregistered experiments (n = 1,590). In the first, participants received a brief written warning about sycophancy before conversing with a sycophantic chatbot. In the second, participants watched a video of a sycophantic AI validating several other users, including users on opposite sides of the same conflict, before interacting with it themselves. Both interventions changed how participants evaluated the AI. The warning reduced the AI's perceived objectivity, and the video reduced enjoyment of the AI --- an effect mediated by the reduced belief that its validation was uniquely earned. We then pooled our experiments with two prior studies of sycophancy awareness interventions (six interventions total, n = 3,982). The pattern across experiments was consistent: while the interventions made the sycophantic AI appear less objective and trustworthy, none reduced its persuasiveness. These results suggest that individual-level interventions, such as warning labels or AI literacy, may not be enough to protect users from AI harms.

Moderating With Humans, For Humans - Trust Issues

Moderating With Humans, For Humans - Trust Issues

Undersociality: miscalibrated social cognition can inhibit social connection

A person’s well-being depends heavily on forming and maintaining positive relationships, but people can be reluctant to connect in ways that would create or strengthen relationships. Emerging research suggests that miscalibrated social cognition may create psychological barriers to connecting with others more often. Specifically, people may underestimate how positively others will respond to their own sociality across a variety of social actions, including engaging in conversation, expressing appreciation, and performing acts of kindness.

ECHO CHAMBERS AND EPISTEMIC BUBBLES

Discussion of the phenomena of post-truth and fake news often implicates the closed epistemic networks of social media. The recent conversation has, however, blurred two distinct social epistemic phenomena. An epistemic bubble is a social epistemic structure in which other relevant voices have been left out, perhaps accidentally. An echo chamber is a social epistemic structure from which other relevant voices have been actively excluded and discredited. Members of epistemic bubbles lack exposure to relevant information and arguments. Members of echo chambers, on the other hand, have been brought to systematically distrust all outside sources. In epistemic bubbles, other voices are not heard; in echo chambers, other voices are actively undermined. It is crucial to keep these phenomena distinct. First, echo chambers can explain the post-truth phenomena in a way that epistemic bubbles cannot. Second, each type of structure requires a distinct intervention. Mere exposure to evidence can shatter an epistemic bubble, but may actually reinforce an echo chamber. Finally, echo chambers are much harder to escape. Once in their grip, an agent may act with epistemic virtue, but social context will pervert those actions. Escape from an echo chamber may require a radical rebooting of one's belief system.

Misplaced Divides? Discussing Political Disagreement With Strangers Can Be Unexpectedly Positive

Differences of opinion between people are common in everyday life, but discussing those differences openly in conversation may be unnecessarily rare. We report three experiments ( N = 1,264 U.S.-based adults) demonstrating that people’s interest in discussing important but potentially divisive topics is guided by their expectations about how positively the conversation will unfold, leaving them more interested in having a conversation with someone who agrees versus disagrees with them. People’s expectations about their conversations, however, were systematically miscalibrated such that people underestimated how positive these conversations would be—especially in cases of disagreement. Miscalibrated expectations stemmed from underestimating the degree of common ground that would emerge in conversation and from failing to appreciate the power of social forces in conversation that create social connection. Misunderstanding the outcomes of conversation could lead people to avoid discussing disagreements more often, creating a misplaced barrier to learning, social connection, free inquiry, and free expression.

Act or Clarify? Modeling Sensitivity to Uncertainty and Cost in Communication

When deciding how to act under uncertainty, agents may choose to act to reduce uncertainty or they may act despite that uncertainty. In communicative settings, an important way of reducing uncertainty is by asking clarification questions (CQs). We predict that the decision to ask a CQ depends on both contextual uncertainty and the cost of alternative actions, and that these factors interact: uncertainty should matter most when acting incorrectly is costly. We formalize this interaction in a computational model based on expected regret: how much an agent stands to lose by acting now rather than with full information. We test these predictions in two experiments, one examining purely linguistic responses to questions and another extending to choices between clarification and non-linguistic action. Taken together, our results suggest a rational tradeoff: humans tend to seek clarification proportional to the risk of substantial loss when acting under uncertainty.

A case for AI taboos – Writings and rehearsals by Nathan Schneider

We probably all know the person, and maybe it is ourselves, who has inundated themselves in AI use, and say they are fully aware of all the dangers, but are so, so careful as to not fall into such traps themselves.