We are in the midst of a preference cascade about existential risk from AI.
The Extinction Risk Preference Cascade: Quotes
These are quotes from OpenAI, Anthropic and Google employees, in the wake of Jacob Coxon’s warnings, in which the employees confirm that they think AI might soon kill everyone.
AI existential risk probabilities are too unreliable to inform policy
How speculation gets laundered through pseudo-quantification
AI FOR EPISTEMICS & COORDINATION
Civilization and technology have radically improved the human condition. Nonetheless, the world sometimes goes in directions which essentially nobody would prefer — e.g., nuclear arms races, unexpected financial crashes, predatory marketing, or ubiquitous political misinformation.
Jacob Coxon Warns of Human Extinction and Triggers a Preference Cascade
CEOs of major AI labs, and employees of major AI labs, including OpenAI and Anthropic, often say they plan to build superintelligence soon, as in within a few years create AIs that are superior to humans at essentially all cognitive tasks.
The AI safety vibe shift
Once a fringe obsession of Bay Area rationalists, existential risk is suddenly all anyone is talking about
AI Epistemic Risks: Emerging Mechanisms & Evidence
<p>Advances in artificial intelligence pose risks to humanity's collective capacity to form accurate beliefs, reason well, and maintain a healthy information en
Ranking is already an action - Sensemaker
Before an AI clicks, buys or writes, another system may already have decided what it sees first. That choice needs controls and evidence too.
Women treat AI with greater skepticism than men do, study suggests
Women perceive artificial intelligence (AI) as riskier than men do, according to a study. Beatrice Magistro and colleagues hypothesized that women are both more exposed to risk from AI and are more averse ...
Everybody needs a personal AI policy. Just ask Hank Green.
How can we reap AI’s benefits without melting our brains in the process?
Why women are more skeptical of AI than men, according to a new study
Drawing on survey data, the study finds that women consistently perceive AI as riskier, especially when its economic effects are uncertain.
Beyond Preferences in AI Alignment
The dominant practice of AI alignment assumes (1) that preferences are an adequate representation of human values, (2) that human rationality can be understood in terms of maximizing the satisfaction of preferences, and (3) that AI systems should be aligned with the preferences of one or more humans to ensure that they behave safely and in accordance with our values. Whether implicitly followed or explicitly endorsed, these commitments constitute what we term a preferentist approach to AI alignment. In this paper, we characterize and challenge the preferentist approach, describing conceptual and technical alternatives that are ripe for further research. We first survey the limits of rational choice theory as a descriptive model, explaining how preferences fail to capture the thick semantic content of human values, and how utility representations neglect the possible incommensurability of those values. We then critique the normativity of expected utility theory (EUT) for humans and AI, drawing upon arguments showing how rational agents need not comply with EUT, while highlighting how EUT is silent on which preferences are normatively acceptable. Finally, we argue that these limitations motivate a reframing of the targets of AI alignment: Instead of alignment with the preferences of a human user, developer, or humanity-writ-large, AI systems should be aligned with normative standards appropriate to their social roles, such as the role of a general-purpose assistant. Furthermore, these standards should be negotiated and agreed upon by all relevant stakeholders. On this alternative conception of alignment, a multiplicity of AI systems will be able to serve diverse ends, aligned with normative standards that promote mutual benefit and limit harm despite our plural and divergent values.
AI agents pose untold risk to humanity. We must act to prevent that future | David Krueger
The pieces are falling into place for autonomous artificial intelligence. We must stop unregulated development
The AI future where humans get paid to be creative
