On Mechanistic Interpretability
God Help Us, Let’s Try To Learn About Mechanistic Interpretability Techniques
...
Dario Amodei — The Urgency of Interpretability
In the decade that I have been working on AI, I’ve watched it grow from a tiny academic field to arguably the most important economic and geopolitical issue in the world. In all that time, perhaps the most important lesson I’ve learned is this: the progress of the underlying technology is inexorable, driven by forces too powerful to stop, but the way in which it happens—the order in which things are built, the applications we choose, and the details of how it is rolled out to society—are eminently possible to change, and it’s possible to have great positive impact by doing so. We can’t stop the bus, but we can steer it. In the past I’ve written about the importance of deploying AI in a way that is positive for the world, and of ensuring that democracies build and wield the technology before autocracies do. Over the last few months, I have become increasingly focused on an additional opportunity for steering the bus: the tantalizing possibility, opened up by some recent advances, that we could succeed at interpretability—that is, in understanding the inner workings of AI systems—before models reach an overwhelming level of power.
.jpg)
Tracing the thoughts of a large language model
Anthropic's latest interpretability research: a new microscope to understand Claude's internal mechanisms
Reify This
The authors contend that contemporary efforts to render AI systems interpretable rest on a mistake: reification, the process of treating abstractions and statistical artifacts as if they were concrete realities.…
A global workspace in language models
Interpretability research on Claude's internal thoughts.
Understanding Artificial Neural Networks: Mysterianism about Known Mechanism is Mysticism
Mysterianism is the idea that human cognition, mind, cannot be understood. Taking this concept and applying it to known mechanism — such that claims are made that we do not know how engineered systems, such as artificial neural networks (ANNs), work, or that they constitute black boxes that we can only open with difficulty — is inappropriate at best and malicious at worst. We do know the mechanistic structure of such models because we designed and built them. We also do know their functional role (what they are for) as well as the mathematical function they are asked to approximate (map inputs to target outputs). Because mysterianist beliefs about known systems, such as ANNs, are often expressed, scientists need to sit up and take notice. We provide an error theory as to what is going on to help unpick this metatheoretical blunder. Ultimately, the problem is that 'understanding' is not a technical term in these cases: the word is co-opted for a specific narrative to sell 'artificial intelligence' through mystification. All computational systems, from pendulums to databases, will behave in ways we cannot predict or control — this is not a unique property of ANNs — and experts do indeed grasp the computational properties of these systems nonetheless.
Understanding Artificial Neural Networks: Mysterianism about Known Mechanism is Mysticism
Mysterianism is the idea that human cognition, mind, cannot be understood. Taking this concept and applying it to known mechanism — such that claims are made that we do not know how engineered systems, such as artificial neural networks (ANNs), work, or that they constitute black boxes that we can only open with difficulty — is inappropriate at best and malicious at worst. We do know the mechanistic structure of such models because we designed and built them. We also do know their functional role (what they are for) as well as the mathematical function they are asked to approximate (map inputs to target outputs). Because mysterianist beliefs about known systems, such as ANNs, are often expressed, scientists need to sit up and take notice. We provide an error theory as to what is going on to help unpick this metatheoretical blunder. Ultimately, the problem is that 'understanding' is not a technical term in these cases: the word is co-opted for a specific narrative to sell 'artificial intelligence' through mystification. All computational systems, from pendulums to databases, will behave in ways we cannot predict or control — this is not a unique property of ANNs — and experts do indeed grasp the computational properties of these systems nonetheless.
GEMOT: Towards Mechanistic World Models for Biology
Scientific discovery seeks mechanisms that explain observations and predict beyond the measurements that produced them. Whereas large language models (LLMs) encode knowledge implicitly, mechanistic world models organise it as a parsimonious set of explicit, modular, reusable mechanisms whose predictions can be scored against data.
Computational hermeneutics: evaluating generative AI as a cultural technology
Generative AI (GenAI) systems are increasingly recognized as cultural technologies, yet current evaluation frameworks often treat culture as a variable to be measured rather than fundamental to the system's operation. Drawing on hermeneutic theory from the humanities, we argue that GenAI systems function as "context machines" that must inherently address three interpretive challenges: situatedness (meaning only emerges in context), plurality (multiple valid interpretations coexist), and ambiguity (interpretations naturally conflict). We present computational hermeneutics as an emerging framework offering an interpretive account of what GenAI systems do, and how they might do it better. We offer three principles for hermeneutic evaluation—that benchmarks should be iterative, not one-off; include people, not just machines; and measure cultural context, not just model output. This perspective offers a nascent paradigm for designing and evaluating contemporary AI systems: shifting from standardized questions about accuracy to contextual ones about meaning.
Abeba Birhane on Twitter / X
a classic case of “the human mind is afforded less complexity than is owed, and the computer is afforded more wisdom than is due.” Baria and Cross (2021)that "vibe" is at the core of what makes us human. it defies formalization and datafication https://t.co/AXELAqlYcq— Abeba Birhane (@Abebab) February 17, 2024
Sense-making reconsidered: large language models and the blind spot of embodied cognition
Large Language Models (LLMs) demonstrate a kind of linguistic competence that theories of embodied and enactive cognition have long deemed impossible for systems lacking the meaningful perspective of a living being, i.e., the capacity for sense-making. Facing up to this unexpected technological development requires confronting what I propose to call the “AI dilemma”: either frontier LLMs are capable of sense-making despite lacking biological embodiment, or the kind of linguistic competence they exhibit does not necessarily require sense-making. In their chapter on cognition, Frank, Thompson, and Gleiser (2024) maintain that no AI system comes close to realizing relevance, a position that derives much of its motivation from past practical failures. However, frontier LLMs have effectively overcome Dreyfus’ commonsense knowledge problem, such that their dismissal as categorically mindless risks undermining Frank et al.’s central claim that human cognition is deeply intertwined with lived experience. I therefore argue in favor of the alternative side of the AI dilemma: human-level linguistic competence of LLMs should be recognized as a novel non‑biological form of sense‑making, based on a technologically‑mediated embodiment whose enabling properties are in need of further theoretical analysis. This reorientation invites enactive theory to clarify which aspects of sense-making may be universal and which aspects are specifically contingent on organic life, thereby advancing its conceptual framework in dialogue with contemporary AI.
Ambiguity, legibility, and working in the open – ambiguiti.es
How To Think Real Good | Meta-rationality
Practical epistemology: heuristics for how to think about difficult things, particularly using formal methods.
Towards Post-Interaction Computing: Addressing Immediacy, (un)Intentionality, Instability and Interaction Effects
We situate the debate on intentionality within the rise of cognitive neuroscience and argue that cognitive neuroscience can explain intentionality. We discuss the explanatory significance of ascribing intentionality to representations. At first, we ...
Social Physics in the Age of Generative AI: How, Exactly, Is Society Complex?
The aspiration for a social physics — a science that would reveal universal laws of social life analogous to those governing the natural world—has haunted the social sciences since Comte. Each new wave of quantitative and computational methods breathes fresh life into this vision. The current moment is no exception: advances in artificial intelligence and large language models have reinvigorated attempts to model society through the methods of social physics. This essay argues that these ambitions rest on a confusion about the nature of social complexity. Drawing a distinction between systems complexity and anthropological complexity, we identify the limitations of analyzing society as a complex system akin to natural systems. One crucial obstacle for social physics is the analysis of meaning and interpretation. Generative artificial intelligence (GenAI) offers important opportunities to model language and reflexivity, inviting and perhaps compelling social physics to confront meaning and interpretation as constitutive features of the social world. At the same time, GenAI leaves untouched and may even aggravate social physics’ failure to grasp power, institutions, and inequality.