Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

We demonstrate that sparse autoencoders can extract interpretable features from Claude 3 Sonnet, a production-scale language model, addressing the open question of whether dictionary learning methods scale beyond small transformers. We trained sparse autoencoders with up to 34 million features on the model's middle layer residual stream, using scaling laws to guide hyperparameter selection. The resulting features are multilingual and multimodal (generalizing to images despite text-only training), respond to both concrete instances and abstract discussions of concepts, and can be used to steer model behavior in ways consistent with their interpretations. We find features corresponding to famous entities and locations, as well as more abstract concepts like sarcasm or errors in code. We also identify features relevant to ways in which language models might cause harm--including features representing deception, power-seeking, sycophancy, and bias--and show that these causally influence model outputs when manipulated. Additionally, we conduct analyses of feature interpretability, geometry, and computational function. However, significant limitations remain: our suite of features is incomplete, and we lack rigorous methods for evaluating whether our features faithfully capture model computations.

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

We demonstrate that sparse autoencoders can extract interpretable features from Claude 3 Sonnet, a production-scale language model, addressing the open question of whether dictionary learning methods scale beyond small transformers. We trained sparse autoencoders with up to 34 million features on the model's middle layer residual stream, using scaling laws to guide hyperparameter selection. The resulting features are multilingual and multimodal (generalizing to images despite text-only training), respond to both concrete instances and abstract discussions of concepts, and can be used to steer model behavior in ways consistent with their interpretations. We find features corresponding to famous entities and locations, as well as more abstract concepts like sarcasm or errors in code. We also identify features relevant to ways in which language models might cause harm--including features representing deception, power-seeking, sycophancy, and bias--and show that these causally influence model outputs when manipulated. Additionally, we conduct analyses of feature interpretability, geometry, and computational function. However, significant limitations remain: our suite of features is incomplete, and we lack rigorous methods for evaluating whether our features faithfully capture model computations.

Mapping the Mind of a Large Language Model

We have identified how millions of concepts are represented inside Claude Sonnet, one of our deployed large language models. This is the first ever detailed look inside a modern, production-grade large language model.

Introducing Claude Sonnet 5

Our most agentic Sonnet yet, with top-tier intelligence for coding and everyday professional work.

Maintaining the love for coding in the time of AI

By Alex Band Over the last year, we have seen the profound effects Large Language Models (LLMs) have on our open-source software work at NLnet Labs. These can be grouped into two main areas: community contributions and security reports. Before we dive into this topic, let me first give

Random Tech Thoughts

This article is about standardisation, particularly how it relates to understanding code.  I’ll first go into an historical example that highlights the lack of standardisation in an area where we take it for granted today.  After that I’ll get into code, and how understanding code is like understanding data via visualisations.  In both the historic … Continue reading Standardisation and code

The language of generalization.

Apple Intelligence Foundation Language Models Tech Report 2025

We introduce two multilingual, multimodal foundation language models that power Apple Intelligence features across Apple devices and…

The Beauty of Something Grown

On Mechanistic Interpretability

Tracing the thoughts of a large language model

Anthropic's latest interpretability research: a new microscope to understand Claude's internal mechanisms

Claude Code: An analysis | Notion

Why Claude Code Matters

Jailbreaking Large Language Models: If You Torture the Model Long Enough, It Will Confess!

A Cautionary Tale…

Cognition | Agent Trace: Capturing the Context Graph of Code

We’re excited to join in Cursor, Cloudflare, Vercel, git-ai, OpenCode and others in support of [Agent Trace](https://agent-trace.dev/). As described in the spec, Agent Trace is an open, vendor-neutral spec for recording AI contributions alongside human authorship in version-controlled codebases.

Introducing talkie: a 13B vintage language model from 1930

This is a 24/7 live feed of Claude Sonnet 4.6 prompting talkie-1930-13b-it in order to explore its knowledge, capabilities, and inclinations. talkie’s outputs reflect the culture and values of the texts it was trained on, not the views of its authors.