An Observation on Generalization
Language model harnesses are compositional generalizers
Harnesses can lead to compositional generalization: we observe a property in training RLMs, in which similarly structured tasks are viewed as isomorphic and all individual LM calls in the harness become in-distribution.
Understanding Understanding: A Pragmatic Framework Motivated by...
Motivated by the rapid ascent of Large Language Models (LLMs) and debates about the extent to which they possess human-level qualities, we propose a framework for testing whether any agent (be it...
Empirical Generalization Research Papers - Academia.edu
View Empirical Generalization Research Papers on Academia.edu for free.
Generalised algebraic theories and contextual categories
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
We find a diversity of highly abstract features. They both respond to and behaviorally cause abstract behaviors. Examples of features we find include features for famous people, features for countries and cities, and features tracking type signatures in code. Many features are multilingual (responding to the same concept across languages) and multimodal (responding to the same concept in both text and images), as well as encompassing both abstract and concrete instantiations of the same idea (such as code with security vulnerabilities, and abstract discussion of security vulnerabilities).
Mixture-of-Experts (MoE): The Birth and Rise of Conditional Computation
What two decades of research taught us about sparse language models...
The Lambek Calculus
There is a noticeable revival of categorial grammar these days, as a vehicle for linguistic description. The systems used differ somewhat from the original calculus of Ajdukiewicz and Bar-Hillel, however. In particular, there is a component of rules for ‘type change’ of expressions, making for greater flexibility and elegance. One fundamental system of this kind is the so-called ‘Lambek Calculus’, whose type-change rules show a close analogy with the inference rules of constructive propositional logic. In this paper, we present one calculus of this kind, and survey its theoretical properties as a device in linguistic semantics. Our two main new contributions are a new and complete semantics for this calculus, as well as a modest study of its language-accepting capacity. In this way, we hope to provide a better understanding of the background theory of flexible categorial grammar, in tandem with its descriptive uses.
Computation and its Connotations
A Review of Language Machines by Leif Weatherby
Recursive Language Models: the paradigm of 2026
How we plan to manage extremely long contexts
.png?v=c8c07d4bf43b)
Variety (cybernetics)
In cybernetics, the term variety denotes the total number of distinguishable elements of a set, most often the set of states, inputs, or outputs of a finite-state machine or transformation, or the binary logarithm of the same quantity.[1] Variety is used in cybernetics as an information theory that is easily related to deterministic finite automata, and less formally as a conceptual tool for thinking about organization, regulation, and stability. It is an early theory of complexity in automata, complex systems,[1]: 6  and operations research.[2]
Mapping the Mind of a Large Language Model
We have identified how millions of concepts are represented inside Claude Sonnet, one of our deployed large language models. This is the first ever detailed look inside a modern, production-grade large language model.
The Hitchhiker's Guide to Monoculture
Large language models (LLMs) often produce homogeneous outputs, raising concerns that AI coding assistants may lead to convergence in the software artifacts that developers create. Whether this occurs in practice is unclear because developers interactively prompt, evaluate, modify, and reject model outputs, and because outputs vary with prompt and repository context. I examine code homogenization using Kaggle contest submissions from 2019 to mid-2026. I first document widespread convergence toward the random seed value 42, consistent with LLMs reinforcing a longstanding convention in programming culture. I then study homogenization more broadly, at two levels of aggregation and abstraction. At the submission level, I measure the average pairwise similarity of submissions within contests. At the contest level, I measure the conceptual span of submitted code, motivating distinct measures for each: TF-IDF representations, which capture surface syntax, and Voyage 3 code embeddings, which capture code intent and semantics. The results demonstrate substantial syntactic homogenization at both the individual and collective levels: individual submissions have become more alike in literal syntax and code structure, while the latent dimensionality of syntactic variation has narrowed. In contrast, I find little evidence of semantic homogenization, individually and collectively. Average semantic distance remains essentially flat, and the contest-level latent dimensional span of semantic approaches remains stable. These findings suggest that AI coding assistants are certainly standardizing implementation details, yet they have not yet produced evidence of homogenization in the approaches and problem-solving strategies coders employ.
Abstract syntax and variable binding
We develop a theory of abstract syntax with variable binding. To every binding signature we associate a category of models consisting of variable sets endowed with compatible algebra and substitution structures. The syntax generated by the signature is the initial model. This gives a notion of initial algebra semantics encompassing the traditional one; besides compositionality, it automatically verifies the semantic substitution lemma.
generalized algebraic theory in nLab
A generalized algebraic theory (GAT, Cartmell 1978/86) is a dependent type theory in the syntactic sense: There are judgments declaring types and terms of types, where, critically, a type AA declared in context Γ\Gamma is allowed to depend on terms of Γ\Gamma. The only other judgments allowed are of equality (judgemental equality) of pairs of types or terms.