







While modern language models take raw text as their input and produce raw text as output, they do not operate over text directly. Hidden in the very first step of language model pipelines is...
Stanford CS336 Language Modeling from Scratch I 2025
RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization
This study addresses the challenge of extending Large Language Models (LLMs) to non-English languages, specifically those using non-Roman scripts. We propose an approach that utilizes the romanized form of text as an interface for LLMs, hypothesizing that its frequent informal use and shared tokens with English enhance cross-lingual alignment. Our approach involve the continual pretraining of a English LLM like Llama 2 on romanized text of non-English, non-Roman script languages, followed by instruction tuning on romanized data. The results indicate that romanized text not only reduces token fertility by 2x-4x but also matches if not outperforms native script representation across various NLU, NLG and MT tasks. Moreover, the embeddings computed on romanized text exhibit closer alignment with their English translations than those from the native script. Our approach presents a promising direction for leveraging the power of English LLMs in languages traditionally underrepresented in NLP research.
The ability for language models to write coherent text can be explained as each token develops some affinity for other tokens. During next token prediction each of the previous tokens "vote" for the tokens they have high affinity for, which results in coherent continuation.
The ability for language models to write coherent text can be explained as each token develops some affinity for other tokens.During next token prediction each of the previous tokens "vote" for the tokens they have high affinity for, which results in coherent continuation. https://t.co/APkIpDoi4N— Chomba Bupe (@ChombaBupe) April 21, 2024
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
The success of the large neural language models on many NLP tasks is exciting. However, we find that these successes sometimes lead to hype in which these models are being described as “understanding” language or capturing “meaning”. In this position paper, we argue that a system trained only on form has a priori no way to learn meaning. In keeping with the ACL 2020 theme of “Taking Stock of Where We've Been and Where We're Going”, we argue that a clear understanding of the distinction between form and meaning will help guide the field towards better science around natural language understanding.
What Language is This? Ask Your Tokenizer
Language Identification (LID) is an important component of many multilingual natural language processing pipelines, where it facilitates corpus curation, training data analysis, and cross-lingual evaluation of large language models. Despite near-perfect performance on high-resource languages, existing systems remain brittle in low-resource and closely related language settings. We introduce UniLID, a simple and efficient LID method based on the UnigramLM tokenization algorithm, leveraging its probabilistic framing, parameter estimation technique and inference strategy. In short, to predict a string's language label, we simply ask: under which language's unigram distribution is this string most likely? Our formulation is data- and compute-efficient, supports incremental addition of new languages without retraining existing models, and can naturally be integrated into existing language model tokenization pipelines. Empirical evaluations against widely used baselines, including fastText, GlotLID and CLD3, show that UniLID achieves competitive performance on standard benchmarks, substantially improves sample efficiency in low-resource settings -- reaching ~70% accuracy with as few as five labeled samples per language -- and delivers large gains on fine-grained dialect identification.

Hypothesis generation and updating in large language models
Large language models (LLMs) increasingly help people solve problems, from debugging code to repairing machinery. This process requires generating plausible hypotheses from partial descriptions,...

Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
Autoregressive language models are constrained by their inherently sequential nature, generating one token at a time. This paradigm limits inference speed and parallelism, especially during later stages of generation when the direction and semantics of text are relatively certain. In this work, we propose a novel framework that leverages the inherent knowledge of vanilla autoregressive language models about future tokens, combining techniques to realize this potential and enable simultaneous prediction of multiple subsequent tokens. Our approach introduces several key innovations: (1) a masked-input formulation where multiple future tokens are jointly predicted from a common prefix; (2) a gated LoRA formulation that preserves the original LLM's functionality, while equipping it for multi-token prediction; (3) a lightweight, learnable sampler module that generates coherent sequences from the predicted future tokens; (4) a set of auxiliary training losses, including a consistency loss, to enhance the coherence and accuracy of jointly generated tokens; and (5) a speculative generation strategy that expands tokens quadratically in the future while maintaining high fidelity. Our method achieves significant speedups through supervised fine-tuning on pretrained models. For example, it generates code and math nearly 5x faster, and improves general chat and knowledge tasks by almost 2.5x. These gains come without any loss in quality.

Crowdsourcing the evolution of text parsing with unified
unified is a tool for manipulating content using syntax trees. Today, we’re announcing the unified collective to make it sustainable

Beyond Model Collapse: Scaling Up with Synthesized Data Requires...
Large Language Models (LLM) are increasingly trained on data generated by other LLM, either because generated text and images become part of the pre-training corpus, or because synthetized data is...

The Stanford NLP Group
Performing groundbreaking Natural Language Processing research since 1999.

Words that Work: Using Large Language Models to Generate and Refine Hypotheses from Text
In this paper, we introduce a data-driven framework for generating and refining hypotheses from text. Our three-step approach–Hypothesize, Intervene, and Predic
Language Machines
How generative AI systems capture a core function of language Looking at the emergence of generative AI, Language Machines presents a new theory of meaning i...

Caveman | Lithic Token Compression
A semantic constraint engine for Claude Code & Codex. Forces agentic communication into minimal-token lithic structures. Retain 100% technical accuracy while destroying up to 87% of output latency.
Andrej Karpathy on Twitter / X
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks:Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for… pic.twitter.com/uMFWTcBCsl— Andrej Karpathy (@karpathy) October 2, 2026
I'm creating a series of short form videos about how language models work technically. The goal is to be something in between "you know it's next token prediction" and "now you've taken a machine learning class." I'd love your thoughts so here are the first few! 🧵 youtube.com/shorts/VZB8XCcyllE
How does ChatGPT work? Or rather, language models in general- Part 1 attempting a lay explanation.
www.youtube.com