Why Can’t Powerful LLMs Learn Multiplication?

Why can’t powerful AIs learn basic multiplication?

New research reveals why even state-of-the-art large language models stumble on seemingly easy tasks—and what it takes to fix it

Why Can't Transformers Learn Multiplication?...

Language models are increasingly capable, yet still fail at a seemingly simple task of multi-digit multiplication. In this work, we study why, by reverse-engineering a model that successfully...

Why Can't Transformers Learn Multiplication?...

Language models are increasingly capable, yet still fail at a seemingly simple task of multi-digit multiplication. In this work, we study why, by reverse-engineering a model that successfully...

Can LLMs Be Computers? | Percepta

We build a computer inside a transformer — executing arbitrary C programs for millions of steps with exponentially faster inference via 2D attention heads.

Can LLMs Be Computers? | Percepta

We build a computer inside a transformer — executing arbitrary C programs for millions of steps with exponentially faster inference via 2D attention heads.

Solving a Million-Step LLM Task with Zero Errors

LLMs have achieved remarkable breakthroughs in reasoning, insights, and tool use, but chaining these abilities into extended processes at the scale of those routinely executed by humans,...

What sort of maths are LLMs good at?

For the sake of anyone who might read this blog post in the distant future (a month from now, say), let me mention that I am writing it a few days after OpenAI announced that it had solved ten majo…

Large language models are not the problem

If a Large Language Model (LLM) can replicate your scientific contribution, the problem is not the LLM. What does it say about our field that so much of the anxiety about AI comes down to the fear that a machine could do what we do? Perhaps it says we should be doing something better.

Fast and Hard Code

People pick languages differently with LLMs.

Cognitive exponents and LLM leverage

I know a few people for whom LLMs have been a near-immediate multiplier of attention and effort. I know a lot for whom LLMs clearly make them worse at thinking and doing things. So: why?

Beyond Standard LLMs

Linear Attention Hybrids, Text Diffusion, Code World Models, and Small Recursive Transformers

distil labs — Replace LLMs with Custom Small Language Models

Train and deploy custom small language models that are faster, cheaper, and just as accurate as LLMs.

Why Are LLMs Smart?

A popular way to explain how current LLMs work is to say that “all” they do is predict the next most likely word in a sentence.

We Don't Understand Neural Networks At The Algorithmic Level

The largest ongoing debate about AI is “Are Large Language Models (LLMs) intelligent?” That makes sense, at least: the evidence is ambiguous and the stakes a...

How To Scale Your Model

Training LLMs often feels like alchemy, but understanding and optimizing the performance of your models doesn't have to. This book aims to demystify the science of scaling language models: how TPUs (and GPUs) work and how they communicate with each other, how LLMs run on real hardware, and how to parallelize your models during training and inference so they run efficiently at massive scale. If you've ever wondered “how expensive should this LLM be to train” or “how much memory do I need to serve this model myself” or “what's an AllGather”, we hope this will be useful to you.