Knowledge distillation is a machine learning technique used to transfer the learning of a large pre-trained “teacher model” to a smaller “student model.”
Everything You Need to Know about Knowledge Distillation
A Blog post by Ksenia Se on Hugging Face
Dataset Distillation
Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called {\em dataset distillation}: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to {\em synthesize} a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress $60,000$ MNIST training images into just $10$ synthetic {\em distilled images} (one per class) and achieve close to original performance with only a few steps of gradient descent, given a particular fixed network initialization. We evaluate our method in a wide range of initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods in most settings.
On-Policy Distillation
On-policy, dense supervision is a useful tool for distillation
Distilling the Knowledge in a Neural Network
Our teams advance the state of the art through research, systems engineering, and collaboration across Google.
Distillation Scaling Laws
We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the…
241. Distillation Is Not Anti-American, Weaponizing It Is
Recent attempts at pursuing government intervention in distillation of AI models are short-sighted and technically weak arguments to attempt regulatory capture
The distillation panic
‘Distillation attacks’ is a horrible term for what is happening right now.
Detecting and preventing distillation attacks
Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.
Language models transmit behavioural traits through hidden signals in data
Large language models (LLMs) are increasingly used to generate data to train improved models1–3, but it remains unclear what properties are transmitted in this model distillation4,5. Here we show that distillation can lead to subliminal learning—the transmission of behavioural traits through semantically unrelated data. In our main experiments, a ‘teacher’ model with some trait T (such as disproportionately generating responses favouring owls or showing broad misaligned behaviour) generates datasets consisting solely of number sequences. Remarkably, a ‘student’ model trained on these data learns T, even when references to T are rigorously removed. More realistically, we observe the same effect when the teacher generates math reasoning traces or code. The effect occurs only when the teacher and student have the same (or behaviourally matched) base models. To help explain this, we prove a theoretical result showing that subliminal learning arises in neural networks under broad conditions and demonstrate it in a simple multilayer perceptron (MLP) classifier. As artificial intelligence systems are increasingly trained on the outputs of one another, they may inherit properties not visible in the data. Safety evaluations may therefore need to examine not just behaviour, but the origins of models and training data and the processes used to create them.
Elon Musk confirms xAI used OpenAI’s models to train Grok
He said it was “partly” true that the company had used model distillation to improve xAI’s models.
See what you think
Allegra A. Beal Cohen's blog about knowledge curation, new interfaces, and large-scale qualitative data.
Simply Jekyll
A non-linear thread of important ideas weaved together to explore and understand the fabric of knowledge.
Changing Minds: Computers, Learning, and Literacy
An impassioned guide to how computers can fundamentally change how we learn and think.Andrea diSessa's career as a scholar, technologist, and teacher has b
The Best Tacit Knowledge Videos on Every Subject — LessWrong
TL;DR Tacit knowledge is extremely valuable. Unfortunately, developing tacit knowledge is usually bottlenecked by apprentice-master relationships. Ta…
Backpropagation
In machine learning, backpropagation is a gradient computation method commonly used for training a neural network in computing parameter updates.
LLM Knowledge Bases
A visual breakdown of Andrej Karpathy's approach to building personal knowledge bases powered by LLMs. Learn the 4-phase pipeline: ingest, compile, query, and maintain - with an interactive architecture diagram.
