Knowledge distillation is a machine learning technique used to transfer the learning of a large pre-trained “teacher model” to a smaller “student model.”
Everything You Need to Know about Knowledge Distillation
A Blog post by Ksenia Se on Hugging Face
Dataset Distillation
Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called {\em dataset distillation}: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to {\em synthesize} a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress $60,000$ MNIST training images into just $10$ synthetic {\em distilled images} (one per class) and achieve close to original performance with only a few steps of gradient descent, given a particular fixed network initialization. We evaluate our method in a wide range of initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods in most settings.
On-Policy Distillation
On-policy, dense supervision is a useful tool for distillation
Distilling the Knowledge in a Neural Network
Our teams advance the state of the art through research, systems engineering, and collaboration across Google.
DIKW pyramid
The DIKW pyramid is a model describing relationships between data, information, knowledge and wisdom sometimes also stylized as a chain, refer to models of possible structural and functional relationships between a set of components—often four: data, information, knowledge, and wisdom. The concept has roots predating the 1980s. In the latter years of that decade, interest in the models grew after explicit presentations and discussions, including from Milan Zeleny, Russell Ackoff, and Robert W. Lucky. Subsequent important discussions extended along theoretical and practical lines into the coming decades.
Distillation Scaling Laws
We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the…
241. Distillation Is Not Anti-American, Weaponizing It Is
Recent attempts at pursuing government intervention in distillation of AI models are short-sighted and technically weak arguments to attempt regulatory capture
The distillation panic
‘Distillation attacks’ is a horrible term for what is happening right now.
Detecting and preventing distillation attacks
Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.
What is Machine Learning? | IBM
Machine learning is the subset of AI focused on algorithms that analyze and “learn” the patterns of training data in order to make accurate inferences about new data.
Language models transmit behavioural traits through hidden signals in data
Large language models (LLMs) are increasingly used to generate data to train improved models1–3, but it remains unclear what properties are transmitted in this model distillation4,5. Here we show that distillation can lead to subliminal learning—the transmission of behavioural traits through semantically unrelated data. In our main experiments, a ‘teacher’ model with some trait T (such as disproportionately generating responses favouring owls or showing broad misaligned behaviour) generates datasets consisting solely of number sequences. Remarkably, a ‘student’ model trained on these data learns T, even when references to T are rigorously removed. More realistically, we observe the same effect when the teacher generates math reasoning traces or code. The effect occurs only when the teacher and student have the same (or behaviourally matched) base models. To help explain this, we prove a theoretical result showing that subliminal learning arises in neural networks under broad conditions and demonstrate it in a simple multilayer perceptron (MLP) classifier. As artificial intelligence systems are increasingly trained on the outputs of one another, they may inherit properties not visible in the data. Safety evaluations may therefore need to examine not just behaviour, but the origins of models and training data and the processes used to create them.
Konrad Hinsen's blog
Knowledge refinement is the ever ongoing process in science (and beyond it) that shepherds knowledge from lab notebooks into journal articles and then on to review articles, monographs, reference handbooks, university textbooks, and finally professional domain expertise and school education for a wider public. It has been going on for a few centuries, but we hardly talk about it. In fact, I made up the term because I couldn't find an established one. Computational knowledge has not yet found its place in the knowledge refinement process. Why not? And what can we do to make it happen?
Elon Musk confirms xAI used OpenAI’s models to train Grok
He said it was “partly” true that the company had used model distillation to improve xAI’s models.
What Are Machine Learning Algorithms? | IBM
A machine learning algorithm is the procedure and mathematical logic through which an AI model learns patterns in training data and applies to them to new data.
See what you think
Allegra A. Beal Cohen's blog about knowledge curation, new interfaces, and large-scale qualitative data.
Simply Jekyll
A non-linear thread of important ideas weaved together to explore and understand the fabric of knowledge.