The Little Book of Deep Learning

Introducing beginners to the mechanics of machine learning – Miriam Posner

Every year, I spend some time introducing students to the mechanics of machine learning with neural nets. I definitely don’t go into great depth; I usually only have one class for this. But I try to unpack at least some of the major concepts, so that ML isn’t quite such a black box.

Deep Learning with Python

Start building deep learning applications with Python and Keras today!

tinygrad: A simple and powerful neural network framework

We write and maintain tinygrad, the fastest growing neural network framework

DeepSeek_V4.pdf · deepseek-ai/DeepSeek-V4-Pro at main

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Deep learning gets the glory, deep fact checking gets ignored – Rachel Thomas, PhD

AI, science, education, & ethics

Deep Learning on a Data Diet: Finding Important Examples Early in Training

Recent success in deep learning has partially been driven by training increasingly overparametrized networks on ever larger datasets. It is therefore natural to ask: how much of the data is superfluous, which examples are important for generalization, and how do we find them? In this work, we make the striking observation that, in standard vision datasets, simple scores averaged over several weight initializations can be used to identify important examples very early in training. We propose two such scores—the Gradient Normed (GraNd) and the Error L2-Norm (EL2N) scores—and demonstrate their efficacy on a range of architectures and datasets by pruning significant fractions of training data without sacrificing test accuracy. In fact, using EL2N scores calculated a few epochs into training, we can prune half of the CIFAR10 training set while slightly improving test accuracy. Furthermore, for a given dataset, EL2N scores from one architecture or hyperparameter configuration generalize to other configurations. Compared to recent work that prunes data by discarding examples that are rarely forgotten over the course of training, our scores use only local information early in training. We also use our scores to detect noisy examples and study training dynamics through the lens of important examples—we investigate how the data distribution shapes the loss surface and identify subspaces of the model’s data representation that are relatively stable over training.

deepseek-ai/DeepSeek-OCR-2 · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Neural Networks: Zero To Hero

Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science

View recent discussion. Abstract: Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods. Taking inspiration from the network properties of biological neural networks (e.g. sparsity, scale-freeness), we argue that (contrary to general practice) artificial neural networks, too, should not have fully-connected layers. Here we propose sparse evolutionary training of artificial neural networks, an algorithm which evolves an initial sparse topology (Erdős-Rényi random graph) of two consecutive layers of neurons into a scale-free topology, during learning. Our method replaces artificial neural networks fully-connected layers with sparse ones before training, reducing quadratically the number of parameters, with no decrease in accuracy. We demonstrate our claims on restricted Boltzmann machines, multi-layer perceptrons, and convolutional neural networks for unsupervised and supervised learning on 15 datasets. Our approach has the potential to enable artificial neural networks to scale up beyond what is currently possible.

deepseek-ai/DeepSeek-V4-Flash-0731 at 9e165c30e2704aec5d9d593cce3eebd58bbef1cb

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

The Divergence Machine

Introducing the 2026 Book Club theme

An Introduction to Transformers

The transformer is a neural network component that can be used to learn useful representations of sequences or sets of data-points. The transformer has driven recent advances in natural language processing, computer vision, and spatio-temporal modelling. There are many introductions to transformers, but most do not contain precise mathematical descriptions of the architecture and the intuitions behind the design choices are often also missing. Moreover, as research takes a winding path, the explanations for the components of the transformer can be idiosyncratic. In this note we aim for a mathematically precise, intuitive, and clean description of the transformer architecture. We will not discuss training as this is rather standard. We assume that the reader is familiar with fundamental topics in machine learning including multi-layer perceptrons, linear transformations, softmax functions and basic probability.

Machine Learning  |  Google for Developers

Understanding Deep Learning Algorithms that Leverage Unlabeled Data, Part 2: Contrastive Learning

Theoretical analysis of contrastive learning algorithms for leveraging unlabeled data.