Introducing beginners to the mechanics of machine learning – Miriam Posner
Every year, I spend some time introducing students to the mechanics of machine learning with neural nets. I definitely don’t go into great depth; I usually only have one class for this. But I try to unpack at least some of the major concepts, so that ML isn’t quite such a black box.
Deep Learning with Python
Start building deep learning applications with Python and Keras today!
DeepSeek_V4.pdf · deepseek-ai/DeepSeek-V4-Pro at main
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Jan-Nano : 1st Deep Research LLM
Deep Learning on a Data Diet: Finding Important Examples Early in Training
Recent success in deep learning has partially been driven by training increasingly overparametrized networks on ever larger datasets. It is therefore natural to ask: how much of the data is superfluous, which examples are important for generalization, and how do we find them? In this work, we make the striking observation that, in standard vision datasets, simple scores averaged over several weight initializations can be used to identify important examples very early in training. We propose two such scores—the Gradient Normed (GraNd) and the Error L2-Norm (EL2N) scores—and demonstrate their efficacy on a range of architectures and datasets by pruning significant fractions of training data without sacrificing test accuracy. In fact, using EL2N scores calculated a few epochs into training, we can prune half of the CIFAR10 training set while slightly improving test accuracy. Furthermore, for a given dataset, EL2N scores from one architecture or hyperparameter configuration generalize to other configurations. Compared to recent work that prunes data by discarding examples that are rarely forgotten over the course of training, our scores use only local information early in training. We also use our scores to detect noisy examples and study training dynamics through the lens of important examples—we investigate how the data distribution shapes the loss surface and identify subspaces of the model’s data representation that are relatively stable over training.
deepseek-ai/DeepSeek-OCR-2 · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Neural Networks: Zero To Hero
Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
View recent discussion. Abstract: Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods. Taking inspiration from the network properties of biological neural networks (e.g. sparsity, scale-freeness), we argue that (contrary to general practice) artificial neural networks, too, should not have fully-connected layers. Here we propose sparse evolutionary training of artificial neural networks, an algorithm which evolves an initial sparse topology (Erdős-Rényi random graph) of two consecutive layers of neurons into a scale-free topology, during learning. Our method replaces artificial neural networks fully-connected layers with sparse ones before training, reducing quadratically the number of parameters, with no decrease in accuracy. We demonstrate our claims on restricted Boltzmann machines, multi-layer perceptrons, and convolutional neural networks for unsupervised and supervised learning on 15 datasets. Our approach has the potential to enable artificial neural networks to scale up beyond what is currently possible.
Stanford CS231N Deep Learning for Computer Vision | Spring 2025 | Lecture 1: Introduction
deepseek-ai/DeepSeek-V4-Flash-0731 at 9e165c30e2704aec5d9d593cce3eebd58bbef1cb
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Understanding Deep Learning Algorithms that Leverage Unlabeled Data, Part 2: Contrastive Learning
Theoretical analysis of contrastive learning algorithms for leveraging unlabeled data.
The Roadmap of Mathematics for Machine Learning
A complete guide to linear algebra, calculus, and probability theory
Deep Learning Scaling is Predictable, Empirically
Deep learning (DL) creates impactful advances following a virtuous recipe: model architecture search, creating large training data sets, and scaling computation. It is widely believed that growing training sets and models should improve accuracy and result in better products. As DL application domains grow, we would like a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements to advance the state-of-the-art. This paper presents a large scale empirical characterization of generalization error and model size growth as training sets grow. We introduce a methodology for this measurement and test four machine learning domains: machine translation, language modeling, image processing, and speech recognition. Our empirical results show power-law generalization error scaling across a breadth of factors, resulting in power-law exponents---the "steepness" of the learning curve---yet to be explained by theoretical work. Further, model improvements only shift the error but do not appear to affect the power-law exponent. We also show that model size scales sublinearly with data size. These scaling relationships have significant implications on deep learning research, practice, and systems. They can assist model debugging, setting accuracy targets, and decisions about data set growth. They can also guide computing system design and underscore the importance of continued computational scaling.
I Know We’re in an AI Bubble Because Nobody Wants Me 😭
I first got into deep learning in 2012, when AlexNet came out. I was CTO of Jetpac, a startup that aimed to provide information about bars, hotels, and restaurants by analyzing public photos, for e…
Towards infinite context windows: neural KV cache compaction | Base Labs
Working to advance and democratize open-source intelligence.
