







This is the most step-by-step spelled-out explanation of backpropagation and training of neural networks. It only assumes basic knowledge of Python and a vague recollection of calculus from high school. Links: - micrograd on github: https://github.com/karpathy/micrograd - jupyter notebooks I built in this video: https://github.com/karpathy/nn-zero-to-hero/tree/master/lectures/micrograd - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - "discussion forum": nvm, use youtube comments below for now :) - (new) Neural Networks: Zero to Hero series Discord channel: https://discord.gg/3zy8kqD9Cp , for people who'd like to chat more and go beyond youtube comments Exercises: you should now be able to complete the following google collab, good luck!: https://colab.research.google.com/drive/1FPTx1RXtBfc4MaTkf7viZZD4U2F9gtKN?usp=sharing Chapters: 00:00:00 intro 00:00:25 micrograd overview 00:08:08 derivative of a simple function with one input 00:14:12 derivative of a function with multiple inputs 00:19:09 starting the core Value object of micrograd and its visualization 00:32:10 manual backpropagation example #1: simple expression 00:51:10 preview of a single optimization step 00:52:52 manual backpropagation example #2: a neuron 01:09:02 implementing the backward function for each operation 01:17:32 implementing the backward function for a whole expression graph 01:22:28 fixing a backprop bug when one node is used multiple times 01:27:05 breaking up a tanh, exercising with more operations 01:39:31 doing the same thing but in PyTorch: comparison 01:43:55 building out a neural net library (multi-layer perceptron) in micrograd 01:51:04 creating a tiny dataset, writing the loss function 01:57:56 collecting all of the parameters of the neural net 02:01:12 doing gradient descent optimization manually, training the network 02:14:03 summary of what we learned, how to go towards modern neural nets 02:16:46 walkthrough of the full code of micrograd on github 02:21:10 real stuff: diving into PyTorch, finding their backward pass for tanh 02:24:39 conclusion 02:25:20 outtakes :)
Deep Dive into LLMs like ChatGPT
tinygrad: A simple and powerful neural network framework
We write and maintain tinygrad, the fastest growing neural network framework
Taking AI Doom Seriously For 62 Minutes
I Built an LLM From Scratch
A lot of Machine Learning (ML) I learned during my Ph.D. was from youtube. I didn't have a guide to do this effectively and thus here it is: A complete guide to studying ML from youtube: 13 best and most recent ML courses available on YouTube. 👩🏫🧵⤵️
A lot of Machine Learning (ML) I learned during my Ph.D. was from youtube. I didn't have a guide to do this effectively and thus here it is:A complete guide to studying ML from youtube: 13 best and most recent ML courses available on YouTube. 👩🏫🧵⤵️— Sanju Sinha (@Sanjusinha7) November 7, 2022
Build Your Own Reasoning Model: A Full Video Course
Over 11 hours of coding videos on evaluation, inference-time scaling, reinforcement learning, and distillation, with Python and PyTorch.

Neural Networks: Zero To Hero
Karpathy's LLM Wiki: The Complete Guide to His Idea File
Karpathy's follow-up gist went viral. Complete breakdown with implementation examples.

How to build your own AlphaZero AI using Python and Keras
Teach a machine to learn Connect4 strategy through self-play and deep learning.

AI is Proving Things No One Understands*
What I've learned so far - Hugging Face MCP Course (Module 1)
It’s a buffet out there! If you want to learn about open source + AI, there’s no shortage of ways to start. I began with Hugging Face, the epicenter of machine learning developer communities. I noticed on LinkedIn that they were running an online course to teach folks about

Build A Large Language Model (From Scratch), Published by Manning, ISBN 978-1633437166
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
10 Easy Ways to Enhance Your LLM Wiki or Knowledge Base
My Review on Math Academy after almost 1 year as a university student (15k + exp total)
Build a Reasoning Model (From Scratch)
"An exceptional deep dive into the next frontier of AI.” —Aman Chadha, Google Build a Reasoning Model (From Scratch) is a practical guide to understanding how modern reasoning-oriented LLMs work by building their core methods step by step. The book tells a clear engineering story: start with a conventional pre-trained LLM, learn how text generation works, build reliable evaluation tools, improve reasoning through inference-time methods, then move into training-based approaches such as reinforcement learning and distillation. The progression is deliberate. Early chapters establish the baseline model and explain text generation, KV caching, and evaluation with math verifiers. The middle chapters show how reasoning can be improved without changing model weights, using chain-of-thought prompting, sampling, self-consistency, response scoring, and self-refinement. Later chapters move to changing the model itself through reinforcement learning with verifiable rewards, GRPO improvements, format rewards, and finally distillation from stronger reasoning models into smaller ones. The book is especially useful because it implements the core methods from scratch rather than treating them as black-box library calls. Readers see how self-consistency, self-refinement, Best-of-N, and training-based methods actually work, including their cost and latency trade-offs. It also discusses common failure modes, including cases where refinement can make answers worse. Difficult concepts such as softmax, temperature, and top-p sampling are clarified with code-linked explanations and diagrams, and visual workflows make pipelines and scoring methods easier to follow. Reading the book feels like following a guided technical build rather than a loose survey of AI topics. Each concept is introduced because the project now needs it. Diagrams, roadmaps, code listings, exercises, and repeated workflow summaries help readers stay oriented through advanced material. This structure reflects Sebastian Raschka’s professional strength: explaining complex machine learning topics by making every detail concrete and showing exactly where each section fits in the larger story. He does not treat mechanisms like evaluation, log-probabilities, KL regularization, or distillation as isolated abstractions; he connects them to the goal of making reasoning models understandable and implementable. Physically and organizationally, the book has eight chapters and seven substantial appendixes. That design keeps the main narrative focused while moving supporting material like references, exercise solutions, model source code, larger models, batching, evaluation alternatives, and chat interfaces into ordered appendixes. The result is a logically flowing book that remains hands-on, navigable, and technically deep without constantly interrupting the central build.

hodgesmr/neural_network_from_scratch
A Jupyter notebook building and training a Neural Network from scratch with NumPy