Stanford CS336 | Language Modeling from Scratch (Spring 2025 Archive)

Recursive Language Models: the paradigm of 2026

How we plan to manage extremely long contexts

The Kaitchup – AI on a Budget | Benjamin Marie | Substack

Weekly tutorials and news on adapting large language models (LLMs) to your tasks and hardware using the most recent techniques and models. The Kaitchup proposes a collection of 180+ AI notebooks regularly updated. Click to read The Kaitchup – AI on a Budget, by Benjamin Marie, a Substack publication with tens of thousands of subscribers.

Build A Large Language Model (From Scratch), Published by Manning, ISBN 978-1633437166

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

Extracting Training Data from Large Language Models

Nicholas Carlini, Google; Florian Tramèr, Stanford University; Eric Wallace, UC Berkeley; Matthew Jagielski, Northeastern University; Ariel Herbert-Voss, OpenAI and Harvard University; Katherine Lee and Adam Roberts, Google; Tom Brown, OpenAI; Dawn Song, UC Berkeley; Úlfar Erlingsson, Apple; Alina Oprea, Northeastern University; Colin Raffel, Google

arcee-ai/mergekit

Tools for merging pretrained large language models.

Olmo from Ai2

Our fully open language model and complete model flow.

Recursive Language Models

We propose Recursive Language Models (RLMs), an inference strategy where language models can decompose and recursively interact with input context of unbounded length through REPL environments.

Cheatsheet - DSPy

The framework for programming—rather than prompting—language models.

openbench: Provider-agnostic, open-source evaluation infrastructure for language models

Provider-agnostic, open-source evaluation infrastructure for language models

mozilla/langview

EXPERIMENTAL – A library for language models to respond with GUI.

Self-Instruct: Aligning Language Models with Self-Generated Instructions

Large "instruction-tuned" language models (i.e., finetuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks. Nevertheless, they depend heavily on human-written instruction data that is often limited in quantity, diversity, and creativity, therefore hindering the generality of the tuned model. We introduce Self-Instruct, a framework for improving the instruction-following capabilities of pretrained language models by bootstrapping off their own generations. Our pipeline generates instructions, input, and output samples from a language model, then filters invalid or similar ones before using them to finetune the original model. Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT-001, which was trained with private user data and human annotations. For further evaluation, we curate a set of expert-written instructions for novel tasks, and show through human evaluation that tuning GPT3 with Self-Instruct outperforms using existing public instruction datasets by a large margin, leaving only a 5% absolute gap behind InstructGPT-001. Self-Instruct provides an almost annotation-free method for aligning pre-trained language models with instructions, and we release our large synthetic dataset to facilitate future studies on instruction tuning. Our code and data are available at https://github.com/yizhongw/self-instruct.

ModelArk-Coding-Plan

First month from $4.5 USD. Flexible models, unlimited tools.

sgl-project/sglang

SGLang is a high-performance serving framework for large language models and multimodal models.