1.1K votes, 64 comments. I build a 7-node ESP32-S3 cluster to run a ~0.4B LLM using 1.58-bit quantization. This builds on the idea of distributing…
How I fit a 28.9M LLM on an ESP32-S3 (~9 tok/s, fully on-chip)
1.2K votes, 69 comments. I wanted to see how big a language model I could actually run on an ESP32. Not the 260K-param TinyStories model that's been…
A Visual Guide to Quantization
Exploring memory-efficient techniques for LLMs
A 4-Bit Model and a 1-Bit Index
Running NVFP4 Nemotron on a CPU, then mapping every embedding-compression method at matched byte budgets. The two quantizations compose.
Run LLMs locally on your Mac · mlx-optiq
Quantize, fine-tune and serve LLMs locally on Apple Silicon. MLX-native, no PyTorch, no cloud. On PyPI.
mzau/broke-cluster
A Poor Man's Apple Silicon LLM Cluster — tuned for MLX, scalable without shame.
omlx/docs/oQ_Quantization.md at main · jundot/omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar - jundot/omlx
Reverse-engineering GGUF | Post-Training Quantization
Megakernel: Matching Apple Silicon Efficiency at 2x the Throughput on a RTX 3090
The first megakernel for hybrid DeltaNet/Attention LLMs. 413 tok/s at 1.87 tok/J on a 2020 RTX 3090, matching M5 Max efficiency at 1.8x throughput.
Quantization from the ground up | ngrok blog
A complete guide to what quantization is, how it works, and how it's used to compress large language models
The Kaitchup Index: A Leaderboard for LLMs and Their Quantized Versions
Comparing formats like GGUF, GPTQ, and AWQ, with different bitwidths
On-Device LLM Throughput Calculator - a Hugging Face Space by FL33TW00D-HF
This tool estimates and visualizes the throughput of Large Language Models on devices with memory bandwidth constraints. Users input device and model configurations, and the tool generates a plot s...
Introducing LFM2: The Fastest On-Device Foundation Models on the Market | Liquid AI
Today, we release LFM2, a new class of Liquid Foundation Models (LFMs) that sets a new standard in quality, speed, and memory efficiency for on-device deployment. Built on a hybrid architecture, LFM2 delivers 200% faster decode and prefill performance than Qwen3 and Gemma 3 on CPU. It also significantly outperforms models in each size class on instruction-following and function calling—the core capabilities that make LLMs reliable for building AI agents.
[Tool Release] Finetune & Quantize 1–3B LLMs on 8GB RAM using LoFT CLI (TinyLlama + QLoRA + llama.cpp)
23 votes, 16 comments. Hey folks — I’ve been working on a CLI tool called LoFT (Low-RAM Finetuning Toolkit), and I finally have a working release. 🔧…
Quantizer support in Mediabunny v1.52.0 | Mediabunny
Mediabunny v1.52.0 adds quantizer-based video encoding for AVC, HEVC, VP9, and AV1, enabling constant-quality video encoding.
Benchmarks | EXO
Transparent benchmarks for LLMs tested on real hardware. Coming soon.