1.2K votes, 69 comments. I wanted to see how big a language model I could actually run on an ESP32. Not the 260K-param TinyStories model that's been…
On-Device LLM Throughput Calculator - a Hugging Face Space by FL33TW00D-HF
This tool estimates and visualizes the throughput of Large Language Models on devices with memory bandwidth constraints. Users input device and model configurations, and the tool generates a plot s...
I built a 7-node ESP32-S3 cluster to run a ~0.4B LLM using 1.58-Bit (BitNet) ternary quantization...
1.1K votes, 64 comments. I build a 7-node ESP32-S3 cluster to run a ~0.4B LLM using 1.58-bit quantization. This builds on the idea of distributing…
Can You Run This LLM? VRAM Calculator (Nvidia GPU and Apple Silicon)
Calculate the VRAM required to run any large language model.
How To Scale Your Model
Training LLMs often feels like alchemy, but understanding and optimizing the performance of your models doesn't have to. This book aims to demystify the science of scaling language models: how TPUs (and GPUs) work and how they communicate with each other, how LLMs run on real hardware, and how to parallelize your models during training and inference so they run efficiently at massive scale. If you've ever wondered “how expensive should this LLM be to train” or “how much memory do I need to serve this model myself” or “what's an AllGather”, we hope this will be useful to you.
Scaling Laws Across Model Architectures: A Comparative Analysis of...
The scaling of large language models (LLMs) is a critical research area for the efficiency and effectiveness of model training and deployment. Our work investigates the transferability and...
Demystifying llm-d and vLLM: The race to production
Learn how vLLM and llm-d work together for efficient and scalable large language model (LLM) inference. Discover the benefits of disaggregated scaling, expert-parallel scheduling, and KV cache-aware routing.
The Big LLM Architecture Comparison
From DeepSeek-V3 to Kimi K2: A Look At Modern LLM Architecture Design
How to Make Small Language Models Outperform Large Language Models Using DSPy!
How a 3B Language Model Surpasses an 8B Counterpart with DSPy? “In an era where language models (LMs) are revolutionising countless tasks, their potential is only as powerful as we interpret …
distil labs — Replace LLMs with Custom Small Language Models
Train and deploy custom small language models that are faster, cheaper, and just as accurate as LLMs.
Running local models on an M4 with 24GB memory | jola.dev
Experiments with getting usable outputs out of local models on a standard Macbook
Your Laptop Isn’t Ready for LLMs. That’s About to Change
The quest to run large AI models locally on an individual's machine are driving the biggest change in laptop architecture in decades.
Large language models are not the problem
If a Large Language Model (LLM) can replicate your scientific contribution, the problem is not the LLM. What does it say about our field that so much of the anxiety about AI comes down to the fear that a machine could do what we do? Perhaps it says we should be doing something better.
Build Bigger With Small Ai: Running Small Models Locally