A high-throughput and memory-efficient inference and serving engine for LLMs
Demystifying llm-d and vLLM: The race to production
Learn how vLLM and llm-d work together for efficient and scalable large language model (LLM) inference. Discover the benefits of disaggregated scaling, expert-parallel scheduling, and KV cache-aware routing.
Inference Time Memory Module | Research | Tiles
Simple inference-time memory module that treats memory management as a series of LLM calls and agent loops over a markdown-based file tree.
Announcing Burn-LM (alpha): LLM Inference Engine
We're happy to announce the Burn-LM, an LLM inference engine based on Burn! The goal is to support any large model, LLM, VLM, and others, for inference but also for training (pre-training, post-training, and fine-tuning).
LLM in a Flash: Efficient Large Language Model Inference with Limited Memory
Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks…
Overview - GroqDocs
Fast LLM inference, OpenAI-compatible. Simple to integrate, easy to scale. Start building in minutes.
Wolfram LLM Benchmarking Project
Results from Wolfram's ongoing tracking of LLM performance. The benchmark is based on a Wolfram Language code generation task.
Patrick Collison on Twitter / X
I want some kind of LLM workflow tool.• Ability to manage a set of input files (Markdown or similar), plus other general-purpose context.• With real-time collaboration. (And maybe some concept of snapshots or VCS integration.)• And the ability to create/manage a inference…— Patrick Collison (@patrickc) June 6, 2026
Constructing an LLM-Computer | Percepta
How we analytically compiled a WebAssembly VM into a transformer using a code-to-weights compiler we built.
Docker Model Runner Integrates vLLM for High-Throughput Inferencing
How Docker Model Runner integrates vLLM as an inference backend, letting developers run safetensors models with high-throughput serving, PagedAttention, streami
On-Device LLM Throughput Calculator - a Hugging Face Space by FL33TW00D-HF
This tool estimates and visualizes the throughput of Large Language Models on devices with memory bandwidth constraints. Users input device and model configurations, and the tool generates a plot s...
Can LLMs Be Computers? | Percepta
We build a computer inside a transformer — executing arbitrary C programs for millions of steps with exponentially faster inference via 2D attention heads.
Can LLMs Be Computers? | Percepta
We build a computer inside a transformer — executing arbitrary C programs for millions of steps with exponentially faster inference via 2D attention heads.
An Internet for the KV Cache: Rethinking Classical Infrastructure Boundaries in the LLM Inference Age
LLM inference has become a global-scale, heterogeneous workload spanning agents, retrieval, tool-use, code execution and multi-modal reasoning. These workloads naturally enable context reuse from overlapping inputs, creating a major opportunity to store and reuse the contexts' KV Caches instead of recomputing them. However, model-side advances that shrink the KV Cache and system-side advances that reduce compute, storage, and transfer costs are evolve independently within legacy cloud boundaries. We argue that future inference infrastructure should allow decoupling of compute and KV Cache storage across cloud and datacenters. The network becomes an active distribution channel; bandwidth, latency and pricing directly determines how the KV Cache should be managed. We propose a vision for an Internet for the KV Cache, with KV Cache management working as a content-distribution system. In this view, KV Cache storage and recompute decisions are driven by model, infrastructure, and application metrics, to enable adaptive, content-driven decisions for minimizing latency and cost.
