Fast, flexible LLM inference
LLM in a Flash: Efficient Large Language Model Inference with Limited Memory
Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks…
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
Overview - GroqDocs
Fast LLM inference, OpenAI-compatible. Simple to integrate, easy to scale. Start building in minutes.
Announcing Burn-LM (alpha): LLM Inference Engine
We're happy to announce the Burn-LM, an LLM inference engine based on Burn! The goal is to support any large model, LLM, VLM, and others, for inference but also for training (pre-training, post-training, and fine-tuning).
Demystifying llm-d and vLLM: The race to production
Learn how vLLM and llm-d work together for efficient and scalable large language model (LLM) inference. Discover the benefits of disaggregated scaling, expert-parallel scheduling, and KV cache-aware routing.
Economic futures for LLM inference
Inference providers will look for other lines of business.
Equivariance Encryption for Private LLM Inference — Davide Cifarelli
This article shows how we can leverage equivariant transformations to run LLM inference on encrypted data without losing model performance.
Dria on Twitter / X
Introducing Inference Arena v2.0.An agentic experience that searches, analyzes, and delivers insights about LLM inference.When we first launched, our goal was simple: make it easier for developers to compare models, engines, and hardware without digging through scattered… pic.twitter.com/fgWgos48lW— Dria (@driaforall) September 30, 2025
Defeating Nondeterminism in LLM Inference
Reproducibility is a bedrock of scientific progress. However, it’s remarkably difficult to get reproducible results out of large language models. For example, you might observe that asking ChatGPT the same question multiple times provides different results. This by itself is not surprising, since getting a result from a language model involves “sampling”, a process that converts the language model’s output into a probability distribution and probabilistically selects a token. What might be more surprising is that even when we adjust the temperature down to 0This means that the LLM always chooses the highest probability token, which is called greedy sampling. (thus making the sampling theoretically deterministic), LLM APIs are still not deterministic in practice (see past discussions here, here, or here). Even when running inference on your own hardware with an OSS inference library like vLLM or SGLang, sampling still isn’t deterministic (see here or here).
Towards Feasible, Private, Distributed LLM Inference
Exploring how the Secure Transformer Inference Protocol (STIP) protects inputs, outputs, and model weights with lightweight permutations enabling efficient, privacy-safe LLM inference at scale.