Google's TurboQuant landed for vLLM.cpp's KV cache – but it's not just useful for KV! A possibly more impactful implementation is for quantizing embedding vectors for RAG. github.com/oaustegard/remex is a Python and Mojo for just that purpose.
GitHub - oaustegard/remex: Retrieval-validated embedding compression. 4-8x smaller vectors, proven recall. Based on TurboQuant (ICLR 2026).
github.comMay 2, 2026 at 3:10 AM
Qdrant
Qdrant is an Open-Source Vector Search Engine written in Rust. It provides fast and scalable vector similarity search service with convenient API.
An Internet for the KV Cache: Rethinking Classical Infrastructure Boundaries in the LLM Inference Age
LLM inference has become a global-scale, heterogeneous workload spanning agents, retrieval, tool-use, code execution and multi-modal reasoning. These workloads naturally enable context reuse from overlapping inputs, creating a major opportunity to store and reuse the contexts' KV Caches instead of recomputing them. However, model-side advances that shrink the KV Cache and system-side advances that reduce compute, storage, and transfer costs are evolve independently within legacy cloud boundaries. We argue that future inference infrastructure should allow decoupling of compute and KV Cache storage across cloud and datacenters. The network becomes an active distribution channel; bandwidth, latency and pricing directly determines how the KV Cache should be managed. We propose a vision for an Internet for the KV Cache, with KV Cache management working as a content-distribution system. In this view, KV Cache storage and recompute decisions are driven by model, infrastructure, and application metrics, to enable adaptive, content-driven decisions for minimizing latency and cost.
[RFC PATCH 00/73] KVM: x86/PVM: Introduce a new hypervisor
Constructing an LLM-Computer | Percepta
How we analytically compiled a WebAssembly VM into a transformer using a code-to-weights compiler we built.
A 4-Bit Model and a 1-Bit Index
Running NVFP4 Nemotron on a CPU, then mapping every embedding-compression method at matched byte budgets. The two quantizations compose.
Benjamin Marie on Twitter / X
Quantization and Qwen3-VL✔️ AutoRound W4A16 (INT4)✔️ AutoRound NVFP4 (llm compressor format)❌ support by vLLM (tried stable and dev releases)VLMs are now very easy to quantize. And then we can’t use the models with fast inference frameworks like vLLM and SGLang.It…— Benjamin Marie (@bnjmn_marie) November 5, 2025
Benchmarking Tesla GPUs - esologic
Decommissioned enterprise tesla GPUs are widely available and extremely cheap. In this post, I benchmark tesla GPUs to see if they are useful
shady-gang
Home to the Shady IR and Vcc compiler. shady-gang has 17 repositories available. Follow their code on GitHub.
Thireus/GGUF-Tool-Suite
Produce your own Dynamic 3.0 Quants and achieve optimum accuracy & SOTA quantization performance! Input a target size and the toolchain will create a GGUF recipe tuned to your hardware within seconds — flexible model sizing and lowest achievable perplexity/kld for GGUF enthusiasts seeking precise and automated dynamic quant production.
Vaibhav (VB) Srivastav on Twitter / X
🚨 Apple just released FastVLM on Hugging Face - 0.5, 1.5 and 7B real-time VLMs with WebGPU support 🤯> 85x faster and 3.4x smaller than comparable sized VLMs> 7.9x faster TTFT for larger models> designed to output fewer output tokens and reduce encoding time for high… pic.twitter.com/7cPBWdTQw3— Vaibhav (VB) Srivastav (@reach_vb) August 29, 2025
FOSDEM 2022 - LibVF.IO: vGPU & SR-IOV on Consumer GPUs using Nim
I'd like to showcase LibVF.IO's new LIME Runtime feature (Lime Is Mediated Emulation) and do a deep dive on open source vGPU technology in general.
NVlabs/cuda-oxide
cuda-oxide is an experimental Rust-to-CUDA compiler that lets you write (SIMT) GPU kernels in safe(ish), idiomatic Rust. It compiles standard Rust code directly to PTX — no DSLs, no foreign language bindings, just Rust.
