Luce-Org/lucebox-hub

tanishqkumar/ssd

A lightweight inference engine supporting speculative speculative decoding (SSD).

Benchmarks | EXO

Transparent benchmarks for LLMs tested on real hardware. Coming soon.

jundot/omlx

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

predibase/lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

omlx/docs/experimental/dflash_mlx_integration.md at main · jundot/omlx

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar - jundot/omlx

leobeeson/llm_benchmarks

A collection of benchmarks and datasets for evaluating LLM.

LM Studio 0.3.10: 🔮 Speculative Decoding

Inference speed up with Speculative Decoding for `llama.cpp` and `MLX`

mzau/broke-cluster

A Poor Man's Apple Silicon LLM Cluster — tuned for MLX, scalable without shame.

Demystifying llm-d and vLLM: The race to production

Learn how vLLM and llm-d work together for efficient and scalable large language model (LLM) inference. Discover the benefits of disaggregated scaling, expert-parallel scheduling, and KV cache-aware routing.

omlx/docs/oQ_Quantization.md at main · jundot/omlx

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar - jundot/omlx

apple-silicon-llm-bench/results/complete_results.html at main · AlexHiesch/apple-silicon-llm-bench

Systematic LLM inference benchmark for Apple Silicon: 8 backends, 7 models, 791 measurements - AlexHiesch/apple-silicon-llm-bench

turbopuffer: fast search on object storage

Inaugural blog post about the development of turbopuffer, a search engine that uses object storage and SSD caching for cost-effective, low latency search. This post describes into the motivation behind its creation, its unique architecture, and how it significantly reduces costs for large-scale vector searches. Discover how turbopuffer is transforming search infrastructure for companies like Cursor and Suno, offering a scalable and reliable solution.

Overview - GroqDocs

Fast LLM inference, OpenAI-compatible. Simple to integrate, easy to scale. Start building in minutes.

State of AI 2025: 100T Token LLM Usage Study | OpenRouter

Read OpenRouter's 2025 State of AI report — an empirical 100 trillion token study of real LLM usage, model trends, and developer insights.

mem-agent: Equipping LLM Agents with Memory Using RL

The insights and the technical report behind Mem-Agent, our 4B model for persistent memory in LLMs

EricLBuehler/mistral.rs

Fast, flexible LLM inference