Exploring memory-efficient techniques for LLMs
Understanding AI/LLM Quantisation Through Interactive Visualisations
AI/LLM Quantisation Visualised
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
omlx/docs/oQ_Quantization.md at main · jundot/omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar - jundot/omlx
Solving a Million-Step LLM Task with Zero Errors
LLMs have achieved remarkable breakthroughs in reasoning, insights, and tool use, but chaining these abilities into extended processes at the scale of those routinely executed by humans,...
Understanding Multimodal LLMs
An introduction to the main techniques and latest models
The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities (Version 1.0)
Can LLMs Be Computers? | Percepta
We build a computer inside a transformer — executing arbitrary C programs for millions of steps with exponentially faster inference via 2D attention heads.
Can LLMs Be Computers? | Percepta
We build a computer inside a transformer — executing arbitrary C programs for millions of steps with exponentially faster inference via 2D attention heads.
Reverse-engineering GGUF | Post-Training Quantization
Run LLMs locally on your Mac · mlx-optiq
Quantize, fine-tune and serve LLMs locally on Apple Silicon. MLX-native, no PyTorch, no cloud. On PyPI.
The Kaitchup Index: A Leaderboard for LLMs and Their Quantized Versions
Comparing formats like GGUF, GPTQ, and AWQ, with different bitwidths
Benchmarks | EXO
Transparent benchmarks for LLMs tested on real hardware. Coming soon.
On-Device LLM Leaderboard
Intelligence × decode speed × memory × quantization retention, under real iPhone limits. Same protocol for every model; Apple's built-in FM on the board.
Tethered Reasoning: Decoupling Entropy from Hallucination in Quantized LLMs via Manifold Steering
A fundamental challenge in quantized inference is the temperature-entropy trade-off: low sampling temperatures yield repetitive, mode-collapsed outputs, while high temperatures (T>2.0T{>}2.0) cause what is conventionally termed “hallucination”—semantic incoherence and factual errors. Quantization exacerbates this: 4-bit models exhibit earlier collapse than full-precision counterparts [5, 6]. Most inference frameworks cap temperature at T=2.0T{=}2.0, treating high-entropy regimes as inherently unstable.
On-Device LLM Throughput Calculator - a Hugging Face Space by FL33TW00D-HF
This tool estimates and visualizes the throughput of Large Language Models on devices with memory bandwidth constraints. Users input device and model configurations, and the tool generates a plot s...
i can’t help but think we’re far from nailing memory systems this one here is extremely interesting. two LLMs at once, one just managing and surfacing memory for the other
Asa
I'm not a fan of the decoupled 'memory retrieval → task execution' loop, so my agent has a subconscious background thread that looks for relevant, unique memory context in its experiential database while it runs and injects it on top of the live context window.
