A far more detailed explanation of prompt caching than anyone asked for.
Prompt caching: 10x cheaper LLM tokens, but how? | ngrok blog
A far more detailed explanation of prompt caching than anyone asked for: how tokens, embeddings, and attention make cached LLM tokens 10x cheaper and faster.
Prompt Caching In Agents | EARENDIL
How prompt caching shapes the cost, latency, tools, and architecture of coding agents, and what Pi does to keep cache behavior visible.
How LLMs Actually Work
A from-the-ground-up walkthrough of how modern LLMs work, from tokens to transformer blocks to the next-token loop
Mem0 Research Paper: Token-Efficient Memory Algorithm
Benchmarked across LoCoMo, LongMemEval, and BEAM, achieves competitive accuracy while using under 7,000 tokens per retrieval call. For comparison, full-context approaches on these benchmarks routinely consume 25,000+ tokens per query.
State of AI 2025: 100T Token LLM Usage Study | OpenRouter
Read OpenRouter's 2025 State of AI report — an empirical 100 trillion token study of real LLM usage, model trends, and developer insights.
jundot/omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
Compression is prediction | ngrok blog
Compression and LLMs are trying to solve the exact same problem: predicting what comes next. Learn the fundamentals of compression and how better prediction leads to better shrinkage.
any-llm platform: Cloud Vault and Usage Tracking for LLMs
any-llm managed platform adds end-to-end encrypted API key storage and usage tracking to the any-llm ecosystem. Keys are encrypted client-side, never visible to us, while you monitor token usage, costs, and budgets in one place. Supports OpenAI, Anthropic, Google, and more.
any-llm platform: Cloud Vault and Usage Tracking for LLMs
any-llm managed platform adds end-to-end encrypted API key storage and usage tracking to the any-llm ecosystem. Keys are encrypted client-side, never visible to us, while you monitor token usage, costs, and budgets in one place. Supports OpenAI, Anthropic, Google, and more.
Models & Pricing | DeepSeek API Docs
The prices listed below are in units of per 1M tokens. A token, the smallest unit of text that the model recognizes, can be a word, a number, or even a punctuation mark. We will bill based on the total number of input and output tokens by the model.
Prince Canuma on Twitter / X
Prompt caching with SSD offloading now active and works with all your favourite harnesses: Pi, Hermes, OpenCode, Claude code and more.Here is the performance of Qwen3-VL-4B-Instruct when using our new prompt caching. pic.twitter.com/ikMktUgpqB— Prince Canuma (@Prince_Canuma) May 6, 2026
tokens are getting more expensive
"language models will get cheaper by 10x" will not save ai subscriptions from the short squeeze
vincent on Twitter / X
Session management is working and now it will keep the model in memory instead of constantly loading. Now to fix the hardest part, gibberish issue.ALMOST THERE ~80% DONE 🔨🔨🔨Demo: @UnslothAI Llama 3.2 1B on CPU3.6 tokens/sec to about 4.1 tokens/sec (gibberish) pic.twitter.com/1tp8dXQrLo— vincent (@t0kenl1mit) August 5, 2025
i can’t help but think we’re far from nailing memory systems this one here is extremely interesting. two LLMs at once, one just managing and surfacing memory for the other
Asa
I'm not a fan of the decoupled 'memory retrieval → task execution' loop, so my agent has a subconscious background thread that looks for relevant, unique memory context in its experiential database while it runs and injects it on top of the live context window.
We've recently made Pi's cache behavior more visible. This site has been debating whether agent harnesses are helping or quietly torching their caches. That seemed like a good excuse to explain how KV caches actually work and how Pi helps (or doesn't). earendil.com/posts/prompt-caching/
Prompt Caching In Agents | EARENDIL
earendil.comsomething that has come up fairly recently with LLMs - for coding, specifically - is that it’s become a lot easier to burn stupefying amounts of tokens on stuff very fast, with agents running 24/7 or managing more agents (see: Yegge’s Gas Town) even with low inference costs that adds up in a hurry
Jesse Felder
‘While some cling to the promise of an AI “revolution,” the cost of adoption is proving a stubborn bottleneck. These developments also suggest that the economics of replacing human labor with AI may be more complicated than some early forecasts originally implied.’ fortune.com/2026/05/22/microsoft-ai-cost-…
