A far more detailed explanation of prompt caching than anyone asked for: how tokens, embeddings, and attention make cached LLM tokens 10x cheaper and faster.
Prompt caching: 10x cheaper LLM tokens, but how? | ngrok blog
A far more detailed explanation of prompt caching than anyone asked for.
Prompt Caching In Agents | EARENDIL
How prompt caching shapes the cost, latency, tools, and architecture of coding agents, and what Pi does to keep cache behavior visible.
Tokenization Tax 2026: LLM Tokenizer Comparison | ellamind
A measurement study of 20 LLM tokenizers across 12 languages and 21 kinds of text. On German, Claude 4.7+ needs 2.01x the tokens of the OpenAI reference, and Cohere Command A+ the fewest.
Mem0 Research Paper: Token-Efficient Memory Algorithm
Benchmarked across LoCoMo, LongMemEval, and BEAM, achieves competitive accuracy while using under 7,000 tokens per retrieval call. For comparison, full-context approaches on these benchmarks routinely consume 25,000+ tokens per query.
How LLMs Actually Work
A from-the-ground-up walkthrough of how modern LLMs work, from tokens to transformer blocks to the next-token loop
State of AI 2025: 100T Token LLM Usage Study | OpenRouter
Read OpenRouter's 2025 State of AI report — an empirical 100 trillion token study of real LLM usage, model trends, and developer insights.
Compression is prediction | ngrok blog
Compression and LLMs are trying to solve the exact same problem: predicting what comes next. Learn the fundamentals of compression and how better prediction leads to better shrinkage.
Sliding-window beats linear attention
Due to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy. Every new token costs more than the previous one. For each additional token, the keys and values must be stored in memory indefinitely, which is unsustainable.
tokens are getting more expensive
"language models will get cheaper by 10x" will not save ai subscriptions from the short squeeze
elie on Twitter / X
nice pre training work by nous claiming ~2.5x efficiency gains, building on previous research like MTP/SuperBPE. overall intuition is that at each step you want the model to process and predict more tokens https://t.co/K33QtJiF5C pic.twitter.com/hDlSbv1Tg9— elie (@eliebakouch) May 13, 2026
The Ads Model for Prompts Vertically Integrates AI
Meta's dual-tier AI pricing introduces an explicit barter : a 92% discount on inference in exchange for your data. Applying Michael Mauboussin's insight that prices contain information reveals the clearing price of prompt data : $1.24 per million tokens.
i can’t help but think we’re far from nailing memory systems this one here is extremely interesting. two LLMs at once, one just managing and surfacing memory for the other
Asa ⭔
I'm not a fan of the decoupled 'memory retrieval → task execution' loop, so my agent has a subconscious background thread that looks for relevant, unique memory context in its experiential database while it runs and injects it on top of the live context window.
We've recently made Pi's cache behavior more visible. This site has been debating whether agent harnesses are helping or quietly torching their caches. That seemed like a good excuse to explain how KV caches actually work and how Pi helps (or doesn't). earendil.com/posts/prompt-caching/
Prompt Caching In Agents | EARENDIL
earendil.comsomething that has come up fairly recently with LLMs - for coding, specifically - is that it’s become a lot easier to burn stupefying amounts of tokens on stuff very fast, with agents running 24/7 or managing more agents (see: Yegge’s Gas Town) even with low inference costs that adds up in a hurry
Jesse Felder
‘While some cling to the promise of an AI “revolution,” the cost of adoption is proving a stubborn bottleneck. These developments also suggest that the economics of replacing human labor with AI may be more complicated than some early forecasts originally implied.’ fortune.com/2026/05/22/microsoft-ai-cost-…
