morgan — on Twitter / X

Muse Glimmer: Meta’s 30B Model Built for Efficient Inference

Inside Meta’s 30B local reasoning model and its tiny KV cache

Planetary-Scale Inference: Building a Distributed Inference Engine for the Public Internet

We are excited to share a preview of our distributed inference stack — engineered for consumer GPUs and the 100ms latencies of the public internet—plus a research roadmap that scales it into a planetary-scale inference engine.

OpenAI's open source LLM is a reasoning model, coming Next Thursday!

1.1K votes, 257 comments. 756K subscribers in the LocalLLaMA community. Subreddit to discuss locally hostable AI.

docs/models.md at main

docs - Public documentation for the Inference Cooperative

Hatice Ozen on Twitter / X

PSA: @OpenAI is putting the Open back in OpenAI and @GroqInc has Day 0 support. 🤗GPT-OSS 20B and 120B, hybrid-reasoning models with built-in browser search and code execution are now live for instant inference.P.S. We've also launched OpenAI Responses API compatibility. pic.twitter.com/CK7StvMSpr— Hatice Ozen (@ozenhati) August 5, 2025

Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

Large language model (LLM) queries are predominantly processed by frontier models in centralized cloud infrastructure. Demand growth strains this paradigm faster than providers can scale. Two advances create an opportunity to rethink it: small, local LMs (<=20B active parameters) now achieve competitive performance to frontier models on many tasks, and local accelerators (e.g., Apple M4 Max) can host these models at interactive latencies. This raises the question: can local inference viably redistribute demand from centralized infrastructure? This requires measuring both whether local LMs can accurately answer real-world queries and whether they can do so efficiently on power-constrained devices (e.g., laptops). We propose intelligence per watt (IPW), task accuracy per unit of power, as a unified metric for the capability and efficiency of local inference across model-accelerator configurations. We evaluate 20+ state-of-the-art local LMs, 8 hardware accelerators (local and cloud), and 1M real-world single-turn chat and reasoning queries. For each query, we measure accuracy (local LM win rate against frontier models), energy, latency, and power. We find three key results. First, local LMs successfully answer 88.7% of these queries, with accuracy varying by domain. Second, longitudinal analysis from 2023-2025 shows IPW improved 5.3x, driven by both algorithmic and accelerator advances, with locally-serviceable query coverage rising from 23.2% to 71.3%. Third, local accelerators achieve at least 1.4x lower IPW than cloud accelerators running identical models, revealing significant headroom for local accelerator optimization. These findings demonstrate that local inference can meaningfully redistribute demand from centralized infrastructure for a substantial subset of queries, with IPW serving as the critical metric for tracking this transition.

No, local models will not win

Every time a new open-weight AI model is released, people say that local models are the future. Why spend billions of dollars building out datacenters when everyone will just be able to run AI models on their laptops or phones? I think this idea is doomed. No matter how strong open-weight models get, most inference will always happen in AI datacenters.

Cheng on Twitter / X

We have been expecting this since ollama's first pull request to MLX. It is just the beginning, CUDA & CPU backends are still improving and hopefully we will have one framework unifying inference & training for all platforms. https://t.co/EaBmEaNJhZ— Cheng (@zcbenz) March 31, 2026

Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio

MLX on Mac with Apple silicon lets AI researchers and developers experiment locally with the latest models and methods in a private…

Open Models Inference for Coding · Umans AI

Hosted Kimi K3, GLM 5.2, and DeepSeek V4 Flash. Pay per token, on infrastructure we own.

Alex Cheema on Twitter / X

This is why we need open benchmarks for local AI.Otherwise it turns into tribalism and name calling.We will be publishing the largest database of open benchmarks for local AI, tested on 1,000+ real hardware setups. Every device, every interconnect, different… https://t.co/ZsU3PCdSsZ— Alex Cheema (@alexocheema) March 9, 2026

Liquid AI on Twitter / X

Today, we announce 𝙻𝚒𝚚𝚞𝚒𝚍 𝚁𝙸𝙿𝙻 (real-time inference pipeline), our ultra low-latency inference engine built to put LFMs directly in the request path of real-time workloads like search, ads, fraud, recommendations, and payments.> complete generated responses end-to-end… pic.twitter.com/m8zzApzOCT— Liquid AI (@liquidai) October 22, 2026

Gajesh on Twitter / X

TL;DRapple has turn on this switch for everyone to participate in decentralized inferenceppl can rent out their unused compute space and anyone can use this with privacy guarantees https://t.co/LTP4zyjsdt pic.twitter.com/8Dvo7XK8jJ— Gajesh (@gajesh) February 18, 2026

Public AI Inference Utility

A nonprofit, open-source service to make public and sovereign AI models more accessible.

View @Jefflau's post on Firefly

I think what's interesting is that the desire for local AI may inadvertently create a lot more local node users. It's a fraction of the actual compute local AI needs and just requires a decent amount of disk space.

Local-First AI Inference: A Cloud Architecture Pattern for Cost-Effective Document Processing

The Local-First AI Inference pattern routes 70–80% of documents to deterministic local extraction at zero API cost, reserving Azure OpenAI calls for edge cases and flagging low-confidence results for human review. Deployed on 4,700 engineering drawing PDFs, it cut API costs by 75% and processing time by 55%, while bounding errors through a human review tier.