Reproducibility is a bedrock of scientific progress. However, it’s remarkably difficult to get reproducible results out of large language models. For example, you might observe that asking ChatGPT the same question multiple times provides different results. This by itself is not surprising, since getting a result from a language model involves “sampling”, a process that converts the language model’s output into a probability distribution and probabilistically selects a token. What might be more surprising is that even when we adjust the temperature down to 0This means that the LLM always chooses the highest probability token, which is called greedy sampling. (thus making the sampling theoretically deterministic), LLM APIs are still not deterministic in practice (see past discussions here, here, or here). Even when running inference on your own hardware with an OSS inference library like vLLM or SGLang, sampling still isn’t deterministic (see here or here).
The Myth of Deterministic Software
How I've come to terms with LLM non-determinism in software and abandoned the comforting lies we've been telling ourselves about traditional software.
LLM in a Flash: Efficient Large Language Model Inference with Limited Memory
Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks…
chad/whichlang
What programming language do LLMs default to when you don't tell them? A small benchmark.
Tethered Reasoning: Decoupling Entropy from Hallucination in Quantized LLMs via Manifold Steering
A fundamental challenge in quantized inference is the temperature-entropy trade-off: low sampling temperatures yield repetitive, mode-collapsed outputs, while high temperatures (T>2.0T{>}2.0) cause what is conventionally termed “hallucination”—semantic incoherence and factual errors. Quantization exacerbates this: 4-bit models exhibit earlier collapse than full-precision counterparts [5, 6]. Most inference frameworks cap temperature at T=2.0T{=}2.0, treating high-entropy regimes as inherently unstable.
Demystifying llm-d and vLLM: The race to production
Learn how vLLM and llm-d work together for efficient and scalable large language model (LLM) inference. Discover the benefits of disaggregated scaling, expert-parallel scheduling, and KV cache-aware routing.
Take caution in using LLMs as human surrogates | PNAS
Recent studies suggest large language models (LLMs) can generate human-like responses, aligning with human behavior in economic experiments, survey...
Forcing Generative Models to Degenerate Ones: The Power of Data...
Growing applications of large language models (LLMs) trained by a third party raise serious concerns on the security vulnerability of LLMs.It has been demonstrated that malicious actors can...
Why Are LLMs Smart?
A popular way to explain how current LLMs work is to say that “all” they do is predict the next most likely word in a sentence.
LLMs and World Models, Part 1
How do Large Language Models Make Sense of Their “Worlds”?
Large language models are not the problem
If a Large Language Model (LLM) can replicate your scientific contribution, the problem is not the LLM. What does it say about our field that so much of the anxiety about AI comes down to the fear that a machine could do what we do? Perhaps it says we should be doing something better.
Dan Shipper 📧 on Twitter / X
this is true and is a big reason why you don’t need to be a highly technical researcher to use LLMs in surprising and novel ways https://t.co/TuxNzXzToU— Dan Shipper 📧 (@danshipper) July 27, 2025
Google DeepMind Paper Argues LLMs Will Never Be Conscious
Philosophers said the paper’s argument is sound, but that “all these arguments have been presented years and years ago.”
