Does Temperature 0 Guarantee Deterministic LLM Outputs?

Large Language Models (LLMs) are often used in applications where consistency matters. Developers frequently assume that setting the temperature to zero will make the model's output deterministic (always the same for a given prompt). In practice, many have noticed slight variations in outputs even with temperature=0. Why does this happen? This article breaks down the reasons and provides guidance on how to maximize consistency in LLM responses.

Temperature and Determinism

Temperature is a parameter that controls the randomness of an LLM's token selection. Higher temperature values produce more varied (creative) outputs, while lower values make the model focus on more likely predictions. At temperature = 0, the model performs greedy decoding: it always picks the most probable next token at each step, essentially the argmax choice from the predicted probability distribution. In theory, greedy decoding should eliminate the intended randomness in generation. If the model and input are fixed, one would expect the same sequence of tokens every time.

However, setting temperature to zero does not guarantee 100% determinism in practice. There are a few reasons for this:

• Ties or equal probabilities: If two or more next-token options have (practically) the same highest probability, the model or decoding library might break ties in an arbitrary way. This situation is rare but possible. In such a case, even with temperature 0, the choice between tokens could be nondeterministic.

• Upstream variability: Randomness can creep in before the greedy selection happens. Ideally, the model's predicted probabilities for the next token are a pure, deterministic function of the input and the model weights. In reality, how those probabilities are computed can introduce slight variability run-to-run (as we'll explore below). If something causes the model's computed probabilities to differ ever so slightly between runs, the identity of the "most probable token" could change. Essentially, temperature=0 removes deliberate randomness from sampling, but it doesn't account for subtle randomness in the model's computation itself.

In short: temperature=0 greatly increases determinism but is not an absolute guarantee of identical outputs on every run. To understand why, we need to look at how modern LLM architectures and hardware behavior can introduce variability.

Mixture-of-Experts (MoE) and Routing Effects

One big factor influencing determinism is the model's architecture. Many cutting-edge LLMs (including those behind some popular APIs) use a Mixture-of-Experts (MoE) architecture. In an MoE model, the network is divided into multiple "expert" sub-models rather than one monolithic model. A gating mechanism dynamically routes each input token (or segment of the input) to one or a few of these expert networks. This design allows models to scale up effectively — different tokens can be handled by different experts specialized in different things.

However, MoE models can introduce nondeterminism in a way that dense (single-expert) models do not. The key issue is how tokens are routed to experts during inference. The routing is typically based on the content of the token (the model chooses the experts with the highest computed affinity for that token). If each token simply went to its top expert independently, the process would be deterministic for a given input. In practice, capacity limits and batch processing come into play:

• Batch competition: In a production setting, LLM services often perform batched inference – processing multiple user requests or multiple tokens together for efficiency. For MoE, tokens are usually routed in groups (batches) and there's a limit on how many tokens each expert can handle at once. If too many tokens in a batch want to go to the same expert, some of those tokens have to be routed to a secondary choice or "dropped" (not processed by that expert layer).

• Sequence-level vs batch-level determinism: The consequence of the above is that an MoE model might be deterministic when the same batch of inputs is processed together, but not deterministic for an individual input sequence in isolation. In other words, if you send the exact same prompt twice, but behind the scenes it gets grouped with different other user prompts each time, the routing decisions for your prompt's tokens could differ. The model is then deterministic at the batch level but not at the single-request level. Your prompt's output may vary slightly depending on which other queries happened to be processed alongside it.

Consider GPT-4 as an example (which is rumored to use MoE internally). Users observed that even at temperature 0, the same prompt could yield different continuations on different tries. A likely explanation is that GPT-4's backend packs multiple queries together for efficiency, and tokens from different users might contend for the same expert. If in one run your prompt's token got the primary expert and in another run it got a backup expert (due to a busy primary), the resulting token output could diverge. In effect, MoE routing can create a race condition – your tokens might "race" against others for expert capacity. The model's output can thus vary across API calls, even though no traditional random sampling is happening.

It's worth noting that not all LLMs use MoE, but those that do must carefully design their routing to avoid this issue. Research proposals (like "soft" MoE variants) aim to make MoE routing more flexible or deterministic, but in many current systems the nondeterminism from MoE is a known trade-off for scalability. For an API user, this is mostly invisible – except for the puzzling output inconsistencies it can produce.

Hardware-Level Factors and Floating-Point Precision

Even if an LLM model is a standard (dense) transformer without any MoE tricks, you may still see nondeterministic behavior at temperature 0. The culprit here is the underlying hardware and parallel computation.

Modern neural networks are typically executed on GPUs (or TPUs) for speed. These hardware accelerators perform massive parallel math computations, which can introduce subtle nondeterminism due to the nature of floating-point arithmetic:

• Floating-point rounding errors: Neural network calculations involve a lot of floating-point numbers. These have limited precision, meaning operations on them can accumulate tiny rounding errors. Importantly, floating-point addition is not associative – the order in which additions are performed can change the final result slightly. On a GPU, when summing up thousands of values (for example, accumulating the scores from many neurons to compute a logit), the addition might be split across many threads and combined in an arbitrary order. The final summed value might differ by a tiny fraction from one run to another depending on the timing/order of operations. This difference is minuscule (many decimal places out), but if two token probabilities are extremely close, that tiny change can swap their rank order (which one is the max). Thus, a different token may be chosen as the argmax on different runs purely due to numerical rounding differences.

• Parallelism and race conditions: Aside from summation order, any non-deterministic GPU kernel or multi-threading race condition can cause divergence. Deep learning frameworks often prioritize performance, sometimes at the expense of bit-for-bit reproducibility. Unless explicitly using deterministic algorithms, operations like matrix multiplication, convolution, or reduction can have nondeterministic implementations. For example, using atomic operations or multi-pass algorithms on GPU can yield slightly different results run-to-run. Most of these differences are so small that they don't matter for final accuracy or quality – but they can matter for exact reproducibility.

• Precision and hardware variation: The precision used (FP32, FP16, BF16, etc.) also affects reproducibility. Lower precision (fp16/bfloat16) has more rounding error and thus more potential variability than 32-bit floats. If an LLM service switches hardware or uses a mix of hardware (say, different GPU models or a distributed setup), the floating-point behavior might not be identical on each. This means the same model and input might produce slightly different outputs depending on which hardware your request ran on.

In summary, even a conceptually deterministic model (no sampling, no MoE) can behave non-deterministically when deployed on parallel hardware. The model's output probabilities are deterministic in math theory, but not always exactly repeatable in practice due to these implementation details. One OpenAI engineer noted that their models do not output identical token scores between runs for the same input, which makes true per-request determinism elusive.

Ensuring Consistent Outputs in API Calls

If you require consistent outputs from an LLM API (for example, for unit tests or reliable user experience), what can you do? While you cannot force absolute determinism in most public APIs, you can take steps to maximize consistency:

• Use deterministic decoding settings: Always set temperature=0 (greedy decoding) for consistency. Likewise, avoid stochastic sampling parameters like nucleus sampling (top_p) or top-k sampling. For example, ensure top_p=1 (consider the full distribution) unless the API documentation specifies that top_p is ignored at temp=0. Greedy decoding means the model won't intentionally inject randomness. This is the first and most important step.

• Check for a seed parameter: Some providers have introduced a seed parameter to help reproducibility. For instance, OpenAI's API (for certain models) allows a seed value in the request to attempt deterministic sampling. In theory, this fixes the sequence of pseudorandom numbers used in token selection, so if you did use a non-zero temperature, you could reproduce that same random choice. However, note that even with a seed, the underlying model must produce the same token probabilities each time for it to work. As discussed, current models sometimes don't – OpenAI's documentation admits that results are only "mostly" deterministic with a fixed seed. In fact, users have found that even with seed set and temperature=0, outputs can still vary. The seed can control intentional randomness, but it can't solve MoE routing or floating-point quirks.

• Avoid prompt variations: This may sound obvious, but ensure you send exactly the same prompt (including punctuation, casing, and formatting) if you expect the same output. Even a minor difference in input will lead to a different response. Sometimes what looks identical (say, an extra space or hidden character) can cause divergence. Double-check that your prompts are truly consistent across calls.

• Request single outputs: If an API allows requesting multiple completions in one call (e.g., n=5 to get 5 responses), avoid this when you want determinism. Typically, with temperature=0 all n outputs would end up identical (since greedy picks the same tokens each time) – which is wasteful anyway. In some cases, however, generating multiple outputs might internally activate some diversity mechanism. It's safest to request one output at a time for reproducibility.

• Understand the service's limits: Some providers openly acknowledge the nondeterminism. Anthropic, for example, notes in their documentation that even with temperature 0.0, the results will not be fully deterministic. Knowing this, you might decide that for critical deterministic behavior (say, in a unit test for your application), an external API may not be the best choice. If possible, you could use a smaller local model for testing purposes where you can control the environment more tightly.

• Control the environment (if self-hosting): If you are running an open-source LLM on your own hardware, you have more control. You can set random seeds at the framework level (e.g., PyTorch or TensorFlow) to eliminate randomness in sampling. You can also enable deterministic mode in these libraries (for example, turning off certain non-deterministic GPU ops, or running on CPU for exact reproducibility if performance allows). This way, a given prompt should produce the same result every time on your machine. Keep in mind you may still need to fix the model version and ensure identical hardware each run, as changes there can also alter outputs slightly.

Finally, it's wise to design your system to tolerate minor variations if possible. If you are using LLM output in an automated pipeline, consider that two outputs might be semantically the same even if not textually identical. For instance, if you ask an LLM to extract a date from text and it sometimes responds "The date is March 5, 2021." and other times "March 5, 2021.", those are effectively equivalent. Instead of string-matching the whole output, you might post-process the result (e.g., use a regex to find the date). By focusing on the content rather than exact wording, you make your application robust to the small fluctuations that can occur.

Model Differences and Determinism

Not all LLMs are built the same, and this affects determinism:

• Dense vs. MoE architectures: As discussed, a dense transformer model (one big set of weights, like older GPT-3 models or Meta's LLaMA series) doesn't have the routing issues of MoE. In principle, a purely dense model running on the same hardware with the same code should give identical results for identical input (aside from floating-point rounding issues). In contrast, Mixture-of-Experts models add another layer of complexity – they can behave nondeterministically if the implementation doesn't guarantee the same expert assignments every time. GPT-4 and some other very large models are believed to use MoE to achieve their scale, which is why users see more unpredictability at T=0 compared to smaller models.

• Different approaches to MoE: Even among MoE models, design choices matter. Some research models enforce deterministic routing per sequence or use "soft" expert assignments (averaging experts' outputs) which could reduce variability. Others, like the Switch Transformers or certain MoE variants, have hard caps and random tie-breaking which increase variability. It's possible to build an MoE such that it's batch deterministic (always the same output given the same batch of inputs), which is fine when you control the batches entirely. But for a public API serving many users, batch composition is variable, leading to sequence-level nondeterminism. In short, the way MoE is implemented (soft vs hard routing, capacity decisions, etc.) will influence how deterministic the model's outputs are.

• Model size and hardware parallelism: Very large models that require model-parallelism (splitting across multiple GPUs or machines) may introduce more nondeterminism than smaller models that run on a single device. Splitting a model means merging results from different shards, often in parallel, which can amplify floating-point differences. Smaller models (or distilled versions) that run in one go might avoid some of those pitfalls. For example, an open-source 7B parameter model running on one GPU could be more stable in output than a 175B model spread over dozens of GPUs, simply because there are fewer moving pieces in the computation.

• Provider-specific handling: Different API providers use different models and infrastructure. OpenAI's GPT-3.5 Turbo and GPT-4, Anthropic's Claude, Google's PaLM/Bison, and DeepSeek's models each have their own stack. Some might use MoE, others might not. Some might perform aggressive optimizations or request batching, while others might run more straightforwardly. This means the likelihood of nondeterministic output can vary. For instance, if DeepSeek's model is a single expert (dense) and they process each request independently, you might rarely see variation at temp 0 (aside from rare tie cases or hardware bits). On the other hand, if Anthropic's Claude uses a similar strategy as OpenAI's (with possible MoE and batching), it will exhibit similar nondeterminism. In any case, no major provider currently promises fully deterministic outputs for their generative models. It's generally understood as a limitation of today's LLM technology.

Key Takeaways

Setting temperature=0 is the correct approach to minimize randomness in LLM outputs – it ensures the model picks the highest-probability completion at each step. But as we've explored, this alone doesn't guarantee identical results across runs. The complexities of Mixture-of-Experts architectures, the quirks of floating-point arithmetic on parallel hardware, and other implementation details mean that two calls to the same model with the same prompt can occasionally diverge even without any "randomness" parameter.

For developers, the key takeaways are:

• Don't assume perfect reproducibility. If you see slight differences at temperature 0, it's not your imagination – it's a known behavior.
• Design for resilience. If exact repeatable output is critical, consider alternatives or workarounds, and at minimum use the most deterministic settings available.
• Stay informed. Keep an eye on provider documentation for features like seed parameters or future options that might improve determinism. Likewise, be aware of model updates or changes in the backend that could affect output consistency.

In the end, deterministic LLM outputs remain a bit of a moving target. As LLMs and their deployment infrastructure evolve, we may get closer to reproducible behavior (or at least tools to control variation). Until then, understanding why temperature=0 doesn't always give the same answer empowers you to make better decisions in using LLMs – and to build applications that handle the wonderfully complex nature of these AI models.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com