How Does Mixture-of-Experts Routing Affect LLM Repeatability?

If you send the same prompt to a mixture-of-experts (MoE) model several times at temperature 0, the answers can still differ. Routing is one possible reason. In each MoE layer, a router sends every token to a few experts, which are smaller feed-forward networks inside the layer. A tiny numerical difference between two runs can make the router choose a different expert for a token. The token is then computed differently, which sometimes changes the rest of the answer.

The routing step itself is not random. Given the same inputs and the same implementation, the router picks the same experts every time. The differences start before the router. When batching, GPU kernels, precision or software versions change, the numbers that reach the router can shift slightly. In some MoE designs, tokens also compete for limited expert capacity, so other requests in the same batch can change where a token goes. Temperature 0 removes sampling randomness, but it does not make these numbers identical between runs.

What the router does

In a dense transformer, every token goes through the same feed-forward network in each layer. An MoE layer has many experts and runs each token through only a few of them, which the Switch Transformer paper describes as selecting different parameters for each input.

In simplified form, an MoE layer does this for each token:

scores = router(h)                  # one score per expert
chosen = top_k(scores, k)           # the k experts with the highest scores
output = sum(gate[e] * expert[e](h) for e in chosen)

Here h is the token's hidden state. The router is a learned projection that turns it into one score per expert. Each chosen expert's output is weighted by its gate value, gate[e], and the weighted outputs are added up. The result goes back into the residual stream, as the output of a dense feed-forward layer would.

In the top-k step, each expert is either used for the token or skipped. Most of the computation changes only slightly when its inputs change slightly, but this decision can flip. Top-k picks the same experts whenever the scores, the tie-breaking rule and the capacity rules are the same.

How a tiny difference becomes a different answer

Suppose a token uses four experts. The expert with the fourth-highest score is the last one that makes the cut, and the fifth is the first one that misses. The gap between these two scores is the router margin. When the margin is large, small changes in the scores leave the route alone. When it is small, a tiny change is enough to push the fifth expert above the fourth.

Swapping one expert for another is a bigger change than a small shift in a weighted sum. Each expert has its own weights, so a different expert adds something different to the token's hidden state. The later layers and the next-token logits then work from a different input. Most floating-point differences never cross a routing boundary, though. A small difference only matters for routing if it changes the order of the scores at the cutoff or the way a tie is broken. With capacity limits, it can also decide whether an expert still has room for the token.

At temperature 0 the model uses greedy decoding and takes the token with the highest logit from the logits it computed in that run. If the run computed slightly different logits, greedy decoding follows them. A fixed seed does not help either. The seed sets the random state for sampling and has no effect on the kernels, the batch or the router's inputs.

The whole chain looks like this:

1. A change in batching, kernels, precision or hardware gives slightly different intermediate values.
2. A later MoE router gets a slightly different hidden state. Its scores cross the top-k cutoff, break a tie differently or run into a different capacity limit.
3. The token goes through a different set of experts, which changes its hidden state and the next-token logits.
4. If the change puts a different token on top, greedy decoding picks that token.
5. The new token becomes part of the context. Every later step starts from a different input, and the two outputs drift further apart.

Not every route change gets that far. A different route can still leave the same token on top, and sometimes the logit difference is too small to change the choice. A different completion is more likely when the top candidates for the next token have close logits. It is also more likely when the route changes early in the answer, or when later layers make the difference larger.

Can other requests in the batch change the output?

Yes, in two ways, depending on the serving setup. The batch affects the numbers, and in some MoE designs it also affects which experts a token gets.

The batch changes the numbers

Serving engines group requests into batches to use the GPU efficiently. The batch size, padding, request order and sequence lengths all affect how the engine runs the math. When they change, the engine may split a matrix multiplication differently, add numbers in a different order or pick a different attention kernel. The same prompt can then produce slightly different hidden states when it runs alone, in a busy batch, or at a different position in the batch. The Thinking Machines Lab analysis describes this batch-invariance problem for RMSNorm, matrix multiplication and attention.

Different hidden states give the router different scores. If the scores near the top-k cutoff are close, the token can get a different expert. If they are far apart, the route stays the same.

Dense models also produce slightly different numbers when batch-dependent kernels change. MoE adds the routing cutoff, where a tiny difference sometimes turns into a different expert.

Tokens compete for expert capacity

Some MoE designs limit how many tokens each expert can take per batch. In the Switch Transformer design, this expert capacity is the number of tokens in the batch divided by the number of experts, times a capacity factor. When more tokens choose an expert than it has room for, the extra tokens overflow. In Switch Transformer, an overflowed token skips the expert and goes straight to the next layer through the residual connection. Other implementations send it to another expert or use a different fallback.

This ties a token's computation to the rest of the batch. Its path depends on its own router scores and on how many other tokens want the same expert. A prompt that reaches expert 12 when it runs alone might overflow when it shares a batch with many tokens that also chose expert 12. If the dispatcher fills capacity in order, the request's position in the batch matters as well.

In the Switch Transformer experiments, usually fewer than 1% of tokens were dropped once the load-balancing loss was strong enough. That figure belongs to that architecture and its settings. Other systems avoid overflow or handle it differently, with dropless dispatch, larger buffers, expert replication or rerouting.

Expert-choice routing

The routing described so far is called token-choice routing, because each token picks its experts. Many decoder-only MoE models use it. Expert-choice routing turns it around. Each expert picks its highest-scoring tokens from all the tokens in the batch. The tokens an expert takes then depend on what else is in the batch, even when a token's own hidden state stays the same.

The paper says the method does not directly apply to autoregressive text generation, because its implementation uses past and future tokens for the top-k selection. It names very small batches during serving as another case where it does not directly apply.

What the published evidence shows

The Qwen3 test from Thinking Machines Lab

Thinking Machines Lab sent the prompt "Tell me about Richard Feynman" to Qwen/Qwen3-235B-A22B-Instruct-2507 1,000 times at temperature 0 and generated 1,000 tokens each time. The runs produced 80 different completions, and the most common one appeared 78 times. All completions were identical for the first 102 tokens. After "Feynman was born on May 11, 1918, in", 992 runs continued with "Queens, New York" and 8 with "New York City". With the lab's batch-invariant kernels turned on, all 1,000 completions were identical.

According to the model card of its FP8 release, the model uses 128 experts and activates 8 of them for each token.

The test does not show how much of the variation came from routing. The write-up published no expert-route traces and did not separate router changes from numerical differences in RMSNorm, matrix multiplication, attention and other operations.

The cost of deterministic execution

The same write-up measured speed on Qwen3-8B, a dense model. One GPU served 1,000 sequences with 90 to 110 output tokens each:

One test does not give a general number. Deterministic or batch-invariant execution may lower throughput and limit how freely the engine schedules requests. Better kernels win back part of the cost.

Router replay

NVIDIA's Megatron Core, a library for training large models, has a router replay feature. Its documentation notes that MoE routing decisions in distributed training can vary slightly because of floating-point precision. Router replay stores the top-k expert indices from one run and uses them again in a later run, so the MoE path is identical across runs. NVIDIA presents it as a way to reproduce experiments and to tell whether a problem comes from the routing or from the expert computations.

Replay only fixes the expert path. Attention, normalization, reductions, communication between GPUs and decoding can still vary. It is a debugging tool and does not make output repeatable on its own.

How to test whether routing is the cause

No broad public benchmark covers these questions yet:

• how often routes flip in major MoE models and serving engines
• how close router scores are in production workloads
• how batch size affects route changes
• how often a route change alters the next token
• how much of the divergence between completions comes from routing

Without route traces or a replay test on your own stack, you cannot tell whether routing is behind the variation you see.

1. Measure the baseline

Send the same request many times at temperature 0. Record how many completions match exactly, how many unique outputs you get and the first token position where the outputs differ. If the stack exposes them, also record per-token logits and expert routes. Repeat the test with the request running alone, at different batch sizes, in different request orders and under realistic load from other requests.

2. Add one control at a time

Compare normal serving with each of these:

• batch-invariant kernels
• deterministic kernels
• fixed batches with a fixed request order
• route replay, if your stack supports it
• route replay combined with deterministic or batch-invariant kernels

The vLLM batch-invariance documentation describes a mode that makes a model's output independent of the batch size and the order of requests, on supported hardware. You turn it on with VLLM_BATCH_INVARIANT=1. The models it lists as tested include MoE models such as DeepSeek-V3 and Qwen3-30B-A3B. vLLM's reproducibility guide adds that vLLM only gives reproducible results on the same hardware and the same vLLM version.

3. Record the routing data

If the serving stack exposes it, log these for every token and every MoE layer:

• the chosen expert IDs, router scores and gate weights
• expert loads, overflow events and any fallback or residual paths
• the next-token logits

Then compute each token's router margin, the gap between its k-th and (k+1)-th highest score. Start with the tokens that have small margins, because a small numerical change is more likely to change their experts.

4. Read the results

Line up the runs by token position and MoE layer, and find the first point where they differ. Look separately at the first route difference, the first logit difference and the first different output token.

Matching text does not prove matching routes. Two runs can take different routes and still pick the same token at every step. Matching routes do not prove that everything else matched either. They only show that the MoE path was the same.

How to get repeatable output from an MoE model

Start by deciding which target you need:

• bitwise-identical intermediate values
• identical output tokens
• identical complete responses
• stable meaning and task quality
• variation within a limit measured under realistic load

Scientific experiments, regression tests and reinforcement-learning rollouts often require token-exact output, which takes the strictest controls. For many production systems, documented limits on variation and stable task quality are enough.

For any repeatability test, keep these fixed and record them:

• temperature, top-p, top-k, repetition penalties, maximum tokens, stop rules and the seed
• the prompt, retrieved documents, tool results, timestamps and any other outside context
• the model identifier and revision, the weight hash, the tokenizer and the quantization format
• the inference engine and its version, the GPU and driver, the accelerator runtime, and the tensor-parallel and expert-parallel setup

A runtime update or a quantization change can move router scores even when the prompt stays the same.

For token-exact tests, limit dynamic batching and use deterministic or batch-invariant kernels where the engine supports them. Test with fixed batches and again with varied, production-like batches. Fixed batches show whether the setup is reproducible under controlled conditions. Varied batches show whether that holds up in normal serving.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com