What Is Batch Invariance in LLM Inference?

Batch invariance means a request produces the same inference result when the server runs it alone, alongside other requests, at a different batch position, or under another supported batching schedule. It matters because large language model (LLM) servers group requests dynamically to use GPUs efficiently, and that grouping can change the floating-point calculations used to produce the next token.

With a declared hardware and software environment, batch-invariant inference removes batch size, batch composition, request order, and scheduling as sources of output drift. It does not guarantee identical results across every GPU, CUDA version, precision, model revision, or sampling configuration.

Define Batch Invariance

Here, x is one logical request, including its prompt, model, decoding settings, and request-local state. A batch B contains x, and E is a fixed execution environment.

A system is batch-invariant for x when:

F(x; B_1, E) = F(x; B_2, E)

for supported batches B_1 and B_2 that contain the same request. The batches may differ in size, companion requests, request order, or scheduling.

At minimum, the controlled environment must include:

• Model revision and model weights
• Tokenizer and chat template
• Prompt and decoding configuration
• Random state, when sampling
• GPU model and topology
• Serving framework and software versions
• Precision and quantization
• Parallelism configuration
• Supported execution mode, such as prefill or decode

A strong form of batch invariance preserves intermediate values or logits: the numerical outputs before token selection remain identical. An application-level form preserves the generated token IDs or final text. The second guarantee is weaker because different intermediate values can still produce the same selected tokens.

Batch invariance concerns numerical behavior. It does not mean that batch size leaves throughput, latency, memory use, or GPU utilization unchanged. It also does not promise equal treatment between requests, which is a scheduling and fairness question.

The vLLM batch-invariance documentation describes the feature operationally: a request should produce outputs independent of batch size and request order under supported conditions.

Separate Invariance From Determinism

Batch invariance is narrower than full determinism.

A random seed controls the pseudorandom sequence used during sampling. It does not force the server to calculate identical logits if a different batch shape changes the arithmetic path before sampling. Reproducible sampled generation therefore requires both a controlled random-state policy and stable numerical execution.

Batch effects do not require other requests to share prompt data with the target request. Their presence can change the total tensor shape, kernel selection, reduction partitioning, or execution schedule. The target request still uses its own logical input, but the GPU may perform the arithmetic differently.

Batch invariance is also narrower than cross-platform reproducibility. A guarantee that covers batch composition on one GPU configuration does not automatically cover:

• A different GPU architecture
• A different GPU count
• A different tensor-parallel size
• A different CUDA or library version
• A different precision or quantization method
• A different model or tokenizer revision

The term is unrelated to Batch Normalization, a training-time neural-network layer. Batch invariance concerns how inference kernels calculate a request when the serving batch changes.

Trace the Numerical Cause

Dynamic batching is the starting point. In continuous batching, requests join or leave decoding steps as other requests finish. The active batch can therefore change while a request is generating tokens.

A change in batch shape can cause the runtime to select different:

• Matrix-multiplication tile layouts
• GPU block sizes
• Split-K or non-Split-K reduction strategies
• Attention partitions
• Workspace sizes
• Communication or collective-operation paths.

The model weights and prompt remain unchanged, but the numerical execution path does not.

The underlying issue is finite-precision arithmetic. Floating-point addition is not associative after rounding:

(a+b)+c \ne a+(b+c)

Adding the same values in different orders during a reduction can therefore produce slightly different results. Normalization, matrix multiplication, attention, softmax-related calculations, and distributed communication all contain reductions that are sensitive to this effect.

A simple mental model is:

Batching changes the route used to perform the arithmetic, not the question given to the model.

In exact arithmetic, those routes would produce the same result. In finite precision, they can produce a small difference.

Explain Logit Drift

A logit is the model’s numerical score for a possible next token before those scores become probabilities or a token is selected. Most small logit changes do not alter the selected token because the highest-scoring candidate remains highest.

A near tie between two candidate tokens increases the risk. Under temperature-zero, or greedy, decoding, the server selects the highest-logit token. A small batch-induced change can reverse the order of a near tie and select a different token.

With nonzero-temperature sampling, changed logits alter token probabilities. Sampling then adds a separate source of variation unless the random state and sampling implementation are controlled.

Once one token changes, autoregressive generation follows a new path. The newly selected token changes the context and key-value cache, which stores previous attention information for faster decoding, so every later token distribution is computed from that changed state; even a small numerical difference at one position can produce a visibly different completion or reasoning trace.

Identify the Production Problems

Batch invariance matters wherever engineers need to reproduce a result independently of changing traffic.

Evaluation and benchmarks

A benchmark run can change when the evaluator changes batch size, GPU count, GPU type, precision, or serving framework. The NeurIPS 2025 study on numerical nondeterminism reported up to 9 percentage points of accuracy variation and a 9,000-token difference in response length for DeepSeek-R1-Distill-Qwen-7B across tested runtime configurations, including batch and hardware changes.

Those findings do not establish that every model or hosted API behaves this way. They do show that evaluation configuration can affect results materially, rather than only changing low-order bits.

For a meaningful comparison, record the batch size and scheduling mode along with the model, tokenizer, precision, GPU configuration, and software versions. A single greedy run with a fixed seed is not enough documentation.

Long reasoning generations face greater exposure because they contain more token decisions. Each additional decision provides another opportunity for a small numerical difference to branch the sequence.

Debugging and regression tests

A request that passes when run alone but fails under production load is difficult to replay if batch composition is part of the hidden input. Batch invariance helps these tests depend less on unrelated traffic.

It supports:

• Golden-output tests
• Token-level regression tests
• Model-upgrade comparisons
• Cached-result validation
• Replay of incidents
• Evaluation audits

Compare token IDs before comparing rendered text. Text formatting can hide or introduce differences unrelated to model numerics.

Reinforcement learning and agents

Reinforcement-learning rollouts depend on the policy that generates tokens. If rollout inference uses a different numerical path from the scoring or training path, the generated data can differ from the behavior expected by the training procedure.

This concern also applies to agentic systems. An early token difference can alter a tool call, selected action, or reasoning branch. The tensor-parallel invariance preprint separates batch-related variation from variation caused by changing tensor-parallel sizes, which is a useful distinction when training and rollout infrastructure use different distributed configurations.

Load-dependent behavior

Suppose a prompt produces one answer when run alone and another when placed beside requests with longer sequences. The difference alone does not indicate prompt contamination or data leakage. Dynamic batching may have changed the execution shape and, therefore, the floating-point reduction order.

Batch invariance guarantees behavioral isolation: unrelated traffic should not change a request merely by changing how the server groups work. Investigate kernels, scheduling, precision, and distributed execution before treating the difference as evidence that another request’s content entered the target request.

Design Batch-Invariant Kernels

A batch-invariant implementation keeps each request on a stable numerical path even when the enclosing batch changes. The direct method is to fix per-request reduction orders and avoid shape-dependent kernel choices that change those reductions.

That design can disable optimizations such as adaptive Split-K execution or custom collective paths. The supported scope must be explicit because a kernel can be invariant across batch composition while remaining sensitive to hardware, parallelism, or software changes.

Stabilize normalization reductions

RMSNorm, a normalization method that scales a hidden-state vector using the root mean square of its values, reduces many values across a hidden dimension.

A small-batch optimization might split one request’s reduction across multiple GPU cores, while a larger batch uses a different partition. The result can differ slightly because the additions occur in a different order.

A batch-invariant RMSNorm kernel preserves the reduction order for each logical row regardless of the surrounding batch size. The goal is stable arithmetic, not merely a stable scheduler decision.

Fix matrix multiplication paths

Matrix multiplication computes many dot products, and each dot product is a reduction. Different batch shapes can select different tile layouts, Tensor Core instructions, or Split-K strategies.

A batch-invariant matrix-multiplication path uses the same reduction scheme for each logical output element across supported shapes. This often means giving up some shape-specific optimization. The performance cost depends on the model, GPU, precision, and workload, so no single percentage applies universally.

Control attention and caches

Attention combines a query with keys and values across sequence positions. Prefill, decoding, chunked processing, prefix caching, and split-key-value strategies can use different partitions for this work.

For each token, a stable attention implementation keeps the reduction order independent of neighboring requests and chunking arrangements. Key-value cache handling must also preserve the same logical partitioning when requests enter or leave a continuous batch.

Control distributed reductions

Tensor parallelism splits model computation across GPUs. All-reduce order, GPU topology, NCCL behavior, custom collectives, and tensor-parallel size can each introduce numerical variation.

A batch-invariant mode therefore does not automatically provide tensor-parallel invariance. The vLLM implementation notes describe deterministic kernels and the disabling of some optimizations that can introduce variation, including certain custom all-reduce paths. That behavior remains limited to the documented hardware, models, and framework version.

Define the Determinism Boundary

Run-to-run determinism and batch invariance answer different questions.

• Run-to-run determinism: Does a fixed execution configuration produce the same result on every run?
• Batch invariance: Does the same request produce the same result when batch size, batch composition, order, or scheduling changes within the supported environment?

For one fixed shape, a kernel can repeat bit-for-bit yet select a different implementation when the shape changes. Workspace conditions, stream usage, toolkit version, and library heuristics can all affect that choice. NVIDIA’s cuBLAS reproducibility documentation limits bitwise guarantees by GPU architecture, toolkit version, and execution conditions.

Batch invariance removes batch and scheduling variation as causes of drift. It does not remove every reproducibility variable. A reproducibility claim should name its scope rather than imply universal bitwise identity.

Temperature zero is not enough

Temperature zero generally selects the highest-logit token. It does not guarantee that every serving configuration computes the same logits.

If two candidate logits are close, a small change in floating-point arithmetic can change greedy selection. A seed helps with sampling randomness, but it cannot repair a difference that occurred before token selection.

Compare Costs and Alternatives

Adaptive kernels respond to workload shape to maximize occupancy and throughput, whereas batch-invariant kernels restrict those choices to preserve a stable arithmetic path, trading peak adaptive performance for reproducible numerics.

Three broad strategies are available:

1. Fixed numerical paths: Control reduction order and kernel selection directly. This gives the clearest batch-related guarantee but requires specialized kernel work and may reduce throughput.
2. Higher or hybrid precision: Use FP32 for sensitive operations while retaining lower-precision weights or less sensitive computation elsewhere. This reduces rounding sensitivity without guaranteeing identical execution paths.
3. Verification and rollback: Run a fast path, then verify selected steps and recompute or roll back when the result fails the declared contract. This preserves a more optimized execution path but adds scheduler, instrumentation, and recovery complexity.

The NeurIPS study found much lower cross-configuration divergence in FP32 than in BF16 or FP16 in its tested settings. That result supports higher precision as a mitigation, not as a universal replacement for batch-invariant kernels.

Emerging preprints explore selective alternatives. LLM-42 uses verification and rollback under a fixed-shape schedule. MarginGate proposes verifying low-margin token decisions, where a small numerical change is more likely to flip the selected token. Both should be treated as preliminary research rather than established solutions.

Test a Serving Stack

Test batch invariance against a declared environment and a declared contract. The contract should say whether the requirement is identical logits, identical token IDs, or equivalent final text. Exact logits are the strongest and most demanding target.

For the same request, run it:

• Alone
• In supported batches of 2, 8, 32, and 128
• At different positions within the batch
• Alongside short and long companion prompts
• Alongside requests in different decoding states
• Under changing concurrent load
• Through synchronous, continuous-batching, prefill, decode, and chunking paths

Use greedy decoding first, then repeat with seeded nonzero-temperature sampling. Compare token IDs before logits or rendered text where instrumentation allows. Record the first differing token and the top-two logit margin at that position.

Also record:

• Model identifier and weight checksum
• Tokenizer and chat-template revision
• Serving framework, including exact version
• CUDA, driver, and kernel-library versions
• GPU model, count, and topology
• Tensor, data, or expert parallelism
• Precision and quantization
• Sampling parameters and seed
• Batch-invariance setting
• Tested batch sizes, positions, companion prompts, and load conditions.

Diagnose a failed test

If the output changes only when batch composition or request order changes on the same stack, investigate batch-dependent kernels and scheduling first.

If the output changes across GPU types, CUDA versions, or tensor-parallel sizes, batch invariance may not be the cause. Change one variable at a time to keep batch effects separate from changes in model files, tokenizer behavior, hardware, or sampling.

A first differing token with a small top-two margin is consistent with numerical sensitivity. It is not proof of that cause, but it provides a useful diagnostic direction. Do not infer how common the behavior is across hosted APIs from local tests because providers may use undisclosed runtimes, batching policies, and model revisions.

vLLM support

The vLLM v0.17.1 documentation documents batch invariance as a beta feature. At that version, the documented mode is enabled with:

export VLLM_BATCH_INVARIANT=1
vllm serve meta-llama/Llama-3.1-8B-Instruct

The documented support is limited to specified NVIDIA hardware, models, and same-version, same-hardware conditions. Check the exact framework version before relying on the setting in production.

FAQ

Does temperature 0 guarantee the same LLM output every time?

No. Temperature zero normally selects the highest-logit token, but batching, hardware, kernel selection, or precision can change the logits. Near-tied candidates can therefore produce different greedy tokens.

Is batch invariance the same as setting a seed?

No. A seed controls pseudorandom sampling, while batch invariance controls sensitivity to surrounding requests and execution shape. Reproducible nonzero-temperature generation requires both a stable random-state policy and controlled numerical execution.

Can another user’s prompt affect my output without data leakage?

Yes, in the narrow numerical sense. Another request can change batch shape and kernel scheduling without contributing its prompt content to your computation. A changed result is not, by itself, evidence of information sharing.

Does FP32 make batch invariance unnecessary?

No. FP32 reduces rounding sensitivity and produced stronger stability in the cited experiments, but it does not guarantee identical execution paths across all batch shapes, hardware, or software configurations. It also increases memory and compute costs.

Does batch invariance cover different GPUs and tensor-parallel sizes?

No. It covers batch-related variation within a declared execution scope. Cross-hardware, cross-version, and cross-parallelism reproducibility requires separate guarantees and testing.

Is the performance cost always large?

No universal cost exists. The impact depends on the model, workload, GPU, precision, and kernel implementation. Fixed numerical paths can reduce peak performance, while optimized deterministic paths and selective verification can reduce that penalty.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com