Currently Available: Need a skilled Software Developer for your next project?

AI Engineering Guides

How to Test a Nondeterministic LLM Application

August 11, 2026 by Vincent Schmalbach

Test an LLM application as a workflow that must meet behavioral requirements across repeated runs, not as a function that returns one exact string. The application should produce acceptable answers, make authorized tool calls, preserve required data, and reach the correct external state even when wording or execution paths vary.

Nondeterminism means that the same input can produce different outputs or actions. Sampling is one cause, but provider implementation details, model updates, retrieval results, tool responses, concurrency, retries, and multi-step agent decisions also introduce variation. Temperature zero and fixed seeds reduce some variation, but they do not guarantee identical execution. Research has found residual nondeterminism under those settings for some models, so testing must measure behavior rather than assume reproducibility from configuration alone (Towards Reproducible LLM Evaluation).

The practical approach is to control the inputs and runtime where possible, run important cases repeatedly, evaluate outcomes and side effects, and combine deterministic checks with model-based and human review.

Define Reliability Before Reproducibility

Reproducibility asks whether a test produces the same or equivalent result when repeated under stated conditions. Reliability asks whether the application succeeds often enough and safely enough for its intended use.

An application can be non-reproducible but reliable. A support assistant might use different wording on every run and still give the correct policy-based answer each time. It can also be reproducible but unreliable if it consistently returns the wrong answer or performs an unauthorized action.

Define reliability using observable outcomes such as:

The answer contains the correct decision.
The response follows the required safety policy.
The output matches the required schema.
The agent calls a tool only when authorization exists.
The database reaches the intended state.
The answer cites evidence accurately.
The application escalates when evidence is insufficient.
The response stays within latency and cost limits.

Do not make identical wording the default success condition.

Set Behavioral Contracts

Write a behavioral contract for each important workflow. The contract states what the application must do, must not do, and is allowed to vary on.

Separate hard requirements from soft qualities.

Hard requirements are non-negotiable:

Never disclose another user’s private data.
Never issue a refund without authorization.
Always return valid JSON for an API endpoint.
Never execute shell or SQL input generated by a model until it has been validated.
Update the database only after the required checks pass.
Refuse or escalate when evidence is insufficient.

Soft requirements can receive a score rather than a binary result:

Tone
Concision
Quality of explanation
Formatting preference
Degree of helpfulness

Parse structured output before grading the response. Check required fields, data types, allowed values, numeric ranges, cross-field constraints, and business rules. Two JSON responses with different whitespace or field order should pass if they represent the same valid result.

Identify Sources of Variation

List every component that can change a trial’s result:

Sampling settings such as temperature, top-p, and top-k
Provider-side execution behavior
Model or model-snapshot updates
Retrieval ranking, document freshness, and corpus changes
Tool responses and external API state
Conversation history
Concurrency and request ordering
Hardware and batching for self-hosted models
Timeouts and retries
Agent planning and tool-call trajectories
Safety filters and post-processors

Temperature is only one contributor. A fixed prompt can still produce a different result when the retrieved documents change, an API returns a different record, or a retry takes a different path.

Test the Complete Application

The unit under test should usually be the complete workflow, not only the model call. That workflow includes prompt construction, conversation state, retrieval, tools, business logic, safeguards, storage, retries, and the user-facing response.

Anthropic describes an agent harness as code surrounding the agent that processes inputs, orchestrates tools, records interactions, and returns the final result. Testing only the model response misses failures in that harness (Demystifying evals for AI agents).

Separate the Application Layers

Use conventional software tests for components that should be deterministic:

Input validation
Authentication and authorization
Prompt construction
Message serialization
JSON parsing
Schema validation
Tool parameter validation
Handling retries and timeouts
Redaction
Rate limits
Output escaping
Database writes
Permission checks
Billing and cost controls

These tests should run quickly on every code change. They prevent the model from receiving malformed context and prevent model output from bypassing application safeguards.

Then test model behavior and orchestration statistically. For example, a structured extraction workflow might use the following layers:

Unit-test the parser and schema validator.
Verify that the prompt contains the correct customer record.
Run the model repeatedly on representative records.
Check extracted values against labeled expectations.
Verify downstream database writes against the validated result.

This separation makes failures easier to diagnose. A malformed database write is an application defect even if the model produced correct text. A wrong classification after valid parsing belongs to the model or orchestration evaluation.

Verify Outcomes and Side Effects

For agents and state-changing workflows, inspect the entire execution:

Tool selection
Arguments passed to tools
Intermediate actions
Retry behavior
Final response
Database or environment state
Authorization decisions
Escalation behavior

Never treat the model’s claim that it completed a task as proof that it completed the task. An agent can say that it canceled an order while the cancellation API failed, or claim that it updated a record while a validation rule rejected the write.

For a refund workflow, a passing trial should require all of the following:

The customer is authenticated.
The order belongs to that customer.
The refund policy allows the action.
The agent passes the correct order identifier and amount.
The refund tool succeeds.
The payment state reflects the refund.
The final response accurately reports the result.

A transcript alone cannot verify the last two conditions.

Build Representative Test Cases

Create a test corpus that reflects real usage, important business workflows, known failures, and unsafe conditions. Each case should include an input, expected behavioral requirements, and a grading method.

Combine several sources:

Curated cases for common, high-value, and high-risk tasks
Privacy-controlled production examples
Synthetic variations for rare combinations and formatting changes
Adversarial cases for security and robustness
Historical failures converted into regression cases

Production-derived examples expose assumptions that curated tests miss. Google’s evaluation documentation describes using production logs and synthetic examples to build evaluation datasets (Google Cloud evaluation overview). Redact personal data, restrict access, encrypt stored traces, and define a retention period before using production records.

Synthetic cases expand coverage but do not replace real examples. A generator can reproduce the same assumptions and blind spots as the application under test.

Cover Positive and Negative Cases

Test both when an action should occur and when it should not.

For a customer-service agent, include cases where it should:

Search current documentation.
Answer using supplied evidence.
Call a refund tool after authorization.
Escalate an ambiguous request.
Refuse a prohibited request.

Also include cases where it should not:

Search when the answer is already present and searching creates unnecessary exposure.
Call a refund tool without authorization.
Answer confidently when the retrieved evidence is missing or contradictory.
Follow a user instruction that conflicts with a higher-priority policy.
Continue acting after the task has completed.

A system that always refuses can pass a refusal-heavy test set while failing its actual purpose. Balance successful-action cases with refusal, abstention, and escalation cases.

Include Security and Edge Cases

Test direct and indirect attacks, including instructions embedded in retrieved documents, email, webpages, uploaded files, tool results, and database fields.

Cover:

Prompt injection
Malicious retrieved content
Tool-output manipulation
System-prompt extraction
Sensitive-data extraction
Unauthorized tool use
Excessively long inputs
Repeated requests intended to exhaust resources
Variations in Unicode and encoding
Conflicting instructions
Attempts to bypass confirmation steps

The OWASP LLM risk list identifies prompt injection, sensitive information disclosure, improper output handling, excessive agency, system-prompt leakage, and vector or embedding weaknesses as application risks.

Test the boundary between model output and executable behavior. If generated text reaches SQL, shell commands, HTML, email, file systems, or privileged APIs, validate and authorize it with ordinary code before execution. A refusal from the model is not a substitute for access control.

Evaluate Retrieval Separately

For a retrieval-augmented generation application, test retrieval and generation as separate failure surfaces.

Measure retrieval quality:

Whether relevant documents appear
Whether useful documents rank first
Whether metadata filters work
Whether access controls exclude unauthorized documents
Whether documents are fresh
Whether duplicate or contradictory records are handled
Whether the application responds safely when no evidence is found

Then measure the generated answer:

Whether each material claim is supported
Whether citations point to the correct source
Whether citations cover the important claims
Whether the answer preserves caveats and scope
Whether the application abstains when evidence is missing
Whether retrieved instructions can manipulate behavior

A factually correct answer can still fail if it cites the wrong document or uses evidence the user is not allowed to access. Grade access control, grounding, and citation accuracy separately.

Run Repeated Trials

A single successful run hides the distribution of possible outcomes. Run important cases multiple times with the same controlled inputs, then record how frequently each behavior occurs.

Use more trials when:

The workflow has safety, financial, legal, medical, or privacy consequences.
The observed rate is close to a release threshold.
Failures are rare but severe.
The agent has many tool steps.
The model-based grader gives inconsistent results.
Provider, model, prompt, retrieval, or tool behavior recently changed.

A 2024 study found that three repeats were often enough for a particular prediction-interval target under its tested temperature-zero, fixed-seed conditions. That result does not establish a universal repeat count for production applications because variability depends on the model, benchmark, provider, and workflow (Towards Reproducible LLM Evaluation).

Control Test Conditions

Reduce avoidable variation during regression testing with:

Temperature set to zero
A fixed seed when supported
Fixed top-p and top-k settings
A pinned model version or snapshot
A fixed prompt and message format
A fixed retrieval corpus
Controlled tool responses
Consistent concurrency and batch settings
A pinned runtime for self-hosted models

Treat these controls as measurement aids, not guarantees of determinism. The reproducibility research found that some models remain variable even with temperature zero and a fixed seed.

Run a second suite with realistic variation when that variation exists in production. A fully frozen test can show regression behavior while missing failures caused by changing documents, external APIs, time, or concurrent requests.

Measure Success Distributions

For each test case, record:

Total trials
Successful trials
Success rate per case
Failure categories
Output diversity
Tool-path diversity
Latency distribution
Token use
Cost
Grader disagreement

A basic estimate is:

success rate = successful trials / total trials

Report an uncertainty interval with the rate. A result of 9 successful trials out of 10 does not provide the same evidence as 90 out of 100, even though both have a nominal 90% success rate.

Track failure categories separately. For example, “answer incorrect,” “citation unsupported,” “tool unauthorized,” and “database write failed” should not collapse into one aggregate score. Hard failures should remain visible even when average quality improves.

Use Pass Rates Carefully

Pass@1 measures whether the first attempt succeeds. Pass@k measures whether at least one successful result appears across k attempts. These metrics answer different questions.

Pass@k is useful when the application safely generates multiple candidates in an isolated setting. It is not a substitute for first-attempt reliability when each attempt can send an email, charge a card, modify a record, or delete data.

For state-changing workflows, prioritize:

The probability of safe, correct completion on the first authorized attempt.

Retries require separate testing. A retry after a timeout can duplicate a payment or create two records unless the operation is idempotent, meaning repeated requests produce the same final effect. Test timeout, partial failure, and retry scenarios against a sandbox or a transaction-safe test environment.

Combine Multiple Graders

Match each evaluator to the requirement. No single grader should judge schema validity, tool authorization, groundedness, tone, and safety in one opaque score.

Use Deterministic Checks

Use code-based checks for requirements with an objective answer:

Schema validity
Exact identifiers and values
Names and parameters for tools
Authorization decisions
Database state
Security rules
Latency and token limits
Cost limits
Retry counts
Required citations and citation targets

These checks are fast and reproducible. They become brittle when several answers are valid, so normalize structured data and compare semantic fields rather than raw text.

Calibrate Model-Based Graders

Use a model-based grader for qualities that depend on semantic judgment, including relevance, completeness, groundedness, instruction following, and helpfulness. Give it a rubric with explicit pass and fail conditions, and provide examples of borderline cases.

Test the grader itself by:

Repeating the same grading task.
Comparing results against expert judgments.
Checking borderline examples.
Measuring disagreement.
Testing for position bias in pairwise comparisons.
Checking whether the grader favors its own model family.
Versioning the grader model and rubric.

Model graders are inference systems, so their judgments also vary. OpenAI’s grader documentation exposes sampling controls for grader configurations, which reinforces the need to evaluate grader repeatability rather than treating its score as ground truth.

Review High-Risk Decisions

Use human reviewers to calibrate rubrics, resolve disagreements, and audit high-impact outcomes. Reviewers should examine safety, privacy, fairness, and ambiguous cases, not only average-quality examples.

A practical pattern is:

Code checks enforce hard requirements.
A model grader scores open-ended quality.
Human reviewers audit high-risk and borderline cases.
The rubric changes when reviewers find a recurring grading error.
The test corpus gains a regression case for each significant failure.

Record Evidence for Diagnosis

A failure is actionable only when the recorded evidence shows what happened. Capture the context needed to distinguish a model change from a prompt, retrieval, tool, infrastructure, or application change.

Capture Complete Trial Context

Record, where applicable:

test_case_id
user_input
conversation_state
prompt_versions
model_identifier
model_version_or_snapshot
sampling_settings
seed_if_supported
retrieved_documents
tool_definitions
tool_inputs_and_outputs
full_transcript
retry_and_timeout_events
final_output
external_state_changes
grader_versions
latency
token_usage
cost
timestamp
deployment_and_runtime_details

For an agent, preserve the tool sequence, intermediate messages, and final response. For a retrieval workflow, preserve document identifiers and the retrieved text or a privacy-safe representation that permits later inspection.

Protect Evaluation Logs

Evaluation traces often contain user messages, private documents, credentials accidentally included in tool output, or sensitive business data. Apply redaction, access controls, encryption, retention limits, and privacy review to both test and production logs.

Use synthetic substitutes for secrets and personal data whenever they preserve the behavior under test. Reproducibility does not justify indefinite retention of sensitive user content.

Gate Releases and Monitor Production

Run evaluations after changing code, prompts, models, retrieval data, tool definitions, safety policies, or provider configuration. A prompt-only change can alter tool behavior, and a retrieval-index update can change answers without any model change.

Build Regression Gates

A release gate should include:

Deterministic unit and schema tests
Tool-permission and authorization tests
Representative golden cases
Repeated trials on high-risk workflows
Security and prompt-injection tests
Retrieval relevance and access-control tests
Cost and latency comparisons
Human review of high-impact or borderline cases

Compare results with a pinned baseline. Set explicit thresholds for hard failures, first-attempt success, quality scores, latency, and cost. Do not ship a release that improves a soft quality score by introducing an unacceptable safety or authorization failure.

Document residual risks and properties that remain unmeasured. NIST’s AI Risk Management Framework recommends testing before deployment and regularly during operation, with documented metrics, uncertainty, benchmarks, and independent review.

Monitor Real Usage

Production monitoring should track:

Quality failures
Safety incidents
Refusals and escalations
User corrections
Tool failures
Retrieval failures
Unauthorized-action attempts
Latency and cost
Drift in output and tool paths
Changes after provider or model updates

Sample transcripts for manual review under strict privacy controls. When a production failure reveals a missing case, add a minimized and de-identified version to the regression corpus. Maintain incident-response and rollback procedures for model, prompt, retrieval, and tool changes.

FAQ

Is temperature zero enough to make an LLM deterministic?

No. Temperature zero reduces sampling variation, and a fixed seed can reduce it further, but some models still produce different results under those settings. Test repeatability under the actual provider, model, prompt, retrieval, and tool conditions you deploy (Towards Reproducible LLM Evaluation).

How many times should I run each test case?

There is no universal repeat count. Use more trials for high-risk cases, rare serious failures, unstable graders, and results near a release threshold. A small smoke test can use fewer trials, but its result should not be treated as strong evidence of production reliability.

Should tests compare exact text?

Use exact matching only when exact text is a real requirement, such as a fixed command or a narrow classification label. For open-ended responses, check required facts, structured meaning, citations, policy compliance, tool behavior, and final state instead.

Can an LLM judge another LLM?

Yes, model-based graders can scale semantic evaluation, but they are themselves variable and can contain bias. Calibrate them against expert judgments, repeat borderline grading tasks, and keep deterministic checks and human review for hard or high-impact requirements (Demystifying evals for AI agents).

How should I test an agent?

Test the full agent harness, including input handling, tool selection, arguments, intermediate actions, retries, final responses, authorization, and environmental side effects. Verify the resulting database or external state instead of trusting the agent’s statement that it completed the task.

What matters most in production?

Maintain continuous evaluation, monitoring, transcript review, incident response, and regression-case creation. A one-time benchmark cannot detect failures caused by new model versions, changing retrieval data, provider behavior, tool responses, or real user inputs.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com