How to Test a Nondeterministic LLM Application

When an application calls an LLM, the same input can produce a different response on the next run. A test that compares the response with one expected string will fail on answers that are worded differently but still correct. Test the behavior that has to hold on every run instead. The answer should contain the right decision and follow the safety policy. Tools should be called only with authorization, and the database or other external system should end up in the right state.

Setting temperature to zero and fixing a seed reduces this variation but does not remove it. A 2024 study found that some models still gave different answers with both settings. Sampling is only one cause of the variation. Model updates, the provider's serving setup, concurrency and retries can also change a result. Your own application adds more through retrieval results, tool responses, conversation history and safety filters, and an agent can take a different path through a multi-step task on each run.

In practice, you control what you can, run important cases several times, check what the application did and grade the results with code checks, model graders and human review.

Decide what counts as a pass

Do not use identical wording as the default pass condition. A support assistant can phrase its answer differently on every run and still give the correct policy answer each time. An assistant that gives the same wrong answer, or takes the same unauthorized action, on every run is repeatable and still unreliable. Exact matching makes sense only when the exact text is the requirement, such as a fixed command or a narrow classification label.

For each important task, write down what the application must do, what it must never do and what is allowed to vary. Hard requirements are pass or fail, for example:

• Never disclose another user's private data.
• Never issue a refund without authorization.
• Always return valid JSON from an API endpoint.
• Never execute shell or SQL input generated by a model until it has been validated.
• Update the database only after the required checks pass.
• Refuse or escalate when the evidence is insufficient.

Softer qualities, such as tone, concision, formatting and how helpful the explanation is, can get a score instead of a pass or fail.

If the output is structured, parse it before you grade it. Check required fields, data types and allowed values, then numeric ranges, constraints between fields and business rules. Two JSON responses with different whitespace or field order should both pass if they represent the same valid result.

Test the application in layers

Treat the whole application as the thing under test, including the code around the model call that builds the prompt, runs retrieval and tools, applies business rules and stores the results. For agents, Anthropic's guide to agent evals calls this surrounding system the agent harness and points out that evaluating an agent means evaluating the harness and the model together.

Parts of this code should behave the same way every time, so they get ordinary unit tests that run on each code change:

• Input validation, authentication and permission checks
• Prompt construction and message serialization
• JSON parsing, schema validation and tool parameter validation
• Retries, timeouts and rate limits
• Redaction and output escaping
• Database writes and billing or cost controls

These tests keep malformed context away from the model and stop model output from bypassing your safeguards. The model's behavior and the orchestration around it need repeated runs instead. A structured extraction feature, for example, can be tested in layers:

1. Unit-test the parser and the schema validator.
2. Check that the prompt contains the correct customer record.
3. Run the model several times on representative records.
4. Compare the extracted values with labeled expectations.
5. Check the database writes against the validated result.

Splitting the layers tells you where a failure comes from. A malformed database write is an application bug even when the model produced correct text. A wrong classification after valid parsing points to the model or the orchestration.

Check what the application did

For agents and other features that change data, grade the whole run. Look at which tools were chosen, what arguments were passed, the intermediate actions and any retries. Then check the final response, the authorization and escalation decisions, and the state of the database or environment afterwards.

Do not take the model's statement that it finished a task as proof that it did. An agent can say it canceled an order while the cancellation API failed, or say it updated a record that a validation rule rejected.

In a refund flow, a run should pass only if all of these are true:

• The customer is authenticated.
• The order belongs to that customer.
• The refund policy allows the refund.
• The agent passes the correct order ID and amount.
• The refund tool succeeds.
• The payment state shows the refund.
• The final response reports the result accurately.

A transcript alone cannot verify the last two.

Build a test set

Each test case needs an input, the behavior you expect and a way to grade it. Start with hand-written cases for common, valuable and high-risk tasks. Add examples from production, adversarial cases for security, and synthetic variations for rare combinations and formatting changes. Every past failure should become a regression case.

Production examples can show assumptions that hand-written tests miss. Google's evaluation documentation lists sampling from production logs and generating synthetic data as ways to build an evaluation dataset. Remove personal data and restrict access before you use production records. Synthetic cases add coverage but do not replace real examples, because the generator can share the assumptions and blind spots of the application under test.

Test both when an action should happen and when it should not. For a customer-service agent, include cases where it should:

• Search the current documentation
• Answer from the evidence it was given
• Call the refund tool after authorization
• Escalate an ambiguous request
• Refuse a prohibited request

Also include cases where it should not:

• Search when the answer is already present and a search would create unnecessary exposure
• Call the refund tool without authorization
• Answer confidently when the retrieved evidence is missing or contradictory
• Follow a user instruction that conflicts with a higher-priority policy
• Keep acting after the task is complete

An agent that always refuses can pass a test set full of refusal cases while failing at its job. Balance the cases where it should act against refusal, abstention and escalation cases.

Security and edge cases

Test direct attacks and indirect ones, where the instructions hide in retrieved documents, emails, web pages, uploaded files, tool results or database fields. Beyond prompt injection, cover attempts to extract the system prompt or sensitive data, unauthorized tool use, conflicting instructions and attempts to bypass confirmation steps. For edge cases, try very long inputs, variations in Unicode and encoding, and repeated requests meant to exhaust resources.

The OWASP Top 10 for LLM applications lists prompt injection, sensitive information disclosure, improper output handling, excessive agency, system prompt leakage, and vector and embedding weaknesses among its risks.

Test the boundary between model output and executable behavior. If generated text reaches SQL, shell commands, HTML, email, file systems or privileged APIs, ordinary code has to validate and authorize it before it runs. A refusal from the model does not replace access control.

Retrieval

If the application uses retrieval-augmented generation (RAG), meaning it retrieves documents before it answers, test retrieval and generation as separate sources of failure. For retrieval, check that relevant documents appear and that the most useful ones rank first. Check that metadata filters work, that access controls exclude documents the user may not see, and that the documents are fresh. Also test duplicate and contradictory records, and check that the application responds safely when it finds no evidence.

For the generated answer, check that every material claim is supported, that citations point to the correct source and that they cover the important claims. The answer should also keep the caveats and scope of its sources.

A factually correct answer can still fail if it cites the wrong document or uses evidence the user is not allowed to access. Grade access control, grounding and citation accuracy separately.

Run each case more than once

A single passing run hides the other outcomes the same input can produce. Run important cases several times with the same controlled inputs and record how often each behavior occurs.

There is no universal number of runs. Use more when a task has safety, financial, legal, medical or privacy consequences, when failures are rare but severe, or when the observed pass rate is close to a release threshold. Agents with many tool steps, model graders that give inconsistent results, and recent changes to the provider, model, prompt, retrieval or tools are also reasons to run more.

The 2024 reproducibility study tested models on two benchmarks about reasoning with compass directions. With temperature at zero and a fixed seed, three repeats were usually enough for a tight estimate of a model's average score. That does not give you a repeat count for a production application, where the variation depends on the model, the provider, the task and the rest of the application. A small smoke test with fewer runs is weak evidence of production reliability.

Control what you can

For regression testing, reduce the variation you can avoid:

• Temperature at zero, with fixed top-p and top-k
• A fixed seed, where the API supports one
• A pinned model version or snapshot
• A fixed prompt and message format
• A fixed retrieval corpus and controlled tool responses
• The same concurrency and batch settings
• A pinned runtime for self-hosted models

A fully frozen setup is good for catching regressions, but it misses failures caused by changing documents, external APIs, the current date and time, and concurrent requests. If production has that kind of variation, run a second suite that includes it.

Pass rates and failure categories

For each case, record the number of runs, the number of passes and the pass rate (passes divided by runs). Also record the failure category of each failed run, how much the outputs and tool paths varied, how often graders disagreed, and the latency, token use and cost.

Report the pass rate with an uncertainty interval. Nine passes out of 10 and 90 out of 100 are both 90%, but the second result is stronger evidence.

Count failure categories separately. An incorrect answer, an unsupported citation, an unauthorized tool call and a failed database write should not collapse into one aggregate score. Hard failures should stay visible even when average quality improves.

First-attempt success and retries

Anthropic's guide to agent evals defines pass@1 as success on the first try and pass@k as the chance of at least one success in k attempts. pass@k is useful when the application can safely generate several candidates in an isolated setting. It is no substitute for first-attempt reliability when each attempt can send an email, charge a card, change a record or delete data. The same guide defines pass^k, the chance that all k attempts succeed, and recommends it for agents that have to behave consistently.

For anything that changes state, measure how often the first authorized attempt completes safely and correctly.

Test retries separately. A retry after a timeout can create a duplicate payment or a second record unless the operation is idempotent. An idempotent operation has the same final effect whether the request arrives once or several times. Run timeout, partial-failure and retry cases against a sandbox or a transaction-safe test environment.

Grade with code, model graders and people

Match each grader to the requirement it checks. A single grader that judges schema validity, tool authorization, groundedness, tone and safety in one opaque score hides which of them failed. In a practical setup, code checks enforce the hard requirements, a model grader scores open-ended quality, and human reviewers audit high-risk and borderline cases.

Use code for requirements with an objective answer, such as schema validity, exact identifiers and values, tool names and parameters, authorization decisions and database state. Security rules, latency, token and cost limits, retry counts, and required citations and their targets belong in code checks as well. Code checks are fast and reproducible. They become brittle when several answers are valid, so normalize structured data and compare the fields that carry the meaning instead of raw text.

Use a model grader for qualities that need semantic judgment, such as relevance, completeness, groundedness, instruction following and helpfulness. Give it a rubric with explicit pass and fail conditions and examples of borderline cases. A model grader is itself a model call, so its judgments vary too. In OpenAI's grader documentation, for example, score model graders have their own sampling settings, including temperature and seed.

Before you rely on a model grader, repeat the same grading task and measure how much the results disagree. Compare the grader with expert judgments, especially on borderline examples. In pairwise comparisons, test for position bias, and check whether the grader favors its own model family. Version the grader model and the rubric. Anthropic's guide also recommends calibrating model graders closely against human experts.

Human reviewers calibrate rubrics, resolve disagreements and audit high-impact outcomes. They should look at safety, privacy, fairness and ambiguous cases as well as typical ones. Change the rubric when reviewers find a recurring grading error, and add a regression case to the test set for each significant failure they find.

Log enough to explain a failure

A failed run is only useful if the record shows what happened. Capture enough context to tell a model change apart from a change in the prompt, retrieval, tools, infrastructure or application. Record, where applicable:

test_case_id
user_input
conversation_state
prompt_versions
model_identifier
model_version_or_snapshot
sampling_settings
seed_if_supported
retrieved_documents
tool_definitions
tool_inputs_and_outputs
full_transcript
retry_and_timeout_events
final_output
external_state_changes
grader_versions
latency
token_usage
cost
timestamp
deployment_and_runtime_details

For retrieval, keep the document identifiers and the retrieved text, or a privacy-safe version of it that still allows inspection later.

Evaluation traces can contain user messages, private documents, sensitive business data and credentials that ended up in tool output by accident. Apply redaction, access controls, encryption, retention limits and privacy review to test logs as well as production logs. Use synthetic substitutes for secrets and personal data whenever they preserve the behavior under test. Reproducibility does not justify keeping sensitive user content forever.

Use the tests as a release gate

Run the evaluations after every change to code, prompts, models or retrieval data, and after changes to tool definitions, safety policies or provider configuration. A prompt change alone can alter tool behavior, and a retrieval index update can change answers without any model change.

Run the gate against a pinned baseline and set explicit thresholds for hard failures, first-attempt success, quality scores, latency and cost. The gate should include human review of high-impact or borderline cases alongside the automated tests. Do not ship a release that improves a soft quality score by introducing an unacceptable safety or authorization failure.

Document the risks that remain and the properties you did not measure. NIST's AI Risk Management Framework recommends testing AI systems before deployment and regularly during operation, with measures of uncertainty, comparisons to benchmarks, documented results and independent review.

Monitor production

A one-time test run cannot catch failures caused later by new model versions, changing retrieval data, provider behavior, tool responses or real user inputs. In production, track quality failures, safety incidents, refusals, escalations and user corrections, along with tool failures, retrieval failures and attempts at unauthorized actions. Watch latency, cost and drift in outputs and tool paths, and compare behavior before and after provider or model updates.

Sample transcripts for manual review under strict privacy controls. When a production failure reveals a missing case, add a minimized and de-identified version to the regression set. Keep incident-response and rollback procedures ready for model, prompt, retrieval and tool changes.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com