Designing a Schema to Store Reproducible AI Answer Data

When you compare answers from an AI service over time, the answer text alone does not explain why it changed. The prompt, model, product mode, retrieved documents, source content, or software settings may also have changed. An observation is one saved record of an AI request and its result, including the details needed to investigate that run later.

Store each observation as a record you never overwrite. Keep the exact input and output, record what the service reported it used, and link each citation to the source it supports. Group repeated runs under a stable question or test-case ID, but give every run its own observation ID. This preserves evidence for meaningful comparisons without promising that a future request will reproduce the same answer.

Store one record for each run

A stable case ID identifies the question being tested; an observation ID identifies one execution of that question. A repeat-group ID can connect runs made for the same comparison. Keep these identifiers separate so a new answer never overwrites an earlier one.

The record should distinguish requested settings from reported settings. For example, a client might request a model alias while the response identifies a specific model string. Record both values when available, and use null for a known unavailable value. Define whether omitted fields mean “not collected” or “not applicable,” and apply that policy consistently.

The following is a hypothetical JSON record. Its values illustrate a possible structure, not a real provider response.

{
  "record_schema_version": "1.0.0",
  "observation_id": "obs_example_001",
  "case_id": "shipping_faq_001",
  "repeat_group_id": "shipping_check_001",
  "input": {
    "prompt_text": "How long does standard shipping take?",
    "locale": "en-US",
    "conversation_context": [],
    "prompt_template_id": "customer-faq",
    "prompt_template_version": "17",
    "prompt_sha256": "sha256:example",
    "system_instructions_ref": "artifact://prompts/system/example"
  },
  "execution": {
    "provider": "example-provider",
    "product_mode": "api",
    "endpoint": "example-endpoint",
    "request_id": "provider-request-id",
    "response_id": "provider-response-id",
    "model": {
      "requested": "model-alias",
      "returned": "provider-reported-model-id",
      "provider_model_revision": null
    },
    "settings": {
      "temperature": 0,
      "top_p": null,
      "max_output_tokens": 800,
      "seed": null,
      "stream": false,
      "response_format": "text"
    },
    "tools": [],
    "application_version": "git:example-commit",
    "client_library": {
      "name": "provider-sdk",
      "version": "x.y.z"
    }
  },
  "retrieval": {
    "performed": true,
    "system": "example-search",
    "data_source_id": "shipping-index",
    "retrieval_config_version": "shipping-rag-v1",
    "index_snapshot_id": null,
    "queries": [
      {
        "query_id": "q1",
        "text": "standard shipping delivery time",
        "ranker": "provider-reported-ranker",
        "filters": {},
        "top_k": 5
      }
    ],
    "results": [
      {
        "result_id": "document-123-chunk-4",
        "document_id": "document-123",
        "chunk_id": "chunk-4",
        "query_id": "q1",
        "rank": 1,
        "score": 0.91,
        "score_type": "provider-specific",
        "source_id": "source_example_001"
      }
    ]
  },
  "sources": [
    {
      "source_id": "source_example_001",
      "provider_source_id": "provider-source-id",
      "canonical_uri": "https://example.org/shipping",
      "retrieved_uri": "https://example.org/shipping",
      "title": "Shipping information",
      "version_label": null,
      "retrieved_at": "2026-01-15T12:00:00Z",
      "content_sha256": "sha256:example",
      "snapshot_ref": "artifact://sources/example",
      "snapshot_status": "archived"
    }
  ],
  "output": {
    "text": "Standard shipping takes ...",
    "finish_reason": "stop",
    "citations": [
      {
        "citation_id": "cite_1",
        "source_id": "source_example_001",
        "provider_citation_id": null,
        "answer_span": {
          "start": 0,
          "end": 40,
          "offset_unit": "unicode-code-point"
        },
        "source_locator": {
          "chunk_id": "chunk-4",
          "page": null,
          "start": null,
          "end": null
        }
      }
    ]
  },
  "timing": {
    "request_started_at": "2026-01-15T12:00:00Z",
    "retrieval_started_at": "2026-01-15T12:00:01Z",
    "retrieval_completed_at": "2026-01-15T12:00:02Z",
    "response_received_at": "2026-01-15T12:00:03Z"
  },
  "artifacts": {
    "raw_request_ref": "artifact://requests/example",
    "raw_response_ref": "artifact://responses/example",
    "raw_response_sha256": "sha256:example"
  },
  "status": {
    "state": "completed",
    "error_code": null,
    "error_message": null
  }
}

Save the exact prompt and answer text when your privacy rules permit it. If you restrict raw content, store it in access-controlled storage and keep a durable reference and digest in the record. A digest helps detect whether stored content changed, but it cannot reconstruct that content.

Define what each model and search setting means. Store each retrieval score with its score_type, because scores from different search systems may not be comparable. Record the seed when the provider exposes it, but treat it as request metadata rather than proof of deterministic replay. OpenTelemetry’s GenAI conventions provide useful naming ideas for model and retrieval data, but its documentation notes that the conventions have moved; record which convention version your exporter follows rather than assuming one stable universal schema (OpenTelemetry’s GenAI conventions notice).

Keep retrieval results separate from citations

A retrieval result is a document or passage returned during search. A citation is evidence the answer associates with some portion of its text. Store both: a result might never be cited, and a citation might not include the retrieved passage itself.

For each source, capture the provider’s identifier when present, along with its canonical and retrieved URLs. Record the title and any document or chunk identifiers as well. Also record evidence of which source version you used, such as a content digest or a reference to an archived copy. A URL identifies where content was found, but does not establish which content appeared there at the time. If you cannot archive a source, mark the snapshot as unavailable rather than implying that the URL preserves the original.

State the unit used for citation offsets. The start and end positions might count bytes, Unicode code points (the numeric values Unicode assigns to characters), or another unit defined by the provider. Keep the provider’s original citation data alongside fields put into a consistent format, especially when the provider uses its own indexing scheme.

Google’s Gemini documentation describes grounding metadata that can include search queries, source chunks, and mappings between response segments and source-chunk indices. Preserve those original indices and the raw metadata when using that API; do not assume every provider or product mode returns the same fields (Gemini grounding documentation).

Choose JSONL or relational tables for storage

For portable exports and evaluation runs, store one complete JSON observation per UTF-8 line in JSON Lines (JSONL), so a script can process records one at a time while each run stays self-contained. The JSON Lines format defines this line-oriented approach. Validate each record against a versioned JSON Schema, a document that describes the structure and constraints of the JSON. The JSON Schema getting-started guide explains how these schemas work.

For database queries across many observations, use a relational database, which stores data in linked tables. Put repeated kinds of data in separate child tables. For example, use answer_case for stable test definitions and answer_observation for each run. Use retrieval_query and retrieval_result for search activity. Use source_snapshot for evidence of source versions, and answer_citation to link citations to parts of answers. Use foreign keys from results and citations to their source records. Keep provider-specific raw payloads in a JSON column or in archived artifacts so new provider fields do not disappear during normalization.

A practical setup stores searchable metadata in database rows and larger raw requests, responses, and source snapshots in access-controlled object storage. Keep artifact identifiers and digests in the observation, and do not change them. Export JSONL for transfer. When you mainly analyze data in tables, use a columnar format such as Parquet, which stores data by column. Keep the original record or JSONL export as the portable source of truth.

Store timestamps with a UTC offset, such as an RFC 3339 timestamp ending in Z. Record request start, retrieval start and completion, and response receipt separately when those events are observable. RFC 3339 specifies the Internet date-and-time format and its relationship to UTC.

Compare changes in layers

A useful comparison shows which evidence changed, not just which words changed. Compare the exact input and locale first, then prompt-template and system-instruction versions. Next compare the provider, product mode, requested and returned model identifiers, settings, tools, and software versions. Then compare retrieval queries, configuration and index versions, ranked results, source snapshots, and citation mappings. Finally compare the answer text, completion status, and finish reason.

Keep the original prompt and response as primary evidence. If you create normalized text for easier comparison, such as a whitespace-normalized answer, store it as a derived value and document the normalization method. Do not replace the original with it. Preserve array order in retrieval results and citations because rank and sequence carry meaning.

For structured JSON digests, RFC 8785 specifies a canonicalization method that standardizes object-property ordering and preserves array order. Use it to hash the structured value you intend to compare, and keep the original serialized request and response as separate artifacts. Canonicalization makes JSON representation consistent; it does not prove the origin of a response or preserve an external source.

Protect the evidence you store

Personal or confidential information can appear in prompts and instructions, as well as in queries and answers. Decide what your code captures before enabling raw-content logging. Apply access controls and retention limits to stored text and snapshots, and use redaction where your policy requires it. OpenTelemetry likewise warns that message and query content can contain sensitive data (GenAI attribute registry).

Never change a saved observation. If a parser extracts citations incorrectly or a later schema adds fields, write a corrected or reprocessed record and link it to the original observation. This keeps the history intact and makes it clear which values came from the original provider response and which came from later processing.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com