Detecting Schema Changes in Bing AI Performance Reports
If you already save Bing Webmaster Tools AI Performance exports, a changed column or value format can break an importer or make…
When an AI service generates an answer that cites web pages, the citation list shows which pages it named. A later answer can cite different pages even if the pages you tracked stayed the same. A cited page can also change while the answer keeps citing the same URL. To tell these events apart, save each answer with its citations, then capture a snapshot of every page you fetch.
A snapshot is a saved copy of the answer from a web request, or of the text extracted from a page. Compare snapshots using hashes, compact fingerprints of saved data. Track changes to the citation list and captured page content separately. Also record whether the publisher or server reported a different version. The records document your observations but cannot identify a provider’s private search index or the exact page version it used to generate an answer.
Save one run record for each answer and one source record for every time the answer cites a page. These field names are a practical option; no standard requires them. Separate records let you compare an answer’s citations without mixing them up with the pages you fetched.
| Record | Fields to save | What the fields tell you |
|---|---|---|
| Run | schema_version, run_id, captured_at, query or query_hash, provider, model, model_snapshot, request_settings |
Identifies the run and the conditions you know. Record a fixed model snapshot when the provider supplies one. If it supplies only a moving model name, mark the snapshot as unknown. |
| Run | answer_snapshot, answer_sha256 |
Preserves the answer and fingerprints the saved answer text. |
| Run | sources, source_set_hash, ordered_citations_hash |
Preserves citations in their original order, including repeats. The source-set hash tracks membership; the ordered-citations hash also tracks order and repeated citations. |
| Citation | citation_position, cited_url, requested_url, final_url, canonical_url |
Preserves the cited address, the address your collector fetched, the address after redirects, and your chosen key for matching the source across runs. |
| Citation | retrieved_at, fetch_status, http_status, content_type, content_language |
Records when and how the fetch completed. A failed fetch is evidence that the capture is unavailable, not that the page was deleted. |
| Citation | etag, last_modified, vary, content_encoding |
Saves relevant server response metadata when present. |
| Citation | raw_snapshot_uri, raw_sha256, extracted_text_uri, extracted_text_sha256 |
Points to the saved response and extracted text, with a separate fingerprint for each. |
| Citation | extractor_name, extractor_version, normalization_version, declared_version |
Records how you produced comparable text and any explicit page revision or version marker you found. |
Use an RFC 3339 timestamp for captured_at and retrieved_at so each record states its time consistently. A capture time is when your collector fetched the page; it is not the page’s publication or revision date. The RFC 3339 timestamp format provides a standard way to express dates and times.
Use canonical_url as your own rule for deciding whether two URLs point to the same source. Keep the original cited, requested, and final URLs, and document how your rule handles redirects. Also specify how it treats fragments and query parameters. Do not remove all query parameters: some URLs use them to select different content. Clearly record any manual decision to treat two addresses as the same source.
Calculate one SHA-256 hash for the saved response body and another for the extracted text. NIST’s Secure Hash Standard defines SHA-256 as producing a 256-bit digest. Hashes make comparisons quick, but you need the saved snapshots to inspect what changed.
The two hashes tell you different things. A raw response hash changes when the saved response body changes, including changes to page markup or embedded material. An extracted-text hash changes when your extraction and normalization steps produce different text. A page’s markup can change while the extracted text stays the same. An updated extractor can also produce different text from the same page, so record the extractor and normalization versions for every capture.
If you hash structured JSON, first convert it to the same format each time. The JSON Canonicalization Scheme defines a repeatable format for JSON objects and keeps array order. To calculate a source-set hash, make a sorted list of unique keys that identify sources, then hash that list in the standard format. To calculate an ordered-citations hash, keep the original citation order and any repeated citations. These hashes tell you different things, so do not use one in place of the other.
Save the response body and extracted text alongside their hashes. A web-archive format such as WARC stores captured web content, though a small dataset can use other storage. A provenance model such as W3C PROV-O provides ways to record which activity created a record and which inputs it used. You do not need to use either format to keep useful records.
First compare each citation in the answers. Use your documented source-matching key to identify the same sources across runs. Then report which citations were added or removed, and which citations that appear in both answers moved to a different position. Keep repeated citations when comparing order; count each source only once when comparing membership. To express how much the two sets overlap with one number, calculate Jaccard similarity: divide the number of source identities in both runs by the number in either run. This measures overlap between the runs; it does not identify what caused a change.
Next, compare the saved page records for sources that appear in both runs. Check the raw hashes and extracted-text hashes separately. Then compare server validators and explicit page version markers as separate signals. If the extracted-text hashes differ, create a text diff from the saved snapshots. The diff shows text that was added, removed, or changed, following the general comparison model in the Git diff documentation.
Finally, label each result according to what the comparisons show. For example, report “citation added,” “raw response changed; extracted text unchanged,” or “extracted text changed.” If the later fetch failed, report “later capture unavailable.” Keep these outcomes separate, and avoid a single “source changed” flag.
An ETag is a value the server supplies to identify the version of a resource it returned. In HTTP, the value is opaque, which means you cannot assume it is a readable revision number. A Last-Modified header gives the time the origin server says it last changed that version. Neither header replaces a hash of the content you saved. RFC 9110, HTTP Semantics defines these headers and other validators.
Use these signals together:
| Comparison result | What you can conclude |
|---|---|
| A source identity appears only in the later answer | The observed citation list changed. This does not show that a previously cited page changed. |
| The same source identity has different raw hashes | The response bodies your collector saved differ. |
| Raw hashes differ, but extracted-text hashes match | The captured response changed, while your recorded extraction produced the same text. |
| Extracted-text hashes differ | The extracted content differs under the recorded extractor and normalization versions. Inspect the text diff before describing the change as substantive. |
| An ETag differs, but both hashes match | The server validator changed; the saved content fingerprints did not. |
| An ETag matches, but extracted-text hashes differ | The extracted content differs despite the same validator. Check whether either ETag is weak, whether the server selected different representations, and whether your fetch or extraction settings changed. |
Last-Modified differs |
The server supplied a different modification-time value. That signal alone does not establish that meaningful page content changed. |
| The later fetch fails | Your later capture is unavailable. The failure alone does not establish that the source was removed or deleted. |
A weak ETag, marked with the W/ prefix, does not promise that two responses have identical bytes. A server can return different content depending on request headers. Recording Vary and relevant request settings helps explain why captures differ. If you send conditional requests with If-None-Match or If-Modified-Since, record the validator and response status. The response provides evidence under the server’s validator rules, but it does not replace a saved snapshot and its local hash.
Suppose two AI answers cite the same hypothetical page, https://example.com/guide. Run A records the weak ETag W/"rev-17" and extracted-text hash H1. Run B records W/"rev-18" and extracted-text hash H1; the raw response hashes differ while the extracted-text hashes match.
Report: “The captured response and server validator changed; the extracted text was unchanged under the recorded extractor and normalization versions.” The evidence does not establish that the page’s substantive claims changed, or that an AI provider’s index adopted a new page version.
If the later extracted-text hash is H2, report that the extracted content differs, then inspect the text diff. The placeholder hashes are examples, not measured values.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com