Currently Available: Need a skilled Software Developer for your next project?
Categories
SEO, GEO and AEO

Parsing Citations and Sources from AI Search APIs

When a website owner or developer sends a request to an AI search API, the response can include answer text, records for pages the provider returned, and metadata linking those pages to parts of the answer. A source record identifies a returned page. A citation annotation links a page to a span of answer text. Your code can add a display marker such as [1]; the raw response does not have to include one.

To preserve the API’s response, parse the answer text, source records, and citation annotations separately. Store the original payload alongside fields you have put into a consistent format. OpenAI Responses API web search and Google Gemini grounding return data in different shapes, so each needs its own parser. Google’s terms also restrict storing or analyzing grounded links for another purpose, which affects whether you can keep Gemini results for later analysis.

Separate answer text, sources, and citations

A response can record search activity, such as a search query or a call to a search tool. It can also include entries in a source list and citations. Keep these records separate. OpenAI’s web search documentation puts retrieved URLs in sources and attaches url_citation annotations to answer text. A URL can appear in sources without an inline citation. You can also analyze a citation annotation separately from the source list.

Citation annotations are structured metadata, so they do not have to appear as visible markers in the answer. OpenAI’s Responses API reference documents URL citation fields such as url, title, start_index, and end_index. Keep the original answer text with these positions. If you trim, join, or reformat the text first, the positions may point to a different span.

Parse OpenAI Responses API output

The Responses API returns an output array with items such as tool calls and messages. Read every item in the array; a response can have more than one search call, message, or text block. With the current web_search tool, check the search action for activity and returned sources. Then check message content for answer text and citation annotations. Some responses omit fields, so treat a missing array as empty.

The following JavaScript adapter illustrates the field relationships for an article setup; the example is illustrative and was not captured from an API response.

function parseOpenAIResponse(response) {
  const searches = [];
  const sources = [];
  const textParts = [];
  const citations = [];

  for (const item of response.output ?? []) {
    if (item.type === "web_search_call") {
      const action = item.action ?? {};

      searches.push({
        callId: item.id ?? null,
        status: item.status ?? null,
        actionType: action.type ?? null,
        query: action.query ?? null,
        queries: action.queries ?? [],
        raw: item
      });

      for (const source of action.sources ?? []) {
        sources.push({
          sourceListName: "web_search_action",
          callId: item.id ?? null,
          url: source.url ?? null,
          title: source.title ?? null,
          raw: source
        });
      }

      continue;
    }

    if (item.type !== "message") continue;

    for (const part of item.content ?? []) {
      if (part.type !== "output_text" && part.type !== "text") continue;

      const partId = <code>${item.id ?? "message"}:${textParts.length};

      textParts.push({
        partId,
        text: part.text ?? "",
        rawType: part.type
      });

      for (const annotation of part.annotations ?? []) {
        if (annotation.type !== "url_citation") continue;

        citations.push({
          textPartId: partId,
          citationType: annotation.type,
          url: annotation.url ?? null,
          title: annotation.title ?? null,
          startIndex: annotation.start_index ?? null,
          endIndex: annotation.end_index ?? null,
          raw: annotation
        });
      }
    }
  }

  return {
    responseId: response.id ?? null,
    model: response.model ?? null,
    status: response.status ?? null,
    searches,
    sources,
    textParts,
    citations
  };
}

The code keeps each text part and the original annotation. It also stores search activity separately from citations. Check where sources appears in the response mode you use. The field may be absent, so do not assume every web-search response includes a source list.

Parse Gemini grounding metadata

Gemini’s Generate Content API uses grounding metadata, fields that link answer text to sources. A groundingChunks entry holds source details, such as a web URL and title. Each groundingSupports entry points to one or more chunks by their positions in the array. It can also include a text segment and its start and end positions. Search queries appear in webSearchQueries.

Keep the chunk index when parsing a support. The provider uses that index to link the answer segment to a source record. The following adapter handles Generate Content payloads only. Its code illustrates article setup and is not captured API output.

function parseGeminiCandidate(candidate, candidateIndex = 0) {
  const metadata = candidate.groundingMetadata ?? {};
  const chunks = metadata.groundingChunks ?? [];

  const textParts = (candidate.content?.parts ?? [])
    .filter(part => typeof part.text === "string")
    .map((part, partIndex) => ({
      partId: <code>candidate:${candidateIndex}:part:${partIndex},
      text: part.text
    }));

  const sources = chunks.map((chunk, chunkIndex) => ({
    sourceId: candidate:${candidateIndex}:chunk:${chunkIndex},
    sourceIndex: chunkIndex,
    sourceType: chunk.web ? "web" : "other",
    url: chunk.web?.uri ?? null,
    title: chunk.web?.title ?? null,
    raw: chunk
  }));

  const supports = (metadata.groundingSupports ?? []).map(support => ({
    text: support.segment?.text ?? null,
    startIndex: support.segment?.startIndex ?? null,
    endIndex: support.segment?.endIndex ?? null,
    sourceChunkIndices: support.groundingChunkIndices ?? [],
    raw: support
  }));

  return {
    candidateIndex,
    textParts,
    searchQueries: metadata.webSearchQueries ?? [],
    sources,
    supports,
    searchEntryPoint: metadata.searchEntryPoint ?? null
  };
}

This adapter keeps the provider’s chunk indexes in sourceChunkIndices; it does not silently convert them into source identifiers that apply across responses. Google’s Interactions API migration documentation describes a different response format. Record which endpoint returned the data, and use a separate adapter for the Interactions API. Generate Content field paths do not necessarily apply there.

Store a reproducible record

A shared schema is an optional design choice that makes responses easier to compare; neither provider requires one. Record which provider and API returned the response. Include the endpoint and model, along with the capture time. Keep the raw payload and provider-specific objects so you can review earlier parsing decisions if you change your adapter.

For example, a normalized record can use this shape:

{
  "schema_version": "1.0",
  "provider": "provider-name",
  "api": "endpoint-or-api-name",
  "response": {
    "id": null,
    "model": null,
    "status": null,
    "created_at": null,
    "captured_at": null
  },
  "request_context": {
    "endpoint": null,
    "tool": null,
    "prompt_or_prompt_hash": null
  },
  "text_parts": [
    {
      "part_id": "provider-scoped-part-id",
      "text": "Answer text as returned"
    }
  ],
  "search_activity": [
    {
      "call_id": null,
      "action_type": null,
      "queries": [],
      "raw": {}
    }
  ],
  "sources": [
    {
      "source_id": "response-scoped-source-id",
      "source_index": 0,
      "url": "https://example.org/page",
      "title": "Example page",
      "source_list_name": "provider-specific-list",
      "raw": {}
    }
  ],
  "citations": [
    {
      "text_part_id": "provider-scoped-part-id",
      "citation_type": "provider-specific-type",
      "url": "https://example.org/page",
      "start_index": 0,
      "end_index": 10,
      "offset_unit": "provider-defined",
      "source_ids": [],
      "source_chunk_indices": [],
      "raw": {}
    }
  ],
  "raw_payload": {}
}

The example URL and text are illustrative placeholders; the example does not represent a real API response. Generate source_id values separately for each response, and keep provider array indexes tied to that response. If you match a citation to a source-list entry by URL, also keep the citation’s URL and original annotation. The source list and citation metadata do not have to contain identical records.

For a broader discussion of preserving answer data and capture context, see designing a reproducible AI answer schema.

Validate the mappings before analysis

Record the response status and tool-call status first. Answer text alone does not show whether a search call completed. If the provider omits source or citation metadata, store an empty collection or mark the data as unavailable. Do not create records the provider did not return.

For Gemini, check that each groundingChunkIndices value is an integer pointing to an entry in the groundingChunks array. For both providers, confirm that citation positions are numbers at least zero. Check that they appear in the right order and match the associated text under that provider’s counting rules. Keep the original values if a check fails. Providers may count text positions differently.

Keep every source occurrence in the stored record, including the same URL returned by separate calls. Removing duplicates during ingestion loses evidence about where and how often each source appeared, though you can remove them later in a separate analysis view. If you standardize URLs or follow redirects, store the resulting URL separately from the one the API returned.

Test the parser with responses that have no search metadata or show search activity without citations. Test multiple text parts and multiple search calls as well. Include empty source arrays and invalid source indexes. Keep raw prompts and payloads only according to a defined retention policy because they can contain information users provided.

Check Gemini’s retention terms first

Google’s Gemini API terms restrict caching, analyzing, or collecting grounded results, search suggestions, and links for another purpose. This affects plans to store Gemini grounding links for later analysis. The schema above does not grant permission to build a lasting Gemini citation dataset. Before keeping those results, confirm that the terms allow your specific use or obtain written permission. The terms allow limited storage of some displayed grounded-result text, but that does not grant permission to archive links for analytics.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com

Email updates

Usually a new article and a few links I found interesting.

No spam. Unsubscribe with one click.

Leave a Reply

Your email address will not be published. Required fields are marked *