How to Maintain AI Workflows During System Changes
A business AI workflow connects an AI model to documents and business software, then uses the model’s output to do things such…
When a business uses AI to answer questions from its own documents and records, outdated or conflicting information can appear in the answer as if it were current. In a common setup called retrieval-augmented generation (RAG), a search tool finds relevant passages in company content and gives them to a language model before it answers. The search index, a searchable copy of that content, can still contain an old policy. Or two active pages can disagree. The model may use either source without knowing which one is right.
Check freshness and authority separately. Freshness tells you whether the AI’s copy matches the source and whether the information applies to the time period in the question. Authority tells you which source governs a particular fact. Track both freshness and authority. Compare sources that make the same claim, then decide what the AI should do when it cannot resolve a disagreement.
A file’s “last modified” date does not tell you when its contents became valid. Someone might edit a page yesterday even though it describes a policy that has been in force for years. Or someone might change a policy today, while the AI index still shows last month’s version.
For each important source, track when someone last changed it and when its information applies. Also record when the AI index last processed it. The dates that show when information applies are called effective dates. Record an effective-from date and, if the information expires, an effective-to date. Keep these dates separate from the source’s edit time and the index’s processing time.
Measure how long it takes the index to update after someone changes a source. Set a maximum delay for each kind of information based on how quickly it changes and what an outdated answer could mean. For example, a frequently changing operational status needs a shorter refresh delay than a historical report. Microsoft’s guidance on grounding data recommends measuring this delay against the needs of the workload. It does not set one freshness threshold for every business.
Check that the software copying content to the index also handles removals and replacements. An old page can remain searchable after someone deletes or replaces it if that software does not pass the change to the index. Review indexing logs for failed or incomplete updates. Check that renamed, moved, and deleted documents no longer appear in search results. A recent successful indexing run does not prove that every change reached the index.
Store enough metadata with each piece of content for a person or search tool to judge whether it applies. Metadata is information kept alongside the content, rather than in its main text. Start with fields that answer practical questions: Where did the content come from? Who owns it? Is it approved? Which revision is it? When does it apply? When did the AI index it?
A compact record might include:
| Field | What it helps establish |
|---|---|
| Source location and identifier | Which original page, file, or business record supplied the content |
| Owner and status | Who can verify it, and whether it is approved, draft, superseded, or archived |
| Version and effective dates | Which revision applies and for what period |
| Source-change time and indexed time | Whether the AI’s copy has caught up with the source |
| Scope | Which product, location, customer group, or business unit the content covers |
| Supersedes or is superseded by | How this revision relates to an earlier or later one |
Require owners to fill in the important fields when they publish or approve content. Check that dates are present and status values are familiar. Also look for records marked current even though a newer version has replaced them. Incorrect metadata can make an outdated source look valid, so filters work only when the fields they rely on are accurate. Microsoft’s data-platform standards describe practices such as assigning owners and documenting intended use. They also describe tracking lineage, which records where data came from and how it changed.
Do not make “newest document wins” the company rule. A recent draft or copied FAQ may be less authoritative than an approved policy or the business record that governs the fact in question. Recency helps only after you establish that the sources cover the same subject, scope, and period.
Define authority by information type. An approved handbook might govern employee policy, while a payroll record governs a particular employee’s current pay status. Operational instructions may explain how staff carry out a policy, but they should not silently override it. Drafts and unreviewed notes should not support definitive answers about current rules; archived material should support historical questions only.
A retrieval system can rank newer material more highly, but a ranking preference does not exclude old material. Microsoft’s freshness-aware retrieval documentation describes freshness as a ranking bias and distinguishes it from a filter that limits results to a date range. For a “current approved policy only” answer, filter out drafts and superseded versions using accurate status and effective-date fields. Ranking and filtering features vary by product; check the documentation for the search service you use before relying on a particular control.
Compare claims by subject; matching words alone do not establish a conflict. To check for a conflict, identify what the claim is about, what it says, and where and when it applies. For example, two pages with different return periods conflict only if they cover the same product, customer group, and effective period. If one applies to an older product version, both statements may be valid.
For structured records, start with checks that follow fixed rules. Find duplicate identifiers with different values and check for overlapping effective dates. Also flag missing validity dates and records marked “current” even though their status says they have been superseded. For prose pages, text comparison or an AI review can flag passages that appear to disagree. Have a person verify those results. Similar wording does not prove that two passages refer to the same product, location, or time period.
When a reviewer resolves a conflict, preserve the sources, the chosen authority, the reason for the decision, and who approved the correction. This record helps prevent the same disagreement from returning after the next index update. Research published at ACL in 2025 found that outdated information alongside current information can reduce answer accuracy and mislead retrieval-based systems, even when current information is available. The finding supports removing or clearly separating stale content; it does not establish a single conflict-detection method for every business.
Write the rules for handling disagreements into the search settings or the instructions for the people who maintain them. If one approved source clearly governs the claim and matches the question’s scope, use it and flag the conflicting, less authoritative source for correction. If different versions apply to different periods, retrieve the version that was valid on the date in the question. Use historical evidence for a historical question, rather than automatically choosing the newest document.
If two credible sources disagree about the same scope and period, the AI should report that the records conflict and refer the question to the named owner. It should not silently choose one. If the question lacks the detail needed to choose between different scopes, ask for clarification. If required dates or approval status are missing, treat the content as unverified for a consequential current answer and use an approved fallback or request review.
For an urgent correction, remove the affected content from search or quarantine it so the AI cannot retrieve it. Correct the source, update the index, and test that the obsolete version no longer appears. Keep a record of what the AI retrieved, including source versions and timestamps, so a reviewer can trace an answer to its evidence.
Choose one area where outdated information would cause a clear business problem, such as customer service policy or operating procedures. Name the source that governs each important claim and the person responsible for correcting it. Add the essential metadata, set a source-to-index delay that fits the information, and create checks for duplicate or conflicting values.
Test the setup with a superseded page, a conflicting source, missing effective dates, and an index update that is late. Confirm that the search selects the valid source or shows the unresolved issue. A source register in a spreadsheet and a few automated checks are a reasonable starting point. Fit the controls to the specific risk and search service.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com