Currently Available: Need a skilled Software Developer for your next project?
Categories
AI in Business

How to Maintain AI Workflows During System Changes

A business AI workflow connects an AI model to documents and business software, then uses the model’s output to do things such as update a record or route a request. If any connected part changes, the workflow may return a different answer, reject data it once accepted, or lose permission to act.

Treat each deployed workflow as a software integration with a recorded version. Record the inputs and settings that shape its behavior. Test changes before using the workflow for consequential work, and decide in advance what to do if a check fails. A regression test compares a changed workflow with expected results from representative cases. These records and tests let an operator identify the change that caused a failure and decide whether to approve, pause, or roll back the workflow.

Recheck authoritative documents when their contents change

When a workflow relies on a policy, procedure, price list, or other authoritative document, track the exact revision it uses. A filename or modified date alone may not show whether the document’s text changed. A content fingerprint is a calculated value that changes when the file’s contents change. The document repository’s revision identifier also gives the workflow a more reliable way to detect changes.

After a document changes, refresh any extracted text or search index the workflow uses. A search index is a searchable copy of document content, and changing the original file does not necessarily update that copy. Compare the new revision with the approved one, then judge how the change affects the workflow. A typo correction may leave the answer unchanged, while a changed eligibility rule, threshold, or definition may alter decisions.

Agree with the authoritative document’s owner on which changes make an earlier approval no longer valid. For material changes, test the workflow against the new revision and get the owner’s approval before using it for consequential outputs. No universal rule requires approval after every edit, so the organization must decide which changes are material. The NIST AI Risk Management Framework’s management guidance supports change management and post-deployment monitoring without prescribing a document-specific approval threshold.

Validate connected fields and permissions

A connector passes data between an AI workflow and another service. The interface contract specifies field names and data types. It also defines required values, allowed choices, and response fields. Check data against that agreement before the workflow writes to a business record. If a required field disappears or changes type, stop the workflow before it sends incomplete or malformed data to the next service.

For example, a workflow might read a customer status field and send a summary to a case-management system. If the connected service renames the field, the workflow should report a contract error and hold the case for review instead of silently sending a blank value. Microsoft documents this kind of coupling in Power Apps and Power Automate: changes to a flow’s connections or outputs can break apps that depend on them. This behavior applies to those products, but it shows why operators should test connected interfaces before releasing changes.

Check permissions separately from whether the fields match. An authentication error means something different from a missing field. Microsoft’s Power Automate guidance maps 401 errors to authentication problems. It maps 403 errors to permission or policy blocks. It maps 404 errors to resources that were deleted, renamed, or moved. These codes are specific to that product; for other platforms, check the connector’s documentation.

If a permission or schema check fails, stop later actions that would write data and alert the person responsible for the connection. Retrying an unauthorized request will not restore access. For consequential work, send the case to a person or follow a documented backup procedure. For guidance on preventing duplicate actions when a failed run is replayed, see handling retries and changed inputs.

Test model changes against the same cases

Record the model identifier and, if the provider supports it, the exact model version used in production. It identifies the service or version that generated the answer. A version alias is a label that points to a version, and the provider can change which version the label points to. Google’s documentation explains that aliases are mutable, so a label such as production does not by itself identify an immutable model version.

Before switching models, run the same representative test cases through the current and proposed versions. Compare the results with criteria set for the business task. For document extraction, the criteria might check whether the workflow finds all required fields and extracts them correctly. For routing, they might check whether the model chooses the right destination and handles exceptions as expected. Include difficult cases and past failures, not just routine examples. OpenAI’s evaluation best practices recommend evaluating workflows as they change and adding newly observed cases to the test set.

A test score shows how the candidate performed on the cases tested, without establishing that every result will be correct. Keep the test cases, criteria, and results with the proposed release. An operator can then compare later changes against the same baseline. Also watch for provider retirement notices. Pinning a version makes it easier to trace results, but does not guarantee that the provider will keep offering that version. Google’s model lifecycle documentation describes retirement and migration information for its products.

Evaluation tools also change over time. OpenAI states that its Evals platform became read-only for existing users on October 31, 2026, and is scheduled to shut down on November 30, 2026. Those dates apply to that platform, not to evaluation practices in general. Keep test cases and results in a format or location you can still use if an evaluation service changes.

Validate structure and business rules separately

When a model sends output to software, define the required structure and check it before taking action. A schema describes the fields and data types the output must contain. For example, it can require a case identifier and limit a status field to allowed values.

A correctly structured answer is not necessarily true, and its suggested action is not necessarily appropriate. OpenAI’s Structured Outputs documentation explains that its feature limits completed outputs that are not refusals to a supplied JSON Schema. The software must still handle refusals, incomplete responses, API errors, and business rules. Check factual claims against the source document, and use regular code or human review to enforce exact thresholds, calculations, and eligibility rules.

Keep a release record and a safe fallback

Keep a release record that links each production result to the settings that produced it. Record the source-document revision and model version. Include the prompt and configuration version, plus the connector version. Also record the expected input and output structure and the relevant connection reference. Add the test results and approval status. This is one way to implement change tracking, not a required standard. An operator can compare a failed run with the release record to narrow down what changed.

Decide what the workflow should do when each check fails. It might pause, save the input for review, or send the case to a person. It might also use an approved non-AI procedure or return to a previously approved configuration. If the workflow writes to another service, record whether each write succeeded before replaying a run. Without that record, a retry could repeat the same update. NIST’s management guidance recommends planning for monitoring, response, recovery, and contingency procedures, while leaving the specific fallback design to the organization.

For workflows with greater consequences, test the fallback too. Confirm that a paused case reaches someone who can resolve it and that the original input remains available. Also verify that a failed authorization check stops the workflow from writing data. Choose the fallback based on the business consequences of delay or error. Assigning ownership when an AI workflow fails covers the related question of who handles that handoff.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com

Email updates

Usually a new article and a few links I found interesting.

No spam. Unsubscribe with one click.

Leave a Reply

Your email address will not be published. Required fields are marked *