How to Roll Out a New LLM Model Version Safely
A new large language model (LLM) version is not a routine dependency update. It changes the behavior of the production system that…
DNS, or the Domain Name System, maps a hostname such as www.example.com to an IP address. A DNS time to live (TTL) is the period for which a caching resolver may retain a DNS answer before checking for updated data.
An expired TTL does not guarantee that the new server appears immediately. TTL applies to one cached record at one resolver. After the entry expires, the resolver must refresh it, but that lookup can still reach an outdated authoritative server, use stale data during an outage, follow cached delegation information, or be bypassed by a client-side cache. Troubleshooting requires identifying which layer still contains the old destination.
A TTL is a cache lifetime for a specific resource-record set, or RRset. An RRset is the group of records with the same name and type, such as all A records for api.example.com.
A recursive resolver normally stores an answer for no longer than the TTL published with that answer. Each resolver starts its own countdown when it retrieves the RRset. The countdown does not start when an operator edits the DNS zone.
DNS normally uses pull-based refreshes. An operator publishes new authoritative data, but DNS does not send a global invalidation message to every recursive resolver. Each resolver continues using its cached answer until that entry expires, then performs another lookup. The DNS caching model in RFC 1035 describes authoritative TTL values and the separate countdown used by cached records.
Changing the TTL does not affect every layer involved in name resolution:
A TTL is a maximum cache lifetime, not a required one. A resolver may discard a record earlier, as described in RFC 2181. Under normal conditions, however, an operator should not expect a compliant resolver to use an ordinary cached answer beyond its published TTL unless a stale-serving policy, local policy, or implementation defect applies.
Suppose api.example.com initially returns:
api.example.com. 3600 IN A 198.51.100.10
The sequence can look like this:
R retrieves 198.51.100.10 with a 3,600-second TTL.203.0.113.20 and publishes a 60-second TTL.R still returns 198.51.100.10. Its original cache entry does not expire until 12:59.R begins a new lookup.R returns the new address only if the authority path is reachable and serves consistent data.The 60-second TTL applies to resolvers that retrieve the new RRset. It does not retroactively shorten the 3,600-second entry stored at 11:59.
For a planned change, lower the TTL before changing the address. Wait through the previous, higher TTL before relying on the shorter value. Lowering the TTL at the moment of the address change does not provide that benefit.
TTL expiry triggers a refresh attempt. It does not determine which answer the resolver receives.
The normal sequence is:
The resolver can cache the old address again if the authoritative response still contains it. Common causes include:
Do not treat every delay as “DNS propagation.” Separate cache expiry, re-querying, authoritative publication, and authoritative inconsistency. Different recursive resolvers can refresh at different times and follow different authoritative servers, so users can receive different addresses during an incomplete deployment.
A zone is not fully updated until every authoritative server that can answer for it serves the intended version. A control-panel change confirms that the provider accepted the update, not that every serving node has deployed it.
DNS secondaries use the zone’s Start of Authority (SOA) record to discover changes. The SOA contains a serial number and timing values such as REFRESH and RETRY. A secondary checks for a newer serial and transfers the zone when it detects one. DNS NOTIFY, specified in RFC 1996, accelerates that process, but it is not an atomic worldwide publication event. A notified secondary still checks the SOA and transfers the zone only when the serial is newer.
Query every authoritative server directly:
dig +norecurse @AUTHORITATIVE_SERVER_IP example.com SOA
dig +norecurse @AUTHORITATIVE_SERVER_IP www.example.com A
dig +norecurse @AUTHORITATIVE_SERVER_IP www.example.com AAAA
dig +norecurse @AUTHORITATIVE_SERVER_IP www.example.com CNAME
Interpret the results as evidence:
A direct query with recursion disabled and the AA flag helps distinguish an authoritative response from a recursive-cache response. The ISC dig guidance provides examples of this testing method.
Changing a host address under the same DNS provider is simpler than moving a zone to different authoritative nameservers.
A nameserver migration can involve several independently cached records:
NS RRset.NS RRset.A and AAAA records.A and AAAA records for nameserver hostnames.DS records in the parent.DNSKEY records in the child.These records can have different TTLs and different administrative owners. The TTL for www.example.com A does not bound the time needed to replace the authoritative infrastructure for example.com.
During migration, keep the old authoritative servers operational for the maximum relevant parent and child TTLs. RFC 9199 discusses this approach because resolver behavior around parent and child delegation data is not completely uniform.
The parent zone stores the delegation that tells resolvers which nameservers serve the child zone. The child zone also publishes its own NS records. Those records can be cached independently and can have different TTLs.
Glue adds another dependency when a nameserver is inside the delegated zone. For example, if ns1.example.com serves example.com, resolvers need an address for ns1.example.com before they can contact it. A stale glue address can send a resolver to an old authoritative server even after the leaf record’s TTL expires.
Research summarized in RFC 9199 found measurable differences in resolver behavior. In the cited measurements, about 90% of resolvers followed the child-side view in the tested parent and child TTL scenario, while about 10% appeared parent-centric. The result does not predict every resolver, but it shows why a nameserver migration can outlast the apparent TTL of the changed hostname.
When changing authoritative infrastructure, inspect the delegation path, not only the final A or AAAA record.
A recursive resolver can intentionally serve stale DNS data when it cannot successfully refresh an expired answer. RFC 8767 defines this behavior as a resilience mechanism for failures such as authority outages, routing problems, denial-of-service attacks, or DNSSEC validation problems.
The normal decision path is:
Unexpired cache entry?
Yes: return it.
No unexpired entry:
Attempt an authoritative refresh.
Refresh succeeds?
Yes: cache and return the refreshed answer.
No: return stale data if the resolver policy permits it.
Stale serving is conditional. It is not a general permission to ignore TTLs whenever an old record remains in memory. The resolver first attempts to refresh the data, then may choose stale data when the refresh fails and its policy allows that choice.
BIND 9 provides one implementation-specific example. Its 9.16.25 configuration documentation describes configurable stale-answer behavior, including stale retention and a maximum stale TTL. Those settings are not universal DNS rules, and other resolver software can use different defaults or policies.
Resolver policies can also impose local minimum cache lifetimes. A DNS-OARC measurement tested more than 27,000 resolvers across 9,500 autonomous systems and reported that 8.74% of the measured subset arbitrarily extended TTLs. That sample does not establish a rate for every Internet resolver, but it demonstrates that published TTLs are not an absolute operational guarantee in all real deployments. The DNS-OARC measurement describes the methodology and result.
Two behaviors need separate explanations:
RFC 2181 treats TTL as a maximum cache lifetime, so early eviction is allowed. Post-expiry use normally requires a resilience policy, a local implementation policy, or behavior that does not follow the protocol. This distinction also explains why one client can see the new address before another client’s nominal TTL window ends.
DNS might already return the new address while the client continues to use the old one. Long-lived processes frequently reuse a resolved IP address without querying DNS for every connection.
Possible caching layers include:
Java illustrates the runtime-specific case. Java SE 26 networking properties documents positive and negative name caches, along with a stale name cache that can retain results after lookup failures. The exact behavior depends on the Java version and configuration, so it should not be generalized to every runtime.
Check for local overrides and alternate resolution paths:
/etc/hosts or the Windows hosts file.If the client process resolves the new address but traffic still reaches the old backend, DNS is not the remaining explanation.
An old response can come from an intermediary rather than name resolution. A CDN, reverse proxy, load balancer, service-discovery system, HTTP redirect, or persistent connection can still direct traffic to an old backend even when DNS returns the expected IP.
Record the destination used by the client process, not only the result of a command-line lookup. Then compare:
A new DNS answer combined with old traffic is evidence to investigate the application and network path. It does not prove that a public recursive resolver still holds stale DNS.
Use evidence from each layer instead of waiting for an assumed propagation period.
Record:
A, AAAA, or CNAME.AA flag.The resolver address matters because an internal forwarder, public resolver, browser, or enterprise encrypted-DNS service can produce a different result.
Run direct, nonrecursive queries against each authoritative server:
dig +norecurse @AUTHORITATIVE_SERVER_IP example.com SOA
dig +norecurse @AUTHORITATIVE_SERVER_IP www.example.com A
dig +norecurse @AUTHORITATIVE_SERVER_IP www.example.com AAAA
Compare SOA serials and record values. Any mismatch indicates incomplete publication or provider-side inconsistency. Do not use one authoritative response to represent the entire serving fleet.
If nameservers changed, query both parent and child data:
dig +trace example.com NS
dig +norecurse @PARENT_AUTHORITATIVE_IP example.com NS
dig +norecurse @CHILD_AUTHORITATIVE_IP example.com NS
Repeat the checks for glue addresses, DS, DNSKEY, A, AAAA, and CNAME records when those records participate in the change. Stale delegation or glue can keep resolvers on the old authority path.
Test the affected network’s recursive resolver, internal forwarders, selected public resolvers, and any encrypted-DNS resolver configured by the client.
Compare each answer with the direct authoritative responses:
If authoritative data is correct but a resolver still returns the old address after the expected cache window, inspect resolver configuration and logs for serve-stale behavior. Also check whether the resolver can reach the authoritative servers and whether DNSSEC validation or upstream forwarding is failing.
Do not infer stale serving from a low response TTL alone. Resolver implementations differ, and RFC 8767 defines stale answers around failed refreshes rather than ordinary cache use.
Restart or instrument the affected process and compare its resolved destination with independent resolver results. Inspect runtime DNS settings, local caches, hosts-file overrides, connection reuse, proxies, and load-balancer logs.
If all authoritative and recursive answers are new but the process still reaches the old endpoint, the remaining fault is probably in the client or traffic-routing path.
dig CommandsThese commands compare authority, delegation, and recursive results without claiming that one query proves global convergence.
Query each authoritative server:
dig +norecurse @AUTHORITATIVE_SERVER_IP example.com SOA
dig +norecurse @AUTHORITATIVE_SERVER_IP www.example.com A
Inspect the delegation path:
dig +trace example.com NS
dig +norecurse @PARENT_AUTHORITATIVE_IP example.com NS
dig +norecurse @CHILD_AUTHORITATIVE_IP example.com NS
Query the recursive resolver used by the affected client:
dig @RESOLVER_IP www.example.com A
dig @RESOLVER_IP www.example.com AAAA
dig @RESOLVER_IP www.example.com CNAME
For an authority migration, repeat the relevant checks for:
NSA and AAAADSDNSKEYA and AAAA records for the serviceCompare the answer, remaining TTL, flags, and source resolver. A direct authoritative query with recursion disabled should normally show the AA flag. A recursive response generally does not indicate that the queried resolver is authoritative for the zone.
For a planned host-address change:
A, AAAA, CNAME targets, aliases, and DNS-based traffic-management records.For a nameserver or delegation migration, keep the old authoritative infrastructure operational for the maximum relevant parent and child TTLs. Include nameserver addresses, glue, delegation records, and DNSSEC data in the migration plan.
Long TTLs reduce authoritative query load and can improve lookup performance, but they reduce migration agility. Research on DNS TTLs found that longer TTLs can substantially reduce lookup latency in measured environments, which explains why operators use them in stable configurations. The ACM Internet Measurement Conference study documents this performance trade-off.
Do not promise a universal convergence time. Publication delays, independent cache age, delegation state, resolver policies, authority failures, and application caches exist outside the zone owner’s control.
For a hard failover deadline, combine DNS with endpoint overlap, health-aware traffic management, a load balancer, or another mechanism that does not depend solely on cache expiry.
An expired DNS TTL means that one cached answer is ready for refresh. It does not mean that every resolver has discarded the old address or that the next lookup will reach a consistent, reachable authority.
Check the authoritative servers first, then delegation and glue, recursive resolvers, stale-serving policies, and finally the client and backend-routing path. Plan cutovers with lower TTLs in advance and keep the old endpoint available long enough to cover the relevant caches and migration dependencies.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com