Designing a Webhook Replay Strategy

When a provider sends a webhook, it makes an HTTP request to notify your service about an event, such as a changed order or account. A delivery can fail before your service acknowledges it, or your service can acknowledge it and then fail while doing the work. A replay deliberately sends or processes an earlier event again to recover from either kind of gap.

A replay strategy needs enough information to identify and verify each event. It must prevent duplicate business effects and check that an event still makes sense to apply. It also needs to distinguish replays from automatic retries and compare local records with the provider's current data when an old event is missing or out of date.

Retries and replays solve different failures

A retry is another delivery attempt made under the provider’s rules, usually after a timeout, connection failure, or unsuccessful HTTP response. An operator or recovery service starts a replay to recover an event. The replay can ask the provider to deliver the event again, resend a message held by a webhook service, or process an event that the receiver already stored.

Providers handle failed deliveries differently. GitHub does not automatically redeliver failed deliveries, while Stripe and Shopify document automatic retries. Providers also differ in how operators can recover deliveries and how long they keep them. Stripe, for example, documents Dashboard resends for up to 15 days and CLI resends for up to 30 days. Check the provider’s current documentation before relying on its retry schedule or replay window.

The failure point determines how to recover. If the receiver does not return a successful response, the provider can try the delivery again. But if the receiver acknowledges the request before finishing its business work, the provider can count the delivery as successful even when that work fails. Shopify’s guidance recommends reconciliation as a backup for missed data, addressing the gap between acknowledging a delivery and completing its processing.

Record processing results separately from HTTP responses. A successful response tells the provider that the receiver accepted the delivery under its rules. It does not prove that the receiver completed the intended business action.

Preserve the event and its processing history

A replay record needs to show which event arrived, whether it passed the provider’s checks, and what the receiver did with it. Store the exact request body as raw bytes alongside a parsed payload. To verify a provider's signature, a value used to check a request's source and integrity, the receiver may need the original bytes. Both Stripe and Shopify document raw-body requirements for verification.

Store relevant request headers and the content type, along with the provider and the account or tenant that received the event. Also record the endpoint or subscription, event type, and action so the receiver can send the event to the right handler and integration. Preserve the provider’s event ID and delivery ID when available. An event ID identifies an event; a delivery ID may identify one attempt or a delivery to a specific subscription. The distinction matters: Shopify documents separate delivery IDs for related deliveries that share an event ID, while GitHub documents a delivery identifier that remains the same for a requested redelivery.

Record when the event occurred, when the receiver got it, and when each delivery attempt happened. Also record the payload or API version when applicable, the verification result, and the processing history. Timestamps help diagnose delays but do not show the order in which events should be processed. Stripe warns against using its event creation timestamp to determine processing order. Store signing secrets and other credentials in a secret-management system, not in event records.

Follow the provider’s rules for checking signatures. A signature helps verify where a request came from and whether it was changed, but it does not prevent duplicate business effects. An old signature also cannot be treated as a new live request. Stripe, for example, generates a new signature and timestamp for each retry and documents a default five-minute timestamp tolerance in its libraries. For a local replay, follow the provider’s rules for the original request and the receiver’s own access controls.

Prevent duplicate effects before replaying

A provider can deliver an event more than once. A receiver can also finish the work but lose the acknowledgment. Plan for duplicate delivery attempts.

Use stored identifiers to recognize repeat deliveries, and make the business operation idempotent. That means applying the same intended change again does not create an extra effect, such as a second payment or duplicate notification. Choose an identifier based on what needs deduplication: a delivery ID can distinguish separate attempts, while an event ID identifies the underlying event. The provider’s rules determine which identifier fits.

Make the duplicate check part of a reliable way to record the work. If the receiver marks an event as handled before it safely records the work, a crash can cause it to discard a later replay even though the work never happened. Record the event and the work it should trigger together, or prevent a failure between those steps from losing the work. For consequential actions, use a business-level idempotency key when provider identifiers alone do not prevent duplicate effects.

Do not use timestamps to identify duplicates. Different events can have the same timestamp, and providers do not guarantee delivery order in every context. Stripe also notes that separate event objects can sometimes describe changes that look like duplicates. When two events look alike, check their identifiers and the current resource data instead of assuming they are the same event.

Decide whether an event is safe to replay

A valid payload can still be unsafe to apply. Before replaying it, check that the event belongs to the right provider account and integration, that its signature passes the provider’s checks, and that the receiver recognizes its event type and action. GitHub recommends checking both event type and action because they determine what the delivery represents.

Check whether the receiver already processed the event and whether the payload still describes a change worth applying. If an old event says a resource changed but the handler needs the resource's current state, fetch that state from the provider or compare it with local records before applying the event. Stripe documents fetching objects through its API, and Shopify recommends API-based reconciliation when webhook deliveries leave gaps.

Check what happens if the handler runs twice. Before replaying a batch, make sure an event that issues a payment, sends an email, changes access, or triggers another command has duplicate protection or gets reviewed. Also check whether event order matters. If one event depends on an earlier change, do not assume the events were created in the order they should be applied. Use resource versions or the provider’s current state when available, or reconcile instead.

Set a policy that places events in one of three groups: safe to replay automatically, replay only after checking the provider’s current state, or require manual review. For example, a notification that refreshes local data can be replayed automatically if the handler prevents duplicate work. An event that triggers an irreversible financial action needs stronger checks. The event’s meaning and the receiver’s business rules determine its group.

Use reconciliation when replay cannot recover the state

Replay processes a particular event or delivery again. Reconciliation compares local records with the provider’s authoritative data and fixes any differences. Use reconciliation when the provider’s replay window has expired, the receiver never stored the event, or the old payload no longer describes the resource’s current state.

Each provider sets its own replay window. GitHub’s documentation describes redelivering recent deliveries from the past three days. Stripe sets different resend limits for different methods, and Shopify recommends importing missing data through APIs after extended downtime. Provider history is not a complete archive. Choose how to reconcile important data, such as fetching resources changed since a saved checkpoint and comparing them with local records.

Control each replay. Start with a small batch and record which events you selected and why. Monitor delivery responses and business-processing results. If the replay causes unexpected effects, pause it and investigate before sending more events. In the replay log, record the operator or recovery job, event identifiers, attempt results, and any business actions that followed.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com