Currently Available: Need a skilled Software Developer for your next project?
Categories
APIs

Designing Webhook Retry Strategies and Backoff Algorithms

When a service sends a webhook, it makes an HTTP request to the URL where another service receives event notifications. The sender must decide what to do when the receiver is slow or unavailable. It must also decide how to handle an error response. A retry is another attempt to deliver the same event; backoff is the increasing wait between failed attempts.

A reliable sender limits retries and uses exponential backoff, which increases the wait after each failure, with random variation in the wait. It retries only failures its policy treats as temporary. A receiver must also expect duplicate deliveries: a timeout does not tell the sender whether the receiver processed the event before the response was lost. Make repeated events safe to handle, and check the sending provider’s documented retry behavior rather than assuming every sender retries.

Set retry limits and a delivery contract

A sender should define what counts as a successful delivery, which failures it retries, how long it waits for each response, and when it stops trying. It should also document what happens after retries end, such as exposing the failed delivery for manual replay or alerting the endpoint owner.

Exponential backoff increases the delay after each failed attempt. A common schedule sets a delay ceiling by doubling a starting delay after each failure, up to a configured maximum:

delay ceiling = min(maximum delay, base delay * 2^(failed attempts - 1))

Jitter adds a random amount to the schedule. With full jitter, the sender chooses a delay between zero and the calculated ceiling. This spreads out retries from senders that failed at the same time and reduces synchronized traffic spikes, known as a thundering herd. AWS describes how jitter spreads retry traffic in its guidance on timeouts and backoff.

Set a maximum attempt count, a maximum elapsed retry window, or both. A firm limit keeps a prolonged outage from creating an indefinitely growing backlog. The right delay and limit depend on how quickly the event matters, how much traffic the receiver can handle, and what the sender promises about delivery. No single schedule fits every webhook service.

If a receiver returns a Retry-After header, treat it as a signal for when to try again. The header specifies either a time or a number of seconds; the sender should not retry before the requested delay. RFC 9110 describes the header’s HTTP semantics, while the Standard Webhooks specification recommends considering it when scheduling a later attempt. A sender’s published contract should say whether and how it honors the header.

Retry failures according to what they mean

Do not apply the same retry rule to every failed request. A timeout, for example, means the sender did not receive a response in time. The receiver may still have completed the work, so the sender should retry only within its policy and the receiver should be ready to recognize the event as a duplicate.

Result Practical sender policy
Connection failure or timeout Count the attempt as failed and retry within the configured limits. Treat the outcome as uncertain because the receiver may have processed the event.
5xx response Retry with backoff when the sender’s contract treats the error as temporary. A server error does not guarantee that the next attempt will succeed.
429 Too Many Requests Slow down. If the response includes Retry-After, use it to schedule the next attempt.
3xx redirect Follow the sender’s documented policy. Standard Webhooks recommends treating redirects as failures and updating the configured endpoint URL.
410 Gone Consider disabling the endpoint if the sender’s contract gives this status that meaning. Standard Webhooks recommends this behavior.
Other 4xx responses Usually stop retrying an unchanged request and surface the failure for correction. Make explicit exceptions for statuses such as 429 and follow the provider’s rules.

The table is a starting point, not a universal HTTP-to-retry map. In particular, a sender should publish its handling of redirects and client errors, and endpoint owners should check that policy before diagnosing a failed delivery. HTTP also cautions against automatically repeating non-idempotent requests unless the sender has a way to know the retry is safe, as described in RFC 9110.

Make duplicate events safe

A receiver should use a stable event identifier to detect when it has already accepted or processed an event. If the same identifier arrives again, the receiver can avoid repeating the associated business action, such as charging a customer twice or creating a duplicate order. This property is called idempotency: repeating the same logical operation does not create an additional unwanted effect.

The receiver needs to keep deduplication information in storage that survives a restart. It should also distinguish duplicate deliveries from distinct event records that describe the same underlying change. Stripe recommends tracking processed event IDs and notes that some duplicate cases require considering both the object ID and event type. Stripe also warns that events can arrive out of order, so receivers should not assume that arrival order matches event order. See Stripe’s webhook guidance.

Deduplication does not remove every ordering problem. If later events replace earlier state, the receiver should use the event’s meaning and its own data rules to decide how to apply them. When the event does not provide enough information to make that decision, the receiver should verify the provider’s event semantics rather than infer an ordering guarantee.

Acknowledge after the receiver records the event

A receiver should return a success response once it has safely accepted responsibility for the event. A practical approach is to verify the request, store the event or put it on a durable queue, return success, and then handle longer business tasks in the background. A queue stores work for later processing, which helps a receiver absorb bursts without making the sender wait for every downstream action.

The receiver should not acknowledge an event before it has recorded it durably. Otherwise, a crash after the response can lose work that the sender believes was delivered. Conversely, waiting for lengthy processing can exceed the sender’s deadline and trigger another delivery even when the first attempt is still running.

Deadlines vary by provider. GitHub says its webhook deliveries fail if it does not receive a response within 10 seconds, and recommends a queue when processing cannot finish in that window. Standard Webhooks recommends a request timeout in the range of 15 to 30 seconds. These are different scopes, not interchangeable defaults. Check the sender’s deadline and keep the receiver’s acknowledgment path within it. Stripe also recommends handling events asynchronously.

Check the provider’s retry and replay rules

Webhook providers do not share one retry contract. Stripe documents automatic retries for live-mode events for up to three days, while GitHub says it does not automatically redeliver failed deliveries. GitHub documents manual redelivery and API-based options in its failed-delivery guidance. These policies show why a receiver should not assume a failed delivery will return on its own.

Before relying on a provider, check which responses count as success and how long its request deadline lasts. Check which failures it retries and how long it keeps retrying. Also check whether it offers replay. On the sender side, keep delivery history and provide a safe replay path for exhausted or manually recovered events. Replaying should use the same event identity so the receiver can apply its duplicate-handling logic.

Monitor delivery and processing separately

For each attempt, the sender should record the result and response status. It should also record how long it waited and whether it will try again. These records help an endpoint owner identify a slow receiver. They also show whether the receiver rejected the request or a network failure occurred. Alert on sustained delivery failures and make it clear when the sender has stopped retrying.

A receiver should separately track accepted events and completed business work. A successful HTTP response can mean the receiver stored the event, while a background worker may still fail later. Monitoring those stages separately helps operators find events that were delivered successfully but did not finish processing.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com

Email updates

Usually a new article and a few links I found interesting.

No spam. Unsubscribe with one click.

Leave a Reply

Your email address will not be published. Required fields are marked *