Turning Webhook Docs Into an Incident Checklist

When a provider sends an event notification to your service, it sends a webhook, an HTTP request to an endpoint (the URL that receives the request). If delivery fails, the provider’s documentation says whether it will retry or stop trying. It also explains whether it queues the event or disables the subscription. Your endpoint acknowledges receipt by returning an HTTP status code the provider accepts.

During an incident, use the provider’s documentation to find four rules: how it retries, how quickly your endpoint must acknowledge a delivery, what status or consequence it records after a failure, and how to recover missed deliveries. Record those rules in a checklist for that provider before replaying events. A successful acknowledgment confirms delivery under the provider’s rules; it does not necessarily mean your code finished processing the event.

Confirm which webhook contract applies

First identify the provider, product or event API, environment, endpoint, API version, and subscription type. A provider can apply different rules to different products or environments. For example, Shopify’s subscription guidance distinguishes app-specific subscriptions from shop-specific subscriptions, which behave differently after repeated delivery failures.

Then find the provider’s official pages. Look for troubleshooting and delivery attempts, along with information on timeouts, retries, redelivery, and recovery. Search results can point you to a page, but use the documentation itself to verify the instructions. If a value is missing, record it as “not specified in the documentation reviewed.” Do not fill a gap with another provider’s limit or a guessed default.

Record retries, deadlines, failure states, and recovery controls

A retry policy explains whether the provider sends a failed event again, how many times it tries, and when it stops. The response deadline is how long your endpoint has to acknowledge a request. The failure state is the status the provider records, or what it does, after one or more failed attempts. A recovery control is a documented way to restore delivery or recover missed data, such as a dashboard button, an API operation, a setting, or a reconciliation procedure.

Provider rules vary. Slack’s Events API requires a response within three seconds and documents up to three retries. If failures exceed 95% of delivery attempts within 60 minutes, Slack can temporarily disable subscriptions, subject to an exception for apps receiving fewer than 1,000 events per hour. The documented recovery is to re-enable event subscriptions after correcting the problem. See Slack’s Events API guidance.

GitHub expects a 2XX response within 10 seconds and does not automatically redeliver failed deliveries. Its delivery history covers recent deliveries for three days. The redelivery instructions describe manual recovery. Shopify’s troubleshooting documentation describes a five-second response deadline and up to eight failed attempts in a four-hour period. Shopify’s subscription guidance qualifies what happens after persistent failures: app-specific and shop-specific subscriptions have different removal behavior.

Adyen marks an endpoint Failing and queues messages when it does not receive an acknowledgment within 10 seconds. Its troubleshooting guidance says the retry queue can continue for up to 30 days, and operators can retry or ignore a queued message in Customer Area. Stripe documents retries for live-mode events for up to three days. Sandbox events receive three attempts over a few hours. Its reviewed webhook guidance does not specify a numeric response deadline; dashboard resends are available for events up to 15 days old, while Stripe CLI resends are available up to 30 days. See Adyen’s troubleshooting guidance and Stripe’s webhook documentation.

These examples show that each provider sets its own rules; they are not shared defaults. For each incident, record the exact rule for the affected product and environment. Also separate the HTTP result of one delivery attempt from the provider’s current status for the endpoint or event. A 500 response, for example, describes an attempt; a status such as Failing or Failed describes a provider-tracked state.

Turn the documented rules into a checklist

Fill in provider-specific values before using this checklist. If the documentation gives no numeric value, state that explicitly and verify whether another official page applies.

• [ ] Confirm the provider, product or event API, environment, API version, endpoint, and subscription type.
• [ ] Record the affected event type and delivery ID, plus the first failure time and most recent attempt.
• [ ] Record the retry policy: automatic or manual, attempt schedule or duration, and when retries stop.
• [ ] Record the acknowledgment deadline and accepted status codes. Mark a deadline as “not specified in the documentation reviewed” rather than guessing.
• [ ] Check the provider’s delivery log or dashboard. Capture the current event or endpoint state, response code, attempt number, and any error details.
• [ ] Record what the provider does after failures, such as stopping retries, queuing an event, disabling an endpoint, or removing a subscription.
• [ ] Find the documented recovery control and confirm its permissions, eligibility window, and exact action.
• [ ] Verify that the endpoint is healthy and can acknowledge within the documented deadline before replaying events.
• [ ] Check how your code handles duplicate events and event ordering before replaying.
• [ ] Replay or retry the delivery using the provider’s documented control. If events were exhausted or a subscription was removed, follow the provider’s documented restoration and reconciliation steps.
• [ ] Confirm that new deliveries succeed, the provider’s state reflects recovery, and the business processing completed.
• [ ] Check for remaining failed, queued, delayed, or missing events; record any gap and the evidence for the recovery action.

The checklist treats getting the provider to send events again separately from recovering missing data. A provider may let you replay a delivery without restoring a removed subscription, or stop retries before your service processes every event. Shopify, for example, advises reconciliation because webhook delivery is not guaranteed. If a provider does not document a replay control for the failure you see, verify whether it offers another recovery path, such as restoring the subscription or importing missing data.

Acknowledge first, then process safely

A provider’s acknowledgment deadline can be shorter than the time your code needs to finish business work. Adyen recommends storing a verified message and acknowledging it before processing it afterward. GitHub also recommends using a queue so the receiver can respond promptly and process events later, outside the request that received them. Apply that pattern only if your own receiver can reliably store the event before it acknowledges receipt.

Replay can send an event your code has already handled. Check whether your receiver tracks event IDs and prevents duplicate processing; Stripe recommends tracking event IDs, and GitHub states that a delivery identifier stays the same on redelivery. Also check ordering: Stripe says events are not guaranteed to arrive in order. A successful replay does not prove that your code processed every related event once and in the correct order.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com