Building a Webhook Incident Runbook
A webhook is a message that a provider sends to your service when an event occurs, such as a payment update or…
Categories
Categorizing Webhook Failures for API Consumers
When a service sends a webhook, it makes an HTTP request to an endpoint you control to report an event, such as a changed account or a completed payment. Your endpoint receives the request and returns a response. The service sending the webhook acts as the HTTP client; your endpoint acts as the server.
Webhook failures fit into six useful categories: transport, authentication, payload, timing, provider retry state, and terminal failure. The categories help you identify what happened and what to check next. They are an organizing framework, not a universal standard: HTTP standards define response semantics, while each provider sets its own delivery, retry, and subscription rules.
Separate the delivery result from the provider’s next action
A failed delivery attempt describes an outcome, such as a connection error, an invalid signature, or a timeout. The provider’s retry state describes what happens next. Providers may take different actions after the same outcome.
HTTP Semantics, RFC 9110 defines response-code classes: 2xx responses indicate success, 4xx responses indicate a client error, and 5xx responses indicate a server error. Those meanings do not establish a universal webhook retry policy. Check the provider’s documentation and delivery records before assuming that a particular status code triggers a retry or ends delivery.
The categories follow a delivery from the connection through the provider’s eventual decision. A single attempt can involve several of them: for example, a slow endpoint can time out, the provider can schedule another attempt, and repeated failures can eventually stop delivery.
Transport: could the provider reach the endpoint?
A transport failure occurs when the provider cannot connect to your endpoint or finish sending the request. A DNS lookup failure, which prevents the provider from finding the endpoint’s network address, a refused connection, or an invalid security certificate can stop the request before it reaches the code that handles webhooks.
The provider may report a network or connection error without an HTTP response because the endpoint never returned one. GitHub documents connection failures when a receiver refuses a connection, while Slack includes connection and certificate-validation errors among its delivery failures. Check the provider’s delivery log for the reported failure, then verify that the endpoint is reachable and its certificate is valid.
Authentication: does the request prove its origin and integrity?
Webhook authentication helps your endpoint check whether a request came from the expected provider and whether anyone changed its body. Some providers use a digital signature: the provider calculates a value from the request body and a secret shared with the consumer. The consumer calculates a value from the received body and checks whether it matches. An HMAC is a hash that uses a secret key for this check.
A failed signature means the request did not pass the configured check. That failure alone does not show whether the secret was wrong, the algorithm differed, the request body changed, or something else failed to match. GitHub’s signature-validation guidance lists the secret, algorithm, header, encoding, and body modification as points to check.
Provider mechanisms vary. For example, Shopify’s HTTPS webhook guidance uses an HMAC-SHA-256 signature in the X-Shopify-Hmac-SHA256 header and requires verification against the raw request body. That rule applies to its HTTPS delivery method; Shopify documents different delivery methods as well. Use the verification method documented for the provider and delivery method you configured.
Payload: can the consumer parse and interpret the event?
A payload failure occurs when the request reaches your endpoint but the consumer cannot use its body. The consumer might be unable to read the body’s format. It might read the body but not recognize the event type or fields. Or it might understand the event but fail while carrying out the related business logic. These distinctions help diagnose the failure, but providers do not use them as a universal classification.
An unknown event type does not necessarily mean the payload is malformed. Providers add event types and actions, and payload structures can depend on product versions. GitHub documents distinct webhook event types and payloads; Stripe documents that an event’s structure depends on the API version associated with the webhook endpoint or account. Check the event type and version before treating unfamiliar fields as evidence of invalid data.
GitHub documents a 25 MB payload cap and says it will not deliver a webhook payload when an event generates a larger payload. When an expected event is missing, check the provider’s delivery record and payload limits rather than assuming the consumer rejected it.
Timing: did the endpoint acknowledge in time?
Providers set their own deadlines for receiving a successful response. If your endpoint takes too long to respond, the provider can record the attempt as timed out even if your code eventually finishes its work.
The documented deadlines differ by provider and product:
| Provider and scope | Documented response timing |
|---|---|
| GitHub webhooks | GitHub expects a 2xx response within 10 seconds. |
| Shopify HTTPS webhooks | Shopify documents a 1-second connection timeout and a 5-second timeout for the full request. |
| Slack Events API | Slack expects a valid 2xx response within 3 seconds. |
These limits come from GitHub’s troubleshooting guidance, Shopify’s delivery-verification guidance, and the Slack Events API documentation. They are provider-specific rules, not recommended universal thresholds.
Acknowledgment time is separate from processing time. An endpoint can acknowledge receipt before completing longer-running work; Slack and GitHub recommend keeping webhook responses within their deadlines rather than waiting for all downstream work to finish. Also distinguish a slow response from a late or out-of-order event. GitHub says deliveries can arrive out of order, and Shopify says delivery order is not guaranteed. If sequence matters, use event details and provider-specific recovery guidance to determine how to reconcile changes.
Provider retry state: what happens after a failed attempt?
A provider’s retry policy says whether it retries a failed delivery. It also sets how long the provider waits and whether the consumer must request another delivery. The provider sets these rules; HTTP status-code classes do not.
| Provider | Documented behavior |
|---|---|
| Shopify webhooks | Shopify retries failed calls up to eight times within four hours, with increasing intervals between failures. |
| Slack Events API | Slack retries a failed request up to three times: nearly immediately, then after one minute, then after five minutes. It includes headers that identify the retry number and reason. |
| GitHub webhooks | GitHub does not automatically redeliver failed deliveries. Consumers can manually redeliver them or arrange their own process to identify failures. |
The Shopify retry guidance, Slack Events API documentation, and GitHub failed-delivery guidance describe different policies. Read the relevant provider’s delivery record to find out whether an attempt is scheduled for retry, requires manual redelivery, or has entered another state.
A timeout does not prove that your endpoint did no work. Your endpoint might process the event before the provider receives its acknowledgment. The provider can then send the event again. Shopify recommends handling duplicates or identifying them by delivery ID. Use the method your provider documents for identifying duplicates.
Terminal failure: has the provider stopped delivery?
Delivery has failed terminally when the provider has used all its attempts, stopped sending events, or disabled or removed the subscription. This describes what happened to delivery, not whether the underlying business event is final.
The provider’s terminal behavior depends on its product and configuration. Shopify documents subscription removal after persistent failures; its verification guidance specifies automatic deletion after eight consecutive failures for subscriptions configured through the Admin API. Slack documents temporary disabling when more than 95% of delivery attempts fail over 60 minutes, with an exception for apps receiving fewer than 1,000 events per hour. Slack requires a manual action to re-enable delivery after the endpoint is repaired. GitHub’s documented approach relies on manual redelivery rather than automatic retries. See Shopify’s retry guidance, the Slack Events API rules, and GitHub’s redelivery instructions.
After the provider stops delivery, check whether it allows you to send events again and how to find events the consumer missed. Shopify recommends reconciling webhook data with its APIs because it does not guarantee delivery. Recovery depends on the provider: neither HTTP nor a shared webhook standard guarantees that every event will arrive.
What I'm buildingDelegate tasks. Get software.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com