Routing Failed Webhooks to an Error Endpoint
When a provider sends a webhook, it delivers an event to an HTTP endpoint operated by the receiving service. If the endpoint…
Categories
Designing a Webhook Observability Model
A webhook is an HTTP request a provider sends to a URL you control to report an event, such as a changed record or completed payment. Observability means keeping enough evidence to see what the provider sent, how your receiving endpoint responded, whether later processing succeeded, and how an operator recovered from a failure.
Keep three separate records: the event, the delivery to a particular subscription or endpoint, and each attempt to send it. Preserve the provider’s identifiers and raw results. Then link those records to your own processing records with a local correlation ID, an identifier that connects related records. This separation makes retries, duplicate notifications, and manual replays easier to investigate, even when providers use different identifiers and recovery rules.
Read the provider’s delivery contract first
Before choosing fields or setting alert thresholds, check the documentation for the specific provider and webhook subscription. Check how the provider identifies events and deliveries, and which response it accepts as an acknowledgment. Then check its retry period and whether it offers redelivery or replay. Also check how long its logs remain available and whether repeated failures can disable a subscription.
Store these details with each delivery so an operator can interpret the record. Shopify’s troubleshooting documentation, for example, describes delivery logs covering the past seven days and warns that the logs can be delayed by several minutes. GitHub says a delivery can take a few minutes to appear in its recent-deliveries log. A provider dashboard provides useful evidence, but it is not a complete, real-time record.
For each delivery, record the subscription or endpoint and the payload version, which determines the structure of the data the provider sends. Shopify delivery details include a subscription ID and API version; Stripe notes that an event’s payload structure is tied to the version in effect for that event. This information helps an operator tell whether a delivery failed or the endpoint configuration or payload format changed. Provider policies also vary: Shopify documents retries over four hours, while Stripe documents live-mode retries for up to three days. Do not hard-code one provider’s retry window as a general webhook rule.
Keep event, delivery, and attempt identities separate
An event ID identifies an event from the provider. Store it with the provider name and account or tenant, meaning the provider account that owns the event. An ID from one provider or account is not guaranteed to be unique in another.
A delivery ID identifies one provider delivery to a particular destination. It helps you find the matching entry in the provider’s delivery logs or use its tools to send the delivery again. Shopify distinguishes webhook IDs from event IDs: separate subscriptions can receive different webhook IDs for the same event. GitHub documents that its X-GitHub-Delivery value stays the same when a delivery is redelivered. Stripe recommends using event IDs to recognize duplicate deliveries, and notes that separate Event objects can sometimes refer to the same underlying object and event type.
An attempt is one try to send a delivery using HTTP, the protocol for requests between web services. Give each attempt its own local identifier, and keep the provider’s attempt number when it supplies one. Keep both records when an attempt fails and a later one succeeds: the failed attempt shows what happened before recovery, and the successful one shows when delivery resumed.
Store each identifier in its own field. To find a record, combine the provider name, provider account or tenant, and the provider’s event or delivery ID. Use the local attempt ID to identify each try. Shopify’s delivery details show why this distinction matters: the provider exposes delivery-level information such as the webhook ID as well as an event ID that can connect related notifications.
Record each attempt and both kinds of success
An attempt record should preserve what the provider sent and what your receiver observed. Keep the original provider values alongside normalized fields, meaning values converted to a consistent format, so operators can compare providers without losing the original evidence.
| Record area | Useful fields |
|---|---|
| Identity | Provider, provider account or tenant, event ID, delivery ID, local attempt ID, attempt number |
| Destination | Endpoint or subscription ID, event type or topic, payload version |
| Timing | Provider event time when available, receipt time, attempt start and end, response duration |
| Request | Relevant headers, payload size, secure payload reference, signature-validation result |
| Outcome | Transport result, HTTP status, provider error code and message, normalized result |
| Recovery | Retry state, replay indicator, replay-run ID, final disposition |
This model is a recommendation, not a provider-wide format. Provider consoles show some of the same details: Shopify shows response time, attempt number, payload size, and headers; GitHub exposes delivery status, status code, duration, and whether a delivery was sent again.
Track delivery acknowledgment separately from application processing. The acknowledgment records whether your endpoint returned a response the provider accepts. The processing result records whether work that follows receipt of the webhook has finished. Stripe recommends asynchronous processing to handle delivery spikes, and GitHub recommends prompt acknowledgment followed by background processing. In that design, a successful HTTP response can mean that your receiver accepted the event for later work; it does not prove that the work finished. Record queue acceptance and processing completion as separate outcomes.
Group failures while preserving the original evidence
Use a small set of internal failure categories to group incidents and build dashboards. Keep the exact HTTP status, transport error, and provider message in separate fields. The categories below are a practical way to classify failures, not a standard that webhook providers require.
| Category | Examples |
|---|---|
| Connection or transport | DNS resolution failure, refused connection, TLS problem |
| Timeout | Connection or response deadline exceeded |
| HTTP response | Redirect, client error, or server error, with the exact status |
| Authentication or integrity | Invalid signature, missing credentials |
| Receiver processing | Invalid payload, unsupported event type, handler exception |
| Throttling or capacity | Rate limit, overloaded endpoint, full queue |
| Provider or subscription state | Disabled subscription, provider delay, event not subscribed to |
| Terminal recovery state | Retries exhausted, event dropped, event awaiting operator action |
Record the observed failure separately from the event’s final outcome. A timeout describes a problem with one attempt; “delivered after retry” or “dropped after recovery ended” describes what ultimately happened to the event. Amazon EventBridge’s event-delivery records illustrate this distinction by tracking attempt outcomes and final results. EventBridge is an event-delivery service, not a general webhook provider.
Do not apply one success-code rule to every provider. Shopify treats responses outside the 200 range, including redirects, as errors. GitHub troubleshooting guidance recommends a 2xx response, while its REST delivery filters classify status codes from 200 through 399 as success. Store the actual response code. When reporting whether delivery succeeded, use the provider’s documented definition of success.
Connect provider IDs to local processing and replay
A correlation ID is a local identifier that lets an operator follow a webhook from receipt through later work. Create or attach one when the webhook arrives, then include it in queued work and processing logs. Keep it separate from the provider’s event ID: the provider ID identifies a provider object, while a correlation ID joins records inside your own services. If you use distributed tracing, store the trace ID and span IDs separately as well. The reviewed provider documentation does not define a common trace or correlation header that all senders use.
Record each replay, meaning a deliberate attempt to process or send an event again, as a separate recovery action. Keep the original event and delivery identity, add a replay-run ID, and record the reason, who started it, start time, target, and outcome. This lets an operator distinguish automatic provider retries from a manual resend or an internal replay.
The recovery path depends on the provider. GitHub exposes repository webhook deliveries through its API, letting operators list, retrieve, and redeliver repository webhook deliveries. Stripe supports dashboard resends for up to 15 days after event creation and CLI resends for up to 30 days; a manual resend does not stop its automatic retry behavior. Shopify documents a different failure mode: repeated failed deliveries can lead to subscription removal, after which recovery can require recreating the subscription and importing missing data. These paths should remain distinct in records and operator procedures.
For internal recovery, make the consumer idempotent: it can receive the same logical event more than once without repeating an unintended action. Retries and replays can create duplicate deliveries, and provider identifiers do not always reveal when two events represent the same business action. Where that distinction matters, retain the underlying object ID and event type as well as the provider event ID.
Measure delivery and processing separately
A webhook dashboard should show attempt volume, successful acknowledgments, and failure categories. It should also show response-time percentiles, retries per event, and the age of the oldest undelivered event. Add filters for provider, endpoint or subscription, event type, and tenant where those fields exist. Show downstream processing success and completion time separately from HTTP acknowledgment. Also show counts of events that exhausted retries, were dropped, or were replayed.
Use provider metrics as context, not as targets for every service. Shopify reports failed delivery rate and 90th-percentile response time. Its troubleshooting guidance describes a failure rate above 0.5% as higher than average in Shopify’s context. EventBridge recommends monitoring invocation attempts and successful attempts. It also recommends tracking retries and failures, along with delivery latency. Set service objectives based on your traffic and business impact. If provider and receiver counts disagree, compare their records.
What I'm buildingDelegate tasks. Get software.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com