Routing Failed Webhooks to an Error Endpoint

When a provider sends a webhook, it delivers an event to an HTTP endpoint operated by the receiving service. If the endpoint cannot receive the event, or the receiving service fails while processing it, the event needs a separate path for investigation and recovery. A provider often sends undelivered events to a queue or storage location called a dead-letter queue (DLQ). The queue or storage keeps the events for later investigation and recovery.

Providers define their own retry and failure rules, and many do not support a second webhook URL as a built-in fallback. A receiver can also create a separate failure queue for events that fail during its own processing. Choose the design based on who controls the failure path, the provider or the receiver, and when each considers delivery complete.

Route exhausted deliveries to a separate destination

To route failures on the provider’s side, configure a normal webhook destination and a DLQ or other durable store. The provider sends the event to the normal endpoint, evaluates the response or connection result, and retries according to its policy. If delivery still fails after the provider’s retry window or attempt limit, it writes the event to the failure destination, when that feature is configured.

For example, Azure Event Grid can store undelivered events in an Azure Storage blob container. Dead-lettering is off by default, so an operator must configure the destination; otherwise, events that cannot be delivered can be dropped. Event Grid’s delivery settings also let operators configure retry limits and event time-to-live.

A DLQ is usually a queue or storage resource, rather than a second HTTP URL that the provider calls automatically. A separate program can read the stored records, investigate the cause, and replay or discard them. If an integration needs an error-handling HTTP endpoint, that worker can forward records to it. That forwarding step is a separate design choice.

Separate receipt from business processing

A receiving service can separate webhook receipt from later processing by accepting each request, storing the event durably, and processing it in the background. The receiver returns a success response only after it has stored the event safely. A background worker then performs the business operation and sends processing failures to a separate failure queue.

This design lets the receiver close the webhook connection without waiting for slow or failing business logic. GitHub recommends queueing payloads for asynchronous processing so the receiver can respond promptly. GitHub considers a delivery failed if the server takes longer than 10 seconds to respond, and it does not automatically redeliver failed deliveries. Operators need to monitor failed delivery records and initiate recovery through GitHub’s documented methods. See GitHub’s troubleshooting guidance and failed-delivery instructions.

The receiver must tell the provider whether it has stored the event or finished processing it. The CloudEvents HTTP Webhook specification describes 202 Accepted for a request accepted while processing remains incomplete or unknown. However, each provider defines how it interprets status codes, so the receiver must follow that provider’s delivery contract. Returning success before storing the event risks losing it if the receiver then fails before processing.

Check what each provider calls a failure

Each webhook provider sets its own retry rules. Those rules define which responses trigger another attempt and which failures end delivery. They also set how long the provider keeps trying. Azure Event Grid treats HTTP 200–204 as successful delivery. Its retry behavior depends on the failure: some response codes, including 400, 403, and 413, are not retried under the documented policy. Its default policy ends delivery at the first of 30 attempts or 24 hours. These are Event Grid settings, not general webhook rules. See Event Grid’s retry behavior.

Other services use different limits and recovery paths. GitHub does not automatically redeliver failed webhooks. Google Cloud Pub/Sub forwards undeliverable messages to a dead-letter topic, but describes forwarding as best effort, so its configured attempt count is not an exact guarantee. Check the specific Pub/Sub dead-letter topic guidance before relying on its attempt tracking.

CloudEvents standardizes event metadata and HTTP delivery details, but its webhook specification does not define a universal failure destination. The destination and retry contract remain provider-specific. Before implementation, confirm which responses count as success, which failures trigger retries, how long retries continue, and what happens after the limit.

Make the failure path recoverable

A failure destination keeps failed deliveries separate from ordinary processing, but it does not guarantee recovery. The receiver or provider must be able to write to the destination, and an operator needs a way to detect, inspect, and resolve queued failures. For every stored record, preserve the original event and the delivery details needed to diagnose the failure, such as the error reported by the provider or receiver.

Check the destination for new records and a growing backlog. When a record appears, identify and correct the cause. Then replay the event deliberately, or discard it and record the reason. Replaying an event can repeat work if the receiver partly processed it the first time. The receiver should make repeated handling safe, for example by recognizing events it has already completed. Also verify the destination’s permissions and retention settings: a misconfigured destination or expired record defeats the recovery path.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com