Designing a Dead-Letter Queue for Webhooks
A webhook sends an event from one service to another over HTTP. If the receiver cannot handle an event, the sender may…
When a service sends an event to your webhook endpoint, it makes an HTTP request to deliver the event. A webhook delivery log records what happened to that request. A delivery attempt is one try to send an event. After a failure, the provider may try again.
Read the log as a timeline. Count the provider’s attempts, then note when each started and ended. Check the HTTP status or error recorded for each attempt, and whether the provider considers delivery finished. These details help you tell a temporary outage from a problem with the URL or endpoint that needs fixing. Providers use different field names and failure categories, so interpret each entry according to that provider’s documentation.
An attempt count shows how many delivery tries the provider recorded for an event. Some providers count attempts for each event and subscription together. Check what the count includes before comparing it across events. Hook0, for example, documents request attempts with statuses such as pending, in progress, successful, and failed in its request-attempt documentation.
With a pending or in-progress status, the provider has not recorded a final result for the attempt. The provider considers the delivery successful according to its rules when the status is successful. A failed status marks an unsuccessful attempt, and the provider may make further attempts. Before treating the delivery as abandoned, check for later attempts or a separate status showing that the provider has finished trying.
The attempt count helps explain the status. One failed attempt followed by a successful one points to a different situation from repeated failures with no success. Providers use different field names and labels. For example, EventHooks’ documentation describes logs that show an attempt’s status code, the content of the response, any error message, and the times of key events.
A delivery log may record when the provider created an attempt, started sending it, and marked it successful or failed. Hook0 documents fields for these phases in its performance-monitoring guidance. The timestamps show whether the provider waited before sending, the endpoint took a long time to respond, or the provider waited a long time before trying again.
Compare timestamps in order. If the provider created an attempt well before it started sending it, the delay happened before the request went out. If the provider started sending promptly but recorded a failure much later, check how long the endpoint took to respond and read the error details. If a later attempt appears after a failed one, the provider retried the delivery. Providers may wait longer between each retry. A long gap between attempts does not by itself mean the delivery is stuck.
Confirm the time zone and timestamp meaning in the provider’s documentation before comparing log entries with your own server logs. The time when the provider starts sending an attempt and the time when the endpoint receives the request mark different points in the delivery.
An HTTP status code is a number that records the response the endpoint returned to the provider. Each provider defines which responses count as successful. A response with a status code outside the 200–299 range commonly appears as a failed delivery, but check the provider’s rules because they do not all handle status codes the same way. Stripe’s webhook documentation describes its delivery requirements.
A 4xx response usually means the endpoint rejected the request or could not handle it as sent. A 5xx response usually means something went wrong while the endpoint handled the request. These categories help narrow down the problem; its persistence remains uncertain. For example, repeated 5xx responses can point to a server-side bug. Retrying alone will not fix that bug. Hookdeck describes the distinction between protocol errors, commonly associated with 4xx responses, and consumer errors, commonly associated with 5xx responses, in its error-recovery guidance.
Some failures have no HTTP status because the provider could not finish sending the request or receive a response. Hook0 documents provider-specific categories such as connection errors and timeouts, along with DNS failures and HTTP errors. Its categories also cover invalid target URLs and internal errors. Read the message along with the category. A timeout means the request did not finish in time. A DNS error means the provider could not find the network address for the target hostname.
Decide whether a failure is temporary or persistent by checking the error, whether it repeats, and what the provider says about the delivery. A connection failure or timeout may clear when the network or service recovers. If the same error keeps appearing, check whether the endpoint is available and review its logs. Do not assume the next retry will succeed.
An invalid target URL means the configured destination is wrong. The problem will remain until someone corrects it. If 4xx responses keep appearing, check the endpoint’s route and how it handles the incoming request. For a 5xx response, check the endpoint’s logs. A temporary outage may clear on its own, but a recurring server error needs a code or configuration fix.
A terminal failure means the provider has stopped waiting or retrying and marked the delivery as failed. The exact meaning and label vary by provider. Before deciding whether to fix the endpoint, change its URL, or retry manually, check the final attempt’s error and the provider’s documentation about retry status. Do not infer that a single failed attempt is terminal.
Repeated responses with status codes outside the 200–299 range, more failed attempts, or a growing backlog of undelivered events signal that deliveries need attention. Compare these patterns with successful deliveries and endpoint logs to see whether failures affect one event, one destination, or more deliveries.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com