Designing a Dead-Letter Queue for Webhooks
A webhook sends an event from one service to another over HTTP. If the receiver cannot handle an event, the sender may…
When a provider sends a webhook, your endpoint must tell the provider whether it received the event. If your server performs slow business work before replying, the provider can time out, mark the delivery as failed, and send the same event again. With the acknowledge-fast pattern, your endpoint first verifies the request and saves the event for background work. Once the save is confirmed, the endpoint returns a success response.
A background worker is a separate process that takes saved work and handles it after the endpoint responds. The acknowledgment means your service accepted responsibility for the event; it does not mean the event’s business effects are complete. The endpoint should acknowledge only after it has durably stored the event or confirmed that a durable queue accepted it.
Webhook providers set response deadlines, and the exact limit depends on the provider. When a receiver spends that time calling other services, updating records, or waiting on a slow database, the provider may receive no response before its deadline. A timeout and a non-success response both count as failed deliveries for many providers, which leads to retries. Hookdeck discusses webhook timeouts, but you should check the sending provider’s documentation for its actual deadline and success requirements.
Your endpoint should do only what it needs to verify the request and save the event:
A durable queue stores work so a worker can retrieve it later, including after a temporary interruption. A database-backed inbox can serve the same purpose: the endpoint writes each event to a table, and a worker reads pending rows. In either design, the endpoint must not return success before the handoff is durable. If the queue or database is unavailable, return an error that matches the provider’s retry rules rather than acknowledging an event that was never saved.
Use the status code the provider documents as successful. Many integrations use 200 OK; 202 Accepted also indicates that the request was accepted for processing that has not finished. However, 202 does not report the eventual result, and a provider that requires a particular response may not treat it as success. Check the provider’s accepted status codes before choosing one. For example, Hookdeck’s retry guidance recommends a quick acknowledgment, while MDN’s description of 202 Accepted explains that the status is noncommittal.
A fast response is useful only if the event came from the expected provider. Before storing the event or scheduling work, verify the webhook signature, a value that lets you check whether the provider sent the request and whether anyone changed it. Otherwise, an attacker could submit forged requests that trigger business actions.
Signature checks often depend on the exact bytes the sender transmitted. For Stripe, the signature uses the raw request body, so parsing the JSON and then turning it back into text can invalidate verification. The Stripe signature notes describe this requirement. Keep the original body available until verification finishes, then save the verified event for the worker.
If verification fails, do not queue the event as trusted work. Return the response your provider’s documentation specifies for invalid requests, and record enough diagnostic information to investigate without exposing secrets or sensitive payload data.
A provider can retry after the receiver times out or returns an error. It can also retry when the receiver’s success response never reaches the provider. As a result, the same event can arrive more than once even when the first request was handled correctly. If the endpoint completed a business action before timing out, a retry can repeat that action unless the receiver deduplicates the event.
Store a stable event identifier with each accepted event and enforce uniqueness in the database; a unique constraint makes the database check for an existing record and insert a new one as a single operation. The first delivery creates the record; a duplicate finds it and does not create a second unit of work. This is idempotency: handling the same event again does not repeat its effects. Hookdeck’s idempotency guidance describes tracking processed webhook IDs this way.
Confirm that the provider’s event ID is actually unique for the events you receive. HubSpot’s community discussion, for example, reports that its eventId is not guaranteed to be unique, so relying on that field alone may not be enough. When uniqueness is unclear, check the provider’s documentation and choose a deduplication key that identifies the event reliably.
Keep deduplication records for at least as long as the provider can retry or replay events. If the record expires sooner, a late retry can pass the uniqueness check and repeat the business action. The worker should also make its own changes idempotent, because a worker can fail after performing an action but before marking the event complete.
After the endpoint acknowledges an event, the worker is responsible for completing the business work. It should retry temporary failures, such as an unavailable downstream service, without asking the webhook provider to resend work that the receiver already saved. Use increasing retry delays with randomized variation, known as exponential backoff with jitter, to prevent many failed jobs from retrying together. Hookdeck’s retry best practices cover this approach.
When a job reaches its retry limit, move it to a dead-letter queue, a holding area for work that needs investigation or manual replay. Track the event ID, attempt count, failure reason, and processing state so an operator can distinguish an unaccepted webhook from an accepted event that the worker could not finish. The Svix explanation of webhook retries discusses exhausted retries and failed deliveries.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com