Designing a Dead-Letter Queue for Webhooks

A webhook sends an event from one service to another over HTTP. If the receiver cannot handle an event, the sender may deliver it again, or the receiver may retry its own processing after storing the event. A dead-letter queue (DLQ) is a separate queue or store for failed events. It holds events after they reach the limit set by the retry policy, the rules for how and when to try again, so operators can investigate and recover them.

A DLQ gives repeated failures a clear endpoint. When the retry limit is reached, the sender or receiver saves the failed event and records why it failed instead of retrying forever or discarding it. Operators can then fix the cause and replay the event. The sender controls retries for delivering a webhook; the receiving service controls retries for work it has accepted into its own queue.

Decide which component owns each retry

A webhook integration has two separate failure boundaries. The sender may fail to deliver an HTTP request to the receiver. Or the receiver may accept the request but fail later while processing the event. The sender can retry delivery, and the receiver can retry its own processing. Neither side can retry the other side’s work.

For example, a receiving endpoint can verify the webhook’s signature, save the event to an internal queue, and return a success response after the queue confirms that it stored the event. A worker then processes the event asynchronously. The response confirms that the receiver stored the event, not that it finished the business task triggered by the event. If the endpoint returns an error or fails to respond, the sender may try delivery again, depending on its retry policy. If the worker fails, the receiver’s retry policy applies.

This distinction helps prevent retry storms, where repeated attempts add pressure to an already struggling endpoint. A queue-first design separates incoming HTTP traffic from slower processing, as described in Hookdeck’s webhook reliability guidance. The receiver should acknowledge an event only after it has saved the event to persistent storage, such as a queue. Otherwise, the sender can receive a success response and believe the event was saved when it was not.

Set a retry limit and move exhausted events aside

Retries help when a failure is temporary. A downstream service, meaning a service the receiver depends on, may have a brief outage. A rate limit, which restricts how many requests the service accepts, may also lift before the next attempt. Exponential backoff waits longer between each retry, giving the failed service time to recover and reducing unnecessary load.

A persistent problem needs a different outcome. A malformed payload, invalid destination, broken handler, or business-rule rejection will not necessarily improve with repeated attempts. A DLQ sits at the end of the retry policy. When an event reaches the set attempt limit or another stopping point for retries, the worker moves it out of the active queue and records the failure. Queue products expose this behavior differently; for example, Amazon SQS redrive policies use a maximum receive count to determine when to move a message to a DLQ.

Retrying forever keeps failed work active, consumes worker capacity, and delays recognition of persistent problems. Discarding an event avoids those repeated costs but loses data that may matter. A DLQ preserves the event while stopping automatic retries. There is no universal retry limit: choose a budget that fits the event’s importance, the expected duration of temporary outages, and the cost of holding up later work.

Use the available error details to classify failures. Retry temporary failures; send clearly permanent failures for investigation without using the full retry limit. Business-logic failures need special care because replaying the same event against unchanged rules will produce the same failure.

Preserve enough information to diagnose and replay

A DLQ record needs the original event and the context required to understand its failure. Store the payload and relevant request headers. Include event or delivery identifiers, timestamps, and the attempt count. Also record the last error and processing context. Without the original data and a useful error record, operators may know an event failed but lack the information needed to fix the cause or replay the event safely.

A DLQ can use the same queue technology as the main queue or a database table. A queue lets operators capture and handle failed messages through the same queueing approach as the main queue. A database can make the messages easier to search and analyze. Some designs use both, with a queue for immediate capture and a database for longer-term review. Choose based on how operators will inspect, retain, and recover failed events.

Set retention deliberately. If the DLQ deletes events before operators can investigate them, those events are lost later instead of immediately. In Amazon SQS standard queues, message expiration uses the original enqueue timestamp, so AWS recommends a DLQ retention period longer than the source queue’s retention period. That is an SQS-specific rule; verify how the selected storage handles expiration.

Protect the DLQ as carefully as the main event store. Webhook payloads and headers can contain sensitive data. Restrict access to the people and services that need it, and use encryption and secure transport where the storage provider supports them. AWS’s SQS security guidance describes encryption, least-privilege access, and HTTPS for SQS.

Replay only after addressing the cause

A DLQ supports recovery, but replaying an event does not guarantee that it will succeed. First find and fix the cause, such as a service the handler depends on being down, an incorrect configuration, or a bug in the handler. Then replay the affected event or a selected group of events and check whether processing completes. Avoid releasing a large backlog all at once if the downstream service cannot handle the added traffic.

Duplicate handling matters during both ordinary delivery and replay. A sender can retry after a receiver has completed work but before the sender receives confirmation; replay can also repeat an event that partially succeeded. Make handlers idempotent: when a handler receives an event it has already processed, it recognizes the repeat and does not apply the same business change twice. Use a stable event identifier or another reliable deduplication key when the webhook contract provides one.

If events must stay in order, check what moving one event to a DLQ does to later events. Some queue configurations can allow later messages to proceed while an earlier one waits for investigation. AWS specifically warns that using a DLQ with FIFO queues can affect exact ordering. The right behavior depends on whether the business operation requires strict order or whether independent events can continue.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com