Choosing a Webhook Error Queue Design

A webhook receiver accepts event notifications that a provider sends to a service. When the receiver cannot process an event, it needs a place to store it so an operator can investigate and decide whether to try again. A dead-letter queue (DLQ) holds messages that have used all their normal processing attempts or need investigation. Redrive moves a message from that holding path back into processing.

Most designs send failures to one shared queue, separate queues for each provider, or separate dead-letter paths for each event type. One shared queue is simpler to operate; separate paths make some failures easier to isolate and recover. Choose the simplest layout that gives operators the ability to identify failures, separate them where needed, and control replay. In every layout, store enough event details to investigate failures.

Compare the three layouts

These are architectural tradeoffs, not measured guarantees. Product documentation describes queue and retry behavior, but does not establish that one layout consistently improves recovery time or reduces failures compared with the others.

One shared queue keeps operations simple

A shared queue gives operators one place to inspect failures from different providers and event types. This works well when providers have similar retry rules and the same people handle recovery.

Each queue record should include enough information to identify the failure: a stable provider delivery or event ID, the internal event type, the affected tenant (customer or organization account) or destination when relevant, the attempt count, timestamps, the latest error, and a reference to the original payload. Without those details, operators may see a backlog but struggle to identify which messages belong to an incident or which are safe to replay. Queue products also illustrate the value of failure context: EventBridge DLQ records, for example, include error and event details.

One queue does not automatically mean that a failed message blocks every other message. The effect depends on how the queue handles retries and message order. Google Cloud Pub/Sub documents a delay for each message that lets other messages continue during a failure, except when ordered delivery requires messages to stay in sequence. Widespread negative acknowledgments, which tell Pub/Sub that messages were not processed successfully, can also slow delivery more broadly. Check the queue’s ordering and retry behavior instead of using the number of queues to judge how well failures are isolated. (Pub/Sub retry behavior)

A shared queue fits when common handling is more valuable than provider-specific boundaries. Give operators filters for provider, event type, and type of failure, and let them replay only the messages they select. A broad redrive can put unrelated messages back into service or overwhelm a downstream dependency that has only just recovered. Amazon’s SQS redrive guidance recommends starting at a low rate and increasing it while monitoring the source queue. That is SQS-specific guidance, but the operational principle applies broadly: control replay volume and watch what receives the messages.

Provider queues separate provider-specific failures

Use separate queues when providers need different access, retention periods, alerts, retry rules, or recovery procedures. A provider-specific route also helps operators identify the source of a failure while investigating it.

Separate queues keep provider messages on different routes, but they do not by themselves separate the computing capacity that handles those messages. If every provider queue feeds the same set of message-handling workers or relies on the same overloaded service, one provider’s backlog can still compete for shared resources. To limit that competition, separate the relevant workers, capacity limits, or dependencies as well as the queues.

Separate provider paths make it easier to pause or replay one provider’s backlog without mixing it with others. They also require more configuration and monitoring. As the number of providers grows, operators must keep each path’s alerts, permissions, and recovery procedures consistent.

Provider behavior is one reason to consider this design, but a separate queue does not replace provider-side recovery. GitHub says it does not automatically redeliver failed deliveries, while PayPal and Shopify document their own retry behavior. Preserve provider delivery IDs and check the provider’s delivery records when reconciling a local retry with a provider resend.

Event paths support event-specific recovery

Route failures by event type when, for example, an account-state update and a one-time transaction need different owners, ordering, or repair procedures. An account-state update may require a different replay decision from a one-time transaction. This routing helps when each path clearly maps to the handler and recovery procedure responsible for its event.

Event-specific paths separate failures more closely by event type than provider queues do, but they add routing rules and queues to maintain. A separate DLQ also does not guarantee separate capacity if the paths share workers or services that receive the messages. Avoid creating a path for every event name unless each path marks a real boundary in how operators handle failures.

Recovery needs to account for event dependencies and ordering. An operator may be able to replay one event after fixing its handler, while another event needs a check against current state before replay. Keep stable event identity, attempt history, and the original payload or a durable reference to it so the recovery procedure can make that distinction. Google Cloud Pub/Sub’s dead-letter topic behavior offers a product-level example of failures associated with a specific subscription; it does not prescribe one queue for every webhook event type.

Design recovery around stable identity

Retries, redrive, and provider redelivery are separate actions. A retry is another attempt during normal processing. Redrive returns work stored in the receiver’s queue to processing. Provider redelivery asks the original sender to send the event again. These actions can create duplicate deliveries, so the receiver should identify events by a stable provider or business event ID, not only by the queue’s message ID.

Redrive can also change identifiers that the queue assigns to messages. Amazon SQS, for example, documents that redriven messages receive a new message ID and enqueue time. Keep the original event ID in the message data or a stored record so operators can connect the redriven message to its earlier attempts.

Before enabling replay, define who can start it and how to control the replay rate. Decide how the receiver handles events it has already processed. Also check how long the queue, DLQ, provider dashboard, and any event store retain the data. Retention rules vary by queue type; SQS DLQ guidance, for example, describes different timestamp behavior for standard and FIFO queues.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com