Designing a Dead-Letter Queue for Webhooks
A webhook sends an event from one service to another over HTTP. If the receiver cannot handle an event, the sender may…
When a service sends a webhook to your server and your server does not accept it, the provider may record the delivery as failed. A webhook is an HTTP request that sends an event, such as a payment update or repository change, to a URL you control. To recover a missed event, use the provider’s API to find its delivery record and, if supported, request that the provider send the event again.
Recovery steps vary by provider. Some APIs let you view failed deliveries and request that the provider send them again. Others show delivery logs but offer no way to replay an event. Retention limits set how long the provider keeps the records or events you need. Check those limits before relying on the provider to recover missed events.
First, find out how the provider records each delivery and how its API marks failures. A delivery record usually includes a status and an event identifier, but providers define these fields and the available API actions differently.
GitHub does not automatically resend failed webhook deliveries. Its documented recovery approach is to use the REST API to inspect delivery attempts, find those whose status is not OK, and redeliver them manually or with a scheduled script. The script should record when it last checked so it can request records for newer attempts the next time it runs.
GitHub records a delivery as failed when the receiving server is down or takes more than 10 seconds to respond. For GitHub App webhooks, the API endpoints for finding and redelivering deliveries require a JSON Web Token generated from the app ID and private key. The recovery script can run on a server that handles webhooks or in GitHub Actions, as described in GitHub’s GitHub App redelivery instructions. GitHub does not provide API endpoints for delivery data for GitHub Marketplace or GitHub Sponsors webhooks, so this API-based method does not cover those event types.
Akamai Webhooks v3 stores webhook events in an event store, a provider-side record of those events. To find failures, query the /events endpoint and set the state parameter to return failed events. Query each Webhooks subscription separately, because each subscription has its own results.
Some providers show delivery history but do not offer a way to replay events. Struct’s delivery-log API returns records newest first, with the send time and delivered payload. It uses keyset pagination: each response includes a pagination_key that you send with the next request to get more records. These logs show what the provider sent, but they do not mean the provider supports redelivery.
After you find a failed record, use the provider’s documented operation to redeliver that event. The API usually requires an event or delivery identifier. Check the provider’s documentation for the required identifier and authentication method before sending the request.
Akamai’s /redeliver endpoint reschedules an event only if it remains in the event store and has a failed state. The endpoint does not redeliver events marked as successful. When Akamai redelivers an event, it resets the event to an awaiting-execution state, treats it as new, and places it in a lower-priority delivery queue. The endpoint allows one request per minute per Webhooks subscription. A recovery script must leave at least that much time between requests for the same subscription. For incidents involving hundreds or thousands of failed events, Akamai recommends bulk export instead of individual redelivery requests.
You must explicitly request a resend from GitHub; it does not resend failed deliveries automatically. For ongoing recovery, a scheduled script can find and redeliver failures. For a one-time incident, use GitHub’s manual redelivery controls. Before requesting a resend, make sure the original failure is resolved. GitHub documents two failure conditions: the receiving server is unavailable, or it takes longer than 10 seconds to respond.
A provider’s retention period determines how long you have to recover a failed event. Once the provider deletes the event or its delivery record, the recovery API cannot retrieve that data.
Akamai keeps failed events in its event store for seven days, then deletes them permanently. Request redelivery before the seven days expire. Akamai permits repeated redelivery of the same failed event. Each request puts the event back in the delivery queue and lets it remain in the store, as long as you request redelivery within seven days.
Struct keeps delivery logs for seven days. The API returns logs newest first and uses pagination_key to fetch more records. If you need a complete incident record, retrieve all the pages promptly. Struct also notes that large payloads and error messages can be truncated, so the logs may not include everything you need for a later investigation.
A retry schedule and a retention period describe different things. edrv.io documents six retries, then drops the message if delivery still fails. Its documentation does not describe a separate API for retrieving dropped events. Billingplatform.uk retries indefinitely and recommends using its X-Idempotency-Key header to detect duplicate deliveries. Neither provider’s retry behavior tells you how long its API keeps failed-event history.
GitHub’s failed-delivery documentation does not specify how long it keeps records. Check the current documentation and test how far back the API returns delivery records. Do not assume a fixed recovery window.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com