Building a Webhook Incident Runbook

A webhook is a message that a provider sends to your service when an event occurs, such as a payment update or a repository change. A webhook incident means the expected event is missing, delivery failed, or delivery appeared successful but the expected work did not happen.

A useful runbook separates two questions: Did the provider attempt delivery? and Did your service process the event and complete the intended work? Check provider delivery records alongside your own logs, then choose a recovery method that the provider supports. A replay sends an earlier event again, so your handler also needs to prevent repeated events from causing unintended duplicate effects.

Identify which part failed

Start with the event you expected. Record the provider, environment, and subscription, then note the event type and affected account or resource. Add the approximate event time and delivery ID if you have one. Describe the symptom precisely: missing event, failed HTTP request, processing error, delayed update, or duplicate effect. This scope helps distinguish a single-event problem from an outage affecting one event type or all webhook traffic.

Then compare the symptom with the provider’s delivery history:

• No matching delivery appears: Check that the event occurred in the expected account or resource, that the subscription is active and includes the event type, and that provider limits or service issues do not explain the gap. A missing record does not by itself prove that your endpoint failed. GitHub also advises allowing for a short delay before its delivery record appears. See GitHub’s webhook troubleshooting guidance.
• The provider records a failed request: Follow the reported failure through the connection and endpoint. Check DNS resolution, network restrictions, the TLS certificate, endpoint availability, timeouts, and the HTTP status your service returned. Providers use different error labels, so treat these as diagnostic areas rather than a universal set of status meanings.
• The provider records success, but the expected result is absent: Trace the event inside your service. Check validation, queue acceptance, processing, database writes, and downstream calls. A successful HTTP response tells the provider that its delivery met that provider’s acknowledgment rules; it does not prove that your service completed the business operation.

Correlate provider records with service logs

Use the provider’s delivery record to establish what it sent and what response it received. Depending on the provider, the record may show when delivery occurred and which attempt and event type it covered. It may also include request headers and payload, the response status, and request duration. GitHub documents request and response details in its webhook delivery history. Shopify’s delivery logs include fields such as response code, attempt number, and response time, and Shopify warns that those logs can take several minutes to appear. Check the Shopify webhook troubleshooting details before treating an empty log as proof that no attempt occurred.

Search your service logs using the provider’s delivery ID or another stable correlation key. Follow each matching request from receipt through signature validation and parsing. Then check durable acceptance and processing, and record the response returned to the provider. A signature is a value used to check that the request came from the expected provider and was not altered. Log whether validation passed or failed, but avoid storing secrets or unnecessary personal information.

If the service acknowledges a request before processing finishes, record both the acknowledgment and the later processing result. That distinction shows whether the provider completed delivery while your queue or worker failed afterward. If your logs do not capture these stages today, add the records and correlation IDs as a follow-up; without them, provider delivery success cannot locate a failure inside your service.

Check replay eligibility before recovery

A provider retry, a manual redelivery, an event archive replay, and a dead-letter queue redrive are different recovery mechanisms. A dead-letter queue (DLQ) holds messages or failure records that could not be delivered through the normal path. The provider’s documentation determines which mechanism applies and how long an event remains available. It also specifies who has permission to replay it and whether a replay preserves its identifier.

For each affected event, check the exact product and delivery configuration. GitHub allows manual redelivery for deliveries from the past three days and says it does not automatically redeliver failed deliveries; permissions depend on the webhook type. See GitHub’s redelivery instructions. Shopify documents up to eight retries over four hours. After repeated failures, a subscription may be removed, so extended downtime can require restoring the subscription and importing missing data rather than relying on replay alone. See Shopify’s recovery guidance.

EventBridge policies also depend on the destination type. For an Amazon EventBridge Custom Event Bus subscriber, the documented default is five attempts or 300 seconds, whichever limit comes first; a configured DLQ supports a separate redrive path, subject to event retention. EventBridge’s Custom Event Bus retry documentation describes that behavior. EventBridge API destinations have different retry settings and response handling, as their destination documentation shows. Do not apply one EventBridge policy to every target.

Before replay, confirm that the cause is fixed and that repeated processing will not trigger an unintended charge, notification, or provisioning action. Idempotent processing means handling the same event more than once without repeating its intended effect. Use a durable record of processed event IDs or another provider-supported duplicate check, and verify the provider’s identifier semantics rather than assuming all providers reuse IDs in the same way.

Recover a bounded set of events

Choose the recovery path only after confirming eligibility. If the provider permits redelivery, define the affected event IDs or time interval and replay those events. If a DLQ or archive applies, follow the documented redrive procedure and check its retention limits. If replay is unavailable or the event has expired, reconcile against the provider’s source-of-truth data and import only the missing records.

Keep the recovery set narrow enough to measure. Track the events selected, their outcomes, and any manual corrections. Replay in manageable batches when the provider’s tooling allows it. While batches run, watch processing errors and queue depth, along with latency, rate limits, and downstream effects. If the service cannot safely recognize events it has already processed, resolve that duplicate risk before replaying.

Reconciliation also needs duplicate controls. Record the query’s time bounds and matching criteria, then compare imported records with existing state before applying changes. Shopify’s guidance for extended downtime specifically calls for importing missing data and restoring subscriptions where applicable; that is a provider-specific recovery path, not a general replay guarantee.

Confirm delivery and business recovery

Close the incident only after confirming both delivery and the intended result. Check that a controlled or known event appears in provider history, that your service validates and durably accepts it, and that processing completes. Then compare the source-of-truth state with your service’s state for the affected interval. Account for every replayed or reconciled event, and confirm that duplicate handling worked.

During an agreed observation period, watch the backlog and retry counts. Also track DLQ population, error rate, and processing latency. Use your service’s own operating thresholds; provider-specific figures are not universal alerting standards. Shopify’s successful-delivery acknowledgment alone does not confirm that downstream processing finished. See Shopify’s delivery verification guidance.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com