Designing a Webhook Failure Dashboard
A webhook sends an HTTP notification from one service to an endpoint hosted by another service. When delivery fails, operators need to…
Categories
Catching Invalid Webhook Integrations Early
A webhook is a request one service sends to another over HTTP, the protocol used to transfer data on the web, when an event occurs, such as a payment completing or a repository changing. The receiving service must be reachable, recognize the event, validate the request, and respond in the way the sender expects. A mistake in any of those steps can leave events undelivered or unprocessed.
Check the destination and receiver configuration before activating the webhook. Then test it with a representative event and compare the sender’s delivery record with the receiver’s logs. A successful ping proves basic connectivity; it does not prove that a real event, secret, or production route is configured correctly. Each provider sets its own response deadlines and retry rules, so use the provider’s documentation when deciding what counts as a successful delivery.
Check the destination and receiver before activation
Start with the exact public URL, including its path. Confirm that the webhook is active and set to send the events the receiver handles. A service cannot send an event that the webhook does not subscribe to. GitHub’s setup, for example, asks you to enter a payload URL, choose a content type and secret, and select the events. Activate the webhook, and GitHub sends a ping when it is created. Its webhook setup guidance also recommends selecting only the events the receiver needs.
Check reachability from outside your network. A URL that works on a developer’s machine may be inaccessible to the provider because of an incorrect DNS entry, a firewall, or a network restriction. The receiver also needs a valid HTTPS certificate. HTTPS encrypts the connection, while the certificate lets the sender verify that it has connected to the intended host. GitHub lists unresolved hostnames, blocked connections, self-signed certificates, and incomplete certificate chains among causes of delivery failure in its troubleshooting guidance.
Match the receiver to the request the provider sends. Confirm that the configured content type matches the format of the request body the receiver can read. Also check that the chosen route accepts the provider’s request method and payload. Keep the webhook secret, the shared value used to check that a request came from the provider, aligned between the provider and receiver. The receiver typically validates a signature, a value calculated from the request and secret. It must verify the actual request body. Middleware, software that reads and prepares a request, can change the body before verification and cause a valid delivery to fail signature checks. GitHub documents signature validation in its delivery validation guidance.
Check that the URL points to the intended environment, such as test rather than production. Confirm that the secret and event subscription are configured for that same environment. If the receiver only accepts traffic from listed IP addresses, check that the list includes the provider’s current addresses. GitHub notes that its delivery IP addresses can change, so an outdated allowlist can block requests. Its webhook security recommendations cover secret handling and other protective measures. Provider requirements are specific: for example, PayPal documents HTTPS on port 443 for its REST webhook listener, a requirement that should not be assumed for every provider.
Validate the first delivery end to end
After setup checks pass, send a test event and follow it through the receiver. A provider’s ping or mock event helps check whether a request reaches the endpoint, but it might not exercise the event subscription, real payload shape, or business logic. PayPal describes its simulator payloads as mock events, with limitations on how they relate to real transactions. Stripe’s testing guidance describes triggering events with test tools and using sandbox events tied to test transactions to check real event notifications.
For a representative test, check the provider’s delivery record, then find the matching request in the receiver’s logs. Confirm that the handler recognizes the event type and any relevant action. Then confirm that signature validation succeeds and that the receiver returns the success response required by that provider. A 2xx status means the HTTP request succeeded, but the provider’s timing and acceptance rules still apply. GitHub, for example, expects a 2xx response within 10 seconds. Shopify documents a one-second connection timeout and a five-second total request timeout, and treats responses outside the 200 range, including redirects, as errors. These are provider-specific limits, not shared webhook standards.
Keep the acknowledgment path short. A receiver can validate the request, record that it accepted the event, and return the required success response before doing slower work. It can then process the event asynchronously, meaning a separate worker handles it after the receiver acknowledges delivery. GitHub and Shopify discuss queue-based handling in their webhook guidance. Check the provider’s requirements before choosing this pattern. The receiver still needs to store and process the event after it responds.
Use delivery evidence to narrow the cause
A delivery status points toward an area to inspect; it rarely identifies the root cause on its own. Compare the provider’s delivery record with the receiver’s logs, including timestamps, response status, and response time. If the provider reports a delivery but the receiver has no corresponding request, investigate the path between them. If the receiver logged the request, check how it handles the route and validates the signature, and look for errors in its code.
| Delivery evidence | Likely area to check | What the evidence does and does not establish |
|---|---|---|
| Repeated DNS resolution or connection failures | URL, DNS, firewall, network, or IP allowlist | A persistent failure points toward reachability or configuration. A brief network incident can produce similar evidence, so compare attempts over time. |
| Certificate validation errors | HTTPS certificate or certificate chain | Check the certificate presented by the public endpoint. A certificate error alone does not show whether another runtime problem also exists. |
Repeated 404 or other 4xx responses |
Route, authorization, or request validation | The route may be wrong, or the receiver may be rejecting the request by policy. Read the receiver’s response and logs before changing configuration. |
| Repeated signature failures | Secret, signature handling, or request body | Confirm that both sides use the same secret and that the receiver checks the unaltered body and expected signature header. |
| Timeouts or rising response times | Slow handler, service load, or a dependency | A timeout means the sender did not receive the expected response in time. Compare receiver latency and infrastructure data to distinguish slow code, load, and dependency trouble. |
| Failures across many webhook types at once | Shared receiver or infrastructure | Broad failures suggest a common problem, but do not prove one. Check whether other requests to the receiver also fail. |
| One event type or payload repeatedly fails while others succeed | Event subscription or event-specific handler | Compare the failing event’s type, payload, and response with successful deliveries. The route may work generally while one handler path fails. |
GitHub and PayPal both describe unsuccessful HTTP responses as delivery failures, but a status code alone does not explain why the receiver returned it. A 5xx response, for example, could reflect a code defect, an overloaded receiver, or a failing dependency. Likewise, a missing event may mean that the provider never sent it because the subscription excludes that event, or that the receiver rejected or mishandled a request it did send. GitHub recommends checking recent deliveries after triggering an expected event; its troubleshooting steps help locate where delivery stopped.
Use repetition and scope as diagnostic clues, not as proof. The same certificate error across repeated attempts suggests a stable certificate configuration problem. Failures that appear during a latency spike or affect many webhook types point more strongly toward runtime trouble. If only one event type fails, inspect that event’s subscription and handler first.
Plan for retries and duplicate deliveries
A receiver should handle the same delivery more than once without causing problems. Some providers retry after a failed response, and an event might reach the receiver more than once. Idempotent handling means that processing the same event again does not repeat an action that should happen only once. For example, a receiver can track a provider’s delivery or event identifier before applying a change. Shopify documents duplicate-delivery risks and recommends idempotent handling in its delivery verification guidance.
Do not assume every provider retries. GitHub says failed deliveries are not automatically redelivered, while Shopify and PayPal document retry behavior for their webhook products. Their schedules and consequences also differ. Check the provider’s current documentation for that webhook type. Find out whether the provider retries, how long it retries, and whether it disables the webhook after repeated failures. A retry policy cannot fix an incorrect URL or secret, and a provider that does not retry requires a separate recovery plan for missed events.
What I'm buildingDelegate tasks. Get software.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com