Building a Webhook Failure Triage Tree

When a service sends a webhook, it makes an HTTP request to another service’s endpoint to report an event, such as a payment update or a new issue. If the receiver does not accept or process the request, the provider marks the delivery as failed, and code that depends on the event may not receive it.

Start with the provider’s delivery record and use it to narrow the cause: a timeout, an HTTP error, a request the receiver rejects, a configuration mismatch, or a retry state. The record shows whether the provider tried to send the event and, when available, the endpoint’s response. Providers set their own timeout limits and retry rules, so check the provider’s documentation before deciding what the record means.

Start with the delivery record

Open the provider’s delivery log and select a failed attempt. Check whether the provider sent the event, whether the endpoint replied, and what status code the endpoint returned. For example, Stripe displays delivery attempts and endpoint responses, while GitHub provides webhook logs.

Follow the branch that matches the record:

• No timely response: investigate a timeout.
• An HTTP 500-level response: investigate an endpoint error.
• An HTTP 400-level response or a validation error: inspect the request and the receiver’s expectations.
• No attempt, or a connection or certificate error: check endpoint configuration.
• A retry is pending, scheduled, exhausted, or manual: check the provider’s delivery state and retry rules.

These clues can overlap. For example, a wrong URL can produce an HTTP error response, while a slow endpoint can time out before it returns any status. Use the delivery record to identify the immediate failure, then check configuration if the response or connection points there.

If the provider reports a timeout

A timeout means the provider did not receive a response within its allowed wait. The limit depends on the provider: GitHub documents a 10-second timeout, while Allo documents a 20-second limit. Check the relevant provider’s limit instead of assuming one threshold applies everywhere.

Compare the delivery time with the receiver’s logs. If the request reached the endpoint, use the logs to find what delayed its response, such as a slow database call or a request to another service. If the logs show no request, check the endpoint configuration.

Have the endpoint check and record the incoming request, then return a successful HTTP response while slower work continues separately. A queue or database can store that work for later. Hookdeck’s troubleshooting guidance recommends acknowledging the request promptly rather than making the provider wait for all downstream work.

If the endpoint returned a 500-level error

A 500-level status means the receiver returned an error while handling the request. Use the response body and server logs for that attempt to find which part of the code failed. Check whether a required dependency failed or the handler raised an error before sending its response.

If the failure is temporary, a 503 Service Unavailable communicates that the endpoint cannot handle the request at that time. A provider that supports retries may try again after a 503, but retry behavior varies, so confirm the provider’s rules before relying on another attempt. The GitHub webhook guidance describes how response codes affect delivery handling.

If the receiver rejects the request

An HTTP status in the 400 range or a validation error means the receiver could not accept the request as sent. Compare the request in the provider’s delivery record with what the endpoint expects. Check that the endpoint accepts POST requests, handles JSON when the provider sends JSON, and recognizes the required fields and payload structure. Providers can use different payload formats, so follow the documentation for the specific event.

Also inspect the receiver’s parser and validation logs. The receiver can reject an otherwise valid delivery if the provider sends a field under a different name or in a different structure than the receiver expects. A ServerScout troubleshooting guide recommends checking request format and response status; the endpoint generally needs to return a 2xx status to acknowledge successful receipt.

If the endpoint configuration looks wrong

Make sure the configured URL points to the endpoint that is actually running. Verify its protocol and hostname, then check that the path and port match. Confirm that the provider has the intended endpoint enabled. A wrong path can send a request to a route that does not handle webhooks; a wrong hostname can send it somewhere else entirely.

If the provider reports a connection or certificate error, check that the hostname points to the intended service and that the service’s TLS certificate, which identifies the service during an encrypted connection, is valid for that hostname. Certificate expiration, a hostname mismatch, or an incomplete certificate chain can prevent delivery. Some providers also block private network addresses for security reasons, so check the provider’s documented restrictions rather than assuming every network destination is allowed.

Finally, confirm that the provider is configured for the right environment or mode and has the relevant event types enabled. For example, Stripe’s configuration instructions call for using the exact endpoint URL, the correct mode, and the required event selection.

If the provider shows a retry state

A retry state tells you what the provider plans to do next with a failed delivery. It does not identify the original cause. Read the attempt history to see whether the provider will send the event again, has stopped retrying, or requires a manual retry. Then use the response or connection details from each attempt to follow the relevant branch above.

Do not assume every provider retries automatically. Allo documents at-least-once delivery with retries, while the GitHub webhook guidance says failed deliveries require manual redelivery. Check the provider’s current delivery status before deciding whether another attempt is expected.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com