Currently Available: Need a skilled Software Developer for your next project?
Categories
APIs

How to Debug Webhook Timeouts and 500 Errors

A webhook is an HTTP request one service sends to another when an event occurs, such as a payment update or a repository change. The receiving server returns an HTTP status code showing whether it accepted the request.

The sender times out when it receives no acceptable response before its deadline. A 500 Internal Server Error reports an error from a server-side component, which could be the application, a gateway, or another intermediary. Neither result alone tells you whether the receiver processed the event.

First, find out what the sender observed and trace the delivery through the receiving infrastructure. Then fix the specific failure. Providers set different deadlines and retry rules, so check the sender’s documentation before choosing a response time or relying on another delivery attempt.

Identify what the sender observed

A timeout means the sender did not receive an acceptable response in time, even if the receiver completed the work. A 500 means a component returned an error status, but it does not show which component returned it or why.

Other server errors need separate investigation. A 502 indicates a gateway or proxy received an invalid response from an upstream service. A 503 indicates that a service was unavailable. A 504 indicates that a gateway or proxy timed out while waiting for an upstream service. The response status and any available response body help narrow the search.

Sender’s result What to check first
Timeout, with no matching receiver request Check DNS, network access, TLS certificates, and gateway ingress.
Timeout, with a matching request Compare the request’s arrival and response times. Find where it waited or stalled.
500 Identify which component returned the status, then inspect its logs and the next service it called.
502, 503, or 504 Check the gateway or proxy and its connection to the upstream service.
Duplicate event Check whether the first request completed and whether repeated processing is safe.

Record the provider’s delivery or event ID and the status it received. Note how long the request took and the request ID the receiver generated. Use these details to match the provider’s delivery record with the receiver’s logs.

Trace one delivery across the receiving path

First, check the provider’s delivery history. Confirm that it attempted to send the event and that the webhook subscribes to that event. Check whether the provider reports a timeout or connection failure. Also note any HTTP status it reports. GitHub’s webhook troubleshooting guidance recommends checking recent deliveries and the webhook’s event configuration.

If the provider shows an attempt but you cannot find a matching request in your application, check how the request enters your infrastructure before changing the code. Confirm that the hostname resolves to an address and that the endpoint is publicly reachable. Check that network rules allow the request and that the TLS certificate is valid. A reverse proxy or load balancer can reject the request or time out before it reaches your application.

If the request reached your infrastructure, trace it through the receiving path, starting at the gateway or proxy. Then follow it through the application handler (the code that receives the request), any queue, and the worker that handles queued work. Compare the status and timing at each point. A gateway’s 500 does not prove that the handler threw an error. For example, AWS lists missing permission to invoke a Lambda function and an invalid function response format as possible causes of errors in API Gateway HTTP API Lambda integrations. For that integration type, AWS recommends logging $context.integrationErrorMessage in its Lambda integration troubleshooting guidance.

Measure the handler and worker separately. Record the event ID, when the request arrived, when the receiver responded, the status, and how long the receiver spent writing to a database or calling an external service. Also record how many messages are waiting in the queue and how long the oldest one has waited. These measurements show whether the delay occurred before the handler, while it was running, or after the receiver acknowledged the event.

Check the provider’s deadline and retry rules

Webhook providers set different deadlines for responses and different rules for handling failures. GitHub expects a 2xx response within 10 seconds. Microsoft Graph considers a notification delivered when it receives a 2xx within 3 seconds. If processing takes longer, Microsoft Graph recommends putting the work in a queue and returning 202 Accepted. Zendesk documents a 12-second request timeout and retries timed-out requests up to five times. These limits apply to the named providers, not to webhooks in general. See GitHub best practices, Microsoft Graph delivery guidance, and Zendesk webhook documentation.

A 2xx response tells the sender that the receiver accepted the event. The receiver does not have to finish every later business task before responding. It can validate the request, save the event so it survives a failure, and let a worker handle slower tasks.

Save the event before acknowledging it. If the receiver reports success before saving the event and then loses it, the sender has no reason to send it again. If the receiver waits for every later task, those tasks can make it miss the provider’s deadline.

Move slow work out of the request handler

A handler can miss the sender’s deadline if it calls several external services or sends email. It can also miss the deadline while waiting for a long-running task. Keep the request path short. First authenticate the sender and check the event’s basic structure. Then save the event reliably and return a successful response. A worker can then handle the slower tasks and record failures in its own logs and retry mechanism.

If a handler takes too long while waiting for an external API, save the event to a queue that keeps messages safe and return 202 Accepted after the queue confirms the write. The worker can call the external API afterward. Monitor worker failures and how long messages wait in the queue so a quick response does not hide a growing backlog.

Increase a timeout only when the work has a known maximum duration and both the receiver’s infrastructure and the sender allow enough time. A longer gateway timeout will not help if the provider gives up first. It also will not fix a handler that gets stuck unpredictably while waiting for a database or external service.

Make retries safe

A provider’s retry policy describes how it handles failed deliveries. GitHub says it does not automatically redeliver failed webhook deliveries. Operators can inspect and redeliver them with supported tools, as described in its failed-delivery documentation. Microsoft Graph and Zendesk document different retry rules. Do not assume that a 500 will always trigger another attempt.

A sender can time out after the receiver finishes processing the event but before the response reaches the sender. The sender may then send the same event again. Use the provider’s stable event or delivery ID as an idempotency key, a value the receiver checks to recognize an event it has already handled. Track each event’s processing state and prevent duplicate effects, such as creating a second order for the same event.

Track sender redeliveries separately from worker retries. If both retry without coordination, they can multiply requests during an outage. Document how to send failed events again and confirm that the handler can process them safely. For GitHub, the same X-GitHub-Delivery value identifies a delivery when it is sent again. GitHub also warns that webhook events can arrive out of order. Follow the provider’s documented identifiers and ordering rules; arrival order does not always match event order.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com

Email updates

Usually a new article and a few links I found interesting.

No spam. Unsubscribe with one click.

Leave a Reply

Your email address will not be published. Required fields are marked *