How Downstream Bottlenecks Cause Webhook Failures

A webhook is an HTTP request that an event-producing service sends to a server you operate. The receiving server must return an HTTP response, such as a success status, to tell the provider whether it accepted the delivery. When the server waits for a slow database, a queue that holds messages until they can be processed, or another service’s API, an interface software uses to communicate with that service before responding, that dependency can use up the provider’s time to respond and cause a timeout or error response.

a downstream bottleneck causes a webhook delivery failure when it delays or prevents the receiver’s response. Work that continues after the receiver has sent a successful response can still be slow, but it does not make the provider’s delivery fail. The provider’s documentation states how long the receiver has to respond and which status codes count as success. It also explains what the provider does after a failed delivery. The receiver’s logs and measurements of its dependencies help locate the bottleneck.

How a slow dependency delays the response

A webhook handler that does work before replying is called synchronous. It receives the request, checks it, and then might write to a database, publish a message to a queue, or call another API. The handler responds to the provider only after that work finishes.

The provider’s response timer runs throughout the wait. It counts database delays, including time the receiver waits for an available connection and time a query takes to finish or a lock to clear. It also counts time spent waiting for the queue to confirm acceptance or for a remote API to respond. If those waits push the handler past the provider’s deadline, the provider can end the connection and record a timeout. For example, GitHub’s webhook guidance says the receiver must return a 2xx status within 10 seconds; otherwise, GitHub terminates the connection and considers the delivery failed. That is GitHub’s rule, not a universal webhook limit.

A handler can also return a 5xx status, an HTTP code that indicates a server error. A database failure, rejected queue publish, or downstream API error might cause the handler to return one. The receiver’s server or a proxy in front of it might also return an error if the handler fails. In either case, the provider sees an unsuccessful response. A timeout is different: the provider might end the connection before the receiver returns any status.

A timeout does not prove the receiver did no work. The handler might finish its database write or external API call after the provider has already given up. If the provider sends the event again, the receiver might repeat work it already completed. Make the receiver recognize duplicate deliveries, and use the provider’s delivery identifier to match a redelivery to the original delivery. GitHub documents that its delivery identifier remains the same when a failed delivery is redelivered.

Where database, queue, and API delays enter

A database slows acknowledgment when the handler waits for a connection or lock, or waits for a query or transaction to finish. Measure how long the handler waits to get a database connection separately from how long its query and transaction take. A handler waiting for a free connection has a different bottleneck from one running a slow query. For PostgreSQL, activity and wait-event statistics help operators inspect active work and waits.

A queue slows acknowledgment when the receiver waits for a message to be accepted before responding. If the handler also waits for a worker that takes messages from the queue to process the message, that worker’s processing time becomes part of the webhook request too. Separate the time spent publishing from the time spent waiting for processing. For Amazon SQS, CloudWatch metrics such as visible-message count and age of the oldest message help show backlog and consumer speed. These metrics describe the whole queue, so they do not prove that a specific webhook request waited for it. Compare them with timing for that request.

A downstream API slows acknowledgment when the handler waits for the outbound connection or for the remote service’s response. Record how long the receiver’s call to the other API takes and the response status. Also record any timeout and the number of retries. Then compare these details with how long the incoming request takes. If a remote service returns an error and the handler passes that failure back as a 5xx, the provider sees the receiver’s unsuccessful response, not the full chain of internal causes.

What provider records can and cannot show

Provider documentation tells you which HTTP statuses count as success, how long the receiver has to respond, and what the provider does after a failed delivery. Delivery records can also include useful evidence. GitHub’s repository webhook delivery API exposes details such as delivery duration and response status, which help identify a slow or unsuccessful request.

These records describe what the provider observed. A timeout or 5xx alone does not identify the cause. The failure could come from the database or queue, or a downstream API. It could also come from the receiver, a proxy, or the network. Match the delivery identifier and timestamp to receiver logs and traces, records that show how a request moved through the receiver and its dependencies. Then examine the time spent waiting on each dependency. If the provider reports a long delivery and the receiver trace shows a long database wait, the two observations support a database bottleneck. The provider record by itself does not establish that cause.

Retry rules also depend on the provider. GitHub says it does not automatically redeliver failed webhook deliveries; operators can redeliver them manually or through code, as described in its failed-delivery guidance. The PayPal REST webhook guide describes automatic retries of failed deliveries, up to 25 times over three days. Check the documentation for the exact product and integration you use before treating a failure as something the provider will retry.

Keep long-running work out of the acknowledgment path

A receiver can shorten the provider-facing request by validating the event, storing it somewhere that can keep it until workers process it, and returning success before those workers perform slower business tasks. GitHub recommends considering a queue for this pattern. The receiver should acknowledge only after it has accepted the event into a place that can preserve it for processing; returning success before safe acceptance risks losing work.

A queue does not eliminate downstream delays. If the receiver waits too long to publish the message, the queue still holds up the acknowledgment. If the receiver responds promptly but messages later build up in the queue, the provider may have recorded a successful delivery while the receiver’s business processing falls behind. In that case, investigate consumer speed and queue age separately from webhook delivery failures.

Retries also add pressure to an overloaded dependency. A provider retry can overlap with receiver-side retries or queue redelivery, sending more work toward the same bottleneck. Google’s guidance on cascading failures recommends bounded retries and randomized exponential backoff, which increases the wait between attempts and varies each wait time instead of sending attempts at fixed intervals. Apply retry limits and backoff at the layers you control, and account for the provider’s own retry policy.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com