Webhook Failures Behind a Proxy

A webhook is an event notification that one service sends to another as an HTTP request. When a provider sends an event to a service behind a proxy, the request passes through one or more intermediaries before it reaches the code that handles it. A proxy accepts traffic for another server and forwards requests and responses between the server and the provider. A content delivery network (CDN) or load balancer can also sit in this path.

Each intermediary uses its own connection, timers, and logs. The provider can report a timeout even when the service finishes processing later. A proxy can also send a 5xx error status that the service did not send. To find the failure, follow the request from the provider through each intermediary to the service, then follow the response back. The provider’s delivery record shows what the provider observed. Proxy and service logs show where the request or response changed or stalled.

Trace the request and response through each hop

A typical path is:

Webhook provider → edge proxy or CDN → load balancer or reverse proxy → service

The provider decides whether delivery succeeded based on the response it receives and whether it arrives before the provider’s deadline. The service’s successful completion does not prove the provider received that response. An intermediary can time out or close the connection first.

Start by identifying which component handled each part of the exchange. For a single delivery, compare the provider’s delivery record with the proxy’s access log and the service’s request and response logs. Match the records by the provider’s delivery ID or another request identifier and by timestamp.

Provider deadlines and proxy timeouts measure different things

Webhook providers set their own rules for what counts as a successful delivery. GitHub, for example, expects a 2xx response within 10 seconds and records a slower response as a failure. Its webhook troubleshooting guidance recommends queuing work so the service can acknowledge the request promptly and process the event afterward.

A proxy adds separate timers. NGINX’s default proxy_read_timeout is 60 seconds between successive reads from the upstream server, meaning the server to which NGINX forwards the request. It is not a limit on the total duration of every request. If the upstream sends data during the interval, the exchange can last longer; if it sends nothing for the configured interval, NGINX closes the connection. The NGINX proxy documentation describes this timer and its other proxy settings.

Other proxy services and load balancers use timers that measure different conditions. Cloudflare documents a default 125-second Proxy Read Timeout for its 524 error, while an AWS Application Load Balancer has a documented default 60-second connection idle timeout. These timers measure different things, so their values are not interchangeable. Compare what each timer measures, not just its duration.

A handler that waits on database writes or another service before acknowledging an event can exceed the provider’s deadline while remaining within a proxy’s timeout. Conversely, an intermediary can time out before the provider’s own deadline. When either component closes its connection, the provider may record a failure even if the service later finishes its work.

Request and response buffering affect different moments

Request-body buffering determines when the service starts receiving the webhook. With NGINX’s default request buffering, the proxy reads the full request body before forwarding it upstream. With buffering disabled, it forwards the body as it arrives. A slow or large request can therefore take longer to reach the service when buffering is enabled.

Response buffering controls how the proxy handles data coming back from the service. NGINX reads an upstream response and stores it in memory buffers; it can write some data to a temporary file if the buffers are insufficient. With response buffering disabled, NGINX passes the response to the client as it receives it. Buffering changes the flow, but it does not establish that the proxy always waits for the full response before forwarding any part of it. Check the proxy configuration and timestamps before attributing an acknowledgement delay to buffering.

For a suspected delay, compare when the provider began sending the request, when the proxy received it, when the service received the body, and when the service sent response headers. Those points help distinguish a slow upload from a slow handler or a delayed response path.

A 5xx status does not identify who generated it

The provider’s status code describes the response it received. It does not, by itself, identify which component produced that response. Under HTTP semantics, a 502 means a gateway or proxy received an invalid response from an upstream server; a 504 means it did not receive a timely response. A proxy can send either status when the service connection fails, even if the service did not deliberately return that code.

Intermediaries can also handle or replace upstream responses according to configuration. For example, NGINX can intercept proxied responses with status codes of 300 or higher and pass them to its error-page handling. Treat a 5xx in the provider dashboard as a clue. It does not prove that the service returned that status.

Compare the status sent to the provider with the upstream status recorded by the proxy and the status logged by the service. AWS Application Load Balancer access logs, for example, distinguish the load balancer’s elb_status_code from the target’s target_status_code, and record separate processing times. That distinction can show whether the target responded and whether the load balancer sent a different result.

Correlate records to locate the failing hop

Start with the provider’s records, then follow the request toward the service:

1. Confirm the provider attempted delivery. Check its delivery record and capture the delivery ID and attempt time. Also record the status, error description, and duration. GitHub notes that recent delivery records can take a few minutes to appear.
2. Find the same request in proxy logs. Compare the status sent to the provider with any upstream status and timing fields. A proxy-generated 504 alongside no upstream response points to a different failure path than an upstream 500 that the proxy passed through.
3. Check the service log. Determine whether the service received the full request, when it began handling it, and what response it sent. A service success recorded after the provider’s timeout does not prove that the provider received the success.
4. Inspect the relevant proxy settings. Check request and response buffering, along with connection and read timeouts. Also check error interception and any retry rules. Then test through the public webhook URL, because a direct request to the service bypasses intermediaries that can affect the result.

The failure stage matters. The proxy can fail to connect to the service, or it can connect and then wait too long for the service to send a response. Cloudflare’s documentation, for instance, separates connection-handshake and request-acknowledgement timeouts for its 522 error. Those thresholds describe Cloudflare behavior, not universal network limits.

Sender retry policies also differ, so a failed attempt does not predict what happens next. PayPal documents retries for specified delivery failures, while GitHub Enterprise Cloud says failed deliveries are not automatically redelivered. Check the relevant provider’s retry guidance or failed-delivery documentation before relying on retries for recovery.

Acknowledge promptly when processing takes longer

When the provider requires a quick response, the service can handle receipt separately from longer-running work. It can validate the request, place the event in a queue, and return a success response. A background worker can then take the event from the queue and finish processing it. The provider’s documentation sets the response deadline and defines which status counts as success. The service’s logs should show whether it accepted the event for later processing.

This approach reduces how long the provider waits for the service to finish handling the request, but you still need to check the proxy. Confirm that the proxy forwards the request body as expected and sends the success response back to the provider.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com