Using an API Gateway for Webhook Reliability

A webhook is an HTTP request a service sends to report an event, such as a payment update or code change. An API gateway is a managed service that receives these requests. It checks each request, then sends it to the right destination. Receiving a webhook does not store it durably or guarantee that someone will process it.

The strongest reliability design connects the gateway to durable storage, such as a queue that keeps events until workers process them. The endpoint returns success only after the storage confirms that it saved the event. The sender can then stop waiting while workers handle the business logic. Success means the receiver has accepted responsibility for the event, not that it has finished every action tied to it.

Buffer events before acknowledging them

A durable queue handles bursts and holds events while workers are busy or unavailable. The gateway can send each webhook to the queue directly, or send it to a lightweight intake service that saves it to the queue. Workers process the saved events separately. This gateway-to-queue pattern lets the sender receive a response without waiting for downstream work to finish.

Return an acknowledgment only after the queue confirms that it saved the event. If the queue fails and the endpoint still returns success, the sender believes the receiver stored an event that it did not save. Choose the failure response based on the sender’s documented retry behavior and the receiver’s recovery plan. Providers do not all handle failed deliveries the same way.

If the sender does not receive the acknowledgment, it may send the event again even if the queue already stored it. Save a stable event or delivery identifier with the event data, and make workers idempotent. That means processing the same event again does not repeat an unintended business effect. Keep enough delivery details to find duplicates and investigate problems.

Monitor how many events are waiting in the queue and how long the oldest event has been waiting. Decide how workers will increase processing after an outage. If workers clear a large backlog too quickly, they can overload downstream services as those services recover.

Route requests to the right processing path

A gateway can send a request to an intake destination based on its path or host. It can also use the request method or selected headers. For example, separate webhook paths can send events from different providers to different queues or intake services. Versioned paths let an operator change one integration without redirecting every sender. These are examples of gateway routing rules; gateways do not all support the same matching options.

Reject requests that match no route, or send them to a quarantine path set up for unknown traffic. Do not silently send them to a valid handler. Treat a routing header as untrusted until you authenticate the sender. The gateway can read the header, but that does not prove the sender set it legitimately.

A shared gateway gives several integrations one entry point to manage, but a faulty route change can disrupt them all. Limit each change to the routes it needs, and test that every route reaches its intended destination before sending production traffic through it. The path-routing pattern describes the benefits and risks of directing requests by path.

Separate request checks from sender verification

A gateway can check whether a request has required headers or whether its body matches a configured structure. These checks help reject malformed requests early, but they do not identify the sender or show whether the event is safe to apply to business data. Gateway validation features vary. For example, AWS documents specific limits for request checks in its REST API.

Webhook providers often use a signature, a value the receiver checks to verify the sender and detect changes to the request body. The check may depend on the exact bytes in the original body. Stripe’s official Node SDK requires a raw string or buffer for its signature verification method. If a gateway parses the body and then serializes it again, check that the receiving code still meets the provider’s verification requirements.

Choose where to check signatures based on what the gateway can preserve and verify. If the gateway cannot reliably handle the provider’s signature method and the original body, send the request to a lightweight intake service that can. Keep business checks, such as whether an event fits the account’s current state, in the code that applies the event.

Isolate workloads that should not fail together

Routing helps isolate failures when it sends different workloads to separate queues or worker capacity. If one provider creates a large backlog or one event type keeps failing, dedicated workers can continue processing another workload. This is the bulkhead pattern: separating workloads keeps a problem in one from using all shared capacity. Microsoft’s bulkhead guidance describes separate queues and processing instances as one way to isolate asynchronous workloads.

Choose what to separate based on the harm a shared failure would cause. A separate queue for every sender adds operating cost and management work. One shared queue lets a high-volume workload crowd out other events. Give workloads separate queues or workers when their traffic, urgency, or processing needs justify the extra capacity and complexity.

Gateway rate limits protect the capacity to receive requests, but a rejected request counts as a delivery failure for the sender. Set limits with the queue’s capacity and the workers’ processing speed in mind. Also account for the provider’s retry rules. Throttling guidance also recommends setting limits throughout the system, including at components beyond the gateway.

Plan for duplicates, ordering, and retries

A gateway and queue cannot guarantee that each event causes a business effect exactly once. A worker might make a change and then fail before marking the event complete, so the event is processed again. A sender might also retry if it loses the acknowledgment. Design handlers to tolerate duplicates, and decide how they will handle events that arrive out of order. GitHub, for example, says its webhook deliveries can arrive out of order in its troubleshooting guidance.

Providers have different retry rules. GitHub asks webhook endpoints to respond within 10 seconds and recommends asynchronous processing when the work could delay the response, as described in its webhook best practices. That timing applies to GitHub. Check each provider’s current delivery and recovery instructions; some providers do not retry failed deliveries automatically.

Set a limit on how many times workers retry an event, and provide a way to recover events that still fail. The HTTP standard cautions against automatically repeating requests that are not idempotent unless the client has a reason to know repetition is safe. Webhook receivers need explicit duplicate handling because repeated POST requests are not necessarily harmless.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com