Currently Available: Need a skilled Software Developer for your next project?
Categories
APIs

Error Handling Patterns for Webhook Endpoints

A webhook endpoint receives an HTTP request from another service when an event occurs, such as a payment update or a repository change. The endpoint must decide whether it received a valid event, whether it can safely record that event, and what response to send. That response acts as an acknowledgment: it tells the sender whether to consider the delivery successful or try again.

First verify the request and save it in storage that survives a service restart; then acknowledge it quickly. A background worker, a separate process that handles saved events, then performs the business action. If that action fails, your service retries the work instead of asking the sender to redeliver an event it already accepted. Providers set different time limits, success codes, and retry rules, so check each sender’s requirements before choosing your endpoint’s responses.

Acknowledge only after saving the event

Keep the endpoint’s synchronous work short, but do not send a success response before the event is safely recorded. If the service returns success and then crashes before saving the request, the sender has no reason to try again, and your service may lose the event.

First verify the request and identify the event. Then save it in storage that survives a service restart before returning the success response the provider requires. A worker can read the saved event and handle longer tasks, such as updating an account or calling another service. If the database or queue is unavailable and the service cannot save the event, return the response the provider treats as retryable.

A 202 Accepted response means that a server accepted a request but has not finished processing it, according to RFC 9110. That definition does not guarantee a webhook provider will treat 202 as successful. Shopify, for example, requires 200 OK and treats responses outside the 200 range as errors. Follow the sender’s documented requirements rather than assuming all 2xx responses work.

Match error responses to the provider

Webhook providers set different deadlines and retry rules. A response that prompts a retry from one provider might be treated as a permanent failure by another.

Provider Documented response requirement What happens after a failure
GitHub.com Return a 2xx response within 10 seconds. Failed deliveries are not automatically redelivered.
Shopify HTTPS webhooks Return 200 OK; the full request has a five-second timeout. Shopify retries eight times over four hours. After eight consecutive failures, it deletes subscriptions created through the Admin API.
PayPal REST webhooks Return a 2xx response. PayPal retries up to 25 times over three days.
Amazon SNS HTTP/S subscriptions SNS retries 5xx and 429 responses; other errors count as permanent failures. Its documented example allows 50 attempts, and SNS supports a dead-letter queue for exhausted deliveries.

These details come from the providers’ respective documentation for GitHub, Shopify, PayPal, and Amazon SNS. The GitHub deadline here applies to GitHub.com; GitHub Enterprise Server 3.19 documents a different deadline. Check the documentation for the exact product you use.

Choose the response based on what happened. If the endpoint could not verify or store the request, respond according to the provider’s documented failure behavior. If the endpoint saved the event but a later business action failed, acknowledge the delivery and retry that action internally. Sending an error response after durable acceptance can trigger a second delivery without fixing the original processing failure.

Verify requests before acting

A webhook request can trigger real changes, so verify that it came from the expected provider before allowing it into business processing. Providers describe different verification methods. GitHub uses a shared secret with an X-Hub-Signature-256 header, and Shopify documents HMAC verification.

A signature is a value the provider calculates from the request body so your service can check that the body came from the provider and has not changed. When the signature covers the body, verify the original bytes before parsing and re-serializing the JSON. Re-serialization can change those bytes and cause verification to fail. PayPal’s troubleshooting guidance identifies body-parsing middleware as a source of verification failures and recommends preserving the raw payload.

Reject invalid signatures and record the reason without logging secrets or unnecessary sensitive data. A rejection might prompt the provider to retry, so monitor repeated verification failures. A burst can indicate a configuration problem or unwanted traffic; it should not enter the ordinary event-processing queue.

Make duplicate deliveries harmless

A sender retries when it does not receive the response it expects. That means the same event can reach an endpoint more than once, including when the endpoint completed the work but its response timed out. Design the business action so that processing a repeated event does not repeat its effect.

A persistent inbox can store each event identifier in a database column with a unique constraint, which prevents the database from storing the same identifier twice. Workers check that record before applying the business change, so the endpoint saves and processes each event only once. Choose storage that survives a restart: an in-memory cache loses its history and cannot coordinate separate server instances. If a downstream service supports idempotency keys, pass it a stable event identifier so it can recognize repeated requests. AWS describes this approach in its guidance on idempotent operations.

Choose identifiers according to what they represent. Shopify distinguishes a delivery ID, which identifies a particular subscription delivery, from an event ID, which can connect deliveries caused by the same merchant action. A delivery ID helps suppress retries of one delivery; an event ID helps correlate related deliveries.

Retry internal work separately

Provider retries cover delivery only, and each provider sets its own retry limit. GitHub.com does not automatically redeliver failed deliveries, while Shopify and PayPal set limits on their retry schedules. After your endpoint saves an event, set a limit on your own retries for temporary failures in business processing.

Retry a worker’s task after temporary problems, such as a short database or downstream-service outage. Wait between attempts so repeated failures do not create an immediate loop. If the worker reaches its retry limit, move the event to a dead-letter queue, a holding area for work that needs investigation or controlled replay. Record enough information to diagnose the failure while protecting sensitive data in the event payload.

For malformed requests, repeated attempts are unlikely to help. For a valid but unfamiliar event type, consider saving it for review rather than repeatedly failing delivery. The endpoint can then accept new event types while limiting business actions to events the code understands.

Handle late and out-of-order events

Webhook events do not always arrive in the order they occurred. GitHub warns that deliveries can arrive out of order, and PayPal documents the same issue for invoicing events. A later-arriving event can therefore describe older state than an event your service has already processed.

When event order affects correctness, compare timestamps or versions if the provider supplies them. If an event describes a change rather than the full current state, fetch the latest object from the provider before making an important update. Choose based on what the provider’s events mean and whether each payload gives a complete snapshot or only a partial change.

Monitor delivery and business completion

A successful HTTP response confirms that the sender received an acknowledgment. It does not prove that a worker completed the intended business action. Monitor intake and processing separately: track response times and status codes at the endpoint, then track queue age, worker failures, and events that remain unfinished.

Alert when dead-letter queues grow. Also alert on repeated signature failures or unusual duplicate rates. Shopify describes response times near its five-second limit as a timeout concern; that guidance applies to Shopify’s delivery contract, not to every provider. For GitHub, use its documented manual redelivery or scheduled recovery options when a delivery fails, since GitHub.com does not automatically redeliver it.

After an outage, compare your records with the provider’s current data where possible. This reconciliation can reveal events that exhausted their delivery attempts or that your service acknowledged but failed to complete. Shopify recommends retrieving missing data after an outage and feeding it through the normal processing logic.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com

Email updates

Usually a new article and a few links I found interesting.

No spam. Unsubscribe with one click.

Leave a Reply

Your email address will not be published. Required fields are marked *