Why Webhook Events Arrive Out of Order
A webhook sends a notice from one service to another when something changes, such as a customer record or an order. Events…
When an API provider sends a webhook, it sends an HTTP request to a consumer’s endpoint to report an event. The endpoint returns an HTTP response, usually with a status code. The provider uses that response to decide whether it considers the delivery successful and whether it will try again.
Provider documentation should explain what each response means, which failures consumers can recover from, and who must act next. In particular, explain whether a success response confirms that the endpoint received the event, stored it for later processing, or completed the related business action. Those outcomes are different: a consumer that acknowledges an event before processing it still needs its own way to recover if later work fails.
Tell consumers exactly when to return a success response. If the endpoint acknowledges receipt as soon as it stores an event durably, say so. If the provider expects the endpoint to finish processing first, state that instead. “Durably stored” means the consumer has saved the event somewhere it can retrieve after a crash or restart.
This distinction matters when consumers process events asynchronously, meaning they acknowledge a request first and handle its contents later. GitHub recommends prompt responses and background processing; Shopify documents queueing to help receivers meet its response deadline. In either case, a successful HTTP response can tell the provider that delivery reached the endpoint without proving that the consumer completed the business action.
Document the response deadline and the provider’s interpretation of status codes alongside the acknowledgment rule. These details vary by provider. Shopify treats responses outside the 200 range, including redirects, as errors; PayPal’s REST webhook documentation says non-2xx responses can trigger retries. Consumers should not have to guess whether a particular status code signals success or failure.
“Bad payload” can describe several different problems, and each needs its own guidance. A consumer might be unable to parse the request body, find a required field, validate a field value, or verify that the request came from the provider. Explain how consumers can recognize each problem and whether they should expect another delivery to help.
For checks that verify a request came from the provider, called signature checks, document any required handling of the request body. PayPal’s invoicing guidance identifies body changes and an incorrect webhook ID as possible causes of signature-verification failure, and advises validating the raw request body before parsing it. A consumer that changes the body before verification could therefore reject a legitimate delivery. Also explain how consumers can distinguish a verification configuration problem from a malformed or unauthenticated request.
The right HTTP status code for an invalid payload depends on the provider. Providers differ in how they handle malformed bodies and failed signature checks. State whether the provider retries those responses, whether a repeated delivery can fix the problem, and how consumers should report a suspected provider-side payload defect.
A receiver failure occurs when the endpoint is unreachable, takes too long to respond, or returns an error while handling the request. Tell consumers which observable results count as a failed delivery and what the provider does next. For example, a delivery log that shows the response code, response time, and attempt number helps a consumer distinguish a timeout from a request rejected during validation. Shopify documents these details in its webhook troubleshooting guidance.
A valid event can still be unfamiliar to a consumer. An unknown event type or action means the consumer does not recognize what the provider is reporting. A consumer can recognize an event but still be unable to apply its requested change to the data it has now.
Give consumers separate instructions for these cases. GitHub says it continues to add event types and actions, and advises receivers to inspect them before processing. Tell consumers how to identify each event. For an unrecognized event, explain whether they may acknowledge it after recording it, ignore it safely, or route it for review. PayPal’s invoicing guidance describes recording and alerting on unexpected event types, an approach that gives consumers visibility without treating every new event as a malformed request.
For a recognized event that conflicts with local state, explain the recovery path. The consumer might need to fetch the provider’s current resource state, defer the event for reconciliation, or send it for manual review. If the provider can deliver events in a different order from the order in which it sent them, say so and explain whether sending the same event again can resolve the conflict. PayPal’s invoicing guidance discusses out-of-order events and checking current resource state, but providers do not share a universal response rule for these conflicts.
Consumers need the provider’s exact retry policy: which responses trigger another attempt, how long the provider waits, how many attempts it makes, and what happens when it stops. Also state how long the provider waits for a response before timing out. An automatic retry can succeed after a temporary outage ends. A request-format problem that causes the same failure on every attempt usually requires a correction or a different recovery path. Repeating an unchanged request does not, by itself, fix a payload incompatibility.
Provider policies illustrate why consumers cannot infer retry behavior from the word “webhook.” Shopify documents a five-second whole-request timeout and retries failed deliveries up to eight times over four hours. Its troubleshooting guidance says delivery attempts stop after those retries, and that persistent failures can lead to removal of some subscriptions, depending on their configuration. PayPal documents up to 25 attempts over three days for unsuccessful REST webhook deliveries, followed by a failed status and a manual resend option. GitHub says it does not automatically redeliver failed webhooks, though consumers can use manual or scripted redelivery.
These are product-specific rules, not defaults. The following comparison summarizes the documented examples:
| Provider and scope | Documented failure behavior | Recovery after failure |
|---|---|---|
| Shopify webhooks | A response outside the 200 range or no response counts as an error. Shopify documents a five-second request timeout and up to eight retries over four hours. | After retries stop, consumers need to recover missed data; some subscription types can be removed after persistent failures. See Shopify’s delivery rules. |
| PayPal REST webhooks | Non-2xx responses can be retried up to 25 times over three days. | A failed event can be resent manually from the dashboard. See the PayPal REST webhook overview. |
| GitHub webhooks | GitHub does not automatically redeliver failed deliveries. Its best-practices guidance calls for a response within 10 seconds. | Consumers can redeliver manually or use code to identify and redeliver failures. See GitHub’s failed-delivery guidance. |
Also state who must recover an event after the endpoint acknowledges it. When the consumer returns success after saving an event, the provider marks delivery successful. If processing later fails, the consumer generally needs its own recovery method. It can retry the work or replay the saved event. It can also compare its data with the provider’s current state and make any needed corrections. Name the available recovery option and when consumers should use it.
A provider retry can deliver the same event more than once. Consumers need to know which identifier to use to recognize duplicates and how long the provider might resend an event. Idempotent processing means that handling the same event again produces the same result as handling it once. Shopify recommends idempotent handling and documents using a webhook ID to detect duplicates; PayPal’s invoicing guidance also discusses duplicate deliveries.
Tell consumers what delivery information they can use when investigating a failure. Useful details include an event or delivery identifier, attempt count, timestamp, response code, and the reason the provider recorded the attempt as failed. Document where consumers can view that information, whether they can request a manual redelivery, and how to contact support about an event that cannot be recovered through replay.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com