Designing Webhook Contracts for Partial Failures

A webhook sends an HTTP request from one service to another when an event occurs. The receiving service must decide whether the request arrived, whether its contents make sense, and whether it completed the work the event requires. Those outcomes can differ: the receiver can accept a valid event even while a database or another downstream service is unavailable.

The webhook contract, the agreement between sender and receiver, should describe each kind of failure and explain what each response means. In particular, it should state whether an HTTP 2xx response confirms that the receiver got the request, saved it so it can recover it later, or finished the business work. That distinction lets the sender retry delivery when appropriate and lets the receiver handle later processing failures without creating avoidable duplicate work.

Transport failures leave delivery uncertain

A transport failure occurs when the sender cannot reliably deliver a request or receive its response. A timeout, connection reset, or network problem can leave both sides unsure whether the receiver accepted the event. The receiver might have stored it even though the sender never received the acknowledgment.

Because the sender might not receive an acknowledgment, it may send the same event again. The contract should give each event an identifier that stays the same across delivery attempts, so the receiver can recognize duplicates. It should also make event handling idempotent, meaning that processing the same event again does not repeat its side effects.

The contract should state how long the sender waits for a response before timing out, when it retries, which responses prompt a retry, how long it keeps retrying, and what happens when it stops. It should also say what a 2xx response confirms: that the receiver got the request, saved it so it can recover it, or finished processing it. HTTP response codes alone do not answer every question about a particular sender’s retry behavior. For example, GitHub’s webhook guidance recommends a response within 10 seconds and background processing, while other providers set different deadlines.

The receiver should verify the request’s identity and save the event somewhere recoverable before acknowledging it. For example, it can store the event in a durable queue. This way, a service restart will not erase accepted work. If the receiver cannot do that, it should return the response the sender’s contract defines for a temporary failure. A 2xx response before durable acceptance can stop retries while leaving the event unrecorded.

Semantic failures require an explicit rejection policy

A payload has a semantic failure when it reaches the receiver but the receiver cannot safely understand or accept its contents. Examples include malformed JSON or a missing required field. A payload can also contain an invalid value or use an unsupported schema version. The contract should define the payload format and required fields. It should list supported versions and explain how the receiver handles unknown event types and fields.

Before accepting a payload, the receiver should check it and return the rejection response named in the contract if it is invalid. The sender might try again after a rejection, but retries will not fix a payload that is malformed every time. The contract should say which failures end delivery attempts and which responses prompt another attempt.

A receiver that handles only certain event types may receive a valid event type it does not use. The contract should say whether the receiver should reject it, acknowledge it without taking action, or route it for inspection. Otherwise, the sender might repeatedly retry an event the receiver will never support, or the receiver might silently discard an event it was expected to process. GitHub’s guidance advises receivers to check event type and action, and notes that the available set can expand.

Do not assume that a particular status code has the same retry meaning across webhook providers. GitHub records 4xx and 5xx responses as delivery failures, while Amazon EventBridge API destinations retry selected 4xx responses, including 401, 407, 409, and 429, as well as 5xx responses. The sender’s documentation must determine how the receiver signals a permanent payload problem.

Downstream failures happen after acceptance

A downstream processing failure occurs after the receiver has accepted a valid event but cannot complete the requested business work. An internal service might be unavailable, a database write might fail, or a business rule might prevent an action.

Many receivers acknowledge an event after saving it somewhere recoverable, then do the business work later, outside the request. The receiver retries temporary processing failures and keeps failures that continue for investigation and later replay, or processing again. It should also make those failures visible through operational records or alerts. The receiver should not acknowledge an event just because it started work. It should first save the event somewhere recoverable.

If the receiver waits for all business work before responding, a later failure can cause the sender to resend the event. Earlier steps might already have succeeded, so the receiver needs to handle the event idempotently, meaning that handling it again does not repeat its side effects, or use another way to prevent those effects from happening twice. In either design, the contract should say whether the HTTP response reports delivery acceptance or business completion. GitHub recommends background processing to help receivers respond within its deadline. That recommendation applies to GitHub’s webhook service. The contract should make the durable-acceptance boundary explicit as a design choice.

Batch webhooks need item-level rules

When a request contains several items that the receiver processes separately, one HTTP response normally reports success or failure for the whole request. A 2xx response does not automatically identify which items succeeded. A response body containing item-level results helps only if the sender’s contract says it will read and act on those results.

For a batch webhook, the contract should define identifiers that stay the same for each item, how the receiver reports failed items, and whether the sender retries only those items or sends the whole request again. It should also specify what the sender does when the response is missing or invalid. If the sender supports only whole-request success or failure, the receiver should assume that a retry can repeat items it already processed and use idempotency to handle that replay safely. As an example of an explicit partial-batch mechanism in a different event-delivery context, AWS Lambda’s SQS integration lets a consumer identify failed messages so successful messages do not have to be treated as failures too.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com