Building Reliable Webhook Receivers for SaaS

When an event occurs, such as a subscription changing or an invoice being paid, one service sends another an HTTP webhook request. In a software-as-a-service (SaaS) product, the receiving endpoint often needs to update customer data or trigger other work based on that event.

Reliable delivery takes more than keeping the endpoint online. A sender may retry after a timeout even if the receiver already handled the request. Some providers do not automatically retry failed deliveries. Authenticate each request, save it before acknowledging it, and process it in a way that tolerates duplicates and supports recovery.

Acknowledge only after durable intake

Keep the HTTP request path short. After the receiver verifies and checks a request, it should save the delivery in durable storage that keeps it available after a restart, then return a successful response. A worker can perform slower business tasks after the sender has received the acknowledgment. GitHub recommends responding within 10 seconds, and describes queueing work as a way to meet that deadline in its webhook best practices.

The acknowledgment means the SaaS has accepted responsibility for the event. It does not mean that every downstream action has finished. If the receiver returns success before saving the event, a crash can erase the work while the provider believes delivery succeeded. If the receiver saves the event first, a worker can resume processing after a restart.

A database-backed event inbox, a database table that holds incoming events, makes that boundary clear. Save the provider, event identity, payload, receipt time, and processing status in the table, then let a worker read events that are waiting to be processed. If the receiver also publishes to a separate queue, it must account for failures between saving the record and publishing the message. The storage design should ensure that an accepted event remains available to a worker even if one of those steps fails.

Providers set different response deadlines and retry rules, so do not use one provider’s behavior as a general standard. For example, Shopify documents a one-second connection timeout and a five-second total request timeout for HTTPS webhooks. It retries failed deliveries eight times over four hours; after eight consecutive failures, it deletes subscriptions configured through the Admin API. GitHub, by contrast, says it does not automatically redeliver failed deliveries. Check the current rules for each provider before choosing response behavior or designing recovery.

Make repeated deliveries safe

Assume that a webhook or an internal queue can deliver the same event more than once. Idempotent processing means that repeating an event produces the same effective business result as processing it once. A handler that grants an entitlement should not grant it twice because a sender retried after a timeout.

Store a stable identifier and enforce uniqueness in durable storage. The exact identifier depends on the provider: some identifiers refer to an event, while others identify a particular delivery. CloudEvents, for example, uses the pair of source and id to identify an event, and a duplicate resend can reuse the same ID. Shopify distinguishes a delivery ID from an event ID: separate deliveries related to the same merchant action can have different delivery IDs but share an event ID. Follow the provider’s documented delivery identity rules rather than choosing a key based on its name alone.

Record processing status alongside the identifier. A worker that receives an already completed event can skip it; an operator can also see whether a saved event is still pending or failed. Keep the uniqueness check and the business change coordinated so two workers handling the same event at once do not both perform the effect.

A queue does not remove the need for idempotency. Amazon SQS documents that a message can be delivered again, including when processing takes longer than the message’s visibility timeout. The worker must therefore remain safe to retry even when the queue provides delivery guarantees.

Handle events that arrive late or out of order

Do not make business correctness depend on webhook arrival order unless the provider explicitly guarantees that order. A later event can reach the receiver before an earlier one, so applying each payload as a fresh state update can overwrite newer data with older data. Shopify says it does not guarantee webhook ordering and recommends reconciliation for keeping local data aligned with provider state.

Decide what each event tells the receiver to do. Some events report a fact, such as a payment being recorded. Others signal that a resource changed and the receiver should fetch it again. To keep local data in sync, the receiver can use the event to retrieve the provider’s current resource state and update its local copy. If the provider documents a version or sequence number that only moves forward, the receiver can use it to reject updates with an older number. Do not treat timestamps as a reliable ordering mechanism unless the provider defines that guarantee.

Verify requests before trusting their contents

Use HTTPS and verify the provider’s signature before acting on event fields. HTTPS encrypts traffic in transit; a valid signature helps verify that the request came from the expected provider and that the signed content was not changed. Signature formats differ across providers, so follow the provider’s exact procedure. GitHub, for example, documents HMAC-SHA256 validation using the X-Hub-Signature-256 header and advises using a constant-time comparison.

When a provider signs the raw request body, verify those original bytes before parsing or re-serializing the JSON. Whitespace or field ordering can change the bytes and cause a valid signature check to fail. The OWASP webhook security guidance describes differences between provider signing schemes, including schemes that sign a timestamp along with the body.

After signature verification, validate the event type, required fields, tenant or account identity, and values that drive business actions. A valid signature does not guarantee that the payload matches the receiver’s expectations. Limit accepted request sizes and methods. Keep signing secrets out of source code and logs. IP allowlisting can add a layer of control, but it should not replace signature verification; provider IP ranges can change.

Give failed work a recovery path

Track delivery recovery separately from processing recovery. Delivery recovery asks whether the provider sent the event and whether it offers a way to retry or redeliver it. Processing recovery asks whether the SaaS accepted the event but failed to finish the work. A provider retry cannot repair a job that the receiver acknowledged and then lost.

Retry temporary worker failures a limited number of times, waiting longer between attempts. Then move events that still fail into a dead-letter queue, a separate queue for failed events, or another failure store that operators can inspect. Give operators a way to examine and replay failed events. Replay must use the same idempotency checks as first-time processing so it does not repeat a completed business effect.

For provider-side recovery, check what each sender actually supports. GitHub says it does not automatically redeliver failed deliveries. It provides delivery records and redelivery options. Shopify recommends reconciliation because webhook delivery can be missed or mishandled. For important customer or financial state, periodically compare local records with the provider’s current records and repair discrepancies.

Monitor the whole delivery path

Logs and metrics should let an operator determine whether a request arrived, passed authentication, reached durable storage, and finished processing. Record structured metadata such as provider, tenant, event type, and event or delivery ID. Also record receipt time, response status, retry count, and final outcome. Avoid logging signing secrets or full payloads, which may contain sensitive customer data.

Watch both the HTTP endpoint and the worker. Endpoint errors and slow acknowledgments point to intake trouble. A growing number of queued events, or a longer wait for the oldest queued event, points to processing lag. Track signature failures and dead-letter volume. Also track replay outcomes and reconciliation discrepancies. GitHub’s delivery records provide one example of provider-side visibility, but the SaaS still needs its own view of post-acknowledgment work.

Test the failure paths before relying on them in production. Send duplicate and out-of-order events, reject an invalid signature, pause workers, interrupt queue access, and replay a failed event. Confirm that the receiver does not acknowledge an event before durable intake and that a worker restart does not lose accepted work.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com