Designing and Implementing Webhook Integrations

Define webhook events, subscriptions, and delivery contracts

A webhook contract defines which changes a sender reports, where it sends them, and what the receiver can expect in each request. Keep event types narrow enough that a subscriber can tell what happened and decide what action to take. A subscription connects selected event types, such as order.updated, to a destination URL.

Give every request a documented envelope: the outer fields that identify and describe the event, separate from the event’s detailed payload. Include an event ID, event type, timestamp, and schema version, which identifies the format of the event data. Define what each field means: for example, whether the timestamp records when the change happened or when the sender created the request. Also specify whether the event ID stays the same across redelivery and whether a separate delivery ID identifies an individual attempt. Providers use these identifiers differently; Shopify distinguishes event and delivery IDs.

The payload should contain the data subscribers need to interpret the event. State whether it represents a complete resource or only changed fields, and document which fields can be absent or null. Set a maximum payload size and describe what happens when an event exceeds it. The OWASP webhook guidance cites the Standard Webhooks specification’s recommendation to keep payloads under 20 KB; that is a recommendation, not a universal webhook limit. For larger records, send a stable resource identifier and let the receiver fetch details through an API.

Document delivery behavior alongside the data shape. Specify the expected HTTP response, timeout, retry policy, and any ordering guarantees. These rules vary by provider: GitHub’s guidance says failed deliveries are not automatically redelivered, while Shopify documents retries. Design for duplicate delivery and avoid promising exactly-once delivery.

When a contract changes, publish the new schema and explain which subscriptions it affects. Existing receivers can tolerate additive fields more easily. Removing or redefining fields requires a clear versioning and migration plan. Keep old and new versions available during a stated transition period when feasible, and announce the change through the provider’s documentation or release notices. The CloudEvents webhook specification offers a reference for a common delivery shape, but providers do not all follow one universal contract.

For each event type, document the exact state change that triggers it. For example, define whether invoice.paid fires when payment succeeds or when the invoice record is later updated to show payment. Clear trigger rules help subscribers interpret events consistently.

Secure webhook endpoints and authenticate deliveries

Webhook receivers accept HTTP requests from another service, so protect the endpoint before allowing a payload to trigger work. Use HTTPS to encrypt traffic in transit, then verify each delivery with the authentication method the sender documents. A valid signature helps confirm who sent the request and whether its body changed; it does not establish that every value in the body is safe.

For a shared-secret signature, the sender and receiver use the same secret to calculate a cryptographic signature. Verify that signature against the raw request body, before parsing or reformatting JSON, because changing the bytes can invalidate the check. Compare signatures with a constant-time function to reduce information leaks from timing differences. Providers use different signature formats, so follow the provider’s instructions or maintained library rather than assuming one header or algorithm applies to every webhook. GitHub’s verification guidance describes its HMAC-SHA256 method; OWASP’s webhook guidance covers signature verification more broadly.

A signature does not prevent an attacker from replaying a previously valid request. When the provider’s scheme includes a signed timestamp, reject deliveries whose timestamp falls outside the freshness window you configure. Also use an event or delivery identifier to detect duplicates, since senders can retry deliveries. Check the provider’s documentation to learn which identifier represents an event and which represents an individual attempt.

Treat signing secrets like credentials. Store them where access is restricted. Keep them out of source code and logs. Never place them in callback URLs. When rotating a secret, coordinate the change with the sender: accept the old and new secrets during a supported transition, then remove the old secret once the sender signs with the replacement. Confirm that the provider supports overlapping secrets; otherwise, follow its documented rotation procedure.

Authentication answers, “Did this request come from a sender with the signing secret, and was the signed content preserved?” Authorization answers, “Is this sender or event allowed to cause this action here?” After authentication, validate the event type, payload fields, and the permissions or account context relevant to the requested action. A valid signature alone should not grant access to unrelated records or operations.

If your service lets customers register callback URLs and then sends webhooks to those URLs, validate destinations to prevent server-side request forgery (SSRF), an attack that tricks your server into contacting internal or otherwise restricted network addresses. Restrict URL schemes to HTTPS and block non-public destinations. Enforce address checks when connecting, since DNS resolution can change between validation and delivery; recheck any redirected destination as well. These controls apply to services that make requests to user-supplied URLs, rather than to a fixed inbound endpoint. OWASP’s guidance details these URL and network checks.

Build reliable delivery with timeouts, retries, and replay

A webhook receiver should save each valid event durably, then acknowledge it to tell the sender it has accepted the event. If the receiver returns success before saving the event durably, a later failure can cause the receiver to lose it. After saving the event, return the response the provider expects and process the event asynchronously. Set a bounded request timeout so a slow database or downstream service does not leave the sender waiting indefinitely. Check the provider’s response deadline and keep your receiver’s deadline within it; providers set different limits.

Design consumers for at-least-once delivery: the sender attempts delivery until it considers the event accepted or its retry window ends, so the same event can arrive more than once. A timeout can leave the sender unsure whether the receiver saved the event, prompting a retry even when the first request succeeded. Store a stable event identifier with a uniqueness constraint, and make business changes safe to repeat. Providers may use separate identifiers for an event and for an individual delivery attempt. Follow the provider’s documentation to choose which identifier to use to recognize duplicate events. Shopify documents this distinction.

Keep a delivery record with a state such as pending, processing, succeeded, or failed, plus attempt count and error details. Operators can check the record to see whether an event is waiting, still running, or needs attention. Treat temporary conditions, such as timeouts or service errors, as retryable. Treat permanent problems, such as an invalid signature or an event the receiver cannot process, as failures requiring correction or investigation. Follow the sender’s response rules: providers interpret HTTP status codes differently, and a response that stops retries for one provider may not do so for another.

For retries you control, use exponential backoff, which increases the delay after each failed attempt, and add random jitter so many deliveries do not retry at the same moment. Set a retry window that limits how long automatic attempts continue. After that window, move the event to a dead-letter queue, a holding area for events that exhausted automatic retries, and alert an operator. Preserve the event and its failure details so the operator can correct the cause and replay it. Replay should pass through the same deduplication and processing safeguards as an ordinary delivery.

Apply rate limits when replaying or processing a backlog. If the receiver returns a rate-limit response, the sender’s documented behavior determines whether and when it retries. Internal workers should process events more slowly or limit concurrency to avoid overwhelming downstream services. Monitor failed deliveries. Track both the age and size of the backlog. Retry policies are provider-specific: GitHub does not automatically redeliver failed deliveries, while Shopify documents automatic retries over a defined window. Confirm each provider’s timeout, retry period, and manual recovery options.

Do not assume events arrive in the order they occurred. A sender may delay or retry an earlier event while delivering a later one. If order matters, design for it explicitly, such as by serializing work for the relevant resource or checking the provider’s current resource state before applying an update. Shopify states that it does not guarantee webhook ordering.

Process webhook requests safely and idempotently

A webhook receiver should verify and validate each request, save it durably, and only then acknowledge it. This lets the sender stop waiting while a worker handles slower business logic, and it gives the receiver a safe basis for handling duplicate deliveries.

Read the unmodified request body before parsing JSON or changing its encoding. Check the signature against those raw bytes using the provider’s documented method; parsing and re-serializing the body can change the bytes and cause verification to fail. After verification, validate the event envelope, the outer structure that identifies the event. Check that required fields such as the event ID and type exist. Confirm that the receiver handles the event type and that the payload matches the expected shape. A valid signature proves the request’s origin and integrity, but the receiver still needs to validate its contents. OWASP’s webhook guidance covers signature checks and safe event handling.

Use the provider’s stable event ID to prevent duplicate work. Store that ID with a uniqueness constraint in the database, so concurrent copies cannot both pass a separate “check, then insert” step. Some providers distinguish an event ID from a delivery ID, which identifies an individual delivery attempt; use the identifier that matches the deduplication goal. Shopify documents both identifiers. Make the business action safe to repeat as well. A worker could complete an action outside the database, then fail before recording that it finished.

Save the verified event or a durable queue message before returning success. A queue stores events durably until a separate worker reads and processes them. If the database or queue write fails, return a failure response rather than implying the receiver accepted work it did not save. Once the event is safely recorded, return the success status required by the provider. 202 Accepted signals that the receiver accepted the event for later processing, though each provider’s delivery contract determines which responses count as successful. The CloudEvents webhook specification distinguishes accepted work from completed processing.

For example, a payment service sends an event saying an invoice was paid. The endpoint verifies the raw-body signature, checks the event ID, type, and payload, then inserts the event into a durable queue. If the insert succeeds, it acknowledges the request. A worker later marks the invoice paid and records its result. If the sender repeats the delivery, the unique event ID prevents the receiver from enqueuing it twice. If the save fails, the endpoint returns an error so the provider can follow its documented recovery policy. Providers differ in whether and how they retry, so check their rules rather than assuming every failure triggers redelivery; GitHub, for example, documents a separate failed-delivery recovery process.

When event order affects results, do not apply arrivals blindly. A delayed older update could overwrite newer state. Compare provider-supplied timestamps or sequence information where available, or fetch the resource’s current state from the provider before updating local data. Some providers do not guarantee ordering, so verify their delivery contract and choose a state-update rule that accounts for that behavior.

Test, observe, and operate the integration

Test webhook handling against a provider’s sandbox, a test environment, or a local endpoint before sending production events. Check that valid signatures pass and altered bodies or invalid signatures fail. Send the same event more than once, then confirm the receiver records it once and does not repeat its business effect. Test the receiver with a slow handler. Also simulate a database or queue outage and a worker failure. Verify that the receiver acknowledges an event only after it has stored the event durably, and that recovery follows the provider’s documented retry and redelivery rules. Those rules vary: GitHub’s webhook guidance describes no automatic redelivery for failed deliveries, while Shopify’s delivery guidance documents retries.

Keep delivery logs that let an operator trace a request from receipt through processing. Record the event ID and delivery ID when the provider supplies both, along with event type, receipt time, response status, processing result, and latency. A correlation ID connects records for the same event across the receiver, queue, and worker. Pass the same ID through each of those components. Avoid logging full payloads, signing secrets, or signature and authorization headers. Payloads can contain personal or confidential data, so log only fields needed for diagnosis, restrict access, and set a retention period. OWASP’s webhook security guidance recommends logging delivery details while excluding sensitive request content and credentials.

Track endpoint response latency and failure rates, plus the age and size of any unprocessed queue. Alert when failures persist. Also alert when latency rises above the service’s established baseline or the oldest queued event keeps getting older. Check endpoint health from outside the receiver as well as inside it: a successful health check should confirm that the endpoint can accept work, while queue and worker metrics reveal whether accepted events are actually progressing. Use provider dashboards and delivery logs to find failed attempts; Shopify’s troubleshooting guidance describes its own delivery metrics and logs.

Review subscriptions as an operational control. Subscribe only to event types the integration uses, and verify that required subscriptions remain enabled after endpoint changes or repeated failures. Providers handle failed subscriptions differently, so check the provider’s current rules before relying on automatic retries or assuming a subscription remains active. Keep a runbook that tells the on-call operator how to find a failed delivery, distinguish authentication errors from endpoint or processing failures, restore service, and safely replay or reconcile missed events. Record who may change subscriptions and how operators confirm that recovery has completed.

Before enabling a production subscription, test the receiver against a local endpoint or provider sandbox. Send valid events and confirm signature verification uses the exact raw request body. Test a bad signature and an expired signed timestamp. Then test a duplicate event and a malformed payload. Each case should produce the documented response without repeating a business action.

Simulate timeouts, server errors, and a database or queue outage. Confirm the provider retries failures as expected, the receiver acknowledges an event only after durable storage, and recovery drains queued work without losing or duplicating effects. Test replay and out-of-order events as well, especially if updates depend on sequence or current provider state.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com