Handling Email Webhooks Reliably: Duplicates, Ordering, Retries and Reconciliation

A webhook handler looks simple: receive a request, update a record, return 200. In production it has to cope with duplicates, events in the wrong order, bursts of traffic and your own outages. Email events are a good example, because they drive suppression, billing, user messages and analytics. A handler that mishandles them can email a person who unsubscribed or show a customer a wrong status. This guide describes a design that stays correct under those conditions.

What to assume about delivery

Nearly every webhook system makes the same basic trade-off. Rather than risk losing an event, it delivers at least once and retries when it does not see a success response. That keeps data safe but means your code will sometimes see an event twice, and sometimes see events in a different order from the one in which they happened. Check the documentation for your platform for its exact retry rules, which we do not assume here.

Step 1: Verify before anything else

Check the signature on the raw request body before you parse, store or act on it. Compare using a constant-time function, and reject requests whose timestamp is too old, if your platform provides one, to limit replay. Our guide to webhook signature verification shows how. A request that fails verification gets a rejection and no processing.

Step 2: Acknowledge fast

Once the signature is valid, write the raw event to durable storage, such as a database table or a queue, and return a success status. Do nothing slow in the request. The goal is to answer within a short time, so that the sender does not see a timeout and retry.

POST /hooks/mail-events
1. verify signature on raw body      -> 401 if invalid
2. insert raw event into inbox table  -> 500 if this fails (sender retries)
3. return 200 immediately

worker: read inbox -> process -> mark done

The sketch is generic. Use whatever storage you already operate, and keep the request path short.

Step 3: Deduplicate

Each event should carry a unique identifier in its payload. Store it in a table with a unique constraint. When an event arrives, attempt to insert the identifier. If that fails because it already exists, you have seen the event, so acknowledge it and move on. Insert-then-process is safer than check-then-process, because two parallel requests with the same identifier cannot both pass.

CREATE TABLE event_inbox (
  event_id   VARCHAR(100) PRIMARY KEY,
  received_at TIMESTAMP NOT NULL,
  payload    TEXT NOT NULL,
  status     VARCHAR(20) NOT NULL DEFAULT 'new'
);

Keep identifiers long enough to cover your platform retry window and your reconciliation periods. A few weeks is a common choice, and you can set your own.

Step 4: Make processing safe to repeat

Deduplication by identifier covers most repeats, but workers can crash halfway. Write the work so that running it twice gives the same result. Setting a status to bounced is safe to repeat. Adding one to a counter is not. If you must count, derive the count from stored events instead of incrementing blindly.

Step 5: Do not trust arrival order

Events about one message have a natural order: accepted, delivered, opened, clicked. They may reach you in another order. Handle this with explicit rules.

Step 6: Handle failures

• Retry internal errors. If a worker fails, leave the event in the queue and retry with backoff, as in our guide to rate limits and retries.
• Limit attempts. After several failures, move the event to a dead-letter store with the error, so it does not block the queue.
• Alert on the dead-letter store. Any growth means something needs attention.
• Keep a replay tool. Fix the bug, then replay stored events through the same worker.
• Scale workers separately. Bursts after a large send can produce many events at once. A queue absorbs them.

Step 7: Reconcile

Webhooks are the fast path. They are not a perfect record. To catch gaps from outages, rejected requests or bugs, compare against the platform source of truth on a schedule. Fetch events or message statuses for a recent window from the event log or API, check that each exists in your store, and process any that do not. Run this at least daily for important data and after any outage.

A worked example: unsubscribes that did not stick

A company stores unsubscribe events in its CRM. After a large newsletter, some people who unsubscribed received the next campaign. The investigation finds two causes. First, the handler did the CRM update inside the request, which sometimes took longer than the sender timeout, so the sender retried and the handler processed the same event twice, creating conflicting records. Second, an overnight outage meant that about an hour of events was never received.

The team rebuilds the handler. The endpoint verifies the signature, inserts the raw event with a unique identifier and returns quickly. A worker applies unsubscribes using a safe, repeatable update. A daily reconciliation job compares the last day of platform unsubscribe events with the CRM and fixes any gaps. Before the next campaign, a pre-send check also compares the platform suppression list with the CRM. The next send respects every opt-out. The example is illustrative, but both failures, slow handlers and missed events, are very common.

Be liberal with payloads

Event payloads change over time. Platforms add fields, introduce new event types and sometimes send several events in one request. A handler that fails on an unknown field or an unknown event type will start rejecting traffic after a harmless upgrade, and the sender will retry the rejected events until they are abandoned. Parse only the fields you need, ignore the rest, and store unknown event types for later instead of returning an error. If a request contains a batch of events, handle each one independently, so that one bad entry does not cause the whole batch to be retried and duplicated.

• Keep the original raw body next to the parsed fields, so you can reprocess it after fixing a bug.
• Log the event type counts per hour. A sudden change often means a new type appeared or a sender setting changed.
• Treat missing optional fields as normal, and give them defaults in code instead of failing.
• Version your own processing code, so that you can tell which release handled each event.

Security and operations notes

• Serve the endpoint over HTTPS only.
• Keep signing secrets in a secrets store, not in code, and rotate them with a plan. Our guide to API key security and rotation applies.
• Limit the request body size, and reject unexpected content types.
• Log the event identifier and outcome for every request, but avoid logging full personal data.
• Test with replayed and duplicated events before going live, and test an outage by switching the endpoint off for a while.

Further reading and sources

• RFC 9110: HTTP Semantics (status codes)
• RFC 6585: Additional HTTP Status Codes