Every webhook integration starts the same way. You register an endpoint, the platform posts JSON at it, you do the thing. It works on the first order. Then you get two copies of the same order, or a supplier API times out halfway through, or the platform redelivers an event from three hours ago because your server returned a 500 once. Each time, the thing you did happens twice.
I hit all of that building the operations service behind my print store. The store runs on Shopify; a separate service decides which supplier fulfils each line, sends the order once I have approved it, and tracks what happened. Shopify owns checkout and payment. My service owns every decision after that, which means it owns every way those decisions can go wrong. Below is what the flow actually looks like now, and why each piece is there.
The endpoint's only job is to record the event
The first version of my handler did the work inline: verify, parse, route the order, call the supplier. That is the version you should never ship, for a boring reason. Shopify expects a fast response. If your handler is still talking to a supplier API when the timeout fires, Shopify records a failed delivery and sends the event again. Your first attempt is still running. You have now doubled an order with no bug in your own logic.
So the endpoint does three things and stops:
- Verify the signature.
- Write the event to Postgres.
- Hand the event id to a durable workflow engine, then return 200.
Nothing irreversible and nothing slow happens in the request. Everything that can fail slowly happens later, in a place designed to be retried.
Verify before you parse
Shopify signs each webhook with base64 HMAC-SHA256 over the raw request body. The word doing the work there is raw. If your framework has already parsed the JSON and you re-serialise it to check the signature, you are hashing your own output, not theirs. Key order and whitespace shift, the digest no longer matches, and the natural next move is to “fix” it by loosening the check. That is how a forged payload gets in.
My route captures the unparsed bytes and hands them to one small function, which is also the only place in the codebase that touches the secret:
export function verifyShopifyHmac(rawBody: Buffer | Uint8Array, headerHmac: string | undefined, secret: string): boolean {
if (!headerHmac) return false;
const digest = createHmac("sha256", secret).update(Buffer.from(rawBody)).digest("base64");
const a = Buffer.from(digest, "utf8");
const b = Buffer.from(headerHmac, "utf8");
// timingSafeEqual throws on length mismatch; compare lengths first but still
// run the constant-time compare on the equal-length path.
if (a.length !== b.length) return false;
return timingSafeEqual(a, b);
}Two details worth keeping. Use a constant-time comparison rather than ===,
because a plain string compare returns early on the first differing byte and leaks how much
of a guess was correct. And guard the length first, because Node's
timingSafeEqual throws rather than returning false when the buffers differ in
length. A crash inside your security check reads as a 500 to the platform, which means a
redelivery.
Duplicate delivery is normal, not an error
“At least once” is the delivery guarantee almost every platform gives you, and it is not a weasel word. Retries after a slow response, retries after a deploy, operational replays. Duplicates are a routine input, not an incident.
The fix is to make the duplicate visible. Shopify sends its own identifier for each delivery in a header; I store it with a unique constraint and treat a collision as success. A second copy of an event I already recorded gets a 200 and goes no further. I also store a SHA-256 hash of the body alongside it, which costs nothing and answers a question you will eventually have: was this a plain redelivery of the same payload, or the same event carrying different content?
The insert and the “please process this” signal happen in that order deliberately. The row is committed first; only then does the workflow engine get told. If the hand-off call itself fails, I log it loudly and leave the row sitting there unprocessed. That is recoverable, because of the next two pieces.
Idempotency keys come from the business, not from the clock
The endpoint can be perfectly safe and you can still double-submit to a supplier, because the retry now lives downstream. The only durable protection is that the outbound write carries a key the supplier can de-duplicate on, and that the key is the same every time you retry.
This is where it usually goes wrong. A key built from Date.now(), a fresh
UUID, or an attempt counter is a new key on every attempt, which defeats the entire
mechanism: the supplier sees two different requests and cheerfully creates two orders. The
key has to be derived from the identity of the thing you are doing. Mine is a
purchase-order number built from the supplier, the Shopify order id, and the specific line
items in the request. Same order, same lines, same supplier means the same key an hour
later, after a redeploy, on a different machine, forever. It is also a string a human can
read out over the phone, which matters more than it sounds like it should when you are on a
support thread with a supplier.
Retries with an end, and a defined landing place
Transient failures get retried with backoff. Permanent ones should not be: if the event refers to a record that does not exist, retrying two hundred times just delays the moment someone looks at it. My workflows distinguish the two explicitly and throw a non-retriable error for the second kind.
Retries that are legitimate get a ceiling. Each attempt increments a counter on the event row; past the ceiling the event stops being retried and becomes two things at once: a dead letter, and a task addressed to a human, de-duplicated on the event id so one stuck order does not generate forty identical tasks. “Silently dropped” is not an option in the system and neither is “retried forever”.
Reconciliation is what makes a missed event survivable
Delivery is best-effort no matter how good your endpoint is. A webhook can be lost while you were mid-deploy, or never sent, or dropped because a subscription quietly disappeared. If your only path into the system is the webhook, a lost event is invisible.
So a scheduled job sweeps for events that were recorded but never finished (anything still below the attempt ceiling and not yet complete) and pushes them through again. Between that and the dead-letter rule, an event has exactly three end states: processed, awaiting a human, or picked up on the next sweep. There is no fourth state where it just isn't there.
Failure isolation and a hard stop on spending
Two more rules earned their place the hard way. One broken supplier must not take the store down, so failures are tracked per supplier and a breaker opens on the one that is failing while everything else keeps flowing. And anything that spends money runs behind a guard that fails closed at its cap: if the guard cannot confirm the spend is within budget, the answer is no. A system that spends money should fail by not spending it.
What this adds up to
The current service runs seventeen durable workflow functions, ten of them on a schedule, against eighteen data models. On the revision in production 968 test cases pass, and the ones that matter here cover the parts most likely to bite: signature verification, duplicate handling, idempotency-key construction and retry policy. Those tests are the reason I can change routing logic on a Tuesday without wondering whether I have just re-enabled double-ordering.
None of this is clever. It is five rules: verify the raw body before you parse it, acknowledge duplicates, derive idempotency keys from business identity, give retries both a ceiling and a landing place, and reconcile on a schedule because delivery is only best-effort. What they buy you is the ability to retry anything without holding your breath. In a system where a mistake becomes a physical parcel, that is the whole game.
I designed this service and its failure policy and I run it in production. The code quoted above is from the running service; the store is ovrel.co. Questions welcome: ridgenelson1234@gmail.com.