Idempotent Webhook Ingestion at Scale: Surviving Duplicate, Out-of-Order, and Retried Payment Events

In any commercial system integrating multichain stablecoin checkout, digital-goods delivery, or automatic SaaS provisioning, HTTP webhook callbacks are the most fragile part of the payment-to-fulfillment chain.

Most teams initially implement payment notifications with logic resembling this handler of fewer than ten lines:

// A common basic webhook-consumption antipattern
app.post('/api/payment/webhook', async (req, res) => {
  // 1. Basic signature check
  if (!verifySignature(req.headers, req.body)) {
    return res.status(401).send('Invalid signature');
  }

  // 2. Fulfill directly based on event type
  if (req.body.type === 'payment.paid') {
    await grantProductLicense(req.body.data.orderId);
    await db.orders.update({
      where: { id: req.body.data.orderId },
      data: { status: 'PAID' }
    });
  }

  return res.status(200).send({ received: true });
});

Locally, or under a light load of a few orders per hour, this code appears sound. Public-network instability, network partitions, edge-gateway 504 timeouts, or hundreds of concurrent orders can turn it into three serious production financial incidents:

1. Duplicate fulfillment: a merchant completes fulfillment, such as decrementing inventory or calling an external code-delivery API, but congestion causes its 200 OK response to be lost. The payment gateway times out after 10 seconds and retries the same event 60 seconds later under its backoff policy. Without strict idempotency, the merchant fulfills twice, issuing two redemption codes or crediting twice the tokens for one order.
2. Out-of-order state regression: blockchain payment events often arrive in rapid streams, such as payment.confirming → payment.paid. If a network delay holds the earlier confirming event back by 200 milliseconds, the later paid event may arrive first and mark the order paid. The delayed confirming event then regresses the database state to confirming. Customers refresh, see the regression, and raise alarm and support tickets.
3. Stale snapshot overwrite: after a partial refund (refund.completed, refund_status: partial), an old payment.paid event delayed by 5 minutes of retries arrives. If the handler overwrites the database with the pushed snapshot, it silently erases the recorded refund history and breaks end-of-day reconciliation.

The enterprise payment gateway StableIn rejects the assumption that every received event should immediately trigger processing. This article examines its full-stack design: raw-byte HMAC verification, payload-hash deduplication, revision-based monotonic state transitions, and distributed lease locks.

Payment Event Processing: From Raw Bytes to Monotonic Fulfillment

To handle unreliable networks while preserving financial-state integrity, StableIn uses four layers of webhook consumption and delivery defenses:

+-----------------------------------------------------------------------------------+
|                       StableIn Idempotent, Order-Independent Webhook Processing    |
+-----------------------------------------------------------------------------------+
|  1. Byte-Level Signature Verification                                               |
|     - Enforce timestamp skew: abs(nowSeconds - timestamp) <= 300 seconds              |
|     - Constant-time comparison: timingSafeEqual(expectedSig, actualSig)               |
|     - Hash original bytes: sign rawBody to prevent serialization-format changes       |
+-----------------------------------------------------------------------------------+
                                       |
                                       v
+-----------------------------------------------------------------------------------+
|  2. Payload-Hash Deduplication Gate                                                  |
|     - Idempotent primary key: shop_events(event_id CHAR(36) PRIMARY KEY)               |
|     - Payload fingerprint: payload_hash = SHA-256(rawBytes)                            |
|       * Same fingerprint: network retry; return 200 and safely skip (No-op)            |
|       * Different fingerprint: same event_id, different content; block with 409 EVENT_ID_CONFLICT |
+-----------------------------------------------------------------------------------+
                                       |
                                       v
+-----------------------------------------------------------------------------------+
|  3. Monotonic State Machine & Revision Fencing                                       |
|     - Fetch authoritative current snapshot: current = client.orders.get(orderId)     |
|     - Reject revision regression: if (current.revision < event.order.revision) throw 503 |
|     - Monotonic transitions: never regress high-water states (paid/refunded)          |
+-----------------------------------------------------------------------------------+
                                       |
                                       v
+-----------------------------------------------------------------------------------+
|  4. Lease-Locked Outbox & Delivery Retry                                             |
|     - Separate payment accounting from fulfillment via async shop_fulfillment_outbox |
|     - lease_version optimistic lock prevents stale workers acknowledging newer leases |
|     - Backoff: [60s, 300s, 1800s, 7200s, 21600s, 43200s, 86400s]                      |
+-----------------------------------------------------------------------------------+

Comparing Three Webhook Consumer Architectures

Core Implementation: From SDK Verification to Merchant Integration

All source excerpts below come from StableIn’s production repository (stablein-platform).

1. Raw-Byte Verification and Timing Attack Protection (@stablein/sdk)

A common serious mistake is computing HMAC after serializing the JSON object parsed by Express or NestJS with JSON.stringify(). Property iteration order, whitespace, and Unicode escaping can vary across environments; reserialization is almost certain to break verification.

StableIn requires retaining the original binary Buffer received by middleware and combines it with a timestamp and fixed prefix to generate the signature:

// From StableIn-Platform/packages/sdk/src/webhook.ts
import { createHmac, timingSafeEqual } from 'node:crypto'
import type { WebhookPayload } from '@stablein/protocol'
import { StableInError } from './errors.js'

export type VerifiedWebhook = WebhookPayload

export function verifyWebhook(
  rawBody: Uint8Array,
  headers: Record<string, string>,
  secret: string,
  nowSeconds: number
): VerifiedWebhook {
  const normalized = Object.fromEntries(
    Object.entries(headers).map(([k, v]) => [k.toLowerCase(), v])
  )
  const t = normalized['stablein-timestamp']
  const sig = normalized['stablein-signature']

  // 1. Enforce timestamp skew within 300 seconds (replay protection)
  if (
    !/^(?:0|[1-9][0-9]{0,11})$/.test(t ?? '') ||
    !/^v1=[a-f0-9]{64}$/.test(sig ?? '') ||
    Math.abs(nowSeconds - Number(t)) > 300
  ) {
    throw new StableInError('INVALID_WEBHOOK')
  }

  // 2. Build expected HMAC signature: v1 = HMAC-SHA256(timestamp + "." + rawBody)
  const expected = Buffer.from(
    'v1=' + createHmac('sha256', secret).update(t + '.').update(rawBody).digest('hex')
  )
  const actual = Buffer.from(sig)

  // 3. Constant-time comparison prevents timing side channels from revealing signatures
  if (expected.length !== actual.length || !timingSafeEqual(expected, actual)) {
    throw new StableInError('INVALID_WEBHOOK')
  }

  // 4. Parse JSON and verify the unique event ID
  let parsed: any
  try {
    parsed = JSON.parse(Buffer.from(rawBody).toString('utf8'))
  } catch {
    throw new StableInError('INVALID_WEBHOOK')
  }

  if (parsed?.id !== normalized['stablein-event-id']) {
    throw new StableInError('INVALID_WEBHOOK')
  }

  return parsed
}

Our unit tests enforce this strictness: even logically equivalent JSON must fail verification if a single space or property line break changes:

// From StableIn-Platform/packages/sdk/test/webhook.test.ts
it('verifies exact bytes, event id and 300 second clock tolerance', () => {
  const raw = Buffer.from('{"id":"event", "data":"支付"}')
  const secret = 'test-secret'
  const timestamp = '1000'
  const headers = {
    'stablein-event-id': 'event',
    'stablein-timestamp': timestamp,
    'stablein-signature': 'v1=' + createHmac('sha256', secret).update(timestamp + '.').update(raw).digest('hex')
  }

  // Accept the 300-second boundary; reject 301 seconds immediately
  expect(verifyWebhook(raw, headers, secret, 1300).id).toBe('event')
  expect(() => verifyWebhook(raw, headers, secret, 1301)).toThrow()

  // Parsing and reserializing JSON changes its bytes even when content matches; reject it!
  expect(() => verifyWebhook(Buffer.from(JSON.stringify(JSON.parse(raw.toString()))), headers, secret, 1000)).toThrow()
})

For more on signature verification, see Is a Valid Payment Webhook Signature Enough to Fulfill an Order?.

2. Database Event Deduplication and Payload Hash Verification (store.mjs)

A merchant cannot preserve idempotency across nodes or service restarts by recording processed IDs only in memory. It needs a dedicated event table in a relational database such as MySQL or PostgreSQL:

-- From StableIn-Platform/examples/merchant-store/src/store.mjs
CREATE TABLE IF NOT EXISTS shop_events (
  event_id CHAR(36) PRIMARY KEY,
  order_id CHAR(36) NOT NULL,
  payload_hash BINARY(32) NOT NULL,
  created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
  FOREIGN KEY(order_id) REFERENCES shop_orders(id)
) ENGINE=InnoDB;

When a webhook arrives, the consumption transaction first checks whether its event_id already exists:

• Same ID, same payload (idempotent replay): this is a normal upstream retry after a lost network acknowledgment. Return HTTP 200 and stop; never trigger downstream fulfillment again.
• Same ID, different payload (payload mismatch conflict): differing data under one event ID indicates a serious abnormal replay or forgery. Raise 409 EVENT_ID_CONFLICT immediately and write a security log:

// From StableIn-Platform/examples/merchant-store/src/store.mjs
const [[prior]] = await c.query('SELECT payload_hash FROM shop_events WHERE event_id=?', [event.id]);
if (prior) {
  // Strictly compare the SHA-256 hash of the original payload
  if (!prior.payload_hash.equals(digest(raw))) {
    throw new StoreError(409, 'EVENT_ID_CONFLICT');
  }
  return; // Idempotent no-op; exit safely
}

3. Revision Fencing and Out-of-Order Events

Event-ID deduplication alone cannot prevent reordering: an earlier payment.confirming event (E1) may arrive after a later payment.paid event (E2).

StableIn uses a classic distributed-systems pattern: monotonically increasing revision fencing. Every platform-side state transition or refund change atomically increments the order’s revision field (1 → 2 → 3):

// From StableIn-Platform/src/modules/payment/chain/projector.ts
// Increment revision on every platform state-machine update
Object.assign(r, {
  payment_status: status,
  refunded_amount: String(cumulative),
  refund_status: refundStatus,
  revision: String(BigInt(r.revision) + 1n) // Monotonically increment revision
})
await m.query(
  'UPDATE pay_orders SET payment_status=?, refunded_amount=?, refund_status=?, revision=? WHERE id=?',
  [status, r.refunded_amount, refundStatus, r.revision, id]
)

Merchant webhook consumers must strictly follow two core principles:

1. Fetch the latest authoritative view: call client.orders.get(orderId) for the current snapshot.
2. Reject stale revisions: if the locally recorded or remotely fetched revision exceeds the webhook’s revision, the event is late and obsolete. Never roll the order state backward.
3. Keep terminal states irreversible: after an order reaches paid, delayed confirming events are invalid noise. After a full refund (refundStatus === 'full'), a late paid event must never reactivate fulfillment.

// From StableIn-Platform/examples/merchant-store/src/rules.mjs
export function canFulfill(order, localRevision) {
  try {
    assertCurrentRisk(order)
  } catch {
    return false
  }

  // Strict monotonic fulfillment guard:
  // 1. Risk and chain enforcement must allow the operation
  // 2. Payment status must be paid
  // 3. No full refund (refundStatus !== 'full'); refunded amount must be below total
  // 4. Snapshot revision must be >= locally recorded revision (prevent regression)
  return (
    order.fulfillmentDecision === 'allow' &&
    order.merchantRisk.serviceStatus === 'enabled' &&
    order.merchantRisk.chainEnforcement === 'allowed' &&
    order.paymentStatus === 'paid' &&
    order.refundStatus !== 'full' &&
    BigInt(order.refundedAmount) < BigInt(order.amount) &&
    BigInt(order.revision) >= BigInt(localRevision)
  )
}

For the state-mapping rationale, see Mapping Payment Provider Events to Order States and Partial Refunds and Digital Product Entitlements.

4. Fulfillment Outbox and Lease Locking

To prevent failures in a merchant’s code-delivery, email, or inventory service from blocking a webhook until timeout, StableIn recommends separating webhook acknowledgment from actual fulfillment through an asynchronous local fulfillment outbox.

To prevent duplicate fulfillment when multiple nodes claim outbox work concurrently, the system uses optimistic locking with a lease version (lease_version):

-- From StableIn-Platform/src/migrations/1792108804000-PaymentWebhooks.ts
CREATE TABLE pay_webhook_deliveries (
  id CHAR(36) PRIMARY KEY,
  event_id CHAR(36) NOT NULL,
  endpoint_id CHAR(36) NOT NULL,
  status VARCHAR(16) NOT NULL DEFAULT 'pending',
  attempts INT UNSIGNED NOT NULL DEFAULT 0,
  next_attempt_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
  lease_owner CHAR(36) NULL,
  lease_until DATETIME(6) NULL,
  lease_version BIGINT UNSIGNED NOT NULL DEFAULT 0, -- Optimistic lease version
  last_http_status SMALLINT NULL,
  last_error VARCHAR(60) NULL,
  created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
  UNIQUE KEY event_endpoint(event_id, endpoint_id),
  KEY due(status, next_attempt_at)
) ENGINE=InnoDB;

A worker must increment lease_version when claiming a task. If an earlier worker resumes after a garbage-collection pause or network interruption and its lease has expired, the database rejects that stale lease. Two servers cannot process the same fulfillment order simultaneously:

// From StableIn-Platform/src/modules/payment/__tests__/webhooks.spec.ts
it('blocks stale workers from acknowledging a newer lease and permits recovery', async () => {
  await endpoint()
  await event()
  await worker.relay()
  const [row] = await f.db.query('SELECT * FROM pay_webhook_deliveries')

  // Worker 1 claims the lease
  const stale = await worker.claim(row.id)

  // Simulate lease timeout
  await f.db.query('UPDATE pay_webhook_deliveries SET lease_until=DATE_SUB(UTC_TIMESTAMP(6),INTERVAL 1 SECOND) WHERE id=?', [row.id])

  // Worker 2 claims a new lease; lease_version increases
  const current = await worker.claim(row.id)
  expect(current).not.toBeNull()

  // Worker 1's stale acknowledgment is rejected; Worker 2 succeeds!
  expect(await worker.ack(stale, true)).toBe(false)
  expect(await worker.ack(current, true)).toBe(true)
})

Five Production Requirements for Webhook Consumers

Before deploying any payment callback consumer, check these five engineering requirements:

1. Never compute signatures over parsed JSON: use the unchanged binary Buffer (rawBody) of the HTTP request body for HMAC-SHA256, and use crypto.timingSafeEqual against timing side channels.
2. Atomically persist a deduplication record for every event: use event_id as a unique primary key and store the raw payload’s SHA-256 hash. Return 200 safely for identical replays and 409 for conflicting payloads.
3. Use monotonic revisions to prevent regression: never blindly run UPDATE orders SET status = event.status. Check whether the current order snapshot’s revision exceeds the event version, and reject any transition from a completed state back to an intermediate one.
4. Separate payment accounting from actual fulfillment asynchronously: the webhook endpoint should quickly verify the signature, record the event, and write the local outbox. Background queues handle slow inventory updates, external API calls, and email notifications.
5. Use distributed lease locks for asynchronous jobs: in multi-instance deployments, control task claims with lease_version optimistic locks to prevent duplicate fulfillment from split-brain conditions or timed-out workers.

Enterprise Payment Infrastructure and Technical Consulting

The StableIn team builds robust multichain stablecoin acceptance and automated clearing and settlement for cross-border merchants, digital-goods platforms, and Web3 protocols. Alongside ready-to-use merchant checkout, we deliver high-availability infrastructure designs tested at production scale:

• Explore enterprise multichain stablecoin acceptance: StableIn Solutions
• Understand server-side order idempotency: Build an Idempotent Order API with Python and SQLite
• Explore multichain reorg handling: Block Reorg Defense and Finality Confirmation Tiers for Multichain Stablecoin Payments
• Arrange custom payment gateway development and architecture assessment: Discuss Your Technical Requirements with Fone