Stop discovering failures
from your customers.

Last month, an automation pipeline processing order webhooks started losing events. The workflow reported 100% success in the dashboard, but downstream inventory was completely desynchronized. Here is the forensic autopsy.

1. The Trap: Status 200 From an Intermediate Proxy

When the upstream database stalled, the ingress cloud proxy hit a gateway timeout. But instead of bubbling a raw 504 to the workflow engine, an intermediate CDN caught the timeout and rendered an HTML "Friendly Maintenance" error page with a 200 OK response header.

Because the automation engine only checked response.status < 400, it marked the execution as successfully completed and discarded the payload!

2. The 4-Line Defensive Trap

To defend against status-spoofing proxies, every critical webhook receiver must assert schema integrity before acknowledging receipt:

// 1. Assert content-type is strictly JSON
if (!response.headers.get("content-type")?.includes("application/json")) {
  throw new Error(`Invalid content-type: received HTML maintenance page`);
}
// 2. Validate payload schema signature before downstream mutation
const payload = await response.json();
if (!payload?.id || !payload?.event_type) {
  throw new Error(`Schema mismatch: expected event payload, received empty body`);
}

3. The Golden Rule of Ingestion

Decouple intake from execution. Never execute multi-step business logic directly inside the webhook HTTP response lifecycle. Store the raw payload in an idempotent dead-letter buffer first, return 202 Accepted, and let a reconciled worker process the queue.