Handling FastAPI Webhooks with HMAC and Idempotent Retries

Consuming external events reliably requires defensive engineering. When receiving fastapi webhooks from providers like Stripe, Shopify, or GitHub, your service must solve two core problems immediately: prove that the payload actually came from the provider, and process the payload without duplicating side effects when the provider retries the request.

If your handler does too much work synchronously, the external provider times out and queues an aggressive retry schedule. If your handler blindly executes database updates or sends emails on every incoming delivery, network drops and duplicates will leave your data in an inconsistent state. Here is a pattern to verify signatures, return 200 immediately, and process jobs safely.

Verifying the Raw Payload for FastAPI Webhooks with HMAC

Most webhook systems use an HMAC signature computed over the raw request body with a shared secret. A common mistake in FastAPI is parsing the body as a Pydantic model or calling request.json() before running hmac signature validation. JSON parsers re-serialize whitespace and key order differently, which invalidates cryptographic hashes.

To verify correctly, read request.body() as raw bytes. Compare the calculated hash against the incoming header using hmac.compare_digest to prevent timing attacks. Only parse the JSON payload after the signature passes inspection.

Decoupling Processing with FastAPI Background Tasks

Webhook endpoints should do as little work as possible before returning a response. The caller does not care about your database writes or cache invalidations; it only wants an HTTP 200 acknowledgment that the message reached you. When designing a scalable API, if your handler takes longer than two to five seconds, platforms like Shopify or Stripe treat the request as failed.

FastAPI background tasks let you hand off processing to a worker thread while immediately returning an empty 200 OK response. For higher scale or long-running work, push the job to an external message broker like SQS or Redis. For straightforward services, the built-in BackgroundTasks abstraction keeps your architecture light.

import hmac
import hashlib
import json
from fastapi import APIRouter, Request, Header, HTTPException, BackgroundTasks, status

router = APIRouter()
WEBHOOK_SECRET = b"super-secret-key"
PROCESSED_EVENTS = set()  # In production, back this with Redis or PostgreSQL

def process_event(event_id: str, data: dict):
    # Business logic runs here out of band
    print(f"Processing event {event_id}: {data.get('type')}")

@router.post("/api/v1/webhooks")
async def receive_webhook(
    request: Request,
    background_tasks: BackgroundTasks,
    x_signature: str = Header(None),
):
    if not x_signature:
        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Missing signature")

    # 1. Read raw bytes for strict validation
    raw_body = await request.body()
    computed_signature = hmac.new(
        WEBHOOK_SECRET,
        raw_body,
        hashlib.sha256
    ).hexdigest()

    # 2. Timing-safe comparison
    if not hmac.compare_digest(computed_signature, x_signature):
        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid signature")

    # 3. Parse JSON only after authentication passes
    payload = json.loads(raw_body)
    event_id = payload.get("id")

    # 4. Check idempotency before queueing heavy work
    if not event_id or event_id in PROCESSED_EVENTS:
        return {"status": "ignored", "reason": "duplicate or missing id"}

    PROCESSED_EVENTS.add(event_id)
    background_tasks.add_task(process_event, event_id, payload)
    return {"status": "accepted"}

Designing an Idempotent API Pipeline

Webhooks guarantee at-least-once delivery, never exactly-once. Upstream servers will retry requests when an intermediate proxy drops a connection or when your service takes slightly too long to respond. Because of this, building an idempotent api pipeline is not optional.

• Track unique identifiers: Most providers include a unique event ID in headers or payload fields. Store this ID in an atomic datastore like Redis or an indexed column with a UNIQUE constraint in PostgreSQL.
• Set an expiration TTL: Webhook retries usually stop after 24 to 72 hours. Expire deduplication keys in Redis after three to seven days to manage memory.
• Lock state during processing: If you rely on external background workers, use a distributed lock or set a pending status in the database to prevent duplicate jobs from running in parallel when two retries arrive simultaneously.
• Handle partial failures gracefully: Ensure state transitions inside your database run inside a single transaction so retries can clean up half-finished state if a process crashes mid-flight.

By verifying the raw body with constant-time HMAC comparisons, acknowledging receipt quickly through background execution, and recording unique event IDs, your webhook integration becomes resilient against dropped packets, traffic spikes, and upstream provider retries.