FastAPI Redis Queue for Reliable Webhooks

Webhooks from payment providers, CRMs and online stores expect a quick answer. If your endpoint does the real work while the sender waits, such as updating several systems or calling slow APIs, requests time out, senders retry and duplicates appear. A common fix is middleware that accepts each event, stores it on a queue and responds immediately, while separate workers process events at their own pace. A FastAPI Redis queue is a lightweight way to build that: FastAPI receives and validates requests, Redis Streams hold the events, and worker processes read them through consumer groups, acknowledging each one when it is done. This guide explains when the pattern fits, how the pieces work, how to handle failures and duplicates, and what to monitor.

When a FastAPI Redis queue fits

The pattern suits workloads where:

• Senders need a fast acknowledgement, but processing takes longer
• Bursts of events arrive, such as end-of-day order batches, and you want to smooth them out
• Processing calls APIs with rate limits, so you need to control the pace
• You want to retry failed processing without asking the sender to resend

It is not always necessary. For light work that finishes quickly, FastAPI's own background tasks may be enough. FastAPI's background tasks documentation notes that for heavy background computation that does not need to run in the same process, bigger tools that use a message or job queue, such as RabbitMQ or Redis, may be more suitable. If you already run a managed queue service, that may be a better choice than adding Redis.

How the pieces work

FastAPI handles the incoming request: it checks the signature or shared secret, validates the payload shape, writes the event to the queue and returns a 202 Accepted response.

Redis Streams store events as an append-only log. According to Redis's streams documentation, consumer groups let several consumers share the work of reading a stream, each message is delivered to one consumer in the group, and consumers acknowledge processed messages with XACK. Messages delivered but not yet acknowledged sit in the group's pending entries list, which XPENDING reports on.

Workers run as separate processes. Each reads a batch of new messages for its group, processes them, and acknowledges each one. If a worker crashes before acknowledging, the message stays pending and can be claimed by another worker later.

Worked example: queuing order webhooks

Suppose a distributor receives order webhooks from its online store and must update its CRM and warehouse system for each one. This simplified, untested sketch shows the two parts in one listing: the FastAPI endpoint and a worker that runs as a separate process. Signature checking, logging and configuration are reduced to comments.

# api.py: receives webhooks and queues them
import redis.asyncio as aioredis
from fastapi import FastAPI, Request

app = FastAPI()
queue = aioredis.Redis(host="localhost", port=6379, decode_responses=True)
STREAM, GROUP = "order-events", "order-workers"

@app.post("/webhooks/orders", status_code=202)
async def receive_order(request: Request):
    body = (await request.body()).decode()
    # verify the sender's signature here and reject invalid requests
    event_id = request.headers.get("X-Event-Id", "")
    await queue.xadd(STREAM, {"event_id": event_id, "payload": body}, maxlen=100000)
    return {"queued": True}

# worker.py: run one or more copies as separate processes
import json
import redis

def run_worker(name, handle):
    r = redis.Redis(host="localhost", port=6379, decode_responses=True)
    try:
        r.xgroup_create(STREAM, GROUP, id="0", mkstream=True)
    except redis.ResponseError:
        pass  # the group already exists
    while True:
        batches = r.xreadgroup(GROUP, name, {STREAM: ">"}, count=10, block=5000)
        for _stream, messages in batches or []:
            for msg_id, fields in messages:
                handle(fields["event_id"], json.loads(fields["payload"]))
                r.xack(STREAM, GROUP, msg_id)

The handle function updates the CRM and warehouse. If it raises an exception, the message is not acknowledged and stays pending. The maxlen argument keeps the stream from growing without limit; the figure is the distributor's own choice, and trimming is approximate by default in the Redis client. In production, the worker would also catch errors per message, log them and move repeatedly failing messages aside, as described below.

Validating requests at the edge

The endpoint is the only part exposed to the internet, so it should reject bad input before anything reaches the queue. Verify the sender's signature or shared secret using a constant-time comparison. Check the content type and a maximum body size. Validate the payload against a schema, which FastAPI makes straightforward with typed models, and return a clear error for malformed requests rather than queuing them. Rate-limit by sender if abuse is a risk. Store the raw body alongside any parsed fields, because signature checks and later debugging both depend on the exact bytes received.

Handling duplicates

Senders retry, networks repeat requests and a worker can crash after doing the work but before acknowledging it. Duplicates will happen, so make processing idempotent:

• Use the sender's event ID where one exists, and record processed IDs in a database table with a unique constraint
• Skip events whose ID has already been processed
• Prefer upserts keyed on stable identifiers when writing to other systems

Idempotent processing turns retries from a risk into a safety net. Our guide to webhook queues covers the wider design of queue-based event processing.

Handling failures

Plan for three kinds of failure:

• Temporary errors, such as a downstream API timing out or returning a rate-limit response: leave the message unacknowledged, back off and retry
• Stuck messages, left pending by a crashed worker: periodically claim messages that have been idle too long, for example with XAUTOCLAIM, so another worker can process them
• Permanent errors, such as invalid data: after a set number of attempts, copy the message to a separate stream for review, acknowledge the original and alert someone

Track the number of delivery attempts per message, which XPENDING reports, so a message that fails every time does not cycle forever.

Running it in production

A few operational points keep the pattern reliable:

• Configure Redis persistence appropriately for your tolerance of data loss, or use a managed Redis service with persistence
• Run at least two workers so one crash does not stop processing
• Shut workers down gracefully, finishing the current message before exiting
• Keep the endpoint thin; any slow work inside it defeats the purpose
• Secure Redis with authentication and network restrictions

What to monitor

Watch the signals that show the queue is healthy:

• Stream length and the number of pending messages per group
• Age of the oldest pending message
• Processing rate and error rate per worker
• Messages moved to the review stream
• Endpoint response times and rejected requests

An ageing oldest message is often the first sign of trouble, such as a downstream system being down. Alert on it with a threshold that reflects how quickly the business needs events processed. Our guide to React and Python covers how Python services like this fit into wider business applications.

Common mistakes

• Doing slow work inside the webhook endpoint
• Acknowledging messages before processing them
• No idempotency, so retries create duplicates
• Letting failed messages cycle forever without review
• Unbounded stream growth
• Running Redis without persistence or authentication

Frequently Asked Questions

What is a FastAPI Redis queue?

It is a pattern where a FastAPI endpoint accepts events and writes them to Redis, often Redis Streams, and separate worker processes read and process them.

Why not process webhooks directly in the endpoint?

Slow processing makes senders wait, time out and retry. Queuing lets the endpoint respond quickly and workers process events at a controlled pace.

What happens if a worker crashes?

Unacknowledged messages stay in the consumer group's pending list. Another worker can claim them after an idle period and process them.

Is FastAPI's BackgroundTasks enough?

For light, quick work, often yes. FastAPI's documentation suggests bigger tools with a message queue for heavy work that does not need to run in the same process.

Build event processing that keeps up

We design and build Python middleware that queues, processes and monitors business events reliably. See our custom app development services, or talk to us about your integration load.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.