Celery Task Queue: Run Background Jobs Reliably

Run a Celery task queue reliably: idempotent tasks, late acknowledgement where safe, retries with backoff, Redis visibility timeout and worker monitoring.

Web applications and integrations often need to do work that should not happen inside a user's request: sending emails, generating reports, syncing records with a CRM or calling slow third-party APIs. Doing this work in the background keeps the application responsive and lets failures be retried without the user waiting. In Python, Celery is one of the most widely used tools for this. A Celery task queue lets your application hand jobs to workers through a message broker such as Redis or RabbitMQ. This guide explains how Celery works, how to configure acknowledgements and retries safely, how to make tasks idempotent and how to monitor workers in production.

Why background jobs need a queue

Running slow work inside web requests causes problems:

Users wait for operations they do not need to see complete
Timeouts cut off work part-way through
Failures surface as errors to users instead of being retried quietly
Traffic spikes overload the application server

A queue separates accepting work from doing it. The application records a job and returns quickly; workers process jobs at a sustainable pace and retry when something fails.

How a Celery task queue works

A Celery set-up has three main parts:

The application, which defines tasks as Python functions and sends them to the queue
The broker, such as Redis or RabbitMQ, which stores messages until a worker takes them
Workers, which are separate processes that receive messages and run the tasks

An optional result backend stores task results if the application needs them. Many integration jobs do not need results stored, because their effect is recorded in the target system or your own database.

Understand acknowledgements

Acknowledgement controls when the broker considers a message handled. The Celery documentation explains that, by default, a task message is acknowledged just before the task is executed, so a task that crashes part-way will not be redelivered. Setting acks_late makes the worker acknowledge after the task returns instead (Celery tasks guide).

Late acknowledgement means work interrupted by a crash can be run again. The documentation advises using it only if the task is idempotent, because the task may then run more than once.

Make tasks idempotent

An idempotent task produces the same result whether it runs once or several times. The Celery guide notes that tasks should ideally be idempotent. Practical techniques include:

Using upserts keyed on a stable external ID instead of blind inserts
Recording processed job IDs and skipping duplicates
Passing idempotency keys to APIs that support them
Checking current state before making a change

Idempotency is what makes retries and late acknowledgement safe.

Configure retries with backoff

External APIs fail temporarily for many reasons. Celery supports automatic retries for chosen exceptions through autoretry_for, with options such as max_retries, retry_backoff, retry_backoff_max and retry_jitter, as described in the tasks guide. Exponential backoff spaces retries further apart, and jitter randomises them so many tasks do not retry at the same moment.

Retry only errors that are likely to succeed later, such as timeouts and rate limits. Validation errors and bad data should fail clearly and be reviewed.

Worked example: a Celery task queue job

The Python sketch below shows a task that pushes an order to an external CRM, with late acknowledgement, automatic retries and an idempotent upsert. It is simplified and untested; names such as crm_upsert_order and load_order stand in for your own code.

# Simplified, untested sketch
import requests
from celery import Celery

app = Celery("integrations", broker="redis://localhost:6379/0")
app.conf.broker_transport_options = {"visibility_timeout": 3600}

@app.task(
    bind=True,
    acks_late=True,
    autoretry_for=(requests.Timeout, requests.ConnectionError),
    retry_backoff=True,
    retry_backoff_max=600,
    retry_jitter=True,
    retry_kwargs={"max_retries": 5},
)
def push_order(self, order_id):
    order = load_order(order_id)  # read current state from your database
    if order is None:
        return "missing"
    # Upsert keyed on the order ID, so a repeated run updates rather than duplicates
    crm_upsert_order(external_id=order_id, payload=order)
    return "ok"

# In the web application, after saving the order:
# push_order.delay(order_id)

The task receives only the order ID and reads current data when it runs, so a retried task never sends stale values. The upsert keeps repeated runs safe.

Understand the Redis visibility timeout

When Redis is the broker, Celery uses a visibility timeout: the number of seconds to wait for a worker to acknowledge a task before the message is redelivered to another worker. The Celery Redis documentation states the default is one hour and that it can be changed through broker_transport_options (Celery Redis broker guide).

If a task can run longer than the visibility timeout, it may be delivered again while still running. Keep tasks short, or set the timeout above your longest expected task, and make tasks idempotent regardless.

Keep tasks small and focused

Small tasks are easier to retry, monitor and scale:

Pass identifiers rather than large objects
Split long jobs into smaller tasks that each handle a batch
Avoid tasks that wait on other tasks synchronously
Set time limits so stuck tasks do not block workers forever

Our guide to a FastAPI Redis queue covers a related approach for event-driven middleware where incoming events are queued for processing.

Choose a broker deliberately

Redis is simple to run and is often already present for caching, which makes it a popular starting point. RabbitMQ is a dedicated message broker with richer routing and delivery features. Managed queue services are another option for teams that prefer not to operate brokers themselves. Whichever you pick, check how it handles persistence, redelivery and monitoring, and test what happens to queued tasks when the broker restarts.

Separate queues by priority

Not all jobs are equally urgent. Route tasks to different queues, such as one for user-facing work like password reset emails and another for bulk syncs, and run workers for each. This prevents a large batch job from delaying urgent tasks.

Queue incoming webhooks

Webhook receivers should respond quickly and process later. A common pattern is to validate the webhook, store or enqueue it and return a success response, then let a worker do the heavy processing. Our guide to the webhook queue explains this pattern and how to avoid losing events.

Monitor workers and queues

In production, track:

Queue length and how long tasks wait before starting
Task success, failure and retry counts
Task duration compared with normal
Worker availability and memory use

Celery's monitoring guide describes command-line inspection tools and Flower, a real-time web monitor (Celery monitoring guide). Alert on growing queues and rising failure rates, not just worker crashes.

Handle tasks that keep failing

After retries are exhausted, a task should not simply disappear. Log the failure with its arguments and error, record it somewhere visible such as a failed jobs table, and provide a way to replay it after the cause is fixed.

Frequently Asked Questions

What is a Celery task queue?

It is a system where a Python application sends tasks through a message broker, such as Redis or RabbitMQ, to separate worker processes that run them in the background.

When should I use acks_late in Celery?

When you want tasks interrupted by a crash to be redelivered, and the task is idempotent so running it more than once is safe.

What is the Redis visibility timeout in Celery?

It is how long the broker waits for a task to be acknowledged before redelivering it. Celery documents a default of one hour, configurable through broker transport options.

How should Celery tasks handle API failures?

Retry temporary errors such as timeouts with exponential backoff and jitter, fail clearly on bad data, and record exhausted tasks for review and replay.

Run background work reliably

We design Celery task queues with idempotent tasks, safe retries, priority queues and monitoring for integrations and web applications. See our custom app development services, or talk to us about background job processing.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.