Exponential Backoff: Retry Failed Syncs Safely

Every integration that calls another system will eventually see a failure: a timeout, a rate limit response or a brief outage on the other side. Retrying is the obvious response, but retrying immediately and repeatedly can make things worse by adding load to a service that is already struggling. Exponential backoff is the standard answer: wait a little after the first failure, longer after the second and longer again after the third, with some randomness added so that many clients do not retry at the same moment. This guide explains how to apply exponential backoff to sync jobs and API integrations, which errors deserve a retry and how to know when to stop.

Why immediate retries cause trouble

Imagine a sync job that pushes changes from your CRM to an accounting system. The accounting API slows down for a minute. If every worker retries instantly, the API receives the original load plus a wave of retries, which slows it further and causes more failures. The AWS Builders' Library describes retries as selfish: when a client retries, it spends more of the server's time to get a higher chance of success, which is fine when failures are rare but adds load when a service is struggling (AWS Builders' Library: Timeouts, retries and backoff with jitter).

Backoff spaces those retries out so that the other system has time to recover.

How exponential backoff works

The idea is simple. After the first failure, wait a base delay. After each further failure, multiply the delay, commonly doubling it, up to a maximum cap. Stop after a fixed number of attempts or a total time budget and treat the item as failed.

Two refinements make it work in practice:

• A cap on the delay, so that a long outage does not lead to waits of hours between attempts.
• Jitter, a random element in each delay, so that clients that failed together do not all retry together.

The AWS Architecture Blog post on exponential backoff and jitter compares several approaches, including one it calls Full Jitter, where the wait is a random value between zero and the current backoff ceiling, and explains how adding randomness reduces the clustering of retries (AWS Architecture Blog: Exponential Backoff And Jitter).

Decide which failures to retry

Not every error should be retried. Retrying a request that can never succeed wastes time and can hide real problems. Google Cloud's retry guidance for Cloud Storage lists HTTP 408, 429 and 5xx responses, along with socket timeouts and TCP disconnects, as transient problems that are useful to retry, and says that whether a request is safe to retry also depends on whether it is idempotent (Google Cloud: Retry strategy).

A practical rule set for sync jobs:

• Retry: timeouts, connection resets, 429 Too Many Requests and most 5xx server errors.
• Do not retry: 400 Bad Request, 401 or 403 authorisation failures, 404 Not Found and validation errors. These need a fix, not another attempt.
• Respect explicit instructions: if the response includes a Retry-After header, wait at least that long.

Our guide to API rate limit handling covers how to read rate limit headers and pace requests before you hit the limit.

Make retries safe with idempotency

Backoff controls when you retry. Idempotency controls what happens if a retried request had in fact succeeded the first time. If a create request timed out after the other system processed it, a blind retry can create a duplicate invoice or contact. The AWS article recommends designing APIs to be idempotent so that they can be safely retried.

For sync work, that usually means sending an idempotency key with create requests, using upserts keyed on an external ID, or checking for an existing record before creating one. Our guide to idempotency keys explains the patterns in detail.

Retry at one layer, not every layer

Retries multiply when several layers each retry independently. If a job retries three times, the HTTP client inside it also retries three times and a queue redelivers failed messages three times, a single failing call can turn into many attempts. The AWS Builders' Library recommends retrying at a single point in the stack for this reason.

Decide where retries live. For many integrations, the cleanest choice is to keep the HTTP client simple, retry inside the job with backoff and let the queue handle only the final failure by moving the message to a dead letter queue.

Worked example: a retry helper with backoff and jitter

The simplified, untested sketch below shows a small Python helper that retries a call with capped exponential backoff and full jitter. It retries only errors marked as transient and respects a server-supplied delay when one is given. Names are hypothetical.

# Simplified, untested sketch
import random
import time

class TransientError(Exception):
    def __init__(self, message, retry_after=None):
        super().__init__(message)
        self.retry_after = retry_after

def call_with_backoff(func, max_attempts=5, base=1.0, cap=30.0):
    for attempt in range(1, max_attempts + 1):
        try:
            return func()
        except TransientError as err:
            if attempt == max_attempts:
                raise
            ceiling = min(cap, base * pow(2, attempt - 1))
            delay = random.uniform(0, ceiling)
            if err.retry_after is not None:
                delay = max(delay, err.retry_after)
            time.sleep(delay)

In a real job, the function that calls the API would raise TransientError for timeouts, 429 and 5xx responses, and a different exception for errors that should not be retried. The values for attempts, base delay and cap are placeholders to tune for each API.

Set a budget and a final destination

Backoff needs an end point. Decide, per job, how many attempts or how much total time a single item may consume before it is treated as failed. Then decide where failed items go: a dead letter queue, a failed-records table or an alert. The worst outcome is a job that retries silently forever or drops the item without trace.

When items do fail permanently, record the last error, the number of attempts and the payload, so that someone can fix the cause and replay the item.

Watch the retry rate

Retries are a useful signal. A rising retry rate often appears before outright failures, and it can reveal an API that is approaching its limits or a credential that is about to expire. Track attempts per item, the share of calls that needed a retry and the number of items that exhausted their budget. Alert on sudden changes rather than on single failures.

The AWS article also describes limiting retries locally with a token bucket, so that calls can retry while tokens remain and then retry at a fixed rate once tokens run out. For busy integrations, a limit like this stops a widespread outage from turning into a flood of retries.

Frequently Asked Questions

What is exponential backoff?

It is a retry strategy where the wait between attempts grows after each failure, usually by doubling up to a cap, so that a struggling service is not overwhelmed by immediate retries.

Why add jitter to backoff?

Without randomness, clients that failed at the same time retry at the same time. Jitter spreads retries out and reduces the chance of repeated bursts.

Which errors should not be retried?

Errors that will not change on their own, such as bad requests, authorisation failures, missing resources and validation errors. These need a fix before trying again.

How many times should a sync job retry?

There is no universal number. Set a limit per job based on how long the data can wait and how the API behaves, then send items that exhaust it to a dead letter queue.

Build syncs that recover on their own

We design integrations with sensible retries, idempotent writes and clear failure handling so that brief outages do not become data problems. See our real-time CRM and database synchronization service, or talk to us about retry design.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.