Idempotency Keys: Make Retries Safe

A connector sends a request to create an order. The network drops before the response comes back. Did the order get created? The connector cannot tell, so it retries, and now there may be two orders, two invoices or two charges. Retries are essential for reliable integrations, but without protection they create duplicates. Idempotency keys solve this. The client attaches a unique key to each operation, and the server uses it to recognise retries and return the original result instead of doing the work again. This guide explains why retries cause duplicates, how idempotency keys work on both sides, and how to store, check and expire them.

Why retries create duplicates

Network failures are ambiguous. When a request times out, there are three possibilities:

• The request never reached the server
• The server processed it, but the response was lost
• The server is still processing it

Retrying is correct in the first case and harmful in the second and third. Without a way to tell them apart, a client has to choose between missing data and duplicate data.

Some HTTP methods are already safe to repeat. RFC 9110, the HTTP semantics specification, states that PUT, DELETE and safe request methods are idempotent, meaning repeating the request has the same intended effect as sending it once. POST, which most APIs use to create records, is not. Idempotency keys make POST-style operations safe to retry.

How idempotency keys work

The pattern has two halves.

On the client:

• Generate a unique key for each logical operation, such as "create order for cart 12345", before the first attempt
• Send the same key with every retry of that operation
• Use a new key for a genuinely new operation

On the server:

• When a request arrives with a key, check whether that key has been seen
• If not, record the key, process the request and store the result against the key
• If the key has been seen and processing finished, return the stored result without repeating the work
• If the key has been seen but the request parameters differ, reject it as an error

Stripe's documentation on idempotent requests is a well-known example. It suggests V4 UUIDs or another random string with enough entropy, allows keys up to 255 characters, saves the status code and body of the first request for each key, and compares incoming parameters with the original to catch accidental misuse. It also notes that keys can be removed after they are at least 24 hours old.

Choosing keys

Good keys are unique per operation and stable across retries:

• Random UUIDs generated once and stored with the pending operation work well
• Keys derived from business data, such as a source system's order ID plus an operation name, make retries naturally consistent across restarts
• Avoid using sensitive data, such as email addresses, directly as keys; Stripe's guidance warns against personal identifiers

Whatever the scheme, store the key with the outgoing operation before the first attempt, so a crash and restart reuses the same key rather than generating a new one.

Worked example: an idempotency keys table

The simplified, untested SQL below sketches a table a server might use to track idempotency keys. Column names and sizes are placeholders.

CREATE TABLE idempotency_keys (
    idem_key       VARCHAR(255) PRIMARY KEY,
    request_hash   CHAR(64)     NOT NULL,  -- hash of the request parameters
    status         VARCHAR(20)  NOT NULL,  -- 'in_progress' or 'completed'
    response_code  INTEGER,
    response_body  TEXT,
    created_at     TIMESTAMP    NOT NULL DEFAULT CURRENT_TIMESTAMP
);

The flow for each request:

1. Try to insert a row with the key, a hash of the parameters and status "in_progress". The primary key makes this fail if the key already exists.
2. If the insert succeeds, process the request, then update the row with the response and status "completed", ideally in the same transaction as the business change.
3. If the insert fails, read the existing row. If the hash differs, return an error. If the status is "completed", return the stored response. If it is still "in_progress", return a conflict response asking the client to retry later.
4. A scheduled job deletes rows older than your retention period.

Handle concurrent retries

Aggressive clients sometimes send a retry while the first attempt is still running. The "in_progress" state handles this: the second request sees the key, finds the work unfinished and receives a response telling it to wait. Without that state, two requests could both pass the check and both do the work.

Also decide what happens if processing crashes midway. A row stuck in "in_progress" should either expire after a timeout, so a later retry can proceed, or be resolved by checking whether the business change actually happened.

Idempotency keys when you are the client

Many APIs you call will not support idempotency keys. In that case, protect yourself:

• Before creating a record, search for an existing one with the same external reference
• Use upsert operations keyed on a unique external ID where the API offers them; our guide to the Zoho CRM upsert explains how that works in Zoho
• Store a unique external ID on created records so duplicates can be detected and merged later

When an API does support idempotency keys, use them for every create or payment request, not just the ones you expect to retry.

Idempotent consumers

The same idea applies to events. Webhooks and queues usually deliver at least once, so consumers should record processed event IDs and skip repeats. Our guide to the webhook dead letter queue covers what to do with events that keep failing, and replaying them safely depends on this kind of idempotent processing.

Test the duplicate path

Duplicate handling is easy to get wrong and rarely exercised in normal operation. Add tests that send the same request twice with the same key, send two requests with the same key at the same moment, reuse a key with different parameters, and simulate a crash between the business change and the key update. Each should produce the expected response and exactly one business record.

Expire keys sensibly

Keys cannot be kept forever, but expiring them too soon reopens the duplicate risk. Set the retention period longer than the longest realistic retry window, including queued retries after an outage. Document the period so clients know how long a key remains valid.

Common mistakes

• Generating a new key on each retry, which defeats the purpose
• Storing the key only in memory, so a restart loses it
• Not comparing request parameters, so a reused key silently returns the wrong result
• Recording the key and the business change in separate transactions, so a crash leaves them inconsistent
• No handling for concurrent requests with the same key

Frequently Asked Questions

What are idempotency keys?

They are unique values a client sends with a request so the server can recognise retries of the same operation and return the original result instead of repeating the work.

Which requests need idempotency keys?

Mainly requests that create records or move money, usually POST requests. PUT, DELETE and safe methods such as GET are idempotent by definition in HTTP.

How long should idempotency keys be kept?

Longer than the longest realistic retry window for your clients. Stripe, for example, notes that keys can be removed once they are at least 24 hours old.

What if an API does not support idempotency keys?

Search for existing records by external reference before creating, use upserts keyed on unique external IDs, and store those IDs so duplicates can be found.

Make retries safe across your integrations

We design idempotent APIs and connectors, add key handling and duplicate protection and make retries safe end to end. See our CRM and API integration services, or talk to us about idempotency.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.