Webhook Dead Letter Queue: Triage and Replay

Every reliable webhook integration eventually needs somewhere to put events it cannot process. Retrying forever blocks the queue and wastes resources; discarding failures loses data. A webhook dead letter queue is the middle path: after a set number of failed attempts, the event moves to a separate queue where it waits, intact, until someone works out what went wrong and replays it. Setting one up takes minutes on most platforms. Running it well, with triage, alerting, safe replay and sensible retention, is where most teams fall short. This guide focuses on that operational side: how to configure a webhook dead letter queue, how to triage what lands there, how to replay events safely and how to keep it from becoming a graveyard, with a worked example.

What a webhook dead letter queue is for

A webhook dead letter queue holds events that failed processing repeatedly. It serves three purposes:

• It protects the main queue, so one bad event does not block or slow everything behind it
• It preserves the failed event exactly as received, so nothing is lost
• It creates a clear work list of failures that need attention

It is not a substitute for good error handling. Temporary failures, such as timeouts or rate limits, should be retried with backoff before an event is dead-lettered. The dead letter queue is for events that keep failing.

Configuring it

On managed queues, a dead letter queue is usually a setting on the main queue. In Amazon SQS, for example, a redrive policy moves a message to the dead-letter queue after it has been received a set number of times without being deleted. RabbitMQ uses dead letter exchanges to route rejected or expired messages to another queue. Whatever the platform, decide:

• Maximum attempts before dead-lettering, high enough to ride out brief outages
• Retention on the dead letter queue, longer than the time you need to investigate; AWS advises making it longer than the source queue's retention
• What metadata to keep with each event: original receipt time, attempt count and the last error

Capturing the last error matters. A dead letter queue full of events with no record of why they failed turns every investigation into guesswork. Many teams have workers attach an error category and message before giving up, or write a separate failure record keyed by event ID.

Triage: sorting what lands there

Dead-lettered events usually fall into a few categories, each with a different fix:

CategoryExampleTypical fixReplay?Bad data from senderMissing required field, invalid emailCorrect at source or transform, then replayAfter correctionDownstream rejectionTarget system rejects a duplicate or invalid valueFix mapping or target dataYes, after fixBug in processing codeUnhandled format, null pointerDeploy fixYes, after deployProlonged outageTarget system down longer than retries lastedWait for recoveryYesObsolete eventUpdate for a record since deletedNoneNo, archiveMalicious or invalid requestFailed signature that slipped throughInvestigateNo

Grouping by error category lets you fix one cause and replay hundreds of events at once, instead of handling each by hand.

Replaying safely

Replay means moving events back to the main queue for another attempt. It is safe only if processing is idempotent, so an event that partly succeeded before failing does not create duplicates when it runs again. Before replaying:

1. Confirm the cause is fixed, by testing one event first
2. Check that events are still relevant; an old status update may now be wrong
3. Replay in batches at a controlled rate, so you do not flood downstream systems
4. Watch the dead letter queue: events that fail again point to a second cause

SQS supports redriving messages from a dead-letter queue back to the source queue. A custom replay tool adds filtering by error category and a dry-run mode.

Ordering and stale events

Replaying old events can overwrite newer data. If a contact was updated twice, and the first update failed and is replayed after the second succeeded, the older value wins unless processing checks timestamps or version numbers. Compare the event's timestamp with the record's last update and skip events that are older. This check belongs in normal processing too, not just replay.

Worked example: a CRM-to-accounting webhook integration

Suppose a company sends invoice events from its CRM to an accounting system through a queue, with five attempts before dead-lettering. One Monday, 340 events land in the dead letter queue over an hour. The team's failure records show three categories:

CategoryCountCauseActionDownstream rejection: unknown tax code318A new tax code added in the CRM was never mappedAdd mapping, test one, replay allBad data: missing customer email19Records created by an import without emailsFix records in CRM, replayObsolete: invoice deleted3Invoices deleted after the eventArchive, no replay

They replay with a simple tool that filters by category. This is a simplified, untested illustration; failure_category stands for however the team records the last error for each message:

def replay(sqs, dlq_url, main_url, category, limit=100, dry_run=True):
    moved = 0
    while moved < limit:
        resp = sqs.receive_message(QueueUrl=dlq_url, MaxNumberOfMessages=10,
                                   WaitTimeSeconds=1)
        msgs = resp.get("Messages", [])
        if not msgs:
            break
        for m in msgs:
            if failure_category(m) != category:
                continue  # becomes visible again after the visibility timeout
            if not dry_run:
                sqs.send_message(QueueUrl=main_url, MessageBody=m["Body"])
                sqs.delete_message(QueueUrl=dlq_url, ReceiptHandle=m["ReceiptHandle"])
            moved += 1
    return moved

A dry run reports 318 matching events. After deploying the mapping and testing one event, they replay the rest in batches of 100. Processing upserts invoices by CRM invoice ID, so none are duplicated. The 19 events with missing emails are replayed after the CRM records are fixed, and the 3 obsolete events are archived with a note. The team adds an alert so a new unmapped tax code will be caught the first time it appears.

Alerting and ownership

• Alert when anything arrives in the dead letter queue, or when it grows beyond a small threshold
• Alert on the age of the oldest message, so nothing sits until retention deletes it
• Assign a named owner for each integration's dead letter queue
• Review recurring categories monthly and fix root causes

Retention is a deadline. An event that expires from a dead letter queue is gone, so alerts must give people time to act.

Common mistakes

• No error details stored with dead-lettered events
• No alerting, so the queue fills silently until events expire
• Replaying without fixing the cause, which just refills the queue
• Replaying old events that overwrite newer data
• Non-idempotent processing that duplicates records on replay
• Treating the dead letter queue as an archive rather than a work list

For the overall design of queue-based webhook processing, see our guide to the webhook queue. For handling errors in Zoho CRM integrations specifically, our guide to Zoho CRM API error handling covers retries and failure categories.

Frequently Asked Questions

What is a webhook dead letter queue?

A separate queue where webhook events go after repeatedly failing processing, so they are preserved for investigation and replay instead of blocking the main queue or being lost.

How many retries before an event is dead-lettered?

Enough to ride out brief outages and rate limits, often a handful of attempts with increasing delays. The right number depends on how long downstream problems typically last.

How do you replay events from a dead letter queue?

Fix the cause, test one event, then move matching events back to the main queue in controlled batches, with idempotent processing so retries cause no duplicates.

How long should a dead letter queue keep messages?

Longer than the time you need to notice and investigate failures, and longer than the source queue's retention, with alerts well before messages expire.

Make your integrations recoverable

We build webhook integrations with dead letter queues, triage tooling and safe replay, so failures never mean lost data. See our CRM and API integration services, or talk to us about your integration.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.