Outbox Pattern: Publish Events Without Gaps

Many integrations need to do two things when data changes: update their own database and tell other systems about it. The outbox pattern is a reliable way to do both. An invoice is marked paid, and the CRM, the reporting warehouse and a customer notification service all need to know. The obvious code updates the database, then publishes a message to a queue or calls a webhook. That works until the process crashes between the two steps, or the message broker is briefly unreachable. Then the database says the invoice is paid but nobody else heard about it, or worse, a message announces a change that was rolled back. This guide explains how it works, how to implement the relay and what consumers must handle.

The dual write problem

Writing to two separate systems, a database and a message broker, cannot be made atomic with ordinary code. The microservices.io description of the transactional outbox pattern explains that a service must atomically update the database and send messages to avoid data inconsistencies, and that a traditional distributed transaction (two-phase commit) spanning the database and broker is not viable: one or both may not support it, and even when they do, coupling them that way is often undesirable (microservices.io: Transactional outbox).

Retrying the publish step does not fully solve it either. If the process crashes after committing the database change but before publishing, there is nothing left to retry from.

How the outbox pattern works

The solution, as the same page describes it, is to store the message in the database as part of the transaction that updates the business data. A separate process then sends stored messages to the message broker. Because the business change and the outbox row are committed together, either both exist or neither does.

The flow:

1. The application begins a database transaction.
2. It updates the business tables, such as marking an invoice paid.
3. It inserts a row into an outbox table describing the event, such as invoice_paid with the invoice ID and amount.
4. It commits the transaction.
5. A relay process reads unsent outbox rows, publishes them to the broker or calls the webhook, and marks them as sent.

Worked example: an outbox table and relay query

The simplified, untested sketch below shows a PostgreSQL outbox table, a transaction that writes an event alongside a business update, and a query a relay could use to claim a batch of unsent events. Table and column names are hypothetical.

-- Simplified, untested sketch
CREATE TABLE outbox (
    id           BIGSERIAL PRIMARY KEY,
    event_type   TEXT NOT NULL,
    aggregate_id TEXT NOT NULL,
    payload      JSONB NOT NULL,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    sent_at      TIMESTAMPTZ
);

BEGIN;
UPDATE invoices SET status = 'paid', paid_at = now() WHERE id = 1042;
INSERT INTO outbox (event_type, aggregate_id, payload)
VALUES ('invoice_paid', '1042', '{"invoice_id": 1042}');
COMMIT;

SELECT id, event_type, aggregate_id, payload
FROM outbox
WHERE sent_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;

After publishing each claimed event, the relay sets sent_at in the same transaction that holds the lock, then commits.

Implement the relay

The microservices.io page names two ways to implement the relay: the polling publisher pattern, which queries the outbox table for unsent messages, and the transaction log tailing pattern, which reads changes from the database's log. Polling is simple and works with any relational database. Log tailing avoids repeated queries and can deliver events sooner, but needs change data capture tooling.

For polling with several relay workers, PostgreSQL's SKIP LOCKED option helps. The PostgreSQL documentation explains that with SKIP LOCKED, rows that cannot be locked immediately are skipped, and that while this gives an inconsistent view unsuitable for general work, it can avoid lock contention when multiple consumers access a queue-like table (PostgreSQL: SELECT). Be aware that parallel workers can publish events for the same record out of order. If order per record matters, use a single relay or route each record's events to one worker.

Consumers must handle duplicates

The outbox pattern guarantees that events are not lost, not that they are delivered exactly once. The microservices.io page notes that the relay might publish a message more than once, for example if it crashes after publishing but before recording that it did. Consumers must therefore be idempotent: processing the same event twice must have the same effect as processing it once. Include a unique event ID in every message, and have consumers record processed IDs or use naturally idempotent operations such as upserts. Our guide to idempotency keys explains these techniques in detail.

Keep event order where it matters

The same page explains that messages must be sent to the broker in the order the service produced them, for example events from a series of transactions on the same record. Ordering by the outbox row ID within a single relay preserves that. Downstream, brokers have their own ordering guarantees, which often apply per partition or per message group rather than globally. Design consumers to cope with gaps, such as ignoring an event older than the version they already have.

Clean up and monitor the outbox

Outbox tables grow. Delete or archive sent rows after a retention period that suits your debugging and audit needs. Monitor:

• The number of unsent events and the age of the oldest one
• Relay errors and retries
• Events that repeatedly fail to publish, which should move to a dead letter table for investigation

An ageing backlog is the clearest sign that downstream systems are not hearing about changes.

Know when you may not need it

The outbox adds a table, a relay process and monitoring. It is worth that cost when a lost or phantom event would cause real harm, such as an unpaid invoice reported as paid or a customer never receiving a confirmation. It may be unnecessary when downstream systems can safely catch up on a schedule, for example a nightly report that reads directly from the database, or when a periodic reconciliation already compares both sides and fixes differences. In some designs, log-based change data capture on the business tables can serve a similar purpose without a separate outbox table. Choose the simplest design that meets the consistency your process actually needs, and write down why.

Fit the outbox into your integration design

The outbox pattern pairs well with webhook delivery and queues. When the target is an external system that receives webhooks, the relay can call it with retries and backoff, and our guide to the webhook queue covers how to process incoming events reliably on the receiving side. For updates that span several systems and need to be undone on failure, the outbox is often combined with sagas.

Frequently Asked Questions

What is the outbox pattern?

It is a way to update a database and publish an event reliably: the event is written to an outbox table in the same transaction as the business change, and a separate relay publishes it afterwards.

Why not just publish after committing?

If the process fails between commit and publish, the event is lost and other systems never hear about the change. The outbox keeps the event safe in the database until it is sent.

Does the outbox pattern guarantee exactly-once delivery?

No. The relay may publish an event more than once, so consumers must be idempotent and use event IDs to detect duplicates.

Polling or log tailing for the relay?

Polling is simpler and works with any relational database. Log tailing needs change data capture tooling but avoids repeated queries.

Publish changes without gaps

We design reliable event publishing between databases, CRMs and accounting systems, including outbox tables, relays and idempotent consumers. See our real-time CRM and database synchronization service, or talk to us about the outbox pattern.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.