Message Broker Choice: RabbitMQ, Kafka or SQS?

Once systems start sending events to each other, a message broker usually appears in the design. It sits between producers, which send messages, and consumers, which process them, so neither has to wait for the other or even know the other exists. The hard part is choosing one. RabbitMQ, Apache Kafka and Amazon SQS are all popular, all reliable in the right setting, and all built around different ideas. Picking the wrong one leads to awkward workarounds for years. This guide explains the difference between queues and event logs, what delivery and ordering guarantees mean in practice, and how to choose a message broker for business integrations.

What a message broker does

A message broker accepts messages from producers, stores them, and delivers them to consumers. That simple job brings several benefits:

• Decoupling: producers and consumers can be changed, scaled or restarted independently
• Buffering: bursts of messages wait in the broker instead of overwhelming consumers
• Reliability: messages survive if a consumer is temporarily down
• Fan-out: one message can reach several consumers where the broker supports it

In integration work, a broker often sits behind webhook endpoints, between a CRM and an ERP, or at the heart of an event-driven architecture. Our guide to event-driven ERP shows where brokers fit in finance and operations systems.

Queues and event logs are different models

The most important difference between brokers is how they treat a message after it is processed.

Queue model: a message waits in a queue until a consumer takes it and acknowledges it, then it is removed. RabbitMQ's documentation on queues describes a queue as an ordered collection of messages, enqueued and delivered in a first in, first out manner. Amazon SQS also follows the queue model. Queues suit work items: "process this order", "send this email".

Log model: events are appended to a log and kept for a configured period, whether or not anyone has read them. The Apache Kafka introduction explains that events are not deleted after consumption; instead, each topic has a retention setting, after which old events are discarded. Several consumers can read the same events independently, and a consumer can re-read history. Logs suit event streams: "this customer changed", "this payment happened", where several systems may care and replay is useful.

The lines are not absolute. RabbitMQ, for example, also offers streams with non-destructive consumer semantics, where consumers can read the same messages as many times as they want. But each product is strongest in its original model.

Delivery guarantees in practice

Brokers describe delivery in terms such as at-least-once and exactly-once. In practice:

• At-least-once means a message will be delivered, but possibly more than once. Amazon's documentation for SQS standard queues states that they ensure at-least-once delivery, that more than one copy of a message might be delivered, and that messages may occasionally arrive out of order.
• Exactly-once is usually narrower than it sounds. SQS FIFO queues, for example, offer strict ordering and exactly-once processing within their design, but your consumer can still fail halfway through its own work.

The safe assumption is that any message may arrive twice. Design consumers to be idempotent, so processing a duplicate has no extra effect.

Ordering guarantees in practice

Ordering matters when later messages depend on earlier ones, such as "customer created" before "customer updated".

• Kafka guarantees order within a partition: any consumer of a given topic-partition reads its events in exactly the order they were written. Messages with the same key, such as a customer ID, can be sent to the same partition to keep their order.
• RabbitMQ queues deliver in order, but multiple consumers, requeues and retries can change the order in which messages are processed.
• SQS standard queues make a best-effort attempt at ordering; FIFO queues preserve it strictly.

Often, global ordering is unnecessary. Ordering per entity, such as per customer or per order, is enough and scales far better.

Retries, failures and dead letters

Every broker needs a plan for messages that cannot be processed:

• RabbitMQ uses consumer acknowledgements; unacknowledged messages can be requeued, and dead letter exchanges route rejected or expired messages elsewhere.
• SQS uses a visibility timeout: a received message is hidden from other consumers, and if it is not deleted in time, it becomes visible again. The visibility timeout defaults to 30 seconds and can be adjusted.
• Kafka consumers track their position in the log; failed events are often sent to a separate retry or dead letter topic by the application.

Worked example: choosing a message broker

The table below compares the three options for a hypothetical company integrating a CRM, ERP and e-commerce platform. Ratings reflect the general design of each product and should be checked against your own requirements.

NeedRabbitMQKafkaAmazon SQSWork queue for tasksStrongPossibleStrongSeveral systems reading the same eventsVia exchanges or streamsStrongNeeds extra fan-out serviceReplay past eventsStreams onlyStrong, within retentionNot designed for itPer-entity orderingPossible with careStrong, per partition keyFIFO queuesOperations effortSelf-hosted or managed serviceHigher, or managed serviceFully managed by AWSFits small integration workloadsYesOften more than neededYes

In this example, the company chooses SQS for task queues because it already runs on AWS and has a small team, and decides to revisit Kafka only if several systems need to replay a shared event history.

Questions to ask before choosing a message broker

• Are messages tasks to be done once, or events that several systems need?
• Do consumers ever need to replay past messages?
• Is per-entity ordering required, and what is the entity?
• What volume and burst sizes are expected?
• Who will run the broker, and is a managed service available on your cloud?
• What happens to messages that fail repeatedly?

Common mistakes

• Choosing Kafka for a few thousand task messages a day, then running a cluster nobody understands
• Assuming exactly-once delivery and skipping idempotent consumers
• Demanding global ordering when per-entity ordering would do
• No dead letter handling, so poison messages block queues or disappear
• No monitoring of queue depth, consumer lag or message age

Our guide to microservices communication patterns explains when to use asynchronous messaging at all, versus direct synchronous calls.

Frequently Asked Questions

What is a message broker?

It is software that receives messages from producers, stores them and delivers them to consumers, so systems can communicate without waiting for each other.

What is the difference between RabbitMQ and Kafka?

RabbitMQ is built around queues where messages are removed once processed. Kafka is built around a log where events are kept for a set period and can be read by many consumers and replayed.

Does any broker guarantee exactly-once delivery?

Some offer exactly-once features within limits, but consumers can still fail mid-task. Design consumers to handle duplicates safely whatever the broker promises.

Is Amazon SQS enough for business integrations?

For many integration workloads on AWS, yes. It is fully managed and suits task queues well, though it is not designed for replaying event history.

Pick the right backbone for your integrations

We help choose and set up message brokers, design idempotent consumers and add monitoring and dead letter handling. See our CRM and API integration services, or talk to us about message brokers.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.