Message Queues vs Event Streams: Kafka, RabbitMQ, and SQS Compared

Every distributed system eventually needs a way to move data between services without tightly coupling them. The moment you split a monolith into services, or add a background worker, or need to fan out an event to five consumers, you're in messaging territory. The problem is that "messaging" spans two fundamentally different patterns — message queues and event streams — and the tools that implement them (Apache Kafka, RabbitMQ, AWS SQS) are frequently chosen for the wrong reasons. Teams pick Kafka because it's fashionable, then spend six months operating a cluster they didn't need. Or they use SQS for an event-sourcing workload and discover ordering guarantees they can't get. This article breaks down the message queue vs event stream Kafka decision the way an engineering manager actually has to make it: against ordering, durability, throughput, and operational cost.

• Background / Why This Matters
• Option A: Apache Kafka
• Option B: RabbitMQ Comparison
• Decision Framework: How to Choose
• Common Mistakes / What to Avoid
• Frequently Asked Questions
• Conclusion

Background / Why This Matters

The core distinction is deceptively simple. A message queue treats each message as a task to be consumed and destroyed. Once a consumer acknowledges a message, it's gone. Queues are about work distribution: hand a job to one of N workers, guarantee it gets processed, then move on. RabbitMQ and AWS SQS are queue-first systems.

An event stream treats messages as an immutable, append-only log. Consumers read from the log at their own pace by tracking an offset, and the same event can be read by many independent consumers days apart. Nothing is deleted on read — data is retained on a time or size policy. Apache Kafka is the canonical event streaming platform (along with AWS Kinesis and Apache Pulsar).

Why does this matter to an engineering manager? Because the pattern you choose determines what your system can and cannot do architecturally, and that constraint is expensive to reverse later.

• Replayability: Streams let you reprocess history — invaluable for rebuilding a read model, backfilling a new service, or debugging. Queues generally can't replay consumed messages.
• Fan-out: Streams let many consumer groups read the same data independently. Queues typically deliver each message to exactly one consumer within a group.
• Ordering: Streams provide strong per-partition ordering. Queues offer weaker or optional ordering, usually at a throughput cost.
• Operational burden: Queues (especially managed SQS) are dramatically cheaper to run than a self-managed Kafka cluster.

Takeaway: Decide whether your workload is "process this task once" (queue) or "record this fact for many readers over time" (stream) before you evaluate specific products. The tool follows the pattern, not the other way around.

Option A: Apache Kafka

Kafka is a distributed, partitioned, replicated commit log. Producers write events to topics, which are split into partitions. Each partition is an ordered, immutable sequence, and consumers track their position with an offset. This design is what makes Kafka's headline capabilities possible.

What Kafka does well

• High throughput. Kafka is built for sustained, high-volume ingestion — think clickstreams, IoT telemetry, and log aggregation. Benchmarks vary widely by hardware and message size, but Kafka routinely handles hundreds of thousands to millions of messages per second on modest clusters.
• Durable, replayable history. Retain events for hours, days, or forever. A new service can bootstrap itself by replaying the topic from offset zero.
• Ordering per partition. Messages sharing a partition key (e.g., customer_id) are strictly ordered. This is essential for event sourcing and CDC (change data capture).
• Ecosystem. Kafka Connect for integrations, Kafka Streams and ksqlDB for stream processing, and Schema Registry for contract enforcement.

What Kafka costs you

Kafka's power comes with real operational weight. Self-hosting means managing brokers, replication, partition rebalancing, and — historically — ZooKeeper (now replaced by KRaft in modern versions). Capacity planning around partition counts is a genuine skill; too few partitions caps parallelism, too many strains the cluster. Managed offerings like Confluent Cloud, AWS MSK, and Redpanda Cloud reduce this burden substantially but shift cost to your bill.

Kafka is the right answer when you need replayability, high-volume fan-out, or an event log as a system of record. It is the wrong answer when you just need a background job queue for a handful of workers.

Takeaway: Choose Kafka when event history and multi-consumer fan-out are first-class requirements — and budget for either a managed service or a dedicated platform engineer to run it well.

Option B: RabbitMQ Comparison

RabbitMQ is a mature, feature-rich message broker implementing AMQP. Its mental model is exchanges and queues: producers publish to an exchange, which routes messages to bound queues based on rules (direct, topic, fanout, headers). Consumers pull from queues, acknowledge messages, and the broker removes them.

Where RabbitMQ shines

• Flexible routing. RabbitMQ's exchange types let you build sophisticated routing topologies without application logic — routing by key, wildcard patterns, or broadcasting.
• Per-message workflows. Priority queues, dead-letter exchanges, message TTLs, and delayed delivery are first-class features. This is ideal for task processing, RPC-style request/reply, and retry semantics.
• Lower conceptual overhead. For "give this job to a worker and make sure it completes," RabbitMQ is simpler to reason about than Kafka's offset model.
• Fast to stand up. A single-node RabbitMQ instance is easy to deploy; clustering and quorum queues add durability when you need it.

Where AWS SQS fits

AWS SQS deserves a mention here because it competes directly with RabbitMQ for queue workloads on AWS. SQS is fully managed — no brokers to patch, near-infinite scaling, and pay-per-request pricing. It comes in two flavors: Standard (at-least-once delivery, best-effort ordering, extremely high throughput) and FIFO (exactly-once processing, strict ordering, but capped throughput per message group). SQS trades RabbitMQ's rich routing for zero operational overhead.

Side-by-side comparison

Dimension	Apache Kafka	RabbitMQ	AWS SQS
Pattern	Event stream (log)	Message queue / broker	Message queue
Message retention	Configurable, days to forever	Until consumed/acked	Up to 14 days, deleted on ack
Replay	Yes (reset offset)	No (native)	No
Ordering	Strict per partition	Per queue (single consumer)	FIFO only, per message group
Fan-out to many consumers	Excellent (consumer groups)	Via exchanges, but destructive	Requires SNS + multiple queues
Throughput	Very high	Moderate to high	Very high (Standard), capped (FIFO)
Routing flexibility	Basic (partition keys)	Excellent (exchange types)	Minimal
Operational burden	High (or managed)	Moderate	Very low (fully managed)

Takeaway: RabbitMQ wins on routing sophistication and task-oriented workflows; SQS wins on operational simplicity for queue workloads on AWS; Kafka wins on streaming, replay, and fan-out.

Decision Framework: How to Choose

Instead of comparing feature lists in the abstract, walk through these questions in order. The first "yes" that changes your answer usually settles the decision.

1. Do consumers need to replay history? If a new service must rebuild state from past events, or you need CDC and event sourcing — choose Kafka. Queues don't do this natively.
2. Do multiple independent consumers need the same data? If yes, Kafka's consumer groups are the natural fit. With RabbitMQ or SQS you'd bolt on fanout exchanges or SNS-to-SQS fan-out, which adds complexity.
3. Is your dominant need "run this job once, reliably"? Then you want a queue. On AWS with no routing needs, start with SQS. If you need priorities, delayed delivery, or complex routing, choose RabbitMQ.
4. What are your ordering requirements? Strict ordering at high throughput points to Kafka partitioning. Strict ordering at modest volume points to SQS FIFO. If ordering doesn't matter, SQS Standard or RabbitMQ scale easily.
5. What is your operational appetite? A small team without a platform group should lean toward managed services — SQS, or a managed Kafka like MSK or Confluent Cloud — rather than self-hosting Kafka.

A common and entirely valid outcome is using more than one. Many mature architectures run Kafka as the event backbone for analytics and cross-service events, while using SQS or RabbitMQ for discrete task queues. These patterns aren't mutually exclusive.

This is exactly the kind of trade-off Halkwinds' engineering teams evaluate when designing cloud infrastructure for clients — we've seen teams save significant operational cost by not