The problem: a queue that forgets is not enough

An order service needs to tell billing, notifications and analytics that an order was paid. Calling three services synchronously couples their availability to yours. A classic message queue removes the coupling, but once a message is consumed it is gone, so a new consumer that appears next month cannot see history, and a consumer that had a bug cannot replay yesterday's messages.

Kafka answers this with a different model: it is a distributed, replicated, append-only log. Producers append records; consumers read them at their own pace and remember their own position. Reading does not delete anything. Records disappear only when retention says so.

The core vocabulary

  • Topic: a named log, for example orders.paid. It is a logical category, not a single file.
  • Partition: a topic is split into partitions. Each partition is an ordered, immutable sequence of records. Partitions are the unit of parallelism and of ordering.
  • Offset: the position of a record inside one partition. Offsets are meaningful only per partition.
  • Record: key, value, headers and a timestamp. The key matters more than it looks.
  • Broker: a server in the cluster. Each partition lives on several brokers.
  • Consumer group: a set of consumer instances that share the work of reading a topic.
topic orders.paid (3 partitions)

 P0: [0][1][2][3][4] ->  appended at the end
 P1: [0][1][2]
 P2: [0][1][2][3]

 consumer group "billing"        consumer group "analytics"
   instance A <- P0, P1             instance X <- P0, P1, P2
   instance B <- P2

Each partition is assigned to at most one consumer instance inside a group at a time. Different groups are independent: billing and analytics both receive every record, each with its own committed offsets.

Ordering is per partition, not per topic

This is the single most misunderstood fact. Kafka orders records inside one partition. Across partitions there is no global order. The producer decides the partition: with a key, the default partitioner hashes the key, so all records with the same key land in the same partition and are read in the order they were written.

So if you need "all events of order 42 in order", use orderId as the key. If you need a global order across all orders, you need a single partition, and with it a single consumer per group, which gives up scalability.

Retention, compaction and replication

Retention is time- or size-based: a topic may keep records for a configured period regardless of whether anyone has read them. A compacted topic instead keeps the latest record per key, which suits "current state" topics such as a customer profile.

Replication is configured per topic through the replication factor. For every partition one replica is the leader; producers and consumers talk to it, and followers copy from it. The followers that are caught up form the in-sync replica set (ISR). If the leader dies, a member of the ISR becomes leader. The setting min.insync.replicas says how many replicas must have a write before the producer is told it succeeded, when acks=all is used.

Producer acknowledgements

The acks setting decides when a send counts as successful:

acksMeaningRisk
0Do not wait for any brokerRecords can be lost silently
1Leader wrote itLost if the leader fails before followers copy
allAll in-sync replicas wrote itStrongest durability; higher latency

For business events use acks=all together with min.insync.replicas=2 on a topic with replication factor 3. Enable enable.idempotence=true so that producer retries do not create duplicates inside a partition.

Example: Spring Kafka producer and listener

spring:  kafka:    bootstrap-servers: localhost:9092    producer:      acks: all      properties:        enable.idempotence: true    consumer:      group-id: billing      enable-auto-commit: false      auto-offset-reset: earliest    listener:      ack-mode: record
@Serviceclass PaymentEvents {    private final KafkaTemplate<String, String> kafka;    PaymentEvents(KafkaTemplate<String, String> kafka) { this.kafka = kafka; }    void publishPaid(String orderId, String json) {        kafka.send("orders.paid", orderId, json)   // key = orderId             .whenComplete((result, ex) -> {                 if (ex != null) { /* log, alert, retry policy */ }             });    }}
@Componentclass BillingListener {    @KafkaListener(topics = "orders.paid", groupId = "billing")    void on(ConsumerRecord<String, String> record) {        // process; the offset is committed after this method returns        invoiceService.createFor(record.key(), record.value());    }}

The listener commits the offset only after the method returns without an exception. If the process crashes in the middle, the record is delivered again. That behaviour is called at-least-once, and it means your handler must tolerate duplicates; the next article in this series covers that in detail.

Consumer groups and rebalancing

When an instance joins or leaves a group, partitions are reassigned in a rebalance. During a rebalance, consumption pauses for the affected partitions. Slow processing is a frequent cause: if a handler takes longer than max.poll.interval.ms, the broker assumes the consumer is dead and reassigns its partitions, and the record is processed again by someone else.

The number of partitions caps the useful parallelism of a group. Ten instances reading a topic with six partitions leaves four idle. Plan partition counts from the highest consumer parallelism you expect, with some headroom, because adding partitions later affects key placement.

Common mistakes

  • Assuming a global order across partitions, then being surprised by interleaved events.
  • Using a random or null key for events that belong to one entity.
  • Doing slow I/O inside the listener without tuning max.poll.records and max.poll.interval.ms, causing rebalance storms.
  • Leaving enable-auto-commit on, so offsets may be committed before the work finishes.
  • Treating a topic like a database: querying by arbitrary field is not what a log is for.
  • Putting huge payloads in records instead of a reference to blob storage.
  • Setting retention shorter than the longest time a consumer may be down or need to replay.

Trade-offs and when not to use Kafka

Kafka gives durability, replay, high throughput and decoupled consumers. You pay with operational weight (brokers, upgrades, capacity planning), a harder mental model (partitions, offsets, rebalances) and eventual consistency between services. For a modest system with one consumer and low volume, a database table or a simple managed queue may be easier to run. If you need request/response semantics, use HTTP or gRPC; a log is a poor fit.

Consumer lag: the number to watch

Lag is the distance between the latest offset in a partition and the offset a group has committed. A growing lag means consumers are slower than producers. It is the most useful single health signal for a Kafka-based flow, because it turns an invisible backlog into something you can alert on. Look at lag per partition, not only the total: one stuck partition, often caused by a poison record or a hot key, hides inside a healthy-looking average.

Checklist

  1. The key is chosen from the entity whose order matters.
  2. Partition count is planned for consumer parallelism and future growth.
  3. Producer uses acks=all and idempotence for important events.
  4. Replication factor and min.insync.replicas are set deliberately.
  5. Offsets are committed after processing, not before.
  6. Handlers are safe to run twice for the same record.
  7. Retention covers the longest replay or outage you need to survive.

Where to go next