The problem: the order is saved, the event is not

An order service saves an order in PostgreSQL and then publishes OrderCreated to Kafka. Two things can go wrong. The database commit succeeds and the process dies before the send: the order exists, nobody hears about it. Or the send succeeds and the commit then fails: downstream services react to an order that does not exist. This is the dual-write problem: two systems, no shared transaction, so no atomic outcome.

Before fixing it we need precise vocabulary for what a message system can promise.

The three delivery semantics

  • At-most-once: a message is delivered zero or one time. You commit the offset first, then process. A crash between the two loses the message. Acceptable for lossy metrics, rarely for business events.
  • At-least-once: delivered one or more times. You process first, then commit the offset. A crash between the two causes redelivery, so duplicates happen. This is the practical default.
  • Exactly-once: the effect happens once. In a distributed system with failures this cannot be obtained from the network alone; it is built from at-least-once delivery plus deduplication or atomic commits.

What Kafka's exactly-once covers, and what it does not

Kafka offers three mechanisms that combine:

  • the idempotent producer (enable.idempotence=true): the broker discards duplicate sends caused by producer retries within a partition;
  • transactions (transactional.id): a producer writes to several partitions and commits consumed offsets atomically with its output records;
  • read_committed consumers: they skip records from aborted transactions.

Together they give exactly-once for a read-process-write pipeline that stays inside Kafka: consume from topic A, transform, produce to topic B, and the offsets and outputs commit as one unit. Kafka Streams uses this.

It does not extend to external side effects. If your consumer sends an email, calls a payment API, or writes to a SQL database, Kafka cannot roll that back or deduplicate it. For those you need idempotent consumers or, on the publishing side, the outbox pattern.

Idempotent consumers

Because delivery is at-least-once, the handler must produce the same result when it sees the same event twice. The simplest robust technique is a processed-events table, written in the same database transaction as the business change:

CREATE TABLE processed_event (    consumer   text        NOT NULL,    event_id   uuid        NOT NULL,    handled_at timestamptz NOT NULL DEFAULT now(),    PRIMARY KEY (consumer, event_id));
@KafkaListener(topics = "orders.paid", groupId = "billing")@Transactionalvoid on(ConsumerRecord<String, OrderPaid> record) {    UUID eventId = UUID.fromString(        new String(record.headers().lastHeader("event-id").value()));    int inserted = jdbc.update(""        INSERT INTO processed_event(consumer, event_id)        VALUES ('billing', ?) ON CONFLICT DO NOTHING"", eventId);    if (inserted == 0) return;          // already handled    invoiceService.createFor(record.value());}

The insert and the invoice commit or roll back together, so a redelivery either finds the row and skips, or finds nothing and does the work. Note that the Kafka offset commit and the database commit are two separate operations: the listener container commits the offset after the method returns, and the database transaction commits when the method exits, and no single transaction spans both. A crash between them redelivers a message whose database work is already committed, which is exactly why the processed-events table is what makes redelivery safe. Sometimes natural idempotency is enough: "set status to PAID" is safe to repeat, "add 10 to balance" is not. See the article on idempotency patterns for request keys.

The transactional outbox

To fix the dual write, make the event part of the same local transaction as the state change. Write it to an outbox table; a separate relay publishes rows to Kafka.

 +-------------- one DB transaction --------------+
 | INSERT orders ...                              |
 | INSERT outbox (id, topic, key, payload)        |
 +------------------------------------------------+
                    |
          relay polls unpublished rows
                    v
          Kafka topic  --->  consumers (idempotent)
CREATE TABLE outbox (    id           uuid PRIMARY KEY,    aggregate_id text        NOT NULL,    topic        text        NOT NULL,    payload      jsonb       NOT NULL,    created_at   timestamptz NOT NULL DEFAULT now(),    published_at timestamptz);CREATE INDEX outbox_unpublished ON outbox (created_at)    WHERE published_at IS NULL;
@Scheduled(fixedDelay = 500)void relay() {    for (OutboxRow r : repo.lockNextBatch(100)) { // FOR UPDATE SKIP LOCKED        kafka.send(r.topic(), r.aggregateId(), r.payload()).join();        repo.markPublished(r.id());    }}

The relay itself is at-least-once: if it crashes after sending but before markPublished, the row is sent again. That is why the consumers must be idempotent, and why the id should travel in a header as the event id. Using FOR UPDATE SKIP LOCKED lets several relay instances share the table. Keying by aggregate_id preserves per-entity order in Kafka, but the relay must publish rows of one aggregate in creation order for that to hold.

An alternative to polling is change data capture reading the database log; it lowers latency and load but adds an infrastructure component to operate.

Dead-letter topics and poison messages

Some records will never succeed: malformed JSON, a violated business rule, a missing reference. Retrying forever blocks the partition. Spring Kafka's DefaultErrorHandler with a DeadLetterPublishingRecoverer retries a bounded number of times with backoff, then publishes the record to a topic such as orders.paid.DLT and moves on.

@BeanDefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> t) {    var recoverer = new DeadLetterPublishingRecoverer(t);    return new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3));}

Separate transient failures (a database timeout: retry) from permanent ones (a validation failure: dead-letter immediately). Monitor the dead-letter topic, because an unwatched one is just a slower way to lose data, and define who replays records and how.

Common mistakes

  • Calling kafka.send inside a @Transactional method and assuming the database transaction covers it.
  • Believing enable.idempotence makes the whole system exactly-once.
  • Committing offsets before processing, then discovering lost events.
  • An idempotency check that is not in the same transaction as the side effect.
  • Outbox rows that are never cleaned up, so the table grows without bound.
  • Dead-letter topics with no alert and no replay procedure.
  • Publishing from the relay without a stable event id, making deduplication impossible.

Trade-offs

The outbox adds a table, a relay and extra latency, and the processed-events table adds a write per event. In return you remove the lost-event window between database and broker. If losing an occasional notification is acceptable, direct publishing after commit is simpler. If the event drives money or inventory, the outbox cost is usually justified.

Ordering, retries and the relay

Retries can reorder records. If a producer has several requests in flight and one fails and is retried, a later record may land first. The idempotent producer preserves order within a partition for retries, which is one more reason to keep it enabled. In the outbox relay, publish rows of the same aggregate sequentially and in creation order, and do not run two relays on the same aggregate without row locking. When ordering is not essential for a given consumer, say so explicitly: it makes later parallelism changes safer.

A practical example: payment and invoice

The payment service publishes PaymentCaptured through the outbox. If the billing consumer receives the event twice, the primary key in processed_event stops the second attempt and no duplicate invoice is created. The notification consumer calls an external email provider; if the provider accepts an idempotency key, pass the event id as that key, and if it does not, decide in advance how harmful a duplicate email is for the business.

Checklist

  1. State per flow whether at-least-once or at-most-once is acceptable.
  2. Never publish and write to the database as two independent steps for important events.
  3. Write the event to an outbox in the same transaction as the state change.
  4. Give every event a stable id and carry it in a header or field.
  5. Make every consumer idempotent, in the same transaction as its side effect.
  6. Use bounded retries and a dead-letter topic with monitoring.
  7. Clean up published outbox rows on a schedule.

Where to go next