Kafka and Event-Driven Architecture

• Explain how topics, partitions and replication affect ordering and throughput
• Preserve ordering through key choice
• Predict consumer group, offset commit and rebalancing behaviour
• Distinguish the real meaning of at-most-once, at-least-once and exactly-once semantics
• Write a consumer that tolerates duplicates (idempotent)
• Keep a database and Kafka consistent with a transactional outbox
• Change an event schema in a backward-compatible way with a Schema Registry
• Set up monitoring and alerting on consumer lag and other metrics

No education program can guarantee employment or a promotion.

1. 01

   The Kafka model

   Understanding what a log-based model gives and what it does not.

   • Topics, partitions, segments and the log model
   • Replication, leaders and the ISR
   • Retention and log compaction
   • Producer batching, acks and latency
   • How Kafka differs from a queue

   Hands-on lab: Create a topic on a local cluster and observe how partition and retention settings change behaviour.

2. 02

   Ordering and partitioning

   Ordering holds only within a partition; we turn that fact into a design decision.

   • Message keys and the partitioner
   • Hot partitions and skewed distribution
   • Choosing a partition count, and the consequences of increasing it later
   • How ordering breaks (retries, multiple producers)
   • Choosing the key from the domain

   Hands-on lab: Choose a key that preserves the order of events for one customer and reproduce a case where order is broken.

3. 03

   Consumer groups and rebalancing

   How consumers share partitions, and what that can cause.

   • Consumer groups and partition assignment
   • Offset commit strategies
   • Rebalance types and cooperative rebalancing
   • max.poll.interval and the slow-consumer problem
   • Consumer lag and the limit of parallelism

   Hands-on lab: Slow a consumer down to trigger a rebalance storm, find the cause in logs and metrics and fix it.

4. 04

   Delivery semantics and idempotent consumption

   Duplicates are unavoidable; we learn to write consumers that tolerate them.

   • At-most-once, at-least-once, exactly-once
   • Idempotent producers and Kafka transactions
   • Consumer-side idempotency: unique keys and deduplication
   • Retry topics and dead letter topics
   • Handling a poison message

   Hands-on lab: Write a consumer that receives the same message twice and prove with a test that the effect is not applied twice.

5. 05

   The transactional outbox

   Removing the inconsistency between writing to the database and publishing an event.

   • The dual-write problem
   • The outbox table and a polling publisher
   • A CDC-based approach (the Debezium concept)
   • Outbox cleanup and ordering
   • The inbox pattern

   Hands-on lab: Build an outbox in an order service and show that an event is not lost when the service crashes and that duplicates are handled.

6. 06

   Event modelling and schema evolution

   Events are long-lived contracts; we learn to change them without breaking consumers.

   • Event versus command versus state transfer
   • Naming and shaping events
   • Avro/Protobuf and a Schema Registry
   • Backward and forward compatibility rules
   • Versioning and living with old events

   Hands-on lab: Add and remove a field in an existing event schema and observe which change the compatibility check rejects.

7. 07

   Choreography and orchestration

   Comparing two ways to coordinate a multi-step business flow with events.

   • Choreography: benefits and hidden coupling
   • Orchestration: central control and its cost
   • Visibility of the flow and debugging
   • Compensation and timeout handling

   Hands-on lab: Build the same business flow with both approaches and compare how easy each is to trace and to change.

8. 08

   Operations and monitoring

   Deciding what must be watched to run a Kafka-based system reliably.

   • Consumer lag and alerting
   • Broker and topic metrics
   • Capacity planning and the cost of retention
   • Replay: reprocessing events
   • Security basics: authentication and ACLs

   Hands-on lab: Set up a dashboard and alert rule for consumer lag and verify them with a scenario that creates lag.

Articles related to this program

• Distributed systemsFailure handling in distributed systems
• Event-driven architectureEvent-driven architecture trade-offs
• Distributed systemsIdempotency patterns for safe retries
• Event-driven architectureDelivery semantics, idempotent consumers and the outbox pattern
• KafkaKafka fundamentals for backend engineers