Kafka and Event-Driven Architecture: Modules
The program aims not just at using Kafka but at understanding its behaviour well enough to build a reliable event-driven system. Ordering, delivery semantics, idempotent consumption, the transactional outbox and schema evolution are learned through realistic failures. Operations and monitoring are treated as part of the design.
01
The Kafka model
Understanding what a log-based model gives and what it does not.
- Topics, partitions, segments and the log model
- Replication, leaders and the ISR
- Retention and log compaction
- Producer batching, acks and latency
- How Kafka differs from a queue
Lab: Create a topic on a local cluster and observe how partition and retention settings change behaviour.
02
Ordering and partitioning
Ordering holds only within a partition; we turn that fact into a design decision.
- Message keys and the partitioner
- Hot partitions and skewed distribution
- Choosing a partition count, and the consequences of increasing it later
- How ordering breaks (retries, multiple producers)
- Choosing the key from the domain
Lab: Choose a key that preserves the order of events for one customer and reproduce a case where order is broken.
03
Consumer groups and rebalancing
How consumers share partitions, and what that can cause.
- Consumer groups and partition assignment
- Offset commit strategies
- Rebalance types and cooperative rebalancing
- max.poll.interval and the slow-consumer problem
- Consumer lag and the limit of parallelism
Lab: Slow a consumer down to trigger a rebalance storm, find the cause in logs and metrics and fix it.
04
Delivery semantics and idempotent consumption
Duplicates are unavoidable; we learn to write consumers that tolerate them.
- At-most-once, at-least-once, exactly-once
- Idempotent producers and Kafka transactions
- Consumer-side idempotency: unique keys and deduplication
- Retry topics and dead letter topics
- Handling a poison message
Lab: Write a consumer that receives the same message twice and prove with a test that the effect is not applied twice.
05
The transactional outbox
Removing the inconsistency between writing to the database and publishing an event.
- The dual-write problem
- The outbox table and a polling publisher
- A CDC-based approach (the Debezium concept)
- Outbox cleanup and ordering
- The inbox pattern
Lab: Build an outbox in an order service and show that an event is not lost when the service crashes and that duplicates are handled.
06
Event modelling and schema evolution
Events are long-lived contracts; we learn to change them without breaking consumers.
- Event versus command versus state transfer
- Naming and shaping events
- Avro/Protobuf and a Schema Registry
- Backward and forward compatibility rules
- Versioning and living with old events
Lab: Add and remove a field in an existing event schema and observe which change the compatibility check rejects.
07
Choreography and orchestration
Comparing two ways to coordinate a multi-step business flow with events.
- Choreography: benefits and hidden coupling
- Orchestration: central control and its cost
- Visibility of the flow and debugging
- Compensation and timeout handling
Lab: Build the same business flow with both approaches and compare how easy each is to trace and to change.
08
Operations and monitoring
Deciding what must be watched to run a Kafka-based system reliably.
- Consumer lag and alerting
- Broker and topic metrics
- Capacity planning and the cost of retention
- Replay: reprocessing events
- Security basics: authentication and ACLs
Lab: Set up a dashboard and alert rule for consumer lag and verify them with a scenario that creates lag.