Apache Kafka Essentials

Core model: topics and partitions

Kafka stores events in topics. A topic is split into partitions, each an append-only, ordered log. Every record in a partition gets a sequential offset. Ordering is guaranteed only within a partition, never across the whole topic.

Records have an optional key, a value, a timestamp and headers. When a record has a key, the default partitioner hashes it so the same key always lands in the same partition, which keeps per-key ordering (for example all events for one customer). Records without a key are spread across partitions. Data is not removed when it is read; it is kept until retention expires.

Partition count sets the upper bound on consumer parallelism and on throughput. Choose it with growth in mind, because adding partitions later changes which partition a key maps to.

Producers

A producer serialises records (strings, JSON, Avro, Protobuf), picks a partition and sends batches to the partition leader. Key settings:

Setting	Effect
acks	How many acknowledgements before a send counts as successful
retries	How often failed sends are retried
delivery.timeout.ms	Total time budget for a send, including retries
enable.idempotence	Prevent duplicates caused by retries
linger.ms / batch.size	How long and how much to batch before sending
compression.type	Compress batches (for example lz4, snappy, zstd)

Acknowledgements

• acks=0: fire and forget. Fastest, but data can be lost silently.
• acks=1: the leader confirms after writing. Data can still be lost if the leader fails before followers copy it.
• acks=all: the leader waits for all in-sync replicas. Combine with min.insync.replicas=2 (and replication factor 3) for strong durability.

Retries and idempotence

Retries protect against transient errors but can duplicate a message (the broker wrote it, the acknowledgement was lost, the producer resent) and can reorder messages when several batches are in flight. An idempotent producer attaches a producer id and sequence numbers so the broker discards duplicates and keeps order. Recent client versions enable it by default along with acks=all; for older clients set it explicitly. A "safe" producer therefore uses idempotence, acks=all, and a generous delivery.timeout.ms.

For throughput, a small linger.ms (a few milliseconds) plus larger batches and compression usually trades a little latency for much better efficiency.

Consumers and groups

Consumers pull records and track progress by committing offsets (stored in the internal __consumer_offsets topic). Consumers sharing a group.id form a consumer group: each partition is assigned to exactly one member, so the group divides the work. More consumers than partitions leaves the extras idle. Different groups each receive all the data independently.

Rebalancing

When members join, leave or crash, partitions are reassigned. The classic eager protocol revokes everything and pauses the group; the cooperative (incremental) protocol moves only the affected partitions. Static membership (group.instance.id) lets a restarted consumer keep its partitions within the session timeout and avoids needless rebalances.

Delivery semantics

Committing after processing gives at-least-once delivery, so make processing idempotent. Committing before processing risks at-most-once. Exactly-once is available within Kafka using idempotent producers and transactions.

Offset reset behaviour for a new group is controlled by auto.offset.reset (earliest or latest).

Brokers and replication

A broker is a server that stores partitions. Clients connect to any broker (the bootstrap server), receive cluster metadata and then talk directly to the right leaders.

Each partition has a replication factor. One replica is the leader and handles reads and writes; the others are followers that copy it. Followers that keep up form the in-sync replica (ISR) set. If the leader dies, a new leader is elected from the ISR. Setting unclean.leader.election.enable=true allows an out-of-sync replica to become leader, trading data loss for availability, so keep it false for important data.

A common production baseline is replication factor 3 with min.insync.replicas=2. Consumers can also fetch from the nearest follower replica in recent versions, which reduces cross-zone traffic.

KRaft versus ZooKeeper

Historically Kafka used an external ZooKeeper ensemble to hold cluster metadata and elect a controller. KRaft mode replaces it with a Raft-based quorum of controllers built into Kafka. Benefits: one system to operate, faster controller failover and better scaling to many partitions. Newer Kafka releases have removed ZooKeeper entirely, so new clusters should use KRaft.

Everyday CLI commands

# create a topic
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 3 --replication-factor 1

# inspect topics
kafka-topics.sh --bootstrap-server localhost:9092 --list
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

# produce with keys, consume with metadata
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders \
  --property parse.key=true --property key.separator=:
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders \
  --from-beginning --property print.key=true --property print.timestamp=true

# consumer groups
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-group \
  --topic orders --reset-offsets --to-earliest --execute

In --describe output, the lag column (log end offset minus committed offset) is the first metric to check when consumers fall behind. Reset offsets only while the group is stopped.

Connect, Streams and Schema Registry

• Kafka Connect moves data between Kafka and external systems without custom code. Source connectors ingest from databases or files; sink connectors write to stores or search engines. Workers run connectors as tasks and can be scaled in distributed mode, with configuration submitted over a REST API.
• Kafka Streams is a Java library for stateful stream processing (filtering, joins, windowed aggregations) that runs inside your application.
• Schema Registry stores schemas for Avro, Protobuf or JSON Schema and enforces compatibility rules so producers and consumers can evolve independently.

Durability and retention

Durability is the combination of replication factor, acks=all and min.insync.replicas. With factor 3 and a minimum of 2, you can lose one broker with no data loss, and writes are rejected rather than silently risked if too few replicas remain.

Retention is governed by a cleanup policy:

• delete (default): discard segments older than retention.ms or beyond retention.bytes.
• compact: keep only the latest record per key, useful for changelogs and state.