Kafka Fundamentals - Guide to Distributed Messaging

Why Does Kafka Exist?

You are building a backend system with multiple services (auth, payments, notifications, analytics). Each service needs to communicate with other services. Issues come when Service A needs to call Service B, and if Service B is down, Service A also fails. This type of communication increases system complexity and scaling problems. This is why Kafka is built.

Kafka introduces a new paradigm; instead of sending requests and waiting for responses, Kafka lets you treat data as a continuous stream of events. You publish events, allowing others to consume them independently.

What Does Kafka Provide?

  • Decoupling at Scale - By using Kafka, your services do not need to establish direct communication with other services; instead, they push events in Kafka, and other services receive them independently.
  • Massive Throughput - Kafka is designed for high-throughput event streaming. The throughput a cluster can achieve depends on factors such as message size, partition count, hardware, replication, compression, and producer and consumer configuration.
  • Durability - Kafka retains records according to the topic’s retention policy rather than deleting them simply because a consumer read them. This allows consumers to replay data while it remains retained.
  • Fault Tolerance - Partitions can be replicated across brokers, allowing the cluster to continue serving data after some failures when replication and acknowledgement settings are configured appropriately.

What Is Kafka?

Apache Kafka is a distributed event streaming platform used to handle real-time data feeds at massive scale. Think of Kafka like a high-speed messaging system where producers send messages, Kafka stores messages, and consumers read them. But unlike traditional messaging systems, Kafka is built for high throughput, fault tolerance, scalability, and real-time processing.

Kafka vs. Traditional Message Queues

Kafka is very different from traditional message queues. Traditional message queues are built for task distribution; they tell who will process the job. But Kafka is built for event streaming that says what happened and who all should know. In traditional queues a message is deleted once a consumer processes it, but in Kafka it is retained so that we can replay the events if we want. Kafka is optimized for high-throughput event streaming, while traditional queues make different trade-offs around delivery, routing, and task processing.

Core Concepts

Let’s go through the main Kafka concepts step by step, starting with how messages are produced and consumed.

Producer

The producer is the application that is responsible for sending messages to Kafka.

Topic

A topic is a logical category used to organize and store messages. For example, messages related to payment can go to the “payments” topic.

Consumer

The consumer is the application that is responsible for fetching messages from Kafka and processing them.

Partitions

A partition is a subdivision of a topic that allows Kafka to split data for scalability. Each partition stores messages in an ordered sequence, ensuring order within that partition. Partitions enable parallel processing by allowing multiple consumers to read data simultaneously.

Partitions Are Way More Important

Partitions are more important than you think because they provide us with the biggest advantages:

  • Parallelism
  • Ordering Guarantee

If a topic has three partitions, up to three consumers in the same consumer group can process those partitions in parallel. This is very important for scalability. When you need to process more data from a parallel topic, you need to create more partitions.

If messages “A” and “B” are written to the same partition, and “A” is appended first, consumers of that partition will read “A” before “B”. Kafka does not provide a global ordering guarantee across all partitions of a topic.

How Does a Consumer Actually Get Data?

Kafka consumers use a pull-based mechanism to fetch messages. Instead of Kafka saying, “Okay, take this message,” to a consumer, the consumer says to Kafka that “we need this message.” But the question is how do consumers actually know which message to read? This is where offset comes into play.

In Kafka, an offset is a unique, sequential and immutable integer assigned to each message within a partition. When a consumer starts, its starting position depends on whether it already has a committed offset and on the configured offset-reset behavior. It is not necessarily offset 0. Once it reads the 0th offset, it can go to the 1st, 2nd, 3rd and so on. A consumer controls its position by committing offsets. A message being read is not the same thing as its offset being committed; applications choose when to commit based on their processing semantics. In case any consumer crashes when a new consumer instance starts, it will use the last committed offset to resume reading from where the previous one left off.

Consumer Group — An Important Concept

A consumer group is a collection of consumers that work together to consume messages from a topic. Here are a few important points about consumer groups.

  • Each partition will be processed by exactly one consumer in the group.
  • If there are more consumers than partitions, then some consumers will sit idle.
  • If there are fewer consumers than partitions, then some consumers will process multiple partitions.

Consumer groups are used to enable horizontal scalability, high-throughput parallel processing, and fault tolerance.

💡 If consumers are added or removed from the group, Kafka can rebalance the partition assignments across the active members of the consumer group.

Kafka Architecture

Broker

A broker is a core server component in Kafka that is responsible for receiving messages from producers, storing them and sending them to consumers. All the magic happens inside this component. It is often referred to as the Kafka server.

Cluster

A cluster is a group of multiple brokers working together as a single system. Instead of relying on a single broker, Kafka distributes data across multiple brokers to handle more load and avoid a single point of failure. Each broker in the cluster stores a part of the data and coordinates with others to serve requests. This setup makes Kafka highly scalable and fault-tolerant, meaning even if one broker goes down, the cluster can still continue to operate without data loss.

ZooKeeper (Historical Architecture)

ZooKeeper was the coordination and metadata system used by older Kafka architectures. It did not store Kafka message data; it stored cluster metadata and coordination state. Modern Kafka clusters can use KRaft instead, which moves metadata management into Kafka itself and removes the ZooKeeper dependency.

KRaft

KRaft is Kafka’s new way of managing the cluster without using ZooKeeper. Instead of relying on an external system, Kafka now handles its own metadata and coordination internally.

In KRaft mode, a set of brokers form a controller quorum that uses the Raft consensus algorithm to manage cluster state. This includes things like broker registration, topic metadata, and leader election. Everything that ZooKeeper used to do is now handled inside Kafka itself.

Data Replication

Replication gives Kafka fault tolerance by keeping multiple replicas of a partition. Each partition has a leader replica and follower replicas. Writes go to the leader and are replicated to followers; reads are typically served by the leader, although Kafka can also be configured for follower reads. With an appropriate replication and acknowledgement configuration, another replica can take over after a broker failure.

In-Sync Replicas

ISR is the set of replicas that are considered sufficiently caught up with the leader according to Kafka’s replication rules. Not all replicas are always in sync; some might lag behind because of network or processing delays. Kafka tracks the replicas that are in the ISR. Under the normal clean leader-election path, a new leader is chosen from eligible in-sync replicas, helping avoid losing committed data during failover. If a replica is too far behind, it is removed from ISR until it catches up again.

Delivery Guarantees

Kafka provides different levels of delivery guarantees depending on how you configure producers and consumers. This defines how safe your data is vs how fast your system runs.

  • At most once — The consumer commits its position before processing. If processing fails afterward, the message may not be processed again.
  • At least once — The consumer processes the message and then commits its position. If it fails between those operations, the message can be processed again.
  • Exactly once — Kafka provides transactional and idempotent mechanisms that can support exactly-once processing semantics when used correctly. Exactly-once does not automatically make arbitrary external side effects exactly once.

That’s It

So these were the fundamentals that everyone must know. If we go further, we can dive into other concepts such as scaling Kafka, data retention & storage, setting up a Kafka cluster, etc. But in order to keep this post simple and beginner-friendly, we are not going to dive into those.

Before You Go

If you made it this far, thank you.

I usually write about backend engineering, distributed systems, and things I learn while working on real problems. Not theory — mostly practical stuff that I wish someone had explained to me earlier.

I run a free newsletter where I share these kinds of write-ups. No spam. Just occasional backend engineering notes.