Kafka Internals: Why It Is Fast and How It Scales

An order service publishes an OrderPlaced event. Billing charges the customer, inventory reserves stock, and analytics updates a dashboard. Tomorrow, a fraud service needs the same events, including last week's history.

A service-to-service call can deliver the event now. A work queue can hold it until a worker processes it. Kafka solves a broader problem: keep a shared event history that independent applications can read at their own pace.

Its performance comes from that choice. Kafka organizes work around partitioned logs, large batches, and sequential access. Its scaling limits come from the same design.

This guide follows a record through the system, then examines failures and capacity. It assumes basic backend knowledge, not prior Kafka operations experience. Configuration and guarantees use Apache Kafka 4.1 documentation as the baseline, with conventional consumer groups unless stated otherwise.

Start with a Log Rather Than a Queue

A log is an ordered sequence of records. Append adds another record to its end. Reading a record does not remove it.

Kafka retains records according to time, size, or compaction policies. Billing can finish processing an event while analytics remains several minutes behind. Neither application's progress decides whether the other may read it.

Each application keeps a position in the log. A new application can start at an earlier retained position and rebuild its state. This is useful for search indexes, fraud models, audit pipelines, and recovering from processing bugs.

Contrast this with a conventional work queue: workers compete for jobs, then acknowledge completed work so the queue can remove it. Both models are useful. Kafka's advantage becomes substantial when several independent applications need the same durable, replayable stream.

The distinction is about consumption and retention semantics, not product labels. RabbitMQ also offers streams, and Kafka has a separate share-group model. This article uses the classic retained-log model to explain Kafka's core design.

Build the Mental Model

A topic, such as orders, names a stream. Kafka splits it into partitions. Each partition is a separate ordered log.

A record contains a key, value, timestamp, and optional headers. Within a partition, Kafka assigns it an offset: a position such as 1042. An offset is neither a global event ID nor a timestamp. Partition 0 and partition 1 can both contain offset 1042.

A broker stores partition replicas and handles client requests. Each partition has one leader and follower replicas. Writes go to that partition's leader; followers fetch its log to maintain copies.

The diagram shows two replicas per partition for clarity. A common production starting point is three, spread across independent failure domains.

Notice that leadership is distributed. Broker A can lead one partition and follow another. A Kafka cluster does not have one data leader through which every event passes.

Keys define the ordering boundary

Using order_id as the key normally sends events for one order to the same partition, assuming a stable partitioning scheme. That gives the consumer a log order for OrderPlaced, PaymentCaptured, and OrderShipped.

It does not establish a global business-event order across producers. If two services race to publish, the partition records arrival order. The application still needs rules about valid state transitions.

Ordering also depends on processing. Reading in order and launching uncoordinated asynchronous tasks can produce out-of-order side effects.

KRaft manages metadata

Modern Kafka uses KRaft, a Raft-based metadata quorum, instead of ZooKeeper. Controllers track brokers, topics, partition assignments, and leadership changes. One controller is active; a majority replicates metadata decisions.

Three controllers tolerate one controller failure. Five tolerate two, with additional operating cost. Production deployments commonly separate controller and broker roles.

This is the control plane. Order records travel between producers, brokers, and consumers, not through the controller quorum. Losing a controller majority threatens metadata changes and failure recovery; it does not mean every established data request instantly routes through a broken controller.

Follow a Record from the Producer

The producer initially contacts a bootstrap broker to discover cluster metadata. It learns which broker leads each partition, then connects directly to the relevant leaders.

That explains a common networking failure: the bootstrap address is reachable, but the broker addresses returned through advertised.listeners are not. Discovery succeeds; producing fails when the client follows those addresses.

Before sending, the producer serializes the key and value, chooses a partition, and accumulates records in a per-partition batch. A sender thread sends ready batches, potentially including several partitions in one broker request.

Batching trades a little waiting for less overhead

Sending 1,000 small records as 1,000 requests repeats network and protocol overhead. Sending them in batches amortizes that cost.

Two settings shape this behavior:

• batch.size controls the target batch capacity in bytes for a partition.
• linger.ms allows a short wait for more records before sending an incomplete batch. A full batch can leave sooner.

These are not end-to-end latency guarantees. Network congestion, broker queues, replication, and retries add their own delays.

Compression works across a batch, where repeated JSON field names and similar values compress well. The batch is stored and fetched in compressed form. Brokers still validate data, and some configurations require recompression; compression does not eliminate broker CPU work.

Here is an illustrative Java producer properties file. It assumes the named brokers are reachable and shows the throughput/durability settings; add the authentication and TLS configuration required by your cluster.

bootstrap.servers=broker-a:9092,broker-b:9092,broker-c:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
 
# Require replicated acknowledgement and deduplicate producer retries.
acks=all
enable.idempotence=true
max.in.flight.requests.per.connection=5
 
# Starting values to measure with the real payload distribution.
compression.type=lz4
batch.size=65536
linger.ms=5
delivery.timeout.ms=120000

Blindly increasing batch.size will not create large batches if traffic is thinly spread across thousands of partitions. Measure actual batch sizes, compression ratio, request latency, and buffer exhaustion.

The producer's buffer is also not durable storage. If a process exits before buffered records are delivered, those records can disappear. Handle send results, flush during orderly shutdown, and use an outbox when database changes and event publication must stay coordinated.

Inside Broker Storage

A partition is not one endlessly growing file. Kafka divides its log into segments, usually named by their starting offset. New records append to the active segment; size or time limits eventually roll it into a closed segment.

Segments have companion indexes. An offset index maps selected offsets to approximate file positions. A time index helps locate records by timestamp.

The indexes are sparse: they do not need one entry for every record. Kafka finds a nearby position, then scans forward. This balances lookup speed against index size.

These indexes locate offsets and times. They do not turn Kafka into a database that efficiently answers arbitrary queries such as “find every order above $500.” Build a read model for that.

The page cache does much of the work

Kafka relies heavily on the operating system's page cache. File writes initially enter memory managed by the kernel. The OS later writes dirty pages to storage.

Consumers close to the head of the log often read those same cached pages. Several consumer groups can reuse them instead of each triggering a fresh disk read.

This is why broker memory planning is not simply “give the JVM all available RAM.” The OS needs room to cache log data. A large historical replay can read cold segments from disk, compete with live traffic, and change the cluster's performance profile.

Retention and compaction solve different problems

With delete retention, Kafka removes eligible old segments according to configured time or size limits. Deletion is segment-based and asynchronous, not a precise per-record timer. A slow consumer can lose access to data it has not processed.

With log compaction, Kafka eventually removes superseded values for the same key. An account-status stream can retain the latest known value for each account while discarding older versions.

Compaction does not immediately leave exactly one record per key. Recent duplicate keys can remain until cleaning runs. A null-valued record acts as a deletion marker, called a tombstone, with its own retention behavior.

Offsets remain stable through compaction, so gaps are normal. A compacted topic is also not a complete historical audit trail: old values are deliberately removed.

Replication and What an Acknowledgement Means

Suppose a partition has replication factor three. Broker A leads; B and C follow. Followers fetch batches from A and append them to their own logs.

Kafka tracks in-sync replicas, or ISR: replicas considered sufficiently caught up under Kafka's liveness and lag rules. The leader is included. A slow follower can leave the ISR and later rejoin after catching up.

The producer's acks setting controls the acknowledgement requirement:

acks=all means all current ISR replicas, not every configured replica forever. Pair it with min.insync.replicas to define the minimum acceptable ISR size.

For replication factor three and minimum ISR two:

• With A, B, and C in the ISR, success requires their replication progress.
• With A and B in the ISR, writes can still succeed.
• With only A in the ISR, acks=all writes fail rather than accept single-copy success.

The minimum is an admission threshold, not an instruction to pick any two replicas and ignore the third while it remains in the ISR.

The diagram assumes all three replicas remain in sync and simplifies the fetch loop. Follower replication is pull-based, not a leader pushing one RPC to each follower per record.

The high watermark marks the replication commit boundary. Ordinary consumers cannot read the uncommitted tail above it. Kafka 4.1's strict minimum-ISR behavior also prevents this boundary advancing when the ISR falls below the configured minimum.

A transactional consumer using read_committed has another boundary: the last stable offset, which prevents it from reading past unresolved transactions. Aborted transactional records are filtered out.

Replicated does not mean flushed on every disk

Appending a batch can place it in the OS page cache. Kafka does not normally force a disk flush on every replica before acknowledging each batch.

Its usual durability strategy combines replication across independent machines with background flushing. This handles individual machine failures efficiently. Correlated failures that lose every relevant volatile copy are a different failure model.

So acks=all is not equivalent to “three physical disks have synchronously persisted this record.” Placement, power-loss behavior, storage reliability, and failure assumptions still matter.

To create the example topic, run this from a Kafka distribution against an existing three-broker lab cluster. Add --command-config with your admin client properties when authentication is enabled.

bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create \
  --topic orders \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000

This keeps approximately seven days by time policy, subject to segment cleanup behavior and any other retention limits. The twelve-partition count is an example, not a universal default.

Follow the Record into a Consumer Group

A consumer requests batches starting at an offset. Fetching is pull-based, and the broker can hold a fetch briefly while waiting for data. This avoids a tight empty-poll loop.

A consumer group distributes a subscription across its members. In a conventional group, one partition belongs to at most one consumer at a time. One consumer may own several partitions.

Billing and analytics each see the stream. Billing workers share its processing; they do not each receive every record.

Position and committed offset are different

The consumer's current position tracks fetching progress. Its committed offset is a restart checkpoint stored through the group coordinator, backed by Kafka's internal __consumer_offsets topic.

Committing offset 105 means “resume at 105,” not “105 was the last processed record.” The application should only checkpoint past records whose required work has completed.

Consider a worker that charges a payment and then commits its offset. If it crashes between those steps, Kafka redelivers the record. Committing first reverses the risk: a crash can skip the charge entirely.

For external effects, use an idempotency key such as the event ID, or atomically store a deduplication marker with the database update. An offset commit alone cannot make a payment API call exactly once.

Rebalances move ownership

When workers join, leave, or fail, the group adjusts partition ownership. This is a rebalance. It is separate from moving partition files between brokers.

Kafka's group protocols differ in how much work pauses during reassignment. Incremental approaches reduce disruption, but an application still needs to handle revoked partitions, incomplete work, and offset management correctly.

Long processing intervals can also make a consumer exceed max.poll.interval.ms. Increasing the timeout may postpone the symptom; it does not improve processing capacity. Bound work per poll and account for downstream latency.

Inspect the group rather than guessing from CPU utilization:

bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group billing

The output includes per-partition current offsets, log-end offsets, lag, and ownership. Offset lag is not a direct measurement of seconds or bytes. Compaction, transactions, payload variation, and processing cost all affect its interpretation.

Why This Design Is Fast

Kafka combines several efficiencies:

1. Append-oriented storage: normal writes avoid finding and updating arbitrary existing records.
2. Batching: requests, compression, and storage operations amortize fixed costs across many records.
3. Page-cache reuse: recently appended data can serve several readers without repeated disk access.
4. Efficient transfer: suitable plaintext paths use file-to-socket transfer to avoid unnecessary user-space copying.
5. Small progress state: conventional groups checkpoint offsets rather than maintain a separate completion record for every event.

The fourth point is often called zero-copy. It does not mean no bytes ever move. It means avoiding particular copies between kernel and application buffers. In Kafka 4.1, TLS uses a different path and does not use this sendfile optimization.

Sequential access helps SSDs too, although the old comparison with mechanical disk seeks is less dramatic. Real deployments also interleave many partitions, run compaction, and serve historical reads. Actual I/O is more complicated than one perfectly sequential file.

These choices favor sustained throughput and replay. They do not prove that Kafka has the lowest latency for one tiny message, or that adding consumers is free. Each independent reader still consumes network bandwidth and application resources.

How Kafka Scales and Where It Stops

The partition is the main unit of data parallelism. Different leaders can serve different writes on different brokers. More brokers help when partition replicas and leadership are distributed onto them.

Adding a broker requires moving work

An empty broker does not automatically remove load from existing leaders. Existing partitions need reassignment, whether performed by an operator, a balancing tool, or a managed service.

Follower copies are omitted here. Replica migration consumes disk reads, network bandwidth, and writes while live traffic continues. Plan headroom and throttle movement when necessary.

Consumers scale only as far as the partition model allows

For twelve partitions, a conventional consumer group can have at most twelve consumers actively owning those partitions. A thirteenth does not create another partition.

Application-level concurrency can increase work inside an owner, but then the application must preserve any required ordering and commit only completed contiguous progress.

Kafka 4.1 also documents share groups as a preview with different record-sharing semantics. Their ability to share partitions should not be confused with conventional consumer-group ownership.

Hot keys defeat even partition counts

Suppose one large merchant produces half of all events. Partitioning by merchant ID concentrates that traffic on one partition. Increasing from twelve to forty-eight partitions does not split the merchant's key.

Possible responses include choosing a finer key, isolating that tenant, or explicitly splitting its stream. Each changes the ordering boundary. You cannot divide one strictly ordered stream across arbitrary workers and retain its original serial semantics for free.

Adding partitions can also change key-to-partition mapping for subsequent records. Old records stay where they were. A key can then span old and new partitions, complicating ordering during the transition. Treat repartitioning as an application migration.

Too many partitions have a cost

Partitions create replicas, files, indexes, metadata, replication work, and recovery overhead. A huge count can increase election and restart costs without increasing useful throughput.

Choose counts from measured producer and consumer capacity, expected growth, and skew. Also distribute replicas across racks or availability zones; three copies in one failure domain provide less protection than the number suggests.

Cross-region replication usually deserves separate clusters and an explicit replication design. Putting a synchronous replication path across a long-distance link adds latency and couples availability to that link.

What Happens When Something Fails

A partition leader disappears

In the simple case, another in-sync replica survives. The controller selects a new leader, clients refresh metadata, and requests resume.

Replication and leadership changes are not instantaneous. In-flight requests may time out, and retries are expected.

The full election rules depend on the Kafka version and feature state. Kafka 4.1 enables eligible leader replicas, or ELR, by default on new clusters. This tracks certain out-of-ISR replicas known to be safe candidates under the strict minimum-ISR rules. Therefore, “only a current ISR member can ever be elected safely” is too simplistic for modern Kafka.

An unclean election that chooses a replica without the required data is different: it can restore availability by discarding acknowledged history. Understand that trade-off before enabling it.

A producer loses the success response

The batch might already be committed even though the producer sees a timeout. Retrying is necessary, but it can duplicate records without protection.

The idempotent producer uses producer identity and sequence numbers so brokers can recognize retry duplicates. It does not deduplicate two separate business-level send calls containing the same order event, nor does it solve a database/event dual write.

A consumer repeats completed work

This is the payment-and-offset gap described earlier. At-least-once processing accepts redelivery and makes effects idempotent.

For Kafka-to-Kafka processing, transactions can atomically write output records and consumed offsets. Downstream read_committed consumers exclude aborted output. Transaction abort handling must also reset local processing state and consumer position appropriately.

That is a useful exactly-once boundary. It does not automatically include email, payment providers, or arbitrary database writes. Define the boundary before claiming the guarantee.

When Kafka Is Better Than the Alternatives

Kafka is strongest when retention, replay, independent consumers, and a streaming ecosystem matter together. Its architecture explains why.

Pulsar and Redpanda also support durable streaming. “Kafka has a log” does not distinguish it from them. Compare actual workloads, compatibility requirements, operating effort, and recovery behavior.

For a small application sending background email jobs, Kafka's partition planning and broker operations may buy little. For a shared event backbone feeding billing, search, analytics, and rebuilding workflows, the same machinery can remove substantial application complexity.

The streaming-platform comparison explores the broader selection decision.

Work Through a Capacity Estimate

Assume a workload with these illustrative inputs:

• 20,000 records per second, each averaging 1,000 bytes before compression.
• A measured 2:1 compression ratio for these payloads.
• Seven days of retention and replication factor three.
• Three independent consumer groups, each reading the whole stream.

Using decimal units and ignoring indexes and protocol overhead:

Raw ingress           = 20,000 x 1,000 = 20 MB/s
Compressed ingress    = 20 / 2         = 10 MB/s
One retained copy     = 10 x 604,800   = 6.048 TB
Three retained copies = 6.048 x 3      = 18.144 TB
Follower replication  = 10 x (3 - 1)   = 20 MB/s
Consumer egress       = 10 x 3         = 30 MB/s

These are aggregate steady-state estimates, not a broker count. Consumer compression behavior is assumed unchanged, and replays, migrations, and catch-up traffic are excluded.

At a target maximum disk occupancy of 70%, the retained copies alone imply about 18.144 / 0.7 = 25.92 TB of provisioned capacity. Add segment overhead, growth, and the ability to recover after a broker loss. Tiered storage can change the local-storage calculation, but introduces remote-read and retention considerations.

Now suppose a load test shows a consumer can sustainably process 2,500 records per second from one partition. The ideal lower bound is eight partitions. Twelve gives a starting margin, but only with reasonably balanced keys and sufficient downstream capacity.

Backlog recovery needs spare capacity too. At 20,000 incoming records per second and 30,000 processing capacity, a 36-million-record backlog takes about one hour to drain:

Catch-up rate = 30,000 - 20,000 = 10,000 records/s
Recovery time = 36,000,000 / 10,000 = 3,600 seconds

If processing merely equals ingress, the backlog never shrinks. Adding storage extends the time available; it does not fix the processing bottleneck.

Monitor per-partition traffic, ISR changes, under-replicated partitions, request latency, disk space, network saturation, and consumer lag trends. A cluster-wide average can hide the one hot partition that limits the application.

Keep the Core Model in Mind

Kafka is fast because it does fewer expensive operations per record. It scales because independent partition leaders and consumers can work in parallel. It supports replay because consuming data does not delete it.

Those strengths have boundaries: ordering is partition-local, independent readers consume bandwidth, replica placement matters, and external side effects need their own correctness rules.

Once those boundaries are clear, settings stop looking like magic. They become explicit choices about batching, failure tolerance, recovery time, and the amount of parallel work your application can safely perform.

References

• Kafka 4.1 design: storage, replication, consumers, and delivery semantics
• Producer configuration and topic configuration
• KRaft operations
• Eligible leader replicas and strict minimum ISR
• Consumer rebalance protocol