Follow
Subscribe via Email!

Enter your email address to subscribe to this platform and receive notifications of new posts by email.

Apache Kafka Architecture and How Real-Time Event Streams Work

Apache Kafka handles real-time data streams through distributed append-only commit logs. Built at LinkedIn and open-sourced through the Apache Software Foundation, it lets producers publish records while consumers read them independently.
A speaker presents on stage in front of presentation projection screens to an audience.

Apache Kafka provides a shared event store and stream platform that holds real-time data in partitioned commit logs with high speed and low latency. Originally built by engineers at LinkedIn, it was donated to the Apache Software Foundation in 2011 to solve data bottlenecks across web systems. Unlike old message queues that delete records once read, Kafka keeps event streams in durable files on disk, letting many client programs read the same log at their own pace without slowing down the cluster [8].

Why Apache Kafka Was Created at LinkedIn

In the early days of LinkedIn, software teams faced a sharp split between two ways of handling data. Web systems ran remote procedure calls that needed sub-second answers, while offline analytics relied on Apache Hadoop batch jobs that ran hours or days later. Chris Riccomini at LinkedIn said this time gap made it hard to show real-time newsfeed updates, member alerts, and ad click metrics. In his InfoQ talk, Riccomini explained that stream tasks need to run in milliseconds or minutes to bridge the gap between synchronous calls and slow batch runs [7].

To bridge that gap, Jay Kreps, Neha Narkhede, and Jun Rao designed a new logging system at LinkedIn. Kreps said he chose to name it after writer Franz Kafka because it was “a system optimized for writing,” matching his love for the author’s books. The team also piped database changes from their internal Databus tool into this stream, turning database mutations into clean event records [7]. Unifying these events helped remove fragile point-to-point links across backend services [8].

Mentors Alan D. Cabrera, Chris Douglas, Geir Magnusson Jr., and Owen O’Malley guided Kafka through the Apache Incubator, reviewing committers like Jakob Homan and Chris Burroughs. On October 23, 2012, it graduated to become a top-level Apache project [1]. Adoption grew as tech firms sought reliable tools to replace scattered message queues [8].

Architecture diagram of KIP-932 share group components and RPC interactions in Apache Kafka.
The KIP-932 architectural proposal outlines share group components and coordinator protocols for cooperative queue processing. (Credit: Apache Software Foundation Confluence)

Append-Only Commit Logs in Apache Kafka

At the core of the system sits an append-only commit log. Rather than sorting index trees in memory, new records go to the end of a permanent file on disk. The broker runs a custom binary TCP protocol that packs records into sets. These batched message sets allow the host system to write sequentially to disk, turning messy network streams into smooth, linear disk writes without random seeking penalties [8].

In their research on replicated logs, G. Wang and fellow LinkedIn engineers said that treating the log as the main building block helped Apache Kafka act as a reliable backbone for storage and stream tasks [10]. Each record holds a key, a payload value, and a timestamp, sorted into named topics [13]. Because data is saved in strict order, standard disk drives reach high write speeds without head movement delays. Replicating log files across server nodes protects data when a machine crashes. A single leader broker serves writes for each log slice, while follower replicas pull copies in the background to keep data safe across the cluster [10].

Unlike traditional brokers that erase messages after a client reads them, Kafka keeps log files for set retention days. Messages stay on disk whether one or ten consumer apps have read them. Each consumer keeps track of its own position with a number called an offset, moving through the log at its own pace without slowing down other readers or locking records for fellow consumers [8].

Topic Partitions and Message Ordering

To spread load across servers, topics are split into logs called partitions. In any single partition, records receive an ordered number. Ordering stays partition-bound. Kafka does not enforce a global order across separate partitions, which lets independent worker threads handle slices of the topic stream side by side without blocking one another [8].

Choosing partition counts needs a careful balance between server resources and message delays. In a 2024 study in Future Generation Computer Systems, researchers T. P. Raptis, C. Cicconetti, and A. Passarella tested topic partitioning heuristics for real-time data flows. They found that while more partitions raise parallel intake, too many partitions overload broker memory and coordinator tasks, especially on smaller edge boxes. By matching stream speed with server hardware bounds, their heuristics pick partition counts that hit latency targets without wasting CPU or bogging down cluster metadata updates across broker nodes [14].

Apache Kafka Architecture and How Real-Time Event Streams Work — GitHub image
(Credit: GitHub)

In a 2023 survey in IEEE Access, Raptis and Passarella grouped Kafka streaming studies into five main areas (algorithms, networks, data models, cyber-physical systems, and security). They showed that partition choices shape network load and consumer balance. When partition assignments fit broker capacity, Kafka clusters keep processing delays under a second even during heavy traffic spikes across wide area networks [11].

Producers and Manual Offset Tracking

Producers are client apps that push records to topic partitions. When sending an event, the producer can pass a key that the client library hashes to pick a destination log slice. Records that share the same key always go to the exact same partition, keeping state changes for a user or device in chronological order. If no key is set, the client sends records across partitions in a round-robin loop to spread network traffic evenly across all broker nodes in the cluster [8].

On the read side, consumers join logical consumer groups to split the work. Each partition in a topic goes to just one consumer in that group, so two workers do not read the same record at once. The consumer tracks its place by saving its offset. Because offset commits can be done by hand, an app that hits a bug can wait to commit, pausing work on that single partition while other partitions keep moving without delay [8].

How do engineers check consumer health when traffic spikes? In a review of Kafka patterns, SoftwareMill engineer Jarosław Kijanowski showed how tools like kafka-consumer-offset-checker compare saved offsets against total log size. Offsets can be saved in coordinator topics or local files like /tmp/connect.offsets in test runs. Saved offsets let restarted workers pick up where they stopped without losing records or re-reading old logs [5].

Kafka Connect for External Storage

Writing new producer and consumer code for every database or file format takes time and causes errors. To simplify data movement, the Apache Software Foundation added the Kafka Connect framework in the 0.9.0.0 release. Connect gives developers a ready-made runtime that handles data conversions, scaling, and crash recovery with plugins called connectors. Source connectors pull records from outside stores into Kafka topics, while sink connectors read topic data and write it to targets like SQL databases or search indexes [5].

Testing the JDBC Source Connector, Kijanowski showed how to sync tables from PostgreSQL 9.5 into Kafka topics. In timestamp and incrementing mode, the connector detects new or edited rows by watching a timestamp column and an auto-incrementing id. Clock precision needs care (specifically millisecond rounding): PostgreSQL makes microsecond timestamps, but Connect stores offsets rounded to milliseconds using LOCALTIMESTAMP(3). If developers keep microsecond timestamps, the offset stored by Kafka stays behind the database value, making the connector pull the same row again every five seconds. Proper timestamp settings stop duplicate message loops while keeping database rows synced cleanly in topic streams [5].

Connect runs in standalone mode or as a clustered service. In distributed mode, workers coordinate over a REST API on port 8083, sharing tasks across servers and rebalancing if a box goes down. The community also built connectors like the Cassandra plugin from DataMountaineer. This plugin links to Cassandra tables, handling bulk loads as well as incremental streaming with timeuuid columns to send device metrics to Kafka topics with reliable delivery [5].

Stateful Stream Processing with RocksDB

While Connect focuses on data ingestion, apps often need to filter and join records on the fly. To support this work, the Apache Software Foundation released the Kafka Streams library in the 0.10.0.0 release. This Java library runs inside client apps, giving developers stream functions like map, filter, and windowing to shape event data without deploying separate stream processing clusters [8].

Stateful tasks (like computing five-minute window counts or joining two live streams) must store intermediate data between incoming events. For this local state, Kafka Streams embeds a RocksDB key-value store right next to each stream task on disk. Because RocksDB flushes data to local drives, the active state can be much larger than host RAM. To keep data safe during server crashes, every change written to RocksDB is also sent to an internal changelog topic in Kafka. If a worker fails, a replacement node can quickly rebuild its local RocksDB store by reading that changelog topic from the start [8].

A related design took shape in Apache Samza, a stream engine built with Kafka at LinkedIn and donated to Apache in 2013 [9]. LinkedIn engineer Chris Riccomini said Samza runs stream tasks inside containers managed by Hadoop YARN, reading records from Kafka and keeping state in local key-value stores. Checkpointing lets a task restart after a crash and pick up where it stopped without losing state, powering LinkedIn newsfeed algorithms, member recommendations, and site health monitoring [7].

How Share Groups Function in Apache Kafka

Standard consumer groups guarantee strict record ordering inside each partition, but they tie consumer scale to partition count. If an app takes time to process each message, teams often over-partition topics just to run more parallel consumers during busy hours. Can Apache Kafka support shared queue semantics without forcing teams to create dozens of extra partitions? In proposal KIP-932, called “Queues for Kafka,” author Andrew Schofield introduced share groups under JIRA ticket KAFKA-16092 to fix this issue [2].

Share groups let many consumers cooperatively read records from the same partition, ending the old one-consumer-per-partition rule [8]. When a worker pulls records in a share group, it gets messages held by an acquisition lock that defaults to 30 seconds under share.record.lock.duration.ms. The worker can acknowledge successful handling, release the record for another try, or reject a bad message. If the worker crashes, the broker automatically unlocks the record when the timer runs out, letting another worker take it. Under the KIP-848 consumer rebalance protocol, a group coordinator uses SimpleAssignor to hand out partition tasks without needing partition fencing barriers [2].

Kafka continues to evolve on GitHub, where code builds run on Java 17 and Java 25 alongside Scala 2.13. Code quality is checked with SpotBugs and Checkstyle [3]. On October 13, 2026, the community tagged release 4.4.0, polishing core modules, client libraries, and stream connectors [4]. By joining append-only commit logs with flexible share groups, Kafka provides unified messaging for modern workloads across cloud and on-premises deployments [2].

Deployments from Bio-Data to Smart Cities

Beyond web services and corporate IT, Apache Kafka handles big data in scientific research. In a 2018 study in GigaScience, B. Lawlor, R. M. Lynch, M. M. Aogáin, and P. Walsh tested Kafka as a store for genetic data. Life scientists often faced a choice between flat files (which do not scale well) and rigid relational schemas. The authors showed that holding NCBI Reference Sequence genetic data in Kafka enabled fast parallel access and stream analysis while keeping data formats flexible for downstream research tools [12].

Sensor networks form another busy use case. In 2023, G. Calderón and co-authors tested an IoT platform linking Apache Kafka with Elastic Stack tools like Elasticsearch, Kibana, and Beats across edge nodes and cloud boxes. Their test runs showed how broker pipelines separate sensor intake from live dashboards, allowing administrators to monitor distributed device health without dropping records during network disruptions [6].

Industrial plants and enterprise tools have also adopted this architecture. Researchers V. Ashiwal, A. M. Gutierrez, K. Aschbacher, and A. Zoitl tested Kafka as middleware for PLC service buses under the IEC 61499 standard. In business systems, teams link software like Microsoft Dynamics 365 to Kafka to sync customer records across branches [6]. By pairing durable append-only commit logs with rich client libraries and share groups, Kafka remains a cornerstone for real-time distributed computing [8].

Sources
  1. WEBSITE Apache Software Foundation. (2011). Kafka Incubation Status. Apache Incubator. [Article Link]
  2. WEBSITE Schofield, A. (2026). KIP-932: Queues for Kafka. Apache Software Foundation Confluence. [Article Link]
  3. WEBSITE Apache Software Foundation. (2011). Apache Kafka Source Code Repository. GitHub. [Article Link]
  4. WEBSITE Apache Software Foundation. (2026). Release 4.4.0. GitHub. [Article Link]
  5. ONLINE NEWS Kijanowski, J. (2016). Kafka Connect – Import Export for Apache Kafka. SoftwareMill. [Article Link]
  6. WEBSITE Google Scholar. (2016). Apache Kafka Scholarly Literature Index. Google Scholar. [Article Link]
  7. ONLINE NEWS Shenoy, R. (2014). How LinkedIn Uses Apache Samza. InfoQ. [Article Link]
  8. WEBSITE Wikipedia. (2026). Apache Kafka. Wikimedia Foundation. [Article Link]
  9. WEBSITE Wikipedia. (2026). Apache Samza. Wikimedia Foundation. [Article Link]
  10. ACADEMIC JOURNAL Wang, G., Koshy, J., Subramanian, S., Paramasivam, K., Zadeh, M., Narkhede, N., Rao, J., Kreps, J., & Stein, J. (2015). Building a replicated logging system with Apache Kafka. Proceedings of the VLDB Endowment, 8(12), 1654-1655. [Article Link]
  11. ACADEMIC JOURNAL Raptis, T. P., & Passarella, A. (2023). A Survey on Networked Data Streaming With Apache Kafka. IEEE Access, 11, 85333-85350. [Article Link]
  12. ACADEMIC JOURNAL Lawlor, B., Lynch, R. M., Aogáin, M. M., & Walsh, P. (2018). Field of genes: using Apache Kafka as a bioinformatic data repository. GigaScience, 7(4). [Article Link]
  13. ACADEMIC JOURNAL Shaheen, J. A. (2017). Apache Kafka: Real Time Implementation with Kafka Architecture Review. International Journal of Advanced Science and Technology, 109, 35-42. [Article Link]
  14. ACADEMIC JOURNAL Raptis, T. P., Cicconetti, C., & Passarella, A. (2024). Efficient topic partitioning of Apache Kafka for high-reliability real-time data streaming applications. Future Generation Computer Systems, 154, 173-188. [Article Link]

Cover image: graysky. / Wikimedia Commons, CC BY 2.0

Leave a Comment

Related Posts
Total
0
Share