Why Apache Kafka is the Best Tool for Data Streaming

In continuing our series on data engineering, the next rational step felt obvious: Apache Kafka. Whether it’s logging events, collecting metrics, or enabling real-time analytics, Apache Kafka has become a go-to solution for building dependable event streaming systems.

But what is an event streaming system, and how is it important to us? More to the point, why is Apache Kafka considered to be so important? Well, good news for you – that’s what this article is all about! 

What is Apache Kafka?

Apache Kafka is an open-source distributed event streaming platform originally developed by LinkedIn and now maintained by the Apache Software Foundation. In English: it’s a tool that is designed to handle a high-volume of streaming (e.g. continuously occurring) data in real-time.

Kafka is commonly used for:

Event-driven architectures – Any app that involves a ton of data going this way and that constantly (text messaging and social media comes to mind!) can be handled appropriately by Kafka.
Log aggregation – Quickly turn many small events into an easy to read log, depending on what information is necessary.
Real-time analytics – Kafka can also be used for pure data analysis, such as keeping tabs of the inflow/outflow of a company’s servers!
Streaming ETL pipelines – We’ve covered a bit on ETL for streaming data before. Kafka is the go-to solution for this!How Kafka Works: Core Concepts

To understand how Kafka works, it’s important to be familiar with its core components:

Topics

A topic is a logical channel where data records (also called messages – don’t get these confused with our text messaging example, we’re just referring to generic data here) are published. Producers send data to topics, and consumers read data from them. Speaking of which…

Producers

Producers are client applications that publish (write) data to Kafka’s topics. Producers can push data to specific partitions within a topic based on keying strategies. For example, a server might occasionally send off data about its current state to the “Server Uptime” topic. This makes the server a producer. 

Consumers

Consumers are applications that subscribe to topics and read (AKA consume) data from them. Kafka keeps track of which messages each consumer has read using offsets.

Offsets

The way that Kafka tracks the position of a consumer in a topic partition. This allows consumers to pick up where they left off, or replay messages as needed.

Partitions

Topics are split into partitions, which allows Kafka to scale horizontally. Each partition is an ordered, immutable sequence of records.

Brokers

A Kafka broker is a server that specifically stores data and serves client requests. In other words, they’re the brains of the whole operation. Kafka clusters tend to be made up of multiple brokers!

Best Practices for Using Kafka

And, as always, let’s finish things off by talking about some best practices you can use when deploying and utilizing Kafka for your own projects!

1. Plan for PartitioningDesign your topic partitioning strategy carefully. A good design will save you from modification headaches down the road!
More partitions allow higher parallelism but also increase overhead. Find the right balance for your needs.
Make sure partitions distribute the load evenly to avoid hotspots!
2. Monitor Lag and ThroughputKeep an eye on consumer lag to ensure consumers are keeping up with producers.
Use tools like Kafka’s internal JMX metrics, or bring in external solutions like Prometheus and Grafana for monitoring!
3. Use CompressionEnable message compression (e.g., gzip, snappy, or lz4) on producers to save bandwidth and storage!
4. Control Data RetentionUse appropriate retention policies for topics based on your use case and regulatory needs.
You can retain data based on time (e.g., 7 days) or size (e.g., 10GB).
5. Avoid Large MessagesKafka is optimized for small-to-medium message sizes (under 1MB).
Because of this, if you need to transmit larger data, consider storing it elsewhere (e.g., in object storage) and passing a reference to the data (e.g., a URI) via Kafka.
6. Secure Your ClusterBasic security practices apply to everything: enable encryption (SSL) and authentication (SASL) to secure data in transit!
In addition, you can use Access Control Lists (ACLs) to restrict topic access, same as any other UAC tool!
7. Implement Idempotency and RetriesEnable idempotent producers (fancy way of saying “unique value producers”) to avoid duplicate records.
Set up appropriate retry strategies to ensure your consumers don’t get overloaded after a server hiccup causes a bunch of data to try and replay itself.
8. Use Kafka Connect and Kafka Streams When AppropriateUse Kafka Connect to integrate with external data sources to your Kafka workflow.
Use Kafka Streams or ksqlDB for real-time stream processing!
9. Test at ScaleAlways benchmark and test Kafka in development at the scale you expect in production. Another universal best practice, but one you’d be surprised how many people don’t follow!
In a similar vein, simulate real workloads to uncover bottlenecks before going live.
Conclusion

Apache Kafka has become a cornerstone technology for building data streaming pipelines. By understanding its architecture and following best practices, you can ensure your Kafka deployment is running as well as the best of them!

Further Reading:

Apache Kafka Documentation
Kafka Streams
Kafka Connect

The post Why Apache Kafka is the Best Tool for Data Streaming appeared first on Jacob Robinson.