Streams for Everyone
If you have come this far it means you have already considered or are considering using event streaming in your data architecture for the wide variety of benefits it can offer. Or perhaps you are looking for something to support a Data Mesh initiative because that’s all the rage right now. In either case, both Amazon Kinesis and Apache Kafka can help but which one is the right fit for you and your goals. Let’s find out!
Real quick disclaimer, I currently work at Rockset but previously worked at Confluent, a company known for building Kafka based platforms and cloud services. My experience and understanding of Kafka is much deeper than Kinesis but I’ve made every attempt to provide a mostly unbiased comparison between the two for the purposes of this article.
Software or Service
Apache Kafka is Open Source Software, governed by the Apache Software Foundation and licensed under Apache License Version 2.0. You can look at the source code, deploy it wherever you want and even fork the source code, create a new product and sell it! Amazon Kinesis is a fully managed service available on AWS. The source code is not available and that’s ok, no one’s judging KFC for keeping their recipe secret. In terms of software deployment and management strategies, Kafka and Kinesis could not be more different. This fundamental difference between software and service makes them interesting to compare since Kinesis has no true Open Source alternative and Kafka has several non-AWS managed service options including Aiven, Instaclustr and Confluent Cloud. This inevitably makes Kafka the more flexible option between the two if hedging against an AWS-only architecture.
Accessible or Convenient
As with many Open Source projects, Kafka gained popularity by being easily accessible to an audience of engineers and developers who had enough hardware to solve their problem but couldn’t find the right software. On the other hand, Kinesis has become one of the top cloud-native streaming services largely based on its convenience and low barrier to entry, especially for existing AWS customers. For the most part these aspects have continued for both parties and you can find lots of different variations of Kafka with a vast and varied ecosystem. While Kinesis remains land locked in the AWS ecosystem, it is still extremely easy to get started with and has tight coupling with several key AWS services like S3 and Lambda. Services like Confluent Cloud and AWS Managed Streaming for Kafka (MSK) are attempts at increasing the convenience of Kafka in the cloud (Confluent Cloud being the most mature option) but compared to Kinesis, they are still works in progress.
Architect or Developer
As with any evaluation we should also consider our audience. For an architect looking at the big picture, Kafka often seems attractive for both its flexibility and industry adoption. The Kafka API is so pervasive even other cloud-native messaging services have adopted it (see Azure Event Hubs). Although as a developer one may be forced into a more tactical decision in need of a well known outcome that makes Kinesis an obvious choice. Kinesis also has a developer-friendly REST-based API and several language specific client libraries. Kafka also has many language specific libraries in the community but officially only supports Java. In other words, if you are reading this article and you need to make a decision tomorrow, that might be too soon to consider a strategic platform like Kafka. If you already have an AWS account, you could have a highly scalable event streaming service today with Kinesis.
Vast or Fast
Performance in a streaming context is often about two things: latency and throughput. Latency being how quickly data gets from one end of the pipe to the other and throughput being how big (think circumference) the pipe is. In general, both Kafka and Kinesis are designed for low-latency and high-throughput workloads and there are plenty of realistic examples out there if you care to search for them. So they are both fast but the real difference in performance between the two comes from a concept called fanout. Since its inception Kafka was designed for very high fanout, write an event once and read it many, many times. Kinesis has the ability to fanout messages but it makes very specific and well-known limits about fanout and consumption rates. A fanout ratio of 5x or less is usually acceptable for Kinesis but I would look to Kafka for anything higher.
Partitions or Shards
In order to achieve scalability both Kafka and Kinesis split data up into isolated units of parallelism. Kafka calls these partitions and Kinesis calls them shards but conceptually they are equivalent in their nature to allow for higher levels of throughput performance. Both have documented limits around the maximum number of partitions and shards but those are changing often enough that it’s more relevant to think about per unit numbers. For information about per partition throughput we have to look at Confluent Cloud documentation as there is no standard for Kafka. In this case Confluent Cloud provides a max 10MB/s write and max 30MB/s read per partition. Kinesis documentation has a clearer but lower number per shard at 1MB/s write and 2MB/s read. This doesn’t inherently mean that partitions are better than shards but when thinking about your capacity needs and costs, it’s important to start with how many of these units of parallelism you are going to need in order to meet your requirements.
Secured or Safe
Kafka and Kinesis both have similar security features like TLS encryption, disk encryption, ACLs and client allow lists. Unfortunately for Kafka it is the lack of enforcement of these features that comes as a detriment. Unless you are using Confluent Cloud, Kafka has these features as options while Kinesis for the most part mandates them. That gives Kinesis a big security advantage and like many other AWS services, it integrates very well with existing AWS IAM roles, making security quick and painless. And if you are thinking, well I don’t need all of those things because I am self managing Kafka in my private network then you need to stop reading this and go read about Zero Trust. For those returning from their Zero Trust update and the rest of us, the bottom line is that both Kafka and Kinesis can be secured but it’s Kinesis and other managed cloud services that are inherently safer as it is part of their cloud rigor.
Summary
Here’s a quick table that summarizes some of the discussion from above.
If you forced me to choose between Kafka or Kinesis, I would choose Kafka every day and twice on Sunday. The reason being that as someone who is more of an architect, I am looking at the big picture. I might be choosing an enterprise standard event store where I need to separate the choice of Cloud provider from my choice for a common data exchange API. Of course, in the absence of competing managed services for Kafka and an existing AWS account I would probably lean towards Kinesis to improve my time to market and lower operational burden. The context of the situation matters more than the feature set of each technology. Everyone has a unique and interesting situation and I hope with some insights from this article, some second opinions and hands-on experience, you can make a decision that is best for you. I don’t think you’ll be disappointed in either case as both technologies have stood the test of time, likely only to be supplanted by something totally new that none of us have heard of yet (just ask JMS).
Rockset is the leading real-time analytics platform built for the cloud, delivering fast analytics on real-time data with surprising efficiency. Rockset provides built-in connectors to both Kafka and Kinesis, so users can build user-facing analytics on streaming data quickly and affordably. Learn more at rockset.com.