More and more applications, with more and more microservices and back-ends, deployed over more and more clouds, generating more and more logs — this is what developers have to deal with every day, and it makes observability harder and harder.
If you think monitoring and debugging applications in this setting is hard, you’re right. But then, of course, we have observability, which is not monitoring, but everyone agrees is important.
Confused yet?
Observability definitions abound. A simple one is “the ability to measure a system’s current state based on the data it generates, such as logs, metrics, and traces”.
Fluentd and Fluent Bit are widely-used open source projects under the auspices of the CNCF, focusing on the logs part of the equation. They work in tandem to collect and process, respectively, information stored in application and service logs.
Combined, Fluentd and Fluent Bit have more than 30K GitHub stars, 2M deployments per day, and 1,300 active contributors. They have been deployed over a billion times and are embedded into industry standard technologies such as Kubernetes and Red Hat OpenShift.
Today, Fluentd and Fluent Bit maintainer Eduardo Silva and former product manager Anurag Gupta announce the launch of Calyptia, a company formed to commercialize Fluent Bit, as well as a $5 million seed round led by Sierra Ventures and Carbide Ventures.
We caught up with Gupta and Silva to discuss Calyptia’s offering and its positioning in the observability landscape.
Contextualizing logs
As Cindy Sridharan, author of “Distributed Systems Observability,” notes: logs, metrics, and traces are often known as the three pillars of observability. While plainly having access to logs, metrics, and traces doesn’t necessarily make systems more observable, these are powerful tools that, if understood well, can unlock the ability to build better systems.
Fluentd is an open source data collector for unified logging layer, enabling users to unify data collection and consumption for a better use and understanding of data. Fluent Bit is an open source logging and metrics processor and forwarder, and it’s what Calyptia’s offering is built around.
As Silva puts it, Fluentd and Fluent Bit are “cousins” of projects such as Jaeger and Zipkin, which focus on traces. Logs, Silva adds, are a well-known mechanism to enable some kind of visibility into what an applications does.
When it’s time to analyze how an application is behaving, the first place to look is application logs. But in a distributed system setting, such as Kubernetes, that’s not trivial.
“You have a handful of applications, and you do want to perform data analysis to see how the applications are behaving. You want to take a look at the logs, but in order to solve that, you cannot go to each one of the applications, every single file, and take a look at what this application is doing. You need to have a specialized tool,” Silva says.
“So Fluentd and Fluent Bit are this specialized tool that can collect this log information, process it, and centralize it in one or multiple endpoints for analysis,” Silva adds.
Observability is all about logs, metrics, and traces. First-Mile Observability is about starting where data gets generated, contextualized, processed and routed, as per Calyptia Getty Images/iStockphoto
The power of the combo comes from the fact that it does not just collect and forward data stored in logs, but it also applies transformation and parsing logic, Gupta explained. This can be done using things such as Lua scripting or adding specific key value pairs in the Kubernetes space.
What Fluent Bit and Fluentd have been doing for the past ten and five years, respectively, is adding the ability to talk to APIs within Kubernetes and reach all that data. This helps contextualize the data in logs and make it more meaningful, so diagnosing and debugging become faster and easier.
A key advantage of working with Fluentd and Fluent Bit is the ability to work with multiple back-ends, Gupta notes. The tools are vendor agnostic when it comes to back-ends, so data can be routed wherever it needs to go, regardless of what they are or how many there are.
Even though back-ends are proliferating, Fluentd and Fluent Bit will be keeping up, Gupta adds.
First-Mile Observability
Gupta and Silva, both with long-standing experience in software systems engineering and stints at Microsoft and Oracle, respectively, met in 2017. They worked closely together on Fluentd, and as time went by, they saw the dynamics and decided to found Calyptia to offer commercial services around it in 2020.
Calyptia Cloud is a key part of the Calyptia Enterprise offering, and it includes two things: a management and monitoring layer, and developer tools. The developer tools are aimed at people who just want to use the open source layer, and the tools can help them figure out key aspects, such as regular expressions or whether or not things are working properly, Gupta says.
The management and monitoring offering aggregates all the information that Fluentd and Fluent Bit have at the edge layer, contextualizes, and visualizes it. This can help answer questions such as: Where is the data routed to? How much data is routed to which IPs? Why did the data choose a specific path? Why was this volume of data sent?
The goal is to also be able to have predictive analytics. In this case, this means being able to surface such insights ahead of the data landing into that back-end and then getting billed for it, as Gupta says. As for Calyptia Cloud’s own back-end, the service is also agnostic, being able to operate with anything ranging from New Relic and Splunk to Amazon and Google object storage or PostgreSQL.
Observability can be seen as a journey, as per Gupta. People can start with simple monitoring, then evolve, and start adding more data sources. The journey starts at the place where the data is created and continues towards the place where debugging, diagnosing, and getting insights happens.
Calyptia sees the place where data gets generated, contextualized, processed, and routed as the first mile in that journey. This, Gupta says, is where Fluentd and Fluent Bit have been operating, and this is also what Calyptia is aiming at.
Silva on his part describes observability as a continuum, with Calyptia addressing the data collection and pre-processing end of the spectrum and tools, such as Grafana, addressing the visualization end. Calyptia works side-by-side and integrates with all of them, he adds.
That may be so, but it does look like at least some overlap does exist.
Gupta and Silva spent a lot of time on calls with people from the likes of AWS, Google Cloud, and Microsoft, as well as enterprises looking to solve observability at scale and open source project contributors. This helped shape their offering. Fluentd and Fluent Bit are used at places like Fidelity and Neiman Marcus, and Calyptia would not be unlikely to have a foot at the door too.
In addition to Calyptia Cloud, Calyptia Enteprise can be self-hosted on any environment. Calyptia offers a subscription model, as well as consulting services and support. The company aims to use the funding to increase its current headcount of 16 and invest in the continued development of its software.