🏏 450 million fans watched the last IPL. What is 'Cricket Scale' for SREs? Know More

Apr 11th, ‘23/Changelog

Tame High Cardinality Data using Streaming Aggregation

Levitate now allows streaming aggregation of metrics to deal with near infinite high cardinality metrics. Define the streaming aggregations rules using PromQL.

High Cardinality data is almost reality in today's cloud native environment world. Levitate has superior defaults for managing high cardinality.

Quota Type Limit Reset Period Possible Actions
Per Time Series Cardinality 1,000,000 Per Hour Can be Raised on Request
Per Time Series Cardinality 20,000,000 Per Day Can be Raised on Request
💪🏻 Managing High Cardinality
This document explains how Levitate provides visibility, superior defaults and control levers to tame high cardinality

Even with these defaults, the metrics data may overflow. Levitate supports advanced features such as streaming aggregations to tackle such cases.

Streaming aggregation has been a long-standing concept in the data processing world, providing new observability capabilities that may have otherwise been avoided due to concerns about performance or resource requirements to store the necessary data. This powerful concept can significantly boost SLO queries, resulting in enhanced customer experiences. Furthermore, it can effectively monitor system performance in real time and detect anomalous components of the infrastructure without requiring large amounts of resources behind the query and storage layers.

Levitate now supports managing high cardinality data using PromQL based streaming aggregations.

Here is a walkthrough on how to start using streaming aggregations today with Levitate.

🚿 Streaming Aggregation
This document explains how to setup streaming aggregation pipelines in Levitate.

We have also written a detailed blog post about the technical details and scenarios in which streaming aggregations can be used.

High Cardinality? No Problem! Stream Aggregation FTW
High cardinality in time series data is challenging to manage. But it is necessary to unlock meaningful answers. Learn how streaming aggregations can rein in high cardinality using Levitate.