🏏 450 million fans watched the last IPL. What is 'Cricket Scale' for SREs? Know More

Sep 17th, ‘23/3 min read

Unwiring High Cardinality - SRE Day 2023

Report from SRE Day 2023, where Piyush Verma - CTO Last9, gave a talk on Unwiring High Cardinality

Unwiring High Cardinality - SRE Day 2023

Piyush Verma gave a talk at SRE Day 2023the on Unwiring High Cardinality. The conference was held in London on September 14-15.

The conference included talks on various talks, including real-time stream processing, running the SRE team's incident management, and how Thanos proved costly for specific organizations.

Piyush's talk was based on the principles of building Levitate - our managed time series data warehouse.

Here is the outline of the talk.

Observability relies on metrics as a crucial aspect, providing a cost-effective and speedy way to address SDLC and Software health queries, which can otherwise be challenging.
With metrics, inevitably, you hit High Cardinality problems. While searching for profound insights from the systems, we often face restrictions due to the cardinality limitations of the observability tools. But what makes high cardinality significant, and why is it an inevitable challenge when monitoring systems on a vast scale?

Piyush delved into the anatomy of a metric and issues that high cardinality can help resolve, from combating Noisy Neighbors to battling in the Streaming Wars and dealing with the pulse of High Cardinality.

High Cardinality? No Problem! Stream Aggregation FTW | Last9
High cardinality in time series data is challenging to manage. But it is necessary to unlock meaningful answers. Learn how streaming aggregations can rein in high cardinality using Levitate.

However, modern systems' limitations make cardinality an unsolved problem. To find the best solution for cardinality, it is crucial to understand the Metric Lifecycle. Lastly, Piyush defined the workflows that enable scaling cardinality to millions, not just thousands.

When software is in production, it's crucial to have telemetry and instrumentation to troubleshoot issues. Unfortunately, this can be a time-consuming and costly process. Often, we resort to using generic solutions that may not address all the unique needs of our specific system. This can lead to missed opportunities for improvement and wasted time looking for answers elsewhere. Most importantly, it will allow architects and engineering leaders to keep things SIMPLE and reach that 9 with much less pain.https://last9.io/blog/high-cardinality-no-problem-stream-aggregation-ftw/

Here is the YouTube video of the talk.

Here are blog posts showcasing how Levitate is built to tackle high cardinality.

How we tame high cardinality in time series databases | Last9
Engineering innovation to solve high cardinality with Levitate - a multi-part series
How we tame High Cardinality by Sharding a stream | Last9
Using ‘Sharding’ to tame High Cardinality data for Levitate - Our Time Series Data Warehouse
Streaming Aggregation vs Recording Rules | Last9
Streaming Aggregation and Recording Rules are two ways to tame High Cardinality. What are they? Why do we need them? How are they different?



Stay updated on the latest from Last9.



Last9 helps businesses gain insights into the Rube Goldberg of micro-services. Levitate - our managed time series data warehouse is built for scale, high cardinality, and long-term retention.

Handcrafted Related Posts