Preeti Dewani, Senior Software Engineer at Last9, is giving a talk at Rootconf SRE Conf 2023 in Bengaluru on Deep Dive into Analyzing High Cardinality Metrics — assuming Prometheus-like systems for reference.
What you’ll learn
- Why high cardinality is unavoidable — microservices with dimensions like tenant, region, and service push label combinations into the millions, and that’s a feature, not a bug.
- What unchecked cardinality actually breaks — resource consumption, slow dashboards, failing alert queries, blank observability windows, shrinking retention.
- Why “just drop a label” gives you a false sense of fixing it — corner cases where dropping the highest-cardinality label has zero impact on reduction, and what to do instead.
- A practical method (and tool) — how to find your TSDB’s cardinality limits, detect when you’re approaching them, identify which metrics and labels are contributing, and reason about what the system looks like after you act. Preeti will walk through an open-source cardinality debugger she’s been using in customer production environments.
If you’ve ever stared at a TSDB chewing through resources and not known where to start cutting, this one’s for you.
