To handle high cardinality in Prometheus, find the offending metrics first (the tsdb status endpoint and a couple of PromQL counts will name them), then cut series where they are created: drop or rewrite labels with metric_relabel_configs, pre-aggregate with recording rules, cap each scrape with sample_limit, and replace unbounded label values like user IDs with bounded buckets.
Cardinality is set at scrape time, when a new label combination creates a new series. That is where every fix in this post operates. Query-side tricks make dashboards faster but do not shrink the TSDB.
| Step | What to run | What it tells you |
|---|---|---|
| 1. Count series | curl http://localhost:9090/api/v1/status/tsdb | Total head series, top 10 metrics by series count |
| 2. Rank offenders | topk(10, count by (__name__)({__name__=~".+"})) | Which metric names own the most series |
| 3. Find the label | count(count by (label_x)(metric_name)) | How many unique values one label contributes |
| 4. Fix at scrape | metric_relabel_configs with labeldrop or value rewrites | Series stop being created |
| 5. Guard the door | sample_limit, label_limit per scrape job | A bad deploy fails its scrape instead of flooding the TSDB |
What is High Cardinality in Prometheus?
Prometheus label cardinality refers to the number of unique label value combinations in a given metric. Labels are key-value pairs attached to each data point, allowing fine-grained identification and categorization of metrics. The cardinality of a label is the number of distinct values it can take. Too many unique values per label, such as user_id or user_email, produce high cardinality.
For instance, in an e-commerce application, monitoring product sales is one metric that can easily cause high cardinality due to dimensions like region, product category, and product type, leading to thousands of unique combinations. But this metric does have use cases for understanding business for product and engineering teams. This is just one example of how high cardinality is a reality today for getting answers from the time series data.
High-label cardinality means that a metric has many unique label combinations. Each distinct combination creates a separate time series, so high cardinality increases memory use, CPU use, query times, and storage requirements.
Keeping label cardinality within reasonable limits is generally recommended to ensure efficient usage of Prometheus. While no specific threshold is defined as “high” cardinality, it is important to strike a balance between providing sufficient granularity for monitoring needs and avoiding excessive unique label combinations.
High cardinality is also a budget and availability problem, not just a performance one. When an instance hits its memory ceiling, teams respond by dropping metrics or shortening retention, and the data you wanted for debugging is the data that disappears. Scaling past the problem costs hardware and operational attention.
What Causes High Cardinality in Prometheus?
Unbounded labels cause most of it: user IDs, emails, session tokens, raw URL paths, container IDs, commit SHAs. Each new value mints a new time series. Kubernetes multiplies the effect, since every metric carries pod, namespace, and instance labels. A hundred pods scraped by one Prometheus instance turn one counter into a hundred series before any application label is added.
More data = more metrics = high cardinality.
Churn matters as much as scale. Deployments that recreate pods, autoscaling, and short-lived jobs all retire old label sets and create new ones, so head series climb even when traffic is flat.
What Are the Prometheus Cardinality Limits?
There is no fixed series limit in Prometheus. Memory is the real ceiling: each active series costs roughly 1 to 3 KiB in the head block, so a 16 GB instance runs comfortably at a few million active series. What you can enforce are per-scrape limits in scrape_configs.
The enforceable knobs: sample_limit fails any scrape returning more than N samples, label_limit caps labels per series (v2.27 and later), label_name_length_limit and label_value_length_limit cap label sizes, and target_limit caps discovered targets per job. When one trips, the whole scrape fails and the target reports up=0 for that interval, so alert on prometheus_target_scrapes_exceeded_sample_limit_total instead of finding out mid-incident.
Prometheus’ index-based storage system inverts the index on the labels and the bitmap of the matching time series. Therefore, the index will correspondingly grow for high-cardinality-prone metrics, increasing memory, cpu and disk utilization and slowing down queries.
Treat a sustained climb in prometheus_tsdb_head_series as the real alarm, and plan capacity around active series rather than samples per second.
How Do You Find High Cardinality Metrics?
Prometheus provides several methods for identifying high cardinality metrics. Here’s a general guide on how you can do it:
1. Using Prometheus UI:
Visit the Prometheus web interface, usually available at http://<your-prometheus-server-address>:9090. Use the following expressions to find high cardinality metrics:
- {__name__=~".+"}: It returns all series currently in memory. If the result is too high, it signifies high cardinality.
- count({__name__=~".+"}) by (__name__): It returns the count of series per metric name. Look for metrics with a significantly higher count than others.
- topk(10, count by (__name__, job)({__name__=~".+"})): It returns the top 10 highest series counts by metric name and job. It helps to identify the jobs that are producing high cardinality metrics.
- topk(10, count by (__name__, instance)({__name__=~".+"})): This is similar to the above but groups by instance. Useful to identify problematic instances.
2. Using the Prometheus Stats API:
Prometheus also exposes TSDB statistics directly. Hit http://<your-prometheus-server>:9090/api/v1/status/tsdb and read numSeries for the total, plus seriesCountByMetricName for the top 10 metrics by series count. The same data sits in the UI under Status > TSDB Status, which is the fastest first stop during an incident.
The usual incident shape: a deploy adds one label, say a request path with an ID in it, and head series triple overnight. Nobody notices until Prometheus is OOMKilled and restarts into a long WAL replay, during which dashboards sit blank and alerts stay silent. The fix took one labeldrop line; finding it at 2 AM took an hour. Run the tsdb check before the deploy, not after.
3. Using Grafana dashboards:
Using Grafana, you can import the official Prometheus Stats dashboard (ID: 1860) or Prometheus 2.0 Stats (ID: 3662) to visualize the total number of series and other valuable metrics. These dashboards provide insights into cardinality across all metrics stored in your Prometheus instance.
After identifying the high cardinality metrics, you may need to revise your metrics design or use techniques such as metric relabeling to drop unnecessary metrics or labels. Be aware that removing labels can cause loss of information, so be sure to evaluate the impact before making any changes.
How Do You Reduce Cardinality in Prometheus?
Work top down: take the two or three worst metrics from the tsdb check above and apply the cheapest fix that holds. In practice that means relabeling first, then aggregation, then scrape-level guards.
Metrics Relabeling
Relabeling in Prometheus is a powerful feature that allows you to modify or filter labels on metrics before they are stored or processed. By using relabeling, you can reduce the cardinality of your metrics, which can help improve performance and resource usage. Here are some relevant examples of relabeling techniques that can be used in Prometheus:
Dropping a label: labeldrop removes the label itself from every series in the job. Use it for labels that carry no analytical value, like internal IDs. Series that were unique only through that label collapse together:
metric_relabel_configs: - regex: unwanted_label action: labeldropKeeping only selected labels: labelkeep is the inverse. Everything not matching the regex is removed. Never let the regex eat __name__, job, or instance, and check dashboards before shipping either action, because both apply to every metric in the job:
metric_relabel_configs: - regex: (job|instance|method|status_class) action: labelkeepReplacing values with a constant works when the label must exist but its detail does not matter for charts. The series for each old value merge into one.
Bucketing values is the workhorse for paths and URLs. Rewrite unbounded values into a bounded set, such as collapsing every /api/users/<id> into one path:
metric_relabel_configs: - source_labels: [path] regex: "/api/users/.+" target_label: path replacement: "/api/users/:id"Regex capture groups let you keep just the meaningful part of a value, for example stripping the replica hash off a pod name so all replicas share one series set:
metric_relabel_configs: - source_labels: [pod] regex: "(.+)-[a-z0-9]+-[a-z0-9]{5}" target_label: workload replacement: "$1"One gotcha: relabel_configs and metric_relabel_configs are not interchangeable. The first runs against target metadata before the scrape, the second against every sample after it. Filtering samples in relabel_configs does nothing, and a heavy regex in metric_relabel_configs runs per sample per scrape, so a careless rule on a 50k-sample job burns visible CPU.
These are just a few examples of how relabeling can be used in Prometheus to reduce cardinality. Your specific relabeling configuration will depend on your specific use case and the nature of your metrics.
The relabeling guide covers every action type with worked examples: Mastering Prometheus Relabeling.
Aggregation
Aggregation is a method that involves combining the metric values of multiple time series to create a single new time series. This approach can significantly reduce the number of time series that Prometheus needs to store and handle, minimizing memory usage, CPU usage, and disk space requirements.
For instance, if you have a metric http_requests_total with two labels: method and status_code, you can use the following query to aggregate the metrics into a single time series:
sum by (method) (http_requests_total)This PromQL-based query sums up the http_requests_total metric by method, removing the status_code label from the result. This approach eliminates the need to store and manage status_code label values.
You can use recording rules and streaming aggregations to perform these aggregations.
Last9 Documentation 
Last9 streaming aggregation rolls up high cardinality series at ingest, before storage. Start for free.
Bucketing or Histogramming
Using bucketing or histogramming techniques for metrics with continuous values to group data into predefined ranges. This reduces cardinality by reducing the number of distinct values while providing insights into data distribution. Prometheus provides the histogram_quantile() function to query aggregated data from histogram metrics.
Rollup and downsampling
You can roll up or downsample data over time, depending on your long-term storage requirements. For example, you might decide to store high-resolution data for a short period and then downsample it to reduce cardinality and save storage space.
Tuning Prometheus server configs
Two different levers live in the server config. Interval settings like scrape_interval cut samples per series, which trims ingest volume and storage but leaves series count untouched. Scrape limits are the cardinality lever:
Set sample_limit on a job to make any scrape that returns more samples fail outright. It is a guard rail against a bad deploy, not a tuning knob: when it trips you lose that scrape, so pair it with an alert on prometheus_target_scrapes_exceeded_sample_limit_total.
scrape_configs: - job_name: my_job sample_limit: 5000 static_configs: - targets: ["my-target:9090"]Trim Histogram Buckets, or Go Native
Classic histograms multiply series: one histogram with 12 buckets is 12 series per label combination, plus _sum and _count. If a histogram’s precision is not earning its cost, drop unused le buckets with metric_relabel_configs, or shorten the bucket list in instrumentation:
metric_relabel_configs: - source_labels: [__name__, le] regex: "http_request_duration_seconds_bucket;(0.005|0.01|0.025)" action: dropNative histograms (experimental since Prometheus v2.40, with mature tooling in the 3.x line) change the math: the whole distribution lives in one series with dynamic buckets, so the per-bucket series explosion disappears. If histograms dominate your series count, they are the structural fix.
Use Efficient PromQL Query Patterns
Query patterns like topk and bottomk help reduce the number of data points queries return. Grafana and other visualization tools also enable the customization of smarter dashboards where you can filter data, highlight essential metrics, and identify teams and environments that contribute the most to Prometheus cardinality.
Optimize labels and tags
Enforce similar labels and values across related metrics. Organize tags into a hierarchy or categories to avoid confusion. ONLY define relevant metrics.
Control data frequency via DPM
Data frequency is a volume lever, not a cardinality one. The default 15s interval produces 4 samples per minute per series; widening it to 30s or 60s halves or quarters ingest and storage while series count stays the same. To see your actual rate:
count_over_time(scrape_samples_scraped[1m])Implement Appropriate Retention Policies
Determine at what granularity metrics will be stored using benchmarks like the purpose of the metric, the importance of the metric, legal and regulatory constraints, and costs. This helps ensure that the most important metrics are not lost due to storage limitations. Metrics that should only be retained briefly should not stay longer.
Learn more about Last9’s automatic data tiering.
Use Kubernetes Annotations
With Kubernetes, the cheapest series are the ones never scraped. Annotations control discovery: set prometheus.io/scrape: “false” on pods you do not chart, and point prometheus.io/path at a metrics endpoint that exposes only what you need.
The same idea extends to discovery-time relabeling: drop entire namespaces or workloads by matching __meta_kubernetes_namespace in relabel_configs before the scrape happens, so test and batch workloads never enter the TSDB at all.
Employ Horizontal Scaling
Use horizontal scaling in Kubernetes to spread the load of metrics collection and queries across multiple instances. You can use Prometheus Operator to manage the installation of the Prometheus stack in your Kubernetes cluster. Also known as federation or sharding, each instance collects metrics from a subset of the monitored targets and stores them locally. After that, queries and alerts are federated or sharded across all instances, each serving a subset of the workload. This approach is especially effective for large-scale, distributed systems where the workload is spread across geographically sparse locations.
FAQs
How do I find which metrics have the highest cardinality in Prometheus?
Hit /api/v1/status/tsdb on your Prometheus server; it lists total head series and the top 10 metric names by series count. For more detail, run topk(10, count by (__name__)({__name__=~”.+”})) in the expression browser, then count(count by (label)(metric)) to see which label is responsible.
What causes high cardinality in Prometheus?
Labels with unbounded values: user IDs, emails, session tokens, raw URL paths, container IDs, commit SHAs. Each new value creates a new time series. Kubernetes makes it worse by multiplying every metric by pod, namespace, and instance labels, so one careless label on a busy service can add millions of series.
How many time series can Prometheus handle?
There is no hard limit; memory is the constraint. Each active series costs roughly 1 to 3 KiB in the head block, so a single instance handles a few million active series on 16 GB of RAM. Watch prometheus_tsdb_head_series and plan to aggregate or shard well before memory pressure hits.
Does dropping labels with relabeling lose data?
Yes, permanently. Series are dropped or rewritten at scrape time, before storage, so there is no recovering them later. Check the label is unused in dashboards and alerts first, or pre-aggregate with recording rules so a coarser version of the data survives. Streaming aggregation keeps the rollup while skipping raw series storage.
What is the difference between recording rules and streaming aggregation?
Recording rules run on data Prometheus has already stored: you pay the cardinality cost first, then compute rollups on a schedule. Streaming aggregation computes the rollup at ingest, so the high cardinality raw series never lands in long-term storage. Rules are built into Prometheus; streaming aggregation needs a backend like Last9 that supports it.
Is high cardinality always a problem?
No. Per-customer or per-endpoint detail is often exactly what debugging needs. It becomes a problem when unbounded labels create series faster than your instance’s memory and query engine can absorb, which is a capacity decision, not a data-modeling sin. Decide which dimensions earn their cost; do not delete detail reflexively.
Conclusion
Run the tsdb status check today, before anything is on fire. In most setups two or three metrics own the bulk of the series, and one labeldrop plus a sample_limit guard buys months of headroom. Defaults are fine below about a million active series on a mid-sized instance; past that, pre-aggregate before storage or shard.
If the high cardinality detail is something you want to keep rather than drop, use a backend that aggregates at ingest. Last9 streams aggregates as data arrives and keeps the raw series queryable, so you decide what to roll up with the data in hand instead of guessing at instrumentation time.
Last9 Documentation 
