When Prometheus starts consuming multiple CPU cores just to scrape basic metrics, something's gone wrong. The monitoring system that's supposed to be invisible starts slowing down your entire infrastructure stack.
Most Prometheus CPU problems boil down to three issues: queries that scan too much data, metrics with explosive label combinations, or scraping configurations that haven't been tuned for your actual workload. Each one creates a different CPU usage pattern, and each needs a different fix.
What Causes Prometheus CPU Spikes
Prometheus processes data in predictable ways, which means CPU spikes have identifiable causes. Understanding these patterns helps you fix the right problem instead of guessing.
Query Engine Load
The query engine does most of the heavy lifting. When you run a query like rate(http_requests_total[5m]), Prometheus loads five minutes of data points for that metric, calculates rates between consecutive points, and returns the results. Simple enough for a few hundred time series.
But scale that up, and the problems become clear:
• Complex calculations amplify CPU usage - A histogram quantile calculation across 100 microservices means loading data for potentially thousands of time series, performing mathematical operations on each bucket, then computing percentiles across the entire dataset. The CPU cost grows exponentially with the number of series involved.
• Dashboard refreshes create continuous load - If your main operational dashboard has 20 panels and refreshes every 10 seconds, that's 120 queries per minute hitting your Prometheus instance continuously. Each query might seem fast individually, but the aggregate load adds up quickly.
High-Cardinality Metrics
High-cardinality metrics create the most dramatic CPU spikes. Every unique combination of label values creates a separate time series that Prometheus must store and process. A metric tracking API response times with labels for endpoint, method, status code, and user ID can explode into millions of individual series if you're not careful about which labels you include.
Ingestion Pipeline Impact
The ingestion pipeline also consumes CPU proportional to how much data you're collecting. Each scrape target requires an HTTP request, response parsing, and updating the internal time series database.
• Scraping frequency matters - Scraping 1,000 targets every 15 seconds means 4,000 HTTP operations per minute, plus all the associated data processing.
• Target count scales linearly - More targets mean more HTTP connections, more parsing work, and more database updates happening simultaneously.
Diagnosing CPU Usage with Built-in Metrics
Before optimizing anything, you need to see where Prometheus is spending CPU time. The built-in metrics tell you most of what you need to know.
Start with overall CPU usage. The process_cpu_seconds_total metric shows cumulative CPU time, but you want the rate of change to see current usage:
rate(process_cpu_seconds_total[5m]) * 100This gives you CPU utilization as a percentage. Values consistently above 70-80% indicate you're approaching the limits of what your current setup can handle.
Query performance shows up in the prometheus_engine_query_duration_seconds histogram. Look at the 95th percentile to see how long your slowest queries take:
histogram_quantile(0.95, rate(prometheus_engine_query_duration_seconds_bucket[5m]))Queries taking more than a second regularly suggest either complex operations or too much data being processed.
The number of active time series directly impacts CPU usage because every query operation scales with the series count. Check prometheus_tsdb_head_series to see how many series Prometheus is currently tracking. Sudden growth in this metric often correlates with CPU spikes.
Memory pressure affects the CPU through garbage collection overhead. Monitor process_resident_memory_bytes and rate(go_gc_duration_seconds_count[5m]) to see if memory constraints are forcing frequent garbage collection cycles that consume CPU.
Set up alerts for these patterns so you catch problems before they become outages:
- alert: PrometheusCPUHigh
expr: rate(process_cpu_seconds_total[5m]) * 100 > 80
for: 5m
- alert: PrometheusSlowQueries
expr: histogram_quantile(0.95, rate(prometheus_engine_query_duration_seconds_bucket[5m])) > 1
for: 2mHow to Optimize Expensive Queries
Query optimization usually provides the biggest CPU reduction with the least operational complexity. Most expensive queries fall into predictable patterns that you can fix systematically.
Enable Query Logging
Enable query logging first so you can see exactly which queries consume the most resources:
./prometheus --query.log-file=/tmp/prometheus-queries.logThe log file shows execution time for each query, making it easy to identify the worst offenders. Look for queries that consistently take more than 500ms or queries that run very frequently.
Optimize Range Vector Size
Range vector size has a huge impact on query cost. A query like rate(metric[1h]) processes 12 times more data points than rate(metric[5m]) if you're scraping every 15 seconds. Reduce the time range when you don't need the longer historical context.
Use Recording Rules for Expensive Operations
Histogram quantile calculations are particularly expensive because they scan every bucket for every series, then perform statistical calculations across the entire dataset. If you're running the same histogram quantile query repeatedly, precompute it with a recording rule:
groups:
- name: latency_rules
interval: 30s
rules:
- record: http_request_duration_95th_percentile
expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))Now your dashboards can query http_request_duration_95th_percentile instead of recalculating the expensive histogram operation every time the dashboard refreshes.
Optimize Aggregation Order
Aggregation order matters for complex queries. Prometheus processes operations left-to-right, so putting the most selective filters first reduces the amount of data processed in subsequent steps. Instead of sum(rate(metric[5m])) by (service) > 0.1, use sum(rate(metric{service="important"}[5m])) by (service) > 0.1 when you only care about specific services.
How to Manage Cardinality to Reduce CPU Load
Cardinality problems are harder to spot but easier to fix once you identify them. A single high-cardinality metric can consume more resources than hundreds of well-designed metrics.
Identify High-Cardinality Metrics
Find your highest-cardinality metrics by counting series per metric name:
topk(10, count by (__name__)({__name__=~".+"}))Any metric with more than 10,000 series deserves investigation. Metrics with hundreds of thousands or millions of series are almost certainly problematic.
Common Cardinality Issues
The most common cardinality issue is including user IDs, request IDs, or other high-variation identifiers as label values. These labels create a new time series for every unique value, which grows without bound as your system processes more users or requests.
Fix Cardinality at Ingestion Time
Remove problematic labels at ingestion time using metric relabeling:
scrape_configs:
- job_name: 'web-servers'
static_configs:
- targets: ['web1:8080', 'web2:8080']
metric_relabel_configs:
- source_labels: [__name__]
regex: 'http_request_duration_seconds'
target_label: user_id
replacement: ''This configuration removes the user_id label from the http_request_duration_seconds metric before storing it, preventing the cardinality explosion while keeping the metric otherwise intact.
Drop Unnecessary Metrics
For debugging and development metrics that you don't need in production, drop them entirely:
metric_relabel_configs:
- source_labels: [__name__]
regex: 'debug_.*'
action: dropUse Histogram Buckets Instead
Instead of high-cardinality labels, use histogram buckets to track distributions. Rather than creating individual metrics for each user's response time, use histogram buckets to show the overall response time distribution across all users.
How to Optimize Scrape Configuration to Minimize CPU Overhead
Scraping inefficiencies are often overlooked but can significantly impact CPU usage, especially in large deployments with hundreds or thousands of targets.
Adjust Scrape Intervals Based on Metric Importance
Not every metric needs the same collection frequency. Critical application metrics might need 15-second resolution, but infrastructure metrics like disk usage can be collected every minute without losing important information:
# High-frequency for critical services
- job_name: 'api-servers'
scrape_interval: 15s
static_configs:
- targets: ['api1:8080', 'api2:8080']
# Lower frequency for infrastructure
- job_name: 'node-exporters'
scrape_interval: 60s
static_configs:
- targets: ['node1:9100', 'node2:9100']Configure Appropriate Scrape Timeouts
Scrape timeouts need to balance reliability with performance. Setting timeouts too low causes failed scrapes and missing data. Setting them too high means slow targets can block scrape processing. A good rule of thumb is timeout = scrape_interval - 5 seconds.
Optimize Service Discovery Configuration
Service discovery can create CPU overhead if configured too broadly. Instead of discovering every pod in your Kubernetes cluster, limit discovery to specific namespaces or use label selectors to only find targets that actually expose metrics:
kubernetes_sd_configs:
- role: pod
namespaces:
names: ['production', 'staging']
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: trueThis configuration only discovers pods in production and staging namespaces that have the prometheus.io/scrape annotation, reducing the overhead of processing every pod in the cluster.
Scaling Prometheus When Optimization Isn't Enough
Sometimes query optimization and cardinality management aren't sufficient, especially for large-scale deployments. That's when you need to consider scaling strategies.
Horizontal Scaling Through Multiple Instances
Horizontal scaling means running multiple Prometheus instances, each handling a subset of your monitoring data.
The simplest approach is functional sharding - dedicating different Prometheus instances to different parts of your infrastructure. One instance monitors frontend services, another handles backend APIs, and a third watches your data pipeline.
For advanced horizontal scaling, consider Thanos which provides a global query view across multiple Prometheus instances, or Cortex for horizontally scalable Prometheus-as-a-Service.
Federation for Multi-Level Monitoring
Federation ties multiple Prometheus instances together by having a higher-level instance scrape summary metrics from the lower-level instances. This gives you both detailed metrics for troubleshooting and aggregated views for high-level monitoring:
# Federation configuration on the aggregation server
scrape_configs:
- job_name: 'federate'
scrape_interval: 15s
honor_labels: true
metrics_path: '/federate'
params:
'match[]':
- '{job=~"frontend-.*"}'
- '{job=~"backend-.*"}'
static_configs:
- targets:
- 'prometheus-frontend:9090'
- 'prometheus-backend:9090'Vertical Scaling Resource Guidelines
Vertical scaling focuses on giving your existing Prometheus instance more resources. CPU scaling generally follows the pattern of one core per 100,000 active time series, but complex queries and high query concurrency require additional cores. Memory scaling depends heavily on cardinality, with roughly 1-3 bytes per sample stored.
Use Grafana dashboards to monitor resource usage and AlertManager to get notified when scaling is needed.
Storage Performance Impact on CPU
Storage performance impacts CPU usage because slow disk operations force Prometheus to wait during query processing. SSDs provide much better query performance than spinning disks, and NVMe drives are even better for write-heavy workloads with high ingestion rates.
Monitor storage performance with Node Exporter metrics to identify bottlenecks.
Remote Storage for Long-Term Efficiency
Remote storage offloads long-term data to external systems, keeping only recent data locally for fast queries. This reduces local disk usage and can improve query performance for recent data while maintaining long-term retention in scalable storage systems.
Popular remote storage options include Thanos, Cortex, VictoriaMetrics, and cloud-native solutions like Amazon Managed Service for Prometheus or Google Cloud Managed Service for Prometheus.
How to Track Optimization Success with Key Performance Metrics
Optimization efforts need quantifiable results to know whether you're moving in the right direction. Track these metrics before and after making changes.
CPU Efficiency Measurement
CPU efficiency shows how much work you're getting per CPU cycle:
rate(prometheus_engine_queries[5m]) / rate(process_cpu_seconds_total[5m])Higher values mean you're processing more queries per unit of CPU time, indicating better efficiency.
Query Performance Tracking
Query performance improvements show up in average query duration:
rate(prometheus_engine_query_duration_seconds_sum[5m]) / rate(prometheus_engine_query_duration_seconds_count[5m])Monitor this metric with Grafana dashboards specifically designed for Prometheus monitoring.
Memory Efficiency Analysis
Memory efficiency indicates how many time series you can handle per gigabyte of RAM:
prometheus_tsdb_head_series / (process_resident_memory_bytes / 1024 / 1024 / 1024)Use Prometheus Node Exporter to get detailed memory usage metrics for deeper analysis.
Performance Benchmarks and Targets
A well-optimized Prometheus setup should achieve CPU usage below 70% during normal operations, 95th percentile query response times under 500ms, and scrape success rates above 99.5%. These targets provide headroom for traffic spikes while maintaining reliable monitoring.
Set up AlertManager or Last9 Alert Studio alerts to notify you when these thresholds are exceeded, and use Grafana to create dashboards that track these key metrics over time for trend analysis.
How to Diagnose and Fix Common CPU Performance Issues
Dashboard Performance Problems
Dashboard performance problems usually manifest as slow loading times and CPU spikes when users access monitoring interfaces.
Check if multiple users are running the same expensive queries simultaneously. Increase dashboard refresh intervals to 30-60 seconds and use recording rules for the most complex dashboard queries.
Use Grafana's query inspector to identify slow queries and consider Grafana Enterprise for query caching capabilities.
Sudden Cardinality Explosions
Sudden cardinality explosions often happen when new code deploys introduce high-variation labels or when existing labels start taking on many more values.
Monitor the rate of change in prometheus_tsdb_head_series to catch these problems quickly. Have emergency metric drop configurations ready to deploy when cardinality issues threaten system stability.
Use tools like PromLens to analyze query performance and Prometheus Operator for automated configuration management during emergencies.
Memory-Related CPU Spikes
Memory-related CPU spikes occur when garbage collection becomes frequent due to memory pressure. Monitor go_memstats_gc_cpu_fraction to see what percentage of CPU time is spent on garbage collection. Values above 0.05 (5%) indicate memory pressure is impacting CPU performance.
Track these metrics with Node Exporter and set up alerts using AlertManager when GC overhead becomes problematic.
Scrape Failure Cascade Effects
Scrape failures can cascade into CPU problems when Prometheus retries failed scrapes or when timeout configurations cause scrapes to queue up. Monitor up metrics and scrape duration histograms to identify problematic targets before they impact overall system performance.
Use Blackbox Exporter to monitor target availability and Prometheus Pushgateway for batch jobs that might be causing scrape issues.
Final Thoughts
Optimizing Prometheus starts with fixing data flow — ingestion, storage, and query execution.
the
Most CPU problems aren’t about hardware limits. They’re about unnecessary work: inefficient queries, high cardinality, and scraping too much, too often. Clean that up, and the CPU drops without losing signal.
But tuning and scaling Prometheus well takes real time and expertise. At Last9, we take that off your plate, no black boxes, no vendor lock-in. Just fast, reliable Prometheus, without the operational drag.
FAQs
How much CPU does Prometheus need?
CPU requirements depend on your workload, but a general baseline is 1 CPU core per 100,000 active time series. A typical setup monitoring a 50,000 series should run comfortably on 1-2 CPU cores. Complex queries, high query concurrency, and frequent dashboard refreshes increase CPU needs significantly. Monitor your actual usage with rate(process_cpu_seconds_total[5m]) * 100 and scale when sustained usage exceeds 70%.
How do you get the CPU usage percentage in Prometheus?
Use this query to get Prometheus's own CPU usage as a percentage:
rate(process_cpu_seconds_total[5m]) * 100For monitoring other applications, use the same pattern with their process_cpu_seconds_total metric. The rate function calculates CPU time consumed per second over a 5-minute window, multiplied by 100 to get a percentage.
Why is 70% of my CPU being used?
High CPU usage usually indicates one of three issues: expensive queries scanning large datasets, high-cardinality metrics creating millions of time series, or aggressive scraping configurations. Check your slowest queries with histogram_quantile(0.95, rate(prometheus_engine_query_duration_seconds_bucket[5m])) and identify high-cardinality metrics with topk(10, count by (__name__)({__name__=~".+"}). Queries taking over 1 second or metrics with over 10,000 series are common culprits.
What does 400% CPU usage mean?
CPU usage above 100% indicates your process is using multiple CPU cores. 400% means roughly 4 CPU cores are fully utilized. This is normal for multi-threaded applications like Prometheus, which can process queries in parallel across multiple cores. However, sustained high multi-core usage suggests you need optimization or more resources.
How can I query CPU usage with Prometheus?
For system-wide CPU usage from node_exporter:
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)For container CPU usage:
rate(container_cpu_usage_seconds_total[5m]) * 100For application-specific CPU usage:
rate(process_cpu_seconds_total{job="your-app"}[5m]) * 100How can I correlate CPU usage with application performance?
Combine CPU metrics with application performance indicators. For example, plot CPU usage alongside request latency:
# CPU usage
rate(process_cpu_seconds_total{job="api-server"}[5m]) * 100
# Request latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job="api-server"}[5m]))Look for patterns where CPU spikes correlate with increased latency or error rates. This helps identify whether CPU constraints are affecting user experience.
What metrics does Prometheus provide?
Prometheus exposes comprehensive metrics about its own operation:
prometheus_tsdb_head_series- active time series countprometheus_engine_query_duration_seconds- query execution timesprometheus_tsdb_compactions_total- database compaction activityprometheus_config_last_reload_successful- configuration reload statusprometheus_notifications_total- alerting statistics
These internal metrics are crucial for monitoring Prometheus health and performance.
Why monitor Kubernetes pod resources?
Kubernetes pods can consume resources unpredictably due to application bugs, traffic spikes, or resource limit misconfigurations. Monitoring pod CPU usage helps with capacity planning, identifying resource-hungry applications, and setting appropriate resource requests and limits. Use container_cpu_usage_seconds_total to track per-container CPU consumption within pods.
How often is container_cpu_usage_seconds_total updated?
The container_cpu_usage_seconds_total metric updates as frequently as you scrape it, typically every 15-30 seconds. However, the underlying CPU accounting data from the kernel updates continuously. The metric represents cumulative CPU time, so you need to use rate() to see current usage: rate(container_cpu_usage_seconds_total[5m]).
Can Prometheus monitor CPU usage of individual containers within a pod?
Yes, container_cpu_usage_seconds_total includes labels that identify specific containers:
rate(container_cpu_usage_seconds_total{pod="my-pod", container="app-container"}[5m])This lets you see CPU usage for each container separately, which is essential for multi-container pods where different containers have different resource patterns.
How do I set up alerts for high CPU usage in Prometheus?
Create alerting rules in your Prometheus configuration:
groups:
- name: cpu_alerts
rules:
- alert: HighCPUUsage
expr: rate(process_cpu_seconds_total[5m]) * 100 > 80
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU usage detected"
description: "CPU usage is {{ $value }}% for 5 minutes"Configure Alertmanager to route these alerts to your notification system.
How do I set up Prometheus to monitor CPU usage?
Configure Prometheus to scrape targets that expose CPU metrics. For system monitoring, use node_exporter:
scrape_configs:
- job_name: 'node-exporter'
static_configs:
- targets: ['localhost:9100']For container monitoring in Kubernetes, scrape cAdvisor metrics:
- job_name: 'kubernetes-cadvisor'
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
