Troubleshooting Common Prometheus Issues: Cardinality & More

Prometheus is the de-facto open-source monitoring powerhouse, but it's not without quirks. However, it's not immune to high cardinality, resource exhaustion, and storage bottlenecks. From the dreaded OOMKilled message to the complexities of managing the Write-Ahead Log (WAL), understanding the problems with Prometheus is crucial for maintaining a robust monitoring system. This guide will walk you through some of the most common issues, including cardinality, resource utilization, and storage, providing practical solutions to keep your Prometheus in top shape.

Cardinality Woes: Symptoms to look for

High cardinality is a frequent challenge in Prometheus, often revealed by error messages like query processing would load too many samples into memory. This problem arises when too many unique combinations of labels and metrics can bog down your system. High cardinality in Prometheus can lead to many issues, from sluggish performance to complete system unresponsiveness.

To tackle this, identify which metrics have the highest cardinality and re-evaluate their necessity. You can use Prometheus's topk() function to determine which metrics are creating the most time series.

Using Prometheus topk()

topk(10, count by (__name__, job)({__name__=~".+"})): It returns the top 10 highest series counts by metric name and job. It helps to identify the jobs that are producing high cardinality metrics.

Upon pinpointing metrics with high cardinality, it may be necessary to reevaluate your metrics schema or implement strategies like metric relabeling to eliminate unused metrics or labels. However, caution is advised as label removal could lead to information loss. Ensure to thoroughly assess the potential consequences before executing modifications.

We have written an extensive guide on using multiple techniques like relabeling, aggregation, and dropping labels to manage high cardinality metrics in Prometheus.

Prometheus Scraping Issues

Scraping issues can be subtle, often only revealing themselves through gaps in data or alerts not firing as expected. Scraping problems can manifest as missing data or gaps in graphs. These can be due to misconfigurations, network issues, or overloaded targets. Prometheus provides metrics like prometheus_target_scrape_pool_sync_total and up to help diagnose scraping problems.

To resolve scraping issues, verify your scrape configurations and ensure your targets are reachable and not overloaded. If you're encountering high scrape series added counts, investigate the target's exported metrics to ensure they're not generating excessive time series due to high cardinality or misconfigurations.

High Memory and CPU Usage in Prometheus

Prometheus can be resource-intensive, mainly when dealing with many metrics or high ingestion rates. You might see your system's memory usage spiking or receive alerts like container_memory_usage_bytes that indicate Prometheus is using more memory than allocated. Similarly, high CPU usage might be flagged by process_cpu_seconds_total increasing rapidly.

To mitigate high resource usage, consider optimizing your scrape intervals, reducing the number of active time series, or increasing the resources allocated to Prometheus. In some cases, implementing a more efficient data model or adjusting query patterns can also help reduce the load.

When Prometheus Falls Over: OOM Killed

The dreaded OOMKilled message indicates that Prometheus has run out of memory, and the kernel has terminated the process to stabilize the system. This can result from high cardinality, resource misconfiguration, or simply underestimating the memory requirements of your Prometheus instance.

To prevent OOM kills, monitor your memory usage closely and set appropriate limits and requests if running in a containerized environment. It's also wise to implement alerting based on memory consumption trends so you can take action before an OOM kill occurs.

Navigating Compaction and Retention Hurdles

Compaction and retention are essential for managing Prometheus's local storage. However, issues can arise, leading to error messages such as compaction failed or not enough disk space. These errors can cause Prometheus to halt data ingestion, leading to gaps in monitoring data.

To address these issues, ensure that your retention policies are set according to your storage capacity and that you monitor disk space usage to prevent running out of space. Adjusting the compaction settings may also be necessary if you're seeing frequent compaction failures.

Prometheus Disk, Storage, and WAL Issues

The Write-Ahead Log (WAL) is a critical component for ensuring data integrity but can also be a source of issues. Error messages like WAL corruption or storage needs too many chunks indicate problems with the storage layer that can lead to data loss or unresponsiveness.

Regularly back up your Prometheus data and monitor the WAL size to ensure it's within expected bounds. If you're consistently encountering storage-related errors, it may be time to scale up your storage capacity or look into long-term storage solutions that can handle the load.

Crash Loop Back-Off

A crash loop occurs when Prometheus repeatedly crashes and restarts, often due to configuration errors or corrupted data. The error message CrashLoopBackOffin Kubernetes is a standard indicator of this issue.

The status message CrashLoopBackOff signifies that a pod is experiencing instability—specifically, one or more containers are repeatedly crashing and restarting. This situation usually arises because pods are configured with a default restartPolicy set to Always.

This default policy dictates that any failing container within the pod must attempt to restart. Nonetheless, a container might continue to fail to start even if other containers within the pod are running normally. Reasons for a pod entering a CrashLoopBackOff state can range from:

• Deployment errors within Kubernetes
• Absence of necessary dependencies
• Complications introduced by recent software updates

To resolve it, check Prometheus's logs for errors on startup, validate your configuration files, and ensure that your storage is not corrupted.

Conclusion

Prometheus is a powerful monitoring tool, but it requires careful tuning and maintenance to avoid common pitfalls. By understanding the typical problems with Prometheus and how to address them, you can ensure a stable and efficient monitoring system. Remember, the best defense against issues in Prometheus is a good offense: regular maintenance, monitoring, and staying informed about best practices will keep your monitoring system healthy and performant. A managed hosted prometheus solution can also help reduce the toil of self-managing your own Prometheus setup.