Prometheus and Grafana: Together!

What is Prometheus and Grafana?

Prometheus and Grafana are two open-source tools that streamline performance monitoring. Prometheus is like the calm, collected superhero that lurks in the background, collecting data, while Grafana swoops all performance metrics into a dashboard with an impressive array of visualization options.

Prometheus collects all the metrics and provides a powerful querying language; Grafana allows for those metrics to be visualized for usage.

How exactly do they work together for monitoring and observability? In this blog post, we'll dive into the nitty-gritty of both tools and provide step-by-step guidelines to connect them seamlessly.

What is Prometheus?

Prometheus is an open-source monitoring system that collects and stores time-series metrics data from various sources, including applications, software systems, and hardware.

It was initially developed by SoundCloud in 2012 and later donated to the Cloud Native Computing Foundation (CNCF) in 2016. Prometheus works with modern cloud-native infrastructure and supports various deployment models. Prometheus functions via its query language—PromQL and has alerting functions that enable you to take a proactive approach to system monitoring.

What is Grafana?

Grafana is an open-source data visualization and analysis tool used to visualize data on customizable dashboards in different formats, such as graphs, charts, and tables. It is widely used for visualizing real-time metrics data, time-series data, and log data from various sources, including Prometheus, Last9, Elasticsearch, InfluxDB, and more.

Grafana was initially released in 2014 to address the need for open-source, modern, and flexible visualization tools. Thanks to its ease of use, scalability, and extensive community support, it has become a leading visualization tool.

Grafana provides an extensive plugin ecosystem that allows users to add new data sources, panels, and features quickly and easily. It also allows users to configure alerts based on thresholds and other conditions, notifying them when specific metrics cross defined thresholds.

Difference between Prometheus and Grafana

Grafana and Prometheus are not the same. Although Prometheus and Grafana work together, they differ in scope, functionality, and purpose — Prometheus pulls metrics data from various systems and stores it in a time-series database (TSDB), while Grafana uses Prometheus as its data source and processes the data for analysis and visualization.

Recommended reading - head over to InfluxDB vs Prometheus for a detailed comparative analysis of the two. In another blog, we are comparing all the popular time series databases. Go check them out.

Using Prometheus and Grafana

To capture and visualize metrics using Prometheus and Grafana, you must configure both tools independently and connect them afterward.

Install Prometheus

We can simply use docker and docker-compose to setup Prometheus.

We will also node exporter to emit metrics that can be scraped by Prometheus and we can visualize them and set alerts.

Node Exporter is a popular open-source software tool used for monitoring Linux and Unix systems. Node Exporter runs on the target system that you want to monitor and collects various metrics about the system's hardware and operating system. It exposes these metrics in a format that Prometheus can scrape and store for further analysis and visualization.

version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.30.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - 9090:9090
  node-exporter:
    image: prom/node-exporter:v1.2.2
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
    ports:
      - 9100:9100

Configure Prometheus

You can configure Prometheus by defining the target endpoints in the prometheus.yml configuration file. This allows Prometheus to scrape metric values from these endpoints.

Add a Target

Add this snippet to the prometheus.yml configuration file:

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
    - targets: ['node-exporter:9100']

After creating these files, run docker-compose up -d to start the services.

Once everything is set up, you can access the Prometheus web UI by navigating to http://localhost:9090. There, you can enter expressions into the expression bar and click Execute to evaluate them. The results are shown in the graph panel.

The Node Exporter provides a wide variety of system metrics, such as CPU usage, memory, disk I/O, network, etc. For example, you can enter the expression node_memory_MemAvailable_bytes in the Prometheus web UI to view the available memory on your system.

Define Alert Rules

Prometheus allows users to define alert rules to alert them when monitored metrics cross defined thresholds. Consider this threshold rule configuration:

groups:
- name: example
  rules:
  - alert: HighCpuLoad
    expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{job="node",mode="idle"}[5m])) * 100) > 85
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "High CPU load detected"
      description: "CPU load is > 85% for 2 minutes"
  - alert: HighMemoryUsage
    expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 80
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "High memory usage detected"
      description: "Memory usage is > 80% for 2 minutes"

This file defines two alert rules: one that triggers when CPU usage is above 85% and another one when memory usage is above 80%.

See the metrics in the Prometheus UI

To visualize a graph of collected metrics in the Prometheus web interface, open http://localhost:9090 in a web browser. Run this query to get the memory usage percentage.

(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100

Complete Prometheus Configuration

global:
  scrape_interval:     15s
  evaluation_interval: 15s

rule_files:
  - "alert.rules.yml"

scrape_configs:
  - job_name: 'node'
    static_configs:
    - targets: ['node-exporter:9100']

Grafana

Grafana is an open-source data visualization and analysis software that provides real-time analytics, visualizations, and alerts. It supports the visualization of metrics data from Prometheus.

Getting started with Grafana and Prometheus

To create a series of dashboards in Grafana to display system metrics for a server monitored by Prometheus, follow these steps.

We already have Prometheus running with node exporter exposing metrics on localhost:9100

Configure Prometheus for Grafana

There are two methods to configure Prometheus for Grafana; using a hosted Grafana instance at Grafana Cloud, or running Grafana locally.

Last9's Managed Time Series Data Warehouse - Levitate comes up with Embedded Grafana which takes away the toil of running and maintaining Grafana or using a paid version. Start for Free today.

Install Grafana via Grafana Cloud

1. Sign up for Grafana here.

2. Remote write your metrics to the Grafana.com Prometheus instance

3. Add the following code to your prometheus.yml file to begin sending metrics to your hosted Grafana instance:

remote_write:
- url: <https://your-remote-write-endpoint>
  basic_auth:
    username: <your user name>
    password: <Your Grafana.com API Key>

Install Grafana using Docker

1. Download Grafana using docker-compose.

version: '3'
services:
  ...
  grafana:
    image: grafana/grafana:8.1.5
    ports:
      - 3000:3000

Run docker-compose up -d to start Grafana.

2. Open Grafana by going to http://localhost:3000 in your web browser.

3. Log in with the default username (admin) and password (admin).

4. Add a data source:

• Click on the Grafana icon in the top left corner, click on "Configuration" and then "Data Sources".
• Click on "Add data source", select Prometheus as the type.
• Enter the URL of the Prometheus server. If you're running it locally on Docker, it would be http://prometheus:9090.
• Click "Save & Test".

5. Create a dashboard:

• Click on the Grafana icon in the top left corner, click on "Create" and then "Dashboard".
• Click on "Add new panel".
• In the query field, enter a Prometheus expression, such as 100 - (avg by(instance) (irate(node_cpu_seconds_total{job="node",mode="idle"}[5m])) * 100), which corresponds to the CPU usage.
• Adjust the graph settings as desired.
• Click on "Apply".

6. Repeat step 5 for each metric you want to add to the dashboard.

Prometheus Metrics in Grafana

In your Grafana instance, go to the Explore view and build queries to experiment with the metrics you want to monitor. Pay special attention to the Prometheus-specific features to avail custom querying experience for Prometheus.

Create dashboards to render system metrics monitored by Prometheus.

Grafana dashboards can also be represented as JSON models. If you want to share a dashboard, you can click on "Share dashboard" and then "Export for sharing externally" to get the JSON model. To import a dashboard, you can paste the JSON model into the "Import" field.

Grafana Support for Prometheus

Grafana has excellent support for Prometheus—the Grafana data source for Prometheus has been included since Grafana 2.5.0 in 2015. Some of the key features of Grafana's support for Prometheus include.

1. Grafana makes integrating with Prometheus as a data source easy, simplifying the configuration process for developers and programmers alike.
2. Grafana provides a simple query builder, which enables users to build complex queries for querying Prometheus data.
3. Grafana supports multi-tenancy and allows users to create separate dashboards for different users or organizations, enabling seamless sharing of Prometheus metrics.
4. Grafana provides a wide range of visualization options for visualizing metrics data obtained from Prometheus. These include graphs, tables, heat maps, logs, and more.

Conclusion

And that, dear developer, is how Prometheus and Grafana work together! So, whether you're experienced, or a newbie, Prometheus and Grafana are a formidable duo to monitor the performance of your applications. We hope you're chomping at the bit to start your monitoring exercise.