What is Prometheus Remote Write

In this blog post, we will cover the following points.

• What is Prometheus Remote Write
• How to configure Remote Write
• Long Term Storage of metrics using Prometheus Remote Write
• Remote Write vs. Federation
• What is Prometheus Agent

If you are new to the world of Prometheus and Metrics, do check our blog ⏤ What is Prometheus?

What is Prometheus Remote Write?

Prometheus Remote Write is a mechanism that enables you to write data to external storage, which are Prometheus compatible backends.

Prometheus syncs with the backend of these external systems by sending HTTP POST requests to the Remote Write endpoint. This technique has batching and deduplication features to help minimize network traffic and avoid data duplication.

Remote Write also allows cross-cluster federation, allowing you to scale your Prometheus instances across multiple clusters horizontally. Prometheus Remote Write utilizes an efficient binary format, preserving Prometheus standard labels and timestamps, ensuring the original metrics' consistency and accuracy.

To use Prometheus Remote Write, you must enable the feature in the Prometheus server and configure the Remote Write endpoint. You must also configure the external storage to ingest remote write metrics.

How to Configure Prometheus Remote Write:

To configure Remote Write in Prometheus, follow these steps:

Ensure you have a target destination that can accept metrics in Prometheus Remote Write format. Examples of such systems are Levitate, Thanos, Cortex.

Edit the Prometheus configuration file (prometheus.yml) to include a remote_write block. Here is a sample prometheus remote write config:

remote_write:
- url: "http://your-remote-write-target/endpoint"
  # Sets the `Authorization` header on every remote write request with the
  # configured username and password.
  # password and password_file are mutually exclusive.
  basic_auth:
    [ username: <string> ]
    [ password: <secret> ]
    [ password_file: <string> ]

Configure the remote_write block according to your needs.

Configuring Multiple Remote Write Destinations

You can add multiple entries to this block to send data to multiple remote write endpoints.

remote_write:
- url: "http://your-remote-write-target/endpoint"
  name: Alpha
  # Sets the `Authorization` header on every remote write request with the
  # configured username and password.
  # password and password_file are mutually exclusive.
  basic_auth:
    [ username: <string> ]
    [ password: <secret> ]
    [ password_file: <string> ]

- url: "http://your-second-remote-write-target/endpoint"
  name: Beta
- url: "http://your-third-remote-write-target/endpoint"
  name: Gamma

Save the Prometheus configuration file and restart Prometheus so that configuration changes take effect.

Collecting Metrics via Prometheus Remote Write

To collect metrics using Prometheus Remote Write, and store them in a remote system for further processing and analysis, follow these steps:

Set up a remote write-enabled target destination like Levitate or Thanos.

Configure Prometheus to send metrics to the remote write endpoint. Use the remote_write configuration block in the prometheus.yml configuration file to add the endpoint information, for example:

remote_write:
- url: "http://your-remote-write-target/endpoint"

Restart Prometheus and verify that the metrics are sent to the remote write target by querying the target endpoint directly.

Configure the retention policies and other settings to ensure the metrics are stored efficiently and securely. Query the remote write destination for the metrics using the appropriate query language and tools.

Prometheus long-term storage — a use-case for remote write

One primary challenge with Prometheus is the lack of a built-in remote storage system that can provide long-term data retention and archiving capabilities.

Prometheus's local storage is limited to a single node's scalability and durability. Instead of trying to solve clustered storage in Prometheus itself, Prometheus offers a set of interfaces that allow integration with remote storage systems.

Prometheus Remote Write Receiver is a crucial component in the ecosystem of Prometheus monitoring. It acts as a bridge, enabling the transmission of metrics from one Prometheus instance to another through the Remote Write functionality. Users can seamlessly export their monitoring data to another Prometheus server or any other remote storage system supporting the Remote Write protocol by configuring the Prometheus Remote Write Receiver. This allows for efficient data sharing, aggregation, and analysis across multiple Prometheus instances, promoting collaborative monitoring efforts and enabling organizations to build robust and scalable monitoring architectures.

There are Prometheus Remote Receivers, such as Levitate and Thanos, that can be used for long-term storage of time series data.

OpenTelemetry and Prometheus Remote Write

OpenTelemetry helps software developers better understand the behavior and performance of their applications, services, and infrastructure. It provides a set of APIs, libraries, agents, and collectors that capture telemetry data — logs, metrics, and traces — from various sources and exports them to backends like logging systems, monitoring tools, and tracing platforms.

For how to use the OpenTelemetry package in Python applications to send metrics to Prometheus Remote Write endpoint, such as Levitate, check the document below.

Using OpenTelemetry Exporter for Prometheus Remote Write | Last9 Documentation

Use OpenTelemetry Prometheus Exporter package in Python applications to send data to Prometheus-compatible systems such as Levitate

Last9

The Future of Prometheus Remote Write

A significant quality stabilizing the future of Prometheus Remote Write is its integration with cloud-native architectures and technologies such as Kubernetes. And as more organizations adopt these platforms, monitoring them effectively becomes increasingly essential. Remote Write helps collect and send metrics to these environments.

Also, as more applications become cloud-native, microservices-based, and hybrid-cloud, collecting and integrating data from various locations and systems will become critical. Remote Write may quickly become part of the solution for centralizing cross-platform monitoring.

In April 2023, the Prometheus Remote-Write Specification was published to standardize Version 1.0. It contains plans that speculate the quality of services Prometheus Remote-Write could offer. We may see further improvements such as caching, compression, and more efficient serialization. These improvements would help handle larger volumes of data and reduce the costs and overhead associated with transmitting large amounts of data over the network.

What Is Agent Mode in Prometheus?

The agent mode in Prometheus is a way to collect and scrape metrics from a target system without requiring the installation of the full Prometheus server. When running in the agent mode, Prometheus acts as a lightweight data collection agent that can be installed on the target system and periodically scrapes specified metrics from that system. The agent mode can also be configured to scrape metrics from multiple targets.

Once the metrics have been scraped, the agent mode exposes them via the remote write HTTP endpoint. This endpoint serves as the target URL for a Prometheus-compatible TSDB, which can then ingest the metrics and store them.

Prometheus Remote Write vs. Federation

Both are mechanisms to handle data across multiple Prometheus servers but are used in different scenarios.

Remote Write

Remote write is the mechanism that allows Prometheus to send data to other systems while it is collected in real-time. Prometheus can send each scraped sample to a remote endpoint immediately. It is designed to export your metrics to third-party long-term storage systems like Levitate, Cortex, Thanos, or any other compatible API system.

In the configuration file, the remote_write the section specifies to which URL Prometheus should send its recorded samples. This feature enables real-time sending and can be helpful for long-term storage, replicability, and data analysis.

Prometheus Federation

On the other hand, Federation is a Prometheus feature that allows a Prometheus server to scrape selected time series from another. This allows you to create hierarchical and/or functional clustering of Prometheus servers for scalability and reliability.

Federation is commonly used when multiple Prometheus servers run across different environments or data centers and want a global view of all metrics. The higher-level Prometheus server can scrape a subset of metrics from the lower-level server.

The primary differences between remote write and federation are:

• The direction of the data: With remote write, data is pushed out from Prometheus to other systems. With federation, a Prometheus server pulls data from other Prometheus servers.
• The purpose: Remote write is typically used for long-term storage and integration with other monitoring systems, while federation creates a hierarchy of Prometheus servers for scalability and reliability.
• The type of data: Remote write sends each scraped sample immediately, while federation deals with aggregated and less granular data over a longer time.

Conclusion

Keep your metrics from being lost in the abyss of unmanageable data. Remote Write them to a centralized location such as Levitate. In this post, you have learned about the key benefits of Remote Write, how to set it up, and the various configurations and options available.