What is Prometheus Remote Write

Are you frustrated with Prometheus’ storage issues? Prometheus Remote Write might absolve your woes. Read this article to bade farewell to the woes of long term Prometheus storage.

What is Prometheus?

Prometheus is an open-source monitoring tool that monitors systems and applications in real-time, and stores time-stamped data. Written in the Go programming language, Prometheus utilizes a pull-based metric collection approach. Its building blocks include the Prometheus Server, instrumented applications, exporters, and clients.

Prometheus long term storage — a headache

One primary challenge with Prometheus is the lack of a built-in remote storage system that can provide long-term data retention and archiving capabilities.

Prometheus's local storage is limited to a single node's scalability and durability. Instead of trying to solve clustered storage in Prometheus itself, Prometheus offers a set of interfaces that allow integrating with remote storage systems.

What is Prometheus Remote Write?

Prometheus Remote Write enables you to write data to external storage or monitoring backends which are Prometheus compatible.

Prometheus syncs with the backend of these external systems by sending HTTP POST requests to the Remote Write endpoint. This technique has batching and deduplication features to help minimize network traffic and avoid data duplication.

Remote Write also allows cross-cluster federation, allowing you to scale your Prometheus instances across multiple clusters horizontally. Prometheus Remote Write utilizes an efficient binary format, preserving Prometheus standard labels and timestamps, ensuring the original metrics' consistency and accuracy.

To use Prometheus Remote Write, you must enable the feature in the Prometheus server and configure the Remote Write endpoint. You must also configure the external storage to ingest remote write metrics.

What Is Agent Mode in Prometheus?

The agent mode in Prometheus is a way to collect and scrape metrics from a target system without requiring the installation of the full Prometheus server. When running in the agent mode, Prometheus acts as a lightweight data collection agent that can be installed on the target system and will periodically scrape specified metrics from that system. The agent mode can also be configured to scrape metrics from multiple targets.

Once the metrics have been scraped, the agent mode then exposes them via the remote write HTTP endpoint. This endpoint serves as the target URL for a Prometheus compatible TSDB, which can then ingest the metrics and store them.

Configuring Prometheus Remote Write

To configure Prometheus Remote Write in Prometheus, follow these steps:

Ensure you have a target destination that can accept metrics in Prometheus Remote Write format. Eg. of such systems are Levitate, Thanos, Cortex.

Edit the Prometheus configuration file (prometheus.yml) to include a remote_write block as follows:

remote_write:
- url: "http://your-remote-write-target/endpoint"
  # Sets the `Authorization` header on every remote write request with the
  # configured username and password.
  # password and password_file are mutually exclusive.
  basic_auth:
    [ username: <string> ]
    [ password: <secret> ]
    [ password_file: <string> ]

Configure the remote_write block according to your needs.

Configuring Multiple Remote Write Destinations

You can add multiple entries to this block to send data to multiple remote write endpoints.

remote_write:
- url: "http://your-remote-write-target/endpoint"
  name: Alpha
  # Sets the `Authorization` header on every remote write request with the
  # configured username and password.
  # password and password_file are mutually exclusive.
  basic_auth:
    [ username: <string> ]
    [ password: <secret> ]
    [ password_file: <string> ]

- url: "http://your-second-remote-write-target/endpoint"
  name: Beta
- url: "http://your-third-remote-write-target/endpoint"
  name: Gamma

Save the Prometheus configuration file and restart Prometheus so that configuration changes take effect.

History of Remote Write Protocol

Remote Write enables the export of Prometheus-generated metrics to long-term or remote storage or monitoring systems, solving the storage constraints and scaling difficulties that Prometheus has. It was introduced in Prometheus version 2.0, released in May 2017, and is considered a replacement for the previous storage backend, which relied on local disk storage. Its introduction was driven by the desire to enable Prometheus’ integration with external storage solutions.

Remote Write is based on the OpenMetrics specification that defines a standard for representing metrics in a line-based text format over HTTP. It can be used with many data stores and monitoring systems that support the OpenMetrics standard.

Collecting Metrics via Remote Write

To collect metrics using Prometheus Remote Write, and store them in a remote system for further processing and analysis, follow these steps:

Set up a remote write-enabled target destination like Levitate or Thanos.

Configure Prometheus to send metrics to the remote write endpoint. Use the remote_write configuration block in the prometheus.yml configuration file to add the endpoint information, for example:

remote_write:
- url: "http://your-remote-write-target/endpoint"

Restart Prometheus and verify that the metrics are sent to the remote write target by querying the target endpoint directly.

If necessary, configure the retention policies and other settings to ensure the metrics are stored efficiently and securely. Query the remote write destination for the metrics using the appropriate query language and tools.

OpenTelemetry and Prometheus Remote Write

OpenTelemetry helps software developers better understand the behavior and performance of their applications, services, and infrastructure. It provides a set of APIs, libraries, agents, and collectors that capture telemetry data — logs, metrics, and traces — from various sources and exports them to backends like logging systems, monitoring tools, and tracing platforms.

For how to use the OpenTelemetry package in Python applications to send metrics to Prometheus Remote Write endpoint, such as Levitate, check the document below.

🐍 Using OpenTelemetry Exporter for Prometheus Remote Write

An example showing how to use OpenTelemetry Exporter for Prometheus Remote Write to send metrics to Prometheus compatible backend such as Levitate.

Last9 Documentation

The Future of Prometheus Remote Write

A major quality that stabilizes the future of Prometheus Remote Write is its integration with cloud-native architectures and technologies such as Kubernetes. And as more organizations adopt these platforms, monitoring them effectively becomes increasingly important. Remote Write is useful for collecting and sending metrics to these environments.

Also, as more applications become cloud-native, microservices based, and hybrid-cloud, collecting and integrating data from various locations and systems will become critical. Remote Write may easily become part of the solution for centralizing cross-platform monitoring.

In April 2023, the Prometheus Remote-Write Specification was published to standardize Version 1.0. It contains plans that speculate the quality of services Prometheus Remote-Write could offer. We may see further improvements such as caching, compression, and more efficient serialization. These improvements would help handle larger volumes of data and reduce the costs and overhead associated with transmitting large amounts of data over the network.

How to Scale Prometheus Remote Write

To improve Prometheus Remote Write performance at scale, several steps can be taken:

Use compression

Compressing data can reduce the amount of network traffic transmitted to the remote write endpoint and improve performance. Consider using a compression algorithm such as gzip. In Thanos, you can enable gzip compression in the config file for the sidecar:

grpc:
  compression:
    enabled: true
    level: 9

Use batching

Consider grouping samples into batches to minimize the number of requests sent to the remote write destination. In Thanos, you can set the chunk_size and max_series_per_request parameters in the configuration file to control batch size:

remote_write:
    urls:
      - http://your-remote-write-target/endpoint
    chunk_size: 500
    max_series_per_request: 100

Optimize sample collection

Limit the number of Prometheus instances and target endpoints to improve the efficiency of data collection. You can also fine-tune scrape intervals to ensure that only relevant data is being sent.

Use caching

If possible, cache the data locally for a certain period. This allows for batching multiple sample writes into a single request, saving network resources and lowering the amount of request processing time.

Optimize chunking

Consider adjusting the chunk size or the number of streams. A larger chunk size can reduce the required requests, while smaller chunks can reduce the overhead of transmitting small sample sets.

Use a higher-performance storage such as Levitate

Consider using a higher-performance storage system to handle large volumes of time-series data. Using a managed time series warehouse like Levitate can improve the scalability and availability of the storage layer, improve query latency, and reduce the overall load on Remote Write.

For extensive real-life experience in improving the performance of Prometheus Remote Write at scale check our blog post below.

How to improve Prometheus remote write performance at scale

Deep dive into how to improve the performance of Prometheus Remote Write at Scale based on real-life experiences

Book a Demo

Conclusion

Keep your metrics from being lost in the abyss of unmanageable data. Remote Write them to a centralized location such as Levitate. In this post, you have learned about the key benefits of Remote Write, how to set it up, and the various configurations and options available. So overcome the short-term retention of Prometheus storage, and move on to more efficient monitoring with Prometheus Remote Write.