May 22nd, ‘23/7 min read
What is Prometheus Remote Write
Everything you need to know about Prometheus Remote Write mechanism and storing metrics in long term storage such as Levitate

Are you frustrated with Prometheus’ storage issues? Prometheus Remote Write might absolve your woes. Read this article to bade farewell to the woes of long term Prometheus storage.
What is Prometheus?
Prometheus is an open-source monitoring tool that monitors systems and applications in real-time, and stores time-stamped data. Written in the Go programming language, Prometheus utilizes a pull-based metric collection approach. Its building blocks include the Prometheus Server, instrumented applications, exporters, and clients.
Prometheus long term storage — a headache
One primary challenge with Prometheus is the lack of a built-in remote storage system that can provide long-term data retention and archiving capabilities.

Prometheus's local storage is limited to a single node's scalability and durability. Instead of trying to solve clustered storage in Prometheus itself, Prometheus offers a set of interfaces that allow integrating with remote storage systems.
What is Prometheus Remote Write?
Prometheus Remote Write enables you to write data to external storage or monitoring backends which are Prometheus compatible.

Prometheus syncs with the backend of these external systems by sending HTTP POST requests to the Remote Write endpoint. This technique has batching and deduplication features to help minimize network traffic and avoid data duplication.
Remote Write also allows cross-cluster federation, allowing you to scale your Prometheus instances across multiple clusters horizontally. Prometheus Remote Write utilizes an efficient binary format, preserving Prometheus standard labels and timestamps, ensuring the original metrics' consistency and accuracy.
To use Prometheus Remote Write, you must enable the feature in the Prometheus server and configure the Remote Write endpoint. You must also configure the external storage to ingest remote write metrics.
What Is Agent Mode in Prometheus?
The agent mode in Prometheus is a way to collect and scrape metrics from a target system without requiring the installation of the full Prometheus server. When running in the agent mode, Prometheus acts as a lightweight data collection agent that can be installed on the target system and will periodically scrape specified metrics from that system. The agent mode can also be configured to scrape metrics from multiple targets.

Once the metrics have been scraped, the agent mode then exposes them via the remote write HTTP endpoint. This endpoint serves as the target URL for a Prometheus compatible TSDB, which can then ingest the metrics and store them.
Configuring Prometheus Remote Write
To configure Prometheus Remote Write in Prometheus, follow these steps:
Ensure you have a target destination that can accept metrics in Prometheus Remote Write format. Eg. of such systems are Levitate, Thanos, Cortex.
Edit the Prometheus configuration file (prometheus.yml
) to include a remote_write
block as follows:
remote_write:
- url: "http://your-remote-write-target/endpoint"
# Sets the `Authorization` header on every remote write request with the
# configured username and password.
# password and password_file are mutually exclusive.
basic_auth:
[ username: <string> ]
[ password: <secret> ]
[ password_file: <string> ]
Configure the remote_write
block according to your needs.
Configuring Multiple Remote Write Destinations
You can add multiple entries to this block to send data to multiple remote write endpoints.
remote_write:
- url: "http://your-remote-write-target/endpoint"
name: Alpha
# Sets the `Authorization` header on every remote write request with the
# configured username and password.
# password and password_file are mutually exclusive.
basic_auth:
[ username: <string> ]
[ password: <secret> ]
[ password_file: <string> ]
- url: "http://your-second-remote-write-target/endpoint"
name: Beta
- url: "http://your-third-remote-write-target/endpoint"
name: Gamma
Save the Prometheus configuration file and restart Prometheus so that configuration changes take effect.
History of Remote Write Protocol
Remote Write enables the export of Prometheus-generated metrics to long-term or remote storage or monitoring systems, solving the storage constraints and scaling difficulties that Prometheus has. It was introduced in Prometheus version 2.0, released in May 2017, and is considered a replacement for the previous storage backend, which relied on local disk storage. Its introduction was driven by the desire to enable Prometheus’ integration with external storage solutions.
Remote Write is based on the OpenMetrics specification that defines a standard for representing metrics in a line-based text format over HTTP. It can be used with many data stores and monitoring systems that support the OpenMetrics standard.
Collecting Metrics via Remote Write
To collect metrics using Prometheus Remote Write, and store them in a remote system for further processing and analysis, follow these steps:
Set up a remote write-enabled target destination like Levitate or Thanos.
Configure Prometheus to send metrics to the remote write endpoint. Use the remote_write
configuration block in the prometheus.yml
configuration file to add the endpoint information, for example:
remote_write:
- url: "http://your-remote-write-target/endpoint"
Restart Prometheus and verify that the metrics are sent to the remote write target by querying the target endpoint directly.
If necessary, configure the retention policies and other settings to ensure the metrics are stored efficiently and securely. Query the remote write destination for the metrics using the appropriate query language and tools.
OpenTelemetry and Prometheus Remote Write
OpenTelemetry helps software developers better understand the behavior and performance of their applications, services, and infrastructure. It provides a set of APIs, libraries, agents, and collectors that capture telemetry data — logs, metrics, and traces — from various sources and exports them to backends like logging systems, monitoring tools, and tracing platforms.
For how to use the OpenTelemetry package in Python applications to send metrics to Prometheus Remote Write endpoint, such as Levitate, check the document below.


The Future of Prometheus Remote Write
A major quality that stabilizes the future of Prometheus Remote Write is its integration with cloud-native architectures and technologies such as Kubernetes. And as more organizations adopt these platforms, monitoring them effectively becomes increasingly important. Remote Write is useful for collecting and sending metrics to these environments.
Also, as more applications become cloud-native, microservices based, and hybrid-cloud, collecting and integrating data from various locations and systems will become critical. Remote Write may easily become part of the solution for centralizing cross-platform monitoring.
In April 2023, the Prometheus Remote-Write Specification was published to standardize Version 1.0. It contains plans that speculate the quality of services Prometheus Remote-Write could offer. We may see further improvements such as caching, compression, and more efficient serialization. These improvements would help handle larger volumes of data and reduce the costs and overhead associated with transmitting large amounts of data over the network.
How to Scale Prometheus Remote Write
To improve Prometheus Remote Write performance at scale, several steps can be taken:
Use compression
Compressing data can reduce the amount of network traffic transmitted to the remote write endpoint and improve performance. Consider using a compression algorithm such as gzip. In Thanos, you can enable gzip compression in the config file for the sidecar:
grpc:
compression:
enabled: true
level: 9
Use batching
Consider grouping samples into batches to minimize the number of requests sent to the remote write destination. In Thanos, you can set the chunk_size
and max_series_per_request
parameters in the configuration file to control batch size:
remote_write:
urls:
- http://your-remote-write-target/endpoint
chunk_size: 500
max_series_per_request: 100
Optimize sample collection
Limit the number of Prometheus instances and target endpoints to improve the efficiency of data collection. You can also fine-tune scrape intervals to ensure that only relevant data is being sent.
Use caching
If possible, cache the data locally for a certain period. This allows for batching multiple sample writes into a single request, saving network resources and lowering the amount of request processing time.
Optimize chunking
Consider adjusting the chunk size or the number of streams. A larger chunk size can reduce the required requests, while smaller chunks can reduce the overhead of transmitting small sample sets.
Use a higher-performance storage such as Levitate
Consider using a higher-performance storage system to handle large volumes of time-series data. Using a managed time series warehouse like Levitate can improve the scalability and availability of the storage layer, improve query latency, and reduce the overall load on Remote Write.
For extensive real-life experience in improving the performance of Prometheus Remote Write at scale check our blog post below.


Conclusion
Keep your metrics from being lost in the abyss of unmanageable data. Remote Write them to a centralized location such as Levitate. In this post, you have learned about the key benefits of Remote Write, how to set it up, and the various configurations and options available. So overcome the short-term retention of Prometheus storage, and move on to more efficient monitoring with Prometheus Remote Write.
Contents
Newsletter
Stay updated on the latest from Last9.
Related Reading

Prometheus vs Datadog
Comparison between Prometheus and Datadog - two of the most popular monitoring tools in the market today
Last9

What is Prometheus
What is Prometheus, how to use it and challenges of scaling Prometheus
Last9

Prometheus Monitoring
Prometheus is a popular open-source monitoring system. In this blog, we'll cover the basics of Prometheus monitoring, including its architecture, key features, and alternatives.
Last9