# Prometheus Port Configuration: A Detailed Guide > Learn how to configure Prometheus ports correctly, whether using defaults or custom settings, to keep your monitoring setup running smoothly. Source: https://last9.io/blog/prometheus-port-configuration/ Setting up Prometheus should be straightforward, but when metrics stop flowing, it’s usually something simple—like a port issue. Misconfigure it, and suddenly, your whole monitoring setup feels like a guessing game. This guide breaks down how to configure Prometheus ports properly, whether you're sticking to defaults or need a custom setup. ## The Prometheus Port Architecture Prometheus port is the network endpoint where your Prometheus server listens for incoming connections to scrape metrics from your targets. By default, Prometheus uses port 9090 for its web interface and API. But Prometheus isn't just one port. Your entire monitoring ecosystem involves multiple ports that need to play nice together: - **9090**: Prometheus server web UI and API - **9091**: Pushgateway (when you need to push metrics instead of having them scraped) - **9093**: Alertmanager - **9100**: Node Exporter (for machine metrics) - **9104**: MySQL Exporter - **9114**: Elasticsearch Exporter - **9115**: Blackbox Exporter - **9182**: SNMP Exporter Each component in the Prometheus ecosystem follows the pattern of exposing an HTTP endpoint on a specific port. This modular design allows for distributed deployment and independent scaling. If you're configuring Prometheus ports, chances are you'll need to interact with its API. Here's how to make the most of it: [Prometheus API](https://last9.io/blog/prometheus-api/). ## Why Your Prometheus Port Configuration Makes or Breaks Your Monitoring Strategy Mess up your port configuration and you're flying blind. Here's what's at stake: - **Security vulnerabilities**: Open the wrong ports to the wrong networks and you've left your monitoring data out on the porch with a "take me" sign - **Data black holes**: Misconfigured ports mean metrics don't get collected, and you don't know what you don't know - **Troubleshooting nightmares**: When something breaks at 3 AM, proper port setup makes the difference between a quick fix and an all-nighter - **Performance bottlenecks**: Incorrect port setup can lead to excessive network traffic or TCP connection exhaustion - **Infrastructure scalability limits**: As your infrastructure grows, proper port management becomes essential for maintaining clean network segmentation ## How to Configure Custom Prometheus Server Ports: Command-Line and YAML Methods The simplest way to change your Prometheus server port from the default 9090 is through the command line when you start the service: ```bash prometheus --web.listen-address=:8080 ``` This command tells Prometheus to listen on port 8080 on all network interfaces (0.0.0.0). The colon prefix (`:8080`) is shorthand for "bind to all available network interfaces on port 8080." This is particularly useful in containerized environments where you don't know the IP address in advance. If you want to bind to a specific IP address, you can specify that too: ```bash prometheus --web.listen-address=192.168.1.100:8080 ``` This more restrictive configuration binds Prometheus only to the specified IP address (192.168.1.100) on port 8080. This is useful for multi-homed servers where you want to limit which network interface Prometheus is accessible from, enhancing security by reducing exposure. Alternatively, you can configure this in your `prometheus.yml` configuration file: ```yaml web: listen_address: ":8080" # Just the port for all interfaces # OR # listen_address: "192.168.1.100:8080" # Specific IP:Port ``` This YAML configuration achieves the same result as the command-line options but makes it persistent in your configuration file. The `web` section is completely separate from the `global` section in your config file, which contains settings like `scrape_interval` and `evaluation_interval`. This separation allows for cleaner organization of configuration parameters. Note that when running Prometheus in systemd, you would typically modify the service file to include these options: ```ini [Service] ExecStart=/usr/local/bin/prometheus \ --config.file=/etc/prometheus/prometheus.yml \ --storage.tsdb.path=/var/lib/prometheus \ --web.listen-address=:8080 ``` This systemd service configuration ensures Prometheus will always start with your custom port, even after system reboots. Once your Prometheus ports are configured, the next step is making queries more efficient. Check out how Prometheus functions can help: [Prometheus Functions](https://last9.io/blog/prometheus-functions/). ## Advanced Port Configuration For more complex network setups, you might need Prometheus to listen on multiple interfaces or support IPv6: ```bash # For IPv6 support prometheus --web.listen-address="[::]:9090" # For multiple listeners (requires prometheus 2.37.0+) prometheus --web.listen-address=127.0.0.1:9090 --web.listen-address=192.168.1.100:9090 ``` The first command configures Prometheus to listen on all IPv6 interfaces. The square brackets are part of the IPv6 address notation. The second command (available in newer Prometheus versions) allows Prometheus to listen on multiple specific interfaces simultaneously, giving you granular control over network access. ## How to Configure Target Exporter Ports with Authentication and Timeout Controls Your monitoring targets (the services you want data from) need their ports properly set up. Here's how that works for some common exporters with advanced configuration options: | Exporter | Default Port | Configuration Method | Advanced Options | Common Gotchas | | ----------------- | ------------ | ------------------------------ | -------------------------------------------- | -------------------------------------------------------------------------------- | | Node Exporter | 9100 | `--web.listen-address=":9100"` | `--web.config.file="/path/to/config.yml"` | Firewall rules often block this; SELinux can prevent access to system metrics | | MySQL Exporter | 9104 | `--web.listen-address=":9104"` | `--collect.info_schema.tables.process=false` | Needs MySQL user with stats permissions; connection pooling affects metrics | | Blackbox Exporter | 9115 | `--web.listen-address=":9115"` | `--timeout=5s` | Target URLs need to be reachable; DNS resolution failures can cause high latency | | cAdvisor | 8080 | `--port=8080` | `--docker_only=true` | Container access permissions; high cardinality metrics can cause memory issues | For exporters that use HTTP endpoints, you can configure authentication. Here's an example for the Node Exporter: ```yaml # web-config.yml for node exporter tls_server_config: cert_file: server.crt key_file: server.key basic_auth_users: prometheus: $2y$10$qrPkF6JVZpyf.4g8CXP77OnnXxtiIPljLnECqK.g7ItIwl2WnM/Vi ``` This configuration sets up TLS encryption and basic authentication for your Node Exporter. The password is stored as a bcrypt hash, significantly enhancing security by protecting your metrics endpoint from unauthorized access. In your Prometheus scrape config, you'll need to match this authentication: ```yaml scrape_configs: - job_name: "node" basic_auth: username: prometheus password: secret_password tls_config: insecure_skip_verify: false ca_file: /path/to/ca.crt static_configs: - targets: ["node1:9100", "node2:9100"] ``` This scrape configuration includes the credentials and TLS settings needed to access the secured Node Exporter endpoints. The `insecure_skip_verify: false` option ensures certificate validation, protecting against man-in-the-middle attacks. Configuring Prometheus ports is just the start. Understanding metric types will help you make sense of the data you're collecting: [Prometheus Metric Types](https://last9.io/blog/prometheus-metrics-types-a-deep-dive/). ## Port Conflicts: Resolving the "Address Already in Use" Error Running into the dreaded "address already in use" error? You've got a port conflict. Here's how to fix it systematically: **For systemd services, update and restart**: ```bash sudo systemctl edit prometheus.service # Add the override with the new port sudo systemctl daemon-reload sudo systemctl restart prometheus ``` This sequence creates a systemd override that changes the port, reloads the systemd configuration, and restarts the Prometheus service with the new settings without modifying the original service file. **For temporary testing, kill the conflicting process**: ```bash sudo kill -9 $(sudo lsof -t -i:9090) ``` This command forcibly terminates whatever process is using port 9090. Use with caution, as it doesn't gracefully shut down the service, which could lead to data corruption for some applications. **Update your prometheus.yml scrape configs**: ```yaml scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9091"] ``` This configuration updates Prometheus to scrape itself at the new port (9091). Without this change, self-monitoring would fail since Prometheus would try to scrape the old port. **Pick a different port**: ```bash prometheus --web.listen-address=:9091 ``` This configuration changes Prometheus to use port 9091 instead of the default 9090, avoiding the conflict. Remember that when changing ports, you'll need to update anything that connects to Prometheus. **Find the culprit**: ```bash sudo netstat -tulpn | grep 9090 # OR for a more detailed view sudo ss -tulpn | grep 9090 ``` These commands show all processes using port 9090. The `netstat` command is more widely available, while `ss` is newer and can provide more detailed information. The output will show the PID (process ID) of whatever is using your port. ## Port Management in Containerized Environments Containers add another layer to port configuration. When using Docker, you'll need to map container ports to host ports: ```bash docker run -p 8080:9090 prom/prometheus --web.listen-address=:9090 ``` This Docker command maps the host's port 8080 to the container's port 9090. Inside the container, Prometheus is still listening on its usual port (9090), but from outside, you'll access it via port 8080. The `--web.listen-address=:9090` is redundant here since it's the default, but included for clarity. For Docker Compose, you'd use: ```yaml services: prometheus: image: prom/prometheus ports: - "8080:9090" command: - "--config.file=/etc/prometheus/prometheus.yml" - "--storage.tsdb.path=/prometheus" - "--web.listen-address=:9090" ``` This Docker Compose configuration achieves the same port mapping as the previous command but in a declarative format that's easier to maintain. In Kubernetes, you'll need a Service definition with more nuanced control: ```yaml apiVersion: v1 kind: Service metadata: name: prometheus annotations: prometheus.io/scrape: "true" prometheus.io/port: "9090" spec: selector: app: prometheus ports: - name: web port: 9090 targetPort: 9090 type: ClusterIP # or NodePort/LoadBalancer depending on your needs ``` This Kubernetes Service exposes the Prometheus pod's port 9090 as a service. The `type: ClusterIP` makes it only accessible within the cluster. Use `NodePort` to expose it on each node's IP, or `LoadBalancer` to provision an external load balancer (in cloud environments). For Istio service mesh environments, you'll need to consider port naming conventions: ```yaml ports: - name: http-prometheus port: 9090 targetPort: 9090 ``` This naming convention (`http-prometheus`) tells Istio that this is an HTTP port, enabling features like automatic mTLS, retry logic, and traffic splitting. If you're setting up Prometheus in a containerized environment, check out how to run it with Docker Compose for a smoother deployment: [Prometheus with Docker Compose](https://last9.io/blog/prometheus-with-docker-compose/). ## Enterprise-Grade Security for Prometheus Ports An open Prometheus port can expose sensitive metrics about your infrastructure. Lock it down with these enterprise approaches: **Kubernetes Network Policies for granular access control**: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: prometheus-access namespace: monitoring spec: podSelector: matchLabels: app: prometheus ingress: - from: - namespaceSelector: matchLabels: access: monitoring - podSelector: matchLabels: role: monitoring-dashboard ports: - protocol: TCP port: 9090 ``` This Kubernetes NetworkPolicy restricts access to Prometheus pods to only pods with specific labels and from specific namespaces. This microsegmentation provides defense-in-depth by controlling traffic at the Kubernetes network layer. **Firewall Rules with IP-based allowlisting**: ```bash # Allow access only from specific IP ranges sudo iptables -A INPUT -p tcp --dport 9090 -s 10.0.0.0/8 -j ACCEPT sudo iptables -A INPUT -p tcp --dport 9090 -j DROP ``` These iptables rules allow access to port 9090 only from the 10.0.0.0/8 private network range and drop all other connection attempts. This network-layer protection complements application-layer security measures. **TLS Encryption with certificate verification**: ```yaml web: listen_address: ":9090" tls_server_config: cert_file: prometheus.crt key_file: prometheus.key client_auth_type: "RequireAndVerifyClientCert" client_ca_file: client_ca.crt ``` This advanced TLS configuration encrypts traffic to/from Prometheus and requires clients to present valid certificates for mutual TLS authentication. This provides extremely strong security by ensuring both server and client identities are verified. **Basic Auth with bcrypt hashed passwords**: ```yaml web: listen_address: ":9090" basic_auth_users: admin: $2y$10$zT7JcUX.j0BWQHUwNnG8Ue/eKz8qPB3QQnUt.GHDB.DkNP0YzEfTe # "admin_password" readonly: $2y$10$lQi/hU/7WiYK6U3gXfDnqOTiNQ2L3qH8MxQALQkMn75M6wcGzGT7G # "viewer_password" ``` This configuration sets up two users with different passwords. You can generate these bcrypt hashes using tools like `htpasswd` Or online generators. Multiple users allow for different access levels, enhancing security through the principle of least privilege. Setting up Prometheus is one thing, but making sure it scales without breaking is another. Here’s how to do it right: [Scaling Prometheus](https://last9.io/blog/scaling-prometheus-tips-tricks-and-proven-strategies/). ## Advanced Diagnostic Techniques for Prometheus Port Issues When metrics aren't flowing, check these common port-related issues with advanced troubleshooting: **Container networking issues?** ```bash # Check if port is properly exposed in Docker docker ps | grep prometheus # Inspect the container network settings docker inspect --format '{{.NetworkSettings.Ports}}' prometheus # In Kubernetes, check pod status and events kubectl describe pod prometheus-pod-name ``` These commands help troubleshoot container-specific networking issues. They verify that the container is running, that ports are correctly mapped, and (in Kubernetes) show events that might indicate why a pod can't start or expose its ports properly. **Firewall blocking?** ```bash # Check existing rules sudo iptables -L | grep 9090 # Test connectivity with telnet telnet your-prometheus-server 9090 # For more complex diagnosing, try traceroute + port sudo traceroute -T -p 9090 your-prometheus-server ``` These commands help identify network-level issues. The iptables command shows if there are explicit rules affecting port 9090, telnet tests basic TCP connectivity, and traceroute with the `-T` and `-p` flags show where along the network path TCP connections to port 9090 might be failing. **Exporter not scraping?** ```bash # Test basic connectivity curl http://your-exporter:9100/metrics # Check Prometheus scrape status curl http://your-prometheus:9090/api/v1/targets | jq . # Debug a specific scrape with verbose logging curl -s 'http://your-prometheus:9090/api/v1/query?query=up{job="node"}' | jq . ``` This sequence of commands helps isolate where the problem lies. The first tests if the exporter is accessible, the second checks what Prometheus knows about all targets, and the third checks the status of a specific job. The `jq` tool formats the JSON output for readability. **Can't connect to Prometheus UI?** ```bash # If you've configured a custom port (like 8080) curl -v http://your-prometheus-server:8080/-/healthy # Or default port curl -v http://your-prometheus-server:9090/-/healthy # With TLS and authentication curl -v --cacert ca.crt --cert client.crt --key client.key -u admin:password https://your-prometheus-server:9090/-/healthy ``` The `-v` (verbose) flag in these curl commands shows the entire HTTP transaction, including TLS handshake details and HTTP headers. This provides crucial information for diagnosing connection, authentication, or TLS certificate issues. ## Optimize Prometheus Port Handling for High-Scale Deployments For high-cardinality environments with millions of time series, port configuration needs careful optimization: ```yaml web: listen_address: ":9090" max_connections: 512 read_timeout: 30s write_timeout: 30s max_request_size: 2097152 # 2MB in bytes ``` This configuration tunes Prometheus' HTTP server parameters to handle high loads. The `max_connections` setting limits concurrent connections to prevent resource exhaustion, while the timeouts prevent slow clients from tying up connections indefinitely. The `max_request_size` setting limits the size of incoming HTTP requests to prevent memory exhaustion attacks. For the Prometheus Go runtime itself, you can add: ```bash prometheus --web.listen-address=:9090 --web.max-connections=512 --storage.remote.flush-deadline=1m ``` These additional flags control connection pooling and remote write behavior. The flush deadline setting is particularly important for remote storage scenarios, where it controls how long Prometheus will wait when shutting down before dropping unwritten samples. Getting your Prometheus ports right is crucial, but so is keeping it highly available. Learn how to avoid single points of failure: [High Availability in Prometheus](https://last9.io/blog/high-availability-in-prometheus/). ## Prometheus Port in High-Availability Setups Running Prometheus at scale? Your port configuration needs to account for load balancing and redundancy: ```yaml global: external_labels: prometheus: "prom-1" # Unique identifier cluster: "production" replica: "A" web: listen_address: ":9090" ``` This configuration adds external labels that uniquely identify this Prometheus instance. These labels are critical in federated setups to avoid duplicate metrics. For Thanos sidecars, you'll need additional ports: ```bash thanos sidecar \ --tsdb.path="/path/to/prometheus/data" \ --prometheus.url="http://localhost:9090" \ --grpc-address="0.0.0.0:10901" \ --http-address="0.0.0.0:10902" \ --objstore.config-file="/path/to/bucket.yml" ``` This Thanos sidecar configuration exposes two additional ports: 10901 for gRPC communication between Thanos components and 10902 for the HTTP endpoint. The sidecar connects to the local Prometheus on its port 9090, uploads blocks to object storage, and enables querying of both real-time and historical data. For a complete Thanos setup with Kubernetes, you would need: ```yaml # For Prometheus --- apiVersion: v1 kind: Service metadata: name: prometheus spec: selector: app: prometheus ports: - name: web port: 9090 targetPort: 9090 # For Thanos Sidecar --- apiVersion: v1 kind: Service metadata: name: thanos-sidecar spec: selector: app: prometheus ports: - name: grpc port: 10901 targetPort: 10901 - name: http port: 10902 targetPort: 10902 ``` This configuration creates separate Kubernetes services for Prometheus and its Thanos sidecar, allowing different components to connect to the appropriate ports. This separation is essential for scalability in large deployments. ## Best Practices for Prometheus Port in Enterprise Environments Keep your monitoring stack humming with these enterprise-grade approaches: - **Standardize port assignments** across environments using Infrastructure as Code tools like Terraform or Ansible - **Document your port mapping** in a service registry or CMDB, not just in your configuration files - **Use service discovery** with relabeling for dynamic environments to avoid hardcoded ports - **Set up port monitoring** with synthetic probes that regularly check endpoint availability - **Implement circuit breakers** for remote write endpoints to prevent cascading failures - **Use consistent port naming conventions** across all exporters and services - **Use sidecars** for consistent port exposure in microservice architectures - **Create dedicated service accounts** with least privilege for each Prometheus instance Example of standardized port allocation in Ansible: ```yaml prometheus_components: server: port: 9090 alertmanager: port: 9093 node_exporter: port: 9100 mysql_exporter: port: 9104 custom_exporter: port: 9999 ``` This Ansible variable structure enforces consistent port allocation across your entire infrastructure, making it easier to manage firewall rules and troubleshoot issues. Once your Prometheus setup is running smoothly, recording rules can help optimize queries and reduce load. Learn how to use them: [Prometheus Recording Rules](https://last9.io/blog/prometheus-recording-rules/). ## Wrapping Up With the foundation of a solid monitoring stack you'll have reliable metrics flowing when you need them most – during incidents when every second counts. Remember these key takeaways: - Default ports are just starting points – customize them for your security and network requirements - Port configuration goes beyond just listening addresses – think about TLS, authentication, and connection limits - In containerized environments, you have multiple layers of port configuration (container, service, ingress) - As scale increases, port management becomes increasingly important for reliability and security If you've any questions about your Prometheus setup, jump into our [Discord Community](https://discord.com/invite/W8gMppQC4b) to discuss with other developers.