NATS

Monitor NATS from two angles: server infrastructure metrics scraped via prometheus-nats-exporter, and distributed traces across your Java publish/subscribe operations via a zero-code OTel agent extension.

Prerequisites

Before setting up NATS monitoring, ensure you have:

• NATS server 2.x running with HTTP monitoring enabled (http_port: 8222)
• Administrative access to the server running the collector
• OpenTelemetry Collector installation access
• Last9 account with integration credentials

1. Enable NATS HTTP Monitoring

   NATS exposes monitoring data over HTTP on port 8222. Add the following to your NATS server configuration:

   # nats-server.conf
   port: 4222
   http_port: 8222

   Restart the server and verify the endpoint is reachable:

   curl http://localhost:8222/varz

2. Install prometheus-nats-exporter

   The prometheus-nats-exporter reads NATS monitoring endpoints and exposes them as Prometheus metrics. Install it on the same host as your NATS server or any host with network access to port 8222.

   • Binary
   • Docker

   wget https://github.com/nats-io/prometheus-nats-exporter/releases/latest/download/prometheus-nats-exporter-linux-amd64.zip
   unzip prometheus-nats-exporter-linux-amd64.zip
   sudo mv prometheus-nats-exporter /usr/local/bin/

   Start the exporter and verify it exposes metrics:

   # Start with common monitoring endpoints
   prometheus-nats-exporter -varz -connz -routez -subz -jsz=all http://localhost:8222
   
   # Verify metrics are available
   curl http://localhost:7777/metrics | grep gnatsd_varz_in_msgs

   Flags used above:

   • -varz — general server stats (messages, bytes, connections, CPU, memory)
   • -connz — per-connection stats (connection-level throughput and subscriptions)
   • -routez — cluster route stats (route throughput, pending bytes; relevant for clustered NATS)
   • -subz — subscription routing stats
   • -jsz=all — JetStream stats (if JetStream is enabled)

3. Install OpenTelemetry Collector

   • DEB Package
   • RPM Package

   wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.118.0/otelcol-contrib_0.118.0_linux_amd64.deb
   sudo dpkg -i otelcol-contrib_0.118.0_linux_amd64.deb

   More options: OpenTelemetry Collector installation docs.

4. Configure OpenTelemetry Collector

   Create the collector configuration file:

   sudo nano /etc/otelcol-contrib/config.yaml

   Add the following configuration. It scrapes NATS metrics from prometheus-nats-exporter and also accepts any OTLP data (traces, metrics, logs) from application services sending to port 4318:

   receivers:
     # Scrapes NATS server metrics via prometheus-nats-exporter
     prometheus:
       config:
         scrape_configs:
           - job_name: nats
             scrape_interval: 15s
             static_configs:
               - targets: ["localhost:7777"]
                 labels:
                   service: nats-server
   
     # Accepts OTLP from application services (e.g., Java app on port 4318)
     otlp:
       protocols:
         grpc:
           endpoint: 0.0.0.0:4317
         http:
           endpoint: 0.0.0.0:4318
   
     hostmetrics:
       collection_interval: 60s
       scrapers:
         cpu:
         memory:
         disk:
         network:
   
   processors:
     batch:
       timeout: 20s
       send_batch_size: 10000
       send_batch_max_size: 10000
     resourcedetection/system:
       detectors: ["system"]
       system:
         hostname_sources: ["os"]
     resourcedetection/cloud:
       detectors: ["aws", "gcp", "azure"]
   
   exporters:
     otlp/last9:
       endpoint: "{{ .Logs.WriteURL }}"
       headers:
         "Authorization": "{{ .Logs.AuthValue }}"
     debug:
       verbosity: detailed
   
   service:
     pipelines:
       metrics:
         receivers: [prometheus, otlp, hostmetrics]
         processors: [batch, resourcedetection/system, resourcedetection/cloud]
         exporters: [otlp/last9]
       traces:
         receivers: [otlp]
         processors: [batch, resourcedetection/system]
         exporters: [otlp/last9]
       logs:
         receivers: [otlp]
         processors: [batch, resourcedetection/system]
         exporters: [otlp/last9]

   Replace the exporters placeholder values with your actual Last9 credentials from the Last9 Integrations page.

5. Start and Enable the Service

   sudo systemctl daemon-reload
   sudo systemctl enable otelcol-contrib
   sudo systemctl start otelcol-contrib

Understanding the Setup

NATS HTTP Monitoring

NATS exposes a built-in HTTP monitoring server (default port 8222) with JSON endpoints for server stats, connections, subscriptions, and JetStream. This is the data source for all NATS metrics.

prometheus-nats-exporter

The exporter translates NATS JSON monitoring endpoints into Prometheus-format metrics. The OTel Collector’s prometheus receiver then scrapes those metrics and converts them to OTLP before forwarding to Last9.

OTLP Receiver for Application Metrics

The otlp receiver allows application services — such as Java services using io.nats — to send their own OTel data (traces, metrics, logs) through the same collector to Last9. Configure the Java app with:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Application Tracing

The opentelemetry-nats-java extension instruments the NATS Java client (io.nats:jnats) via the OTel Java agent. No changes to your application code — just add the extension JAR at startup:

java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.javaagent.extensions=/path/to/opentelemetry-nats-java-0.1.0.jar \
  -Dotel.service.name=my-service \
  -jar your-app.jar

Each publish/subscribe pair becomes a linked distributed trace in Last9:

Spans are linked via W3C traceparent header injected into the outgoing NATS message and extracted on receive. The full attribute set includes messaging.destination.name, messaging.operation.type, messaging.message.body.size, server.address, and messaging.client.id.

Key Metrics to Monitor

Server Throughput

Connections and Subscriptions

Cluster Routes

Available when -routez flag is passed and NATS is running in cluster mode.

Server Health

JetStream (if enabled)

Configuration Tips

prometheus-nats-exporter as systemd Service

sudo nano /etc/systemd/system/prometheus-nats-exporter.service

[Unit]
Description=Prometheus NATS Exporter
After=network.target

[Service]
ExecStart=/usr/local/bin/prometheus-nats-exporter -varz -connz -routez -subz -jsz=all http://localhost:8222
Restart=always

[Install]
WantedBy=multi-user.target

Monitoring Multiple NATS Servers

To monitor multiple NATS nodes, add each as a separate scrape target:

prometheus:
  config:
    scrape_configs:
      - job_name: nats
        scrape_interval: 15s
        static_configs:
          - targets:
              - "nats-node-1:7777"
              - "nats-node-2:7777"
              - "nats-node-3:7777"

Run a separate prometheus-nats-exporter instance per NATS node, each on a unique port.

Verification and Monitoring

1. Check Collector Service Status

   sudo systemctl status otelcol-contrib

2. Monitor Collector Logs

   sudo journalctl -u otelcol-contrib -f

3. Verify prometheus-nats-exporter

   curl http://localhost:7777/metrics | grep gnatsd

4. Verify NATS Monitoring Endpoint

   # Server variables
   curl http://localhost:8222/varz | jq '.connections, .in_msgs, .out_msgs'
   
   # Active connections
   curl http://localhost:8222/connz | jq '.num_connections'

5. Verify Data in Last9

   Log into your Last9 account and check that NATS metrics are appearing in Grafana. Search for gnatsd_varz_in_msgs to confirm data is flowing.

Troubleshooting

Exporter Cannot Reach NATS Monitoring Port

# Test connectivity
curl http://localhost:8222/healthz

# Verify NATS is listening on 8222
ss -tlnp | grep 8222

# Check NATS config has http_port set
grep http_port /path/to/nats-server.conf

No Metrics in Collector

# Validate config
sudo /usr/bin/otelcol-contrib --config /etc/otelcol-contrib/config.yaml --dry-run

# Check collector can reach exporter
curl http://localhost:7777/metrics

# View collector logs for scrape errors
sudo journalctl -u otelcol-contrib --no-pager | grep -i nats

Slow Consumers Growing

High gnatsd_varz_slow_consumers values indicate subscribers that can’t keep up with message rate. Check subscriber application performance and consider increasing NATS pending limits or scaling subscribers horizontally.

Need Help?

If you encounter any issues or have questions:

• Join our Discord community for real-time support
• Contact our support team at support@last9.io