# NATS

> Monitor NATS server metrics with OpenTelemetry and Last9 using the prometheus-nats-exporter

Source: https://last9.io/docs/integrations/messaging/nats/

Monitor NATS from two angles: server infrastructure metrics scraped via [prometheus-nats-exporter](https://github.com/nats-io/prometheus-nats-exporter), and distributed traces across your Java publish/subscribe operations via a zero-code OTel agent extension.

:::tip[Application Tracing]
The OTel Java agent doesn't include NATS instrumentation out of the box. The [opentelemetry-nats-java](https://github.com/last9/opentelemetry-nats-java) extension adds it — zero code changes required. Load it alongside the agent and every `publish()` becomes a PRODUCER span linked to its CONSUMER span via W3C trace context.

Available on [Maven Central](https://central.sonatype.com/artifact/io.last9/opentelemetry-nats-java). See the [auto-instrumentation demo](https://github.com/last9/opentelemetry-examples/tree/main/java/nats-auto) to run it end-to-end.
:::

## Prerequisites

Before setting up NATS monitoring, ensure you have:

- NATS server 2.x running with HTTP monitoring enabled (`http_port: 8222`)
- Administrative access to the server running the collector
- OpenTelemetry Collector installation access
- Last9 account with integration credentials

1.  **Enable NATS HTTP Monitoring**

    NATS exposes monitoring data over HTTP on port 8222. Add the following to your NATS server configuration:

    ```conf
    # nats-server.conf
    port: 4222
    http_port: 8222
    ```

    Restart the server and verify the endpoint is reachable:

    ```bash
    curl http://localhost:8222/varz
    ```

2.  **Install prometheus-nats-exporter**

    The prometheus-nats-exporter reads NATS monitoring endpoints and exposes them as Prometheus metrics. Install it on the same host as your NATS server or any host with network access to port 8222.

**Binary**

    ```bash
    wget https://github.com/nats-io/prometheus-nats-exporter/releases/latest/download/prometheus-nats-exporter-linux-amd64.zip
    unzip prometheus-nats-exporter-linux-amd64.zip
    sudo mv prometheus-nats-exporter /usr/local/bin/
    ```

**Docker**

    ```bash
    docker run -d --name nats-exporter \
      -p 7777:7777 \
      natsio/prometheus-nats-exporter:latest \
      -varz -connz -routez -subz -jsz=all \
      http://localhost:8222
    ```

    Start the exporter and verify it exposes metrics:

    ```bash
    # Start with common monitoring endpoints
    prometheus-nats-exporter -varz -connz -routez -subz -jsz=all http://localhost:8222

    # Verify metrics are available
    curl http://localhost:7777/metrics | grep gnatsd_varz_in_msgs
    ```

    Flags used above:

    - `-varz` — general server stats (messages, bytes, connections, CPU, memory)
    - `-connz` — per-connection stats (connection-level throughput and subscriptions)
    - `-routez` — cluster route stats (route throughput, pending bytes; relevant for clustered NATS)
    - `-subz` — subscription routing stats
    - `-jsz=all` — JetStream stats (if JetStream is enabled)

3.  **Install OpenTelemetry Collector**

**DEB Package**

    ```bash
    wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.118.0/otelcol-contrib_0.118.0_linux_amd64.deb
    sudo dpkg -i otelcol-contrib_0.118.0_linux_amd64.deb
    ```

**RPM Package**

    ```bash
    wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.118.0/otelcol-contrib_0.118.0_linux_amd64.rpm
    sudo rpm -ivh otelcol-contrib_0.118.0_linux_amd64.rpm
    ```

    More options: [OpenTelemetry Collector installation docs](https://opentelemetry.io/docs/collector/installation/#linux).

4.  **Configure OpenTelemetry Collector**

    Create the collector configuration file:

    ```bash
    sudo nano /etc/otelcol-contrib/config.yaml
    ```

    Add the following configuration. It scrapes NATS metrics from prometheus-nats-exporter and also accepts any OTLP data (traces, metrics, logs) from application services sending to port 4318:

    ```yaml
    receivers:
      # Scrapes NATS server metrics via prometheus-nats-exporter
      prometheus:
        config:
          scrape_configs:
            - job_name: nats
              scrape_interval: 15s
              static_configs:
                - targets: ["localhost:7777"]
                  labels:
                    service: nats-server

      # Accepts OTLP from application services (e.g., Java app on port 4318)
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

      hostmetrics:
        collection_interval: 60s
        scrapers:
          cpu:
          memory:
          disk:
          network:

    processors:
      batch:
        timeout: 20s
        send_batch_size: 10000
        send_batch_max_size: 10000
      resourcedetection/system:
        detectors: ["system"]
        system:
          hostname_sources: ["os"]
      resourcedetection/cloud:
        detectors: ["aws", "gcp", "azure"]

    exporters:
      otlp/last9:
        endpoint: "{{ .Logs.WriteURL }}"
        headers:
          "Authorization": "{{ .Logs.AuthValue }}"
      debug:
        verbosity: detailed

    service:
      pipelines:
        metrics:
          receivers: [prometheus, otlp, hostmetrics]
          processors: [batch, resourcedetection/system, resourcedetection/cloud]
          exporters: [otlp/last9]
        traces:
          receivers: [otlp]
          processors: [batch, resourcedetection/system]
          exporters: [otlp/last9]
        logs:
          receivers: [otlp]
          processors: [batch, resourcedetection/system]
          exporters: [otlp/last9]
    ```

    Replace the `exporters` placeholder values with your actual Last9 credentials from the Last9 Integrations page.

5.  **Start and Enable the Service**

    ```bash
    sudo systemctl daemon-reload
    sudo systemctl enable otelcol-contrib
    sudo systemctl start otelcol-contrib
    ```

## Understanding the Setup

### NATS HTTP Monitoring

NATS exposes a built-in HTTP monitoring server (default port 8222) with JSON endpoints for server stats, connections, subscriptions, and JetStream. This is the data source for all NATS metrics.

### prometheus-nats-exporter

The exporter translates NATS JSON monitoring endpoints into Prometheus-format metrics. The OTel Collector's `prometheus` receiver then scrapes those metrics and converts them to OTLP before forwarding to Last9.

### OTLP Receiver for Application Metrics

The `otlp` receiver allows application services — such as Java services using `io.nats` — to send their own OTel data (traces, metrics, logs) through the same collector to Last9. Configure the Java app with:

```bash
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

## Application Tracing

The [opentelemetry-nats-java](https://github.com/last9/opentelemetry-nats-java) extension instruments the NATS Java client (`io.nats:jnats`) via the OTel Java agent. No changes to your application code.

**Get the JAR**

**Gradle**

```kotlin
implementation("io.last9:opentelemetry-nats-java:0.1.0")
```

**Maven**

```xml
<dependency>
  <groupId>io.last9</groupId>
  <artifactId>opentelemetry-nats-java</artifactId>
  <version>0.1.0</version>
</dependency>
```

**Direct download**

Download from [GitHub Releases](https://github.com/last9/opentelemetry-nats-java/releases/latest).

**Start your app with the extension**

```bash
java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.javaagent.extensions=/path/to/opentelemetry-nats-java-0.1.0.jar \
  -Dotel.service.name=my-service \
  -jar your-app.jar
```

Each publish/subscribe pair becomes a linked distributed trace in Last9:

| Span                | Kind     | Trigger                    |
| ------------------- | -------- | -------------------------- |
| `{subject} publish` | PRODUCER | `connection.publish()`     |
| `{subject} process` | CONSUMER | Message handler invocation |

Spans are linked via W3C `traceparent` header injected into the outgoing NATS message and extracted on receive. The full attribute set includes `messaging.destination.name`, `messaging.operation.type`, `messaging.message.body.size`, `server.address`, and `messaging.client.id`.

## Key Metrics to Monitor

### Server Throughput

| Metric                  | Description                           |
| ----------------------- | ------------------------------------- |
| `gnatsd_varz_in_msgs`   | Total messages received by the server |
| `gnatsd_varz_out_msgs`  | Total messages sent by the server     |
| `gnatsd_varz_in_bytes`  | Total bytes received                  |
| `gnatsd_varz_out_bytes` | Total bytes sent                      |

### Connections and Subscriptions

| Metric                          | Description                                    |
| ------------------------------- | ---------------------------------------------- |
| `gnatsd_connz_num_connections`  | Current active client connections              |
| `gnatsd_varz_total_connections` | Total connections since server start           |
| `gnatsd_varz_slow_consumers`    | Slow consumer count (consumers falling behind) |
| `gnatsd_varz_subscriptions`     | Active subscription count                      |

### Cluster Routes

Available when `-routez` flag is passed and NATS is running in cluster mode.

| Metric                        | Description                            |
| ----------------------------- | -------------------------------------- |
| `gnatsd_routez_num_routes`    | Number of active routes in the cluster |
| `gnatsd_routez_in_msgs`       | Messages received on cluster routes    |
| `gnatsd_routez_out_msgs`      | Messages sent on cluster routes        |
| `gnatsd_routez_in_bytes`      | Bytes received on cluster routes       |
| `gnatsd_routez_out_bytes`     | Bytes sent on cluster routes           |
| `gnatsd_routez_pending_bytes` | Bytes pending on cluster routes        |

### Server Health

| Metric                        | Description                        |
| ----------------------------- | ---------------------------------- |
| `gnatsd_varz_mem`             | Server memory usage (bytes)        |
| `gnatsd_varz_cpu`             | Server CPU usage (fraction)        |
| `gnatsd_varz_uptime_seconds`  | Time since server start            |
| `gnatsd_varz_max_connections` | Configured max connections         |
| `gnatsd_varz_remotes`         | Currently connected remote servers |
| `gnatsd_varz_routes`          | Current route count (cluster)      |

### JetStream (if enabled)

| Metric                            | Description                  |
| --------------------------------- | ---------------------------- |
| `gnatsd_varz_jetstream_enabled`   | Whether JetStream is enabled |
| `gnatsd_varz_jetstream_memory`    | JetStream memory store usage |
| `gnatsd_varz_jetstream_storage`   | JetStream file store usage   |
| `gnatsd_varz_jetstream_streams`   | Number of active streams     |
| `gnatsd_varz_jetstream_consumers` | Number of active consumers   |

## Configuration Tips

### prometheus-nats-exporter as systemd Service

```bash
sudo nano /etc/systemd/system/prometheus-nats-exporter.service
```

```bash
[Unit]
Description=Prometheus NATS Exporter
After=network.target

[Service]
ExecStart=/usr/local/bin/prometheus-nats-exporter -varz -connz -routez -subz -jsz=all http://localhost:8222
Restart=always

[Install]
WantedBy=multi-user.target
```

### Monitoring Multiple NATS Servers

To monitor multiple NATS nodes, add each as a separate scrape target:

```yaml
prometheus:
  config:
    scrape_configs:
      - job_name: nats
        scrape_interval: 15s
        static_configs:
          - targets:
              - "nats-node-1:7777"
              - "nats-node-2:7777"
              - "nats-node-3:7777"
```

Run a separate prometheus-nats-exporter instance per NATS node, each on a unique port.

## Verification and Monitoring

1.  **Check Collector Service Status**

    ```bash
    sudo systemctl status otelcol-contrib
    ```

2.  **Monitor Collector Logs**

    ```bash
    sudo journalctl -u otelcol-contrib -f
    ```

3.  **Verify prometheus-nats-exporter**

    ```bash
    curl http://localhost:7777/metrics | grep gnatsd
    ```

4.  **Verify NATS Monitoring Endpoint**

    ```bash
    # Server variables
    curl http://localhost:8222/varz | jq '.connections, .in_msgs, .out_msgs'

    # Active connections
    curl http://localhost:8222/connz | jq '.num_connections'
    ```

5.  **Verify Data in Last9**

    Log into your Last9 account and check that NATS metrics are appearing in [Grafana](https://app.last9.io/grafana). Search for `gnatsd_varz_in_msgs` to confirm data is flowing.

---

## Troubleshooting

### Exporter Cannot Reach NATS Monitoring Port

```bash
# Test connectivity
curl http://localhost:8222/healthz

# Verify NATS is listening on 8222
ss -tlnp | grep 8222

# Check NATS config has http_port set
grep http_port /path/to/nats-server.conf
```

### No Metrics in Collector

```bash
# Validate config
sudo /usr/bin/otelcol-contrib --config /etc/otelcol-contrib/config.yaml --dry-run

# Check collector can reach exporter
curl http://localhost:7777/metrics

# View collector logs for scrape errors
sudo journalctl -u otelcol-contrib --no-pager | grep -i nats
```

### Slow Consumers Growing

High `gnatsd_varz_slow_consumers` values indicate subscribers that can't keep up with message rate. Check subscriber application performance and consider increasing NATS pending limits or scaling subscribers horizontally.

Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:support@last9.io) if you have any questions.