# Nginx

> Monitor NGINX web server with comprehensive metrics, access logs, and error logs using OpenTelemetry Collector

Source: https://last9.io/docs/integrations/web-and-cdn/nginx/

Monitor your NGINX web server and reverse proxy with comprehensive metrics and log collection using OpenTelemetry Collector. This integration provides visibility into server performance, request patterns, and error conditions.

## Prerequisites

- NGINX web server installed and running
- OpenTelemetry Collector
- Access to NGINX configuration files
- Last9 account with OTLP endpoint configured

## Features

- **Status Metrics**: Active connections, request rates, server status
- **Performance Metrics**: Request duration, response codes, upstream health
- **Access Logs**: Detailed request information and patterns
- **Error Logs**: Error conditions and debugging information
- **SSL/TLS Metrics**: Certificate status and SSL performance
- **Upstream Monitoring**: Backend server health and load balancing

## Installation

1.  **Install NGINX (if not already installed)**

**Ubuntu/Debian**

    ```bash
    sudo apt-get update
    sudo apt-get install -y nginx

    # Start and enable NGINX
    sudo systemctl start nginx
    sudo systemctl enable nginx
    ```

**CentOS/RHEL**

    ```bash
    sudo yum update -y
    sudo yum install -y nginx

    # Start and enable NGINX
    sudo systemctl start nginx
    sudo systemctl enable nginx
    ```

**Docker**

    ```bash
    # Run NGINX with custom configuration
    docker run -d \
      --name nginx-server \
      -p 80:80 \
      -p 8080:8080 \
      -v $(pwd)/nginx.conf:/etc/nginx/nginx.conf:ro \
      -v $(pwd)/logs:/var/log/nginx \
      nginx:latest
    ```

2.  **Install OpenTelemetry Collector**

**Package Manager**

    ```bash
    # Download and install collector
    wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.118.0/otelcol-contrib_0.118.0_linux_amd64.deb
    sudo dpkg -i otelcol-contrib_0.118.0_linux_amd64.deb
    ```

**Docker**

    ```bash
    docker pull otel/opentelemetry-collector-contrib:0.118.0
    ```

3.  **Enable NGINX Status Module**

    Create `/etc/nginx/conf.d/status.conf`:

    ```nginx
    server {
        listen 8080;
        server_name localhost;

        # Status endpoint for metrics
        location /nginx_status {
            stub_status on;
            access_log off;
            allow 127.0.0.1;
            allow ::1;
            # Add your collector's IP if running remotely
            # allow 10.0.0.100;
            deny all;
        }

        # Health check endpoint
        location /health {
            access_log off;
            return 200 "healthy\n";
            add_header Content-Type text/plain;
        }
    }
    ```

    Verify stub_status module is available:

    ```bash
    nginx -V 2>&1 | grep -o with-http_stub_status_module
    ```

4.  **Configure NGINX Logging**

    Update your main NGINX configuration (`/etc/nginx/nginx.conf`):

    ```nginx
    http {
        # Enhanced logging format for better observability
        log_format json_combined escape=json '{'
            '"timestamp": "$time_iso8601", '
            '"remote_addr": "$remote_addr", '
            '"remote_user": "$remote_user", '
            '"request": "$request", '
            '"status": $status, '
            '"body_bytes_sent": $body_bytes_sent, '
            '"request_time": $request_time, '
            '"upstream_response_time": "$upstream_response_time", '
            '"upstream_addr": "$upstream_addr", '
            '"http_referrer": "$http_referer", '
            '"http_user_agent": "$http_user_agent", '
            '"http_x_forwarded_for": "$http_x_forwarded_for", '
            '"host": "$host", '
            '"request_method": "$request_method", '
            '"request_uri": "$request_uri", '
            '"server_protocol": "$server_protocol", '
            '"ssl_protocol": "$ssl_protocol", '
            '"ssl_cipher": "$ssl_cipher"'
        '}';

        # Access logs with JSON format
        access_log /var/log/nginx/access.log json_combined;

        # Error logs
        error_log /var/log/nginx/error.log warn;

        # Your server configurations...
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
    }
    ```

5.  **Test and Reload NGINX**

    ```bash
    # Test configuration
    sudo nginx -t

    # Reload NGINX
    sudo systemctl reload nginx

    # Verify status endpoint
    curl http://localhost:8080/nginx_status
    ```

## OpenTelemetry Collector Configuration

Create `/etc/otelcol-contrib/config.yaml`:

```yaml
receivers:
  # NGINX metrics via stub_status
  nginx:
    endpoint: "http://localhost:8080/nginx_status"
    collection_interval: 30s

  # NGINX access logs
  filelog/nginx_access:
    include:
      - /var/log/nginx/access.log
    operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.timestamp
          layout: "2006-01-02T15:04:05-07:00"
      - type: move
        from: attributes.remote_addr
        to: attributes["client.ip"]
      - type: move
        from: attributes.request_method
        to: attributes["http.method"]
      - type: move
        from: attributes.status
        to: attributes["http.status_code"]
      - type: move
        from: attributes.request_time
        to: attributes["http.request_duration"]
      - type: move
        from: attributes.body_bytes_sent
        to: attributes["http.response_size"]
      - type: add
        field: attributes["log.source"]
        value: "nginx.access"
      - type: add
        field: attributes["service.name"]
        value: "nginx"

  # NGINX error logs
  filelog/nginx_error:
    include:
      - /var/log/nginx/error.log
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?P<level>\w+)\] (?P<pid>\d+)#(?P<tid>\d+): \*(?P<connection_id>\d+) (?P<message>.*)'
        timestamp:
          parse_from: attributes.timestamp
          layout: "2006/01/02 15:04:05"
      - type: severity_parser
        parse_from: attributes.level
        mapping:
          debug: debug
          info: info
          notice: info2
          warn: warn
          error: error
          crit: fatal
          alert: fatal
          emerg: fatal
      - type: add
        field: attributes["log.source"]
        value: "nginx.error"
      - type: add
        field: attributes["service.name"]
        value: "nginx"

processors:
  # Batch processing for efficiency
  batch:
    timeout: 10s
    send_batch_size: 1024
    send_batch_max_size: 2048

  # Memory limiter
  memory_limiter:
    limit_mib: 256
    spike_limit_mib: 64

  # Resource detection
  resourcedetection/system:
    detectors: ["system", "env"]
    timeout: 5s
    system:
      hostname_sources: ["os"]

  # Add resource attributes
  resource:
    attributes:
      - key: service.name
        value: nginx
        action: upsert
      - key: service.version
        from_attribute: nginx.version
        action: insert
      - key: deployment.environment
        value: production
        action: insert

  # Transform metrics to add useful attributes
  transform/nginx:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["server.type"], "nginx")
          - set(attributes["monitoring.tool"], "opentelemetry")

  # Filter out health checks from access logs
  filter/health_checks:
    logs:
      exclude:
        match_type: regexp
        bodies:
          - '.*"request_uri": "/health".*'
          - '.*"request_uri": "/nginx_status".*'

exporters:
  # Last9 OTLP exporter
  otlp/last9:
    endpoint: $last9_otlp_endpoint
    headers:
      Authorization: $last9_otlp_auth_header
    compression: gzip
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 30s
      max_elapsed_time: 300s

  # Debug exporter for troubleshooting
  debug:
    verbosity: basic
    sampling_initial: 5
    sampling_thereafter: 200

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]
  pipelines:
    # Metrics pipeline
    metrics:
      receivers: [nginx]
      processors:
        [
          memory_limiter,
          resourcedetection/system,
          resource,
          transform/nginx,
          batch,
        ]
      exporters: [otlp/last9]

    # Logs pipeline
    logs:
      receivers: [filelog/nginx_access, filelog/nginx_error]
      processors:
        [
          memory_limiter,
          resourcedetection/system,
          resource,
          filter/health_checks,
          batch,
        ]
      exporters: [otlp/last9]
```

## Docker Deployment

For containerized deployments:

**Docker Compose**

```yaml
# docker-compose.yml
version: "3.8"

services:
  nginx:
    image: nginx:latest
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080" # Status endpoint
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./conf.d:/etc/nginx/conf.d:ro
      - ./logs:/var/log/nginx
      - ./ssl:/etc/nginx/ssl:ro
    networks:
      - web
    labels:
      - "monitoring.service=nginx"

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.118.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-config.yaml:/etc/otel-collector-config.yaml:ro
      - ./logs:/var/log/nginx:ro
    environment:
      - LAST9_OTLP_ENDPOINT=$last9_otlp_endpoint
      - LAST9_OTLP_AUTH_HEADER=$last9_otlp_auth_header
    networks:
      - web
    depends_on:
      - nginx

networks:
  web:
    name: web
```

**Kubernetes**

```yaml
# nginx-monitoring.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    events {
        worker_connections 1024;
    }

    http {
        include       /etc/nginx/mime.types;
        default_type  application/octet-stream;

        log_format json_combined escape=json '{'
            '"timestamp": "$time_iso8601", '
            '"remote_addr": "$remote_addr", '
            '"request": "$request", '
            '"status": $status, '
            '"body_bytes_sent": $body_bytes_sent, '
            '"request_time": $request_time, '
            '"http_user_agent": "$http_user_agent"'
        '}';

        access_log /var/log/nginx/access.log json_combined;
        error_log /var/log/nginx/error.log warn;

        include /etc/nginx/conf.d/*.conf;
    }

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-with-monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - containerPort: 80
            - containerPort: 8080
          volumeMounts:
            - name: nginx-config
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
            - name: status-config
              mountPath: /etc/nginx/conf.d/status.conf
              subPath: status.conf
            - name: logs
              mountPath: /var/log/nginx

        - name: otel-collector
          image: otel/opentelemetry-collector-contrib:0.118.0
          command: ["--config=/etc/otel-collector-config.yaml"]
          volumeMounts:
            - name: collector-config
              mountPath: /etc/otel-collector-config.yaml
              subPath: config.yaml
            - name: logs
              mountPath: /var/log/nginx
          env:
            - name: LAST9_OTLP_ENDPOINT
              valueFrom:
                secretKeyRef:
                  name: last9-credentials
                  key: endpoint
            - name: LAST9_OTLP_AUTH_HEADER
              valueFrom:
                secretKeyRef:
                  name: last9-credentials
                  key: auth-header

      volumes:
        - name: nginx-config
          configMap:
            name: nginx-config
        - name: status-config
          configMap:
            name: nginx-status-config
        - name: collector-config
          configMap:
            name: otel-collector-config
        - name: logs
          emptyDir: {}
```

## Advanced Monitoring

### SSL/TLS Monitoring

Add SSL monitoring to your NGINX configuration:

```nginx
server {
    listen 443 ssl http2;
    server_name your-domain.com;

    ssl_certificate /etc/nginx/ssl/certificate.crt;
    ssl_certificate_key /etc/nginx/ssl/private.key;

    # SSL session information in logs
    log_format ssl_extended '$remote_addr - $remote_user [$time_local] '
                          '"$request" $status $body_bytes_sent '
                          '"$http_referer" "$http_user_agent" '
                          'ssl_protocol="$ssl_protocol" ssl_cipher="$ssl_cipher"';

    access_log /var/log/nginx/ssl_access.log ssl_extended;

    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}
```

### Upstream Health Monitoring

Configure upstream monitoring:

```nginx
upstream backend {
    least_conn;
    server backend1.example.com:8080 max_fails=3 fail_timeout=30s;
    server backend2.example.com:8080 max_fails=3 fail_timeout=30s;
    server backend3.example.com:8080 max_fails=3 fail_timeout=30s backup;
}

server {
    listen 80;

    location / {
        proxy_pass http://backend;
        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_set_header Host $host;
        proxy_set_header X-Upstream-Response-Time $upstream_response_time;
    }
}
```

### Rate Limiting Monitoring

```nginx
http {
    # Rate limiting zones
    limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m;
    limit_req_zone $binary_remote_addr zone=api:10m rate=100r/m;

    server {
        location /api/ {
            limit_req zone=api burst=20 nodelay;
            limit_req_status 429;

            # Log rate limit events
            access_log /var/log/nginx/rate_limit.log json_combined;

            proxy_pass http://api_backend;
        }
    }
}
```

## Custom Metrics

Add custom metric collection using NGINX Plus or third-party modules:

```yaml
# Add to otel-config.yaml
receivers:
  prometheus/nginx:
    config:
      scrape_configs:
        - job_name: "nginx-vts"
          static_configs:
            - targets: ["localhost:9113"]
          scrape_interval: 30s

  httpcheck/nginx:
    targets:
      - endpoint: http://localhost/health
        method: GET
    collection_interval: 60s
```

## Alerting and Dashboards

Example metrics to monitor:

- **nginx_connections_active**: Active client connections
- **nginx_connections_reading**: Connections reading request headers
- **nginx_connections_writing**: Connections writing responses
- **nginx_connections_waiting**: Idle connections waiting for requests
- **nginx_requests_total**: Total client requests
- **HTTP status codes**: 2xx, 3xx, 4xx, 5xx response patterns
- **Request duration**: Response time percentiles
- **Upstream response time**: Backend performance

## Troubleshooting

### Common Issues

1. **No metrics appearing:**

   - Verify stub_status module is enabled: `nginx -V 2>&1 | grep stub_status`
   - Check status endpoint accessibility: `curl localhost:8080/nginx_status`
   - Ensure collector has network access to NGINX

2. **Missing logs:**

   - Verify log file permissions: `ls -la /var/log/nginx/`
   - Check log format is valid JSON
   - Ensure collector can read log files

3. **High resource usage:**
   - Reduce collection frequency
   - Filter out high-volume endpoints
   - Optimize log format

### Debug Commands

```bash
# Check NGINX configuration
nginx -t

# Test status endpoint
curl -s http://localhost:8080/nginx_status

# Monitor logs in real-time
tail -f /var/log/nginx/access.log

# Check collector health
curl http://localhost:13133
```

## Performance Optimization

```nginx
# Optimize NGINX for monitoring
http {
    # Reduce log I/O
    access_log /var/log/nginx/access.log json_combined buffer=32k flush=5s;

    # Optimize worker processes
    worker_processes auto;
    worker_connections 1024;

    # Enable gzip compression
    gzip on;
    gzip_types text/plain application/json;
}
```

Your NGINX infrastructure will now provide comprehensive observability data to Last9, enabling detailed monitoring of web server performance, request patterns, and error conditions.