# Docker

> Monitor Docker containers comprehensively with OpenTelemetry Collector for metrics, logs, and container lifecycle events

Source: https://last9.io/docs/integrations/containers-and-k8s/docker/

Monitor Docker containers and containerized applications using Last9's OpenTelemetry endpoint. This integration collects comprehensive container metrics, logs, and lifecycle events from your Docker infrastructure.

## Prerequisites

- Docker and Docker Compose installed
- Containers logging to stdout/stderr
- Docker daemon with stats API access
- Last9 account with OTLP endpoint configured

## How It Works

- **OpenTelemetry Collector**: Collects metrics from Docker containers using the Docker Stats API
- **Logspout**: Captures logs from all Docker containers and forwards them to the OpenTelemetry Collector
- **Data Processing**: The collector processes, batches, and enriches the telemetry data
- **Export**: The processed data is sent to Last9 for visualization and analysis

## Configuration

1. **Create Docker Compose Configuration**

   Create `last9-docker-monitoring.yaml` for the monitoring stack:

   ```yaml
   version: "3.8"

   services:
     otel-collector:
       image: otel/opentelemetry-collector-contrib:0.144.0
       container_name: last9-otel-collector
       command:
         [
           "--config=/etc/otel-collector-config.yaml",
           "--feature-gates=transform.flatten.logs",
         ]
       volumes:
         - ./otel-config.yaml:/etc/otel-collector-config.yaml:ro
         - /var/run/docker.sock:/var/run/docker.sock:ro
         # Single host filesystem mount so the hostmetrics receiver can read
         # VM CPU / memory / disk / network stats. HOST_PROC / HOST_SYS /
         # HOST_ETC env vars below point each scraper at the right subdirectory.
         # Do NOT use split mounts (/proc:/hostfs/proc, /sys:/hostfs/sys, etc.) —
         # they cause hostmetrics to build paths like /hostfs/snap/core18 that
         # don't exist, producing ENOENT errors on every scrape.
         - /:/hostfs:ro,rslave
       ports:
         - "4317:4317" # OTLP gRPC receiver
         - "4318:4318" # OTLP HTTP receiver
         - "8888:8888" # Prometheus metrics
         - "8889:8889" # Health check endpoint
       restart: unless-stopped
       user: "0" # root user to access docker stats and /proc
       pid: host # so the process scraper sees host PIDs
       environment:
         - LOGSPOUT=ignore
         - HOST_PROC=/hostfs/proc
         - HOST_SYS=/hostfs/sys
         - HOST_ETC=/hostfs/etc
       networks:
         - last9_monitoring
       healthcheck:
         test:
           [
             "CMD",
             "wget",
             "--no-verbose",
             "--tries=1",
             "--spider",
             "http://localhost:8889/",
           ]
         interval: 30s
         timeout: 5s
         retries: 3
       labels:
         - "monitoring.last9=otel-collector"
         - "logging.disable=false"

     logspout:
       image: "gliderlabs/logspout:v3.2.14"
       container_name: last9-logspout
       volumes:
         - /etc/hostname:/etc/host_hostname:ro
         - /var/run/docker.sock:/var/run/docker.sock:ro
       command: syslog+tcp://otel-collector:2255
       depends_on:
         otel-collector:
           condition: service_healthy
       restart: unless-stopped
       environment:
         - LOGSPOUT_MULTICAST=true
         - BACKLOG=false
       networks:
         - last9_monitoring
       labels:
         - "monitoring.last9=logspout"

     # Optional: Cadvisor for additional container metrics
     cadvisor:
       image: gcr.io/cadvisor/cadvisor:v0.47.0
       container_name: last9-cadvisor
       privileged: true
       devices:
         - /dev/kmsg:/dev/kmsg
       volumes:
         - /:/rootfs:ro
         - /var/run:/var/run:rw
         - /sys:/sys:ro
         - /var/lib/docker/:/var/lib/docker:ro
         - /cgroup:/cgroup:ro
       ports:
         - "8080:8080"
       restart: unless-stopped
       networks:
         - last9_monitoring
       labels:
         - "monitoring.last9=cadvisor"

   networks:
     last9_monitoring:
       name: last9_monitoring
       driver: bridge
   ```

   :::note
   Both the OpenTelemetry Collector and Logspout read from the Docker socket (`/var/run/docker.sock`) — they do **not** need to share a Docker network with the containers being monitored to collect metrics and logs. The `last9_monitoring` network above only exists so Logspout can forward logs to the collector over syslog.

   You only need to attach the collector to additional networks if you plan to add receivers that actively scrape endpoints _inside_ a container — for example, `prometheus`, `redis`, `postgresql`, `mysql`, or `httpcheck` receivers — or if your application containers push OTLP data to the collector (see [Application Integration](#application-integration) below).
   :::

2. **Create OpenTelemetry Collector Configuration**

   Create `otel-config.yaml` with comprehensive Docker monitoring:

   ```yaml
   receivers:
     # Docker container metrics
     docker_stats:
       collection_interval: 30s
       timeout: 20s
       api_version: 1.40
       metrics:
         # CPU metrics
         container.cpu.usage.total:
           enabled: true
         container.cpu.usage.kernelmode:
           enabled: true
         container.cpu.usage.usermode:
           enabled: true
         container.cpu.throttling_data.periods:
           enabled: true
         container.cpu.throttling_data.throttled_periods:
           enabled: true
         container.cpu.throttling_data.throttled_time:
           enabled: true
         container.cpu.utilization:
           enabled: true
         container.cpu.percent:
           enabled: true

         # Memory metrics
         container.memory.usage.limit:
           enabled: true
         container.memory.usage.total:
           enabled: true
         container.memory.usage.max:
           enabled: true
         container.memory.percent:
           enabled: true
         container.memory.cache:
           enabled: true
         container.memory.rss:
           enabled: true
         container.memory.swap:
           enabled: true

         # Network metrics
         container.network.io.usage.rx_bytes:
           enabled: true
         container.network.io.usage.tx_bytes:
           enabled: true
         container.network.io.usage.rx_packets:
           enabled: true
         container.network.io.usage.tx_packets:
           enabled: true
         container.network.io.usage.rx_dropped:
           enabled: true
         container.network.io.usage.tx_dropped:
           enabled: true
         container.network.io.usage.rx_errors:
           enabled: true
         container.network.io.usage.tx_errors:
           enabled: true

         # Block I/O metrics
         container.blockio.io_service_bytes_recursive:
           enabled: true
         container.blockio.io_serviced_recursive:
           enabled: true

         # Process metrics
         container.pids.count:
           enabled: true
         container.pids.limit:
           enabled: true

     # Container logs via syslog from logspout
     tcplog/docker:
       listen_address: "0.0.0.0:2255"
       operators:
         - type: syslog_parser
           protocol: rfc5424

     # Host (VM / bare-metal) metrics. root_path: /hostfs + HOST_* env vars
     # point each scraper at the host filesystem rather than the collector
     # container's own /proc.
     hostmetrics:
       collection_interval: 60s
       root_path: /hostfs
       scrapers:
         cpu:
           metrics:
             system.cpu.logical.count:
               enabled: true
             system.cpu.utilization:
               enabled: true
         memory:
           metrics:
             system.memory.utilization:
               enabled: true
             system.memory.limit:
               enabled: true
         load:
         disk:
         filesystem:
           metrics:
             system.filesystem.utilization:
               enabled: true
           exclude_fs_types:
             fs_types:
               - squashfs
               - tmpfs
               - devtmpfs
               - overlay
               - nsfs
               - autofs
               - cgroup
               - cgroup2
               - proc
               - sysfs
             match_type: strict
           exclude_mount_points:
             mount_points:
               - /hostfs/sys/.*
               - /hostfs/proc/.*
               - /hostfs/dev/.*
               - /hostfs/run/.*
               - /hostfs/var/lib/docker/.*
               - /hostfs/snap/.*
               - /hostfs/boot.*
               - /var/lib/docker/.*
             match_type: regexp
         network:
         paging:
         processes:
         # mute_process_*_error: true silences "permission denied" noise for
         # kernel threads and short-lived processes. These flags do NOT cover
         # TOCTOU races — if a process exits between PID discovery and stat
         # read, /proc/<pid>/stat returns ENOENT and is still logged as error.
         # On busy hosts with many short-lived processes, remove the process:
         # scraper entirely and rely on docker_stats for container granularity.
         process:
           mute_process_user_error: true
           mute_process_io_error: true
           mute_process_exe_error: true
           metrics:
             process.cpu.utilization:
               enabled: true
             process.memory.utilization:
               enabled: true

     # Optional: Prometheus scraping for cadvisor
     prometheus/cadvisor:
       config:
         scrape_configs:
           - job_name: "cadvisor"
             static_configs:
               - targets: ["cadvisor:8080"]
             scrape_interval: 30s

   processors:
     # Transform docker logs
     transform/docker_logs:
       error_mode: ignore
       flatten_data: true
       log_statements:
         - context: log
           statements:
             - set(log.body, log.attributes["message"])
             - delete_key(log.attributes, "message")
             - set(resource.attributes["service.name"], log.attributes["appname"])
             - set(log.attributes["container.name"], log.attributes["appname"])
             - set(log.attributes["log.source"], "docker")

     # Add resource attributes for containers
     resource/docker:
       attributes:
         - key: monitoring.tool
           value: last9-otel-docker
           action: insert
         - key: deployment.environment
           from_attribute: docker.container.label.environment
           action: insert
         - key: service.version
           from_attribute: docker.container.label.version
           action: insert

     # Batch processing for performance
     batch:
       send_batch_size: 10000
       send_batch_max_size: 10000
       timeout: 10s

     # Memory limiter to prevent OOM
     memory_limiter:
       limit_mib: 512
       spike_limit_mib: 128

     # Resource detection
     resourcedetection/system:
       detectors: [env, system, docker]
       system:
         hostname_sources: ["os"]
       timeout: 5s
       override: false

     # Cloud-specific VM detectors. Each detector is a no-op on hosts that
     # aren't on the matching cloud, so it is safe to list all three — only
     # the one that matches the runtime environment will populate cloud.*
     # labels. ECS / EKS / AKS detectors are omitted because they require
     # Kubernetes service env vars or ECS task metadata.
     resourcedetection/cloud:
       detectors: [ec2, gcp, azure]
       timeout: 5s

     # Filter out sensitive containers
     filter/exclude_monitoring:
       metrics:
         exclude:
           match_type: regexp
           resource_attributes:
             - key: container.name
               value: "(last9-otel-collector|last9-logspout|last9-cadvisor)"

     # Promote container resource attributes to datapoint attributes so they
     # are guaranteed to land as filterable labels on metrics in Last9. The
     # docker_stats receiver sets container.name / container.image.name as
     # resource attributes; this transform copies them onto each datapoint.
     transform/container_labels:
       error_mode: ignore
       metric_statements:
         - context: datapoint
           statements:
             - set(datapoint.attributes["container.name"], resource.attributes["container.name"])
             - set(datapoint.attributes["container.image.name"], resource.attributes["container.image.name"])
             - set(datapoint.attributes["container.image.tag"], resource.attributes["container.image.tag"])
             - set(datapoint.attributes["container.id"], resource.attributes["container.id"])

     # Promote host + cloud resource attributes onto each host metric
     # datapoint so they appear as filterable labels in Last9.
     transform/hostmetrics:
       error_mode: ignore
       metric_statements:
         - context: datapoint
           statements:
             - set(datapoint.attributes["host.name"], resource.attributes["host.name"])
             - set(datapoint.attributes["host.type"], resource.attributes["host.type"])
             - set(datapoint.attributes["host.image.id"], resource.attributes["host.image.id"])
             - set(datapoint.attributes["cloud.provider"], resource.attributes["cloud.provider"])
             - set(datapoint.attributes["cloud.platform"], resource.attributes["cloud.platform"])
             - set(datapoint.attributes["cloud.region"], resource.attributes["cloud.region"])
             - set(datapoint.attributes["cloud.account.id"], resource.attributes["cloud.account.id"])
             - set(datapoint.attributes["cloud.availability_zone"], resource.attributes["cloud.availability_zone"])

   exporters:
     # Debug exporter for troubleshooting
     debug:
       verbosity: basic
       sampling_initial: 2
       sampling_thereafter: 500

     # Last9 OTLP exporter
     otlp/last9:
       endpoint: $last9_otlp_endpoint
       headers:
         Authorization: $last9_otlp_auth_header
       compression: gzip
       retry_on_failure:
         enabled: true
         initial_interval: 1s
         max_interval: 30s
         max_elapsed_time: 300s
       sending_queue:
         enabled: true
         num_consumers: 10
         queue_size: 5000

   extensions:
     # Health check extension
     health_check:
       endpoint: "0.0.0.0:8889"

     # Performance profiling
     pprof:
       endpoint: "0.0.0.0:1777"

     # Memory ballast for GC optimization
     memory_ballast:
       size_mib: 165

   service:
     extensions: [health_check, pprof, memory_ballast]

     pipelines:
       # Metrics pipeline
       metrics:
         receivers: [docker_stats, hostmetrics, prometheus/cadvisor]
         processors:
           [
             memory_limiter,
             resourcedetection/system,
             resourcedetection/cloud,
             resource/docker,
             filter/exclude_monitoring,
             transform/container_labels,
             transform/hostmetrics,
             batch,
           ]
         exporters: [otlp/last9]

       # Logs pipeline
       logs:
         receivers: [tcplog/docker]
         processors:
           [
             memory_limiter,
             transform/docker_logs,
             resourcedetection/system,
             resourcedetection/cloud,
             resource/docker,
             batch,
           ]
         exporters: [otlp/last9]
   ```

3. **Start the Monitoring Stack**

   ```bash
   # Start the monitoring services
   docker compose -f last9-docker-monitoring.yaml up -d

   # Check the status
   docker compose -f last9-docker-monitoring.yaml ps

   # View logs
   docker compose -f last9-docker-monitoring.yaml logs -f otel-collector
   ```

## Application Integration

### Environment Variables for Containers

Set these environment variables in your application containers for better monitoring:

```yaml
# docker-compose.yml for your applications
version: "3.8"

services:
  your-app:
    image: your-app:latest
    environment:
      # OpenTelemetry configuration for direct instrumentation
      - OTEL_SERVICE_NAME=your-app-service
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://last9-otel-collector:4317
      - OTEL_RESOURCE_ATTRIBUTES=service.version=1.0.0,deployment.environment=production
    labels:
      # Labels for container identification
      - "service.name=your-app-service"
      - "service.version=1.0.0"
      - "environment=production"
      - "component=api"
    networks:
      - your_app_network
      - last9_monitoring # Connect to monitoring network
    depends_on:
      - database

  database:
    image: postgres:15
    environment:
      - POSTGRES_DB=appdb
      - POSTGRES_USER=appuser
      - POSTGRES_PASSWORD=secret
    labels:
      - "service.name=postgres-db"
      - "component=database"
    networks:
      - your_app_network
      - last9_monitoring

networks:
  your_app_network:
    name: your_app_network
  last9_monitoring:
    external: true
    name: last9_monitoring
```

### Custom Log Formatting

For applications producing structured logs:

```dockerfile
# Dockerfile with structured logging
FROM node:18-alpine

WORKDIR /app
COPY package*.json ./
RUN npm ci --production

COPY . .

# Configure structured logging
ENV NODE_ENV=production
ENV LOG_FORMAT=json
ENV LOG_LEVEL=info

# OpenTelemetry configuration
ENV OTEL_SERVICE_NAME=nodejs-app
ENV OTEL_RESOURCE_ATTRIBUTES="service.version=1.0.0,deployment.environment=production"

CMD ["node", "server.js"]
```

### Health Check Integration

```yaml
# Add health checks to your services
services:
  your-app:
    image: your-app:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    labels:
      - "health.endpoint=/health"
      - "monitoring.enabled=true"
```

## Advanced Configuration

### Custom Metrics Collection

```yaml
# Add custom receivers to otel-config.yaml
receivers:
  # JMX metrics for Java applications
  jmx:
    jar_path: /opt/opentelemetry-jmx-metrics.jar
    endpoint: service:jmx:rmi:///jndi/rmi://java-app:9999/jmxrmi
    target_system: java
    collection_interval: 60s

  # StatsD metrics from applications
  statsd:
    endpoint: "0.0.0.0:8125"
    aggregation_interval: 60s

  # HTTP endpoint for custom metrics
  httpcheck:
    targets:
      - endpoint: http://your-app:3000/metrics
        method: GET
    collection_interval: 30s
```

### Log Enrichment

```yaml
# Enhanced log processing
processors:
  transform/enrich_logs:
    log_statements:
      - context: log
        statements:
          # Add container metadata
          - set(attributes["container.id"], resource.attributes["container.id"])
          - set(attributes["container.image"], resource.attributes["container.image.name"])

          # Parse JSON logs
          - merge_maps(cache, ParseJSON(body), "insert") where body != nil
          - set(body, cache["message"]) where cache["message"] != nil
          - set(attributes["log.level"], cache["level"]) where cache["level"] != nil
          - set(attributes["log.timestamp"], cache["timestamp"]) where cache["timestamp"] != nil
```

### Network Monitoring

```yaml
# Add network inspection
receivers:
  tcplog/network:
    listen_address: "0.0.0.0:2256"
    operators:
      - type: json_parser

processors:
  transform/network:
    log_statements:
      - context: log
        statements:
          - set(attributes["network.protocol"], "tcp")
          - set(attributes["monitoring.type"], "network")
```

## Kubernetes Integration

For Kubernetes environments, use a DaemonSet approach:

```yaml
# kubernetes-docker-monitoring.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: last9-otel-collector
  namespace: monitoring
spec:
  selector:
    matchLabels:
      name: last9-otel-collector
  template:
    metadata:
      labels:
        name: last9-otel-collector
    spec:
      serviceAccount: otel-collector
      containers:
        - name: otel-collector
          image: otel/opentelemetry-collector-contrib:0.118.0
          env:
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: LAST9_OTLP_ENDPOINT
              valueFrom:
                secretKeyRef:
                  name: last9-credentials
                  key: endpoint
            - name: LAST9_OTLP_AUTH_HEADER
              valueFrom:
                secretKeyRef:
                  name: last9-credentials
                  key: auth-header
          volumeMounts:
            - name: config
              mountPath: /etc/otel-collector-config.yaml
              subPath: config.yaml
            - name: docker-socket
              mountPath: /var/run/docker.sock
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: otel-collector-config
        - name: docker-socket
          hostPath:
            path: /var/run/docker.sock
```

## Monitoring Capabilities

This integration provides:

- **Container Metrics**: CPU, memory, network, disk I/O, process counts
- **Container Logs**: All stdout/stderr output with metadata
- **Host Metrics**: System-level resource usage
- **Container Lifecycle**: Start, stop, restart events
- **Network Traffic**: Inter-container communication patterns
- **Resource Limits**: Utilization vs. configured limits
- **Health Status**: Container and service health checks

## Best Practices

1. **Resource Management**: Set appropriate memory limits for collector
2. **Network Configuration**: Isolate monitoring traffic when possible
3. **Security**: Use read-only Docker socket access
4. **Retention**: Configure appropriate log retention policies
5. **Alerting**: Set up alerts for high resource usage or container failures

---

## Troubleshooting

- **No container metrics appearing.** Verify the Docker socket is mounted on the collector and readable, check the collector runs as `user: "0"` so it can read the socket, and confirm `docker_stats` is enabled in the metrics pipeline.

  ```bash
  docker exec last9-otel-collector ls -la /var/run/docker.sock
  ```

- **Missing logs.** Verify logspout can access the Docker socket, check containers are logging to stdout/stderr (logspout will not pick up file-based logs), and confirm Logspout and the collector are on the same Docker network so syslog (`syslog+tcp://otel-collector:2255`) is reachable.

- **High resource usage.** Adjust collection intervals, configure memory limits, and use batch processing. Example tuning for high-volume environments:

  ```yaml
  processors:
    batch:
      send_batch_size: 50000
      send_batch_max_size: 50000
      timeout: 30s

    memory_limiter:
      limit_mib: 1024
      spike_limit_mib: 256
  ```

- **Need to inspect collector behavior.** Enable debug logging and check connectivity:

  ```bash
  # Check collector logs
  docker logs last9-otel-collector -f

  # Check logspout logs
  docker logs last9-logspout -f

  # Test connectivity
  docker exec last9-otel-collector wget -qO- http://localhost:8889/
  ```

- **Prometheus scrape failing when target containers use `network_mode: host`.** `host.docker.internal` resolves to the Docker bridge gateway IP (e.g. `172.17.0.1`) on Linux — not loopback. Ubuntu's iptables/ufw blocks connections from bridged containers to host ports, so `extra_hosts: host.docker.internal:host-gateway` fixes DNS but not the firewall. Switch the collector to `network_mode: host`, remove the `networks:` block, and use `localhost:<port>` as the scrape target. Also change the logspout command from `syslog+tcp://otel-collector:2255` to `syslog+tcp://localhost:2255`.

- **`process:` scraper logs ENOENT errors on busy hosts.** Short-lived processes (kworkers, sshd children) exit between PID discovery and stat file read. `mute_process_*_error: true` does not cover this — it only suppresses permission-denied errors for `/exe`, `/io`, and user lookup. Remove the `process:` scraper on busy hosts; `docker_stats` already provides per-container CPU and memory.

- **Auth error 401.** The `Authorization` header for Last9's OTLP endpoint must use `Basic`, not `Bearer`. Use `Authorization: "Basic <token>"`.

Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:support@last9.io) if you have any questions.