OpenTelemetry Collector with Docker: A Detailed Guide

Monitoring and observability have become the backbone of reliable software systems. OpenTelemetry, a CNCF project, has gained immense traction as the go-to framework for collecting and exporting telemetry data.

But what makes it even more powerful is its Collector—a vendor-agnostic tool that simplifies data processing. Combine that with Docker, and you’ve got a robust, portable, and scalable observability solution.

This guide walks you through the nuances of using the OpenTelemetry Collector with Docker, helping you optimize observability workflows in modern systems.

What is the OpenTelemetry Collector?

The OpenTelemetry Collector is a pluggable component designed to collect, process, and export telemetry data—traces, metrics, and logs.

Unlike client-side libraries, the Collector operates as a standalone agent or gateway, enabling centralized telemetry data management.

Key Features

• Vendor Neutrality: Supports various backends, including Prometheus, Jaeger, and Elasticsearch.
• Scalability: Deployable as an agent (sidecar) or gateway in distributed environments.
• Custom Processing: Configurable pipelines for data enrichment and transformation.

Why Use Docker for OpenTelemetry Collector?

Docker provides a lightweight and portable environment to run the OpenTelemetry Collector.

It simplifies deployment and ensures consistent performance across diverse infrastructures—from local machines to cloud environments.

Benefits of Using Docker

1. Portability: Docker containers run consistently on different platforms.
2. Ease of Configuration: The Collector’s configuration can be encapsulated within Docker images.
3. Simplified Scaling: Running multiple instances becomes straightforward with Docker Compose or orchestration tools like Kubernetes.

Setting Up OpenTelemetry Collector with Docker

Here’s how to get started with the OpenTelemetry Collector in a Docker environment.

1. Prerequisites

• Docker installed on your system.
• Basic understanding of YAML for configuring the Collector.
• Access to a backend like Prometheus or Jaeger for testing.

2. Pull the Official Docker Image

Start by pulling the official OpenTelemetry Collector image:

docker pull otel/opentelemetry-collector

3. Create a Configuration File

The Collector’s behavior is governed by a configuration file. Below is an example YAML file for collecting metrics and exporting them to Prometheus:

receivers:
  otlp:
    protocols:
      grpc:
      http:
processors:
  batch:
exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

Save this file as otel-collector-config.yaml.

4. Run the Collector

Use the following command to run the Collector with the custom configuration:

docker run -p 8889:8889 \  
  -v $(pwd)/otel-collector-config.yaml:/etc/otel/config.yaml \  
  otel/opentelemetry-collector --config=/etc/otel/config.yaml

This command maps the configuration file to the Collector container and exposes the Prometheus endpoint at the port 8889.

Environment Setup for OpenTelemetry Collector

To get started with the OpenTelemetry Collector, you'll need to ensure your environment is set up properly.

Here are the steps to get everything ready:

1. Install Dependencies

The OpenTelemetry Collector depends on a few key tools, including Go and Docker. Follow these steps to install them:

Install Go

First, download and install Go from here. After installation, verify it by running the following command:

go version

You should see the installed version printed out. If not, follow the Go installation instructions specific to your OS.

Install Docker

If you're planning to run the OpenTelemetry Collector inside a container, you’ll also need Docker. Download and install Docker from here. Once Docker is installed, verify it by running:

docker --version

This will confirm that Docker is installed and ready to go.

2. Prepare Your System

It’s a good idea to ensure your system is updated and has the necessary tools. If you’re using a Linux-based system, you can run:

sudo apt-get update && sudo apt-get upgrade

For macOS, you can use Homebrew to update your packages:

brew update && brew upgrade

3. Set Up Configuration Files

The OpenTelemetry Collector uses a configuration file to specify how data is collected, processed, and exported.

Here's a basic example configuration file:

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  logging:
    loglevel: debug
  otlp:
    endpoint: "localhost:55680"
    compression: on

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [logging, otlp]

This example sets up a trace pipeline that uses the OTLP receiver to collect telemetry data and the logging exporter to output the data for debugging purposes. You can customize this file based on the telemetry data you want to collect and how you want to export it.

4. Running the Collector

Once everything is set up, you can start the OpenTelemetry Collector. If you're running it in Docker, here’s how to start it with the default configuration:

docker run -d --name=otel-collector --mount type=bind,source=$(pwd)/otel-collector-config.yaml,target=/otel-local-config.yaml \
  open-telemetry/opentelemetry-collector:latest --config /otel-local-config.yaml

This command runs the OpenTelemetry Collector in a Docker container and mounts your configuration file into the container. You can replace otel-collector-config.yaml it with your actual file name if you’ve customized it.

Now you’re all set! You can start collecting telemetry data and integrating it into your monitoring stack.

OpenTelemetry Collector Installation Options

There are several ways to install the OpenTelemetry Collector, each suitable for different environments.

Let’s explore the most popular ones:

1. Docker

Running the OpenTelemetry Collector in Docker is one of the easiest ways to get started. Here’s a simple way to install it using Docker:

Run with Docker

To run the OpenTelemetry Collector with Docker, execute the following command:

docker run -d --name=otel-collector \
  -v $(pwd)/otel-collector-config.yaml:/otel-local-config.yaml \
  open-telemetry/opentelemetry-collector:latest --config /otel-local-config.yaml

This command does the following:

• -d runs the container in detached mode.
• -v $(pwd)/otel-collector-config.yaml:/otel-local-config.yaml mounts your configuration file into the container.
• open-telemetry/opentelemetry-collector:latest specifies the latest OpenTelemetry Collector Docker image.
• --config /otel-local-config.yaml tells the container to use your configuration file.

If you don't have a configuration file, you can create one using the example from the "Environment Setup" section.

2. Docker Compose

If you're managing multiple containers or services, Docker Compose makes it easy to orchestrate the deployment of the OpenTelemetry Collector along with other services (like Prometheus, Jaeger, etc.).

Example docker-compose.yml:

version: '3'
services:
  opentelemetry-collector:
    image: open-telemetry/opentelemetry-collector:latest
    container_name: otel-collector
    volumes:
      - ./otel-collector-config.yaml:/otel-local-config.yaml
    command: --config /otel-local-config.yaml
    ports:
      - "55680:55680"  # OTLP receiver

This Compose file sets up the OpenTelemetry Collector container and maps the necessary ports and configuration files. To run it, just use the following command:

docker-compose up -d

3. Kubernetes

For production environments, Kubernetes is often the go-to platform for orchestrating services at scale. The OpenTelemetry Collector can be deployed to Kubernetes using the official Helm chart or by applying Kubernetes manifests.

Using Helm:

If you have Helm installed, you can quickly deploy the OpenTelemetry Collector with the following commands:

1. Add the OpenTelemetry Helm chart repository:

helm repo add openTelemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

2. Install the OpenTelemetry Collector:

helm install otel-collector openTelemetry/opentelemetry-collector

This command installs the OpenTelemetry Collector on your Kubernetes cluster using the default settings. You can modify the Helm chart's values file to customize the setup according to your needs.

Using Kubernetes Manifests:

Alternatively, you can deploy the OpenTelemetry Collector using raw Kubernetes manifests. Here’s an example of a simple manifest (otel-collector.yaml):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
        - name: otel-collector
          image: open-telemetry/opentelemetry-collector:latest
          volumeMounts:
            - name: config-volume
              mountPath: /otel-local-config.yaml
              subPath: otel-collector-config.yaml
          command:
            - "--config"
            - "/otel-local-config.yaml"
      volumes:
        - name: config-volume
          configMap:
            name: otel-collector-config

To apply the manifest, use the following command:

kubectl apply -f otel-collector.yaml

4. macOS and Windows

macOS (via Homebrew)

For macOS users, installing the OpenTelemetry Collector with Homebrew is quick and easy. Simply run:

brew install opentelemetry-collector

Once installed, you can start the collector using:

otelcontribcol --config /path/to/otel-collector-config.yaml

Windows

For Windows, you can install the OpenTelemetry Collector by downloading the latest release from the OpenTelemetry GitHub Releases page. After downloading the appropriate .zip file, extract it, and run the collector with your configuration file like so:

otelcontribcol.exe --config C:\path\to\otel-collector-config.yaml

Advanced OpenTelemetry: What You Need to Know

1. Docker Compose for Multi-Container Setups

For complex environments, Docker Compose simplifies managing multiple services. Here’s an example docker-compose.yml:

version: "3.9"
services:
  otel-collector:
    image: otel/opentelemetry-collector
    ports:
      - "8889:8889"
    volumes:
      - ./otel-collector-config.yaml:/etc/otel/config.yaml
    command: ["--config=/etc/otel/config.yaml"]

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

This setup runs both the OpenTelemetry Collector and Prometheus as services.

2. Environment Variable Substitution

For dynamic configurations, environment variables can be used. Update your YAML like this:

exporters:
  prometheus:
    endpoint: "${PROMETHEUS_ENDPOINT:0.0.0.0:8889}"

Pass the variable during runtime:

docker run -e PROMETHEUS_ENDPOINT=127.0.0.1:9090 ...

3. Security Best Practices

• Restrict Network Access: Use firewalls to limit access to the Collector’s endpoints.
• Use Non-Root Users: Configure Docker containers to run as non-root users for improved security.
• TLS Encryption: Secure OTLP endpoints with TLS certificates.

Common Issues and How to Fix Them

• Port Conflicts: Ensure the Collector’s ports are not already in use.
• Misconfigured Pipelines: Validate your YAML file using online tools or the Collector’s dry-run mode.
• Resource Constraints: Allocate sufficient memory and CPU for high-throughput environments.

Debugging Logs

Enable detailed logs for better insights:

docker run ... --log-level=debug

Use Cases

1. Centralized Logging: Aggregate logs from microservices and export them to Elasticsearch.
2. Metrics Processing: Convert OTLP metrics to Prometheus-compatible formats.
3. Trace Sampling: Apply sampling rules to reduce trace volume before exporting to Jaeger.

Conclusion

Using the OpenTelemetry Collector with Docker provides a flexible, portable, and scalable way to manage telemetry data.

Experiment with different configurations and exporters to find the setup that best fits your needs. Observability isn’t just about monitoring systems; it’s about understanding them. And tools like the OpenTelemetry Collector make that journey not just possible but efficient.