Managing OpenTelemetry Collector Configuration Across Multiple Clusters

This guide is part of the OpenTelemetry series. Earlier in the series, Deploying OpenTelemetry at Scale made the case for treating Collector configuration as production infrastructure. This guide picks up where that left off and shows the concrete workflow for keeping configurations consistent when you operate Collectors across many clusters and regions.

The patterns described here are the same ones running in production for dozens of Kubernetes clusters across multiple regions. Nothing here is specific to a particular IaC tool — the structure works equally well in Terraform, Helm + Helmfile, Kustomize, or a thin shell-script-on-top approach. A reference implementation in Terraform is shown for concreteness, with sidebars covering the Helm and Kustomize equivalents at each step.

The problem

A single Collector deployment is easy. The configuration is one YAML file, you kubectl apply, and you move on. Once you have more than two clusters or environments, the cracks appear:

• The same processor change has to land in five places, and one cluster always lags.
• A region-specific exporter endpoint or token sneaks into the base config and breaks staging.
• Someone edits a config map directly to debug an incident, and the change never makes it back to source.
• A new cluster takes a day to onboard because the workflow is “copy from another cluster, then fix the bits you forgot.”

The fix is not a new tool. It is structure: a single source of truth in git, a clean separation between the base configuration and per-cluster overrides, and a deterministic apply workflow that any engineer can run.

The mental model

Three things change across clusters, and only three:

1. Identity — which cluster, which region, which environment. These end up as resource attributes (k8s.cluster.name, deployment.environment) so telemetry is queryable per-cluster downstream.
2. Endpoints — where data flows out. Different regions write to different backends; auth tokens are per-cluster.
3. Local policy — the genuine deviations from the base. Common examples:
   • Resource limits (memory and CPU per cluster scale tier)
   • Sampling rate (different in staging vs production, or per high-cardinality service)
   • Pipeline composition (some clusters run logs-only, others run full traces + metrics + logs)
   • Filter or drop rules (skip noisy log files, exclude internal namespaces)
   • Extra processors specific to a cluster’s environment

What stays in the base (and never per-cluster): Receivers, batch sizes, default memory limits, common processors (resource attributes scaffolding, k8sattributes, transform rules that apply universally), exporter type and protocol, retry/backoff policy, common pipeline shape. If something belongs in every cluster, it belongs in the base. Per-cluster overrides are the exception, not the default.

The job of the config-management layer is to merge the variable parts into the base, render YAML, and apply.

Directory layout

This structure scales linearly: one directory per cluster. The _base/ directory holds the shared template and any common modules.

otel-config/
├── .sops.yaml                       # encryption rules for all secrets
├── _base/
│   ├── otel-collector.tpl           # canonical template (shared)
│   ├── variables.tf                 # variable schema
│   └── modules/                     # shared TF modules (or Helm charts, etc.)
├── production/
│   ├── us-east-1/
│   │   ├── platform/                # cluster-wide infra (RBAC, namespaces, operator)
│   │   ├── terraform.tfvars         # identity + endpoints
│   │   ├── overrides.tpl            # cluster-only deviations (optional)
│   │   ├── secrets.enc.yaml         # encrypted credentials
│   │   └── Makefile
│   ├── eu-west-1/
│   │   └── ...
│   └── ap-south-1/
│       └── ...
└── staging/
    └── alpha/
        └── ...

The Makefile in each cluster directory is short — typically 5 lines that delegate to a shared common.mk. It exists so that “apply this cluster’s config” is always make apply regardless of which directory you are in.

Two principles drive this layout:

• A cluster directory is self-contained. Everything needed to apply that cluster lives in its directory plus shared modules in _base/. No cross-cluster references.
• Region and environment are directory boundaries, not flags. production/us-east-1 and staging/us-east-1 are independent directories, not the same code parameterised by an --env flag. They can drift intentionally; they cannot drift accidentally.

Per-cluster variables

terraform.tfvars (or its equivalent in your tool) holds the three things that change across clusters:

# production/us-east-1/terraform.tfvars

cluster_name        = "prod-use1"
region              = "us-east-1"
deployment_env      = "production"

otlp_endpoint       = "https://otlp-aws-use1.example.com:443"
metrics_endpoint    = "https://write-aws-use1.example.com/api/v1/write"

# Local policy
collector_memory    = "2Gi"
sampling_rate       = 0.1
log_paths_include   = ["/var/log/pods/*/*/*.log"]
log_paths_exclude   = ["/var/log/pods/*/otel-collector/*.log"]

The base template reads these and renders the final Collector YAML.

Templating

A single otel-collector.tpl in _base/ defines the full pipeline shape. The same idea expresses cleanly in any major IaC tool:

# _base/otel-collector.tpl (excerpt)
processors:
  batch:
    timeout: 10s
    send_batch_size: 8192

  resource:
    attributes:
      - key: k8s.cluster.name
        value: ${cluster_name}
        action: upsert
      - key: deployment.environment
        value: ${deployment_env}
        action: upsert

exporters:
  otlphttp/last9:
    endpoint: ${otlp_endpoint}
    headers:
      Authorization: Basic ${otlp_token}

Terraform reference

# _base/main.tf
locals {
  collector_yaml = templatefile("${path.module}/otel-collector.tpl", {
    cluster_name     = var.cluster_name
    deployment_env   = var.deployment_env
    otlp_endpoint    = var.otlp_endpoint
    otlp_token       = data.sops_file.secrets.data["otlp_token"]
    # data.sops_file.secrets declared in the cluster directory's tenant.tf
  })
}

resource "kubernetes_config_map" "otel_collector" {
  metadata {
    name      = "otel-collector-config"
    namespace = "observability"
  }
  data = {
    "config.yaml" = local.collector_yaml
  }
  # Mount via your existing collector Deployment, DaemonSet, or the
  # OpenTelemetry Operator OpenTelemetryCollector CR — see the Operator
  # docs at https://opentelemetry.io/docs/kubernetes/operator/
}

Helm + Helmfile equivalent

# helmfile.yaml — one releases block, environments override values
environments:
  production-use1:
    values:
      - production/us-east-1/values.yaml
  production-euw1:
    values:
      - production/eu-west-1/values.yaml

releases:
  - name: otel-collector
    chart: open-telemetry/opentelemetry-collector
    values:
      - _base/values.yaml         # base, shared by all environments
      - {{ .Environment.Values | toYaml | nindent 6 }}

production/us-east-1/values.yaml carries only the per-cluster overrides (cluster name, endpoints, sampling rate). The base _base/values.yaml is identical across the fleet.

Kustomize equivalent

otel-config/
├── _base/
│   └── kustomization.yaml         # base manifests + common patches
└── production/
    └── us-east-1/
        ├── kustomization.yaml     # references ../../_base
        └── patches/
            └── collector-env.yaml # cluster-specific values

Each cluster overlay is a kustomization.yaml that pulls _base/ and layers cluster-specific patches.

For per-cluster deviations beyond the variable substitution, point the template at a cluster-local overrides.tpl and concatenate or merge before rendering. Keep this rare; the more clusters share the base, the easier rollouts get.

Secrets

Two principles, one concrete option.

Principle 1: never commit raw credentials. Tokens, certificates, private keys live in encrypted form in git or are fetched at apply time from a secrets manager.

Principle 2: decrypt at apply time, not at edit time. The CI/CD or local apply step has the decryption key. Engineers editing config do not need raw credentials in their working tree.

Concrete options that satisfy both: SOPS + KMS (works well with Terraform), HashiCorp Vault, External Secrets Operator (Kubernetes-native), AWS/GCP secret managers, sealed-secrets. Pick the one that fits your existing infrastructure.

A SOPS + AWS KMS example, since it requires the least machinery:

# .sops.yaml at the repo root
creation_rules:
  - path_regex: "production/.*\\.enc\\.yaml$"
    kms: "arn:aws:kms:us-east-1:123456789012:key/<prod-key-id>"
  - path_regex: "staging/.*\\.enc\\.yaml$"
    kms: "arn:aws:kms:us-east-1:123456789012:key/<staging-key-id>"

Encrypt a secrets file with sops -e plaintext-secrets.yaml > secrets.enc.yaml, then delete the plaintext version. Terraform reads secrets.enc.yaml via the sops provider; the raw value never lands in plan output or git history.

Apply workflow

Every cluster directory exposes the same three commands, regardless of underlying tool:

make plan      # render config, show diff against current cluster state
make diff      # post the diff somewhere reviewable (Slack, PR comment)
make apply     # apply after review

The Makefile is intentionally thin — it links to shared modules, initializes the IaC tool, and runs the right command. A typical example:

# production/us-east-1/Makefile
CLUSTER := us-east-1-prod
include ../../common.mk

common.mk holds the actual logic: symlink the shared _base/ files, initialize Terraform with a per-cluster remote state key, run plan/apply, post results.

The remote state key matters. Use a path that includes cluster: terraform/${cluster}/state.tfstate. State per cluster means a broken state file blocks one cluster, not the whole fleet.

Helm + Helmfile equivalent

helmfile -e production-use1 diff      # equivalent to make plan
helmfile -e production-use1 apply     # equivalent to make apply

Kustomize equivalent

kubectl diff  -k production/us-east-1   # equivalent to make plan
kubectl apply -k production/us-east-1   # equivalent to make apply

In all three cases, the working directory or environment selector identifies the cluster, and the apply target is the cluster’s kubeconfig.

Diff before apply

Apply is irreversible enough that previewing the change is mandatory. The workflow is the same shape across tools:

1. terraform plan -out=plan.bin (or helm diff, or kustomize build | kubectl diff -f -).
2. Convert to a readable summary. For Terraform: terraform show -json plan.bin > plan.json, then a small script extracts resource changes.
3. Post the summary to a Slack channel or as a PR comment.
4. A reviewer (or the engineer themselves, on small changes) approves before make apply runs.

This step catches the silent-config-drift problem: someone hand-edited a config map last week, and the next apply would silently revert it. The diff makes that visible.

Adding a new cluster

The cost of adding a cluster should scale with the genuine new work, not with the boilerplate. With this structure, a new cluster is a new directory:

1. cp -r production/us-east-1 production/eu-west-1 and update terraform.tfvars (region, endpoints) and secrets.enc.yaml (re-encrypt for the new region’s KMS key if needed).
2. Bring up cluster-level resources first: cd production/eu-west-1/platform && make apply — this creates the namespace, RBAC, and any operator the Collector depends on.
3. Then apply the Collector config: cd production/eu-west-1 && make plan (review the diff) and make apply.
4. Verify telemetry is flowing: query the new k8s.cluster.name value in your backend.

If onboarding takes longer than ~30 minutes, something in the structure is wrong. Either the base template carries too much cluster-specific logic (push it into per-cluster overrides), or apply requires manual steps outside the Makefile (fold them in).

Update propagation

A change to the base template — say, adding a new transform processor — rolls out by:

1. Editing _base/otel-collector.tpl.
2. Running make plan in each cluster directory to see the diff.
3. Applying in waves: staging first, then production one cluster at a time.

The plan output is the safety net. If the change has unintended consequences in one cluster (the new processor relies on an attribute the cluster doesn’t emit), the diff shows it before apply. The cluster can pin its base version until the issue is fixed, or carry a small override.

Backward-incompatible base changes

When you remove or rename a base component (drop a processor, change an exporter type, restructure a pipeline), per-cluster overrides referencing the old shape break. Three protections:

• Run make plan across the entire fleet before merging the base change. The PR shows the diff for every cluster, not just the one you tested. If five clusters’ plans turn red, hold the merge.
• Version the base for high-risk transitions. Keep _base/v1/ and _base/v2/ parallel during migration, point each cluster at one version via tfvars, migrate cluster-by-cluster, then delete the old base.
• Rollback is git revert + make apply per cluster. Treat the base template like any other production code — atomic, reversible, fast to recover.

What to avoid

Six patterns that look reasonable but cost weeks of debugging at scale:

• One giant config map shared across clusters. Saves “duplication” but means any change risks breaking everyone. The duplication is the point — it gives you per-cluster safety.
• Hand-applied changes to debug an incident, not folded back into source. The cluster will revert on the next apply and you will lose the fix. Always update source first, then apply.
• A separate workflow for “small” changes. “Just a one-liner” is how config drift starts. Every change goes through plan-diff-apply, even when the diff is two lines.
• Storing the base template alongside application code. Coupling means an app PR can change Collector behaviour fleet-wide. Keep otel-config/ in its own repo with its own review and CI.
• Concurrent apply across all clusters. Hits cloud API rate limits, creates ingestion storms on the backend, makes incidents harder to attribute. Apply sequentially or in small waves; staging always before production.
• A single shared remote state across clusters. One broken state file blocks the entire fleet. Use a separate state key per cluster.

Going further

Once the base structure is stable, five extensions are worth investing in:

• CI/CD on make plan. A PR that touches production/* automatically generates a plan and posts the diff as a comment. Reviewers see the actual change, not just the source diff.
• Drift detection. A scheduled job runs terraform plan against every cluster and reports any non-zero diff. Catches hand-edits before the next intentional apply does.
• Canary cluster for base changes. Designate one staging cluster as the canary; new base changes apply there first and bake for 24h before fleet rollout. Catches regressions that pass terraform plan but break at runtime.
• otelcol validate in CI. Before any plan runs, validate the rendered YAML against the Collector binary’s schema (see How to debug OpenTelemetry pipelines). Catches typos, removed components, and bad processor names before they reach a cluster.
• GitOps reconciliation (ArgoCD or Flux). An alternative to imperative make apply — the cluster pulls the desired state from git on a loop. Trades immediate-apply control for automatic drift correction. Works well with the same directory structure; replace the Makefile with an ArgoCD Application per cluster pointing at production/${cluster}/.

These are not necessary on day one. They become valuable once you have more than 5–10 clusters, when manual review starts missing things.

Related

• Deploying OpenTelemetry at Scale: production patterns that work — the strategic patterns this guide implements.
• The OpenTelemetry Collector deep dive — what the components in otel-collector.tpl actually do.
• How to debug OpenTelemetry pipelines — for when the apply lands but the data doesn’t.