How to Exclude Health Check Endpoints from Python OTel Traces

You instrument a Python service with OpenTelemetry, set OTEL_TRACES_SAMPLER=always_on, and push traces to your backend. Within hours, your trace dashboard is 80% noise: identical /health-check spans generated every 30 seconds by your load balancer's liveness probe.

At 30-second intervals, that's 2,880 non-actionable spans per day — per service. In a fleet of 20 services, that's 57,600 health-check traces flooding your storage and making it harder to find real issues.

This post covers the two approaches to fix this, when each applies, and the subtle correctness issue that most implementations get wrong.

How Python OTel Sampling Works

If you haven't instrumented your Python service yet, start with OpenTelemetry Python instrumentation — this post assumes auto-instrumentation is already in place.

The OTel SDK makes a sampling decision at span creation time. If a sampler returns DROP, the span is created as a no-op — it exists in memory (so context propagation still works) but is never exported. For a deeper look at what spans contain, see OpenTelemetry Spans Explained.

OTEL_TRACES_SAMPLER=always_on tells the SDK to sample 100% of spans. That includes every HTTP request your app handles — useful endpoints and health checks alike.

There are two layers where you can intercept and drop spans:

1. Instrumentation layer — before the span is created (URL-based exclusion via env var)
2. Sampler layer — at span creation decision time (custom sampler logic)

Both approaches are zero-cost at export time: dropped spans are never serialized or sent.

Approach 1: OTEL_PYTHON_EXCLUDED_URLS

The simplest fix. The Python OTel instrumentation libraries check this env var before creating a span. If the incoming URL matches any listed pattern, no span is created.

export OTEL_PYTHON_EXCLUDED_URLS="health-check,ping,ready,live,metrics"

Key behavior: - Format: comma-separated regex patterns - Matching: re.search() against the full URL — partial matches apply - health-check matches /health-check, /api/health-check, /internal/health-check - Works with the opentelemetry-instrument CLI — zero code changes required

Framework-specific variants

If you have multiple Python services with different exclusion needs, you can scope exclusions per framework:

# FastAPI service
export OTEL_PYTHON_FASTAPI_EXCLUDED_URLS="health-check,ping,ready"

# Flask service
export OTEL_PYTHON_FLASK_EXCLUDED_URLS="health,live"

Framework-specific variants take precedence over the generic OTEL_PYTHON_EXCLUDED_URLS. This matters in monorepos where a single environment might have several services with different URL structures.

Using with opentelemetry-instrument

OTEL_PYTHON_EXCLUDED_URLS="health-check,ping,ready,live" \
OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.last9.io" \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64-credentials>" \
opentelemetry-instrument uvicorn app:app --host 0.0.0.0 --port 8000

No changes to your application code. The instrumentation CLI injects the OTel SDK at startup and checks OTEL_PYTHON_EXCLUDED_URLS before creating any span.

When to use this approach: Any case where "exclude these URL patterns entirely" is sufficient. This covers 90% of real-world health-check noise scenarios.

Limitation: You cannot express conditional logic — if you want to keep 5xx health-check spans (because a failing health check is actionable) while dropping 2xx ones, you need a custom sampler.

Approach 2: Custom Sampler

When you need logic that cannot be expressed as URL patterns: - Keep error spans on excluded paths (failing health checks matter) - Filter by tenant ID, user type, or custom span attributes - Combine URL exclusion with probabilistic sampling for high-traffic endpoints

The sampler

# sampler.py
from opentelemetry.sdk.trace.sampling import (
    Sampler, SamplingResult, Decision, ALWAYS_ON, ParentBased
)
from opentelemetry.context import Context
from opentelemetry.trace import SpanKind
from opentelemetry.trace.span import TraceState
from typing import Optional, Sequence

EXCLUDED_PATHS = {"/health-check", "/health", "/ping", "/ready", "/live", "/metrics"}


class DropNoisySpansSampler(Sampler):
    def should_sample(
        self,
        parent_context: Optional[Context],
        trace_id: int,
        name: str,
        kind: Optional[SpanKind] = None,
        attributes=None,
        links: Optional[Sequence] = None,
        trace_state: Optional[TraceState] = None,
    ) -> SamplingResult:
        if attributes:
            # OTel semconv v1: http.target | semconv v2: url.path
            path = attributes.get("http.target") or attributes.get("url.path") or ""
            status = (
                attributes.get("http.status_code")
                or attributes.get("http.response.status_code")
            )

            is_excluded = any(path.startswith(p) for p in EXCLUDED_PATHS)
            is_error = status is not None and int(status) >= 500

            # Keep error spans even on excluded paths — a failing health check is signal
            if is_excluded and not is_error:
                return SamplingResult(Decision.DROP)

        return ALWAYS_ON.should_sample(
            parent_context, trace_id, name, kind, attributes, links, trace_state
        )

    def get_description(self) -> str:
        return "DropNoisySpansSampler"


# ParentBased wrapping is required — see explanation below
sampler = ParentBased(root=DropNoisySpansSampler())

Why ParentBased is not optional

This is the correctness issue most implementations skip.

When a root span is dropped, the trace context still propagates downstream — services receive a traceId with sampled=false. Without ParentBased, child spans created within the same process (database queries, outbound HTTP calls) each get independently re-evaluated by DropNoisySpansSampler. Those child spans don't have http.target or url.path attributes, so the sampler falls through to ALWAYS_ON.

Result: orphaned child spans with no parent in your trace view.

ParentBased fixes this: child spans inherit the sampling decision from their parent context. A dropped root means all children are also dropped.

# Wrong: child spans re-evaluated independently → orphaned spans
sampler = DropNoisySpansSampler()

# Correct: child spans inherit DROP from root → clean trace view
sampler = ParentBased(root=DropNoisySpansSampler())

Wiring up — FastAPI

For a full FastAPI + OTel setup from scratch, see Integrating OpenTelemetry with FastAPI. The snippet below shows only the sampler wiring:

# main.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from sampler import sampler

# TracerProvider must be set BEFORE any instrumentation
provider = TracerProvider(sampler=sampler)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="https://otlp.last9.io",
            headers={"Authorization": "Basic <base64-credentials>"},
        )
    )
)
trace.set_tracer_provider(provider)

from fastapi import FastAPI

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)

Order matters: TracerProvider must be set before FastAPIInstrumentor.instrument_app(). If you call instrument_app() first, it captures the default no-op provider and your sampler is never used.

Wiring up — Flask

# app.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from flask import Flask
from sampler import sampler

provider = TracerProvider(sampler=sampler)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="https://otlp.last9.io",
            headers={"Authorization": "Basic <base64-credentials>"},
        )
    )
)
trace.set_tracer_provider(provider)

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

The same sampler.py works for both frameworks — the sampler logic is SDK-level, not framework-specific.

Using a custom sampler with opentelemetry-instrument

The opentelemetry-instrument CLI supports custom samplers via the OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG env vars, but only for samplers registered as entry points. For a project-local sampler, configure the provider in code instead and skip the CLI for sampler setup:

# Run normally — sampler is configured in app startup code
python main.py

Or use the CLI but set OTEL_TRACES_SAMPLER=always_on and let your in-process sampler override it:

OTEL_TRACES_SAMPLER=always_on \
OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.last9.io" \
opentelemetry-instrument python main.py

Since TracerProvider(sampler=sampler) is called explicitly in your code before any instrumentation, the OTEL_TRACES_SAMPLER env var is overridden by your explicit provider setup.

OTel Semantic Convention Attribute Names

One footgun: HTTP attribute names changed between OTel semconv v1 and v2.

Which set your instrumentation library uses depends on the version of opentelemetry-instrumentation-fastapi (or -flask) installed. Check the span attributes in your trace backend's detail view to verify which names are actually present before writing sampler logic that depends on them.

The custom sampler above checks both to stay version-agnostic:

path = attributes.get("http.target") or attributes.get("url.path") or ""
status = (
    attributes.get("http.status_code")
    or attributes.get("http.response.status_code")
)

Decision Guide

Exclude URLs only, no conditions needed?
  → OTEL_PYTHON_EXCLUDED_URLS — zero code, works immediately

Multiple services with different exclusion needs?
  → OTEL_PYTHON_FASTAPI_EXCLUDED_URLS / OTEL_PYTHON_FLASK_EXCLUDED_URLS

Keep error spans on excluded URLs (e.g., failing health checks)?
  → Custom sampler with status code check

Filter by non-URL attributes (tenant ID, user type, feature flag)?
  → Custom sampler

Both URL exclusion and probabilistic sampling?
  → Custom sampler combining DropNoisySpansSampler + TraceIdRatioBased

Working Example

Full runnable example with both approaches, including a FastAPI app and a Flask app: python/trace-filtering on GitHub

Wrapping Up

Health-check noise is one of the most common sources of trace storage bloat in production Python services. The two approaches cover different points on the complexity spectrum:

• OTEL_PYTHON_EXCLUDED_URLS for the common case — no code required
• Custom sampler for conditional logic — error spans on excluded paths, non-URL attributes, combined strategies

The ParentBased wrapper on any custom sampler is non-negotiable for correct behavior. Skip it and you get orphaned child spans that make the drop behavior worse than not filtering at all.

If you're sending Python traces to Last9, you can verify which spans are actually being exported by checking the trace detail view — the attribute names visible there confirm which semconv version your instrumentation is using, which matters when writing sampler attribute lookups.

References

• opentelemetry-util-http: excluded URLs implementation
• OTel Python sampling docs
• OTel HTTP semconv migration guide
• ParentBased sampler docs
• OpenTelemetry Python Instrumentation — Last9
• Integrating OpenTelemetry with FastAPI — Last9
• OpenTelemetry Context Propagation — Last9
• OpenTelemetry Spans Explained — Last9