# How to Exclude Health Check Endpoints from Python OTel Traces

> Health check endpoints generate thousands of identical, useless spans per day. Here are two production-ready approaches to filter them from your Python OTel traces — and the correctness trap most implementations miss.

Source: https://last9.io/blog/python-otel-exclude-health-check-traces/

You instrument a Python service with OpenTelemetry, set `OTEL_TRACES_SAMPLER=always_on`, and push traces to your backend. Within hours, your trace dashboard is 80% noise: identical `/health-check` spans generated every 30 seconds by your load balancer's liveness probe.

At 30-second intervals, that's 2,880 non-actionable spans per day — per service. In a fleet of 20 services, that's 57,600 health-check traces flooding your storage and making it harder to find real issues.

This post covers the two approaches to fix this, when each applies, and the subtle correctness issue that most implementations get wrong.

---

## How Python OTel Sampling Works

If you haven't instrumented your Python service yet, start with [OpenTelemetry Python instrumentation](https://last9.io/blog/opentelemetry-python-instrumentation/) — this post assumes auto-instrumentation is already in place.

The OTel SDK makes a sampling decision at span creation time. If a sampler returns `DROP`, the span is created as a no-op — it exists in memory (so context propagation still works) but is never exported. For a deeper look at what spans contain, see [OpenTelemetry Spans Explained](https://last9.io/blog/opentelemetry-spans-events/).

`OTEL_TRACES_SAMPLER=always_on` tells the SDK to sample 100% of spans. That includes every HTTP request your app handles — useful endpoints and health checks alike.

There are two layers where you can intercept and drop spans:

1.  **Instrumentation layer** — before the span is created (URL-based exclusion via env var)
2.  **Sampler layer** — at span creation decision time (custom sampler logic)

Both approaches are zero-cost at export time: dropped spans are never serialized or sent.

---

## Approach 1: `OTEL_PYTHON_EXCLUDED_URLS`

The simplest fix. The Python OTel instrumentation libraries check this env var before creating a span. If the incoming URL matches any listed pattern, no span is created.

```bash
export OTEL_PYTHON_EXCLUDED_URLS="health-check,ping,ready,live,metrics"
```

Key behavior: - **Format**: comma-separated regex patterns - **Matching**: `re.search()` against the full URL — partial matches apply - `health-check` matches `/health-check`, `/api/health-check`, `/internal/health-check` - Works with the `opentelemetry-instrument` CLI — zero code changes required

### Framework-specific variants

If you have multiple Python services with different exclusion needs, you can scope exclusions per framework:

```bash
# FastAPI service
export OTEL_PYTHON_FASTAPI_EXCLUDED_URLS="health-check,ping,ready"

# Flask service
export OTEL_PYTHON_FLASK_EXCLUDED_URLS="health,live"
```

Framework-specific variants take precedence over the generic `OTEL_PYTHON_EXCLUDED_URLS`. This matters in monorepos where a single environment might have several services with different URL structures.

### Using with `opentelemetry-instrument`

```bash
OTEL_PYTHON_EXCLUDED_URLS="health-check,ping,ready,live" \
OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.last9.io" \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64-credentials>" \
opentelemetry-instrument uvicorn app:app --host 0.0.0.0 --port 8000
```

No changes to your application code. The instrumentation CLI injects the OTel SDK at startup and checks `OTEL_PYTHON_EXCLUDED_URLS` before creating any span.

**When to use this approach**: Any case where "exclude these URL patterns entirely" is sufficient. This covers 90% of real-world health-check noise scenarios.

**Limitation**: You cannot express conditional logic — if you want to keep 5xx health-check spans (because a failing health check _is_ actionable) while dropping 2xx ones, you need a custom sampler.

---

## Approach 2: Custom Sampler

When you need logic that cannot be expressed as URL patterns: - Keep error spans on excluded paths (failing health checks matter) - Filter by tenant ID, user type, or custom span attributes - Combine URL exclusion with probabilistic sampling for high-traffic endpoints

### The sampler

```python
# sampler.py
from opentelemetry.sdk.trace.sampling import (
    Sampler, SamplingResult, Decision, ALWAYS_ON, ParentBased
)
from opentelemetry.context import Context
from opentelemetry.trace import SpanKind
from opentelemetry.trace.span import TraceState
from typing import Optional, Sequence

EXCLUDED_PATHS = {"/health-check", "/health", "/ping", "/ready", "/live", "/metrics"}


class DropNoisySpansSampler(Sampler):
    def should_sample(
        self,
        parent_context: Optional[Context],
        trace_id: int,
        name: str,
        kind: Optional[SpanKind] = None,
        attributes=None,
        links: Optional[Sequence] = None,
        trace_state: Optional[TraceState] = None,
    ) -> SamplingResult:
        if attributes:
            # OTel semconv v1: http.target | semconv v2: url.path
            path = attributes.get("http.target") or attributes.get("url.path") or ""
            status = (
                attributes.get("http.status_code")
                or attributes.get("http.response.status_code")
            )

            is_excluded = any(path.startswith(p) for p in EXCLUDED_PATHS)
            is_error = status is not None and int(status) >= 500

            # Keep error spans even on excluded paths — a failing health check is signal
            if is_excluded and not is_error:
                return SamplingResult(Decision.DROP)

        return ALWAYS_ON.should_sample(
            parent_context, trace_id, name, kind, attributes, links, trace_state
        )

    def get_description(self) -> str:
        return "DropNoisySpansSampler"


# ParentBased wrapping is required — see explanation below
sampler = ParentBased(root=DropNoisySpansSampler())
```

### Why `ParentBased` is not optional

This is the correctness issue most implementations skip.

When a root span is dropped, the [trace context](https://last9.io/blog/opentelemetry-context-propagation/) still propagates downstream — services receive a `traceId` with `sampled=false`. Without `ParentBased`, child spans created within the same process (database queries, outbound HTTP calls) each get independently re-evaluated by `DropNoisySpansSampler`. Those child spans don't have `http.target` or `url.path` attributes, so the sampler falls through to `ALWAYS_ON`.

Result: orphaned child spans with no parent in your trace view.

`ParentBased` fixes this: child spans inherit the sampling decision from their parent context. A dropped root means all children are also dropped.

```python
# Wrong: child spans re-evaluated independently → orphaned spans
sampler = DropNoisySpansSampler()

# Correct: child spans inherit DROP from root → clean trace view
sampler = ParentBased(root=DropNoisySpansSampler())
```

### Wiring up — FastAPI

For a full FastAPI + OTel setup from scratch, see [Integrating OpenTelemetry with FastAPI](https://last9.io/blog/integrating-opentelemetry-with-fastapi/). The snippet below shows only the sampler wiring:

```python
# main.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from sampler import sampler

# TracerProvider must be set BEFORE any instrumentation
provider = TracerProvider(sampler=sampler)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="https://otlp.last9.io",
            headers={"Authorization": "Basic <base64-credentials>"},
        )
    )
)
trace.set_tracer_provider(provider)

from fastapi import FastAPI

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)
```

Order matters: `TracerProvider` must be set before `FastAPIInstrumentor.instrument_app()`. If you call `instrument_app()` first, it captures the default no-op provider and your sampler is never used.

### Wiring up — Flask

```python
# app.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from flask import Flask
from sampler import sampler

provider = TracerProvider(sampler=sampler)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="https://otlp.last9.io",
            headers={"Authorization": "Basic <base64-credentials>"},
        )
    )
)
trace.set_tracer_provider(provider)

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
```

The same `sampler.py` works for both frameworks — the sampler logic is SDK-level, not framework-specific.

### Using a custom sampler with `opentelemetry-instrument`

The `opentelemetry-instrument` CLI supports custom samplers via the `OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` env vars, but only for samplers registered as entry points. For a project-local sampler, configure the provider in code instead and skip the CLI for sampler setup:

```bash
# Run normally — sampler is configured in app startup code
python main.py
```

Or use the CLI but set `OTEL_TRACES_SAMPLER=always_on` and let your in-process sampler override it:

```bash
OTEL_TRACES_SAMPLER=always_on \
OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.last9.io" \
opentelemetry-instrument python main.py
```

Since `TracerProvider(sampler=sampler)` is called explicitly in your code before any instrumentation, the `OTEL_TRACES_SAMPLER` env var is overridden by your explicit provider setup.

---

## OTel Semantic Convention Attribute Names

One footgun: HTTP attribute names changed between OTel semconv v1 and v2.

| Attribute    | Semconv v1         | Semconv v2                  |
| ------------ | ------------------ | --------------------------- |
| Request path | `http.target`      | `url.path`                  |
| Status code  | `http.status_code` | `http.response.status_code` |
| Method       | `http.method`      | `http.request.method`       |

Which set your instrumentation library uses depends on the version of `opentelemetry-instrumentation-fastapi` (or `-flask`) installed. Check the span attributes in your trace backend's detail view to verify which names are actually present before writing sampler logic that depends on them.

The custom sampler above checks both to stay version-agnostic:

```python
path = attributes.get("http.target") or attributes.get("url.path") or ""
status = (
    attributes.get("http.status_code")
    or attributes.get("http.response.status_code")
)
```

---

## Decision Guide

```
Exclude URLs only, no conditions needed?
  → OTEL_PYTHON_EXCLUDED_URLS — zero code, works immediately

Multiple services with different exclusion needs?
  → OTEL_PYTHON_FASTAPI_EXCLUDED_URLS / OTEL_PYTHON_FLASK_EXCLUDED_URLS

Keep error spans on excluded URLs (e.g., failing health checks)?
  → Custom sampler with status code check

Filter by non-URL attributes (tenant ID, user type, feature flag)?
  → Custom sampler

Both URL exclusion and probabilistic sampling?
  → Custom sampler combining DropNoisySpansSampler + TraceIdRatioBased
```

---

## Working Example

Full runnable example with both approaches, including a FastAPI app and a Flask app: [python/trace-filtering on GitHub](https://github.com/last9/l9_otel_examples/tree/main/python/trace-filtering)

---

## Wrapping Up

Health-check noise is one of the most common sources of trace storage bloat in production Python services. The two approaches cover different points on the complexity spectrum:

- `OTEL_PYTHON_EXCLUDED_URLS` for the common case — no code required
- Custom sampler for conditional logic — error spans on excluded paths, non-URL attributes, combined strategies

The `ParentBased` wrapper on any custom sampler is non-negotiable for correct behavior. Skip it and you get orphaned child spans that make the drop behavior worse than not filtering at all.

If you're sending Python traces to [Last9](https://last9.io), you can verify which spans are actually being exported by checking the trace detail view — the attribute names visible there confirm which semconv version your instrumentation is using, which matters when writing sampler attribute lookups.

---

## References

- [opentelemetry-util-http: excluded URLs implementation](https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/util/opentelemetry-util-http/src/opentelemetry/util/http/__init__.py)
- [OTel Python sampling docs](https://opentelemetry-python.readthedocs.io/en/latest/sdk/trace.sampling.html)
- [OTel HTTP semconv migration guide](https://opentelemetry.io/docs/specs/semconv/http/migration-guide/)
- [ParentBased sampler docs](https://opentelemetry.io/docs/concepts/sampling/)
- [OpenTelemetry Python Instrumentation — Last9](https://last9.io/blog/opentelemetry-python-instrumentation/)
- [Integrating OpenTelemetry with FastAPI — Last9](https://last9.io/blog/integrating-opentelemetry-with-fastapi/)
- [OpenTelemetry Context Propagation — Last9](https://last9.io/blog/opentelemetry-context-propagation/)
- [OpenTelemetry Spans Explained — Last9](https://last9.io/blog/opentelemetry-spans-events/)