Whitespace in OTLP headers and OpenTelemetry Python SDK

If you are using OpenTelemetry Python SDK and trying to send telemetry data to Last9, you may see the following warning in the logs:

(.venv) ➜  with-flask git:(flask-example) ✗ OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.last9.io" OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <header>" opentelemetry-instrument flask run
Header format invalid! Header values in environment variables must be URL encoded per the OpenTelemetry Protocol Exporter specification: Authorization=Basic <header>
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit

This means the telemetry data is not being sent to the backend.

Failed to export metrics to otlp.last9.io, error code: StatusCode.UNAUTHENTICATED

Note that you will run into this error not just with Last9 but any time you have whitespace in the value field of the OTLP headers. This may happen if your backend supports basic authorization or bearer authorization.

This warning arises from the following code when sending a http request:

    Parse ``s``, which is a ``str`` instance containing HTTP headers encoded
    for use in ENV variables per the W3C Baggage HTTP header format at
    https://www.w3.org/TR/baggage/#baggage-http-header-format, except that
    additional semi-colon delimited metadata is not supported.

To fix this, we need to escape the value part of the basic auth header as follows:

OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.last9.io" OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20<header>" opentelemetry-instrument flask run

Notice the addition of %20 above. It is important to note that only the value field needs to be escaped; the entire authorization header does not need to be escaped.

The reason for this is as follows. Otel uses the following spec for the OTLP headers.

baggage-string         =  list-member 0*179( OWS "," OWS list-member )
list-member            =  key OWS "=" OWS value *( OWS ";" OWS property )
property               =  key OWS "=" OWS value
property               =/ key OWS
key                    =  token ; as defined in RFC 7230, Section 3.2.6
value                  =  *baggage-octet
baggage-octet          =  %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
                          ; US-ASCII characters excluding CTLs,
                          ; whitespace, DQUOTE, comma, semicolon,
                          ; and backslash
OWS                    =  *( SP / HTAB ) ; optional white space, as defined in RFC 7230, Section 3.2.3

As we can see, only the value the field does not allow whitespace, double quotes, commas, semicolons, and backslashes. If we have any one of these characters in the value, then we need to escape the value.

More details on this behavior can be found in this GitHub issue and docs for the http client.

Interestingly, as mentioned in the above issue, this behavior differs from JavaScript Otel SDK, where we don't need to escape the value part of the header in similar situation.

How does OpenTelemetry handle HTTP header propagation for tracing?

OpenTelemetry handles HTTP header propagation for tracing through a mechanism called context propagation. Here's a concise overview of how it works:

1. Trace Context: OpenTelemetry uses the W3C Trace Context standard, which defines two HTTP headers:
   • traceparent: Contains the trace ID, parent span ID, and trace flags
   • tracestate: Allows vendors to add custom trace information
2. Injecting Headers: When an outgoing HTTP request is made, OpenTelemetry:
   • Retrieves the current trace context
   • Serializes it into the appropriate header format
   • Adds these headers to the outgoing request
3. Extracting Headers: When receiving an incoming HTTP request, OpenTelemetry:
   • Reads the trace context headers
   • Deserializes them to reconstruct the trace context
   • Uses this information to create a new span that's properly connected to the trace
4. Propagators: OpenTelemetry uses propagators to handle the injection and extraction of context data. The default is the W3C Trace Context propagator, but others can be configured.
5. Automatic Instrumentation: Many OpenTelemetry SDKs provide automatic instrumentation for popular HTTP clients and servers, handling propagation without manual coding.

This process ensures that trace information is passed between services, allowing for distributed tracing across microservices architectures.

Let's consider a scenario where Service A makes a request to Service B, which then makes a request to Service C. Here's how the headers might look at each step:

1. Service A initiates the trace and makes a request to Service B:

GET /api/data HTTP/1.1
Host: service-b.example.com
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: rojo=00f067aa0ba902b7

Breakdown:

• traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
  • 00: version
  • 0af7651916cd43dd8448eb211c80319c: trace ID
  • b7ad6b7169203331: parent span ID
  • 01: trace flags (sampling bit is set)
• tracestate: rojo=00f067aa0ba902b7: vendor-specific tracing info

2. Service B receives the request, processes it, and makes a request to Service C:

GET /api/process HTTP/1.1
Host: service-c.example.com
traceparent: 00-0af7651916cd43dd8448eb211c80319c-5bd66ef5a71a2af3-01
tracestate: rojo=00f067aa0ba902b7,congo=t61rcWkgMzE

Note the changes:

• The trace ID remains the same (0af7651916cd43dd8448eb211c80319c)
• The parent span ID has changed (5bd66ef5a71a2af3), representing the new span created by Service B
• A new key-value pair has been added to the tracestate

3. Service C receives the request from Service B:

It would see the same headers as in step 2, and if it made any further requests, it would generate a new parent span ID while keeping the same trace ID.

This example demonstrates how:

1. The trace ID remains consistent across the entire trace.
2. Each service generates a new parent span ID for outgoing requests.
3. The tracestate can accumulate vendor-specific information.
4. The sampling decision (last bit of traceparent) is propagated.

This propagation allows for distributed tracing, enabling you to follow a request as it moves through your microservices architecture, even across different hosts and services.