Identify Root Spans in Otel Collector

As a developer who's spent countless hours working with OpenTelemetry (OTel), I've come to appreciate the power of effective span filtering and manipulation. Today, I want to share my experience and insights on a crucial aspect of distributed tracing: identifying and working with root spans in OTel's filter and transform processors.

Understanding Root Spans in OpenTelemetry

Before we dive into the nitty-gritty of filtering and transforming, let's clarify what a root span is. In the context of distributed tracing, a root span is the topmost span in a trace hierarchy. It represents the entire operation from start to finish, while child spans represent sub-operations within that trace.

As someone who's debugged my fair share of complex distributed systems, I can't stress enough how important it is to identify root spans correctly. They provide a high-level view of your system's performance and are often the starting point for in-depth analysis.

The Challenge of Root Span Identification

When I first started working with OTel, I quickly realized that identifying root spans wasn't always straightforward. In OTel, spans are related through parent-child relationships, and the root span is the one without a parent.

Filter Processor: Identifying Root Spans

The filter processor is a powerful tool for selecting specific spans based on various criteria. Let's explore how to use it to identify root spans.

For OpenTelemetry versions 0.104.0 and above

In newer versions of OpenTelemetry (0.104.0+), there's a straightforward way to identify root spans:

filter/root_spans:
  error_mode: ignore
  traces:
    span:
      - 'IsRootSpan() == true'

Pull request where isRootSpan was added.

This filter uses the IsRootSpan() function, which returns true for root spans and false for non-root spans.

isRootSpan is a OTTL converter function which returns true if the span in the corresponding context is root, that means its parent_span_id equals to hexadecimal representation of zero. In all other scenarios function returns false.

For older versions of OpenTelemetry

If you're working with an older version of OTel, you can use the parent span ID method:

filter/root_spans:
  error_mode: ignore
  traces:
    span:
      - 'parent_span_id.string == "0000000000000000"'

Complex Filtering Example

Here's a more complex example that combines filtering for root spans with additional span attributes:

filter/important_root_spans:
  error_mode: ignore
  traces:
    span:
      - 'IsRootSpan() == true and attributes["http.method"] == "POST" and attributes["service.name"] == "order-processing"'

This filter selects root spans from the order-processing service that represent POST requests.

Transform Processor: Advanced Span Manipulation

While the filter processor is great for selecting spans, the transform processor allows us to modify spans based on conditions. It's built on OTTL (OpenTelemetry Transformation Language) functions, providing powerful capabilities for span manipulation.

Understanding OTTL Functions

OTTL is a domain-specific language designed for OpenTelemetry transformations. It provides a set of functions that allow you to access and modify various aspects of spans, metrics, and logs.

Example: Using the Transform Processor for Root Span Identification

Here's an example of how we can use the transform processor to add a custom attribute to root spans:

processors:
  transform:
    traces:
      span:
        - context: span
          statements:
            - set(attributes["is_root"] = IsRootSpan())

This configuration:

1. Applies to all spans (context: span).
2. Uses the IsRootSpan() OTTL function to check if the span is a root span.
3. Sets a new attribute called is_root with the result of IsRootSpan().

After this transformation, all spans will have an is_root attribute that's true for root spans and false for non-root spans.

Performance Considerations

While both filter and transform processors are powerful, it's important to use them judiciously. Here are some tips:

1. Keep filter conditions and OTTL statements as simple as possible.
2. Apply transformations only when necessary. Use the filter processor to reduce the number of spans before applying transformations.
3. Consider batch processing for transformations to amortize costs over multiple spans.
4. Regularly benchmark your pipeline to ensure neither filtering nor transformations are causing bottlenecks.

Remember, the goal of observability is to improve your system, not slow it down!

Common Pitfalls and How to Avoid Them

Throughout my journey with OTel, I've encountered a few common pitfalls when working with root span identification and manipulation:

1. Ignoring span events and links: While filtering and transforming based on span attributes is common, don't forget that span events and links can provide valuable context for analysis.
2. Overfiltering or overtransforming: It's tempting to create very specific filters or complex transformations, but this can lead to missing important data or performance issues. Start broad and refine as needed.
3. Not keeping up with OTel updates: As we've seen with the introduction of IsRootSpan(), new versions of OTel can introduce more efficient ways of working with spans. Stay updated with the latest releases and best practices.

Conclusion

Mastering root span identification and manipulation in OpenTelemetry is a game-changer for effective distributed tracing. By understanding and effectively using both the filter and transform processors, you'll be well-equipped to tackle even the most complex observability challenges.

Happy tracing!