How to Use Jaeger with OpenTelemetry

Distributed tracing, a key aspect of observability, enables developers to track and analyze requests spanning multiple services.

OpenTelemetry and Jaeger are comprehensive solutions for distributed tracing. But how do they complement each other? This article is a deep dive into how you can use Jaeger with OTel.   

What is Jaeger?

Jaeger is a Cloud Native Computing Foundation (CNCF) open-source, end-to-end distributed tracing system built for monitoring and troubleshooting microservices-based architectures.

Usually incorporated as an OpenTelemetry backend, it is used to track and visualize user request behavior across the distributed components of complex systems. 

Jaeger’s functions

To complement its distributed tracing functions, Jaeger also has the following.

1. Rich Visualization

Jaeger provides a user-friendly web interface for analyzing distributed traces. It offers features like charts, waterfall-style dependency visualization graphs, flame graphs, and detailed service-level breakdowns.

2. Root Cause Analysis

Jaeger offers in-depth visualization of entire event chains, provides tracing and logging of request paths, and details when and how a series of related software issues began, as well as which problematic microservice triggered issues in others. 

3. Scalability

Jaeger scales horizontally to handle large amounts of tracing data. It uses storage backends like Elasticsearch, Cassandra, or other compatible databases.

4. Flexibility

Jaeger supports multiple instrumentation methods for various programming languages (e.g., native OpenTracing instrumentation, Prometheus, and client libraries). 

What is OpenTelemetry?

OpenTelemetry (OTel) is an open-source observability framework used to collect telemetry data from applications, and export to prespecified backends. Also, a CNCF project, that provides a unified set of APIs, libraries, agents, and protocol specifications.

OpenTelemetry can also be used for distributed tracing, context propagation (for integrating request metadata across distributed services), and semantic conventions (for ensuring consistency in metric data representation and interpretation). Read more about OTel in our dedicated article. 

Jaeger vs. OpenTelemetry

Both Jaeger and OpenTelemetry are complementary rather than competing tools. Nonetheless, let's compare the two.

Why integrate Jaeger with OTel?

Consider these three advantages:

• End-to-end distributed tracing: Integrating both tools into one observability framework provides you with a complete solution: OpenTelemetry allows you to track and capture user requests as they traverse various microservices and application components, then Jaeger provides a web UI and a storage backend for analysis and storage of tracked user requests.
• Automatic Instrumentation: OTel’s automatic instrumentation reduces the manual effort required to add tracing to your application code. Since you do not need to manually modify the codebase, the risk of inaccurate or missing traces is reduced; meaning that more (accurate) traces can be collected and exported to Jaeger for visualization.  
• Performance Optimization: The end goal of observability is to ensure optimal application function. With inputs from OpenTelemetry and Jaeger, you can use insights from traces and trace metadata collected and analyzed to pinpoint performance bottlenecks in specific microservices, optimize resource usage, and improve overall system efficiency.

Both Jaeger and OTel have a set of exporters that ease the process of using both for end-to-end distributed tracing of your application.

These exporters function as alternatives to one another, and the decision is yours to make on which one to use to sync both systems. However, it is best practice to choose the exporter that best fits your requirements in terms of compatibility, performance, and ease of use.

OpenTelemetry provides three specific Jaeger-compatible exporters. Let us examine each. 

Jaeger Trace Exporter

The Jaeger Trace Exporter is a built-in OpenTelemetry exporter that allows you to export traces to a Jaeger backend server. It supports Thrift over UDP only, as it uses protocols such as gRPC or Thrift HTTP. 

OTLP Exporter

The OpenTelemetry Protocol (OTLP) Exporter is generic in function. It exports telemetry data to various backends and endpoints—Jaeger inclusive. You can configure it to send traces to a Jaeger endpoint using the Jaeger reception format.

Batch Span Processor

OpenTelemetry also provides a Batch Span Processor that can be used to buffer and batch multiple spans before exporting them to a Jaeger backend. This processor can be configured to send the accumulated spans to Jaeger at regular intervals or when specific conditions are met.

Now, what are the set-in-stone steps to use Jaeger and OTel in your application?Below is a 10-step sequence explaining how. 

Using Open Telemetry and Jaeger in your application

Follow these steps. 

1. Determine your Use Case

Identify the specific use case or problem you want to address with distributed tracing. You may want to monitor the performance of your application, troubleshoot issues, or optimize resource usage.

2. Understand OpenTelemetry Concepts

Familiarize yourself with the core concepts of OpenTelemetry, such as spans, traces, and attributes. Understand how instrumentation works and how to propagate context between services.

3. Choose a Programming Language

OpenTelemetry supports multiple languages, including Java, Python, Go, JavaScript, and more. Choose the appropriate OpenTelemetry SDK and instrument your code accordingly.

4. Install Dependencies

Install the necessary OpenTelemetry and Jaeger dependencies for your chosen programming language. These dependencies include the OpenTelemetry SDK, Jaeger receiver,  and any required language-specific libraries or plugins.

5. Configure the OpenTelemetry Collector and Jaeger Receiver

These configurations enable you to send traces to the Jaeger backend. Provide the necessary configuration options, such as the Jaeger endpoint, authentication credentials, and sampling strategies. This configuration can usually be done through environment variables or a configuration file.

6. Instrumentation

Identify the parts of your codebase that you want to instrument with distributed tracing. Use the OpenTelemetry APIs and libraries in your code to create spans and add attributes to capture relevant information. Instrumentation can be done manually by adding code snippets or by utilizing auto-instrumentation features provided by OpenTelemetry, if available for your programming language.

7. Export Traces to Jaeger

Configure OpenTelemetry to export captured traces to Jaeger. This involves setting the appropriate exporter options and endpoints in your OpenTelemetry configuration or code. Ensure that the Jaeger endpoint is correctly configured to receive and store exported trace data.

8. Verify Integration

Test and verify that your application is correctly capturing and sending traces to Jaeger. Generate some test requests that exercise the areas of your codebase you have instrumented. Use the Jaeger UI to visualize and validate that the traces are successfully reaching the backend.

9. Visualize and Analyze Traces

Once your application is instrumented and traces are exported to Jaeger, you can start visualizing and analyzing the distributed traces. Use the Jaeger user interface (UI) to filter traces, view latency distributions, identify bottlenecks, and understand microservices’ behavior.

10. Iterate and Optimize

Continuously monitor and analyze your distributed traces. Use insights gained from trace visualization to optimize your services for performance and efficiency. Fine-tune your sampling strategies, adjust instrumentation, and make improvements based on the data you collect.

Now, let us exemplify the steps above. 

How to use OTel with Jaeger

Step 1: Choose a Programming Language

OpenTelemetry supports various languages like Java, Python, Go, and .NET. Choose the language your application is built in and ensure that both OpenTelemetry and Jaeger support it. For this guide, we’ll focus on Python.

Step 2: Install Required Tools & Libraries

Use Docker to run a Jaeger instance. Make sure to avoid container name conflicts by renaming it if necessary:

docker run -d --name jaeger_instance \
  -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 14250:14250 \
  -p 9411:9411 \
  jaegertracing/all-in-one:latest

Install OpenTelemetry and Jaeger Libraries
For Python, create a Dockerfile to install the required libraries:

FROM python:3.8-slim

# Install OpenTelemetry and Jaeger libraries
RUN pip install --no-cache-dir \
    opentelemetry-api \
    opentelemetry-sdk \
    opentelemetry-instrumentation-requests \
    opentelemetry-exporter-jaeger \
    flask requests

# Set up the working directory
WORKDIR /app
COPY . /app

# Run the Python application
CMD ["python", "app.py"]

Step 3: Configure the OTel Exporter

Set up the Jaeger exporter to send traces from your application to Jaeger. Here’s an example Python snippet:

import os
from opentelemetry import trace
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Get Jaeger host and port from environment variables
jaeger_host = os.getenv("JAEGER_HOST", "localhost")
jaeger_port = int(os.getenv("JAEGER_PORT", 6831))

# Set up the TracerProvider with resource attributes
trace.set_tracer_provider(
    TracerProvider(resource=Resource.create({"service.name": "your-application-name"}))
)

# Configure Jaeger Exporter
jaeger_exporter = JaegerExporter(
    agent_host_name=jaeger_host,
    agent_port=jaeger_port
)

# Add the exporter to the trace provider
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(jaeger_exporter)
)

This configuration will export traces to the Jaeger agent running on the specified jaeger_host and jaeger_port.

Step 4: Enable Auto-Instrumentation

Enable OpenTelemetry’s auto-instrumentation to trace your application code automatically. For a Flask web application, you can do the following:

from opentelemetry.instrumentation.flask import FlaskInstrumentor
from flask import Flask

# Create a Flask app
app = Flask(__name__)

# Instrument the Flask application
FlaskInstrumentor().instrument_app(app)

@app.route("/")
def hello_world():
    return "Hello, OpenTelemetry!"

This automatically instruments all Flask endpoints for trace generation.

Step 5: Verify Tracing

1. Run Your Application
   Build and run your application’s Docker container, linking it to the Jaeger container:

docker build -t your-app .
docker run -d --name your-app --link jaeger_instance -p 8000:8000 your-app

2. Generate Traffic
   Send requests to your application to generate traces:

curl http://localhost:8000/

3. Check Jaeger UI
   Open the Jaeger UI at http://localhost:16686 to view and analyze the traces.

Step 6: Explore and Analyze Traces

Once traces are successfully captured and sent to Jaeger, use Jaeger's rich visualization UI to analyze them and gain insights into application performance.

Step 7: Fine-Tune Configuration

Adjust auto-instrumentation configuration options to meet your specific requirements. This may include tweaking sampling strategies, adding custom attributes to traces, or excluding certain parts of your codebase from instrumentation.

Step 8: Monitor and Optimize

Continuously monitor your application's traces to identify bottlenecks or areas for optimization. 

OpenTelemetry to Jaeger Transformation

Jaeger accepts tracing information formats such as OpenTelemetry Protocol (OTLP), Thrift Batch, and Protobuf Batch.

A range of attributes, including trace, parent and span IDs, start and end times, attributes, events, links, and status can be mapped from OpenTelemetry to Jaeger formats to enable Jaeger to understand and process the data for visualization.

Convert OpenTelemetry spans into Jaeger formats by following the steps outlined below. 

Step 1: Ingesting Spans from Jaeger Using OpenTelemetry

To ingest spans from Jaeger, set up the OpenTelemetry Collector with a Jaeger receiver. The Jaeger receiver acts as a bridge between Jaeger and OpenTelemetry.

Here's an example configuration file for the OpenTelemetry Collector with Jaeger receiver.

receivers:
  jaeger:
    protocols:
      grpc:
      thrift_http:

exporters:
  logging:

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [jaeger]
      processors: [batch]
      exporters: [logging]

Step 2: OpenTelemetry Collector

Set up and run the OpenTelemetry Collector using the configuration file from the previous step. Here's an example of how to run the OpenTelemetry Collector.

bash
docker run --rm -v /path/to/config.yaml:/etc/otel-collector-config.yaml otel/opentelemetry-collector-contrib:latest --config=/etc/otel-collector-config.yaml

Step 3: Jaeger Receiver

With the OpenTelemetry Collector running, you can now configure Jaeger Agent and HotRod to send spans to it. Here's an example of configuring the Jaeger Agent.

bash
docker run --rm -p 6831:6831/udp -p 6832:6832/udp jaegertracing/jaeger-agent:latest --reporter.grpc.host-port=<otel_collector_host>:14250

Replace `<otel_collector_host>` with the hostname or IP address of the machine running the OpenTelemetry Collector.

Step 4: Jaeger Agent and HotRod

To simulate sending spans to Jaeger, you can use the Jaeger Agent and HotRod.

Start the Jaeger Agent Container:

docker run --name jaeger-agent \
  -p 5778:5778/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5775:5775/udp \
  jaegertracing/jaeger-agent:latest

Start the HotRod:

docker run --name hotrod \
  -e JAEGER_AGENT_HOST=your_jaeger_agent_host \
  -e JAEGER_AGENT_PORT=6831 \
  -p 8080-8083:8080-8083 \
  jaegertracing/example-hotrod:latest all

Replace `your_jaeger_agent_host` with the address of the Jaeger Agent container.

Step 5: Examples

Now that everything is set up, you can instrument your applications with OpenTelemetry and start generating and exporting traces.

Here's an example of using OpenTelemetry in Python:

from opentelemetry import trace
from opentelemetry.trace import SpanKind

# Create a tracer
tracer = trace.get_tracer(__name__)

# Create a span
with tracer.start_as_current_span("example_span", kind=SpanKind.SERVER) as span:
    # Add custom attributes or events if needed
    span.set_attribute("example_attribute", "value")
    span.add_event("example_event")

    # Perform your application logic

    # End the span
    span.end()

Once the spans are generated, the OpenTelemetry Collector configured with the Jaeger receiver will pick them up and forward them to downstream exporters.

You should now see spans flowing from the Hot Rod to the Jaeger Agent, and from there to the OpenTelemetry Collector. Afterward, the OpenTelemetry Collector will transform and send the spans to Jaeger.

Conclusion

Integrating Jaeger with OpenTelemetry provides you with a comprehensive distributed tracing solution that allows you to capture, export, and analyze traces, enabling you to gain valuable insights into your microservices ecosystem.

Understanding the basics of OpenTelemetry and Jaeger, and following the steps provided for installing, instrumenting, exporting, and visualizing traces will help you leverage the benefits of and effectively utilize the powerful combination to ensure optimal application performance and improve user experience.