Integrating OpenTelemetry with Elixir: A Step-by-Step Guide

If you're building an Elixir application and want to get a better view of how it's performing, OpenTelemetry is a great tool for the job.

This guide will walk you through the steps to set it up in your project, so you can start tracking traces, metrics, and logs.

Whether you're just starting out or integrating it into an existing app, we’ll make sure you have all the details you need to get going.

What is Elixir?

Elixir is a dynamic, functional programming language built on the Erlang virtual machine (BEAM). It's known for its scalability, fault-tolerance, and ability to handle concurrent applications, making it popular for building distributed systems, web apps, and highly available services.

Setting Up OpenTelemetry in Your Elixir Project

Prerequisites and Setup

Before talking about the implementation, let's make sure you're all set up. You'll need the following tools and dependencies:

• Elixir: Make sure you have Elixir installed on your machine. You can check your version by running elixir -v in your terminal.
• OpenTelemetry SDK: The OpenTelemetry Elixir SDK provides all the necessary components to integrate OpenTelemetry into your application.
• Telemetry: OpenTelemetry in Elixir is built on top of the Telemetry library, so make sure it’s included in your project.

The Role of OpenTelemetry Events in Improving Observability | Last9

Learn how OpenTelemetry events enhance observability by providing detailed insights into application performance and system behavior.

Last9Last9

Step 1: Installing the OpenTelemetry Elixir SDK

First, you need to add the necessary dependencies to your Elixir project. Open your mix.exs file and include the OpenTelemetry dependencies in your deps function:

defp deps do
  [
    {:opentelemetry, "~> 1.0"},
    {:opentelemetry_exporter, "~> 0.5"},
    {:telemetry, "~> 1.0"}
  ]
end

Run mix deps.get to install the dependencies.

Step 2: Configuring OpenTelemetry in Your Elixir Project

Once the dependencies are installed, it’s time to configure OpenTelemetry. You'll need to initialize OpenTelemetry in your application’s start-up process, typically in your application.ex file.

Here's an example of how to set up OpenTelemetry with the exporter configured for Jaeger:

defmodule MyApp.Application do
  use Application

  def start(_type, _args) do
    children = [
      # Start the OpenTelemetry tracer
      {OpenTelemetry.Tracer, []},
      # Add other workers and supervisors
    ]

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)

    # Initialize OpenTelemetry Exporter for Jaeger
    :ok = OpenTelemetry.exporter(
      exporter: :jaeger,
      host: "localhost",
      port: 5775
    )
  end
end

This will set up OpenTelemetry with a Jaeger exporter, so you can start sending traces and metrics to your Jaeger instance.

gRPC with OpenTelemetry: Observability Guide for Microservices | Last9

Learn how to integrate gRPC with OpenTelemetry for better observability, performance, and reliability in microservices architectures.

Last9Last9

Instrumenting Your Elixir Application

Now that OpenTelemetry is set up, it’s time to instrument your Elixir application. Instrumentation allows OpenTelemetry to collect traces, metrics, and logs from various parts of your application.

Instrumenting HTTP Requests

One of the most common places to start instrumenting is your HTTP requests. If you're using the popular HTTPoison or Tesla libraries for HTTP requests, you can add simple instrumentation to trace the requests.

For example, with HTTPoison, you can manually create spans around your HTTP calls:

defmodule MyApp.HTTPClient do
  def get(url) do
    # Start a span to trace this request
    OpenTelemetry.Tracer.with_span("http_request") do
      HTTPoison.get(url)
    end
  end
end

This will create a trace for each outgoing HTTP request made using HTTPoison. You can further enrich the span with metadata, like the HTTP method or status code, by adding tags to the span.

Instrumenting Database Queries

Next, let’s look at instrumenting database queries. If you're using Ecto to interact with your database, OpenTelemetry integrates smoothly with it.

To trace database queries, you can attach OpenTelemetry to your Ecto queries like this:

defmodule MyApp.Repo do
  use Ecto.Repo, otp_app: :my_app

  def run_query do
    OpenTelemetry.Tracer.with_span("db_query") do
      MyApp.Repo.query("SELECT * FROM users")
    end
  end
end

This will trace the database query, allowing you to measure its performance and pinpoint any slow queries that could be affecting your application’s responsiveness.

Sending and Analyzing Your Telemetry Data

Once you’ve instrumented your application, it’s time to collect and export the data.

OpenTelemetry supports a wide range of exporters that send your telemetry data to observability platforms like Jaeger, Prometheus, or Zipkin.

Exporting to Jaeger

To export traces to Jaeger, you can use the opentelemetry_exporter package as shown in the configuration section.

This package allows you to send trace data to Jaeger, where you can visualize the traces and gain insights into your application’s performance.

Exporting Metrics to Prometheus

If you want to collect and export metrics, you can use Prometheus as the exporter. Add the prometheus_ex dependency in your mix.exs file:

defp deps do
  [
    {:prometheus_ex, "~> 3.0"}
  ]
end

Then, you can configure Prometheus metrics collection and expose them through an HTTP endpoint:

defmodule MyApp.PrometheusExporter do
  use Prometheus.Metric

  def setup do
    Counter.declare([name: :http_requests_total, labels: [:status]])
  end
end

Now, Prometheus can scrape your Elixir application’s metrics, allowing you to monitor important metrics like request counts and response times.

OpenTelemetry Context Propagation for Better Tracing | Last9

Learn how OpenTelemetry’s context propagation improves tracing by ensuring accurate, end-to-end visibility across distributed systems.

Last9Last9

Setting Up Sampling and Custom Samplers

One of the powerful features of OpenTelemetry is its ability to control how much data is collected through sampling.

By default, OpenTelemetry collects a certain percentage of traces, but for performance reasons, you may want to adjust the sampling rate or implement custom sampling logic to ensure you only capture the most relevant traces.

What is Sampling

Sampling determines which traces are recorded and exported. Without proper configuration, the volume of traces generated by your application can become overwhelming, especially in high-throughput environments.

Reducing the number of traces through sampling helps maintain an efficient and cost-effective observability system.

There are several sampling strategies you can configure:

• Trace Sampling: Decides how often traces are captured. OpenTelemetry supports on-demand sampling and probabilistic sampling.
• Parent-Based Sampling: The decision to sample is based on whether the parent span is sampled. If the parent span is sampled, the child spans are sampled as well.

Kafka with OpenTelemetry: Distributed Tracing Guide | Last9

Learn how to integrate Kafka with OpenTelemetry for enhanced distributed tracing, better performance monitoring, and effortless troubleshooting.

Last9Last9

How to Configure Sampling in Elixir

You can configure the sampling rate in the OpenTelemetry SDK by setting the sampling strategy.

Here's how you can set a basic probability-based sampling strategy in Elixir:

# In your OpenTelemetry configuration (e.g., in your Application start method)
OpenTelemetry.Tracer.configure(
  sampler: {OpenTelemetry.Sampler, :probabilistic, 0.1} # 10% sampling rate
)

This configuration will sample approximately 10% of all traces. The second argument (0.1) represents the probability (0.1 means 10% of traces will be recorded).

Custom Samplers

In addition to the built-in probabilistic sampler, you can create a custom sampler to implement more complex sampling logic based on specific conditions, such as:

• Sampling traces from a particular service or endpoint.
• Sampling based on the error type or status code.
• Sampling-based on time or load metrics.

Here’s an example of how you can implement a custom sampler that samples traces only when an error is encountered:

defmodule MyApp.CustomSampler do
  @behaviour OpenTelemetry.Sampler

  def sample(span_context, _trace_context) do
    case span_context.attributes[:status] do
      "error" -> :sampled  # Only sample spans where status is "error"
      _ -> :not_sampled
    end
  end
end

# Then configure it in your OpenTelemetry setup
OpenTelemetry.Tracer.configure(
  sampler: MyApp.CustomSampler
)

With a custom sampler, you can fine-tune what gets traced and exported, ensuring that you focus on the most critical parts of your application.

Popular OpenTelemetry Instrumentation Libraries for Elixir

Elixir has a growing ecosystem of OpenTelemetry instrumentation libraries that provide out-of-the-box integration with many common libraries and frameworks.

1. OpenTelemetry Instrumentation for HTTP Clients

For HTTP clients like HTTPoison, Finch, and Tesla, OpenTelemetry provides pre-built instrumentation that automatically tracks outbound HTTP requests. This is one of the most common areas where developers want to add observability.

Example: Instrumenting HTTPoison

To automatically instrument HTTP requests with HTTPoison, add the opentelemetry_instrumentation_http package to your mix.exs:

defp deps do
  [
    {:opentelemetry_instrumentation_http, "~> 0.1"}
  ]
end

This will automatically trace HTTP requests, including details such as the request URL, status code, and duration.

Instrumenting AWS Lambda Functions with OpenTelemetry | Last9

Learn how to instrument AWS Lambda functions with OpenTelemetry to gain valuable insights and improve the performance of your serverless apps.

Last9Last9

2. Ecto Instrumentation

Ecto is Elixir's database library, and OpenTelemetry provides built-in instrumentation for Ecto queries, making it easy to trace database operations.

To instrument Ecto queries, use the opentelemetry_instrumentation_ecto package. Add it to your mix.exs:

defp deps do
  [
    {:opentelemetry_instrumentation_ecto, "~> 0.1"}
  ]
end

This package will automatically trace database queries, including query duration and metadata like the query string and the table involved.

3. Phoenix Instrumentation

If you're using Phoenix for building web applications, OpenTelemetry provides support for tracing requests and responses automatically.

You can install the instrumentation for Phoenix by adding:

defp deps do
  [
    {:opentelemetry_instrumentation_phoenix, "~> 0.1"}
  ]
end

This will enable tracing for incoming HTTP requests, tracking the route, status code, and duration of each request.

Getting Started with Host Metrics Using OpenTelemetry | Last9

Learn to monitor host metrics with OpenTelemetry. Discover setup tips, common pitfalls, and best practices for effective observability.

Last9Last9

4. Background Jobs (e.g., Oban)

If you're using background job processing libraries like Oban, you can instrument the job processing to monitor job execution times and track errors.

To enable instrumentation for Oban jobs, you can use the opentelemetry_instrumentation_oban package:

defp deps do
  [
    {:opentelemetry_instrumentation_oban, "~> 0.1"}
  ]
end

This integration helps trace job lifecycle events, including job start, processing, and completion times.

5. Custom Instrumentation

In addition to the libraries above, you can create your custom instrumentation for any part of your application. For example, if you're working with a unique microservice or internal system, you can manually add spans to trace specific operations.

Here's an example of how to create a custom span for a specific function:

defmodule MyApp.CustomService do
  def perform_task do
    OpenTelemetry.Tracer.with_span("custom_task") do
      # Perform your task here
    end
  end
end

Custom spans provide full flexibility, allowing you to track any operation in your code, regardless of whether a pre-built instrumentation library exists.

OTEL Collector Monitoring: Best Practices & Guide | Last9

Learn how to effectively monitor the OTEL Collector with best practices and implementation strategies for improved system performance.

Last9Last9

6. Prometheus Metrics Integration

For monitoring your application's performance, you may want to collect metrics like request rates, error counts, and latencies. OpenTelemetry supports Prometheus as a metrics exporter, allowing you to send custom and default metrics to a Prometheus server for collection and visualization.

You can use libraries like prometheus_ex to declare and collect custom metrics:

defmodule MyApp.Metrics do
  use Prometheus.Metric

  def setup do
    Counter.declare([name: :http_requests_total, labels: [:status]])
  end

  def increment_request_count(status) do
    Counter.inc([name: :http_requests_total, labels: [status]])
  end
end

Understanding and Managing Spans in OpenTelemetry

In OpenTelemetry, spans are the core building blocks of distributed tracing. They represent operations or units of work within a trace.

What is a Span?

A span represents a single operation within your application, such as an HTTP request, a database query, or an internal function execution. It contains several key components that help track its lifecycle:

Key Span Components

• Name: Identifies the operation (e.g., "http_request", "db_query").
• Start Time: When the span starts.
• End Time: When the span ends (calculated from the start and end times).
• Attributes: Key-value pairs describing the span (e.g., HTTP status code, query string).
• Events: Specific occurrences during the span’s lifetime (e.g., an error or log entry).
• Status: The state of the operation (e.g., "ok", "error").

How to Start and End Spans in OpenTelemetry

Starting a Span:

To start a span, use the function OpenTelemetry.Tracer.with_span/2. This creates a span and executes the block of code as part of the trace.

Example:

defmodule MyApp.SomeService do
  def perform_task do
    # Start a span to track this task
    OpenTelemetry.Tracer.with_span("task_operation") do
      # Task logic here
      IO.puts("Performing task operation...")
    end
  end
end

The span (task_operation) starts when the block executes and ends when it finishes.

Hot Reload for OpenTelemetry Collector: Step-by-Step Guide | Last9

Learn to enable hot reload for the OpenTelemetry Collector to update configurations on the fly, improving your observability system’s agility.

Last9

Ending a Span:

The span automatically ends when the block completes. For manual control (e.g., long-running tasks), you can manually start and end spans.

Example:

defmodule MyApp.LongTask do
  def perform_task do
    span = OpenTelemetry.Tracer.start_span("long_running_task")
    # Task logic here
    IO.puts("Performing long-running task...")
    OpenTelemetry.Tracer.end_span(span) # End the span once the task is done
  end
end

How to Add Attributes to Spans in OpenTelemetry

Attributes provide additional context about the span, such as the HTTP method or data size.

You can set attributes either when creating the span or later during its lifecycle using OpenTelemetry.Span.set_attribute/3.

Example:

defmodule MyApp.HTTPClient do
  def make_request(url) do
    OpenTelemetry.Tracer.with_span("http_request") do
      # Set attributes for the span
      OpenTelemetry.Span.set_attribute(:url, url)
      OpenTelemetry.Span.set_attribute(:method, "GET")
      IO.puts("Sending request to #{url}")
    end
  end
end

Adding Events to a Span

Events capture significant occurrences during the span’s lifecycle, such as errors or specific milestones. Use OpenTelemetry.Span.add_event/2 to record events.

Example (Error Event):

defmodule MyApp.Service do
  def process_data(data) do
    OpenTelemetry.Tracer.with_span("data_processing") do
      try do
        IO.puts("Processing data...")
        raise "Something went wrong!" # Simulating an error
      rescue _error ->
        # Add an event to the span when an error occurs
        OpenTelemetry.Span.add_event("error_occurred", %{message: "Data processing failed"})
      end
    end
  end
end

Propagating Context Between Spans

Context propagation ensures that spans in different services or components are linked properly, especially in distributed systems.

For distributed tracing, you need to manually inject trace context into HTTP headers or messaging protocols. OpenTelemetry handles context propagation within a process automatically.

Example:

defmodule MyApp.HTTPClient do
  def make_request(url) do
    span = OpenTelemetry.Tracer.start_span("http_request")
    
    # Propagate context by adding trace information to request headers
    headers = OpenTelemetry.Tracer.inject(span.context)
    
    # Simulate sending the request with headers
    IO.puts("Sending request to #{url} with headers: #{inspect(headers)}")
    
    OpenTelemetry.Tracer.end_span(span)
  end
end

This ensures that the trace context (like trace and span IDs) is included in requests to other services, linking them in the overall trace.

Ingest OpenTelemetry metrics with Prometheus natively | Last9

Native support for OpenTelemetry metrics in Prometheus

Last9

Advanced Features and Optimization Tips

Context Propagation in Elixir’s Concurrency Model

OpenTelemetry automatically propagates context across processes. Ensure context is properly passed when dealing with Elixir’s concurrency model (e.g., using processes or tasks).

Custom Instrumentation

If pre-built instrumentation isn’t available for certain operations, you can manually create custom spans to track specific tasks. This allows for deeper observability.

Troubleshooting Tips

• Missing Spans: Double-check span creation and exporter configuration.
• Performance Issues: OpenTelemetry adds some overhead, so adjust the sampling rate or limit trace data exported to minimize the impact on performance.

How to Send Distributed Traces from Phoenix to Last9 Using OpenTelemetry

Phoenix is a web framework built with Elixir, designed for high-performance applications.

Prerequisites

• A Phoenix application is already set up.
• You've signed up for Last9, created a cluster, and gathered the OTLP credentials:
  • endpoint: https://otlp.last9.io
  • auth_header: Obtain this from the Last9 dashboard.

Install OpenTelemetry Packages To begin instrumentation, add the following dependencies in your mix.exs file:

defp deps do
  [
    {:opentelemetry, "~> 1.4.0"},
    {:opentelemetry_api, "~> 1.3.0"},
    {:opentelemetry_ecto, "~> 1.0"},
    {:opentelemetry_exporter, "~> 1.7.0"},
    {:opentelemetry_phoenix, "~> 1.2.0"}
  ]
end

Run the following command to fetch the dependencies:

mix deps.get

Set Up OpenTelemetry Instrumentation Configure the necessary environment variables, which can be found in the Last9 cluster's write endpoint section:

export OTEL_SERVICE_NAME=phoenix-app-service
export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.last9.io
export OTEL_EXPORTER_OTLP_AUTH_HEADER="Basic <BASIC_AUTH_HEADER>"

Next, in config/config.exs, add the following configuration:

config :opentelemetry, :processors,
  otel_batch_processor: %{
    exporter: {:opentelemetry_exporter,
      %{
        endpoints: [System.get_env("OTEL_EXPORTER_OTLP_ENDPOINT") || "http://localhost:4317"],
        headers: [{"Authorization", System.get_env("OTEL_EXPORTER_OTLP_AUTH_HEADER")}]
      }}
  }

config :opentelemetry, :resource, service: %{name: System.get_env("OTEL_SERVICE_NAME")}
config :opentelemetry_ecto, :tracer,
  repos: [:timeline, :repo]  # Add your repositories for tracing
config :opentelemetry_phoenix, :tracer, service_name: System.get_env("OTEL_SERVICE_NAME")

In your endpoint.ex file, add the following code:

defmodule PhoenixAppWeb.Endpoint do
  use Phoenix.Endpoint, otp_app: :phoenix_app
  # ...
  plug Plug.Telemetry, event_prefix: [:phoenix, :endpoint]
  # ...
end

This is required for the opentelemetry_phoenix package to track Phoenix requests.

Visualize Traces in Last9 With these changes in place, your Phoenix application will begin sending traces to Last9. To view the data, check the APM dashboard in Last9.

Conclusion

Following the steps in this guide, you'll set up tracing, metrics, and logs in your Elixir applications. This will help you monitor your application's health, debug issues faster, and optimize performance.