Akka HTTP / Pekko HTTP

This guide shows how to instrument a Scala service built with Akka HTTP or Apache Pekko HTTP (the Apache 2.0 drop-in replacement). The setup uses the OpenTelemetry Java agent to auto-instrument HTTP routes, JDBC, Redis, and Kafka, with manual spans for any integrations not covered by the agent.

For a complete working example with PostgreSQL, Redis, Kafka, and Aerospike, see the scala/akka-http example.

Prerequisites

Scala 2.13 or 3.x, Java 11+
Akka HTTP 10.2+ or Pekko HTTP 1.0+
sbt with sbt-assembly for fat JAR builds
Last9 account with OTLP endpoint configured

Step 1: Add dependencies

// build.sbt
val otelVersion         = "1.44.0"
val otelInstrumentation = "2.10.0"

libraryDependencies ++= Seq(
  // OTel API only — SDK comes from the Java agent at runtime
  "io.opentelemetry" % "opentelemetry-api" % otelVersion,

  // Logback appender for OTLP log export
  "io.opentelemetry.instrumentation" % "opentelemetry-logback-appender-1.0" %
    s"$otelInstrumentation-alpha" % Runtime,
  "ch.qos.logback" % "logback-classic" % "1.5.12",
)

// Fat JAR merge strategy — required to preserve OTel SPI registrations
assembly / assemblyMergeStrategy := {
  case PathList("META-INF", "services", _*) => MergeStrategy.concat
  case PathList("META-INF", _*)             => MergeStrategy.discard
  case PathList("reference.conf")           => MergeStrategy.concat
  case _                                    => MergeStrategy.first
}

Step 2: Configure log correlation

<!-- src/main/resources/logback.xml -->
<configuration>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} trace_id=%X{trace_id} span_id=%X{span_id} - %msg%n</pattern>
        </encoder>
    </appender>

    <!-- Ships log records to Last9 via OTLP, correlated with trace_id/span_id -->
    <appender name="OTEL" class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
        <captureExperimentalAttributes>true</captureExperimentalAttributes>
        <captureMdcAttributes>*</captureMdcAttributes>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="OTEL"/>
    </root>
</configuration>

Step 3: Add manual spans to routes

The OTel Java agent auto-instruments Akka/Pekko HTTP server routes in some versions, but adding explicit SERVER-kind spans gives you full control over span names and attributes:

import io.opentelemetry.api.GlobalOpenTelemetry
import io.opentelemetry.api.trace.{SpanKind, StatusCode}
import org.apache.pekko.http.scaladsl.server.Directives.*

val tracer = GlobalOpenTelemetry.getTracer("my-service", "1.0.0")

// Synchronous route — use try/finally to guarantee span.end()
val route = path("orders" / IntNumber) { id =>
  get {
    val span  = tracer.spanBuilder("GET /orders/:id").setSpanKind(SpanKind.SERVER).startSpan()
    val scope = span.makeCurrent()
    try {
      span.setAttribute("order.id", id.toLong)
      val order = orderRepo.findById(id)  // JDBC — auto-traced by agent
      complete(order)
    } catch {
      case e: Exception =>
        span.recordException(e)
        span.setStatus(StatusCode.ERROR, e.getMessage)
        throw e
    } finally {
      scope.close()
      span.end()
    }
  }
}

Async routes (Future-based)

For routes that use onComplete or onSuccess, the span must be ended inside the callback, not in a finally block — the Future has not resolved when finally runs:

// Correct pattern for async routes
path("portfolios" / IntNumber / "price") { id =>
  get {
    val span  = tracer.spanBuilder("GET /portfolios/:id/price").setSpanKind(SpanKind.SERVER).startSpan()
    val scope = span.makeCurrent()
    span.setAttribute("portfolio.id", id.toLong)
    onComplete(pricingClient.getPrice(id)) {
      case Success(price) =>
        span.setAttribute("price.value", price.price)
        scope.close()
        span.end()
        complete(price)
      case Failure(ex) =>
        span.recordException(ex)
        span.setStatus(StatusCode.ERROR, ex.getMessage)
        scope.close()
        span.end()
        complete(StatusCodes.ServiceUnavailable, ex.getMessage)
    }
  }
}

Step 4: Manual spans for non-agent integrations

Some libraries are not auto-instrumented by the OTel Java agent (e.g. Aerospike). Add spans manually:

def get(key: String): Option[Record] =
  val span  = tracer.spanBuilder("aerospike.get").setSpanKind(SpanKind.CLIENT).startSpan()
  val scope = span.makeCurrent()
  try {
    span.setAttribute("db.system", "aerospike")
    span.setAttribute("aerospike.key", key)
    Option(client.get(null, Key(namespace, setName, key)))
  } catch {
    case e: Exception =>
      span.recordException(e)
      span.setStatus(StatusCode.ERROR, e.getMessage)
      throw e
  } finally {
    scope.close()
    span.end()
  }

Step 5: Dockerfile with Java agent

FROM eclipse-temurin:17-jre

WORKDIR /app

ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.10.0/opentelemetry-javaagent.jar \
    /app/otel-javaagent.jar

COPY target/scala-3.3.4/myapp.jar /app/app.jar

ENV OTEL_SERVICE_NAME=my-scala-service
ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
ENV OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
ENV OTEL_TRACES_EXPORTER=otlp
ENV OTEL_METRICS_EXPORTER=otlp
ENV OTEL_LOGS_EXPORTER=otlp

ENTRYPOINT ["java", "-javaagent:/app/otel-javaagent.jar", "-jar", "/app/app.jar"]

What gets instrumented automatically

Library	Signal
Akka HTTP / Pekko HTTP server + client	Traces
JDBC (PostgreSQL, MySQL) + HikariCP	Traces
Redis (Lettuce, Jedis)	Traces
Kafka producer / consumer	Traces + W3C context propagation
JVM (GC, heap, threads)	Metrics
SLF4J / Logback	Logs

Kubernetes: OTel Operator (no Dockerfile changes)

If you run on Kubernetes, the OpenTelemetry Operator can inject the Java agent automatically — you don’t need to ADD the agent JAR in your Dockerfile or pass -javaagent in your ENTRYPOINT.

1. Install the operator (once per cluster):

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

2. Create an Instrumentation resource (once per namespace):

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: scala-instrumentation
spec:
  exporter:
    endpoint: http://otel-collector.observability:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_always_on
  java:
    # Operator pulls the agent image — no JAR download in your Dockerfile
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
    env:
      - name: OTEL_EXPORTER_OTLP_PROTOCOL
        value: http/protobuf
      - name: OTEL_LOGS_EXPORTER
        value: otlp
      - name: OTEL_METRICS_EXPORTER
        value: otlp

3. Annotate your Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: portfolio-service
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
        - name: portfolio-service
          image: your-registry/portfolio-service:latest
          env:
            - name: OTEL_SERVICE_NAME
              value: portfolio-service
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: deployment.environment=production

The operator injects the agent as an init container and sets all OTEL_* env vars on the pod before your container starts. Your Dockerfile stays clean — just java -jar app.jar.

Verify the integration

Start your service and make a few requests. In Last9 you should see:

Traces with spans named by route pattern, with DB and Kafka child spans
Logs with trace_id / span_id fields linking to the corresponding trace
JVM metrics under your service name

Set OTEL_LOG_LEVEL=debug to troubleshoot SDK initialisation issues.