# Akka HTTP / Pekko HTTP > OpenTelemetry instrumentation for Scala services using Akka HTTP or Apache Pekko HTTP — traces, metrics, and logs to Last9 Source: https://last9.io/docs/integrations/frameworks/scala/akka-http/ This guide shows how to instrument a Scala service built with [Akka HTTP](https://doc.akka.io/docs/akka-http/current/) or [Apache Pekko HTTP](https://pekko.apache.org/docs/pekko-http/current/) (the Apache 2.0 drop-in replacement). The setup uses the OpenTelemetry Java agent to auto-instrument HTTP routes, JDBC, Redis, and Kafka, with manual spans for any integrations not covered by the agent. For a complete working example with PostgreSQL, Redis, Kafka, and Aerospike, see the [scala/akka-http](https://github.com/last9/opentelemetry-examples/tree/main/scala/akka-http) example. ## Prerequisites - Scala 2.13 or 3.x, Java 11+ - Akka HTTP 10.2+ or Pekko HTTP 1.0+ - sbt with `sbt-assembly` for fat JAR builds - Last9 account with OTLP endpoint configured ## Step 1: Add dependencies ```scala // build.sbt val otelVersion = "1.44.0" val otelInstrumentation = "2.10.0" libraryDependencies ++= Seq( // OTel API only — SDK comes from the Java agent at runtime "io.opentelemetry" % "opentelemetry-api" % otelVersion, // Logback appender for OTLP log export "io.opentelemetry.instrumentation" % "opentelemetry-logback-appender-1.0" % s"$otelInstrumentation-alpha" % Runtime, "ch.qos.logback" % "logback-classic" % "1.5.12", ) // Fat JAR merge strategy — required to preserve OTel SPI registrations assembly / assemblyMergeStrategy := { case PathList("META-INF", "services", _*) => MergeStrategy.concat case PathList("META-INF", _*) => MergeStrategy.discard case PathList("reference.conf") => MergeStrategy.concat case _ => MergeStrategy.first } ``` ## Step 2: Configure log correlation ```xml %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} trace_id=%X{trace_id} span_id=%X{span_id} - %msg%n true * ``` ## Step 3: Add manual spans to routes The OTel Java agent auto-instruments Akka/Pekko HTTP server routes in some versions, but adding explicit `SERVER`-kind spans gives you full control over span names and attributes: ```scala import io.opentelemetry.api.GlobalOpenTelemetry import io.opentelemetry.api.trace.{SpanKind, StatusCode} import org.apache.pekko.http.scaladsl.server.Directives.* val tracer = GlobalOpenTelemetry.getTracer("my-service", "1.0.0") // Synchronous route — use try/finally to guarantee span.end() val route = path("orders" / IntNumber) { id => get { val span = tracer.spanBuilder("GET /orders/:id").setSpanKind(SpanKind.SERVER).startSpan() val scope = span.makeCurrent() try { span.setAttribute("order.id", id.toLong) val order = orderRepo.findById(id) // JDBC — auto-traced by agent complete(order) } catch { case e: Exception => span.recordException(e) span.setStatus(StatusCode.ERROR, e.getMessage) throw e } finally { scope.close() span.end() } } } ``` ### Async routes (Future-based) For routes that use `onComplete` or `onSuccess`, the span must be ended **inside** the callback, not in a `finally` block — the Future has not resolved when `finally` runs: ```scala // Correct pattern for async routes path("portfolios" / IntNumber / "price") { id => get { val span = tracer.spanBuilder("GET /portfolios/:id/price").setSpanKind(SpanKind.SERVER).startSpan() val scope = span.makeCurrent() span.setAttribute("portfolio.id", id.toLong) onComplete(pricingClient.getPrice(id)) { case Success(price) => span.setAttribute("price.value", price.price) scope.close() span.end() complete(price) case Failure(ex) => span.recordException(ex) span.setStatus(StatusCode.ERROR, ex.getMessage) scope.close() span.end() complete(StatusCodes.ServiceUnavailable, ex.getMessage) } } } ``` ## Step 4: Manual spans for non-agent integrations Some libraries are not auto-instrumented by the OTel Java agent (e.g. Aerospike). Add spans manually: ```scala def get(key: String): Option[Record] = val span = tracer.spanBuilder("aerospike.get").setSpanKind(SpanKind.CLIENT).startSpan() val scope = span.makeCurrent() try { span.setAttribute("db.system", "aerospike") span.setAttribute("aerospike.key", key) Option(client.get(null, Key(namespace, setName, key))) } catch { case e: Exception => span.recordException(e) span.setStatus(StatusCode.ERROR, e.getMessage) throw e } finally { scope.close() span.end() } ``` ## Step 5: Dockerfile with Java agent ```dockerfile FROM eclipse-temurin:17-jre WORKDIR /app ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.30.0/opentelemetry-javaagent.jar \ /app/otel-javaagent.jar COPY target/scala-3.3.4/myapp.jar /app/app.jar ENV OTEL_SERVICE_NAME=my-scala-service ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 ENV OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf ENV OTEL_TRACES_EXPORTER=otlp ENV OTEL_METRICS_EXPORTER=otlp ENV OTEL_LOGS_EXPORTER=otlp ENTRYPOINT ["java", "-javaagent:/app/otel-javaagent.jar", "-jar", "/app/app.jar"] ``` ## What gets instrumented automatically | Library | Signal | | -------------------------------------- | -------------------------------- | | Akka HTTP / Pekko HTTP server + client | Traces | | JDBC (PostgreSQL, MySQL) + HikariCP | Traces | | Redis (Lettuce, Jedis) | Traces | | Kafka producer / consumer | Traces + W3C context propagation | | JVM (GC, heap, threads) | Metrics | | SLF4J / Logback | Logs | ## Kubernetes: OTel Operator (no Dockerfile changes) If you run on Kubernetes, the [OpenTelemetry Operator](https://opentelemetry.io/docs/kubernetes/operator/) can inject the Java agent automatically — you don't need to `ADD` the agent JAR in your Dockerfile or pass `-javaagent` in your `ENTRYPOINT`. **1. Install the operator** (once per cluster): ```bash kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml ``` **2. Create an `Instrumentation` resource** (once per namespace): ```yaml apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: scala-instrumentation spec: exporter: endpoint: http://otel-collector.observability:4318 propagators: - tracecontext - baggage sampler: type: parentbased_always_on java: # Operator pulls the agent image — no JAR download in your Dockerfile image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest env: - name: OTEL_EXPORTER_OTLP_PROTOCOL value: http/protobuf - name: OTEL_LOGS_EXPORTER value: otlp - name: OTEL_METRICS_EXPORTER value: otlp ``` **3. Annotate your Deployment**: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: portfolio-service spec: template: metadata: annotations: instrumentation.opentelemetry.io/inject-java: "true" spec: containers: - name: portfolio-service image: your-registry/portfolio-service:latest env: - name: OTEL_SERVICE_NAME value: portfolio-service - name: OTEL_RESOURCE_ATTRIBUTES value: deployment.environment=production ``` The operator injects the agent as an init container and sets all `OTEL_*` env vars on the pod before your container starts. Your Dockerfile stays clean — just `java -jar app.jar`. :::note `OTEL_SERVICE_NAME` must still be set per-deployment (in the pod env), as the operator cannot infer the service name. All other OTel configuration is inherited from the `Instrumentation` CRD. ::: ## Verify the integration Start your service and make a few requests. In Last9 you should see: 1. **Traces** with spans named by route pattern, with DB and Kafka child spans 2. **Logs** with `trace_id` / `span_id` fields linking to the corresponding trace 3. **JVM metrics** under your service name Set `OTEL_LOG_LEVEL=debug` to troubleshoot SDK initialisation issues.