Java | Last9 Documentation

Use OpenTelemetry to instrument your Java application and send telemetry data to Last9. You can run OpenTelemetry Collector or send the telemetry directly from the application. Read the setup guide for more details.

Instrumentation packages

Download the OpenTelemetry Java agent JAR file and place it in the project root directory.

curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar -o opentelemetry-javaagent.jar

The latest opentelemetry-javaagent.jar can be found in the releases section of the opentelemetry-java-instrumentation repository.

For Java version 8, download the following release of OpenTelemetry agent.

curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v1.32.0/opentelemetry-javaagent.jar -o opentelemetry-javaagent.jar

Environment variables

Set the following environment variables.

OTEL_EXPORTER_OTLP_ENDPOINT={{ .Logs.WriteURL }}
OTEL_SERVICE_NAME=<The name of your service>
OTEL_EXPORTER_OTLP_HEADERS=Authorization={{ .Logs.AuthValue }}
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_TRACES_SAMPLER="always_on"
OTEL_RESOURCE_ATTRIBUTES="deployment.environment=local"
OTEL_INSTRUMENTATION_LOGBACK_APPENDER_EXPERIMENTAL_CAPTURE_MDC_ATTRIBUTES="*"
OTEL_LOG_LEVEL=error

Depending on your Cloud environment, enable following:

OTEL_RESOURCE_PROVIDERS_AWS_ENABLED=true
OTEL_RESOURCE_PROVIDERS_GCP_ENABLED=true
OTEL_RESOURCE_PROVIDERS_AZURE_ENABLED=true

System Properties

Alternatively, you can also set system properties for the configuration.

# otel.properties file
otel.exporter.otlp.endpoint: {{ .Logs.WriteURL }}
otel.service.name: <service_name>
otel.exporter.otlp.headers: Authorization={{ .Logs.AuthValue }}
otel.traces.exporter: otlp
otel.metrics.exporter: otlp
otel.logs.exporter: otlp
otel.exporter.otlp.protocol: http/protobuf
otel.traces.sampler: always_on
otel.resource.attributes: "deployment.environment=staging"
otel.instrumentation.logback-appender.experimental.capture-mdc-attributes: "*"

Depending on your Cloud environment, enable following:

otel.resource.providers.aws.enabled: true
otel.resource.providers.gcp.enabled: true
otel.resource.providers.azure.enabled: true

You can either use a properties file or use the runtime syntax for setting system properties as follows.

java -javaagent:path/to/opentelemetry-javaagent.jar -Dotel.exporter.otlp.endpoint={{ .Logs.WriteURL }} -Dotel.service.name=<service_name> -Dotel.exporter.otlp.headers=Authorization={{ .Logs.AuthValue }} -Dotel.traces.exporter=otlp -Dotel.metrics.exporter=otlp -Dotel.logs.exporter=otlp -Dotel.exporter.otlp.protocol=http/protobuf -Dotel.instrumentation.logback-appender.experimental.capture-mdc-attributes="*" -Dotel.traces.sampler=always_on -Dotel.resource.attributes="deployment.environment=staging" -jar myapp.jar

Caveat

Older Java OpenTelemetry may require adding %20 between the Basic and the actual authorization header. If you receive 401 error from the Last9 Opentelemetry endpoint, update the OTEL_EXPORTER_OTLP_HEADERS as follows.

Authorization=Basic%20<auth_header>

Start the application

You can start your application specifying the -javaagent parameter now:

java -javaagent:path/to/opentelemetry-javaagent.jar -jar myapp.jar

You can specify an environment variable that automatically adds the -javaagent parameter to your JVM:

export JAVA_TOOL_OPTIONS="-javaagent:path/to/opentelemetry-javaagent.jar"
java -jar myapp.jar # or current build command

The application will send telemetry data such as logs, traces, and metrics to Last9.

Capture HTTP request/response bodies

Add HTTP request and response body capture to your traces with the java-otel-body-capture extension. Bodies appear as http.request.body and http.response.body span attributes — no changes to your application code required.

Works with Spring Boot 2.x (javax.servlet) and Spring Boot 3.x (jakarta.servlet), as well as any app running on Tomcat, Jetty, or Dropwizard.

Download the extension JAR

Download last9-otel-body-capture.jar from the latest release.

Add it to your startup command

java -javaagent:opentelemetry-javaagent.jar \
     -Dotel.javaagent.extensions=last9-otel-body-capture.jar \
     -jar your-app.jar

Every HTTP span now carries http.request.body and http.response.body as attributes.

Configuration

All options are system properties. The defaults are production-safe out of the box.

Property	Default	Description
`otel.bodycapture.enabled`	`true`	Master switch
`otel.bodycapture.capture_request_body`	`true`	Capture request bodies
`otel.bodycapture.capture_response_body`	`true`	Capture response bodies
`otel.bodycapture.max_body_size_bytes`	`8192`	Truncates larger bodies with `...[TRUNCATED]`
`otel.bodycapture.capture_on_error_only`	`false`	Only attach bodies on 4xx/5xx
`otel.bodycapture.exclude_paths`	`/health,/ready,/metrics`	Skip these path prefixes
`otel.bodycapture.include_paths`	(all)	Restrict to these path prefixes only
`otel.bodycapture.content_types`	`application/json,application/xml,text/plain`	Only capture these content types

java -javaagent:opentelemetry-javaagent.jar \
     -Dotel.javaagent.extensions=last9-otel-body-capture.jar \
     -Dotel.bodycapture.capture_on_error_only=true \
     -Dotel.bodycapture.include_paths=/api/ \
     -jar your-app.jar

PII/PHI redaction at the collector

Redact sensitive data at the OTel Collector so the pattern is centralized and no app redeployment is needed to update it:

processors:
  transform/redact-pii:
    trace_statements:
      - context: span
        statements:
          # SSN
          - replace_pattern(attributes["http.request.body"], "\\b\\d{3}-\\d{2}-\\d{4}\\b", "***-**-****")
          - replace_pattern(attributes["http.response.body"], "\\b\\d{3}-\\d{2}-\\d{4}\\b", "***-**-****")
          # Email
          - replace_pattern(attributes["http.request.body"], "[a-zA-Z0-9._%+\\-]+@[a-zA-Z0-9.\\-]+\\.[a-zA-Z]{2,}", "****@****.***")
          - replace_pattern(attributes["http.response.body"], "[a-zA-Z0-9._%+\\-]+@[a-zA-Z0-9.\\-]+\\.[a-zA-Z]{2,}", "****@****.***")

service:
  pipelines:
    traces:
      processors: [transform/redact-pii, batch]

For a complete working example, see the HTTP body capture example.

Troubleshooting

Please get in touch with us on Discord or Email if you have any questions.