# Spring Boot

> Learn how to integrate OpenTelemetry with Spring Boot applications and send telemetry data to Last9

Source: https://last9.io/docs/integrations/frameworks/java/spring-boot/

This guide shows you how to instrument your Spring Boot application with OpenTelemetry and send traces, metrics, and logs to Last9.

## Prerequisites

- Java 8 or later (Java 11+ recommended)
- Spring Boot 2.6+ or Spring Boot 3.x
- Last9 account with OTLP endpoint configured

## Installation Methods

Choose between auto-instrumentation (recommended) or manual instrumentation:

### Auto-Instrumentation (Recommended)

1. **Download OpenTelemetry Java Agent**

   ```bash
   curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar -o opentelemetry-javaagent.jar
   ```

   The agent supports Java 8 and above. To pin a specific version instead of tracking the latest release:

   ```bash
   curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.30.0/opentelemetry-javaagent.jar -o opentelemetry-javaagent.jar
   ```

2. **Configure environment variables**

   ```bash
   export OTEL_EXPORTER_OTLP_ENDPOINT="$last9_otlp_endpoint"
   export OTEL_EXPORTER_OTLP_HEADERS="Authorization=$last9_otlp_auth_header"
   export OTEL_SERVICE_NAME="spring-boot-app"
   export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production,service.version=1.0.0"
   export OTEL_TRACES_EXPORTER=otlp
   export OTEL_METRICS_EXPORTER=otlp
   export OTEL_LOGS_EXPORTER=otlp
   export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
   export OTEL_TRACES_SAMPLER=always_on
   export OTEL_INSTRUMENTATION_LOGBACK_APPENDER_EXPERIMENTAL_CAPTURE_MDC_ATTRIBUTES="*"
   ```

3. **Start your application with the agent**

   ```bash
   java -javaagent:opentelemetry-javaagent.jar -jar your-spring-boot-app.jar
   ```

### Manual Instrumentation

1.  **Add OpenTelemetry dependencies**

**Maven (pom.xml)**

    ```xml
    <properties>
        <opentelemetry.version>1.32.0</opentelemetry.version>
        <opentelemetry-instrumentation.version>1.32.0</opentelemetry-instrumentation.version>
    </properties>

    <dependencies>
        <!-- OpenTelemetry API -->
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-api</artifactId>
            <version>${opentelemetry.version}</version>
        </dependency>

        <!-- OpenTelemetry SDK -->
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-sdk</artifactId>
            <version>${opentelemetry.version}</version>
        </dependency>

        <!-- OTLP Exporters -->
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-exporter-otlp</artifactId>
            <version>${opentelemetry.version}</version>
        </dependency>

        <!-- Auto-instrumentation libraries -->
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-spring-boot-starter</artifactId>
            <version>${opentelemetry-instrumentation.version}</version>
        </dependency>

        <!-- Spring Boot specific instrumentations -->
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-spring-webmvc-6.0</artifactId>
            <version>${opentelemetry-instrumentation.version}</version>
        </dependency>

        <!-- Database instrumentations -->
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-jdbc</artifactId>
            <version>${opentelemetry-instrumentation.version}</version>
        </dependency>

        <!-- HTTP Client instrumentations -->
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-okhttp-3.0</artifactId>
            <version>${opentelemetry-instrumentation.version}</version>
        </dependency>
    </dependencies>
    ```

**Gradle (build.gradle)**

    ```groovy
    ext {
        openTelemetryVersion = '1.32.0'
        openTelemetryInstrumentationVersion = '1.32.0'
    }

    dependencies {
        // OpenTelemetry API
        implementation "io.opentelemetry:opentelemetry-api:${openTelemetryVersion}"

        // OpenTelemetry SDK
        implementation "io.opentelemetry:opentelemetry-sdk:${openTelemetryVersion}"

        // OTLP Exporters
        implementation "io.opentelemetry:opentelemetry-exporter-otlp:${openTelemetryVersion}"

        // Auto-instrumentation libraries
        implementation "io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter:${openTelemetryInstrumentationVersion}"

        // Spring Boot specific instrumentations
        implementation "io.opentelemetry.instrumentation:opentelemetry-spring-webmvc-6.0:${openTelemetryInstrumentationVersion}"

        // Database instrumentations
        implementation "io.opentelemetry.instrumentation:opentelemetry-jdbc:${openTelemetryInstrumentationVersion}"

        // HTTP Client instrumentations
        implementation "io.opentelemetry.instrumentation:opentelemetry-okhttp-3.0:${openTelemetryInstrumentationVersion}"
    }
    ```

2.  **Configure OpenTelemetry in Spring Boot**

**application.yml**

    ```yaml
    otel:
      exporter:
        otlp:
          endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT:$last9_otlp_endpoint}
          headers:
            authorization: ${OTEL_EXPORTER_OTLP_HEADERS:$last9_otlp_auth_header}
          protocol: http/protobuf
      service:
        name: ${OTEL_SERVICE_NAME:spring-boot-app}
      resource:
        attributes:
          deployment.environment: ${DEPLOYMENT_ENVIRONMENT:production}
          service.version: ${SERVICE_VERSION:1.0.0}
      traces:
        exporter: otlp
        sampler:
          probability: 1.0
      metrics:
        exporter: otlp
      logs:
        exporter: otlp

    # Spring Boot configuration
    spring:
      application:
        name: spring-boot-app
      datasource:
        url: jdbc:postgresql://localhost:5432/springdb
        username: ${DB_USERNAME:user}
        password: ${DB_PASSWORD:password}
        driver-class-name: org.postgresql.Driver
      jpa:
        hibernate:
          ddl-auto: validate
        show-sql: false
        properties:
          hibernate:
            format_sql: true
      redis:
        host: ${REDIS_HOST:localhost}
        port: ${REDIS_PORT:6379}

    # Management endpoints for health checks and metrics
    management:
      endpoints:
        web:
          exposure:
            include: health, info, metrics, prometheus
        enabled-by-default: true
      endpoint:
        health:
          show-details: always
      metrics:
        export:
          prometheus:
            enabled: true
    ```

**application.properties**

    ```properties
    # OpenTelemetry Configuration
    otel.exporter.otlp.endpoint=${OTEL_EXPORTER_OTLP_ENDPOINT:$last9_otlp_endpoint}
    otel.exporter.otlp.headers.authorization=${OTEL_EXPORTER_OTLP_HEADERS:$last9_otlp_auth_header}
    otel.exporter.otlp.protocol=http/protobuf
    otel.service.name=${OTEL_SERVICE_NAME:spring-boot-app}
    otel.resource.attributes=deployment.environment=${DEPLOYMENT_ENVIRONMENT:production},service.version=${SERVICE_VERSION:1.0.0}
    otel.traces.exporter=otlp
    otel.traces.sampler.probability=1.0
    otel.metrics.exporter=otlp
    otel.logs.exporter=otlp

    # Spring Boot Configuration
    spring.application.name=spring-boot-app
    spring.datasource.url=jdbc:postgresql://localhost:5432/springdb
    spring.datasource.username=${DB_USERNAME:user}
    spring.datasource.password=${DB_PASSWORD:password}
    spring.datasource.driver-class-name=org.postgresql.Driver

    # JPA Configuration
    spring.jpa.hibernate.ddl-auto=validate
    spring.jpa.show-sql=false
    spring.jpa.properties.hibernate.format_sql=true

    # Redis Configuration
    spring.redis.host=${REDIS_HOST:localhost}
    spring.redis.port=${REDIS_PORT:6379}

    # Management Configuration
    management.endpoints.web.exposure.include=health,info,metrics,prometheus
    management.endpoint.health.show-details=always
    management.metrics.export.prometheus.enabled=true
    ```

## Configuration Classes

### OpenTelemetry Configuration

**OpenTelemetryConfig.java**

```java
package com.example.config;

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.propagation.ContextPropagators;
import io.opentelemetry.context.propagation.TextMapPropagator;
import io.opentelemetry.exporter.otlp.logs.OtlpGrpcLogRecordExporter;
import io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter;
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
import io.opentelemetry.extension.trace.propagation.AwsXRayPropagator;
import io.opentelemetry.extension.trace.propagation.B3Propagator;
import io.opentelemetry.extension.trace.propagation.JaegerPropagator;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.logs.LogRecordProcessor;
import io.opentelemetry.sdk.logs.SdkLoggerProvider;
import io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor;
import io.opentelemetry.sdk.metrics.SdkMeterProvider;
import io.opentelemetry.sdk.metrics.export.MetricExporter;
import io.opentelemetry.sdk.metrics.export.PeriodicMetricReader;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
import io.opentelemetry.sdk.trace.export.SpanExporter;
import io.opentelemetry.sdk.trace.samplers.Sampler;
import io.opentelemetry.semconv.ResourceAttributes;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.time.Duration;

@Configuration
public class OpenTelemetryConfig {

    @Value("${otel.exporter.otlp.endpoint}")
    private String otlpEndpoint;

    @Value("${otel.exporter.otlp.headers.authorization}")
    private String authorizationHeader;

    @Value("${otel.service.name}")
    private String serviceName;

    @Value("${otel.resource.attributes:}")
    private String resourceAttributes;

    @Bean
    public OpenTelemetry openTelemetry() {
        // Create resource
        Resource resource = Resource.getDefault()
                .merge(Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, serviceName)))
                .merge(parseResourceAttributes(resourceAttributes));

        // Configure trace exporter
        SpanExporter spanExporter = OtlpGrpcSpanExporter.builder()
                .setEndpoint(otlpEndpoint + "/v1/traces")
                .addHeader("authorization", authorizationHeader)
                .setTimeout(Duration.ofSeconds(30))
                .build();

        // Configure metrics exporter
        MetricExporter metricExporter = OtlpGrpcMetricExporter.builder()
                .setEndpoint(otlpEndpoint + "/v1/metrics")
                .addHeader("authorization", authorizationHeader)
                .setTimeout(Duration.ofSeconds(30))
                .build();

        // Configure logs exporter
        OtlpGrpcLogRecordExporter logExporter = OtlpGrpcLogRecordExporter.builder()
                .setEndpoint(otlpEndpoint + "/v1/logs")
                .addHeader("authorization", authorizationHeader)
                .setTimeout(Duration.ofSeconds(30))
                .build();

        // Create tracer provider
        SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
                .addSpanProcessor(BatchSpanProcessor.builder(spanExporter)
                        .setMaxExportBatchSize(512)
                        .setExportTimeout(Duration.ofSeconds(30))
                        .setScheduleDelay(Duration.ofSeconds(5))
                        .build())
                .setResource(resource)
                .setSampler(Sampler.alwaysOn())
                .build();

        // Create meter provider
        SdkMeterProvider meterProvider = SdkMeterProvider.builder()
                .registerMetricReader(PeriodicMetricReader.builder(metricExporter)
                        .setInterval(Duration.ofSeconds(30))
                        .build())
                .setResource(resource)
                .build();

        // Create logger provider
        LogRecordProcessor logProcessor = BatchLogRecordProcessor.builder(logExporter)
                .setMaxExportBatchSize(512)
                .setExportTimeout(Duration.ofSeconds(30))
                .setScheduleDelay(Duration.ofSeconds(5))
                .build();

        SdkLoggerProvider loggerProvider = SdkLoggerProvider.builder()
                .addLogRecordProcessor(logProcessor)
                .setResource(resource)
                .build();

        // Configure context propagators
        ContextPropagators propagators = ContextPropagators.create(
                TextMapPropagator.composite(
                        TextMapPropagator.composite(),
                        B3Propagator.injectingSingleHeader(),
                        JaegerPropagator.getInstance(),
                        AwsXRayPropagator.getInstance()
                )
        );

        // Build OpenTelemetry SDK
        OpenTelemetry openTelemetry = OpenTelemetrySdk.builder()
                .setTracerProvider(tracerProvider)
                .setMeterProvider(meterProvider)
                .setLoggerProvider(loggerProvider)
                .setPropagators(propagators)
                .build();

        // Set as global instance
        GlobalOpenTelemetry.set(openTelemetry);

        return openTelemetry;
    }

    @Bean
    public Tracer tracer(OpenTelemetry openTelemetry) {
        return openTelemetry.getTracer(serviceName, "1.0.0");
    }

    private Resource parseResourceAttributes(String attributes) {
        Resource.Builder builder = Resource.empty().toBuilder();

        if (attributes != null && !attributes.isEmpty()) {
            String[] pairs = attributes.split(",");
            for (String pair : pairs) {
                String[] kv = pair.split("=", 2);
                if (kv.length == 2) {
                    builder.put(kv[0].trim(), kv[1].trim());
                }
            }
        }

        return builder.build();
    }
}
```

**TelemetryService.java**

```java
package com.example.service;

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.logs.Logger;
import io.opentelemetry.api.logs.Severity;
import io.opentelemetry.api.metrics.Counter;
import io.opentelemetry.api.metrics.Histogram;
import io.opentelemetry.api.metrics.Meter;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Context;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import javax.annotation.PostConstruct;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;

@Service
public class TelemetryService {

    @Autowired
    private OpenTelemetry openTelemetry;

    @Autowired
    private Tracer tracer;

    private Meter meter;
    private Logger logger;

    // Custom metrics
    private Counter httpRequestsCounter;
    private Histogram httpRequestDuration;
    private Counter businessOperationsCounter;
    private Counter errorCounter;

    @PostConstruct
    public void initialize() {
        this.meter = openTelemetry.getMeter("spring-boot-app", "1.0.0");
        this.logger = openTelemetry.getLogsBridge().get("spring-boot-app");

        // Initialize custom metrics
        this.httpRequestsCounter = meter.counterBuilder("http_requests_total")
                .setDescription("Total number of HTTP requests")
                .build();

        this.httpRequestDuration = meter.histogramBuilder("http_request_duration_seconds")
                .setDescription("HTTP request duration in seconds")
                .setUnit("s")
                .build();

        this.businessOperationsCounter = meter.counterBuilder("business_operations_total")
                .setDescription("Total number of business operations")
                .build();

        this.errorCounter = meter.counterBuilder("errors_total")
                .setDescription("Total number of errors")
                .build();
    }

    public <T> T traceOperation(String operationName, Supplier<T> operation) {
        return traceOperation(operationName, operation, Attributes.empty());
    }

    public <T> T traceOperation(String operationName, Supplier<T> operation, Attributes attributes) {
        Span span = tracer.spanBuilder(operationName)
                .setAllAttributes(attributes)
                .setSpanKind(SpanKind.INTERNAL)
                .startSpan();

        try (var scope = span.makeCurrent()) {
            T result = operation.get();
            span.setStatus(StatusCode.OK);
            return result;
        } catch (Exception e) {
            span.recordException(e);
            span.setStatus(StatusCode.ERROR, e.getMessage());
            recordError("operation_error", operationName, e.getClass().getSimpleName());
            throw e;
        } finally {
            span.end();
        }
    }

    public void recordHttpRequest(String method, String route, int statusCode, long durationMs) {
        Attributes attributes = Attributes.builder()
                .put("method", method)
                .put("route", route)
                .put("status_code", statusCode)
                .build();

        httpRequestsCounter.add(1, attributes);
        httpRequestDuration.record(durationMs / 1000.0, attributes);
    }

    public void recordBusinessOperation(String operation, String status) {
        Attributes attributes = Attributes.builder()
                .put("operation", operation)
                .put("status", status)
                .build();

        businessOperationsCounter.add(1, attributes);
    }

    public void recordError(String errorType, String operation, String errorClass) {
        Attributes attributes = Attributes.builder()
                .put("error_type", errorType)
                .put("operation", operation)
                .put("error_class", errorClass)
                .build();

        errorCounter.add(1, attributes);
    }

    public void logInfo(String message, Attributes attributes) {
        logger.logRecordBuilder()
                .setSeverity(Severity.INFO)
                .setBody(message)
                .setAllAttributes(attributes)
                .emit();
    }

    public void logError(String message, Throwable throwable, Attributes attributes) {
        logger.logRecordBuilder()
                .setSeverity(Severity.ERROR)
                .setBody(message)
                .setAllAttributes(attributes)
                .emit();

        // Also add current span context
        Span currentSpan = Span.current();
        if (currentSpan != null && currentSpan.isRecording()) {
            currentSpan.recordException(throwable);
        }
    }

    public Span getCurrentSpan() {
        return Span.current();
    }

    public Context getCurrentContext() {
        return Context.current();
    }
}
```

## REST Controller Examples

**UserController.java**

```java
package com.example.controller;

import com.example.model.User;
import com.example.service.TelemetryService;
import com.example.service.UserService;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.instrumentation.annotations.WithSpan;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

import javax.servlet.http.HttpServletRequest;
import javax.validation.Valid;
import java.util.List;
import java.util.Optional;

@RestController
@RequestMapping("/api/users")
public class UserController {

    private static final Logger log = LoggerFactory.getLogger(UserController.class);

    @Autowired
    private UserService userService;

    @Autowired
    private TelemetryService telemetryService;

    @GetMapping
    public ResponseEntity<List<User>> getAllUsers(
            @RequestParam(defaultValue = "0") int page,
            @RequestParam(defaultValue = "10") int size,
            HttpServletRequest request) {

        long startTime = System.currentTimeMillis();
        Span currentSpan = telemetryService.getCurrentSpan();

        try {
            // Add custom attributes to current span
            currentSpan.setAttributes(Attributes.builder()
                    .put("controller", "UserController")
                    .put("action", "getAllUsers")
                    .put("page", page)
                    .put("size", size)
                    .build());

            List<User> users = telemetryService.traceOperation("user_service.get_all_users", () ->
                    userService.getAllUsers(page, size)
            );

            currentSpan.setAttributes(Attributes.builder()
                    .put("users_count", users.size())
                    .build());

            telemetryService.recordBusinessOperation("get_users", "success");

            telemetryService.logInfo("Successfully retrieved users",
                    Attributes.builder()
                            .put("users_count", users.size())
                            .put("page", page)
                            .put("size", size)
                            .build());

            return ResponseEntity.ok(users);

        } catch (Exception e) {
            log.error("Error retrieving users", e);

            telemetryService.recordBusinessOperation("get_users", "error");
            telemetryService.logError("Failed to retrieve users", e,
                    Attributes.builder()
                            .put("page", page)
                            .put("size", size)
                            .build());

            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).build();

        } finally {
            long duration = System.currentTimeMillis() - startTime;
            telemetryService.recordHttpRequest(
                    request.getMethod(),
                    "/api/users",
                    200, // This should be the actual response status
                    duration
            );
        }
    }

    @GetMapping("/{id}")
    @WithSpan("get_user_by_id")
    public ResponseEntity<User> getUserById(@PathVariable Long id, HttpServletRequest request) {
        long startTime = System.currentTimeMillis();
        Span currentSpan = telemetryService.getCurrentSpan();

        try {
            currentSpan.setAttributes(Attributes.builder()
                    .put("controller", "UserController")
                    .put("action", "getUserById")
                    .put("user_id", id)
                    .build());

            Optional<User> user = telemetryService.traceOperation("user_service.get_by_id", () ->
                    userService.getUserById(id),
                    Attributes.builder().put("user_id", id).build()
            );

            if (user.isPresent()) {
                currentSpan.setAttributes(Attributes.builder()
                        .put("user_found", true)
                        .put("user_email", user.get().getEmail())
                        .build());

                telemetryService.recordBusinessOperation("get_user", "success");
                return ResponseEntity.ok(user.get());
            } else {
                currentSpan.setAttributes(Attributes.builder()
                        .put("user_found", false)
                        .build());

                telemetryService.recordBusinessOperation("get_user", "not_found");
                return ResponseEntity.notFound().build();
            }

        } catch (Exception e) {
            log.error("Error retrieving user with id: " + id, e);

            telemetryService.recordBusinessOperation("get_user", "error");
            telemetryService.logError("Failed to retrieve user", e,
                    Attributes.builder().put("user_id", id).build());

            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).build();

        } finally {
            long duration = System.currentTimeMillis() - startTime;
            telemetryService.recordHttpRequest(
                    request.getMethod(),
                    "/api/users/{id}",
                    200, // Should be actual response status
                    duration
            );
        }
    }

    @PostMapping
    public ResponseEntity<User> createUser(@Valid @RequestBody User user, HttpServletRequest request) {
        long startTime = System.currentTimeMillis();
        Span currentSpan = telemetryService.getCurrentSpan();

        try {
            currentSpan.setAttributes(Attributes.builder()
                    .put("controller", "UserController")
                    .put("action", "createUser")
                    .put("user_email", user.getEmail())
                    .build());

            User createdUser = telemetryService.traceOperation("user_service.create_user", () ->
                    userService.createUser(user),
                    Attributes.builder().put("user_email", user.getEmail()).build()
            );

            currentSpan.setAttributes(Attributes.builder()
                    .put("user_created", true)
                    .put("user_id", createdUser.getId())
                    .build());

            telemetryService.recordBusinessOperation("create_user", "success");

            telemetryService.logInfo("Successfully created user",
                    Attributes.builder()
                            .put("user_id", createdUser.getId())
                            .put("user_email", createdUser.getEmail())
                            .build());

            return ResponseEntity.status(HttpStatus.CREATED).body(createdUser);

        } catch (Exception e) {
            log.error("Error creating user", e);

            telemetryService.recordBusinessOperation("create_user", "error");
            telemetryService.logError("Failed to create user", e,
                    Attributes.builder().put("user_email", user.getEmail()).build());

            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).build();

        } finally {
            long duration = System.currentTimeMillis() - startTime;
            telemetryService.recordHttpRequest(
                    request.getMethod(),
                    "/api/users",
                    201, // Should be actual response status
                    duration
            );
        }
    }

    @PutMapping("/{id}")
    public ResponseEntity<User> updateUser(@PathVariable Long id, @Valid @RequestBody User user, HttpServletRequest request) {
        long startTime = System.currentTimeMillis();
        Span currentSpan = telemetryService.getCurrentSpan();

        try {
            currentSpan.setAttributes(Attributes.builder()
                    .put("controller", "UserController")
                    .put("action", "updateUser")
                    .put("user_id", id)
                    .put("user_email", user.getEmail())
                    .build());

            Optional<User> updatedUser = telemetryService.traceOperation("user_service.update_user", () ->
                    userService.updateUser(id, user),
                    Attributes.builder()
                            .put("user_id", id)
                            .put("user_email", user.getEmail())
                            .build()
            );

            if (updatedUser.isPresent()) {
                currentSpan.setAttributes(Attributes.builder()
                        .put("user_updated", true)
                        .build());

                telemetryService.recordBusinessOperation("update_user", "success");
                return ResponseEntity.ok(updatedUser.get());
            } else {
                currentSpan.setAttributes(Attributes.builder()
                        .put("user_found", false)
                        .build());

                telemetryService.recordBusinessOperation("update_user", "not_found");
                return ResponseEntity.notFound().build();
            }

        } catch (Exception e) {
            log.error("Error updating user with id: " + id, e);

            telemetryService.recordBusinessOperation("update_user", "error");
            telemetryService.logError("Failed to update user", e,
                    Attributes.builder()
                            .put("user_id", id)
                            .put("user_email", user.getEmail())
                            .build());

            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).build();

        } finally {
            long duration = System.currentTimeMillis() - startTime;
            telemetryService.recordHttpRequest(
                    request.getMethod(),
                    "/api/users/{id}",
                    200, // Should be actual response status
                    duration
            );
        }
    }

    @DeleteMapping("/{id}")
    public ResponseEntity<Void> deleteUser(@PathVariable Long id, HttpServletRequest request) {
        long startTime = System.currentTimeMillis();
        Span currentSpan = telemetryService.getCurrentSpan();

        try {
            currentSpan.setAttributes(Attributes.builder()
                    .put("controller", "UserController")
                    .put("action", "deleteUser")
                    .put("user_id", id)
                    .build());

            boolean deleted = telemetryService.traceOperation("user_service.delete_user", () ->
                    userService.deleteUser(id),
                    Attributes.builder().put("user_id", id).build()
            );

            if (deleted) {
                currentSpan.setAttributes(Attributes.builder()
                        .put("user_deleted", true)
                        .build());

                telemetryService.recordBusinessOperation("delete_user", "success");
                return ResponseEntity.noContent().build();
            } else {
                currentSpan.setAttributes(Attributes.builder()
                        .put("user_found", false)
                        .build());

                telemetryService.recordBusinessOperation("delete_user", "not_found");
                return ResponseEntity.notFound().build();
            }

        } catch (Exception e) {
            log.error("Error deleting user with id: " + id, e);

            telemetryService.recordBusinessOperation("delete_user", "error");
            telemetryService.logError("Failed to delete user", e,
                    Attributes.builder().put("user_id", id).build());

            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).build();

        } finally {
            long duration = System.currentTimeMillis() - startTime;
            telemetryService.recordHttpRequest(
                    request.getMethod(),
                    "/api/users/{id}",
                    204, // Should be actual response status
                    duration
            );
        }
    }
}
```

## Service Layer with Database Integration

**UserService.java**

```java
package com.example.service;

import com.example.model.User;
import com.example.repository.UserRepository;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.instrumentation.annotations.WithSpan;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Pageable;
import org.springframework.data.domain.Sort;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import java.util.List;
import java.util.Optional;

@Service
public class UserService {

    private static final Logger log = LoggerFactory.getLogger(UserService.class);

    @Autowired
    private UserRepository userRepository;

    @Autowired
    private TelemetryService telemetryService;

    @Autowired
    private CacheService cacheService;

    @Autowired
    private EmailService emailService;

    @WithSpan("user_service.get_all_users")
    public List<User> getAllUsers(int page, int size) {
        Span currentSpan = telemetryService.getCurrentSpan();

        currentSpan.setAttributes(Attributes.builder()
                .put("operation", "get_all_users")
                .put("page", page)
                .put("size", size)
                .build());

        try {
            Pageable pageable = PageRequest.of(page, size, Sort.by("id").ascending());
            Page<User> userPage = userRepository.findAll(pageable);

            currentSpan.setAttributes(Attributes.builder()
                    .put("total_elements", userPage.getTotalElements())
                    .put("total_pages", userPage.getTotalPages())
                    .put("results_count", userPage.getContent().size())
                    .build());

            return userPage.getContent();

        } catch (Exception e) {
            currentSpan.recordException(e);
            log.error("Error retrieving users", e);
            throw e;
        }
    }

    @WithSpan("user_service.get_by_id")
    public Optional<User> getUserById(Long id) {
        Span currentSpan = telemetryService.getCurrentSpan();

        currentSpan.setAttributes(Attributes.builder()
                .put("operation", "get_user_by_id")
                .put("user_id", id)
                .build());

        try {
            // Try cache first
            Optional<User> cachedUser = telemetryService.traceOperation("cache.get_user", () ->
                    cacheService.getUser(id),
                    Attributes.builder().put("cache_key", "user:" + id).build()
            );

            if (cachedUser.isPresent()) {
                currentSpan.setAttributes(Attributes.builder()
                        .put("cache_hit", true)
                        .build());
                return cachedUser;
            }

            // Fetch from database
            Optional<User> user = telemetryService.traceOperation("database.find_user", () ->
                    userRepository.findById(id),
                    Attributes.builder().put("user_id", id).build()
            );

            if (user.isPresent()) {
                // Cache for future requests
                telemetryService.traceOperation("cache.set_user", () -> {
                    cacheService.setUser(id, user.get());
                    return null;
                });

                currentSpan.setAttributes(Attributes.builder()
                        .put("cache_hit", false)
                        .put("user_found", true)
                        .build());
            } else {
                currentSpan.setAttributes(Attributes.builder()
                        .put("cache_hit", false)
                        .put("user_found", false)
                        .build());
            }

            return user;

        } catch (Exception e) {
            currentSpan.recordException(e);
            log.error("Error retrieving user with id: " + id, e);
            throw e;
        }
    }

    @Transactional
    @WithSpan("user_service.create_user")
    public User createUser(User user) {
        Span currentSpan = telemetryService.getCurrentSpan();

        currentSpan.setAttributes(Attributes.builder()
                .put("operation", "create_user")
                .put("user_email", user.getEmail())
                .build());

        try {
            // Check if user already exists
            Optional<User> existingUser = telemetryService.traceOperation("database.find_by_email", () ->
                    userRepository.findByEmail(user.getEmail()),
                    Attributes.builder().put("email", user.getEmail()).build()
            );

            if (existingUser.isPresent()) {
                throw new IllegalArgumentException("User with email " + user.getEmail() + " already exists");
            }

            // Save user to database
            User savedUser = telemetryService.traceOperation("database.save_user", () ->
                    userRepository.save(user),
                    Attributes.builder().put("user_email", user.getEmail()).build()
            );

            currentSpan.setAttributes(Attributes.builder()
                    .put("user_created", true)
                    .put("user_id", savedUser.getId())
                    .build());

            // Send welcome email asynchronously
            telemetryService.traceOperation("email.send_welcome", () -> {
                emailService.sendWelcomeEmail(savedUser);
                return null;
            });

            // Clear relevant caches
            telemetryService.traceOperation("cache.invalidate", () -> {
                cacheService.invalidateUserCaches();
                return null;
            });

            return savedUser;

        } catch (Exception e) {
            currentSpan.recordException(e);
            log.error("Error creating user", e);
            throw e;
        }
    }

    @Transactional
    @WithSpan("user_service.update_user")
    public Optional<User> updateUser(Long id, User userDetails) {
        Span currentSpan = telemetryService.getCurrentSpan();

        currentSpan.setAttributes(Attributes.builder()
                .put("operation", "update_user")
                .put("user_id", id)
                .put("user_email", userDetails.getEmail())
                .build());

        try {
            Optional<User> existingUser = telemetryService.traceOperation("database.find_user", () ->
                    userRepository.findById(id),
                    Attributes.builder().put("user_id", id).build()
            );

            if (existingUser.isPresent()) {
                User user = existingUser.get();
                user.setName(userDetails.getName());
                user.setEmail(userDetails.getEmail());

                User updatedUser = telemetryService.traceOperation("database.save_user", () ->
                        userRepository.save(user),
                        Attributes.builder()
                                .put("user_id", id)
                                .put("user_email", userDetails.getEmail())
                                .build()
                );

                currentSpan.setAttributes(Attributes.builder()
                        .put("user_updated", true)
                        .build());

                // Invalidate cache
                telemetryService.traceOperation("cache.invalidate_user", () -> {
                    cacheService.invalidateUser(id);
                    return null;
                });

                return Optional.of(updatedUser);
            } else {
                currentSpan.setAttributes(Attributes.builder()
                        .put("user_found", false)
                        .build());

                return Optional.empty();
            }

        } catch (Exception e) {
            currentSpan.recordException(e);
            log.error("Error updating user with id: " + id, e);
            throw e;
        }
    }

    @Transactional
    @WithSpan("user_service.delete_user")
    public boolean deleteUser(Long id) {
        Span currentSpan = telemetryService.getCurrentSpan();

        currentSpan.setAttributes(Attributes.builder()
                .put("operation", "delete_user")
                .put("user_id", id)
                .build());

        try {
            if (userRepository.existsById(id)) {
                telemetryService.traceOperation("database.delete_user", () -> {
                    userRepository.deleteById(id);
                    return null;
                }, Attributes.builder().put("user_id", id).build());

                // Invalidate cache
                telemetryService.traceOperation("cache.invalidate_user", () -> {
                    cacheService.invalidateUser(id);
                    return null;
                });

                currentSpan.setAttributes(Attributes.builder()
                        .put("user_deleted", true)
                        .build());

                return true;
            } else {
                currentSpan.setAttributes(Attributes.builder()
                        .put("user_found", false)
                        .build());

                return false;
            }

        } catch (Exception e) {
            currentSpan.recordException(e);
            log.error("Error deleting user with id: " + id, e);
            throw e;
        }
    }
}
```

## Production Deployment

### Docker Configuration

**Dockerfile**

```dockerfile
FROM openjdk:11-jre-slim

# Install curl for health checks
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*

# Create app directory
WORKDIR /app

# Download OpenTelemetry Java agent
ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar /app/opentelemetry-javaagent.jar

# Copy application jar
COPY target/spring-boot-app-1.0.0.jar app.jar

# Create non-root user
RUN groupadd -r appuser && useradd -r -g appuser appuser
RUN chown -R appuser:appuser /app
USER appuser

# Expose port
EXPOSE 8080

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
  CMD curl -f http://localhost:8080/actuator/health || exit 1

# Start application with OpenTelemetry agent
CMD ["java", "-javaagent:opentelemetry-javaagent.jar", "-jar", "app.jar"]
```

**docker-compose.yml**

```yaml
version: "3.8"

services:
  spring-app:
    build: .
    ports:
      - "8080:8080"
    environment:
      # OpenTelemetry Configuration
      - OTEL_EXPORTER_OTLP_ENDPOINT=$last9_otlp_endpoint
      - OTEL_EXPORTER_OTLP_HEADERS=Authorization=$last9_otlp_auth_header
      - OTEL_SERVICE_NAME=spring-boot-app
      - OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.version=1.0.0
      - OTEL_TRACES_EXPORTER=otlp
      - OTEL_METRICS_EXPORTER=otlp
      - OTEL_LOGS_EXPORTER=otlp
      - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

      # Application Configuration
      - SPRING_PROFILES_ACTIVE=production
      - DB_USERNAME=user
      - DB_PASSWORD=password
      - SPRING_DATASOURCE_URL=jdbc:postgresql://postgres:5432/springdb
      - REDIS_HOST=redis
      - REDIS_PORT=6379

      # JVM Configuration
      - JAVA_OPTS=-Xmx1g -Xms512m
    depends_on:
      - postgres
      - redis
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/actuator/health"]
      interval: 30s
      timeout: 10s
      retries: 3

  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: springdb
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data

volumes:
  postgres_data:
  redis_data:
```

### Kubernetes Deployment

**k8s-deployment.yaml**

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-boot-app
  labels:
    app: spring-boot-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: spring-boot-app
  template:
    metadata:
      labels:
        app: spring-boot-app
    spec:
      containers:
        - name: spring-boot-app
          image: your-registry/spring-boot-app:latest
          ports:
            - containerPort: 8080
          env:
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "$last9_otlp_endpoint"
            - name: OTEL_EXPORTER_OTLP_HEADERS
              value: "Authorization=$last9_otlp_auth_header"
            - name: OTEL_SERVICE_NAME
              value: "spring-boot-app"
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "deployment.environment=production,service.version=1.0.0,k8s.cluster.name=production"
            - name: SPRING_DATASOURCE_URL
              valueFrom:
                secretKeyRef:
                  name: app-secrets
                  key: database-url
            - name: REDIS_HOST
              valueFrom:
                configMapKeyRef:
                  name: app-config
                  key: redis-host
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 45
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 5
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "1Gi"
              cpu: "1000m"
---
apiVersion: v1
kind: Service
metadata:
  name: spring-boot-app-service
spec:
  selector:
    app: spring-boot-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: LoadBalancer
```

## Testing the Integration

1. **Start your application**

   ```bash
   # With auto-instrumentation
   java -javaagent:opentelemetry-javaagent.jar -jar target/spring-boot-app-1.0.0.jar

   # Or with Docker
   docker-compose up
   ```

2. **Test the endpoints**

   ```bash
   # Health check
   curl http://localhost:8080/actuator/health

   # User endpoints
   curl http://localhost:8080/api/users
   curl http://localhost:8080/api/users/1

   # Create a user
   curl -X POST http://localhost:8080/api/users \
     -H "Content-Type: application/json" \
     -d '{"name": "Test User", "email": "test@example.com"}'
   ```

3. **View telemetry in Last9**

   Check your Last9 dashboard for:

   - HTTP request traces with Spring Boot specific attributes
   - Database operation spans with JPA/Hibernate details
   - Cache operation traces
   - Custom business logic spans and metrics
   - Error tracking and correlation