# Tomcat

> Learn how to integrate OpenTelemetry with Apache Tomcat applications and send telemetry data to Last9

Source: https://last9.io/docs/integrations/web-and-cdn/tomcat/

This guide shows you how to instrument your Apache Tomcat web application with OpenTelemetry and send traces, metrics, and logs to Last9.

## Prerequisites

- Java 8 or later (Java 11+ recommended)
- Apache Tomcat 8.5+ or Tomcat 9.x/10.x
- Last9 account with OTLP endpoint configured

## Installation Methods

Choose between auto-instrumentation (recommended) or manual instrumentation:

### Auto-Instrumentation with Java Agent (Recommended)

1.  **Download OpenTelemetry Java Agent**

    ```bash
    curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar -o opentelemetry-javaagent.jar
    ```

    The agent supports Java 8 and above. To pin a specific version instead of tracking the latest release:

    ```bash
    curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.30.0/opentelemetry-javaagent.jar -o opentelemetry-javaagent.jar
    ```

2.  **Place the agent in Tomcat directory**

    ```bash
    # Copy to Tomcat lib directory
    cp opentelemetry-javaagent.jar $TOMCAT_HOME/lib/

    # Or create a dedicated directory
    mkdir -p $TOMCAT_HOME/otel
    cp opentelemetry-javaagent.jar $TOMCAT_HOME/otel/
    ```

3.  **Configure Tomcat with OpenTelemetry**

**Environment Variables**

    Create or modify `$TOMCAT_HOME/bin/setenv.sh` (Linux/Mac) or `setenv.bat` (Windows):

    ```bash
    # setenv.sh
    export CATALINA_OPTS="$CATALINA_OPTS -javaagent:$TOMCAT_HOME/lib/opentelemetry-javaagent.jar"

    # OpenTelemetry Configuration
    export OTEL_EXPORTER_OTLP_ENDPOINT="$last9_otlp_endpoint"
    export OTEL_EXPORTER_OTLP_HEADERS="Authorization=$last9_otlp_auth_header"
    export OTEL_SERVICE_NAME="tomcat-app"
    export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production,service.version=1.0.0,server.type=tomcat"
    export OTEL_TRACES_EXPORTER=otlp
    export OTEL_METRICS_EXPORTER=otlp
    export OTEL_LOGS_EXPORTER=otlp
    export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
    export OTEL_TRACES_SAMPLER=always_on

    # JVM Settings
    export CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m -Xms512m"
    export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseG1GC"
    ```

    For Windows (`setenv.bat`):

    ```batch
    set CATALINA_OPTS=%CATALINA_OPTS% -javaagent:%TOMCAT_HOME%\lib\opentelemetry-javaagent.jar

    set OTEL_EXPORTER_OTLP_ENDPOINT=$last9_otlp_endpoint
    set OTEL_EXPORTER_OTLP_HEADERS=Authorization=$last9_otlp_auth_header
    set OTEL_SERVICE_NAME=tomcat-app
    set OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.version=1.0.0,server.type=tomcat
    set OTEL_TRACES_EXPORTER=otlp
    set OTEL_METRICS_EXPORTER=otlp
    set OTEL_LOGS_EXPORTER=otlp
    set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
    set OTEL_TRACES_SAMPLER=always_on
    ```

**System Properties**

    Create `$TOMCAT_HOME/conf/otel.properties`:

    ```properties
    otel.exporter.otlp.endpoint=$last9_otlp_endpoint
    otel.exporter.otlp.headers=Authorization=$last9_otlp_auth_header
    otel.service.name=tomcat-app
    otel.resource.attributes=deployment.environment=production,service.version=1.0.0,server.type=tomcat
    otel.traces.exporter=otlp
    otel.metrics.exporter=otlp
    otel.logs.exporter=otlp
    otel.exporter.otlp.protocol=http/protobuf
    otel.traces.sampler=always_on

    # Optional: Cloud provider detection
    otel.resource.providers.aws.enabled=true
    otel.resource.providers.gcp.enabled=true
    otel.resource.providers.azure.enabled=true
    ```

    Then modify `setenv.sh` to use the properties file:

    ```bash
    export CATALINA_OPTS="$CATALINA_OPTS -javaagent:$TOMCAT_HOME/lib/opentelemetry-javaagent.jar"
    export CATALINA_OPTS="$CATALINA_OPTS -Dotel.javaagent.configuration-file=$TOMCAT_HOME/conf/otel.properties"
    ```

4.  **Start Tomcat**

    ```bash
    $TOMCAT_HOME/bin/startup.sh

    # Or on Windows
    %TOMCAT_HOME%\bin\startup.bat
    ```

### Manual Instrumentation

For more control over instrumentation, you can manually integrate OpenTelemetry in your web application:

1.  **Add OpenTelemetry dependencies to your WAR project**

**Maven (pom.xml)**

    ```xml
    <properties>
        <opentelemetry.version>1.32.0</opentelemetry.version>
        <opentelemetry-instrumentation.version>1.32.0</opentelemetry-instrumentation.version>
    </properties>

    <dependencies>
        <!-- OpenTelemetry API -->
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-api</artifactId>
            <version>${opentelemetry.version}</version>
        </dependency>

        <!-- OpenTelemetry SDK -->
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-sdk</artifactId>
            <version>${opentelemetry.version}</version>
        </dependency>

        <!-- OTLP Exporters -->
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-exporter-otlp</artifactId>
            <version>${opentelemetry.version}</version>
        </dependency>

        <!-- Servlet instrumentation -->
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-servlet-3.0</artifactId>
            <version>${opentelemetry-instrumentation.version}</version>
        </dependency>

        <!-- HTTP client instrumentations -->
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-apache-httpclient-4.3</artifactId>
            <version>${opentelemetry-instrumentation.version}</version>
        </dependency>

        <!-- Database instrumentations -->
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-jdbc</artifactId>
            <version>${opentelemetry-instrumentation.version}</version>
        </dependency>

        <!-- Servlet API -->
        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>javax.servlet-api</artifactId>
            <version>4.0.1</version>
            <scope>provided</scope>
        </dependency>

        <!-- JSON processing -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.15.2</version>
        </dependency>
    </dependencies>
    ```

2.  **Create OpenTelemetry Configuration**

**OpenTelemetryConfig.java**

    ```java
    package com.example.config;

    import io.opentelemetry.api.GlobalOpenTelemetry;
    import io.opentelemetry.api.OpenTelemetry;
    import io.opentelemetry.api.common.Attributes;
    import io.opentelemetry.api.trace.Tracer;
    import io.opentelemetry.context.propagation.ContextPropagators;
    import io.opentelemetry.context.propagation.TextMapPropagator;
    import io.opentelemetry.exporter.otlp.logs.OtlpGrpcLogRecordExporter;
    import io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter;
    import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
    import io.opentelemetry.extension.trace.propagation.B3Propagator;
    import io.opentelemetry.extension.trace.propagation.JaegerPropagator;
    import io.opentelemetry.sdk.OpenTelemetrySdk;
    import io.opentelemetry.sdk.logs.LogRecordProcessor;
    import io.opentelemetry.sdk.logs.SdkLoggerProvider;
    import io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor;
    import io.opentelemetry.sdk.metrics.SdkMeterProvider;
    import io.opentelemetry.sdk.metrics.export.MetricExporter;
    import io.opentelemetry.sdk.metrics.export.PeriodicMetricReader;
    import io.opentelemetry.sdk.resources.Resource;
    import io.opentelemetry.sdk.trace.SdkTracerProvider;
    import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
    import io.opentelemetry.sdk.trace.export.SpanExporter;
    import io.opentelemetry.sdk.trace.samplers.Sampler;
    import io.opentelemetry.semconv.ResourceAttributes;

    import java.time.Duration;

    public class OpenTelemetryConfig {

        private static final String SERVICE_NAME = "tomcat-app";
        private static OpenTelemetry openTelemetry;

        public static void initialize() {
            if (openTelemetry != null) {
                return; // Already initialized
            }

            String otlpEndpoint = System.getenv("OTEL_EXPORTER_OTLP_ENDPOINT");
            String authHeader = System.getenv("OTEL_EXPORTER_OTLP_HEADERS");

            if (otlpEndpoint == null) {
                System.err.println("OTEL_EXPORTER_OTLP_ENDPOINT environment variable is not set");
                return;
            }

            // Extract authorization header
            String authorization = "";
            if (authHeader != null && authHeader.startsWith("Authorization=")) {
                authorization = authHeader.substring("Authorization=".length());
            }

            // Create resource
            Resource resource = Resource.getDefault()
                    .merge(Resource.create(Attributes.of(
                            ResourceAttributes.SERVICE_NAME, SERVICE_NAME,
                            ResourceAttributes.SERVICE_VERSION, "1.0.0",
                            ResourceAttributes.DEPLOYMENT_ENVIRONMENT,
                            System.getProperty("deployment.environment", "production")
                    )));

            // Configure exporters
            SpanExporter spanExporter = OtlpGrpcSpanExporter.builder()
                    .setEndpoint(otlpEndpoint + "/v1/traces")
                    .addHeader("authorization", authorization)
                    .setTimeout(Duration.ofSeconds(30))
                    .build();

            MetricExporter metricExporter = OtlpGrpcMetricExporter.builder()
                    .setEndpoint(otlpEndpoint + "/v1/metrics")
                    .addHeader("authorization", authorization)
                    .setTimeout(Duration.ofSeconds(30))
                    .build();

            OtlpGrpcLogRecordExporter logExporter = OtlpGrpcLogRecordExporter.builder()
                    .setEndpoint(otlpEndpoint + "/v1/logs")
                    .addHeader("authorization", authorization)
                    .setTimeout(Duration.ofSeconds(30))
                    .build();

            // Create providers
            SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
                    .addSpanProcessor(BatchSpanProcessor.builder(spanExporter)
                            .setMaxExportBatchSize(512)
                            .setExportTimeout(Duration.ofSeconds(30))
                            .setScheduleDelay(Duration.ofSeconds(5))
                            .build())
                    .setResource(resource)
                    .setSampler(Sampler.alwaysOn())
                    .build();

            SdkMeterProvider meterProvider = SdkMeterProvider.builder()
                    .registerMetricReader(PeriodicMetricReader.builder(metricExporter)
                            .setInterval(Duration.ofSeconds(30))
                            .build())
                    .setResource(resource)
                    .build();

            LogRecordProcessor logProcessor = BatchLogRecordProcessor.builder(logExporter)
                    .setMaxExportBatchSize(512)
                    .setExportTimeout(Duration.ofSeconds(30))
                    .setScheduleDelay(Duration.ofSeconds(5))
                    .build();

            SdkLoggerProvider loggerProvider = SdkLoggerProvider.builder()
                    .addLogRecordProcessor(logProcessor)
                    .setResource(resource)
                    .build();

            // Configure context propagators
            ContextPropagators propagators = ContextPropagators.create(
                    TextMapPropagator.composite(
                            TextMapPropagator.composite(),
                            B3Propagator.injectingSingleHeader(),
                            JaegerPropagator.getInstance()
                    )
            );

            // Build OpenTelemetry SDK
            openTelemetry = OpenTelemetrySdk.builder()
                    .setTracerProvider(tracerProvider)
                    .setMeterProvider(meterProvider)
                    .setLoggerProvider(loggerProvider)
                    .setPropagators(propagators)
                    .build();

            // Set as global instance
            GlobalOpenTelemetry.set(openTelemetry);

            System.out.println("OpenTelemetry initialized successfully for Tomcat application");
        }

        public static OpenTelemetry getOpenTelemetry() {
            if (openTelemetry == null) {
                initialize();
            }
            return openTelemetry;
        }

        public static Tracer getTracer() {
            return getOpenTelemetry().getTracer(SERVICE_NAME, "1.0.0");
        }

        public static void shutdown() {
            if (openTelemetry instanceof OpenTelemetrySdk) {
                ((OpenTelemetrySdk) openTelemetry).close();
            }
        }
    }
    ```

**ServletContextListener.java**

    ```java
    package com.example.listener;

    import com.example.config.OpenTelemetryConfig;

    import javax.servlet.ServletContextEvent;
    import javax.servlet.ServletContextListener;
    import javax.servlet.annotation.WebListener;

    @WebListener
    public class OpenTelemetryContextListener implements ServletContextListener {

        @Override
        public void contextInitialized(ServletContextEvent sce) {
            System.out.println("Initializing OpenTelemetry for Tomcat application...");
            OpenTelemetryConfig.initialize();
        }

        @Override
        public void contextDestroyed(ServletContextEvent sce) {
            System.out.println("Shutting down OpenTelemetry...");
            OpenTelemetryConfig.shutdown();
        }
    }
    ```

3.  **Create Servlet Filter for Request Tracing**

**TracingFilter.java**

    ```java
    package com.example.filter;

    import com.example.config.OpenTelemetryConfig;
    import io.opentelemetry.api.common.Attributes;
    import io.opentelemetry.api.trace.Span;
    import io.opentelemetry.api.trace.SpanKind;
    import io.opentelemetry.api.trace.StatusCode;
    import io.opentelemetry.api.trace.Tracer;
    import io.opentelemetry.context.Context;
    import io.opentelemetry.context.Scope;
    import io.opentelemetry.context.propagation.TextMapGetter;
    import io.opentelemetry.semconv.SemanticAttributes;

    import javax.servlet.*;
    import javax.servlet.annotation.WebFilter;
    import javax.servlet.http.HttpServletRequest;
    import javax.servlet.http.HttpServletResponse;
    import java.io.IOException;
    import java.util.Collections;
    import java.util.Enumeration;

    @WebFilter(urlPatterns = "/*")
    public class TracingFilter implements Filter {

        private static final TextMapGetter<HttpServletRequest> GETTER = new HttpServletRequestTextMapGetter();
        private Tracer tracer;

        @Override
        public void init(FilterConfig filterConfig) {
            this.tracer = OpenTelemetryConfig.getTracer();
        }

        @Override
        public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
                throws IOException, ServletException {

            if (!(request instanceof HttpServletRequest) || !(response instanceof HttpServletResponse)) {
                chain.doFilter(request, response);
                return;
            }

            HttpServletRequest httpRequest = (HttpServletRequest) request;
            HttpServletResponse httpResponse = (HttpServletResponse) response;

            // Extract context from incoming request
            Context parentContext = OpenTelemetryConfig.getOpenTelemetry()
                    .getPropagators()
                    .getTextMapPropagator()
                    .extract(Context.current(), httpRequest, GETTER);

            // Create span
            String spanName = httpRequest.getMethod() + " " + getRoute(httpRequest);
            Span span = tracer.spanBuilder(spanName)
                    .setParent(parentContext)
                    .setSpanKind(SpanKind.SERVER)
                    .setAllAttributes(getRequestAttributes(httpRequest))
                    .startSpan();

            long startTime = System.currentTimeMillis();

            try (Scope scope = span.makeCurrent()) {
                // Store span in request for access by servlets
                httpRequest.setAttribute("otel.span", span);

                chain.doFilter(request, response);

                // Set response attributes
                span.setAllAttributes(getResponseAttributes(httpResponse, System.currentTimeMillis() - startTime));
                span.setStatus(StatusCode.OK);

            } catch (Exception e) {
                span.recordException(e);
                span.setStatus(StatusCode.ERROR, e.getMessage());
                throw e;
            } finally {
                span.end();
            }
        }

        private Attributes getRequestAttributes(HttpServletRequest request) {
            return Attributes.builder()
                    .put(SemanticAttributes.HTTP_METHOD, request.getMethod())
                    .put(SemanticAttributes.HTTP_URL, request.getRequestURL().toString())
                    .put(SemanticAttributes.HTTP_ROUTE, getRoute(request))
                    .put(SemanticAttributes.HTTP_SCHEME, request.getScheme())
                    .put(SemanticAttributes.HTTP_HOST, request.getHeader("Host"))
                    .put(SemanticAttributes.HTTP_USER_AGENT, request.getHeader("User-Agent"))
                    .put(SemanticAttributes.HTTP_CLIENT_IP, getClientIp(request))
                    .put("http.request.size", getRequestSize(request))
                    .build();
        }

        private Attributes getResponseAttributes(HttpServletResponse response, long durationMs) {
            return Attributes.builder()
                    .put(SemanticAttributes.HTTP_STATUS_CODE, response.getStatus())
                    .put("http.response.size", getResponseSize(response))
                    .put("http.request.duration_ms", durationMs)
                    .build();
        }

        private String getRoute(HttpServletRequest request) {
            String pathInfo = request.getPathInfo();
            if (pathInfo != null) {
                return request.getServletPath() + pathInfo;
            }
            return request.getServletPath();
        }

        private String getClientIp(HttpServletRequest request) {
            String xForwardedFor = request.getHeader("X-Forwarded-For");
            if (xForwardedFor != null && !xForwardedFor.isEmpty()) {
                return xForwardedFor.split(",")[0].trim();
            }
            String xRealIp = request.getHeader("X-Real-IP");
            if (xRealIp != null && !xRealIp.isEmpty()) {
                return xRealIp;
            }
            return request.getRemoteAddr();
        }

        private long getRequestSize(HttpServletRequest request) {
            String contentLength = request.getHeader("Content-Length");
            if (contentLength != null) {
                try {
                    return Long.parseLong(contentLength);
                } catch (NumberFormatException e) {
                    return 0;
                }
            }
            return 0;
        }

        private long getResponseSize(HttpServletResponse response) {
            // Note: This might not be accurate for all response types
            return 0; // Tomcat doesn't easily expose response size
        }

        @Override
        public void destroy() {
            // Cleanup if needed
        }

        private static class HttpServletRequestTextMapGetter implements TextMapGetter<HttpServletRequest> {
            @Override
            public Iterable<String> keys(HttpServletRequest request) {
                return Collections.list(request.getHeaderNames());
            }

            @Override
            public String get(HttpServletRequest request, String key) {
                return request.getHeader(key);
            }
        }
    }
    ```

## Servlet Examples

### REST API Servlet

**UserServlet.java**

```java
package com.example.servlet;

import com.example.config.OpenTelemetryConfig;
import com.example.model.User;
import com.example.service.UserService;
import com.fasterxml.jackson.databind.ObjectMapper;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.trace.Tracer;

import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.List;

@WebServlet(urlPatterns = {"/api/users", "/api/users/*"})
public class UserServlet extends HttpServlet {

    private UserService userService;
    private ObjectMapper objectMapper;
    private Tracer tracer;

    @Override
    public void init() throws ServletException {
        this.userService = new UserService();
        this.objectMapper = new ObjectMapper();
        this.tracer = OpenTelemetryConfig.getTracer();
    }

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException {

        Span span = tracer.spanBuilder("user_servlet.do_get")
                .setAllAttributes(Attributes.builder()
                        .put("servlet", "UserServlet")
                        .put("action", "doGet")
                        .build())
                .startSpan();

        try {
            String pathInfo = request.getPathInfo();

            if (pathInfo == null || pathInfo.equals("/")) {
                // Get all users
                handleGetAllUsers(request, response, span);
            } else {
                // Get user by ID
                String[] pathParts = pathInfo.split("/");
                if (pathParts.length == 2) {
                    try {
                        Long userId = Long.parseLong(pathParts[1]);
                        handleGetUserById(request, response, userId, span);
                    } catch (NumberFormatException e) {
                        sendErrorResponse(response, HttpServletResponse.SC_BAD_REQUEST,
                                "Invalid user ID format", span);
                    }
                } else {
                    sendErrorResponse(response, HttpServletResponse.SC_NOT_FOUND,
                            "Resource not found", span);
                }
            }

            span.setStatus(StatusCode.OK);

        } catch (Exception e) {
            span.recordException(e);
            span.setStatus(StatusCode.ERROR, e.getMessage());
            sendErrorResponse(response, HttpServletResponse.SC_INTERNAL_SERVER_ERROR,
                    "Internal server error", span);
        } finally {
            span.end();
        }
    }

    @Override
    protected void doPost(HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException {

        Span span = tracer.spanBuilder("user_servlet.do_post")
                .setAllAttributes(Attributes.builder()
                        .put("servlet", "UserServlet")
                        .put("action", "doPost")
                        .build())
                .startSpan();

        try {
            // Parse request body
            User user = objectMapper.readValue(request.getInputStream(), User.class);

            span.setAllAttributes(Attributes.builder()
                    .put("user_email", user.getEmail())
                    .put("operation", "create_user")
                    .build());

            // Create user
            User createdUser = userService.createUser(user);

            span.setAllAttributes(Attributes.builder()
                    .put("user_created", true)
                    .put("user_id", createdUser.getId())
                    .build());

            // Send response
            response.setStatus(HttpServletResponse.SC_CREATED);
            response.setContentType("application/json");
            objectMapper.writeValue(response.getOutputStream(), createdUser);

            span.setStatus(StatusCode.OK);

        } catch (Exception e) {
            span.recordException(e);
            span.setStatus(StatusCode.ERROR, e.getMessage());
            sendErrorResponse(response, HttpServletResponse.SC_INTERNAL_SERVER_ERROR,
                    "Failed to create user", span);
        } finally {
            span.end();
        }
    }

    @Override
    protected void doPut(HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException {

        Span span = tracer.spanBuilder("user_servlet.do_put")
                .setAllAttributes(Attributes.builder()
                        .put("servlet", "UserServlet")
                        .put("action", "doPut")
                        .build())
                .startSpan();

        try {
            String pathInfo = request.getPathInfo();
            if (pathInfo == null) {
                sendErrorResponse(response, HttpServletResponse.SC_BAD_REQUEST,
                        "User ID is required", span);
                return;
            }

            String[] pathParts = pathInfo.split("/");
            if (pathParts.length != 2) {
                sendErrorResponse(response, HttpServletResponse.SC_BAD_REQUEST,
                        "Invalid path", span);
                return;
            }

            Long userId = Long.parseLong(pathParts[1]);
            User user = objectMapper.readValue(request.getInputStream(), User.class);

            span.setAllAttributes(Attributes.builder()
                    .put("user_id", userId)
                    .put("user_email", user.getEmail())
                    .put("operation", "update_user")
                    .build());

            // Update user
            User updatedUser = userService.updateUser(userId, user);

            if (updatedUser != null) {
                span.setAllAttributes(Attributes.builder()
                        .put("user_updated", true)
                        .build());

                response.setStatus(HttpServletResponse.SC_OK);
                response.setContentType("application/json");
                objectMapper.writeValue(response.getOutputStream(), updatedUser);
            } else {
                span.setAllAttributes(Attributes.builder()
                        .put("user_found", false)
                        .build());

                sendErrorResponse(response, HttpServletResponse.SC_NOT_FOUND,
                        "User not found", span);
            }

            span.setStatus(StatusCode.OK);

        } catch (NumberFormatException e) {
            span.recordException(e);
            sendErrorResponse(response, HttpServletResponse.SC_BAD_REQUEST,
                    "Invalid user ID format", span);
        } catch (Exception e) {
            span.recordException(e);
            span.setStatus(StatusCode.ERROR, e.getMessage());
            sendErrorResponse(response, HttpServletResponse.SC_INTERNAL_SERVER_ERROR,
                    "Failed to update user", span);
        } finally {
            span.end();
        }
    }

    @Override
    protected void doDelete(HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException {

        Span span = tracer.spanBuilder("user_servlet.do_delete")
                .setAllAttributes(Attributes.builder()
                        .put("servlet", "UserServlet")
                        .put("action", "doDelete")
                        .build())
                .startSpan();

        try {
            String pathInfo = request.getPathInfo();
            if (pathInfo == null) {
                sendErrorResponse(response, HttpServletResponse.SC_BAD_REQUEST,
                        "User ID is required", span);
                return;
            }

            String[] pathParts = pathInfo.split("/");
            if (pathParts.length != 2) {
                sendErrorResponse(response, HttpServletResponse.SC_BAD_REQUEST,
                        "Invalid path", span);
                return;
            }

            Long userId = Long.parseLong(pathParts[1]);

            span.setAllAttributes(Attributes.builder()
                    .put("user_id", userId)
                    .put("operation", "delete_user")
                    .build());

            // Delete user
            boolean deleted = userService.deleteUser(userId);

            if (deleted) {
                span.setAllAttributes(Attributes.builder()
                        .put("user_deleted", true)
                        .build());

                response.setStatus(HttpServletResponse.SC_NO_CONTENT);
            } else {
                span.setAllAttributes(Attributes.builder()
                        .put("user_found", false)
                        .build());

                sendErrorResponse(response, HttpServletResponse.SC_NOT_FOUND,
                        "User not found", span);
            }

            span.setStatus(StatusCode.OK);

        } catch (NumberFormatException e) {
            span.recordException(e);
            sendErrorResponse(response, HttpServletResponse.SC_BAD_REQUEST,
                    "Invalid user ID format", span);
        } catch (Exception e) {
            span.recordException(e);
            span.setStatus(StatusCode.ERROR, e.getMessage());
            sendErrorResponse(response, HttpServletResponse.SC_INTERNAL_SERVER_ERROR,
                    "Failed to delete user", span);
        } finally {
            span.end();
        }
    }

    private void handleGetAllUsers(HttpServletRequest request, HttpServletResponse response, Span span)
            throws IOException {

        // Get pagination parameters
        int page = getIntParameter(request, "page", 0);
        int size = getIntParameter(request, "size", 10);

        span.setAllAttributes(Attributes.builder()
                .put("operation", "get_all_users")
                .put("page", page)
                .put("size", size)
                .build());

        List<User> users = userService.getAllUsers(page, size);

        span.setAllAttributes(Attributes.builder()
                .put("users_count", users.size())
                .build());

        response.setStatus(HttpServletResponse.SC_OK);
        response.setContentType("application/json");

        // Create response with pagination info
        String jsonResponse = objectMapper.writeValueAsString(Map.of(
                "users", users,
                "pagination", Map.of(
                        "page", page,
                        "size", size,
                        "count", users.size()
                )
        ));

        response.getWriter().write(jsonResponse);
    }

    private void handleGetUserById(HttpServletRequest request, HttpServletResponse response,
                                   Long userId, Span span) throws IOException {

        span.setAllAttributes(Attributes.builder()
                .put("operation", "get_user_by_id")
                .put("user_id", userId)
                .build());

        User user = userService.getUserById(userId);

        if (user != null) {
            span.setAllAttributes(Attributes.builder()
                    .put("user_found", true)
                    .put("user_email", user.getEmail())
                    .build());

            response.setStatus(HttpServletResponse.SC_OK);
            response.setContentType("application/json");
            objectMapper.writeValue(response.getOutputStream(), user);
        } else {
            span.setAllAttributes(Attributes.builder()
                    .put("user_found", false)
                    .build());

            sendErrorResponse(response, HttpServletResponse.SC_NOT_FOUND,
                    "User not found", span);
        }
    }

    private void sendErrorResponse(HttpServletResponse response, int statusCode,
                                   String message, Span span) throws IOException {

        span.setAllAttributes(Attributes.builder()
                .put("error", true)
                .put("error_message", message)
                .put("status_code", statusCode)
                .build());

        response.setStatus(statusCode);
        response.setContentType("application/json");

        String errorJson = objectMapper.writeValueAsString(Map.of(
                "error", message,
                "status", statusCode
        ));

        response.getWriter().write(errorJson);
    }

    private int getIntParameter(HttpServletRequest request, String name, int defaultValue) {
        String value = request.getParameter(name);
        if (value != null) {
            try {
                return Integer.parseInt(value);
            } catch (NumberFormatException e) {
                return defaultValue;
            }
        }
        return defaultValue;
    }
}
```

**HealthCheckServlet.java**

```java
package com.example.servlet;

import com.example.config.OpenTelemetryConfig;
import com.fasterxml.jackson.databind.ObjectMapper;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;

import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.Map;

@WebServlet(urlPatterns = {"/health", "/api/health"})
public class HealthCheckServlet extends HttpServlet {

    private ObjectMapper objectMapper;
    private Tracer tracer;

    @Override
    public void init() throws ServletException {
        this.objectMapper = new ObjectMapper();
        this.tracer = OpenTelemetryConfig.getTracer();
    }

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException {

        Span span = tracer.spanBuilder("health_check")
                .setAllAttributes(Attributes.builder()
                        .put("servlet", "HealthCheckServlet")
                        .put("action", "health_check")
                        .build())
                .startSpan();

        try {
            // Check application health
            boolean healthy = checkApplicationHealth();

            Map<String, Object> healthStatus = Map.of(
                    "status", healthy ? "UP" : "DOWN",
                    "timestamp", System.currentTimeMillis(),
                    "details", Map.of(
                            "application", "tomcat-app",
                            "version", "1.0.0"
                    )
            );

            span.setAllAttributes(Attributes.builder()
                    .put("health_status", healthy ? "UP" : "DOWN")
                    .build());

            response.setStatus(healthy ? HttpServletResponse.SC_OK : HttpServletResponse.SC_SERVICE_UNAVAILABLE);
            response.setContentType("application/json");
            objectMapper.writeValue(response.getOutputStream(), healthStatus);

        } catch (Exception e) {
            span.recordException(e);
            response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
        } finally {
            span.end();
        }
    }

    private boolean checkApplicationHealth() {
        // Add your health checks here
        // For example: database connectivity, external service availability, etc.
        return true;
    }
}
```

## Production Deployment

### Docker Configuration

**Dockerfile**

```dockerfile
FROM tomcat:9.0-openjdk11-slim

# Install curl for health checks
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*

# Download OpenTelemetry Java agent
ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar /usr/local/tomcat/lib/opentelemetry-javaagent.jar

# Copy application WAR
COPY target/tomcat-app.war /usr/local/tomcat/webapps/

# Copy Tomcat configuration
COPY conf/server.xml /usr/local/tomcat/conf/
COPY conf/web.xml /usr/local/tomcat/conf/

# Set up OpenTelemetry environment
COPY setenv.sh /usr/local/tomcat/bin/
RUN chmod +x /usr/local/tomcat/bin/setenv.sh

# Create non-root user
RUN groupadd -r tomcat && useradd -r -g tomcat tomcat
RUN chown -R tomcat:tomcat /usr/local/tomcat
USER tomcat

# Expose port
EXPOSE 8080

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
  CMD curl -f http://localhost:8080/tomcat-app/health || exit 1

# Start Tomcat
CMD ["catalina.sh", "run"]
```

**setenv.sh**

```bash
#!/bin/bash

# OpenTelemetry Java Agent
export CATALINA_OPTS="$CATALINA_OPTS -javaagent:/usr/local/tomcat/lib/opentelemetry-javaagent.jar"

# OpenTelemetry Configuration
export OTEL_EXPORTER_OTLP_ENDPOINT="${OTEL_EXPORTER_OTLP_ENDPOINT:-$last9_otlp_endpoint}"
export OTEL_EXPORTER_OTLP_HEADERS="${OTEL_EXPORTER_OTLP_HEADERS:-Authorization=$last9_otlp_auth_header}"
export OTEL_SERVICE_NAME="${OTEL_SERVICE_NAME:-tomcat-app}"
export OTEL_RESOURCE_ATTRIBUTES="${OTEL_RESOURCE_ATTRIBUTES:-deployment.environment=production,service.version=1.0.0,server.type=tomcat}"
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_TRACES_SAMPLER=always_on

# JVM Configuration
export CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m -Xms512m"
export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseG1GC"
export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=200"

# Security settings
export CATALINA_OPTS="$CATALINA_OPTS -Djava.security.egd=file:/dev/urandom"
```

**docker-compose.yml**

```yaml
version: "3.8"

services:
  tomcat-app:
    build: .
    ports:
      - "8080:8080"
    environment:
      # OpenTelemetry Configuration
      - OTEL_EXPORTER_OTLP_ENDPOINT=$last9_otlp_endpoint
      - OTEL_EXPORTER_OTLP_HEADERS=Authorization=$last9_otlp_auth_header
      - OTEL_SERVICE_NAME=tomcat-app
      - OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.version=1.0.0,server.type=tomcat

      # Database Configuration
      - DB_URL=jdbc:postgresql://postgres:5432/tomcatdb
      - DB_USERNAME=user
      - DB_PASSWORD=password

      # JVM Configuration
      - JAVA_OPTS=-Xmx1g -Xms512m
    depends_on:
      - postgres
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/tomcat-app/health"]
      interval: 30s
      timeout: 10s
      retries: 3

  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: tomcatdb
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:
```

### Kubernetes Deployment

**k8s-deployment.yaml**

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-app
  labels:
    app: tomcat-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: tomcat-app
  template:
    metadata:
      labels:
        app: tomcat-app
    spec:
      containers:
        - name: tomcat-app
          image: your-registry/tomcat-app:latest
          ports:
            - containerPort: 8080
          env:
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "$last9_otlp_endpoint"
            - name: OTEL_EXPORTER_OTLP_HEADERS
              value: "Authorization=$last9_otlp_auth_header"
            - name: OTEL_SERVICE_NAME
              value: "tomcat-app"
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "deployment.environment=production,service.version=1.0.0,server.type=tomcat,k8s.cluster.name=production"
            - name: DB_URL
              valueFrom:
                secretKeyRef:
                  name: app-secrets
                  key: database-url
          livenessProbe:
            httpGet:
              path: /tomcat-app/health
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /tomcat-app/health
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 5
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "1Gi"
              cpu: "1000m"
---
apiVersion: v1
kind: Service
metadata:
  name: tomcat-app-service
spec:
  selector:
    app: tomcat-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: LoadBalancer
```

## Testing the Integration

1. **Start Tomcat**

   ```bash
   # Direct start
   $TOMCAT_HOME/bin/startup.sh

   # Or with Docker
   docker-compose up
   ```

2. **Test the endpoints**

   ```bash
   # Health check
   curl http://localhost:8080/tomcat-app/health

   # User endpoints
   curl http://localhost:8080/tomcat-app/api/users
   curl http://localhost:8080/tomcat-app/api/users/1

   # Create a user
   curl -X POST http://localhost:8080/tomcat-app/api/users \
     -H "Content-Type: application/json" \
     -d '{"name": "Test User", "email": "test@example.com"}'
   ```

3. **View telemetry in Last9**

   Check your Last9 dashboard for:

   - HTTP request traces with Tomcat-specific attributes
   - Servlet execution spans
   - Database operation traces
   - Custom business logic spans
   - Error tracking and correlation