# AWS Lambda

> Instrument AWS Lambda functions with OpenTelemetry using ADOT layers for automatic tracing and observability

Source: https://last9.io/docs/integrations/cloud-providers/aws-lambda/

Use OpenTelemetry to instrument your AWS Lambda functions and send telemetry data to Last9. This integration uses the AWS Distro for OpenTelemetry (ADOT) Layer for automatic instrumentation with no code changes required.

## Prerequisites

Before setting up AWS Lambda monitoring, ensure you have:

- **AWS Account**: With access to Lambda service
- **Lambda Functions**: Deployed functions to instrument
- **AWS Console Access**: Or AWS CLI configured
- **Last9 Account**: With OpenTelemetry integration credentials

## Supported Runtimes

ADOT layers support the following Lambda runtimes:

- **Node.js**: 14.x, 16.x, 18.x, 20.x
- **Python**: 3.8, 3.9, 3.10, 3.11, 3.12
- **Java**: 8, 11, 17, 21
- **.NET**: Core 3.1, 6.0, 8.0

1.  **Find the ADOT Layer ARN**

    Get the latest ADOT layer ARN for your AWS region and runtime from the [AWS ADOT Lambda documentation](https://aws-otel.github.io/docs/getting-started/lambda).

**Popular Regions**

    **US East 1 (N. Virginia) - us-east-1:**

    ```
    # Node.js
    arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4

    # Python
    arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python-amd64-ver-1-20-0:3

    # Java
    arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-32-0:3
    ```

    **US West 2 (Oregon) - us-west-2:**

    ```
    # Node.js
    arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4

    # Python
    arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-python-amd64-ver-1-20-0:3

    # Java
    arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-32-0:3
    ```

**Asia Pacific**

    **Asia Pacific (Mumbai) - ap-south-1:**

    ```
    # Node.js
    arn:aws:lambda:ap-south-1:615299751070:layer:AWSOpenTelemetryDistroJs:9

    # Python
    arn:aws:lambda:ap-south-1:615299751070:layer:AWSOpenTelemetryDistro:5

    # Java
    arn:aws:lambda:ap-south-1:615299751070:layer:AWSOpenTelemetryDistroJava:7
    ```

    **Asia Pacific (Singapore) - ap-southeast-1:**

    ```
    # Node.js
    arn:aws:lambda:ap-southeast-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4

    # Python
    arn:aws:lambda:ap-southeast-1:901920570463:layer:aws-otel-python-amd64-ver-1-20-0:3

    # Java
    arn:aws:lambda:ap-southeast-1:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-32-0:3
    ```

**Europe**

    **Europe (Ireland) - eu-west-1:**

    ```
    # Node.js
    arn:aws:lambda:eu-west-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4

    # Python
    arn:aws:lambda:eu-west-1:901920570463:layer:aws-otel-python-amd64-ver-1-20-0:3

    # Java
    arn:aws:lambda:eu-west-1:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-32-0:3
    ```

2.  **Add the ADOT Layer to Your Lambda Function**

**AWS Console**

    Using the AWS Lambda Console:

    1.  Open your Lambda function in the AWS Console
    2.  Scroll down to the **Layers** section
    3.  Click **Add a layer**
    4.  Select **Specify an ARN**
    5.  Paste the appropriate ADOT layer ARN for your region and runtime
    6.  Click **Add**

**AWS CLI**

    Using the AWS CLI:

    ```bash
    aws lambda update-function-configuration \
      --function-name your-function-name \
      --layers arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4
    ```

    Replace with your function name and appropriate layer ARN.

**Terraform**

    Using Terraform:

    ```hcl
    resource "aws_lambda_function" "example" {
      function_name = "your-function-name"
      # ... other configuration

      layers = [
        "arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4"
      ]
    }
    ```

3.  **Configure Environment Variables**

    Add the following environment variables to your Lambda function configuration:

**AWS Console**

    1. Go to **Configuration** → **Environment variables**
    2. Click **Edit** and add these variables:

    ```
    AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument
    OTEL_SERVICE_NAME=<your-service-name>
    OTEL_EXPORTER_OTLP_ENDPOINT=$last9_otlp_endpoint
    OTEL_EXPORTER_OTLP_HEADERS=authorization=$last9_otlp_auth_header
    OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
    OTEL_TRACES_EXPORTER=otlp
    OTEL_TRACES_SAMPLER=always_on
    OTEL_PROPAGATORS=tracecontext,baggage
    OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production
    ```

    Replace `<your-service-name>` with a descriptive service name (e.g., `payment-lambda`, `user-service`).

**AWS CLI**

    ```bash
    aws lambda update-function-configuration \
      --function-name your-function-name \
      --environment Variables='{
        "AWS_LAMBDA_EXEC_WRAPPER": "/opt/otel-instrument",
        "OTEL_SERVICE_NAME": "your-service-name",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "$last9_otlp_endpoint",
        "OTEL_EXPORTER_OTLP_HEADERS": "authorization=$last9_otlp_auth_header",
        "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
        "OTEL_TRACES_EXPORTER": "otlp",
        "OTEL_TRACES_SAMPLER": "always_on",
        "OTEL_PROPAGATORS": "tracecontext,baggage",
        "OTEL_RESOURCE_ATTRIBUTES": "deployment.environment=production"
      }'
    ```

**Terraform**

    ```hcl
    resource "aws_lambda_function" "example" {
      function_name = "your-function-name"
      # ... other configuration

      environment {
        variables = {
          AWS_LAMBDA_EXEC_WRAPPER          = "/opt/otel-instrument"
          OTEL_SERVICE_NAME                = "your-service-name"
          OTEL_EXPORTER_OTLP_ENDPOINT      = "$last9_otlp_endpoint"
          OTEL_EXPORTER_OTLP_HEADERS       = "authorization=$last9_otlp_auth_header"
          OTEL_EXPORTER_OTLP_PROTOCOL      = "http/protobuf"
          OTEL_TRACES_EXPORTER             = "otlp"
          OTEL_TRACES_SAMPLER              = "always_on"
          OTEL_PROPAGATORS                 = "tracecontext,baggage"
          OTEL_RESOURCE_ATTRIBUTES         = "deployment.environment=production"
        }
      }
    }
    ```

    **Important Configuration Notes:**

    - Replace `<your-service-name>` with a descriptive name (e.g., `payment-lambda`, `user-service`)
    - Use lowercase `authorization` in the headers configuration
    - Update the `deployment.environment` to match your environment (production, staging, development)
    - Replace placeholder values with your actual Last9 credentials

4.  **Test and Verify Installation**

    Test your Lambda function to ensure instrumentation is working:

**AWS Console Test**

    1.  Go to your Lambda function in the AWS Console
    2.  Click **Test**
    3.  Create a test event or use an existing one
    4.  Click **Test** to invoke the function
    5.  Check the execution logs for OpenTelemetry initialization messages

**AWS CLI Invoke**

    ```bash
    aws lambda invoke \
      --function-name your-function-name \
      --payload '{"test": "event"}' \
      response.json

    # View the response
    cat response.json
    ```

**API Gateway (if applicable)**

    If your Lambda is behind API Gateway, test via HTTP request:

    ```bash
    curl -X POST https://your-api-gateway-url/your-endpoint \
      -H "Content-Type: application/json" \
      -d '{"test": "data"}'
    ```

## Understanding the Setup

### AWS Distro for OpenTelemetry (ADOT)

ADOT provides:

- **Zero-Code Instrumentation**: No application code changes required
- **AWS Service Integration**: Built-in support for AWS services and APIs
- **Lambda Optimizations**: Optimized for serverless environments
- **Multiple Language Support**: Comprehensive runtime coverage

### Environment Variables Explained

| Variable                      | Purpose                      | Example                       |
| ----------------------------- | ---------------------------- | ----------------------------- |
| `AWS_LAMBDA_EXEC_WRAPPER`     | Enables ADOT instrumentation | `/opt/otel-instrument`        |
| `OTEL_SERVICE_NAME`           | Service identifier in traces | `payment-service`             |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Last9 traces endpoint        | Last9 provided URL            |
| `OTEL_EXPORTER_OTLP_HEADERS`  | Authentication headers       | Last9 auth token              |
| `OTEL_TRACES_SAMPLER`         | Sampling strategy            | `always_on` or `traceidratio` |
| `OTEL_RESOURCE_ATTRIBUTES`    | Additional metadata          | `deployment.environment=prod` |

### What Gets Traced

The ADOT layer automatically traces:

- **Lambda Handler**: Function entry and exit
- **AWS SDK Calls**: DynamoDB, S3, SQS, etc.
- **HTTP Requests**: Outbound API calls
- **Database Calls**: RDS, DynamoDB operations
- **Message Queue Operations**: SQS send/receive

## Advanced Configuration

### Sampling Configuration

Control trace sampling to manage costs:

```bash
# Production: Sample 10% of traces
OTEL_TRACES_SAMPLER=traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

# Development: Sample all traces
OTEL_TRACES_SAMPLER=always_on
```

### Custom Resource Attributes

Add additional metadata to traces:

```bash
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,team=payments,version=2.1.0,region=us-east-1
```

### Error Handling Configuration

Configure error reporting:

```bash
OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=true
OTEL_INSTRUMENTATION_AWS_SDK_SUPPRESSORS_MESSAGING_RECEIVE_TELEMETRY_ENABLED=true
```

## Verification

1. **Check Lambda Execution Logs**

   Look for ADOT initialization messages in CloudWatch Logs:

   ```bash
   aws logs filter-log-events \
     --log-group-name "/aws/lambda/your-function-name" \
     --filter-pattern "OpenTelemetry"
   ```

2. **Verify Layer Attachment**

   Confirm the ADOT layer is properly attached:

   ```bash
   aws lambda get-function-configuration \
     --function-name your-function-name \
     --query 'Layers[?starts_with(Arn, `arn:aws:lambda`)].Arn'
   ```

3. **View Traces in Last9**

   1. Log into your [Last9 dashboard](https://app.last9.io/)
   2. Navigate to the **Traces** section
   3. Filter by your service name
   4. Traces should appear within 1-2 minutes of invocation

4. **Test Trace Propagation**

   If your Lambda calls other services, verify trace propagation by checking connected spans in the trace view.

## Common Issues

### No Traces Appearing

**Enable Debug Logging:**

```bash
OTEL_LOG_LEVEL=debug
```

**Check Common Issues:**

- Verify ADOT layer is attached to your function
- Confirm environment variables are set correctly (no extra quotes)
- Check CloudWatch Logs for error messages
- Ensure Last9 credentials are valid

### Lambda Cold Start Issues

**Optimize Cold Starts:**

- Use provisioned concurrency for critical functions
- Consider using reserved concurrency to limit concurrent executions
- Monitor cold start metrics in CloudWatch

### Memory and Performance Impact

**Monitor Resource Usage:**

- Check function memory usage in CloudWatch metrics
- Adjust memory allocation if needed
- Monitor execution duration for performance impact

### Error Messages

**Common Error Resolutions:**

| Error               | Solution                                   |
| ------------------- | ------------------------------------------ |
| "Recording is off"  | Set `OTEL_TRACES_SAMPLER=always_on`        |
| "Layer not found"   | Use correct layer ARN for your region      |
| "Permission denied" | Check Lambda execution role permissions    |
| "Timeout"           | Increase function timeout or optimize code |

## Ruby Lambda

Ruby does not have an AWS-managed ADOT language layer. Instead, use the OpenTelemetry Ruby SDK with a direct OTLP export to Last9 — no collector sidecar required.

1. **Add OTel gems to your Gemfile**

   ```ruby
   gem 'opentelemetry-exporter-otlp'
   gem 'opentelemetry-instrumentation-aws_lambda'
   gem 'opentelemetry-instrumentation-aws_sdk'
   gem 'opentelemetry-sdk'
   gem 'rexml'  # required by opentelemetry-instrumentation-aws_sdk for XML parsing
   ```

   Then install:

   ```bash
   bundle install
   ```

2. **Create `setup_otel.rb`**

   Initialize the OTel SDK once. The `return if defined?(OTEL_TRACER)` guard prevents double-initialization when tests configure their own exporter first.

   ```ruby
   # frozen_string_literal: true

   return if defined?(OTEL_TRACER)

   require 'opentelemetry/sdk'
   require 'opentelemetry/exporter/otlp'
   require 'opentelemetry/instrumentation/aws_lambda'
   require 'opentelemetry/instrumentation/aws_sdk'

   OpenTelemetry::SDK.configure do |c|
     c.service_name = ENV.fetch('OTEL_SERVICE_NAME', 'ruby-lambda')
     c.use 'OpenTelemetry::Instrumentation::AwsLambda'  # auto root span per invocation
     c.use 'OpenTelemetry::Instrumentation::AwsSdk'      # auto-instruments S3, SQS, SES, DynamoDB
     c.add_span_processor(
       OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
         OpenTelemetry::Exporter::OTLP::Exporter.new
       )
     )
   end

   OTEL_TRACER = OpenTelemetry.tracer_provider.tracer(
     ENV.fetch('OTEL_SERVICE_NAME', 'ruby-lambda'),
     '1.0.0'
   )
   ```

3. **Instrument your Lambda handler**

   Require `setup_otel` at the top of your handler file. Call `force_flush` in an `ensure` block — Lambda freezes the process after the response returns, which kills buffered spans without this.

   ```ruby
   # frozen_string_literal: true

   require_relative './setup_otel'

   def lambda_handler(event:, context:)
     OTEL_TRACER.in_span('process_event') do |span|
       # your business logic
       span.set_attribute('event.source', event['source'].to_s)
       { status: 'ok' }
     rescue StandardError => e
       span.record_exception(e)
       span.status = OpenTelemetry::Trace::Status.error(e.message)
       raise
     end
   ensure
     # Critical: flush buffered spans before Lambda process freezes.
     OpenTelemetry.tracer_provider.force_flush
   end
   ```

   The trace hierarchy produced:

   ```
   lambda_handler           ← auto (AwsLambda instrumentation)
     └─ process_event       ← manual span
          └─ S3.GetObject   ← auto (AwsSdk instrumentation)
   ```

4. **Set environment variables**

   ```bash
   aws lambda update-function-configuration \
     --function-name your-function-name \
     --environment 'Variables={
       OTEL_SERVICE_NAME=ruby-lambda-example,
       OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-aps1.last9.io:443,
       OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <your-base64-credentials>,
       OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf,
       OTEL_TRACES_SAMPLER=always_on,
       OTEL_PROPAGATORS=tracecontext,baggage,xray,
       OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production
     }'
   ```

   :::note
   Do **not** set `AWS_LAMBDA_EXEC_WRAPPER` for Ruby — that env var only works with the Python/Node/Java ADOT language layers.
   :::

5. **Test locally before deploying**

   The handler is plain Ruby — test OTel export without AWS infrastructure:

   ```ruby
   # run_local.rb
   $LOAD_PATH.unshift(__dir__)
   require_relative 'lambda_function'

   result = lambda_handler(event: { 'message' => 'test' }, context: nil)
   puts result.inspect
   ```

   ```bash
   OTEL_SERVICE_NAME=ruby-lambda-example \
   OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-aps1.last9.io:443 \
   OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <your-credentials>" \
   OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
   OTEL_TRACES_SAMPLER=always_on \
   OTEL_RESOURCE_ATTRIBUTES=deployment.environment=local \
   bundle exec ruby run_local.rb
   ```

   Traces appear in [Last9](https://app.last9.io/traces) within seconds.

6. **Package gems and deploy**

   Lambda requires gems bundled inside the deployment zip:

   ```bash
   bundle config set --local path 'vendor/bundle'
   bundle install
   zip -qr function.zip lambda_function.rb setup_otel.rb Gemfile Gemfile.lock vendor/

   aws lambda update-function-code \
     --function-name your-function-name \
     --zip-file fileb://function.zip
   ```

A complete working example with a deploy script is available at [last9/opentelemetry-examples — aws/lambda-ruby](https://github.com/last9/opentelemetry-examples/tree/main/aws/lambda-ruby).

## SQS-Triggered Lambda — Trace Propagation

When Lambda is triggered by SQS via Event Source Mapping (ESM), traces from the upstream producer are **not automatically linked**. You must extract W3C `traceparent` from SQS MessageAttributes and use it as the parent context for your Lambda spans.

### How It Works

1. The upstream service (producer) injects `traceparent` into SQS MessageAttributes when sending a message
2. SQS delivers the message to Lambda via ESM
3. Lambda extracts the trace context from the record's `messageAttributes` and creates child spans

### ESM Attribute Format

:::caution
SQS Event Source Mapping delivers MessageAttributes with **lowercase** keys, unlike the SQS SDK:
:::

| Field        | ESM Format (Lambda trigger) | SDK Format (ReceiveMessage) |
| ------------ | --------------------------- | --------------------------- |
| String value | `stringValue`               | `StringValue`               |
| Data type    | `dataType`                  | `DataType`                  |

Your extraction code must handle both formats.

### Extract Trace Context in Lambda

**Python**

```python
from opentelemetry.propagate import extract
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def extract_context_from_sqs_record(record):
    carrier = {}
    # Support both Lambda ESM ("messageAttributes") and SDK ReceiveMessage ("MessageAttributes")
    msg_attrs = record.get("messageAttributes") or record.get("MessageAttributes") or {}
    for key, attr in msg_attrs.items():
        # ESM uses lowercase "stringValue"; SDK uses "StringValue"
        value = (
            attr.get("stringValue")
            if "stringValue" in attr
            else attr.get("StringValue")
        )
        if value is not None:
            carrier[key] = value
    return extract(carrier)

def handler(event, context):
    for record in event.get("Records", []):
        ctx = extract_context_from_sqs_record(record)

        with tracer.start_as_current_span(
            "process_message",
            context=ctx,  # Links to producer's trace
            kind=trace.SpanKind.CONSUMER,
        ):
            # Your business logic here
            pass
```

**Node.js**

```javascript
const { propagation, trace, SpanKind } = require("@opentelemetry/api");

const tracer = trace.getTracer("lambda-consumer");

function extractContextFromSqsRecord(record) {
  const carrier = {};
  const attrs = record.messageAttributes || {};
  for (const [key, attr] of Object.entries(attrs)) {
    const value = attr.stringValue || attr.StringValue;
    if (value) carrier[key] = value;
  }
  return propagation.extract(trace.context.active(), carrier);
}

exports.handler = async (event) => {
  for (const record of event.Records) {
    const ctx = extractContextFromSqsRecord(record);

    await tracer.startActiveSpan(
      "process_message",
      { kind: SpanKind.CONSUMER },
      ctx,
      async (span) => {
        // Your business logic here
        span.end();
      },
    );
  }
};
```

**Ruby**

```ruby
require 'opentelemetry'

TRACER = OpenTelemetry.tracer_provider.tracer('lambda-consumer')

def extract_context_from_sqs_record(record)
  carrier = {}
  # Support both Lambda ESM ("messageAttributes") and SDK ReceiveMessage ("MessageAttributes")
  attrs = record['messageAttributes'] || record['MessageAttributes'] || {}
  attrs.each do |key, attr|
    # ESM uses lowercase "stringValue"; SDK uses "StringValue"
    value = attr['stringValue'] || attr['StringValue']
    carrier[key] = value if value
  end
  OpenTelemetry.propagation.extract(carrier)
end

def lambda_handler(event:, context:)
  (event['Records'] || []).each do |record|
    parent_ctx = extract_context_from_sqs_record(record)

    OpenTelemetry::Context.with_current(parent_ctx) do
      TRACER.in_span('process_message', kind: OpenTelemetry::Trace::SpanKind::CONSUMER) do |span|
        # Your business logic here
      end
    end
  end
ensure
  OpenTelemetry.tracer_provider.force_flush
end
```

### Producer Side — Injecting Trace Context

The upstream service must inject trace context into SQS MessageAttributes. See the [AWS SQS integration — Trace Context Propagation](/docs/integrations/messaging/aws-sqs/#trace-context-propagation-through-sqs) for producer-side code examples.

### Batch Processing

Each SQS record in a batch may carry a different trace context (from different producer requests). Always extract and create spans **per-record**, not per-invocation. Enable `ReportBatchItemFailures` on your ESM for partial batch failure reporting.

### Full Example

See the [Python SQS → Lambda trace propagation example](https://github.com/last9/opentelemetry-examples/tree/main/python/aws-sqs-lambda) for a complete working implementation with local testing.

## Lambda Log-Trace Correlation

By default, Lambda logs in CloudWatch have no link to the traces generated by the ADOT layer. To correlate logs with traces in Last9, inject the current `trace_id` and `span_id` into your structured log output before forwarding logs.

### Why Logs and Traces Disconnect

ADOT automatically creates spans for your Lambda invocations. Those spans carry a `trace_id`. Your application logs — written with `print`, `logging`, or a structured logger — don't know about that trace_id unless you explicitly inject it.

Without injection, you see traces in Last9's Traces view and logs separately, with no way to jump from a log line to the trace that produced it.

### Injecting Trace Context into Logs

**Python**

Use the OTel SDK to get the current span context and add `trace_id`/`span_id` to your log records:

```python
import json
import logging
from opentelemetry import trace

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def get_trace_context():
    span = trace.get_current_span()
    ctx = span.get_span_context()
    if ctx.is_valid:
        return {
            "trace_id": format(ctx.trace_id, "032x"),
            "span_id": format(ctx.span_id, "016x"),
        }
    return {}

def handler(event, context):
    log_entry = {
        "level": "INFO",
        "message": "Processing event",
        **get_trace_context(),  # injects trace_id and span_id
    }
    print(json.dumps(log_entry))
    # ... business logic
```

**With AWS Lambda Powertools (recommended for Python):**

```python
from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging import correlation_paths

logger = Logger(service="payment-lambda")

@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
def handler(event, context):
    logger.info("Processing request")
    # Powertools automatically includes xray_trace_id
    # Add OTel trace_id manually via get_trace_context() if needed
```

**Node.js**

```javascript
const { trace } = require("@opentelemetry/api");

function getTraceContext() {
  const span = trace.getActiveSpan();
  if (!span) return {};
  const ctx = span.spanContext();
  return {
    trace_id: ctx.traceId,
    span_id: ctx.spanId,
  };
}

exports.handler = async (event) => {
  console.log(
    JSON.stringify({
      level: "INFO",
      message: "Processing event",
      ...getTraceContext(),
    }),
  );
  // ... business logic
};
```

**With pino:**

```javascript
const pino = require("pino");
const { trace } = require("@opentelemetry/api");

const logger = pino({
  mixin() {
    const span = trace.getActiveSpan();
    if (!span) return {};
    const ctx = span.spanContext();
    return { trace_id: ctx.traceId, span_id: ctx.spanId };
  },
});

exports.handler = async (event) => {
  logger.info("Processing event");
};
```

**Ruby**

```ruby
require 'json'
require 'opentelemetry'

def trace_context
  span = OpenTelemetry::Trace.current_span
  ctx = span.context
  return {} unless ctx.valid?
  {
    trace_id: ctx.hex_trace_id,
    span_id: ctx.hex_span_id,
  }
end

def lambda_handler(event:, context:)
  puts JSON.generate({ level: 'INFO', message: 'Processing event', **trace_context })
  # ... business logic
ensure
  OpenTelemetry.tracer_provider.force_flush
end
```

### Forwarding Lambda Logs to Last9

After injecting trace context into logs, forward your CloudWatch logs to Last9 so they appear alongside traces:

**Option 1 — CloudWatch Subscription Filter (push-based):**

1. Create a Lambda function that forwards logs to Last9's OTLP endpoint
2. Add a CloudWatch Logs subscription filter on your Lambda's log group to trigger it

**Option 2 — OTel Collector sidecar (ECS/fargate only):**

If running Lambda on ECS with a sidecar collector, configure the `filelog` receiver to tail `/proc/1/fd/1`.

**Option 3 — Direct OTLP logs from Lambda:**

Install the OTel Logs SDK and emit logs directly to Last9:

```python
from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor

provider = LoggerProvider()
provider.add_log_record_processor(
    BatchLogRecordProcessor(OTLPLogExporter())
)
set_logger_provider(provider)
```

:::caution
Direct OTLP logs from Lambda require `force_flush` before the handler returns — Lambda freezes the process immediately after the response, which drops buffered log records.
:::

### Searching Correlated Logs in Last9

Once logs carry `trace_id`, filter in Last9 Log Management:

```
trace_id = "4bf92f3577b34da6a3ce929d0e0e4736"
```

Or from a trace in Last9's Traces view, click **View Logs** to jump directly to log lines matching that trace's `trace_id`.

## Best Practices

- **Service Naming**: Use descriptive, consistent service names across your Lambda functions
- **Environment Segregation**: Use different service names or resource attributes per environment
- **Sampling Strategy**: Use appropriate sampling rates for production to control costs
- **Monitoring**: Set up CloudWatch alarms for Lambda errors and duration
- **Resource Attribution**: Include meaningful metadata like version, team, and environment
- **Testing**: Test instrumentation in development before deploying to production

---

## Troubleshooting

Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:support@last9.io) if you have any questions.