AWS Lambda

Use OpenTelemetry to instrument your AWS Lambda functions and send telemetry data to Last9. This integration uses the AWS Distro for OpenTelemetry (ADOT) Layer for automatic instrumentation with no code changes required.

Prerequisites

Before setting up AWS Lambda monitoring, ensure you have:

• AWS Account: With access to Lambda service
• Lambda Functions: Deployed functions to instrument
• AWS Console Access: Or AWS CLI configured
• Last9 Account: With OpenTelemetry integration credentials

Supported Runtimes

ADOT layers support the following Lambda runtimes:

• Node.js: 14.x, 16.x, 18.x, 20.x
• Python: 3.8, 3.9, 3.10, 3.11, 3.12
• Java: 8, 11, 17, 21
• .NET: Core 3.1, 6.0, 8.0

1. Find the ADOT Layer ARN

   Get the latest ADOT layer ARN for your AWS region and runtime from the AWS ADOT Lambda documentation.

   • Popular Regions
   • Asia Pacific
   • Europe

   US East 1 (N. Virginia) - us-east-1:

   # Node.js
   arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4
   
   # Python
   arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python-amd64-ver-1-20-0:3
   
   # Java
   arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-32-0:3

   US West 2 (Oregon) - us-west-2:

   # Node.js
   arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4
   
   # Python
   arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-python-amd64-ver-1-20-0:3
   
   # Java
   arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-32-0:3

2. Add the ADOT Layer to Your Lambda Function

   • AWS Console
   • AWS CLI
   • Terraform

   Using the AWS Lambda Console:

   1. Open your Lambda function in the AWS Console
   2. Scroll down to the Layers section
   3. Click Add a layer
   4. Select Specify an ARN
   5. Paste the appropriate ADOT layer ARN for your region and runtime
   6. Click Add

3. Configure Environment Variables

   Add the following environment variables to your Lambda function configuration:

   • AWS Console
   • AWS CLI
   • Terraform
   1. Go to Configuration → Environment variables
   2. Click Edit and add these variables:

   AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument
   OTEL_SERVICE_NAME=<your-service-name>
   OTEL_EXPORTER_OTLP_ENDPOINT=$last9_otlp_endpoint
   OTEL_EXPORTER_OTLP_HEADERS=authorization=$last9_otlp_auth_header
   OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
   OTEL_TRACES_EXPORTER=otlp
   OTEL_TRACES_SAMPLER=always_on
   OTEL_PROPAGATORS=tracecontext,baggage
   OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production

   Replace <your-service-name> with a descriptive service name (e.g., payment-lambda, user-service).

   Important Configuration Notes:

   • Replace <your-service-name> with a descriptive name (e.g., payment-lambda, user-service)
   • Use lowercase authorization in the headers configuration
   • Update the deployment.environment to match your environment (production, staging, development)
   • Replace placeholder values with your actual Last9 credentials

4. Test and Verify Installation

   Test your Lambda function to ensure instrumentation is working:

   • AWS Console Test
   • AWS CLI Invoke
   • API Gateway (if applicable)
   1. Go to your Lambda function in the AWS Console
   2. Click Test
   3. Create a test event or use an existing one
   4. Click Test to invoke the function
   5. Check the execution logs for OpenTelemetry initialization messages

Understanding the Setup

AWS Distro for OpenTelemetry (ADOT)

ADOT provides:

• Zero-Code Instrumentation: No application code changes required
• AWS Service Integration: Built-in support for AWS services and APIs
• Lambda Optimizations: Optimized for serverless environments
• Multiple Language Support: Comprehensive runtime coverage

Environment Variables Explained

What Gets Traced

The ADOT layer automatically traces:

• Lambda Handler: Function entry and exit
• AWS SDK Calls: DynamoDB, S3, SQS, etc.
• HTTP Requests: Outbound API calls
• Database Calls: RDS, DynamoDB operations
• Message Queue Operations: SQS send/receive

Advanced Configuration

Sampling Configuration

Control trace sampling to manage costs:

# Production: Sample 10% of traces
OTEL_TRACES_SAMPLER=traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

# Development: Sample all traces
OTEL_TRACES_SAMPLER=always_on

Custom Resource Attributes

Add additional metadata to traces:

OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,team=payments,version=2.1.0,region=us-east-1

Error Handling Configuration

Configure error reporting:

OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=true
OTEL_INSTRUMENTATION_AWS_SDK_SUPPRESSORS_MESSAGING_RECEIVE_TELEMETRY_ENABLED=true

Verification

1. Check Lambda Execution Logs

   Look for ADOT initialization messages in CloudWatch Logs:

   aws logs filter-log-events \
     --log-group-name "/aws/lambda/your-function-name" \
     --filter-pattern "OpenTelemetry"

2. Verify Layer Attachment

   Confirm the ADOT layer is properly attached:

   aws lambda get-function-configuration \
     --function-name your-function-name \
     --query 'Layers[?starts_with(Arn, `arn:aws:lambda`)].Arn'

3. View Traces in Last9

   1. Log into your Last9 dashboard
   2. Navigate to the Traces section
   3. Filter by your service name
   4. Traces should appear within 1-2 minutes of invocation

4. Test Trace Propagation

   If your Lambda calls other services, verify trace propagation by checking connected spans in the trace view.

Common Issues

No Traces Appearing

Enable Debug Logging:

OTEL_LOG_LEVEL=debug

Check Common Issues:

• Verify ADOT layer is attached to your function
• Confirm environment variables are set correctly (no extra quotes)
• Check CloudWatch Logs for error messages
• Ensure Last9 credentials are valid

Lambda Cold Start Issues

Optimize Cold Starts:

• Use provisioned concurrency for critical functions
• Consider using reserved concurrency to limit concurrent executions
• Monitor cold start metrics in CloudWatch

Memory and Performance Impact

Monitor Resource Usage:

• Check function memory usage in CloudWatch metrics
• Adjust memory allocation if needed
• Monitor execution duration for performance impact

Error Messages

Common Error Resolutions:

Ruby Lambda

Ruby does not have an AWS-managed ADOT language layer. Instead, use the OpenTelemetry Ruby SDK with a direct OTLP export to Last9 — no collector sidecar required.

1. Add OTel gems to your Gemfile

   gem 'opentelemetry-exporter-otlp'
   gem 'opentelemetry-instrumentation-aws_lambda'
   gem 'opentelemetry-instrumentation-aws_sdk'
   gem 'opentelemetry-sdk'
   gem 'rexml'  # required by opentelemetry-instrumentation-aws_sdk for XML parsing

   Then install:

   bundle install

2. Create setup_otel.rb

   Initialize the OTel SDK once. The return if defined?(OTEL_TRACER) guard prevents double-initialization when tests configure their own exporter first.

   # frozen_string_literal: true
   
   return if defined?(OTEL_TRACER)
   
   require 'opentelemetry/sdk'
   require 'opentelemetry/exporter/otlp'
   require 'opentelemetry/instrumentation/aws_lambda'
   require 'opentelemetry/instrumentation/aws_sdk'
   
   OpenTelemetry::SDK.configure do |c|
     c.service_name = ENV.fetch('OTEL_SERVICE_NAME', 'ruby-lambda')
     c.use 'OpenTelemetry::Instrumentation::AwsLambda'  # auto root span per invocation
     c.use 'OpenTelemetry::Instrumentation::AwsSdk'      # auto-instruments S3, SQS, SES, DynamoDB
     c.add_span_processor(
       OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
         OpenTelemetry::Exporter::OTLP::Exporter.new
       )
     )
   end
   
   OTEL_TRACER = OpenTelemetry.tracer_provider.tracer(
     ENV.fetch('OTEL_SERVICE_NAME', 'ruby-lambda'),
     '1.0.0'
   )

3. Instrument your Lambda handler

   Require setup_otel at the top of your handler file. Call force_flush in an ensure block — Lambda freezes the process after the response returns, which kills buffered spans without this.

   # frozen_string_literal: true
   
   require_relative './setup_otel'
   
   def lambda_handler(event:, context:)
     OTEL_TRACER.in_span('process_event') do |span|
       # your business logic
       span.set_attribute('event.source', event['source'].to_s)
       { status: 'ok' }
     rescue StandardError => e
       span.record_exception(e)
       span.status = OpenTelemetry::Trace::Status.error(e.message)
       raise
     end
   ensure
     # Critical: flush buffered spans before Lambda process freezes.
     OpenTelemetry.tracer_provider.force_flush
   end

   The trace hierarchy produced:

   lambda_handler           ← auto (AwsLambda instrumentation)
     └─ process_event       ← manual span
          └─ S3.GetObject   ← auto (AwsSdk instrumentation)

4. Set environment variables

   aws lambda update-function-configuration \
     --function-name your-function-name \
     --environment 'Variables={
       OTEL_SERVICE_NAME=ruby-lambda-example,
       OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-aps1.last9.io:443,
       OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <your-base64-credentials>,
       OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf,
       OTEL_TRACES_SAMPLER=always_on,
       OTEL_PROPAGATORS=tracecontext,baggage,xray,
       OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production
     }'

5. Test locally before deploying

   The handler is plain Ruby — test OTel export without AWS infrastructure:

   # run_local.rb
   $LOAD_PATH.unshift(__dir__)
   require_relative 'lambda_function'
   
   result = lambda_handler(event: { 'message' => 'test' }, context: nil)
   puts result.inspect

   OTEL_SERVICE_NAME=ruby-lambda-example \
   OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-aps1.last9.io:443 \
   OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <your-credentials>" \
   OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
   OTEL_TRACES_SAMPLER=always_on \
   OTEL_RESOURCE_ATTRIBUTES=deployment.environment=local \
   bundle exec ruby run_local.rb

   Traces appear in Last9 within seconds.

6. Package gems and deploy

   Lambda requires gems bundled inside the deployment zip:

   bundle config set --local path 'vendor/bundle'
   bundle install
   zip -qr function.zip lambda_function.rb setup_otel.rb Gemfile Gemfile.lock vendor/
   
   aws lambda update-function-code \
     --function-name your-function-name \
     --zip-file fileb://function.zip

A complete working example with a deploy script is available at last9/opentelemetry-examples — aws/lambda-ruby.

SQS-Triggered Lambda — Trace Propagation

When Lambda is triggered by SQS via Event Source Mapping (ESM), traces from the upstream producer are not automatically linked. You must extract W3C traceparent from SQS MessageAttributes and use it as the parent context for your Lambda spans.

How It Works

1. The upstream service (producer) injects traceparent into SQS MessageAttributes when sending a message
2. SQS delivers the message to Lambda via ESM
3. Lambda extracts the trace context from the record’s messageAttributes and creates child spans

ESM Attribute Format

Your extraction code must handle both formats.

Extract Trace Context in Lambda

from opentelemetry.propagate import extract
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def extract_context_from_sqs_record(record):
    carrier = {}
    # Support both Lambda ESM ("messageAttributes") and SDK ReceiveMessage ("MessageAttributes")
    msg_attrs = record.get("messageAttributes") or record.get("MessageAttributes") or {}
    for key, attr in msg_attrs.items():
        # ESM uses lowercase "stringValue"; SDK uses "StringValue"
        value = (
            attr.get("stringValue")
            if "stringValue" in attr
            else attr.get("StringValue")
        )
        if value is not None:
            carrier[key] = value
    return extract(carrier)

def handler(event, context):
    for record in event.get("Records", []):
        ctx = extract_context_from_sqs_record(record)

        with tracer.start_as_current_span(
            "process_message",
            context=ctx,  # Links to producer's trace
            kind=trace.SpanKind.CONSUMER,
        ):
            # Your business logic here
            pass

const { propagation, trace, SpanKind } = require("@opentelemetry/api");

const tracer = trace.getTracer("lambda-consumer");

function extractContextFromSqsRecord(record) {
  const carrier = {};
  const attrs = record.messageAttributes || {};
  for (const [key, attr] of Object.entries(attrs)) {
    const value = attr.stringValue || attr.StringValue;
    if (value) carrier[key] = value;
  }
  return propagation.extract(trace.context.active(), carrier);
}

exports.handler = async (event) => {
  for (const record of event.Records) {
    const ctx = extractContextFromSqsRecord(record);

    await tracer.startActiveSpan(
      "process_message",
      { kind: SpanKind.CONSUMER },
      ctx,
      async (span) => {
        // Your business logic here
        span.end();
      },
    );
  }
};

require 'opentelemetry'

TRACER = OpenTelemetry.tracer_provider.tracer('lambda-consumer')

def extract_context_from_sqs_record(record)
  carrier = {}
  # Support both Lambda ESM ("messageAttributes") and SDK ReceiveMessage ("MessageAttributes")
  attrs = record['messageAttributes'] || record['MessageAttributes'] || {}
  attrs.each do |key, attr|
    # ESM uses lowercase "stringValue"; SDK uses "StringValue"
    value = attr['stringValue'] || attr['StringValue']
    carrier[key] = value if value
  end
  OpenTelemetry.propagation.extract(carrier)
end

def lambda_handler(event:, context:)
  (event['Records'] || []).each do |record|
    parent_ctx = extract_context_from_sqs_record(record)

    OpenTelemetry::Context.with_current(parent_ctx) do
      TRACER.in_span('process_message', kind: OpenTelemetry::Trace::SpanKind::CONSUMER) do |span|
        # Your business logic here
      end
    end
  end
ensure
  OpenTelemetry.tracer_provider.force_flush
end

Producer Side — Injecting Trace Context

The upstream service must inject trace context into SQS MessageAttributes. See the AWS SQS integration — Trace Context Propagation for producer-side code examples.

Batch Processing

Each SQS record in a batch may carry a different trace context (from different producer requests). Always extract and create spans per-record, not per-invocation. Enable ReportBatchItemFailures on your ESM for partial batch failure reporting.

Full Example

See the Python SQS → Lambda trace propagation example for a complete working implementation with local testing.

Lambda Log-Trace Correlation

By default, Lambda logs in CloudWatch have no link to the traces generated by the ADOT layer. To correlate logs with traces in Last9, inject the current trace_id and span_id into your structured log output before forwarding logs.

Why Logs and Traces Disconnect

ADOT automatically creates spans for your Lambda invocations. Those spans carry a trace_id. Your application logs — written with print, logging, or a structured logger — don’t know about that trace_id unless you explicitly inject it.

Without injection, you see traces in Last9’s Traces view and logs separately, with no way to jump from a log line to the trace that produced it.

Injecting Trace Context into Logs

import json
import logging
from opentelemetry import trace

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def get_trace_context():
    span = trace.get_current_span()
    ctx = span.get_span_context()
    if ctx.is_valid:
        return {
            "trace_id": format(ctx.trace_id, "032x"),
            "span_id": format(ctx.span_id, "016x"),
        }
    return {}

def handler(event, context):
    log_entry = {
        "level": "INFO",
        "message": "Processing event",
        **get_trace_context(),  # injects trace_id and span_id
    }
    print(json.dumps(log_entry))
    # ... business logic

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging import correlation_paths

logger = Logger(service="payment-lambda")

@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
def handler(event, context):
    logger.info("Processing request")
    # Powertools automatically includes xray_trace_id
    # Add OTel trace_id manually via get_trace_context() if needed

const { trace } = require("@opentelemetry/api");

function getTraceContext() {
  const span = trace.getActiveSpan();
  if (!span) return {};
  const ctx = span.spanContext();
  return {
    trace_id: ctx.traceId,
    span_id: ctx.spanId,
  };
}

exports.handler = async (event) => {
  console.log(JSON.stringify({
    level: "INFO",
    message: "Processing event",
    ...getTraceContext(),
  }));
  // ... business logic
};

const pino = require("pino");
const { trace } = require("@opentelemetry/api");

const logger = pino({
  mixin() {
    const span = trace.getActiveSpan();
    if (!span) return {};
    const ctx = span.spanContext();
    return { trace_id: ctx.traceId, span_id: ctx.spanId };
  },
});

exports.handler = async (event) => {
  logger.info("Processing event");
};

require 'json'
require 'opentelemetry'

def trace_context
  span = OpenTelemetry::Trace.current_span
  ctx = span.context
  return {} unless ctx.valid?
  {
    trace_id: ctx.hex_trace_id,
    span_id: ctx.hex_span_id,
  }
end

def lambda_handler(event:, context:)
  puts JSON.generate({ level: 'INFO', message: 'Processing event', **trace_context })
  # ... business logic
ensure
  OpenTelemetry.tracer_provider.force_flush
end

Forwarding Lambda Logs to Last9

After injecting trace context into logs, forward your CloudWatch logs to Last9 so they appear alongside traces:

Option 1 — CloudWatch Subscription Filter (push-based):

1. Create a Lambda function that forwards logs to Last9’s OTLP endpoint
2. Add a CloudWatch Logs subscription filter on your Lambda’s log group to trigger it

Option 2 — OTel Collector sidecar (ECS/fargate only):

If running Lambda on ECS with a sidecar collector, configure the filelog receiver to tail /proc/1/fd/1.

Option 3 — Direct OTLP logs from Lambda:

Install the OTel Logs SDK and emit logs directly to Last9:

from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor

provider = LoggerProvider()
provider.add_log_record_processor(
    BatchLogRecordProcessor(OTLPLogExporter())
)
set_logger_provider(provider)

Searching Correlated Logs in Last9

Once logs carry trace_id, filter in Last9 Log Management:

trace_id = "4bf92f3577b34da6a3ce929d0e0e4736"

Or from a trace in Last9’s Traces view, click View Logs to jump directly to log lines matching that trace’s trace_id.

Best Practices

• Service Naming: Use descriptive, consistent service names across your Lambda functions
• Environment Segregation: Use different service names or resource attributes per environment
• Sampling Strategy: Use appropriate sampling rates for production to control costs
• Monitoring: Set up CloudWatch alarms for Lambda errors and duration
• Resource Attribution: Include meaningful metadata like version, team, and environment
• Testing: Test instrumentation in development before deploying to production

Troubleshooting

Please get in touch with us on Discord or Email if you have any questions.