AWS Lambda

Use OpenTelemetry to instrument your AWS Lambda functions and send telemetry data to Last9. This integration uses the AWS Distro for OpenTelemetry (ADOT) Layer for automatic instrumentation with no code changes required.

Prerequisites

Before setting up AWS Lambda monitoring, ensure you have:

• AWS Account: With access to Lambda service
• Lambda Functions: Deployed functions to instrument
• AWS Console Access: Or AWS CLI configured
• Last9 Account: With OpenTelemetry integration credentials

Supported Runtimes

ADOT layers support the following Lambda runtimes:

• Node.js: 14.x, 16.x, 18.x, 20.x
• Python: 3.8, 3.9, 3.10, 3.11, 3.12
• Java: 8, 11, 17, 21
• .NET: Core 3.1, 6.0, 8.0

1. Find the ADOT Layer ARN

   Get the latest ADOT layer ARN for your AWS region and runtime from the AWS ADOT Lambda documentation.

   • Popular Regions
   • Asia Pacific
   • Europe

   US East 1 (N. Virginia) - us-east-1:

   # Node.js
   arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4
   
   # Python
   arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python-amd64-ver-1-20-0:3
   
   # Java
   arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-32-0:3

   US West 2 (Oregon) - us-west-2:

   # Node.js
   arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4
   
   # Python
   arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-python-amd64-ver-1-20-0:3
   
   # Java
   arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-32-0:3

2. Add the ADOT Layer to Your Lambda Function

   • AWS Console
   • AWS CLI
   • Terraform

   Using the AWS Lambda Console:

   1. Open your Lambda function in the AWS Console
   2. Scroll down to the Layers section
   3. Click Add a layer
   4. Select Specify an ARN
   5. Paste the appropriate ADOT layer ARN for your region and runtime
   6. Click Add

3. Configure Environment Variables

   Add the following environment variables to your Lambda function configuration:

   • AWS Console
   • AWS CLI
   • Terraform
   1. Go to Configuration → Environment variables
   2. Click Edit and add these variables:

   AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument
   OTEL_SERVICE_NAME=<your-service-name>
   OTEL_EXPORTER_OTLP_ENDPOINT=$last9_otlp_endpoint
   OTEL_EXPORTER_OTLP_HEADERS=authorization=$last9_otlp_auth_header
   OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
   OTEL_TRACES_EXPORTER=otlp
   OTEL_TRACES_SAMPLER=always_on
   OTEL_PROPAGATORS=tracecontext,baggage
   OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production

   Replace <your-service-name> with a descriptive service name (e.g., payment-lambda, user-service).

   Important Configuration Notes:

   • Replace <your-service-name> with a descriptive name (e.g., payment-lambda, user-service)
   • Use lowercase authorization in the headers configuration
   • Update the deployment.environment to match your environment (production, staging, development)
   • Replace placeholder values with your actual Last9 credentials

4. Test and Verify Installation

   Test your Lambda function to ensure instrumentation is working:

   • AWS Console Test
   • AWS CLI Invoke
   • API Gateway (if applicable)
   1. Go to your Lambda function in the AWS Console
   2. Click Test
   3. Create a test event or use an existing one
   4. Click Test to invoke the function
   5. Check the execution logs for OpenTelemetry initialization messages

Understanding the Setup

AWS Distro for OpenTelemetry (ADOT)

ADOT provides:

• Zero-Code Instrumentation: No application code changes required
• AWS Service Integration: Built-in support for AWS services and APIs
• Lambda Optimizations: Optimized for serverless environments
• Multiple Language Support: Comprehensive runtime coverage

Environment Variables Explained

What Gets Traced

The ADOT layer automatically traces:

• Lambda Handler: Function entry and exit
• AWS SDK Calls: DynamoDB, S3, SQS, etc.
• HTTP Requests: Outbound API calls
• Database Calls: RDS, DynamoDB operations
• Message Queue Operations: SQS send/receive

Advanced Configuration

Sampling Configuration

Control trace sampling to manage costs:

# Production: Sample 10% of traces
OTEL_TRACES_SAMPLER=traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

# Development: Sample all traces
OTEL_TRACES_SAMPLER=always_on

Custom Resource Attributes

Add additional metadata to traces:

OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,team=payments,version=2.1.0,region=us-east-1

Error Handling Configuration

Configure error reporting:

OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=true
OTEL_INSTRUMENTATION_AWS_SDK_SUPPRESSORS_MESSAGING_RECEIVE_TELEMETRY_ENABLED=true

Verification

1. Check Lambda Execution Logs

   Look for ADOT initialization messages in CloudWatch Logs:

   aws logs filter-log-events \
     --log-group-name "/aws/lambda/your-function-name" \
     --filter-pattern "OpenTelemetry"

2. Verify Layer Attachment

   Confirm the ADOT layer is properly attached:

   aws lambda get-function-configuration \
     --function-name your-function-name \
     --query 'Layers[?starts_with(Arn, `arn:aws:lambda`)].Arn'

3. View Traces in Last9

   1. Log into your Last9 dashboard
   2. Navigate to the Traces section
   3. Filter by your service name
   4. Traces should appear within 1-2 minutes of invocation

4. Test Trace Propagation

   If your Lambda calls other services, verify trace propagation by checking connected spans in the trace view.

Troubleshooting

No Traces Appearing

Enable Debug Logging:

OTEL_LOG_LEVEL=debug

Check Common Issues:

• Verify ADOT layer is attached to your function
• Confirm environment variables are set correctly (no extra quotes)
• Check CloudWatch Logs for error messages
• Ensure Last9 credentials are valid

Lambda Cold Start Issues

Optimize Cold Starts:

• Use provisioned concurrency for critical functions
• Consider using reserved concurrency to limit concurrent executions
• Monitor cold start metrics in CloudWatch

Memory and Performance Impact

Monitor Resource Usage:

• Check function memory usage in CloudWatch metrics
• Adjust memory allocation if needed
• Monitor execution duration for performance impact

Error Messages

Common Error Resolutions:

SQS-Triggered Lambda — Trace Propagation

When Lambda is triggered by SQS via Event Source Mapping (ESM), traces from the upstream producer are not automatically linked. You must extract W3C traceparent from SQS MessageAttributes and use it as the parent context for your Lambda spans.

How It Works

1. The upstream service (producer) injects traceparent into SQS MessageAttributes when sending a message
2. SQS delivers the message to Lambda via ESM
3. Lambda extracts the trace context from the record’s messageAttributes and creates child spans

ESM Attribute Format

Your extraction code must handle both formats.

Extract Trace Context in Lambda

from opentelemetry.propagate import extract
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def extract_context_from_sqs_record(record):
    carrier = {}
    # Support both Lambda ESM ("messageAttributes") and SDK ReceiveMessage ("MessageAttributes")
    msg_attrs = record.get("messageAttributes") or record.get("MessageAttributes") or {}
    for key, attr in msg_attrs.items():
        # ESM uses lowercase "stringValue"; SDK uses "StringValue"
        value = (
            attr.get("stringValue")
            if "stringValue" in attr
            else attr.get("StringValue")
        )
        if value is not None:
            carrier[key] = value
    return extract(carrier)

def handler(event, context):
    for record in event.get("Records", []):
        ctx = extract_context_from_sqs_record(record)

        with tracer.start_as_current_span(
            "process_message",
            context=ctx,  # Links to producer's trace
            kind=trace.SpanKind.CONSUMER,
        ):
            # Your business logic here
            pass

const { propagation, trace, SpanKind } = require("@opentelemetry/api");

const tracer = trace.getTracer("lambda-consumer");

function extractContextFromSqsRecord(record) {
  const carrier = {};
  const attrs = record.messageAttributes || {};
  for (const [key, attr] of Object.entries(attrs)) {
    const value = attr.stringValue || attr.StringValue;
    if (value) carrier[key] = value;
  }
  return propagation.extract(trace.context.active(), carrier);
}

exports.handler = async (event) => {
  for (const record of event.Records) {
    const ctx = extractContextFromSqsRecord(record);

    await tracer.startActiveSpan(
      "process_message",
      { kind: SpanKind.CONSUMER },
      ctx,
      async (span) => {
        // Your business logic here
        span.end();
      }
    );
  }
};

Producer Side — Injecting Trace Context

The upstream service must inject trace context into SQS MessageAttributes. See the AWS SQS integration — Trace Context Propagation for producer-side code examples.

Batch Processing

Each SQS record in a batch may carry a different trace context (from different producer requests). Always extract and create spans per-record, not per-invocation. Enable ReportBatchItemFailures on your ESM for partial batch failure reporting.

Full Example

See the Python SQS → Lambda trace propagation example for a complete working implementation with local testing.

Best Practices

• Service Naming: Use descriptive, consistent service names across your Lambda functions
• Environment Segregation: Use different service names or resource attributes per environment
• Sampling Strategy: Use appropriate sampling rates for production to control costs
• Monitoring: Set up CloudWatch alarms for Lambda errors and duration
• Resource Attribution: Include meaningful metadata like version, team, and environment
• Testing: Test instrumentation in development before deploying to production

Need Help?

If you encounter any issues or have questions:

• Join our Discord community for real-time support
• Contact our support team at support@last9.io