AWS Lambda

Use OpenTelemetry to instrument your AWS Lambda functions and send telemetry data to Last9. This integration uses the AWS Distro for OpenTelemetry (ADOT) Layer for automatic instrumentation with no code changes required.

Prerequisites

Before setting up AWS Lambda monitoring, ensure you have:

AWS Account: With access to Lambda service
Lambda Functions: Deployed functions to instrument
AWS Console Access: Or AWS CLI configured
Last9 Account: With OpenTelemetry integration credentials

Supported Runtimes

ADOT layers support the following Lambda runtimes:

Node.js: 14.x, 16.x, 18.x, 20.x
Python: 3.8, 3.9, 3.10, 3.11, 3.12
Java: 8, 11, 17, 21
.NET: Core 3.1, 6.0, 8.0

Find the ADOT Layer ARN

Get the latest ADOT layer ARN for your AWS region and runtime from the AWS ADOT Lambda documentation.
US East 1 (N. Virginia) - us-east-1:
# Node.js arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4 # Python arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python-amd64-ver-1-20-0:3 # Java arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-32-0:3
US West 2 (Oregon) - us-west-2:
# Node.js arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4 # Python arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-python-amd64-ver-1-20-0:3 # Java arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-32-0:3
Asia Pacific (Mumbai) - ap-south-1:
# Node.js arn:aws:lambda:ap-south-1:615299751070:layer:AWSOpenTelemetryDistroJs:9 # Python arn:aws:lambda:ap-south-1:615299751070:layer:AWSOpenTelemetryDistro:5 # Java arn:aws:lambda:ap-south-1:615299751070:layer:AWSOpenTelemetryDistroJava:7
Asia Pacific (Singapore) - ap-southeast-1:
# Node.js arn:aws:lambda:ap-southeast-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4 # Python arn:aws:lambda:ap-southeast-1:901920570463:layer:aws-otel-python-amd64-ver-1-20-0:3 # Java arn:aws:lambda:ap-southeast-1:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-32-0:3
Europe (Ireland) - eu-west-1:
# Node.js arn:aws:lambda:eu-west-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4 # Python arn:aws:lambda:eu-west-1:901920570463:layer:aws-otel-python-amd64-ver-1-20-0:3 # Java arn:aws:lambda:eu-west-1:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-32-0:3

Add the ADOT Layer to Your Lambda Function

Using the AWS Lambda Console:

Open your Lambda function in the AWS Console
Scroll down to the Layers section
Click Add a layer
Select Specify an ARN
Paste the appropriate ADOT layer ARN for your region and runtime
Click Add

Using the AWS CLI:

aws lambda update-function-configuration \
  --function-name your-function-name \
  --layers arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4

Replace with your function name and appropriate layer ARN.

Using Terraform:

resource "aws_lambda_function" "example" {
  function_name = "your-function-name"
  # ... other configuration

  layers = [
    "arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-18-1:4"
  ]
}

Configure Environment Variables

Add the following environment variables to your Lambda function configuration:

Go to Configuration → Environment variables
Click Edit and add these variables:

AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument
OTEL_SERVICE_NAME=<your-service-name>
OTEL_EXPORTER_OTLP_ENDPOINT=$last9_otlp_endpoint
OTEL_EXPORTER_OTLP_HEADERS=authorization=$last9_otlp_auth_header
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_TRACES_EXPORTER=otlp
OTEL_TRACES_SAMPLER=always_on
OTEL_PROPAGATORS=tracecontext,baggage
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production

Replace <your-service-name> with a descriptive service name (e.g., payment-lambda, user-service).

aws lambda update-function-configuration \
  --function-name your-function-name \
  --environment Variables='{
    "AWS_LAMBDA_EXEC_WRAPPER": "/opt/otel-instrument",
    "OTEL_SERVICE_NAME": "your-service-name",
    "OTEL_EXPORTER_OTLP_ENDPOINT": "$last9_otlp_endpoint",
    "OTEL_EXPORTER_OTLP_HEADERS": "authorization=$last9_otlp_auth_header",
    "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
    "OTEL_TRACES_EXPORTER": "otlp",
    "OTEL_TRACES_SAMPLER": "always_on",
    "OTEL_PROPAGATORS": "tracecontext,baggage",
    "OTEL_RESOURCE_ATTRIBUTES": "deployment.environment=production"
  }'

resource "aws_lambda_function" "example" {
  function_name = "your-function-name"
  # ... other configuration

  environment {
    variables = {
      AWS_LAMBDA_EXEC_WRAPPER          = "/opt/otel-instrument"
      OTEL_SERVICE_NAME                = "your-service-name"
      OTEL_EXPORTER_OTLP_ENDPOINT      = "$last9_otlp_endpoint"
      OTEL_EXPORTER_OTLP_HEADERS       = "authorization=$last9_otlp_auth_header"
      OTEL_EXPORTER_OTLP_PROTOCOL      = "http/protobuf"
      OTEL_TRACES_EXPORTER             = "otlp"
      OTEL_TRACES_SAMPLER              = "always_on"
      OTEL_PROPAGATORS                 = "tracecontext,baggage"
      OTEL_RESOURCE_ATTRIBUTES         = "deployment.environment=production"
    }
  }
}

Important Configuration Notes:

Replace <your-service-name> with a descriptive name (e.g., payment-lambda, user-service)
Use lowercase authorization in the headers configuration
Update the deployment.environment to match your environment (production, staging, development)
Replace placeholder values with your actual Last9 credentials

Test and Verify Installation

Test your Lambda function to ensure instrumentation is working:
1. Go to your Lambda function in the AWS Console
2. Click Test
3. Create a test event or use an existing one
4. Click Test to invoke the function
5. Check the execution logs for OpenTelemetry initialization messages
aws lambda invoke \ --function-name your-function-name \ --payload '{"test": "event"}' \ response.json # View the response cat response.json
If your Lambda is behind API Gateway, test via HTTP request:
curl -X POST https://your-api-gateway-url/your-endpoint \ -H "Content-Type: application/json" \ -d '{"test": "data"}'

Understanding the Setup

AWS Distro for OpenTelemetry (ADOT)

ADOT provides:

Zero-Code Instrumentation: No application code changes required
AWS Service Integration: Built-in support for AWS services and APIs
Lambda Optimizations: Optimized for serverless environments
Multiple Language Support: Comprehensive runtime coverage

Environment Variables Explained

Variable	Purpose	Example
`AWS_LAMBDA_EXEC_WRAPPER`	Enables ADOT instrumentation	`/opt/otel-instrument`
`OTEL_SERVICE_NAME`	Service identifier in traces	`payment-service`
`OTEL_EXPORTER_OTLP_ENDPOINT`	Last9 traces endpoint	Last9 provided URL
`OTEL_EXPORTER_OTLP_HEADERS`	Authentication headers	Last9 auth token
`OTEL_TRACES_SAMPLER`	Sampling strategy	`always_on` or `traceidratio`
`OTEL_RESOURCE_ATTRIBUTES`	Additional metadata	`deployment.environment=prod`

What Gets Traced

The ADOT layer automatically traces:

Lambda Handler: Function entry and exit
AWS SDK Calls: DynamoDB, S3, SQS, etc.
HTTP Requests: Outbound API calls
Database Calls: RDS, DynamoDB operations
Message Queue Operations: SQS send/receive

Advanced Configuration

Sampling Configuration

Control trace sampling to manage costs:

# Production: Sample 10% of traces
OTEL_TRACES_SAMPLER=traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

# Development: Sample all traces
OTEL_TRACES_SAMPLER=always_on

Custom Resource Attributes

Add additional metadata to traces:

OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,team=payments,version=2.1.0,region=us-east-1

Error Handling Configuration

Configure error reporting:

OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=true
OTEL_INSTRUMENTATION_AWS_SDK_SUPPRESSORS_MESSAGING_RECEIVE_TELEMETRY_ENABLED=true

Verification

Check Lambda Execution Logs

Look for ADOT initialization messages in CloudWatch Logs:

aws logs filter-log-events \
  --log-group-name "/aws/lambda/your-function-name" \
  --filter-pattern "OpenTelemetry"

Verify Layer Attachment

Confirm the ADOT layer is properly attached:

aws lambda get-function-configuration \
  --function-name your-function-name \
  --query 'Layers[?starts_with(Arn, `arn:aws:lambda`)].Arn'

View Traces in Last9
1. Log into your Last9 dashboard
2. Navigate to the Traces section
3. Filter by your service name
4. Traces should appear within 1-2 minutes of invocation
Test Trace Propagation

If your Lambda calls other services, verify trace propagation by checking connected spans in the trace view.

Troubleshooting

No Traces Appearing

Enable Debug Logging:

OTEL_LOG_LEVEL=debug

Check Common Issues:

Verify ADOT layer is attached to your function
Confirm environment variables are set correctly (no extra quotes)
Check CloudWatch Logs for error messages
Ensure Last9 credentials are valid

Lambda Cold Start Issues

Optimize Cold Starts:

Use provisioned concurrency for critical functions
Consider using reserved concurrency to limit concurrent executions
Monitor cold start metrics in CloudWatch

Memory and Performance Impact

Monitor Resource Usage:

Check function memory usage in CloudWatch metrics
Adjust memory allocation if needed
Monitor execution duration for performance impact

Error Messages

Common Error Resolutions:

Error	Solution
”Recording is off”	Set `OTEL_TRACES_SAMPLER=always_on`
”Layer not found”	Use correct layer ARN for your region
”Permission denied”	Check Lambda execution role permissions
”Timeout”	Increase function timeout or optimize code

Best Practices

Service Naming: Use descriptive, consistent service names across your Lambda functions
Environment Segregation: Use different service names or resource attributes per environment
Sampling Strategy: Use appropriate sampling rates for production to control costs
Monitoring: Set up CloudWatch alarms for Lambda errors and duration
Resource Attribution: Include meaningful metadata like version, team, and environment
Testing: Test instrumentation in development before deploying to production

Need Help?

If you encounter any issues or have questions:

Join our Discord community for real-time support
Contact our support team at support@last9.io