AWS Chalice

Use OpenTelemetry to instrument your AWS Chalice Lambda functions and send telemetry data to Last9. Chalice is AWS’s Python serverless framework that manages Lambda deployment, API Gateway, and IAM policies through a single config file. This integration uses the AWS Distro for OpenTelemetry (ADOT) Layer for automatic instrumentation with no code changes required.

Prerequisites

Before setting up AWS Chalice monitoring, ensure you have:

• AWS Account: With access to Lambda service
• Python 3.8+: With Chalice installed (pip install chalice)
• Chalice Project: An existing or new Chalice application
• Last9 Account: With OpenTelemetry integration credentials

Supported Runtimes

Chalice deploys Python Lambda functions. ADOT Python layers support:

• Python: 3.8, 3.9, 3.10, 3.11, 3.12

1. Configure .chalice/config.json

   Add the ADOT Lambda layer and environment variables to your Chalice configuration. The layer provides auto-instrumentation — no application code changes needed.

   • Dev Stage
   • Production Stage

   {
     "version": "2.0",
     "app_name": "your-app-name",
     "stages": {
       "dev": {
         "api_gateway_stage": "dev",
         "lambda_timeout": 30,
         "lambda_memory_size": 256,
         "xray": true,
         "layers": [
           "arn:aws:lambda:ap-southeast-1:901920570463:layer:aws-otel-python-amd64-ver-1-25-0:1"
         ],
         "environment_variables": {
           "AWS_LAMBDA_EXEC_WRAPPER": "/opt/otel-instrument",
           "OPENTELEMETRY_COLLECTOR_CONFIG_FILE": "/var/task/.chalice/collector-config.yaml",
           "OTEL_SERVICE_NAME": "your-service-name",
           "OTEL_PROPAGATORS": "tracecontext,xray",
           "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
           "OTEL_TRACES_EXPORTER": "otlp",
           "OTEL_TRACES_SAMPLER": "always_on",
           "OTEL_RESOURCE_ATTRIBUTES": "deployment.environment=dev"
         }
       }
     }
   }

   Important Configuration Notes:

   • Replace your-app-name and your-service-name with descriptive names
   • Replace the layer ARN with the correct one for your AWS region (see ADOT Lambda docs)
   • "xray": true is optional — it enables X-Ray alongside ADOT for co-existence

2. Create Collector Configuration

   Create collector-config.yaml in your project root, then copy it to .chalice/ so it gets packaged with your Lambda:

   receivers:
     otlp:
       protocols:
         grpc:
           endpoint: localhost:4317
   
   exporters:
     otlp:
       endpoint: $last9_otlp_endpoint
       headers:
         authorization: $last9_otlp_auth_header
       tls:
         insecure: false
   
   service:
     pipelines:
       traces:
         receivers: [otlp]
         exporters: [otlp]
       metrics:
         receivers: [otlp]
         exporters: [otlp]

   Copy to the .chalice/ directory:

   cp collector-config.yaml .chalice/collector-config.yaml

3. Deploy

   • Dev
   • Production

   chalice deploy --stage dev

   Chalice packages your app code, .chalice/ directory (including collector-config.yaml), and requirements into a Lambda deployment.

4. Test and Verify

   • Chalice CLI
   • AWS CLI

   # Get your API URL
   chalice url --stage dev
   
   # Test
   curl $(chalice url --stage dev)/

Understanding the Setup

How Chalice + ADOT Works

1. Chalice deploys your Python function to Lambda with the ADOT layer attached
2. AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument wraps the Python process at startup
3. The ADOT layer injects OpenTelemetry auto-instrumentation before your Chalice app loads
4. All HTTP handlers, scheduled tasks, and AWS SDK calls are traced automatically
5. The in-Lambda ADOT Collector sends traces to Last9 via the collector-config.yaml

Environment Variables Explained

What Gets Traced

The ADOT layer automatically traces:

• Chalice Route Handlers: @app.route() decorated functions
• Scheduled Tasks: @app.schedule() decorated functions
• AWS SDK Calls: DynamoDB, S3, SQS, SNS, etc.
• HTTP Requests: Outbound API calls via urllib, requests, boto3
• Database Calls: RDS, DynamoDB operations

X-Ray Co-existence

If your Chalice app already has "xray": true in config.json, you can keep it alongside ADOT:

• X-Ray traces continue going to the AWS X-Ray service (existing dashboards keep working)
• ADOT/OTLP traces go to Last9

Setting OTEL_PROPAGATORS=tracecontext,xray ensures the ADOT layer reads and writes both W3C traceparent and AWS X-Amzn-Trace-Id headers. Trace context propagates correctly regardless of which format upstream services use.

To use ADOT only (no X-Ray), remove "xray": true from config.json and set propagators to tracecontext only.

Advanced Configuration

Custom Spans via Chalice Middleware

Auto-instrumentation captures handlers and SDK calls. For custom business logic spans, use the OTel API with Chalice middleware:

from chalice import Chalice
from opentelemetry import trace

app = Chalice(app_name="your-app")
tracer = trace.get_tracer(__name__)

@app.middleware("all")
def add_custom_attributes(event, get_response):
    span = trace.get_current_span()
    if span.is_recording():
        span.set_attribute("app.framework", "chalice")
    return get_response(event)

@app.route("/process/{order_id}")
def process_order(order_id):
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", order_id)
        # your business logic
    return {"status": "processed"}

Only opentelemetry-api is needed in requirements.txt. The ADOT layer provides the full SDK.

Per-Function Configuration

Chalice supports per-function overrides for layer, memory, and timeout:

{
  "stages": {
    "dev": {
      "lambda_functions": {
        "periodic_check": {
          "lambda_timeout": 60,
          "lambda_memory_size": 128
        }
      }
    }
  }
}

Sampling Configuration

Control trace sampling to manage costs:

# Development: Sample all traces
OTEL_TRACES_SAMPLER=always_on

# Production: Sample 10% of traces
OTEL_TRACES_SAMPLER=traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

Troubleshooting

No Traces Appearing

Check CloudWatch Logs:

aws logs tail /aws/lambda/your-app-name-dev --follow

Common Issues:

• Verify collector-config.yaml is at /var/task/.chalice/collector-config.yaml inside the Lambda
• Confirm ADOT layer ARN is correct for your region
• Check that Last9 credentials in collector config are valid
• Ensure AWS_LAMBDA_EXEC_WRAPPER is set to /opt/otel-instrument

Module Errors

Do NOT add opentelemetry-sdk or opentelemetry-instrumentation-* to requirements.txt. The ADOT layer provides them. Only opentelemetry-api is needed (for custom spans).

Cold Start Latency

ADOT adds ~500ms-1s to cold starts. Mitigate with:

• Provisioned concurrency for latency-sensitive functions
• Adequate memory allocation (256MB+ recommended)

Error Messages

Best Practices

• Service Naming: Use descriptive, consistent names across Chalice stages
• Sampling: Start with always_on in dev, use traceidratio in production
• X-Ray Transition: Keep X-Ray enabled initially, disable once Last9 dashboards are confirmed
• Memory: Allocate 256MB+ to account for ADOT layer overhead
• Collector Config: Always copy collector-config.yaml to .chalice/ before deploying

Need Help?

If you encounter any issues or have questions:

• Join our Discord community for real-time support
• Contact our support team at support@last9.io