AWS CloudWatch Logs

Ingest AWS CloudWatch logs using OpenTelemetry and send them to Last9 for centralized log management and analysis. This integration automatically discovers and ingests logs from AWS services like ELB, ALB, RDS, Lambda, and custom applications.

CloudWatch logs integration is ideal for collecting logs from multiple AWS services in one place, enabling comprehensive log analysis and correlation across your AWS infrastructure.

Prerequisites

Before setting up AWS CloudWatch logs ingestion, ensure you have:

• AWS Account: With access to CloudWatch Logs service
• Log Groups: CloudWatch log groups containing the logs you want to ingest
• EC2 Instance: Or monitoring server where you can install OpenTelemetry Collector
• IAM Permissions: To read CloudWatch logs
• Last9 Account: With OpenTelemetry log ingestion enabled

1. Configure AWS IAM Permissions

   Create an IAM policy that allows the collector to read CloudWatch logs:

   • IAM Policy JSON
   • AWS CLI
   • Terraform

   Create a policy with the following permissions:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "CloudWatchLogsRead",
         "Effect": "Allow",
         "Action": [
           "logs:DescribeLogGroups",
           "logs:DescribeLogStreams",
           "logs:FilterLogEvents",
           "logs:GetLogEvents"
         ],
         "Resource": "*"
       }
     ]
   }

2. Attach Policy to EC2 Instance Role

   Attach the policy to your EC2 instance role or create IAM credentials:

   • EC2 Instance Role
   • IAM User (Alternative)

   # Attach policy to existing EC2 instance role
   aws iam attach-role-policy \
     --role-name EC2-CloudWatch-Role \
     --policy-arn arn:aws:iam::YOUR-ACCOUNT-ID:policy/CloudWatch-Logs-Read-Policy

3. Install OpenTelemetry Collector

   Install the OpenTelemetry Collector with AWS CloudWatch receiver support:

   • DEB Package
   • RPM Package

   For Debian/Ubuntu systems:

   wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.118.0/otelcol-contrib_0.118.0_linux_amd64.deb
   sudo dpkg -i otelcol-contrib_0.118.0_linux_amd64.deb

4. Configure AWS Credentials (if not using EC2 role)

   If not using an EC2 instance role, configure AWS credentials:

   • AWS Credentials File
   • Environment Variables

   Create ~/.aws/credentials:

   [default]
   aws_access_key_id = YOUR_ACCESS_KEY_ID
   aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
   region = us-east-1

5. Create OpenTelemetry Collector Configuration

   Create the collector configuration file:

   sudo mkdir -p /etc/otelcol-contrib
   sudo nano /etc/otelcol-contrib/config.yaml

   Add the following configuration to collect CloudWatch logs:

   receivers:
     awscloudwatch:
       region: us-east-1 # Change to your AWS region
       logs:
         poll_interval: 2m # How often to check for new logs
         groups:
           # Auto-discover log groups by prefix
           autodiscover:
             limit: 100 # Maximum number of log groups to discover
             prefixes:
               - "/aws/elb" # Elastic Load Balancer logs
               - "/aws/alb" # Application Load Balancer logs
               - "/aws/lambda" # Lambda function logs
               - "/aws/rds" # RDS database logs
               - "/aws/ecs" # ECS container logs
               - "/aws/apigateway" # API Gateway logs
               - "/aws/cloudtrail" # CloudTrail audit logs
               - "/aws/s3" # S3 access logs
               - "/aws/codebuild" # CodeBuild logs
   
           # Specify exact log group names (optional)
           named:
             - "/aws/elb/production-alb"
             - "/aws/lambda/user-authentication"
             # Add your specific log groups here
   
   processors:
     batch:
       timeout: 30s
       send_batch_size: 50000
       send_batch_max_size: 100000
   
     # Add timestamps for logs that don't have them
     transform/add_timestamp:
       error_mode: ignore
       log_statements:
         - context: log
           conditions:
             - time_unix_nano == 0
           statements:
             - set(observed_time, Now())
             - set(time_unix_nano, observed_time_unix_nano)
   
     # Add service identification and source
     transform/logs:
       error_mode: ignore
       log_statements:
         - context: log
           statements:
             - set(resource.attributes["service.name"], "aws-cloudwatch-logs")
             - set(resource.attributes["deployment.environment"], "production")
             - set(attributes["source"], "cloudwatch")
             - set(attributes["cloud.provider"], "aws")
             - set(attributes["cloud.region"], "us-east-1")
   
     # Detect EC2 metadata
     resourcedetection/aws:
       detectors: ["env", "ec2"]
       timeout: 2s
       override: false
   
   exporters:
     otlp/last9:
       endpoint: "$last9_otlp_endpoint"
       headers:
         "Authorization": "$last9_otlp_auth_header"
     debug:
       verbosity: detailed
   
   service:
     pipelines:
       logs:
         receivers: [awscloudwatch]
         processors:
           [
             batch,
             resourcedetection/aws,
             transform/add_timestamp,
             transform/logs,
           ]
         exporters: [otlp/last9]

6. Advanced Log Group Configuration

   Configure specific log groups with custom settings:

   • Service-Specific Groups
   • Multi-Region Setup
   • Log Filtering

   Monitor specific services with targeted configuration:

   receivers:
     awscloudwatch/elb:
       region: us-east-1
       logs:
         poll_interval: 1m # More frequent polling for critical services
         groups:
           named:
             - "/aws/elb/production-frontend"
             - "/aws/elb/production-api"
   
     awscloudwatch/lambda:
       region: us-east-1
       logs:
         poll_interval: 2m
         groups:
           autodiscover:
             limit: 50
             prefixes:
               - "/aws/lambda"
   
   service:
     pipelines:
       logs/elb:
         receivers: [awscloudwatch/elb]
         processors: [batch, transform/logs]
         exporters: [otlp/last9]
       logs/lambda:
         receivers: [awscloudwatch/lambda]
         processors: [batch, transform/logs]
         exporters: [otlp/last9]

7. Create Systemd Service Configuration

   Create a systemd service file:

   sudo nano /etc/systemd/system/otelcol-contrib.service

   Add the service configuration:

   [Unit]
   Description=OpenTelemetry Collector for AWS CloudWatch Logs
   After=network.target
   
   [Service]
   ExecStart=/usr/bin/otelcol-contrib --config /etc/otelcol-contrib/config.yaml
   Restart=always
   User=root
   Group=root
   Environment=AWS_REGION=us-east-1
   
   [Install]
   WantedBy=multi-user.target

8. Start and Enable the Service

   Start the OpenTelemetry Collector service:

   sudo systemctl daemon-reload
   sudo systemctl enable otelcol-contrib
   sudo systemctl start otelcol-contrib

Understanding CloudWatch Log Groups

CloudWatch automatically receives logs from various AWS services:

AWS Service Logs

• Load Balancers: ELB, ALB, NLB access logs and error logs
• Lambda Functions: Function execution logs, error traces, and custom logs
• RDS Databases: Error logs, slow query logs, general logs
• API Gateway: Request/response logs, execution logs
• VPC Flow Logs: Network traffic logs for security analysis

Application Logs

• ECS Containers: Application logs from containerized services
• EC2 Applications: Custom application logs via CloudWatch agent
• Elastic Beanstalk: Application and web server logs
• CodeBuild: Build process logs and debugging information

Audit and Security Logs

• CloudTrail: API call audit logs for security monitoring
• GuardDuty: Security finding logs and threat detection
• Config: Configuration change logs for compliance
• WAF: Web application firewall logs

Advanced Configuration

Log Parsing and Enrichment

Add structured parsing for common AWS log formats:

processors:
  transform/parse_elb:
    log_statements:
      - context: log
        conditions:
          - attributes["aws.log_group"] == "/aws/elb/app"
        statements:
          # Parse ELB access log format
          - set(attributes["elb.type"], "application")
          - set(attributes["http.method"], "GET") # Extract from log body
          - set(attributes["http.status_code"], 200) # Parse from log

  transform/parse_lambda:
    log_statements:
      - context: log
        conditions:
          - attributes["aws.log_group"] matches "/aws/lambda/.*"
        statements:
          - set(attributes["faas.name"], "lambda-function")
          - set(attributes["faas.version"], "$LATEST")

Log Sampling and Volume Control

Control log volume for high-traffic services:

processors:
  # Sample logs to reduce volume
  probabilistic_sampler:
    sampling_percentage: 10 # Keep 10% of logs

  # Rate limiting for log ingestion
  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 128

  # Batch optimization for high volume
  batch:
    timeout: 10s
    send_batch_size: 10000
    send_batch_max_size: 50000

Custom Metadata Enhancement

Add business context to logs:

processors:
  transform/business_context:
    log_statements:
      - context: log
        statements:
          - set(attributes["environment"], "production")
          - set(attributes["team"], "platform")
          - set(attributes["cost_center"], "engineering")
          - set(attributes["log_level"], "info") # Default level
          # Add service mapping based on log group
          - set(attributes["service.name"], "web-server") where attributes["aws.log_group"] == "/aws/elb/web"
          - set(attributes["service.name"], "user-service") where attributes["aws.log_group"] == "/aws/lambda/user-auth"

Verification

1. Check Service Status

   Verify the OpenTelemetry Collector is running:

   sudo systemctl status otelcol-contrib

2. Monitor Service Logs

   Check for any configuration errors:

   sudo journalctl -u otelcol-contrib -f

   Look for successful log group discovery and log ingestion messages.

3. Verify AWS Connectivity

   Test CloudWatch access:

   # List available log groups
   aws logs describe-log-groups --region us-east-1 --max-items 10
   
   # Test log events retrieval
   aws logs filter-log-events \
     --log-group-name "/aws/elb/test" \
     --start-time $(date -d '1 hour ago' +%s)000 \
     --region us-east-1

4. Check Log Group Activity

   Verify there are recent log events in your log groups:

   # Check recent log streams
   aws logs describe-log-streams \
     --log-group-name "/aws/lambda/your-function" \
     --order-by LastEventTime \
     --descending \
     --max-items 5

5. Generate Test Logs

   Create test log entries for verification:

   # For Lambda functions - trigger function execution
   aws lambda invoke \
     --function-name your-test-function \
     --payload '{"test": "data"}' \
     response.json
   
   # For ALB - make HTTP requests to generate access logs
   curl https://your-alb-endpoint.com/health
   
   # For custom log groups - send test log events
   aws logs put-log-events \
     --log-group-name "/custom/application" \
     --log-stream-name "test-stream" \
     --log-events timestamp=$(date +%s)000,message="Test log message"

6. Verify Logs in Last9

   Log into your Last9 account and check that CloudWatch logs are being received in the Logs dashboard.

   Look for:

   • AWS service logs with proper source attribution
   • Log entries with CloudWatch metadata
   • Proper timestamp and service name attribution
   • Regional and service-specific log categorization

Common Log Group Patterns

AWS Service Log Groups

Custom Application Log Groups

Troubleshooting

No Logs Being Ingested

Check AWS Permissions:

# Test log group access
aws logs describe-log-groups --region us-east-1

# Test log event retrieval
aws logs filter-log-events \
  --log-group-name "/aws/lambda/test" \
  --start-time $(date -d '10 minutes ago' +%s)000

Verify Collector Configuration:

# Check collector logs
sudo journalctl -u otelcol-contrib -f

# Look for AWS CloudWatch receiver initialization
grep -i "cloudwatch" /var/log/otelcol-contrib.log

High Memory Usage

Optimize Batch Processing:

processors:
  batch:
    timeout: 10s # Shorter timeout
    send_batch_size: 5000 # Smaller batches
    send_batch_max_size: 10000

  memory_limiter:
    limit_mib: 256 # Lower memory limit

Rate Limiting Issues

Handle AWS API Limits:

receivers:
  awscloudwatch:
    logs:
      poll_interval: 5m # Reduce polling frequency
      groups:
        autodiscover:
          limit: 50 # Reduce log group discovery limit

Missing Recent Logs

Check Polling Configuration:

• Reduce poll_interval for more frequent log retrieval
• Verify log groups have recent activity
• Check CloudWatch log retention settings

Best Practices

Security

• IAM Roles: Use EC2 instance roles instead of hardcoded credentials
• Least Privilege: Grant only necessary CloudWatch permissions
• Network Security: Use VPC endpoints for CloudWatch API calls in private subnets
• Log Retention: Configure appropriate log retention policies in CloudWatch

Performance

• Polling Intervals: Balance between real-time needs and API costs
• Batch Processing: Optimize batch sizes for efficient transmission
• Memory Management: Set appropriate memory limits for the collector
• Regional Deployment: Deploy collectors in the same region as log groups

Cost Management

• Log Group Selection: Be selective about which log groups to monitor
• Retention Policies: Set appropriate retention periods for CloudWatch logs
• Sampling: Use sampling for high-volume, less critical log streams
• API Usage: Monitor CloudWatch API usage and costs

Monitoring Strategy

• Service Categorization: Group logs by service, environment, and team
• Alert Setup: Configure alerts for critical error patterns in logs
• Dashboard Creation: Build dashboards for different log sources and use cases
• Log Correlation: Use consistent service names and metadata across logs and metrics

Need Help?

If you encounter any issues or have questions:

• Join our Discord community for real-time support
• Contact our support team at support@last9.io