# AWS CloudWatch Logs

> Ingest and monitor AWS CloudWatch logs with OpenTelemetry for centralized log management and analysis across AWS services

Source: https://last9.io/docs/integrations/observability/aws-cloudwatch-logs/

Ingest AWS CloudWatch logs using OpenTelemetry and send them to Last9 for centralized log management and analysis. This integration automatically discovers and ingests logs from AWS services like ELB, ALB, RDS, Lambda, and custom applications.

CloudWatch logs integration is ideal for collecting logs from multiple AWS services in one place, enabling comprehensive log analysis and correlation across your AWS infrastructure.

> **Prefer a push-based setup?** If you'd rather not run a Collector — or you've hit duplicate log entries for Lambda invocations — you can stream CloudWatch logs to Last9 via Kinesis Data Firehose or a Lambda forwarder instead. See [AWS CloudWatch Logs (push)](/docs/integrations/observability/aws-cloudwatch-logs-push/).

## Prerequisites

Before setting up AWS CloudWatch logs ingestion, ensure you have:

- **AWS Account**: With access to CloudWatch Logs service
- **Log Groups**: CloudWatch log groups containing the logs you want to ingest
- **EC2 Instance**: Or monitoring server where you can install OpenTelemetry Collector
- **IAM Permissions**: To read CloudWatch logs
- **Last9 Account**: With OpenTelemetry log ingestion enabled

1.  **Configure AWS IAM Permissions**

    Create an IAM policy that allows the collector to read CloudWatch logs:

**IAM Policy JSON**

    Create a policy with the following permissions:

    ```json
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "CloudWatchLogsRead",
          "Effect": "Allow",
          "Action": [
            "logs:DescribeLogGroups",
            "logs:DescribeLogStreams",
            "logs:FilterLogEvents",
            "logs:GetLogEvents"
          ],
          "Resource": "*"
        }
      ]
    }
    ```

**AWS CLI**

    Create the policy using AWS CLI:

    ```bash
    # Create the policy document
    cat << EOF > cloudwatch-logs-policy.json
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "CloudWatchLogsRead",
          "Effect": "Allow",
          "Action": [
            "logs:DescribeLogGroups",
            "logs:DescribeLogStreams",
            "logs:FilterLogEvents",
            "logs:GetLogEvents"
          ],
          "Resource": "*"
        }
      ]
    }
    EOF

    # Create the policy
    aws iam create-policy \
      --policy-name CloudWatch-Logs-Read-Policy \
      --policy-document file://cloudwatch-logs-policy.json
    ```

**Terraform**

    ```hcl
    resource "aws_iam_policy" "cloudwatch_logs_read" {
      name = "CloudWatch-Logs-Read-Policy"

      policy = jsonencode({
        Version = "2012-10-17"
        Statement = [
          {
            Sid = "CloudWatchLogsRead"
            Effect = "Allow"
            Action = [
              "logs:DescribeLogGroups",
              "logs:DescribeLogStreams",
              "logs:FilterLogEvents",
              "logs:GetLogEvents"
            ]
            Resource = "*"
          }
        ]
      })
    }
    ```

2.  **Attach Policy to EC2 Instance Role**

    Attach the policy to your EC2 instance role or create IAM credentials:

**EC2 Instance Role**

    ```bash
    # Attach policy to existing EC2 instance role
    aws iam attach-role-policy \
      --role-name EC2-CloudWatch-Role \
      --policy-arn arn:aws:iam::YOUR-ACCOUNT-ID:policy/CloudWatch-Logs-Read-Policy
    ```

**IAM User (Alternative)**

    ```bash
    # Create IAM user for programmatic access
    aws iam create-user --user-name cloudwatch-logs-reader

    # Attach policy to user
    aws iam attach-user-policy \
      --user-name cloudwatch-logs-reader \
      --policy-arn arn:aws:iam::YOUR-ACCOUNT-ID:policy/CloudWatch-Logs-Read-Policy

    # Create access keys (store securely)
    aws iam create-access-key --user-name cloudwatch-logs-reader
    ```

3.  **Install OpenTelemetry Collector**

    Install the OpenTelemetry Collector with AWS CloudWatch receiver support:

**DEB Package**

    For Debian/Ubuntu systems:

    ```bash
    wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.118.0/otelcol-contrib_0.118.0_linux_amd64.deb
    sudo dpkg -i otelcol-contrib_0.118.0_linux_amd64.deb
    ```

**RPM Package**

    For Red Hat/CentOS systems:

    ```bash
    wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.118.0/otelcol-contrib_0.118.0_linux_amd64.rpm
    sudo rpm -ivh otelcol-contrib_0.118.0_linux_amd64.rpm
    ```

4.  **Configure AWS Credentials (if not using EC2 role)**

    If not using an EC2 instance role, configure AWS credentials:

**AWS Credentials File**

    Create `~/.aws/credentials`:

    ```ini
    [default]
    aws_access_key_id = YOUR_ACCESS_KEY_ID
    aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
    region = us-east-1
    ```

**Environment Variables**

    ```bash
    export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID
    export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
    export AWS_DEFAULT_REGION=us-east-1
    ```

5.  **Create OpenTelemetry Collector Configuration**

    Create the collector configuration file:

    ```bash
    sudo mkdir -p /etc/otelcol-contrib
    sudo nano /etc/otelcol-contrib/config.yaml
    ```

    Add the following configuration to collect CloudWatch logs:

    ```yaml
    receivers:
      awscloudwatch:
        region: us-east-1 # Change to your AWS region
        logs:
          poll_interval: 2m # How often to check for new logs
          groups:
            # Auto-discover log groups by prefix
            autodiscover:
              limit: 100 # Maximum number of log groups to discover
              prefixes:
                - "/aws/elb" # Elastic Load Balancer logs
                - "/aws/alb" # Application Load Balancer logs
                - "/aws/lambda" # Lambda function logs
                - "/aws/rds" # RDS database logs
                - "/aws/ecs" # ECS container logs
                - "/aws/apigateway" # API Gateway logs
                - "/aws/cloudtrail" # CloudTrail audit logs
                - "/aws/s3" # S3 access logs
                - "/aws/codebuild" # CodeBuild logs

            # Specify exact log group names (optional)
            named:
              - "/aws/elb/production-alb"
              - "/aws/lambda/user-authentication"
              # Add your specific log groups here

    processors:
      batch:
        timeout: 30s
        send_batch_size: 50000
        send_batch_max_size: 100000

      # Add timestamps for logs that don't have them
      transform/add_timestamp:
        error_mode: ignore
        log_statements:
          - context: log
            conditions:
              - time_unix_nano == 0
            statements:
              - set(observed_time, Now())
              - set(time_unix_nano, observed_time_unix_nano)

      # Add service identification and source
      transform/logs:
        error_mode: ignore
        log_statements:
          - context: log
            statements:
              - set(resource.attributes["service.name"], "aws-cloudwatch-logs")
              - set(resource.attributes["deployment.environment"], "production")
              - set(attributes["source"], "cloudwatch")
              - set(attributes["cloud.provider"], "aws")
              - set(attributes["cloud.region"], "us-east-1")

      # Detect EC2 metadata
      resourcedetection/aws:
        detectors: ["env", "ec2"]
        timeout: 2s
        override: false

    exporters:
      otlp/last9:
        endpoint: "$last9_otlp_endpoint"
        headers:
          "Authorization": "$last9_otlp_auth_header"
      debug:
        verbosity: detailed

    service:
      pipelines:
        logs:
          receivers: [awscloudwatch]
          processors:
            [
              batch,
              resourcedetection/aws,
              transform/add_timestamp,
              transform/logs,
            ]
          exporters: [otlp/last9]
    ```

6.  **Advanced Log Group Configuration**

    Configure specific log groups with custom settings:

**Service-Specific Groups**

    Monitor specific services with targeted configuration:

    ```yaml
    receivers:
      awscloudwatch/elb:
        region: us-east-1
        logs:
          poll_interval: 1m # More frequent polling for critical services
          groups:
            named:
              - "/aws/elb/production-frontend"
              - "/aws/elb/production-api"

      awscloudwatch/lambda:
        region: us-east-1
        logs:
          poll_interval: 2m
          groups:
            autodiscover:
              limit: 50
              prefixes:
                - "/aws/lambda"

    service:
      pipelines:
        logs/elb:
          receivers: [awscloudwatch/elb]
          processors: [batch, transform/logs]
          exporters: [otlp/last9]
        logs/lambda:
          receivers: [awscloudwatch/lambda]
          processors: [batch, transform/logs]
          exporters: [otlp/last9]
    ```

**Multi-Region Setup**

    Collect logs from multiple AWS regions:

    ```yaml
    receivers:
      awscloudwatch/us-east-1:
        region: us-east-1
        logs:
          groups:
            autodiscover:
              prefixes: ["/aws/elb", "/aws/lambda"]

      awscloudwatch/us-west-2:
        region: us-west-2
        logs:
          groups:
            autodiscover:
              prefixes: ["/aws/elb", "/aws/lambda"]

    processors:
      transform/region:
        log_statements:
          - context: log
            statements:
              - set(attributes["aws.region"], resource.attributes["cloud.region"])
    ```

**Log Filtering**

    Filter logs based on content or metadata:

    ```yaml
    receivers:
      awscloudwatch:
        region: us-east-1
        logs:
          groups:
            named:
              - "/aws/elb/production-alb"
          # Filter by log stream patterns
          streams:
            prefixes:
              - "elb.production"
            names:
              - "specific-stream-name"

    processors:
      # Filter out debug logs
      filter/logs:
        logs:
          exclude:
            match_type: regexp
            bodies:
              - ".*DEBUG.*"
              - ".*TRACE.*"
    ```

7.  **Create Systemd Service Configuration**

    Create a systemd service file:

    ```bash
    sudo nano /etc/systemd/system/otelcol-contrib.service
    ```

    Add the service configuration:

    ```ini
    [Unit]
    Description=OpenTelemetry Collector for AWS CloudWatch Logs
    After=network.target

    [Service]
    ExecStart=/usr/bin/otelcol-contrib --config /etc/otelcol-contrib/config.yaml
    Restart=always
    User=root
    Group=root
    Environment=AWS_REGION=us-east-1

    [Install]
    WantedBy=multi-user.target
    ```

8.  **Start and Enable the Service**

    Start the OpenTelemetry Collector service:

    ```bash
    sudo systemctl daemon-reload
    sudo systemctl enable otelcol-contrib
    sudo systemctl start otelcol-contrib
    ```

## Understanding CloudWatch Log Groups

CloudWatch automatically receives logs from various AWS services:

### AWS Service Logs

- **Load Balancers**: ELB, ALB, NLB access logs and error logs
- **Lambda Functions**: Function execution logs, error traces, and custom logs
- **RDS Databases**: Error logs, slow query logs, general logs
- **API Gateway**: Request/response logs, execution logs
- **VPC Flow Logs**: Network traffic logs for security analysis

### Application Logs

- **ECS Containers**: Application logs from containerized services
- **EC2 Applications**: Custom application logs via CloudWatch agent
- **Elastic Beanstalk**: Application and web server logs
- **CodeBuild**: Build process logs and debugging information

### Audit and Security Logs

- **CloudTrail**: API call audit logs for security monitoring
- **GuardDuty**: Security finding logs and threat detection
- **Config**: Configuration change logs for compliance
- **WAF**: Web application firewall logs

## Advanced Configuration

### Log Parsing and Enrichment

Add structured parsing for common AWS log formats:

```yaml
processors:
  transform/parse_elb:
    log_statements:
      - context: log
        conditions:
          - attributes["aws.log_group"] == "/aws/elb/app"
        statements:
          # Parse ELB access log format
          - set(attributes["elb.type"], "application")
          - set(attributes["http.method"], "GET") # Extract from log body
          - set(attributes["http.status_code"], 200) # Parse from log

  transform/parse_lambda:
    log_statements:
      - context: log
        conditions:
          - attributes["aws.log_group"] matches "/aws/lambda/.*"
        statements:
          - set(attributes["faas.name"], "lambda-function")
          - set(attributes["faas.version"], "$LATEST")
```

### Log Sampling and Volume Control

Control log volume for high-traffic services:

```yaml
processors:
  # Sample logs to reduce volume
  probabilistic_sampler:
    sampling_percentage: 10 # Keep 10% of logs

  # Rate limiting for log ingestion
  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 128

  # Batch optimization for high volume
  batch:
    timeout: 10s
    send_batch_size: 10000
    send_batch_max_size: 50000
```

### Custom Metadata Enhancement

Add business context to logs:

```yaml
processors:
  transform/business_context:
    log_statements:
      - context: log
        statements:
          - set(attributes["environment"], "production")
          - set(attributes["team"], "platform")
          - set(attributes["cost_center"], "engineering")
          - set(attributes["log_level"], "info") # Default level
          # Add service mapping based on log group
          - set(attributes["service.name"], "web-server") where attributes["aws.log_group"] == "/aws/elb/web"
          - set(attributes["service.name"], "user-service") where attributes["aws.log_group"] == "/aws/lambda/user-auth"
```

## Verification

1. **Check Service Status**

   Verify the OpenTelemetry Collector is running:

   ```bash
   sudo systemctl status otelcol-contrib
   ```

2. **Monitor Service Logs**

   Check for any configuration errors:

   ```bash
   sudo journalctl -u otelcol-contrib -f
   ```

   Look for successful log group discovery and log ingestion messages.

3. **Verify AWS Connectivity**

   Test CloudWatch access:

   ```bash
   # List available log groups
   aws logs describe-log-groups --region us-east-1 --max-items 10

   # Test log events retrieval
   aws logs filter-log-events \
     --log-group-name "/aws/elb/test" \
     --start-time $(date -d '1 hour ago' +%s)000 \
     --region us-east-1
   ```

4. **Check Log Group Activity**

   Verify there are recent log events in your log groups:

   ```bash
   # Check recent log streams
   aws logs describe-log-streams \
     --log-group-name "/aws/lambda/your-function" \
     --order-by LastEventTime \
     --descending \
     --max-items 5
   ```

5. **Generate Test Logs**

   Create test log entries for verification:

   ```bash
   # For Lambda functions - trigger function execution
   aws lambda invoke \
     --function-name your-test-function \
     --payload '{"test": "data"}' \
     response.json

   # For ALB - make HTTP requests to generate access logs
   curl https://your-alb-endpoint.com/health

   # For custom log groups - send test log events
   aws logs put-log-events \
     --log-group-name "/custom/application" \
     --log-stream-name "test-stream" \
     --log-events timestamp=$(date +%s)000,message="Test log message"
   ```

6. **Verify Logs in Last9**

   Log into your Last9 account and check that CloudWatch logs are being received in the [Logs dashboard](https://app.last9.io/logs).

   Look for:

   - AWS service logs with proper source attribution
   - Log entries with CloudWatch metadata
   - Proper timestamp and service name attribution
   - Regional and service-specific log categorization

## Common Log Group Patterns

### AWS Service Log Groups

| Service                   | Log Group Pattern                 | Content                       |
| ------------------------- | --------------------------------- | ----------------------------- |
| Application Load Balancer | `/aws/elb/app/app-name`           | HTTP access logs, error logs  |
| Lambda Functions          | `/aws/lambda/function-name`       | Function logs, errors, traces |
| RDS                       | `/aws/rds/instance/db-name/error` | Database error logs           |
| API Gateway               | `/aws/apigateway/api-name`        | API request/response logs     |
| CloudTrail                | `/aws/cloudtrail/trail-name`      | API audit logs                |

### Custom Application Log Groups

| Application Type | Recommended Pattern                 | Use Case                 |
| ---------------- | ----------------------------------- | ------------------------ |
| Web Applications | `/app/web/service-name`             | Application server logs  |
| Microservices    | `/service/service-name/environment` | Service-specific logs    |
| Batch Jobs       | `/jobs/job-name`                    | Scheduled task logs      |
| Security Logs    | `/security/component`               | Security monitoring logs |

## Best Practices

### Security

- **IAM Roles**: Use EC2 instance roles instead of hardcoded credentials
- **Least Privilege**: Grant only necessary CloudWatch permissions
- **Network Security**: Use VPC endpoints for CloudWatch API calls in private subnets
- **Log Retention**: Configure appropriate log retention policies in CloudWatch

### Performance

- **Polling Intervals**: Balance between real-time needs and API costs
- **Batch Processing**: Optimize batch sizes for efficient transmission
- **Memory Management**: Set appropriate memory limits for the collector
- **Regional Deployment**: Deploy collectors in the same region as log groups

### Cost Management

- **Log Group Selection**: Be selective about which log groups to monitor
- **Retention Policies**: Set appropriate retention periods for CloudWatch logs
- **Sampling**: Use sampling for high-volume, less critical log streams
- **API Usage**: Monitor CloudWatch API usage and costs

### Monitoring Strategy

- **Service Categorization**: Group logs by service, environment, and team
- **Alert Setup**: Configure alerts for critical error patterns in logs
- **Dashboard Creation**: Build dashboards for different log sources and use cases
- **Log Correlation**: Use consistent service names and metadata across logs and metrics

---

## Troubleshooting

### No Logs Being Ingested

**Check AWS Permissions:**

```bash
# Test log group access
aws logs describe-log-groups --region us-east-1

# Test log event retrieval
aws logs filter-log-events \
  --log-group-name "/aws/lambda/test" \
  --start-time $(date -d '10 minutes ago' +%s)000
```

**Verify Collector Configuration:**

```bash
# Check collector logs
sudo journalctl -u otelcol-contrib -f

# Look for AWS CloudWatch receiver initialization
grep -i "cloudwatch" /var/log/otelcol-contrib.log
```

### High Memory Usage

**Optimize Batch Processing:**

```yaml
processors:
  batch:
    timeout: 10s # Shorter timeout
    send_batch_size: 5000 # Smaller batches
    send_batch_max_size: 10000

  memory_limiter:
    limit_mib: 256 # Lower memory limit
```

### Rate Limiting Issues

**Handle AWS API Limits:**

```yaml
receivers:
  awscloudwatch:
    logs:
      poll_interval: 5m # Reduce polling frequency
      groups:
        autodiscover:
          limit: 50 # Reduce log group discovery limit
```

### Missing Recent Logs

**Check Polling Configuration:**

- Reduce `poll_interval` for more frequent log retrieval
- Verify log groups have recent activity
- Check CloudWatch log retention settings

Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:support@last9.io) if you have any questions.