# AWS S3

> Automatically ingest logs from AWS S3 buckets using Last9's event-driven log ingestion service with SQS and IAM integration

Source: https://last9.io/docs/integrations/databases/aws-s3/

Automatically ingest logs from your AWS S3 buckets using Last9's managed log ingestion service. This integration uses an event-driven architecture where S3 bucket notifications trigger automatic log processing through a Last9-managed SQS queue.

This integration is ideal for ingesting application logs, AWS service logs (CloudTrail, VPC Flow Logs, ALB logs, etc.), and other structured log data stored in S3 buckets.

## How It Works

This integration enables Last9 to automatically ingest logs from your AWS S3 bucket using an event-driven architecture:

1. **Event Trigger**: When new log files are uploaded to your S3 bucket, S3 sends notifications to a Last9-managed SQS queue
2. **Secure Access**: Last9 accesses your S3 bucket securely through AWS STS (Security Token Service) assume role mechanism
3. **Automatic Processing**: The log ingestion service automatically processes and ingests the log files into Last9
4. **Real-time Ingestion**: New logs are processed within minutes of being uploaded to S3

## Prerequisites

Before setting up AWS S3 log ingestion, ensure you have:

- **AWS Account**: With administrative access to create IAM roles and modify S3 bucket settings
- **S3 Bucket**: Containing the logs you want to ingest
- **Log Files**: Structured log files in supported formats (JSON, text, etc.)
- **Last9 Account**: With log ingestion enabled
- **SQS Queue ARN**: Provided by Last9 team for your specific AWS region

**Contact Last9 Team**: Before starting, contact the Last9 team to obtain:

- SQS queue ARN for your AWS region
- Any specific configuration requirements for your use case

Provide the Last9 team with:

- Your S3 bucket ARN
- AWS account ID
- AWS region where your S3 bucket is located

1.  **Create IAM Role for Last9 Access**

    Create a new IAM role that allows Last9 to securely access your S3 bucket:

**AWS Console**

    1. Open the [IAM console](https://console.aws.amazon.com/iam/)
    2. Navigate to **Roles** and click **Create role**
    3. Select **Custom trust policy**
    4. Contact support@last9.io to get the Last9 principal user ARN to replace `<last9-cold-storage-user>` in the trust policy below
    5. Paste the following trust policy:

    ```json
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "<last9-cold-storage-user>",
            "Service": "s3.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
    ```

    6.  Click **Next** and proceed to attach policies in the next step

**AWS CLI**

    Create a trust policy file (contact support@last9.io to get the Last9 principal user ARN to replace `<last9-cold-storage-user>`):

    ```bash
    cat << EOF > last9-s3-trust-policy.json
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "<last9-cold-storage-user>",
            "Service": "s3.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
    EOF
    ```

    Create the IAM role:

    ```bash
    aws iam create-role \
      --role-name Last9-S3-Access-Role \
      --assume-role-policy-document file://last9-s3-trust-policy.json \
      --description "Role for Last9 to access S3 bucket logs"
    ```

**Terraform**

    Contact support@last9.io to get the Last9 principal user ARN to replace `<last9-cold-storage-user>` in the configuration below.

    ```hcl
    resource "aws_iam_role" "last9_s3_access" {
      name = "Last9-S3-Access-Role"

      assume_role_policy = jsonencode({
        Version = "2012-10-17"
        Statement = [
          {
            Effect = "Allow"
            Principal = {
              AWS = "<last9-cold-storage-user>"
              Service = "s3.amazonaws.com"
            }
            Action = "sts:AssumeRole"
          }
        ]
      })
    }
    ```

2.  **Attach S3 Access Policy**

    Create and attach a policy that grants Last9 the minimum required permissions to access your S3 bucket:

**AWS Console**

    1. In the IAM role creation wizard, click **Create policy**
    2. Switch to the **JSON** tab
    3. Paste the following policy (replace `your-bucket-name` with your actual bucket name):

    ```json
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "Last9S3BucketAccess",
          "Effect": "Allow",
          "Action": ["s3:GetObject", "s3:ListBucket"],
          "Resource": [
            "arn:aws:s3:::your-bucket-name",
            "arn:aws:s3:::your-bucket-name/*"
          ]
        }
      ]
    }
    ```

    4.  Name the policy `Last9-S3-Access-Policy`
    5.  Create the policy and attach it to the role

**AWS CLI**

    Create the policy document:

    ```bash
    cat << EOF > last9-s3-access-policy.json
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "Last9S3BucketAccess",
          "Effect": "Allow",
          "Action": [
            "s3:GetObject",
            "s3:ListBucket"
          ],
          "Resource": [
            "arn:aws:s3:::your-bucket-name",
            "arn:aws:s3:::your-bucket-name/*"
          ]
        }
      ]
    }
    EOF
    ```

    Create and attach the policy:

    ```bash
    # Create the policy
    aws iam create-policy \
      --policy-name Last9-S3-Access-Policy \
      --policy-document file://last9-s3-access-policy.json

    # Attach the policy to the role
    aws iam attach-role-policy \
      --role-name Last9-S3-Access-Role \
      --policy-arn arn:aws:iam::YOUR-ACCOUNT-ID:policy/Last9-S3-Access-Policy
    ```

**Terraform**

    ```hcl
    resource "aws_iam_policy" "last9_s3_access" {
      name = "Last9-S3-Access-Policy"

      policy = jsonencode({
        Version = "2012-10-17"
        Statement = [
          {
            Sid = "Last9S3BucketAccess"
            Effect = "Allow"
            Action = [
              "s3:GetObject",
              "s3:ListBucket"
            ]
            Resource = [
              "arn:aws:s3:::${var.s3_bucket_name}",
              "arn:aws:s3:::${var.s3_bucket_name}/*"
            ]
          }
        ]
      })
    }

    resource "aws_iam_role_policy_attachment" "last9_s3_access" {
      policy_arn = aws_iam_policy.last9_s3_access.arn
      role       = aws_iam_role.last9_s3_access.name
    }
    ```

    **Important Notes:**

    - Replace `your-bucket-name` with your actual S3 bucket name in both resource ARNs
    - The policy grants only the minimum required permissions (`GetObject` and `ListBucket`)
    - For multiple buckets, add additional resource ARNs to the policy

3.  **Configure S3 Event Notifications**

    Set up S3 event notifications to trigger Last9's log ingestion when new objects are created:

**AWS Console**

    1.  Open the [S3 console](https://console.aws.amazon.com/s3/) and select your bucket
    2.  Go to the **Properties** tab
    3.  Scroll down to **Event notifications** and click **Create event notification**
    4.  Configure the event:
        - **Event name**: `Last9-Log-Ingestion`
        - **Event types**: Select `All object create events` or specific events like `s3:ObjectCreated:Put`
        - **Prefix**: (Optional) Specify a prefix if logs are in a specific folder (e.g., `logs/`)
        - **Suffix**: (Optional) Specify file extensions if needed (e.g., `.log`, `.json`)
        - **Destination**: Select **SQS queue**
        - **SQS queue**: Enter the SQS ARN provided by the Last9 team
    5.  Click **Save changes**

**AWS CLI**

    Create a notification configuration file:

    ```bash
    cat << EOF > s3-notification-config.json
    {
      "QueueConfigurations": [
        {
          "Id": "Last9-Log-Ingestion",
          "QueueArn": "arn:aws:sqs:us-east-1:LAST9-ACCOUNT:last9-log-queue",
          "Events": ["s3:ObjectCreated:*"],
          "Filter": {
            "Key": {
              "FilterRules": [
                {
                  "Name": "prefix",
                  "Value": "logs/"
                }
              ]
            }
          }
        }
      ]
    }
    EOF
    ```

    Apply the notification configuration:

    ```bash
    aws s3api put-bucket-notification-configuration \
      --bucket your-bucket-name \
      --notification-configuration file://s3-notification-config.json
    ```

**Terraform**

    ```hcl
    resource "aws_s3_bucket_notification" "last9_log_ingestion" {
      bucket = var.s3_bucket_name

      queue {
        id            = "Last9-Log-Ingestion"
        queue_arn     = var.last9_sqs_arn
        events        = ["s3:ObjectCreated:*"]
        filter_prefix = "logs/"  # Optional: only notify for objects in logs/ prefix
      }
    }
    ```

    **Configuration Options:**

    - **Prefix filtering**: Use to monitor specific folders (e.g., `logs/`, `application/`)
    - **Suffix filtering**: Use to monitor specific file types (e.g., `.json`, `.log`, `.txt`)
    - **Event types**: Choose `s3:ObjectCreated:*` for all create events, or specific events like `s3:ObjectCreated:Put`

4.  **Verify the Setup**

    Test the configuration to ensure everything is working correctly:

**Upload Test File**

    Upload a test log file to your S3 bucket:

    ```bash
    # Create a test log file
    echo '{"timestamp": "2024-01-01T12:00:00Z", "level": "info", "message": "test log"}' > test-log.json

    # Upload to S3
    aws s3 cp test-log.json s3://your-bucket-name/logs/test-log.json
    ```

**Check Event Notifications**

    Verify the notification configuration:

    ```bash
    # Check bucket notification configuration
    aws s3api get-bucket-notification-configuration --bucket your-bucket-name

    # Verify the SQS queue receives messages (if you have access)
    aws sqs get-queue-attributes \
      --queue-url https://sqs.us-east-1.amazonaws.com/ACCOUNT/queue-name \
      --attribute-names ApproximateNumberOfMessages
    ```

5.  **Share Configuration with Last9 Team**

    Provide the following information to the Last9 team to complete the setup:

    **Required Information:**

    - **IAM Role ARN**: The ARN of the role created in Step 1
    - **S3 Bucket ARN**: The ARN of your S3 bucket
    - **Log Format**: Description of your log format (JSON, text, CSV, etc.)
    - **Log Structure**: Sample log entries to help with parsing configuration

    **Example Information Package:**

    ```
    IAM Role ARN: arn:aws:iam::123456789012:role/Last9-S3-Access-Role
    S3 Bucket ARN: arn:aws:s3:::my-application-logs
    S3 Region: us-east-1
    Log Format: JSON
    Sample Log:
    {
      "timestamp": "2024-01-01T12:00:00Z",
      "level": "info",
      "service": "api-server",
      "message": "Request processed",
      "duration": 120
    }
    ```

    **Optional Information:**

    - **Log Prefix/Path**: If logs are stored in specific folders
    - **Retention Requirements**: Any specific data retention needs
    - **Parsing Requirements**: Custom parsing or transformation needs

## Supported Log Formats

Last9's S3 integration supports various log formats:

### Structured Logs

- **JSON**: Recommended format for best parsing and analysis
- **JSONL**: JSON Lines format for streaming logs
- **CSV**: Comma-separated values with headers
- **TSV**: Tab-separated values

### Semi-Structured Logs

- **Apache/NGINX Access Logs**: Common web server log formats
- **AWS CloudTrail**: AWS API call logs
- **AWS VPC Flow Logs**: Network traffic logs
- **Application Logs**: Custom application log formats

### Log Compression

- **GZIP**: Compressed log files (.gz extension)
- **BZIP2**: Alternative compression format
- **ZIP**: Archive files containing log data

## Advanced Configuration

### Multi-Bucket Setup

Configure multiple S3 buckets for log ingestion:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Last9MultipleBucketAccess",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::app-logs-bucket",
        "arn:aws:s3:::app-logs-bucket/*",
        "arn:aws:s3:::audit-logs-bucket",
        "arn:aws:s3:::audit-logs-bucket/*",
        "arn:aws:s3:::infrastructure-logs-bucket",
        "arn:aws:s3:::infrastructure-logs-bucket/*"
      ]
    }
  ]
}
```

### Cross-Account Access

For cross-account S3 access, modify the trust policy (contact support@last9.io to get the Last9 principal user ARN to replace `<last9-cold-storage-user>`):

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "<last9-cold-storage-user>",
          "arn:aws:iam::YOUR-ACCOUNT-ID:root"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "unique-external-id-provided-by-last9"
        }
      }
    }
  ]
}
```

### Selective Log Ingestion

Use S3 event filtering for selective log ingestion:

```json
{
  "QueueConfigurations": [
    {
      "Id": "ErrorLogsOnly",
      "QueueArn": "arn:aws:sqs:region:account:queue",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "logs/error/"
            },
            {
              "Name": "suffix",
              "Value": ".json"
            }
          ]
        }
      }
    }
  ]
}
```

## Verification and Monitoring

### Verify Log Ingestion

1. **Upload Test Logs**

   Upload sample log files to verify the integration:

   ```bash
   # Upload a test log file
   aws s3 cp sample-log.json s3://your-bucket-name/logs/

   # Verify the file was uploaded
   aws s3 ls s3://your-bucket-name/logs/
   ```

2. **Monitor SQS Queue (Optional)**

   If you have access to SQS metrics, monitor queue activity:

   ```bash
   # Check queue attributes
   aws sqs get-queue-attributes \
     --queue-url https://sqs.region.amazonaws.com/account/queue-name \
     --attribute-names All
   ```

3. **Check Last9 Logs Dashboard**

   Verify logs are appearing in your Last9 dashboard:

   - Log into your [Last9 account](https://app.last9.io/)
   - Navigate to the **Logs** section
   - Filter by your service name or log source
   - Verify timestamps and log structure

## Troubleshooting

### No Logs Appearing

**Check IAM Permissions:**

```bash
# Verify role exists and has correct permissions
aws iam get-role --role-name Last9-S3-Access-Role
aws iam list-attached-role-policies --role-name Last9-S3-Access-Role
```

**Verify S3 Event Notifications:**

```bash
# Check notification configuration
aws s3api get-bucket-notification-configuration --bucket your-bucket-name

# Test with a manual upload
aws s3 cp test-file.log s3://your-bucket-name/logs/
```

### Permission Denied Errors

**Common Causes:**

- Incorrect S3 bucket name in IAM policy
- Missing `ListBucket` permission
- Trust policy not allowing Last9's service principal

**Fix IAM Policy:**

```json
{
  "Resource": [
    "arn:aws:s3:::correct-bucket-name",
    "arn:aws:s3:::correct-bucket-name/*"
  ]
}
```

### SQS Notification Issues

**Check Event Configuration:**

- Verify SQS ARN is correct
- Ensure the queue is in the same region as S3 bucket
- Check that event types match your upload patterns

**Test Notifications:**

```bash
# Upload file and check if notifications are sent
aws s3 cp test.log s3://bucket/logs/test.log
```

## Best Practices

### Security

- **Least Privilege**: Grant only minimum required S3 permissions
- **External ID**: Use external ID in trust relationships for additional security
- **Regular Review**: Periodically review and rotate IAM policies
- **Resource Restrictions**: Limit access to specific bucket paths when possible

### Performance

- **Batch Uploads**: Upload files in batches to reduce SQS notification volume
- **File Size**: Keep individual log files reasonably sized (< 100MB recommended)
- **Compression**: Use GZIP compression to reduce storage and transfer costs
- **Lifecycle Policies**: Implement S3 lifecycle policies for cost optimization

### Monitoring

- **CloudTrail**: Monitor S3 API calls for security auditing
- **Cost Monitoring**: Track S3 storage, request, and data transfer costs
- **SQS Metrics**: Monitor queue depth and message processing rates
- **Log Validation**: Verify log completeness and format consistency

### Organization

- **Naming Conventions**: Use consistent naming for buckets and log files
- **Folder Structure**: Organize logs by service, environment, and date
- **Metadata**: Use S3 object tags for additional log categorization
- **Retention**: Implement appropriate log retention policies

## Need Help?

If you encounter any issues or have questions:

- Join our [Discord community](https://discord.com/channels/652153247672729619/652153247672729621) for real-time support
- Contact our support team at [support@last9.io](mailto:support@last9.io)
- For setup assistance, provide the Last9 team with your IAM role ARN and S3 bucket details