AWS S3

Automatically ingest logs from your AWS S3 buckets using Last9’s managed log ingestion service. This integration uses an event-driven architecture where S3 bucket notifications trigger automatic log processing through a Last9-managed SQS queue.

This integration is ideal for ingesting application logs, AWS service logs (CloudTrail, VPC Flow Logs, ALB logs, etc.), and other structured log data stored in S3 buckets.

How It Works

This integration enables Last9 to automatically ingest logs from your AWS S3 bucket using an event-driven architecture:

1. Event Trigger: When new log files are uploaded to your S3 bucket, S3 sends notifications to a Last9-managed SQS queue
2. Secure Access: Last9 accesses your S3 bucket securely through AWS STS (Security Token Service) assume role mechanism
3. Automatic Processing: The log ingestion service automatically processes and ingests the log files into Last9
4. Real-time Ingestion: New logs are processed within minutes of being uploaded to S3

Prerequisites

Before setting up AWS S3 log ingestion, ensure you have:

• AWS Account: With administrative access to create IAM roles and modify S3 bucket settings
• S3 Bucket: Containing the logs you want to ingest
• Log Files: Structured log files in supported formats (JSON, text, etc.)
• Last9 Account: With log ingestion enabled
• SQS Queue ARN: Provided by Last9 team for your specific AWS region

Contact Last9 Team: Before starting, contact the Last9 team to obtain:

• SQS queue ARN for your AWS region
• Any specific configuration requirements for your use case

Provide the Last9 team with:

• Your S3 bucket ARN
• AWS account ID
• AWS region where your S3 bucket is located

1. Create IAM Role for Last9 Access

   Create a new IAM role that allows Last9 to securely access your S3 bucket:

   • AWS Console
   • AWS CLI
   • Terraform
   1. Open the IAM console
   2. Navigate to Roles and click Create role
   3. Select Custom trust policy
   4. Contact cs@last9.io to get the Last9 principal user ARN to replace <last9-cold-storage-user> in the trust policy below
   5. Paste the following trust policy:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "<last9-cold-storage-user>",
           "Service": "s3.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }

   6. Click Next and proceed to attach policies in the next step

2. Attach S3 Access Policy

   Create and attach a policy that grants Last9 the minimum required permissions to access your S3 bucket:

   • AWS Console
   • AWS CLI
   • Terraform
   1. In the IAM role creation wizard, click Create policy
   2. Switch to the JSON tab
   3. Paste the following policy (replace your-bucket-name with your actual bucket name):

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "Last9S3BucketAccess",
         "Effect": "Allow",
         "Action": ["s3:GetObject", "s3:ListBucket"],
         "Resource": [
           "arn:aws:s3:::your-bucket-name",
           "arn:aws:s3:::your-bucket-name/*"
         ]
       }
     ]
   }

   4. Name the policy Last9-S3-Access-Policy
   5. Create the policy and attach it to the role

   Important Notes:

   • Replace your-bucket-name with your actual S3 bucket name in both resource ARNs
   • The policy grants only the minimum required permissions (GetObject and ListBucket)
   • For multiple buckets, add additional resource ARNs to the policy

3. Configure S3 Event Notifications

   Set up S3 event notifications to trigger Last9’s log ingestion when new objects are created:

   • AWS Console
   • AWS CLI
   • Terraform
   1. Open the S3 console and select your bucket
   2. Go to the Properties tab
   3. Scroll down to Event notifications and click Create event notification
   4. Configure the event:
      • Event name: Last9-Log-Ingestion
      • Event types: Select All object create events or specific events like s3:ObjectCreated:Put
      • Prefix: (Optional) Specify a prefix if logs are in a specific folder (e.g., logs/)
      • Suffix: (Optional) Specify file extensions if needed (e.g., .log, .json)
      • Destination: Select SQS queue
      • SQS queue: Enter the SQS ARN provided by the Last9 team
   5. Click Save changes

   Configuration Options:

   • Prefix filtering: Use to monitor specific folders (e.g., logs/, application/)
   • Suffix filtering: Use to monitor specific file types (e.g., .json, .log, .txt)
   • Event types: Choose s3:ObjectCreated:* for all create events, or specific events like s3:ObjectCreated:Put

4. Verify the Setup

   Test the configuration to ensure everything is working correctly:

   • Upload Test File
   • Check Event Notifications

   Upload a test log file to your S3 bucket:

   # Create a test log file
   echo '{"timestamp": "2024-01-01T12:00:00Z", "level": "info", "message": "test log"}' > test-log.json
   
   # Upload to S3
   aws s3 cp test-log.json s3://your-bucket-name/logs/test-log.json

5. Share Configuration with Last9 Team

   Provide the following information to the Last9 team to complete the setup:

   Required Information:

   • IAM Role ARN: The ARN of the role created in Step 1
   • S3 Bucket ARN: The ARN of your S3 bucket
   • Log Format: Description of your log format (JSON, text, CSV, etc.)
   • Log Structure: Sample log entries to help with parsing configuration

   Example Information Package:

   IAM Role ARN: arn:aws:iam::123456789012:role/Last9-S3-Access-Role
   S3 Bucket ARN: arn:aws:s3:::my-application-logs
   S3 Region: us-east-1
   Log Format: JSON
   Sample Log:
   {
     "timestamp": "2024-01-01T12:00:00Z",
     "level": "info",
     "service": "api-server",
     "message": "Request processed",
     "duration": 120
   }

   Optional Information:

   • Log Prefix/Path: If logs are stored in specific folders
   • Retention Requirements: Any specific data retention needs
   • Parsing Requirements: Custom parsing or transformation needs

Supported Log Formats

Last9’s S3 integration supports various log formats:

Structured Logs

• JSON: Recommended format for best parsing and analysis
• JSONL: JSON Lines format for streaming logs
• CSV: Comma-separated values with headers
• TSV: Tab-separated values

Semi-Structured Logs

• Apache/NGINX Access Logs: Common web server log formats
• AWS CloudTrail: AWS API call logs
• AWS VPC Flow Logs: Network traffic logs
• Application Logs: Custom application log formats

Log Compression

• GZIP: Compressed log files (.gz extension)
• BZIP2: Alternative compression format
• ZIP: Archive files containing log data

Advanced Configuration

Multi-Bucket Setup

Configure multiple S3 buckets for log ingestion:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Last9MultipleBucketAccess",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::app-logs-bucket",
        "arn:aws:s3:::app-logs-bucket/*",
        "arn:aws:s3:::audit-logs-bucket",
        "arn:aws:s3:::audit-logs-bucket/*",
        "arn:aws:s3:::infrastructure-logs-bucket",
        "arn:aws:s3:::infrastructure-logs-bucket/*"
      ]
    }
  ]
}

Cross-Account Access

For cross-account S3 access, modify the trust policy (contact cs@last9.io to get the Last9 principal user ARN to replace <last9-cold-storage-user>):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "<last9-cold-storage-user>",
          "arn:aws:iam::YOUR-ACCOUNT-ID:root"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "unique-external-id-provided-by-last9"
        }
      }
    }
  ]
}

Selective Log Ingestion

Use S3 event filtering for selective log ingestion:

{
  "QueueConfigurations": [
    {
      "Id": "ErrorLogsOnly",
      "QueueArn": "arn:aws:sqs:region:account:queue",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "logs/error/"
            },
            {
              "Name": "suffix",
              "Value": ".json"
            }
          ]
        }
      }
    }
  ]
}

Verification and Monitoring

Verify Log Ingestion

1. Upload Test Logs

   Upload sample log files to verify the integration:

   # Upload a test log file
   aws s3 cp sample-log.json s3://your-bucket-name/logs/
   
   # Verify the file was uploaded
   aws s3 ls s3://your-bucket-name/logs/

2. Monitor SQS Queue (Optional)

   If you have access to SQS metrics, monitor queue activity:

   # Check queue attributes
   aws sqs get-queue-attributes \
     --queue-url https://sqs.region.amazonaws.com/account/queue-name \
     --attribute-names All

3. Check Last9 Logs Dashboard

   Verify logs are appearing in your Last9 dashboard:

   • Log into your Last9 account
   • Navigate to the Logs section
   • Filter by your service name or log source
   • Verify timestamps and log structure

Troubleshooting

No Logs Appearing

Check IAM Permissions:

# Verify role exists and has correct permissions
aws iam get-role --role-name Last9-S3-Access-Role
aws iam list-attached-role-policies --role-name Last9-S3-Access-Role

Verify S3 Event Notifications:

# Check notification configuration
aws s3api get-bucket-notification-configuration --bucket your-bucket-name

# Test with a manual upload
aws s3 cp test-file.log s3://your-bucket-name/logs/

Permission Denied Errors

Common Causes:

• Incorrect S3 bucket name in IAM policy
• Missing ListBucket permission
• Trust policy not allowing Last9’s service principal

Fix IAM Policy:

{
  "Resource": [
    "arn:aws:s3:::correct-bucket-name",
    "arn:aws:s3:::correct-bucket-name/*"
  ]
}

SQS Notification Issues

Check Event Configuration:

• Verify SQS ARN is correct
• Ensure the queue is in the same region as S3 bucket
• Check that event types match your upload patterns

Test Notifications:

# Upload file and check if notifications are sent
aws s3 cp test.log s3://bucket/logs/test.log

Best Practices

Security

• Least Privilege: Grant only minimum required S3 permissions
• External ID: Use external ID in trust relationships for additional security
• Regular Review: Periodically review and rotate IAM policies
• Resource Restrictions: Limit access to specific bucket paths when possible

Performance

• Batch Uploads: Upload files in batches to reduce SQS notification volume
• File Size: Keep individual log files reasonably sized (< 100MB recommended)
• Compression: Use GZIP compression to reduce storage and transfer costs
• Lifecycle Policies: Implement S3 lifecycle policies for cost optimization

Monitoring

• CloudTrail: Monitor S3 API calls for security auditing
• Cost Monitoring: Track S3 storage, request, and data transfer costs
• SQS Metrics: Monitor queue depth and message processing rates
• Log Validation: Verify log completeness and format consistency

Organization

• Naming Conventions: Use consistent naming for buckets and log files
• Folder Structure: Organize logs by service, environment, and date
• Metadata: Use S3 object tags for additional log categorization
• Retention: Implement appropriate log retention policies

Need Help?

If you encounter any issues or have questions:

• Join our Discord community for real-time support
• Contact our support team at support@last9.io
• For setup assistance, provide the Last9 team with your IAM role ARN and S3 bucket details