How to View and Understand VPC Flow Logs

If you're running workloads in AWS, you've probably heard about VPC Flow Logs. These logs are your eyes and ears for network traffic in your Virtual Private Cloud, and knowing how to check them properly can save you hours of troubleshooting headaches.

Whether you're tracking down connectivity issues or monitoring for suspicious activity, this guide will walk you through checking VPC flow logs step by step, with practical examples you can apply today.

Understanding VPC Flow Logs

VPC Flow Logs capture information about the IP traffic going to and from network interfaces in your VPC. Think of them as the security cameras of your cloud network—they don't affect network performance or behavior, but they record everything that's happening.

Each flow log record includes detailed metadata such as:

• Source and destination IP addresses (both IPv4 and IPv6)
• Source and destination ports
• Protocol used (TCP, UDP, ICMP, etc.)
• Number of packets and bytes transferred during the flow
• Start and end timestamps of the capture window
• Action taken (accept or reject)
• Traffic direction (ingress or egress)
• VPC, subnet, and instance information

VPC Flow Logs can be published to three destinations:

1. Amazon CloudWatch Logs
2. Amazon S3
3. Amazon Kinesis Data Firehose

Each option has different advantages for storage, querying, and integration with other AWS services.

Enabling VPC Flow Logs: Step-by-Step Configuration Process

Before you can check logs, you need to have them enabled. Here's how to set them up if you haven't already:

1. Open the Amazon VPC console at https://console.aws.amazon.com/vpc/
2. In the navigation pane, select your resource (VPC, subnet, or network interface)
3. Click on the "Flow Logs" tab in the details pane
4. Click the "Create flow log" button
5. Configure the following settings:
   • Filter: Choose from "All", "Accepted", or "Rejected" traffic
   • Maximum aggregation interval: Select 1 minute or 10 minutes (shorter intervals mean more granular data but higher costs)
   • Destination: Choose CloudWatch Logs, S3, or Firehose
   • Log format: Default or custom format (custom lets you select specific fields)
   • IAM role: Select an existing role or create a new one with appropriate permissions
6. For S3 destinations, specify:
   • S3 bucket ARN: The ARN of your destination bucket
   • Log file format: Text or Parquet (Parquet is better for analytical queries)
   • Hive-compatible S3 prefix: Optional organizational structure
   • Per-hour partitioning: Enable for easier time-based queries
7. Click "Create flow log"

It typically takes 5-15 minutes for logs to start appearing after you've created a flow log. During high-traffic periods, this could take longer.

How to Access VPC Flow Logs in CloudWatch

If you've configured your logs to go to CloudWatch, here's the detailed process to access them:

1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/
2. Change to the region where your VPC exists
3. In the navigation pane, select "Logs" then "Log groups"
4. Find your VPC Flow Logs group (typically named like /aws/vpc/flowlogs-xxxxxxxxx)
5. Click on the log group to see individual log streams
6. Each log stream represents a specific network interface (ENI)
7. Use the search bar to filter streams by ENI ID if you have many interfaces
8. Click on a stream to view the chronological log entries

A typical raw flow log entry in CloudWatch looks like this:

2 123456789010 eni-1234567890123456 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK

This might look like gibberish at first, but it's quite useful once you know how to read it.

Using CloudWatch Logs Insights for Powerful Queries

CloudWatch Logs Insights provides a powerful query language for analyzing your flow logs:

1. In CloudWatch, select "Logs Insights" from the left navigation
2. Select your VPC Flow Logs log group
3. Use queries like:

fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, action
| filter action = "REJECT"
| sort @timestamp desc
| limit 100

This will show your most recent 100 rejected connections, which is extremely useful for troubleshooting access issues.

Interpreting VPC Flow Log Records: Field-by-Field Breakdown

Each entry follows this standard format when using the default log format:

Common protocol numbers you'll encounter:

• 1: ICMP (ping)
• 6: TCP (web traffic, SSH, most application traffic)
• 17: UDP (DNS, streaming, VoIP)
• 50: ESP (VPN traffic)

The Action field will be either ACCEPT or REJECT, indicating whether the traffic was allowed or blocked by your security groups or network ACLs.

How Do You Retrieve VPC Flow Logs from S3?

If you're sending your flow logs to S3, the process for accessing them differs:

1. Open the Amazon S3 console at https://console.aws.amazon.com/s3/
2. Navigate to your destination bucket
3. These files are compressed with gzip to save space
4. You can download individual files or use S3 Select for simple queries:
   • Select a .gz file
   • Actions → Query with S3 Select
   • Specify CSV format with space delimiter
   • Write a simple SQL query

Within each day folder, files are named with the following pattern:

account_id_vpcflowlogs_region_flow-log-id_yyyymmddhhmmss_hash.log.gz

Flow logs are organized hierarchically as follows:

bucket_name/optional_folder/AWSLogs/account_id/vpcflowlogs/region/year/month/day/

For larger-scale analysis, you can:

• Set up AWS Glue crawlers to catalog your flow logs
• Create an Athena table to query them (more on this later)
• Use EMR for big data processing
• Export to data lakes for long-term analytics

Advanced Filtering Techniques for VPC Flow Logs

The real power of flow logs comes from filtering them to find exactly what you're looking for. Here are practical filtering techniques:

In CloudWatch Logs:

1. Navigate to your log group
2. Click "Search log group"
3. Use filter patterns such as:

Find unusually large data transfers (potential data exfiltration):

[version, account, eni, source, destination, srcport, destport, protocol, packets, bytes>1000000, start, end, action="ACCEPT", log_status]

Find traffic from a specific CIDR range:

[version, account, eni, source=192.0.2.*, destination, srcport, destport, protocol, packets, bytes, start, end, action, log_status]

Find rejected traffic to a specific port:

[version, account, eni, source, destination, srcport, destport="3306", protocol, packets, bytes, start, end, action="REJECT", log_status]

In CloudWatch Logs Insights:

For more complex patterns, use Logs Insights queries:

fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, bytes, action
| filter dstPort = 22 and action = "ACCEPT"
| stats sum(bytes) as totalBytes by srcAddr
| sort totalBytes desc
| limit 10

This query helps you identify the top 10 IP addresses sending the most data to SSH ports—potentially identifying SSH tunneling or data exfiltration.

Practical Troubleshooting with VPC Flow Logs

Diagnosing Connection Timeouts and Refused Connections

If your application reports connection timeouts:

1. Identify the source and destination IP addresses and ports
2. If you see REJECT entries, check security groups and NACLs
3. If you see no entries at all, check route tables or whether the destination is running

Check for REJECT entries in flow logs:

fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, action| filter srcAddr = "10.0.1.4" and dstAddr = "10.0.2.5" and dstPort = 3306| sort @timestamp desc| limit 20

Detecting Unauthorized Access Attempts

To identify potential brute force attacks:

fields @timestamp, srcAddr, dstAddr, dstPort, action
| filter dstPort = 22 and action = "REJECT" 
| stats count(*) as attemptCount by srcAddr, bin(30m) as timeWindow
| filter attemptCount > 10
| sort timeWindow desc, attemptCount desc

This query finds IPs making more than 10 failed SSH connection attempts in a 30-minute window.

Validating Security Group Changes

After updating security group rules:

1. Make a test connection that should now be allowed/blocked
2. Verify the ACTION field shows ACCEPT or REJECT as expected
3. If you don't see expected results, check for more restrictive NACLs or overlapping security group rules

Query the flow logs with a time filter around your test:

fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, action| filter srcAddr = "192.168.1.10" and dstAddr = "10.0.0.5" and dstPort = 443| filter @timestamp > parse_iso_8601('2023-04-08T14:00:00') and @timestamp < parse_iso_8601('2023-04-08T14:05:00')| sort @timestamp desc

In-Depth Analysis with Amazon Athena

For powerful SQL-based analysis of logs stored in S3, set up an Athena table:

1. Open the Athena console at https://console.aws.amazon.com/athena/

Add partitions for faster queries:

ALTER TABLE vpc_flow_logsADD PARTITION (region='us-east-1', day='2023-04-08')LOCATION 's3://your-bucket/AWSLogs/your-account-id/vpcflowlogs/us-east-1/2023/04/08/';

Create a table for your VPC flow logs:

CREATE EXTERNAL TABLE vpc_flow_logs (  version int,  account string,  interface_id string,  srcaddr string,  dstaddr string,  srcport int,  dstport int,  protocol int,  packets int,  bytes int,  start_time int,  end_time int,  action string,  log_status string)PARTITIONED BY (region string, day string)ROW FORMAT DELIMITEDFIELDS TERMINATED BY ' 'LOCATION 's3://your-bucket/AWSLogs/your-account-id/vpcflowlogs/'TBLPROPERTIES ('skip.header.line.count'='1');

Create a database if you don't have one:

CREATE DATABASE vpc_flow_logs;

Now you can run powerful analytical queries like:

-- Find top talkers in your VPC
SELECT
  srcaddr,
  dstaddr,
  dstport,
  SUM(bytes) as total_bytes,
  COUNT(*) as connection_count
FROM vpc_flow_logs
WHERE day = '2023-04-08' 
  AND region = 'us-east-1'
  AND action = 'ACCEPT'
GROUP BY srcaddr, dstaddr, dstport
ORDER BY total_bytes DESC
LIMIT 20;

-- Find rejected connections to critical services
SELECT
  srcaddr,
  COUNT(*) as rejection_count,
  ARRAY_AGG(DISTINCT dstport) as attempted_ports
FROM vpc_flow_logs
WHERE day BETWEEN '2023-04-01' AND '2023-04-08'
  AND action = 'REJECT'
  AND dstport IN (22, 3389, 1433, 3306, 5432)
GROUP BY srcaddr
HAVING COUNT(*) > 50
ORDER BY rejection_count DESC;

Visualizing Network Traffic Patterns with QuickSight

To create visual dashboards from your flow logs:

1. Create an Athena table as described above
2. Open QuickSight at https://quicksight.aws.amazon.com/
3. Create a new analysis using Athena as the data source
4. Select your vpc_flow_logs table
5. Create visualizations like:
   • Heat maps of traffic by source/destination
   • Line charts of traffic volume over time
   • Bar charts of top rejected connections
   • Geospatial maps of traffic sources (after enriching with IP geolocation)

Example metrics to visualize:

• Traffic volume by hour of day
• Top N source/destination pairs
• Accepted vs. rejected traffic ratio
• Traffic distribution by port/protocol

VPC Flow Logs Cost Management and Performance Tips

Cost-Efficient Log Configuration

• Selective logging: Only enable flow logs on critical VPCs or subnets
• Sampling: Use the sampling option (e.g., record only 1 in every 10 flows)
• Aggregation: Use 10-minute aggregation instead of 1-minute when detailed timing isn't critical
• Field filtering: In custom formats, only include fields you actually need

Storage Management Strategies

CloudWatch Logs retention: Set appropriate retention periods:

aws logs put-retention-policy --log-group-name /aws/vpc/flowlogs --retention-in-days 30

S3 lifecycle policies: Automatically transition older logs to cheaper storage:

{
  "Rules": [
    {
      "ID": "Move to Glacier",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "AWSLogs/vpc-flow-logs/"
      },
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        }
      ],
      "Expiration": {
        "Days": 365
      }
    }
  ]
}

Performance Optimization

• Partitioning in S3: Use Hive-compatible prefixes and time-based partitioning
• Athena partitioning: Always query with partition filters (region, day)
• Custom format for analysis: If you frequently analyze specific fields, create a custom format ordering the most important fields first

Integrating VPC Flow Logs with Security Tools

Setting Up CloudWatch Alarms

Create alarms for security events:

1. In CloudWatch, create a metric filter:
   • Log group: Your VPC Flow Logs group
   • Filter pattern: [version, account, eni, source, destination, srcport, destport="22", protocol, packets, bytes, start, end, action="ACCEPT", log_status]
   • Metric name: SSHConnections
2. Create an alarm:
   • Threshold: > 10 for 5 minutes consecutive
   • Actions: SNS notification to your team

AWS Security Hub Integration

Send flow log anomalies to Security Hub:

1. Create a Lambda function that processes flow logs
2. Have it detect patterns like:
   • Unusual port access
   • Connections to known bad IP addresses
   • Sudden traffic spikes
3. When detected, send findings to Security Hub API

Third-Party SIEM Integration

Many security tools can ingest VPC Flow Logs:

• Sumo Logic
  ELK Stack
  Datadog

These platforms offer pre-built dashboards and detection rules specifically for VPC Flow Logs. Last9 helps you go a step further—bringing context and observability to your cloud traffic without the noise.

Last9 complements SIEM tools by helping you:

• Trace unusual traffic patterns in VPC Flow Logs
• Correlate infrastructure changes with incident timelines
• Cut through alert noise to surface only what needs attention

Conclusion

VPC Flow Logs offer a clear window into the network traffic moving through your AWS environment. They're especially useful when you're trying to figure out why a connection is failing, keeping an eye out for anything suspicious, or just making sure your setup is running as efficiently as possible.

This guide walks you through practical ways to use flow logs so you can:

• Spot and fix network issues faster
• Catch unusual or unwanted traffic early
• Check if your security group and NACL rules are doing what they should
• Analyze traffic patterns to improve performance and keep costs in check