VPC Log Format: Custom and Advanced Configurations

VPC Flow Logs come with a default format that gives you basic network traffic details. But you can tweak the format to capture exactly what you need. This can lower costs, speed up processing, and make your logs fit better with what you’re trying to monitor.

If you want to improve security, keep an eye on performance, or save money, adjusting your VPC logs can make a big difference. Let’s take a look at some practical ways to customize your logs beyond the default settings.

VPC Log Format Customization Options

AWS lets you choose from over 30 different fields when customizing your VPC log format. This flexibility means you don’t have to store—and pay for—data you don’t need. Instead, you can build log formats that fit your specific use cases.

The syntax is simple. Instead of using the default 14-field format, you pick exactly the fields you want, like this:

${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${action} ${bytes}

This example captures the basics of network connections while cutting storage costs by around 60% compared to the default.

What makes custom VPC log formats so useful is the ability to tailor them per environment or purpose. For instance, your production security monitoring might require detailed fields, while your development setup could use a leaner format to save money.

Essential VPC Log Format Fields Every Developer Should Know

When customizing your VPC Flow Logs, selecting the right fields is key. Here’s a detailed breakdown of important fields grouped by their purpose.

Core Connection Fields

• ${srcaddr} and ${dstaddr}
  These are the foundation of any network log. They represent the source and destination IP addresses, showing where traffic originates and where it’s going. Without these, you can’t map network flows.
• ${srcport} and ${dstport}
  These fields indicate the source and destination ports. Ports help identify the type of service or application involved—for example, port 443 usually means HTTPS traffic, while port 22 indicates SSH. Tracking ports helps you understand what kind of communication is happening.
• ${protocol}
  This shows the IP protocol number. The most common values are 6 (TCP), 17 (UDP), and 1 (ICMP). Knowing the protocol is critical for interpreting the connection type and troubleshooting network issues.

Traffic Volume Fields

• ${bytes} and ${packets}
  These fields record how much data passed during the flow. They’re crucial for monitoring bandwidth usage, spotting unusual traffic spikes, and estimating data transfer costs.
• ${pkt-srcaddr} and ${pkt-dstaddr}
  Unlike ${srcaddr} and ${dstaddr}, these represent packet-level addresses. In complex network setups with NAT or proxies, packet addresses may differ from flow addresses, providing additional granularity.

Timing and Flow Control

• ${start} and ${end}
  These fields log the Unix timestamps when a flow started and ended. They’re invaluable for understanding connection duration, spotting long-lived or short bursts of traffic, and correlating events.
• ${tcp-flags}
  TCP flags give insight into connection states and anomalies. For example, SYN flags indicate connection initiation, while RST flags indicate abrupt termination. Monitoring TCP flags can help detect attacks like SYN floods or troubleshoot connectivity problems.

Advanced AWS-Specific Fields

• ${pkt-src-aws-service} and ${pkt-dst-aws-service}
  These fields identify the AWS services involved in the traffic flow. This is useful for mapping dependencies between services and optimizing communication within your AWS environment.
• ${sublocation-type} and ${sublocation-id}
  These provide information about where inside AWS the traffic is flowing, such as specific availability zones or subnets. This helps diagnose performance or routing issues within your cloud infrastructure.
• ${flow-direction}
  This indicates if the traffic is inbound (ingress) or outbound (egress) relative to the network interface. It’s especially important for security analysis, helping you understand potential attack vectors or data exfiltration risks.

Tailoring VPC Log Formats for Different Needs

Your logging needs change depending on what you want to do with the data. Customizing your VPC Flow Logs lets you collect just the right info without extra clutter or cost.

Here are some common setups to consider:

Security-Focused Format

For security teams, it’s all about tracking who connected where, when, and if the connection was allowed:

${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${action} ${start} ${end} ${flow-direction}

This gives you the basics for investigating incidents without bloating your storage.

Performance Monitoring Format

If you’re keeping an eye on app performance or network health, these fields help highlight issues:

${srcaddr} ${dstaddr} ${bytes} ${packets} ${start} ${end} ${tcp-flags} ${sublocation-type}

TCP flags help spot connection problems, and the sublocation info shows traffic moving between availability zones, which can affect speed.

Cost Optimization Format

To keep cloud bills in check, focus on fields that matter for data transfer costs:

${srcaddr} ${dstaddr} ${bytes} ${pkt-src-aws-service} ${pkt-dst-aws-service} ${sublocation-id}

This keeps your logs lean and focused on what impacts your costs.

Compliance and Audit Format

For compliance or audits, you need detailed logs that cover all bases:

${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${flow-log-status}

Using this explicit format helps make sure you collect everything regulators might ask for.

How Field Order Impacts VPC Log Processing Performance

JSON vs Delimited Formats

AWS supports two main formats for VPC Flow Logs: space-delimited (the default) and JSON. Each has its pros and cons.

Delimited Format:

• Simple and compact, easy to read if you’re just scanning logs.
• Requires more careful parsing in automated systems since fields are separated by spaces and their order matters.
• Adding or removing fields can break existing parsers if they rely on fixed positions.

JSON Format:

• Logs are structured as JSON objects, making them easier to parse and process programmatically.
• Flexible—adding or removing fields won’t break your parsing tools, as long as you handle missing keys.
• Works well with modern log processors, ELK stack, or cloud-native tools that natively understand JSON.

Here’s a sample JSON log entry:

{
  "srcaddr": "10.0.1.5",
  "dstaddr": "10.0.2.10",
  "srcport": 443,
  "dstport": 32768,
  "protocol": 6,
  "action": "ACCEPT",
  "bytes": 1024
}

Choosing between the two depends on your tooling and processing needs. JSON often simplifies automation and integration, but might have a slightly larger storage footprint.

Multi-Format Strategy

For advanced monitoring setups, it’s common to use multiple VPC Flow Logs with different formats and destinations simultaneously. This lets you balance performance, detail, and cost:

• Real-time alerts: Use a minimal, compact log format with only essential fields, pushing to a fast, real-time processing system.
• Forensic analysis: Maintain a detailed, comprehensive log format sent to a storage or SIEM system for deep investigation.
• Cost monitoring: Use a lean format focused on data transfer and service details to help optimize your cloud spending.

You can create multiple flow logs on the same VPC and assign different formats and destinations to each, tailoring them for different teams or tools.

Field Ordering for Performance

The order of fields in your VPC log format isn’t just cosmetic— it can affect how quickly your logs get parsed and filtered downstream.

Put your most frequently queried fields at the start of the log line. For example:

${action} ${srcaddr} ${dstaddr} ${dstport} ${bytes} ${start}

If your queries often filter on connection action (ACCEPT, REJECT), putting ${action} first helps parsers quickly skip irrelevant entries, speeding up processing and alerting.

How to Optimize VPC Log Format for Different Destinations

Different places you send your VPC logs have their quirks and costs. Adjusting your log format for each destination helps keep things efficient and avoids unnecessary bills or slowdowns.

CloudWatch Logs Optimization

CloudWatch Logs works best with simpler, text-based log formats. A minimal set of fields makes querying faster and reduces storage costs. For example:

${srcaddr} ${dstaddr} ${action} ${bytes} ${start}

This format gives you the essentials to track who’s talking to whom, what happened, and how much data moved. CloudWatch Logs Insights can run queries efficiently on this, making it ideal for alerting and troubleshooting without bloating your bill.

S3 Analytics Optimization

When you send VPC logs to S3, you’re usually storing them for batch analysis later. This allows for more detailed log formats because the processing isn’t real-time:

${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${bytes} ${packets} ${start} ${end} ${action} ${tcp-flags} ${flow-direction}

Using a detailed format helps you run in-depth queries or audits with tools like AWS Athena.

For better storage efficiency and faster queries, consider converting logs to Parquet format once stored in S3. Parquet compresses data well and is optimized for analytical queries.

Kinesis Firehose Optimization

Kinesis Firehose streams logs in real-time, so there’s a trade-off between detail and throughput.

A balanced format looks like this:

${srcaddr} ${dstaddr} ${dstport} ${action} ${bytes} ${start}

This captures enough information for real-time alerting and monitoring without overwhelming your stream processor or causing delays.

VPC Log Format Performance Benchmarks

Knowing how your choice of VPC log format affects performance and cost can help you plan better. Here’s a rough comparison of three common formats:

These numbers can change depending on your traffic volume and which fields you include, but they offer a useful starting point to understand trade-offs between detail, processing speed, and cost.

Processing Performance Tips

• Choosing the right fields matters more than just how many you include. For example, ${bytes} is easy to process, but ${tcp-flags} needs more work from your logging system.
• Keeping the order of fields consistent helps parsers work faster and makes it easier for everyone on your team to handle the logs.
• Skip extra timestamp fields if you’re not using them for detailed timing or performance analysis—they just add unnecessary processing overhead.

Parsing Custom VPC Log Formats Programmatically

When you customize your VPC log format, you control which fields appear and in what order. To work with these logs programmatically, your parsing logic needs to match the format you defined.

A common approach is to first extract the list of fields from your format string (for example, ${srcaddr} ${dstaddr} ${action} ${bytes}), then split each log line by spaces and map each value to the corresponding field.

Most programming languages—like Python, JavaScript, or Go—can handle this easily by:

1. Extracting field names from the format string
2. Splitting log lines into values
3. Creating a dictionary or object that pairs fields with values

This method keeps your parser flexible and ready to handle changes in your log format without hardcoding field positions.

Common VPC Log Format Mistakes and How to Avoid Them

Over-Engineering Your Format

It’s tempting to include every possible field “just in case,” but that usually leads to bloated logs and higher costs. Start with a minimal set—around 5 to 7 carefully chosen fields—and add more only as you identify real needs.

Inconsistent Field Ordering

When different teams use different field orders, parsing becomes a headache. Set a standard order that everyone follows, like this:

• Core identifiers (srcaddr, dstaddr)
• Port info (srcport, dstport)
• Protocol and action
• Volume metrics (bytes, packets)
• Timing (start, end)
• Advanced fields (tcp-flags, AWS service fields)

Having a shared format makes automation and troubleshooting smoother.

Ignoring Destination Capabilities

Each log destination handles data differently. Using a complex format for CloudWatch Logs or a minimal one for S3 analytics doesn’t play to their strengths. Tailor your log format to the tools you’re sending data to.

Not Planning for Growth

Your logging needs will evolve. Make sure your parsing and processing systems can handle changes in log formats without breaking. Planning for flexibility upfront saves headaches later.

Conclusion

Customizing your VPC log format helps you focus on the data that matters, reducing costs, speeding up processing, and cutting down on irrelevant alerts. Using a platform designed for custom formats makes the whole process easier.

Last9 lets you register your log schemas, create focused dashboards, and set up meaningful alerts—all to help you get better visibility into your network traffic without extra hassle.

Book some time with us to know more, or get started for free today!

FAQs

Q: Can I change my custom VPC log format after creating the flow log? A: No, you can't modify an existing flow log's format. You'll need to create a new flow log with the desired format and delete the old one. Plan your formats carefully.

Q: What's the most cost-effective VPC log format for basic monitoring? A: For basic monitoring, use ${srcaddr} ${dstaddr} ${dstport} ${action} ${bytes}. This gives you essential connectivity and volume information while minimizing storage costs.

Q: How do I test a custom VPC log format before deploying it? A: Create the flow log in a test environment first, generate some traffic, and verify the logs contain the expected fields. Always test parsing with your analysis tools before production deployment.

Q: Can I use the same custom format across multiple AWS regions? A: Yes, but you'll need to create separate flow logs in each region. The format definition itself can be identical across regions.

Q: Which fields should I prioritize for security monitoring? A: Focus on ${srcaddr}, ${dstaddr}, ${srcport}, ${dstport}, ${protocol}, ${action}, and ${flow-direction}. Add timing fields if you need to correlate with other security events.