Access Logs: Format Specification and Practical Usage

Your server's been logging everything—it’s just easy to overlook until something breaks.

Every incoming request, database call, or auth check ends up in your access logs. They’re not flashy, but they quietly document every interaction your system handles. For developers, they’re often the most reliable starting point when things go wrong.

In this blog, we'll take a look at what an access log is, its format, types, and a few best practices.

What Exactly Is an Access Log?

An access log is a structured text file that records every incoming request to your server or application. Each log entry captures metadata about the request, such as the client IP address, HTTP method, path, status code, response size, referrer, and user agent.

Here’s a standard access log entry from a web server:

192.168.1.100 - - [18/Jun/2025:10:30:45 +0000] "GET /api/users/123 HTTP/1.1" 200 1024 "https://example.com/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

This single line tells you:

• Client IP: 192.168.1.100
• Timestamp: [18/Jun/2025:10:30:45 +0000]
• HTTP Method and Path: GET /api/users/123
• Protocol: HTTP/1.1
• Status Code: 200
• Response Size: 1024 bytes
• Referrer: https://example.com/dashboard
• User Agent: The client’s browser or tool

Access Logs Categories and What They’re Used For

Depending on what you’re tracking, they usually fall into three categories:

1. Activity Logs

These logs capture user-level actions inside your application—clicks, commands, page transitions, API calls made from the frontend, etc. They're especially useful when you need to understand how users interact with your system.

Examples:

• Application audit logs
• CLI command histories
• API usage patterns tied to user sessions

Use these when you’re tracing a user journey or debugging session-specific issues like unexpected state changes or permission errors.

2. Server Access Logs

These are the classic logs from your web servers, proxies, or databases. They capture high-level request metadata—who connected, what they asked for, and how the system responded. You won’t get user behavior details, but you will see patterns in traffic, latency, and errors.

Examples:

• Nginx/Apache access logs
• Load balancer logs (e.g., ELB)
• FTP and DB connection logs

Good for tracking request volumes, identifying abusive clients, or analyzing how upstream services are behaving.

3. Error Logs

These logs capture application or system-level failures: stack traces, exception messages, failed DB queries, timeouts, etc. They’re typically written by the app or service itself, separate from access logs, but when used together, they give a more complete picture of what went wrong.

Examples:

• Application crash reports
• Web server error logs
• Database query errors

If something’s failing silently, this is often where you’ll find the cause, especially when paired with timestamps from access logs.

Access Logs You’ll Work With

Access logs show up in different parts of your stack, each with a slightly different purpose. Some tell you who hit your server, others show what your app or database did in response.

Here’s a breakdown of the kinds you’ll run into most often:

1. Web Server Logs – The Front Door

This is where every HTTP request lands—your first line of visibility. Web servers like Apache, Nginx, or IIS log every incoming request: IP address, method, path, status code, response size, user agent, referrer—all timestamped.

• Apache uses the Combined Log Format
• Nginx lets you define custom formats with log_format

Great for catching traffic spikes, 4xx/5xx storms, or figuring out which routes are getting hammered.

2. Application Logs – What Your Code Sees

Frameworks like Express, Django, or Rails log at the app level, so you get a bit more context.

• Express uses morgan for basic HTTP logging
• Django has built-in middleware for logging
• Rails logs controller actions, request IDs, and parameters

These logs are usually closer to business logic. If a user action triggers a bunch of internal steps, this is where you’ll see them show up.

3. Database Logs – Who’s Asking What, and How Often

Most databases can log incoming connections, queries, and errors. Super handy when a rogue query is slowing everything down, or when a service is opening far too many connections.

• PostgreSQL and MySQL can log query times, slow queries, and failed logins
• Helpful for tuning indexes or spotting inefficient query patterns

You won’t look at these every day, but when something’s slow and you’re out of guesses, this is where to go.

4. Cloud Provider Logs – API Calls and Infra Access

If you’re in AWS, GCP, or Azure, chances are you’ve seen logs like CloudTrail or GCP Audit Logs. These track actions like spinning up an EC2 instance, hitting a cloud API, or someone fiddling with IAM policies.

• AWS: CloudTrail
• GCP: Audit Logs
• Azure: Activity Logs

These are more infra-facing, but they’re critical for security audits or when things suddenly break due to a config change no one remembers making.

What Each Field in an Access Log Means

Access logs are structured records. Once you know what each part means, they start telling real stories about what’s happening in your system.

Here’s a breakdown of the most common fields you’ll run into:

Let’s Break Down a Log Entry

203.0.113.42 - admin [21/Jun/2025:14:22:30 +0000] "POST /api/orders HTTP/1.1" 201 856 "https://shop.example.com/cart" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

Here’s what’s going on in simple terms:

• IP 203.0.113.42 made a request
• The authenticated user was admin
• It happened at 2:22 PM UTC on June 21st, 2025
• They sent a POST to /api/orders using HTTP/1.1`
• The server responded with a 201 Created and returned 856 bytes
• The request originated from the cart page
• The client was using a Windows browser (Chrome)

That’s a lot of insight from one line. Once you’re used to reading these, they’re a quick way to debug failed calls, trace traffic, or see what a user (or bot) has been up to.

Where to Look for Access Logs

Access logs aren’t always in the same place—it depends on what stack you're using and how your environment is set up. But some common defaults’ll help you get started.

Apache

If you're running Apache, log files usually land here:

• RHEL/CentOS/Fedora: /var/log/httpd/access_log
• Ubuntu/Debian: /var/log/apache2/access.log

Apache also lets you define custom log paths per virtual host, so in production, it’s common to see different log files for different domains or services. Worth checking the site config if things aren't where you'd expect.

Nginx

Nginx is more consistent across distros:

• Default location: /var/log/nginx/access.log

If you’re using multiple server blocks, you might see separate logs for each block, depending on how your access_log directives are configured.

IIS (Windows)

For IIS on Windows, logs live under:

• C:\inetpub\logs\LogFiles\W3SVC[site_id]\

Each IIS site has a numeric site ID. You can find it in the IIS Manager or by running appcmd list site from the command line. Each site’s logs go into its folder.

App-Level Logs

Some frameworks and applications keep their logs separate from the web server.

• Linux: usually something like /var/log/<app_name>/
• Windows: often inside the application’s install or config directory

Where exactly these live depends on the framework—Express apps with morgan, for instance, may log to stdout unless you explicitly direct output to a file. Always check the app’s config or logging middleware.

Common Access Log Formats and How to Configure Them

Access logs follow specific formats that tell your server what to log and how to structure it. You’ve probably seen the two most common ones in the wild: Common Log Format and Combined Log Format.

Common vs Combined Log Format

The Common Log Format (CLF) gives you the essentials—good enough for basic auditing:

%h %l %u %t "%r" %>s %b

The Combined Log Format builds on that by adding user agent and referrer data super useful for debugging client behavior or tracing where requests came from:

%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"

Here’s what each piece means:

Configuring Access Logs in Apache and Nginx

To use one of these formats, you configure it in your web server.

Apache

In Apache, the log format lives in your main config file—either httpd.conf or apache2.conf, depending on your distro.

# Server-wide access log using Combined format
CustomLog "/var/log/httpd/access_log" combined

If you’re running multiple sites, you can set different log paths per virtual host:

<VirtualHost *:80>
    ServerName mysite.com
    CustomLog "/var/log/httpd/mysite_access.log" combined
</VirtualHost>

Nginx

Nginx lets you define named log formats and then apply them wherever you need:

http {
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent"';

    access_log /var/log/nginx/access.log main;
}

You can tweak this format to include things like request times, upstream info, or any custom header you're interested in.

Controlling What Gets Logged with Log Levels

While access logs capture request traffic, you’ll also want to manage log verbosity, especially when it comes to error logs.

In Apache, you can set a LogLevel to control how much noise makes it into your logs:

LogLevel warn

Here are the standard Apache log levels, from most to least verbose:

• debug – All the internal details
• info – General runtime info
• notice – Normal but significant events
• warn – Warning conditions
• error – Standard application errors
• crit – Critical failures
• alert – Requires immediate attention
• emerg – System is unusable

If your logs are getting too noisy in production, drop down to warn or error usually hits the sweet spot—enough signal, not too much noise.

Why Access Logs Deserve More of Your Attention

Access logs aren’t just something you glance at during a postmortem. They’re quietly recording how your application behaves—every request, every user interaction, every odd spike. If you know what to look for, they’re a goldmine of insights.

Spot Performance Issues Early

Access logs can show you which endpoints are slow, what times of day your servers get hammered, or whether that new feature rollout is adding load in unexpected places.

No extra instrumentation needed—just parsing the logs gives you enough to answer questions like:

• Are response times creeping up over time?
• Do certain paths (like /checkout or /search) regularly spike under load?
• Is latency tied to specific regions or clients?

This kind of data is great for catching problems before users start complaining.

Catch Security Red Flags

You don’t need a full-blown SIEM to notice something’s off. Access logs can tip you off when things get weird:

• Lots of 404s from the same IP? Could be a bot crawling for vulnerabilities.
• Suspicious user agents or unusual paths? Might be an automated scanner.
• Repeated failed login attempts or brute-force patterns? Worth flagging.

It’s basic, but effective—especially in environments where you’re not running heavyweight security tools.

Mine Useful Business Insights

Not everything in access logs is backend noise. Product and marketing teams can pull real value from it, too:

• Which pages get the most traffic?
• Where are users bouncing off in your flow?
• When are your peak usage hours?

You can even break things down by geo, device type, or referrer to get a better sense of who’s using what, and when. Pair that with conversion data or feature flags, and access logs become a low-cost BI layer.

Using Access Logs for Deeper Security Monitoring

Access logs are often the first place where signs of trouble show up. Before your security tools raise alerts or before users even notice your logs are quietly recording the patterns that matter. Here's what to watch for:

Request Spikes and DDoS Patterns

Sudden bursts of traffic from a single IP or IP range, especially hitting the same endpoint over and over, is a classic DDoS signature. If /login or /search is getting hammered non-stop, it's time to dig deeper.

Directory Traversal and Recon Attempts

You’ll often see a stream of 404s from the same IP trying things like:

/admin/
/etc/passwd
/config.php

This usually means a bot is probing your server, scanning for known paths or misconfigurations. These attempts might not break anything today, but they’re a warning sign.

Brute Force Login Behavior

Access logs will show repeated POST requests to your login or auth endpoints, often with no referrer and different usernames. If the failure rate is high and clustered by IP, you’ve probably got a brute force attempt in progress.

Bot and Crawler Detection

Not all bots are malicious, but if you're seeing requests with odd user agents, no referrer, or patterns like:

1. Hit robots.txt
2. Crawl every link on the page
3. Repeat with slight variations

…it’s worth separating those requests from human traffic. It helps reduce noise and tightens your monitoring.

SQL Injection Attempts

Keep an eye out for suspicious query strings with keywords like:

?id=1;DROP TABLE users
?q=SELECT+*+FROM

These often show up in URLs or form submissions and can lead to 500s or large, weird-looking responses if your app isn't filtering inputs properly.

Even basic pattern matching or rate analysis on your access logs can give you early warnings. Once you spot something odd, you can act, rate-limit the IP, block it entirely, or step up auth requirements for sensitive endpoints.

Managing Access Logs Before They Take Over Your Server

Access logs can quietly balloon out of control. On a high-traffic API or web service, they can generate hundreds of MBs or even GBs per day. If you don’t actively manage them, they’ll eat disk space, slow I/O, and eventually crash things you care about.

Log Rotation:

Most Linux distros ship with logrotate, and it’s usually configured for services like Nginx or Apache out of the box. Here's what it does:

• Monitors log file size or age (e.g., rotate daily or when a file exceeds 100MB).
• Archives the current log (e.g., access.log.1), then creates a fresh access.log.
• Compresses older logs using gzip (access.log.2.gz, etc.).
• Optionally purges logs after N cycles (rotate 7 keeps 7 generations).

Sample config:

/var/log/nginx/access.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    postrotate
        systemctl reload nginx > /dev/null 2>&1 || true
    endscript
}

This keeps logs manageable while still retaining historical data for audits, security forensics, or ad-hoc analysis.

Centralized Logging for High-Throughput Systems

If you’re logging from multiple services or running a noisy system (e.g., hundreds of requests per second), local log files stop being practical. Instead:

• Stream logs to a collector like the OpenTelemetry Collector, Fluent Bit, or Vector.
• Send them to a remote sink: Elasticsearch, S3, ClickHouse, or a managed log analytics platform like Last9.
• Set up filtering or sampling at the collector level to avoid drowning in noisy or redundant data.

This offloads I/O pressure from your application server and lets you centralize retention, search, and alerting policies.

Don’t Skip Compression and Retention Policies

If you do keep logs locally:

• Always enable compress in your rotation config.
• Use tools like zcat, zgrep, or zless to query compressed logs without unzipping.
• Cap total log size using maxsize or rotate limits.

Tools and Techniques for Access Log Analysis

Once you’ve got your access logs under control, the next step is extracting signal from the noise.

Command-Line One-Liners That Still Hold Up

For many devs, plain-text logs and shell tools are still the fastest way to get answers, especially on production boxes.

# Find all 500 errors
grep ' 500 ' access.log

# Count requests by IP
awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -20

# Top endpoints hit
awk '{print $7}' access.log | sort | uniq -c | sort -nr | head -20

Need timestamps? cut, awk, or even sed can help parse and bucket logs by time. It’s not elegant, but it works—and sometimes that’s what matters in a fire.

Writing Custom Log Parsers

For anything beyond basic grep, a small Python or Go script can make life much easier.

• Use re or str.split() to break down each line.
• Parse timestamps with datetime.strptime().
• Bucket requests, status codes, and response times for basic performance analysis.
• Generate reports, export to CSV, or visualize with matplotlib or Plotly.

For example, you could track:

• Top 10 slowest endpoints (based on 95th percentile latency)
• Requests per minute/hour
• Breakdown of HTTP status codes over time

Log Analysis with Observability Platforms

When logs start scaling out of control—or you want to correlate logs with metrics and traces—you’ll need something more powerful.

Our platform, Last9, is built to handle high-volume, high-cardinality telemetry. You can:

• Stream logs via OpenTelemetry
• Enrich logs with metadata (region, service, pod ID, etc.)
• Correlate errors with spike in latency or dropped traces
• Visualize traffic patterns alongside CPU, memory, or DB latency

Last9 supports OpenTelemetry-native ingestion and works well with Prometheus setups. It’s especially useful if you want full observability without racking up unpredictable costs.

Advanced Access Log Techniques

Structured Logging

Instead of plain text logs, consider structured formats like JSON. They're easier to parse, search, and analyze programmatically. You can include nested data, arrays, and complex objects that would be difficult to represent in traditional formats.

Correlation IDs

Add unique request IDs to correlate log entries across different services. When a request flows through multiple microservices, correlation IDs let you trace the entire journey and understand how different components interact.

Sampling for High-Volume Systems

If you're processing millions of requests daily, logging everything might not be practical. Implement sampling to log a representative subset of requests while reducing storage costs and processing overhead.

Troubleshooting Common Access Log Issues

Below are common issues and what to check, especially in production environments.

1. Missing Log Entries

If requests you know are happening aren't showing up in your logs, consider these causes:

• Incorrect Configuration Path: Confirm that your application or web server is writing logs to the correct file. In Nginx and Apache, misconfigured access_log directives can silently discard logs.
• Logging Disabled: Ensure logging isn't turned off globally or per-virtual host. Apache’s CustomLog or Nginx’s access_log may be commented out or redirected to /dev/null.
• Permissions Issues: The logging process must have write access to the destination directory. Check file and directory ownership (ls -l and stat) and use strace to detect permission errors.
• Buffering or Delayed Writes: Some logging libraries and platforms buffer writes for performance. If logs appear only intermittently, check for buffering settings in the logger (e.g., flush_interval in fluent-bit, auto_flush in some Python logging setups).

2. Timestamp and Timezone Discrepancies

Timestamps are critical for correlating logs across systems. Inconsistent times make tracing incidents much harder.

• Standardize to UTC: Most teams use UTC in production to simplify correlation. In Apache, set LogFormat to include timezone-aware timestamps (e.g., %t includes [day/month/year:hour:minute:second zone]).
• Align Time Sync Across Nodes: Use ntpd or chronyd to sync clocks across servers. Drifted clocks can cause log entries to appear out of order in distributed systems.
• Rotation and Timezone Mismatches: Tools like logrotate use local time unless explicitly told to use UTC. Set the dateext option with %Y-%m-%dT%H:%M:%SZ format to preserve correct timezone handling.

3. Log Volume and Performance Impact

Logging isn’t free. Writing too much or logging synchronously can degrade performance, especially under load.

• Asynchronous Logging: If you're using frameworks like Java (Log4j2 async appenders), Python (with QueueHandler), or Go (channel-based logging), ensure that your log writers are async. This prevents blocking on I/O.
• Selective Logging: You may not need to log every HTTP 200 in high-traffic paths. Consider sampling or filtering logs at the edge (e.g., using Envoy, HAProxy, or nginx’s log_not_found off;).
• Log to DevNull in Dev/Test: When testing throughput, disable access logging entirely with access_log off; in Nginx, or CustomLog /dev/null common in Apache.

Log Levels and Their Impact

Different logging levels provide varying granularity and have different performance footprints:

When possible, log levels should be configurable at runtime or reloadable without restarting the service (e.g., using SIGHUP with Nginx or runtime flags in Java apps).

Tip: Verify Your Logs

• Use tail -f or multitail to live-inspect logs and verify real-time logging.
• Pipe logs through logger or rsyslog to test if systemd or syslog-based ingestion is working as expected.
• For apps using structured logging (JSON, LTSV), use jq, grep, or vector to validate schema correctness and extract missing fields.

Access Log Security and Privacy Considerations

Access logs can expose more than you think. They often contain IP addresses, request parameters, and user identifiers—data that may be subject to privacy regulations like GDPR, CCPA, or internal compliance policies.

Protecting Sensitive Data

• IP Address Anonymization: Mask the last octet (e.g., 192.168.1.xxx) or use a one-way hash to anonymize client IPs when full tracking isn’t required.
• Query Parameters: Avoid logging URLs with sensitive query strings. Passwords, API keys, tokens, or session IDs can accidentally end up in access logs if GET or POST parameters are not sanitized.
• Authorization and Cookie Headers: Filter out or redact headers like Authorization, Set-Cookie, or any custom headers carrying credentials or tokens before logging requests.

Securing Log Access

• Set restrictive file permissions. On Linux, access logs should be owned by root or the application user and readable only by the necessary system users. A typical permission setting is chmod 640.
• Use centralized logging systems that provide access control and audit trails. Only authorized team members should have access to production logs.
• Encrypt archived logs if they’re stored long-term, and ensure backup storage complies with your organization’s security policies.

Best Practices for Access Log Implementation

A solid access log setup doesn't need to be complicated, but it should be intentional. Here are some practical guidelines:

Use Consistent and Parseable Formats

Stick with formats like the Combined Log Format unless you have a specific need for customization. Most analysis tools, log shippers, and monitoring stacks expect a predictable structure.

Log Useful, Not Excessive, Data

Include fields that help with debugging and correlation:

• Trace ID or request ID (especially in distributed systems)
• User ID (anonymized if required)
• HTTP status code, response time, and request method

Avoid logging:

• Internal debug traces in production
• Excessively verbose data or payloads
• Sensitive internal implementation details unless necessary

Monitor Early, Not Just During Incidents

Don’t wait for an outage to start analyzing logs. With regular monitoring in place, you can:

• Establish baselines for latency and traffic
• Detect unusual access patterns or auth failures
• Surface performance regressions before users report them

Validate Parsing Logic

If you're sending logs to tools like fluentd, vector, or writing custom parsers in Python or Go, make sure your setup can handle:

• Malformed or incomplete log entries
• Unexpected characters from bots or external clients
• Variations in log structure across environments

Wrapping Up

Access logs offer a detailed record of request-level activity essential for understanding system behavior, tracking performance bottlenecks, and spotting anomalous patterns.

Last9 helps teams ingest, store, and query high-cardinality log data at scale. With native support for OpenTelemetry and Prometheus, we enable end-to-end visibility so you can trace an outlier request, analyze associated logs, and correlate it with system metrics in one place.

Book a time with us to know more or get started for free today!

FAQs

What's the difference between access logs and error logs?

Access logs record all requests to your server, regardless of success or failure. Error logs specifically capture problems, exceptions, and diagnostic information. You need both for complete visibility into your application's behavior.

How long should I keep access logs?

This depends on your compliance requirements and storage costs. Many organizations keep 30-90 days of detailed logs online, with compressed archives for longer periods. Consider your debugging needs and regulatory obligations when setting retention policies.

Can access logs impact server performance?

Yes, but usually not significantly if configured properly. Synchronous logging to slow disks can create bottlenecks. Use asynchronous logging, fast storage, and consider sampling for high-traffic applications to minimize performance impact.

What's the best format for access logs?

Combined Log Format works well for web servers and is widely supported by analysis tools. For applications, JSON provides more flexibility and easier parsing. Choose based on your analysis tools and specific requirements.

How do I handle access logs in containerized environments?

In containers, you typically log to stdout/stderr and let the container runtime handle file management. Use centralized logging solutions to collect logs from multiple containers and make them searchable across your entire infrastructure.