What Is a Logging Formatter and Why Use One?

Logs play a crucial role in DevOps and software development, especially when troubleshooting issues. However, raw, unformatted logs can quickly become overwhelming and difficult to navigate. This is where logging formatters help by turning messy log entries into clear, structured data, making it easier to pinpoint problems.

In this guide, we’ll cover everything you need to know about logging formatters—how they work, why they matter, and tips for implementing them effectively in your workflow.

What Is a Logging Formatter?

A logging formatter defines how your log messages appear when they're output. Think of it as the template that structures your log data – determining what information gets included and how it's presented.

When your application generates logs, the formatter controls:

• The timestamp format
• Where the log level appears
• How the actual message is displayed
• What contextual data gets included
• The overall layout and organization

Without a formatter, logs tend to be inconsistent and harder to parse, both for engineers and automated tools.

Why Logging Formatters Matter for DevOps Teams

As a DevOps engineer, you're often the first responder when systems misbehave. Well-formatted logs can mean the difference between a quick fix and hours of frustrating investigation.

Good log formatting delivers several benefits:

• Faster troubleshooting: When logs follow a consistent pattern, you can spot anomalies quicker
• Better searchability: Structured logs make it easier to filter and find exactly what you need
• Improved monitoring: Well-formatted logs integrate better with observability tools
• Easier automation: Consistent log formats enable better parsing for automated analysis

Essential Elements Found in Most Logging Formatters

Most logging formatters include some combination of these elements:

How Different Languages Handle Default Logging Formatters

Most programming languages have built-in logging libraries with default formatters. Let's look at some examples:

Understanding Python's Default Logging Formatter

Python's logging module comes with a default formatter that produces output like:

ERROR:root:Database connection failed

The default format string behind this is:

'%(levelname)s:%(name)s:%(message)s'

This basic formatter includes just the log level, logger name, and message – without timestamps or other contextual info.

When you create a logger without specifying a formatter, Python uses this minimal default. If you want timestamps included (which most people do), you need to create a formatter explicitly:

import logging

# Create a basic logger
logging.basicConfig(
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    level=logging.INFO
)

# Now your logs will include timestamps
logging.info("This is a test message")
# Output: 2023-04-15 14:32:17,345 - root - INFO - This is a test message

How Python Format Strings Work with Logging

Python's logging formatter uses a string with placeholders that get replaced with actual values when logging. These placeholders follow this pattern: %(name)s where:

• name is the attribute name from the LogRecord object
• s is the formatting specifier (similar to string formatting)

You can use formatting specifiers to control width, alignment, and precision:

# Right-aligned level name in 8-character field
'%(asctime)s - %(levelname)8s - %(message)s'
# Output: 2023-04-15 14:32:17,345 -     INFO - This is a test message

# Left-aligned level name in 8-character field
'%(asctime)s - %(levelname)-8s - %(message)s'
# Output: 2023-04-15 14:32:17,345 - INFO     - This is a test message

All Available LogRecord Attributes You Can Use

The LogRecord object in Python contains numerous attributes that you can include in your format string:

You can use any of these attributes in your format string to customize your logs.

Other Language Logging Systems: Node.js and Java

Node.js/JavaScript Logging Formatters

In Node.js applications, many developers use libraries like Winston or Bunyan. Winston's default format looks something like:

info: User logged in successfully {"timestamp":"2023-04-15T14:32:17.345Z","user":"john"}

Java Logging with Log4j Formatters

Java's popular Log4j library uses patterns like:

2023-04-15 14:32:17,345 [main] ERROR com.example.App - Failed to connect to database

How to Build Your Custom Logging Formatters

Off-the-shelf formatters rarely meet all your needs. That's why customizing them is such a valuable skill.

Step-by-Step: Creating a Custom Python Formatter

Let's look at a simple custom formatter in Python:

import logging

# Create a custom formatter
formatter = logging.Formatter(
    '%(asctime)s | %(levelname)8s | %(name)s | %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

# Create handler and set the formatter
handler = logging.StreamHandler()
handler.setFormatter(formatter)

# Add the handler to the logger
logger = logging.getLogger(__name__)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)

# Now logs will appear with our custom format
logger.info("This is a test message")

This creates logs that look like:

2023-04-15 14:32:17 |     INFO | my_module | This is a test message

Notice how we used:

• Fixed-width formatting for the level name (%(levelname)8s)
• Pipe separators for better visual parsing
• Custom date formatting

Step-by-Step Process to Create JSON Logs for Machine Processing

For machine-parseable logs, JSON formatters are incredibly useful:

import logging
import json
from datetime import datetime

class JsonFormatter(logging.Formatter):
    def format(self, record):
        log_record = {
            "timestamp": datetime.fromtimestamp(record.created).isoformat(),
            "level": record.levelname,
            "name": record.name,
            "message": record.getMessage(),
            "path": record.pathname,
            "line": record.lineno
        }
        
        if hasattr(record, 'props'):
            log_record.update(record.props)
            
        return json.dumps(log_record)

This produces logs like:

{"timestamp":"2023-04-15T14:32:17.345","level":"ERROR","name":"app.auth","message":"Login failed","path":"/app/auth.py","line":42,"user_id":123}

JSON logs are perfect for shipping to centralized logging systems since they're easily parsed and indexed.

You can use this JSON formatter in your Python application like this:

import logging
from json_formatter import JsonFormatter  # Import the class we defined above

# Create logger
logger = logging.getLogger("myapp")
logger.setLevel(logging.DEBUG)

# Create handler
handler = logging.StreamHandler()
handler.setLevel(logging.DEBUG)

# Add JsonFormatter to handler
formatter = JsonFormatter()
handler.setFormatter(formatter)

# Add handler to logger
logger.addHandler(handler)

# Now your logs will be in JSON format
logger.info("User logged in", extra={"props": {"user_id": 42, "ip": "192.168.1.1"}})

Alternative JSON Logging Method Using Standard Library

Python's standard library also provides tools to create JSON logs without custom classes:

import logging
import json

class CustomAdapter(logging.LoggerAdapter):
    def process(self, msg, kwargs):
        extra = kwargs.get('extra', {})
        extra.update(self.extra)
        kwargs['extra'] = extra
        return msg, kwargs

# Create a logger
logger = logging.getLogger("myapp")
logger.setLevel(logging.INFO)

# Create a handler with default formatter
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter('%(message)s'))
logger.addHandler(handler)

# Create adapter with extra fields that should be in every log entry
adapter = CustomAdapter(logger, {
    "service": "auth-service",
    "environment": "production"
})

# Log using JSON
def log_json(level, message, **kwargs):
    log_data = {
        "message": message,
        **kwargs
    }
    getattr(adapter, level)(json.dumps(log_data))

# Usage
log_json("info", "User logged in", user_id=42, ip="192.168.1.1")

Popular Formatter Patterns With Examples

Let's look at some common formatter patterns and what they produce:

Everyday Formatter Styles You Can Use

Standard Formatter with Timestamp

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

Output:

2023-04-15 14:32:17,345 - myapp - ERROR - Database connection failed

Compact Formatter for Console

formatter = logging.Formatter('%(levelname).1s %(asctime)s %(message)s')

Output:

E 2023-04-15 14:32:17,345 Database connection failed

Detailed Formatter for Debugging

formatter = logging.Formatter('%(asctime)s | %(process)d | %(threadName)s | %(name)s | %(levelname)s | %(filename)s:%(lineno)d | %(message)s')

Output:

2023-04-15 14:32:17,345 | 12345 | MainThread | myapp | ERROR | app.py:42 | Database connection failed

Web Server Style Formatter

formatter = logging.Formatter('%(asctime)s %(name)s[%(process)d]: [%(levelname)s] %(message)s')

Output:

2023-04-15 14:32:17,345 myapp[12345]: [ERROR] Database connection failed

Syslog Style Formatter

import time
formatter = logging.Formatter('%(asctime)s %(name)s %(levelname)s: %(message)s', datefmt='%b %d %H:%M:%S')

Output:

Apr 15 14:32:17 myapp ERROR: Database connection failed

Advanced Techniques for Power Users

Once you've mastered the basics, you can employ some more advanced techniques.

Color-Coded Console Logs for Better Readability

Color-coding logs by severity level makes them much easier to scan:

import logging
from colorama import Fore, Style, init

# Initialize colorama
init()

class ColorFormatter(logging.Formatter):
    COLORS = {
        'DEBUG': Fore.BLUE,
        'INFO': Fore.GREEN,
        'WARNING': Fore.YELLOW,
        'ERROR': Fore.RED,
        'CRITICAL': Fore.RED + Style.BRIGHT
    }
    
    def format(self, record):
        log_message = super().format(record)
        return f"{self.COLORS.get(record.levelname, '')}{log_message}{Style.RESET_ALL}"

Adding Context to Logs with MDC (Mapped Diagnostic Context)

MDC allows you to attach context to logs across your application:

import logging
import threading

class ContextFilter(logging.Filter):
    """
    This filter adds contextual information to log records.
    """
    def __init__(self):
        super().__init__()
        self.local = threading.local()
        
    def set_context(self, **kwargs):
        if not hasattr(self.local, 'context'):
            self.local.context = {}
        self.local.context.update(kwargs)
        
    def filter(self, record):
        if hasattr(self.local, 'context'):
            for key, value in self.local.context.items():
                setattr(record, key, value)
        return True

This lets you add user IDs, request IDs, and other context to all logs within a request:

context_filter = ContextFilter()
logger.addFilter(context_filter)

# Later, when handling a request
context_filter.set_context(request_id='abc-123', user='john')
logger.info("Processing user request")  # Will include the context

Best Practices for Log Formatting Success

Follow these tips to get the most out of your logging formatters:

1. Always Include Timestamps with Timezone

Always include timestamps with timezone information (preferably UTC) to avoid confusion when tracking down issues across distributed systems.

formatter = logging.Formatter('%(asctime)s.%(msecs)03dZ | %(levelname)s | %(message)s', 
                             datefmt='%Y-%m-%dT%H:%M:%S')

2. Keep Formats Consistent Across Services

When working with microservices, use consistent log formats across all services. This makes correlation much easier when troubleshooting.

3. Add Request IDs for Request Tracing

For web applications, include a unique request ID in all logs related to that request:

@app.middleware("http")
async def add_request_id(request, call_next):
    request_id = str(uuid.uuid4())
    context_logger = logging.LoggerAdapter(
        logger, {"request_id": request_id}
    )
    request.state.logger = context_logger
    response = await call_next(request)
    return response

4. Balance Detail and Readability

Include enough information to be useful without making logs so verbose they become hard to read. Consider having different formatters for different environments:

• Development: Colorized, human-readable formats
• Production: JSON or structured formats for machine parsing

5. Never Log Sensitive Information

Be careful not to log passwords, tokens, or personally identifiable information (PII). Consider creating a sanitizer for your formatted:

def sanitize_log(message):
    # Remove passwords, tokens, etc.
    sanitized = re.sub(r'password["\']?\s*[:=]\s*["\']?([^"\']+)["\']?', 
                       r'password=*****', message)
    return sanitized

Integrating Formatters with Observability Tools

Your logs are most valuable when they feed into your broader observability strategy.

Using Formatters with Log Aggregation Systems

Well-formatted logs make integration with log aggregation systems much easier. Here's how to optimize your formatters for common scenarios:

Sending Logs to a Centralized System

When sending logs to a centralized system, structure is key:

import logging
import socket
import json

class StructuredFormatter(logging.Formatter):
    def format(self, record):
        log_data = {
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
            "hostname": socket.gethostname(),
            "service": "user-service",  # Your service name
            "environment": "production"  # Your environment
        }
        
        # Add exception info if present
        if record.exc_info:
            log_data["exception"] = self.formatException(record.exc_info)
            
        # Add any extra fields
        for key, value in record.__dict__.items():
            if key not in ["args", "asctime", "created", "exc_info", "exc_text", 
                          "filename", "funcName", "id", "levelname", "levelno",
                          "lineno", "module", "msecs", "message", "msg", "name", 
                          "pathname", "process", "processName", "relativeCreated", 
                          "stack_info", "thread", "threadName"]:
                log_data[key] = value
                
        return json.dumps(log_data)

This produces highly enriched, structured logs ready for ingestion by systems like Elasticsearch, Loki, or other aggregation tools.

Unified Observability with Last9

If you're looking for a managed observability solution that's kinder to your budget without compromising performance, Last9 is worth checking out. Our telemetry data platform seamlessly integrates with your logging stack, bringing together metrics, logs, and traces for a complete observability picture.

Last9 has handled some of the largest live-streaming events in history and works natively with OpenTelemetry and Prometheus data. This means your carefully formatted logs can be correlated with metrics and traces for faster troubleshooting.

Structured Logging for Better Analysis

When designing your formatters, consider how they'll integrate with your analysis tools:

# Adding structured data to logs
logger.info("User purchase completed", extra={
    "user_id": user.id,
    "item_id": item.id,
    "amount": amount,
    "currency": "USD"
})

This structured approach makes it much easier to:

• Filter logs based on specific fields
• Create dashboards based on log data
• Set up alerts based on log patterns

When to Update Your Logging Format

Your logging needs will evolve as your applications grow. Consider updating your formatters when:

• You're adding new services that need to be correlated with existing ones
• Engineers are struggling to find the information they need in logs
• You're integrating with new monitoring or observability tools
• Performance issues arise from overly verbose logging

Common Mistakes to Avoid With Logging Formatters

Even experienced DevOps engineers make these mistakes:

1. Inconsistent timestamp formats across services
2. Logging too much information, creating noise
3. Logging too little context to be useful
4. Hardcoding formatter settings instead of making them configurable
5. Using different formatters across environments, making it hard to transition from development to production debugging

Troubleshooting Formatter Problems

When your logging formatter isn't working as expected:

1. Test with a simple standalone script
2. Check for configuration overrides in different parts of your application
3. Verify the logger hierarchy and inheritance
4. Look for encoding issues, especially with JSON formatters
5. Check for performance impacts with complex formatters

Conclusion

A well-designed logging formatter can greatly improve troubleshooting and observability. By following the practices in this guide, you'll create logs that are both human-readable and machine-parseable.

However, logging is just one part of a complete observability strategy. When combined with metrics and traces, formatted logs give you a more comprehensive view of your system's health. Last9 helps you achieve this without compromising on performance or unexpected costs.

Talk to us to know more about the platform's capabilities!

FAQs

What's the difference between a formatter and a handler?

A formatter controls how log messages look, while a handler determines where logs go (console, file, remote server, etc.). They work together – handlers use formatters to structure the output before sending it to its destination.

Should I use different formatters for different log levels?

It's generally better to use a consistent format for all levels, but you might want to include additional context for error and critical logs. Instead of different formatters, consider adding context to those specific log calls.

How do formatters affect application performance?

Complex formatters (especially ones that do JSON conversion or heavy string processing) can impact performance in high-throughput applications. Consider benchmarking your formatter if you're logging thousands of messages per second.

Can I have different formatters for different outputs?

Absolutely! It's common to have human-readable formats for console output and machine-parseable formats (like JSON) for files or remote logging.

What's the best formatter for container environments?

For containerized applications, JSON logging to stdout/stderr is generally the best practice. Container orchestration systems can then collect these logs and forward them to your observability platform.

What's the best logging formatter for microservices?

JSON formatters with correlation IDs are ideal for microservices, as they allow you to trace requests across multiple services while keeping logs machine-parseable.