How to Master Log Management with Logrotate in Docker Containers

Docker containers continuously generate logs during operation, and without proper management, these logs can consume significant disk space, impact system performance, and create operational issues. Logrotate offers an effective solution for managing these logs in containerized environments.

This guide covers the implementation of logrotate in Docker containers – from initial setup through advanced configurations that ensure stable, maintainable container deployments. The techniques provided address common challenges faced in production environments and offer concrete solutions for effective log management.

What is Logrotate and Why Use it with Docker?

Docker containers generate substantial log data during normal operation. These logs come from multiple sources:

Container stdout/stderr: Captured by Docker’s logging drivers
Application-specific logs: Written to files inside the container
Service logs: Generated by components running within the container

Without a structured log rotation strategy, these files grow continuously, potentially causing:

Disk space exhaustion leading to service outages
Degraded I/O performance affecting container operations
Increased backup duration and storage costs
Challenges in log analysis and troubleshooting

Logrotate provides a robust solution by handling log rotation, compression, removal, and mailing of log files. When implemented in Docker environments, it offers:

Configurable retention policies based on size, age, or both
Automated compression to reduce storage requirements
Customizable naming conventions for archived logs
Rotation triggering that can integrate with application signal handling
Scriptable pre/post-processing for logs during rotation events

While Docker does provide basic logging controls through logging drivers (JSON-file, local, syslog, etc.), these mechanisms often lack the granularity and flexibility required for production environments. Logrotate delivers precise control over log lifecycle management with minimal overhead.

💡

For a deeper look at how log rotation works in Linux, check out this guide on setting it up efficiently: Log Rotation in Linux.

Setting Up Logrotate in Docker: Implementation Approaches

Implementing logrotate in Docker environments can be accomplished through several architectural patterns, each with specific advantages depending on your infrastructure requirements. The following methods provide a foundation for integrating log rotation capabilities with containerized applications:

Method 1: Embedding Logrotate in the Application Container

This approach integrates logrotate directly into your application container, creating a self-contained solution where each container manages its logs. This method is particularly useful for stateful applications that maintain persistent logs.

FROM ubuntu:20.04

# Install logrotate and dependencies
RUN apt-get update && apt-get install -y \
    logrotate \
    cron \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

# Configure logrotate with specific settings
COPY my-logrotate.conf /etc/logrotate.d/

# Set up cron to run logrotate hourly
RUN echo '0 * * * * /usr/sbin/logrotate -f /etc/logrotate.conf >/proc/1/fd/1 2>/proc/1/fd/2' > /etc/cron.d/logrotate-cron \
    && chmod 0644 /etc/cron.d/logrotate-cron \
    && crontab /etc/cron.d/logrotate-cron

# Create log directories with appropriate permissions
RUN mkdir -p /var/log/app \
    && touch /var/log/logrotate.status \
    && chmod 644 /var/log/logrotate.status

# Start both cron daemon and application
ENTRYPOINT ["/bin/bash", "-c", "service cron start && exec $@"]
CMD ["cron", "-f"]

This configuration ensures that:

Logrotate runs on an hourly schedule
Rotation status is properly tracked
Cron output is directed to container stdout for visibility
The container has appropriate permissions for log management

Method 2: Sidecar Container Pattern

The sidecar pattern employs a dedicated container that runs alongside your application container, with both sharing access to the same log volume. This approach adheres to the single-responsibility principle, keeping your application container focused on its core function while delegating log management to a specialized container.

version: "3.8"
services:
  app:
    image: your-application:latest
    volumes:
      - logs:/var/log/app
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"

  logrotate:
    image: blacklabelops/logrotate:latest
    volumes:
      - logs:/var/log/app
    environment:
      - LOGS_DIRECTORIES=/var/log/app
      - LOGROTATE_STATUSFILE=/var/log/logrotate/logrotate.status
      - LOGROTATE_INTERVAL=daily
      - LOGROTATE_COPIES=7
      - LOGROTATE_SIZE=50M
      - LOGROTATE_COMPRESSION=compress
      - LOG_FILE_ENDINGS=log
      - LOGROTATE_DATEFORMAT=-%Y%m%d
    restart: unless-stopped
    depends_on:
      - app

volumes:
  logs:
    driver: local

This configuration provides:

Complete separation of concerns between application and log management
Fine-grained control over rotation parameters via environment variables
Automatic restart capability if the logrotate container fails
Shared access to log volumes with proper sequence guarantees

💡

You can also explore how sidecar containers in Kubernetes help with log management and other tasks: Sidecar Containers in Kubernetes.

Method 3: Host-Based Rotation

This method leverages the host system’s existing logrotate infrastructure by mounting container log directories to specified host paths. The approach simplifies container configuration while centralizing log management at the host level.

# Run container with logs mounted to host directory
docker run \
  --name app-container \
  -v /var/log/containers/app1:/var/log \
  -d your-container

# Alternative using docker-compose
# docker-compose.yml excerpt:
# services:
#   app:
#     image: your-container
#     volumes:
#       - /var/log/containers/app1:/var/log

On the host system, create a dedicated logrotate configuration:

# /etc/logrotate.d/docker-containers
/var/log/containers/*/*.log {
    daily
    rotate 7
    size 100M
    compress
    delaycompress
    missingok
    notifempty
    dateext
    dateformat -%Y%m%d
    create 0644 root root
    sharedscripts
    postrotate
        # Optional: notify containers about rotation
        for container in $(docker ps --format '{{.Names}}'); do
            docker kill --signal=USR1 $container >/dev/null 2>&1 || true
        done
    endscript
}

Benefits of this approach:

Centralizes log rotation policy across multiple containers
Utilizes existing host system maintenance workflows
Reduces container complexity and image size
Provides unified log access for host-based monitoring tools

This method works particularly well in environments where:

Host systems are long-lived compared to containers
Operations teams prefer monitoring logs from the host’s perspective
Container images need to be kept minimal

Troubleshooting Logrotate in Docker Environments

When implementing logrotate in containerized environments, several common issues can arise. This section addresses these challenges with specific diagnostic steps and solutions.

Problem: Logrotate Not Executing on Schedule

This issue typically occurs when the container lacks a proper scheduler mechanism or when process management is misconfigured.

Diagnostic Steps:

Examine cron logs for execution attempts:

docker exec -it container-name grep cron /var/log/syslog

Verify crontab configuration:

docker exec -it container-name crontab -l

Check if cron is installed and running in the container:

docker exec -it container-name ps aux | grep cron

Solution: Implement a robust scheduling mechanism with proper logging:

# Add to your Dockerfile
RUN apt-get update && apt-get install -y cron rsyslog

# Configure cron with logging enabled
RUN touch /var/log/cron.log
COPY crontab /etc/cron.d/logrotate-cron
RUN chmod 0644 /etc/cron.d/logrotate-cron
RUN crontab /etc/cron.d/logrotate-cron

# Startup script to ensure services run
COPY start.sh /start.sh
RUN chmod +x /start.sh
CMD ["/start.sh"]

Your crontab file should include verbose logging:

0 0 * * * /usr/sbin/logrotate -v /etc/logrotate.conf >> /var/log/cron-logrotate.log 2>&1

The start.sh script ensures both services run:

#!/bin/bash
service rsyslog start
service cron start
# Keep container running
tail -f /var/log/cron.log

💡

Fix production Docker issues instantly—right from your IDE, with AI and Last9 MCP.
[Setup Last9 MCP] -> [View demo]

Problem: Permission Denied Errors During Rotation

Logrotate requires specific permissions to read, write, and manage log files. In Docker environments, permission issues commonly arise due to user mismatches between the application generating logs and the process rotating them.

Diagnostic Steps:

Examine logrotate error messages:

docker exec container-name logrotate -d -v /etc/logrotate.conf

Check existing log file permissions:

docker exec container-name ls -la /var/log/

Identify the user running your application:

docker exec container-name ps aux

Solution: Implement comprehensive permission management in your Dockerfile:

# Add to your Dockerfile - adjust user/group as needed
RUN groupadd -r appgroup && useradd -r -g appgroup appuser

# Create log directory with appropriate ownership
RUN mkdir -p /var/log/app \
    && touch /var/log/app/app.log \
    && chown -R appuser:appgroup /var/log/app \
    && chmod 755 /var/log/app \
    && chmod 644 /var/log/app/*.log

# Ensure logrotate has access
RUN chown root:root /etc/logrotate.d/* \
    && chmod 644 /etc/logrotate.d/*

# Configure logrotate to create new logs with correct permissions
# In your logrotate config file:
# create 0644 appuser appgroup

For scenarios where the application user differs from the logrotate user, use the su directive in your logrotate configuration:

/var/log/app/*.log {
    daily
    rotate 7
    # Run as specific user
    su appuser appgroup
    # Other directives...
}

Problem: Managing Docker Engine Logs (JSON Log Files)

While application logs inside containers can be managed with logrotate, Docker’s own JSON log files (typically stored in /var/lib/docker/containers/[container-id]/[container-id]-json.log) require a different approach since they’re generated outside the container context.

Diagnostic Steps:

Verify current Docker daemon configuration:

docker info | grep "Logging Driver"
cat /etc/docker/daemon.json

Check current Docker log usage:

sudo find /var/lib/docker/containers -name "*-json.log" -exec ls -sh {} \;

Solution: Implement a dual-layer approach:

Configure container-level log limits in your docker-compose.yml:

services:
  app:
    image: your-application
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "5"
        compress: "true"
        labels: "production_app"
        env: "staging,production"

For system-wide defaults, configure the Docker daemon by editing /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "3",
    "labels": "production",
    "env": "os,customer"
  }
}

After updating this file, restart the Docker daemon:

sudo systemctl restart docker

For existing containers with excessive logs, use:

sudo sh -c 'truncate -s 0 /var/lib/docker/containers/*/*-json.log'
docker restart $(docker ps -q)

This comprehensive approach ensures both internal application logs and Docker’s container logs are properly managed. For production environments, consider advanced solutions like forwarding logs to centralized logging systems (Last9, ELK Stack, Graylog, etc.) using the appropriate log drivers (fluentd, syslog, etc.).

💡

If you’re working with security logs, this guide on SIEM logs breaks down what you need to know: SIEM Logs.

Crafting the Perfect Logrotate Config for Docker

A well-tuned logrotate configuration makes all the difference. Here’s a solid starting point:

/var/log/app/*.log {
    daily
    rotate 7
    size 10M
    missingok
    compress
    delaycompress
    notifempty
    create 0644 root root
    sharedscripts
    postrotate
        kill -USR1 $(cat /var/run/app.pid 2>/dev/null) 2>/dev/null || true
    endscript
}

Let’s break down the key options:

Option	Description	Recommended Setting
daily/weekly/monthly	How often to rotate	daily for busy systems
rotate N	Number of old logs to keep	7-14 days for most apps
size N	Rotate when log reaches this size	10M-100M depending on disk space
compress	Compress old logs	Enable to save space
missingok	Don’t error if log missing	Usually enable this
notifempty	Don’t rotate empty logs	Enable to avoid empty archives

Advanced Logrotate Techniques for Docker Environments

Beyond basic configuration, logrotate offers sophisticated capabilities that address complex logging requirements in containerized infrastructures. The following techniques provide enhanced control and efficiency:

Dynamic Log Path Discovery and Pattern Matching

Containerized applications often generate logs with dynamic names including timestamps, UUIDs, or runtime-generated identifiers. Logrotate can handle these scenarios using pattern matching combined with script execution controls.

Implementation:

/var/log/app/*.log /var/log/app/*/*.log {
    daily
    rotate 7
    maxage 30
    sharedscripts
    olddir /var/log/archive
    createolddir 750 appuser appgroup
    dateext
    dateformat -%Y%m%d-%s
    extension .gz
    sharedscripts
    postrotate
        # Send USR1 signal to application to reopen log files
        pid=$(cat /var/run/app.pid 2>/dev/null)
        if [ -n "$pid" ]; then
            kill -USR1 "$pid"
        else
            # Alternative: find process by name and signal it
            pkill -USR1 -f "app_process" || true
        fi
    endscript
}

This configuration:

Matches logs in both the root directory and subdirectories
Archives rotated logs to a separate directory
Creates the archive directory if it doesn’t exist
Uses both date and timestamp in rotated filenames for uniqueness
Sends the USR1 signal to the application after rotating all matched logs (once only)

For microservice architectures where each service might have its logging pattern, you can create separate configuration files:

# /etc/logrotate.d/auth-service
/var/log/app/auth-*.log {
    hourly
    rotate 24
    # Service-specific settings...
}

# /etc/logrotate.d/api-service
/var/log/app/api-*.log {
    daily
    rotate 14
    # Service-specific settings...
}

Implementing Customized Log Naming and Retention Strategies

Structured log file naming is crucial for effective log management in container environments, particularly for automated log analysis and retention compliance.

Implementation Options:

/var/log/app/server.log {
    daily
    dateext
    # ISO 8601 date format for international standardization
    dateformat _%Y-%m-%d
    # Keep 7 daily logs, 4 weekly logs, and 3 monthly logs
    rotate 7
    # After daily rotation complete, perform weekly rotation
    prerotate
        if [ $(date +%u) -eq 7 ]; then
            # Sunday - move the newest rotated log to weekly archive
            if [ -f /var/log/app/server.log_$(date +%Y-%m-%d -d "yesterday") ]; then
                cp /var/log/app/server.log_$(date +%Y-%m-%d -d "yesterday") /var/log/app/weekly/server.log_week$(date +%V)
            fi
        fi

        # First day of month - archive previous month's log
        if [ $(date +%d) -eq 1 ]; then
            lastmonth=$(date +%Y-%m -d "last month")
            if [ -f /var/log/app/server.log_${lastmonth}-* ]; then
                # Find most recent log from last month and archive it
                newest=$(ls -1t /var/log/app/server.log_${lastmonth}-* | head -1)
                cp $newest /var/log/app/monthly/server.log_${lastmonth}
            fi
        fi
    endscript
    # Additional options...
    compress
    compresscmd /usr/bin/xz
    compressext .xz
    compressoptions -9
    delaycompress
}

This advanced configuration:

Uses ISO 8601 date format for standardized naming
Implements a tiered retention strategy (daily, weekly, monthly)
Uses highly efficient XZ compression for long-term storage
Automatically manages multiple retention periods from a single configuration

For containers that need to maintain logs according to regulatory requirements (e.g., PCI-DSS, HIPAA), you can extend this with an audit trail:

/var/log/app/financial-*.log {
    daily
    dateext
    dateformat _%Y-%m-%d
    rotate 365  # 1 year retention for financial data
    compress
    notifempty
    # Record all rotation events for compliance
    prerotate
        echo "$(date --iso-8601=seconds) - Beginning rotation of financial logs" >> /var/log/rotation-audit.log
    endscript
    postrotate
        echo "$(date --iso-8601=seconds) - Completed rotation of financial logs" >> /var/log/rotation-audit.log
        # Optional: Generate hash of rotated logs for integrity verification
        find /var/log/app/ -name "financial-*.log_*" -type f -mtime -1 -exec sha256sum {} \; >> /var/log/rotation-audit.log
    endscript
}

💡

Want to keep an eye on container performance? Check out this guide on monitoring resources with docker stats: Container Resource Monitoring.

Integration with Monitoring and Observability Systems

Effective log rotation should complement your overall observability strategy. Logrotate can be configured to interact with monitoring systems, providing visibility into the rotation process and maintaining consistent log analysis.

Implementation:

/var/log/app/*.log {
    daily
    rotate 7
    missingok
    compress
    dateext
    sharedscripts
    postrotate
        # Notify monitoring system about rotation event
        EVENT_DATA=$(cat <<EOF
        {
          "service": "app-container",
          "event": "log_rotation",
          "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
          "details": {
            "files_rotated": "$(find /var/log/app -name '*.log.*' -mmin -10 | wc -l)",
            "total_size_bytes": "$(du -sb /var/log/app | cut -f1)",
            "host": "$(hostname)"
          }
        }
EOF
        )

        # Send to monitoring endpoint (with retry logic)
        for i in 1 2 3; do
            if curl -s -X POST \
                -H "Content-Type: application/json" \
                -H "Authorization: Bearer ${MONITORING_API_KEY}" \
                -d "$EVENT_DATA" \
                https://your-monitoring-service.com/api/events; then
                break
            else
                sleep $((2**$i))
            fi
        done

        # Update Prometheus metrics file for node_exporter textfile collector
        METRICS_DIR="/var/lib/node_exporter/textfile"
        mkdir -p $METRICS_DIR
        cat > $METRICS_DIR/logrotate.prom <<EOF
        # HELP logrotate_last_run_timestamp_seconds Unix timestamp of last logrotate execution
        # TYPE logrotate_last_run_timestamp_seconds gauge
        logrotate_last_run_timestamp_seconds $(date +%s)
        # HELP logrotate_rotated_files_total Number of files rotated in last execution
        # TYPE logrotate_rotated_files_total gauge
        logrotate_rotated_files_total $(find /var/log/app -name '*.log.*' -mmin -10 | wc -l)
        # HELP logrotate_total_log_size_bytes Total size of log directory in bytes
        # TYPE logrotate_total_log_size_bytes gauge
        logrotate_total_log_size_bytes $(du -sb /var/log/app | cut -f1)
EOF
    endscript
}

This configuration:

Provides detailed event information to external monitoring systems
Includes retry logic to handle temporary network issues
Exposes Prometheus metrics for integration with Grafana dashboards
Tracks both rotation events and overall log volume trends

For containerized environments using centralized logging, you can also notify log shippers to reload:

postrotate
    # Signal Filebeat to check for rotated files
    pkill -SIGHUP -f filebeat || true

    # Update fluentd configuration if using tail input
    if [ -f /var/run/td-agent/td-agent.pid ]; then
        systemctl reload td-agent
    fi
endscript

💡

Don’t forget to check out our docs for integration steps if you’re using Last9 as your observability solution.

Best Practices for Docker Logrotate Implementation

Implementing logrotate in containerized environments requires careful planning and adherence to operational best practices. The following guidelines help ensure reliable, efficient log management:

Configuration Management

Version control your logrotate configurations alongside application code
Parameterize configurations using environment variables for different deployment environments
Validate configurations before deployment using logrotate -d (debug mode)
Test rotation impact on application performance in staging environments
Use separate configuration files for different application components or log types
Implement configuration templating for consistency across multiple environments

Resource Optimization

Schedule rotations during low-traffic periods to minimize the impact
Balance rotation frequency and log sizes based on I/O performance characteristics
Implement size-based triggers alongside time-based rotation to prevent disk space issues
Use copytruncate for applications that cannot reopen log files when rotated
Optimize IO patterns by using delay-compress to defer CPU-intensive operations
Consider log volume growth rates when setting rotation thresholds

Consider compression algorithm tradeoffs:

Algorithm	CPU Usage	Compression Ratio	Best For
gzip	Medium	Medium	General use
bzip2	High	High	Long-term archives
xz	Very High	Very High	Archival storage
zstd	Low	Medium-High	Production environments

Operational Excellence

Include log rotation validation in your CI/CD pipeline testing
Monitor rotation events through application logs or dedicated metrics
Set up alerts for rotation failures or skipped rotation cycles
Document your rotation strategy in infrastructure as code repositories
Review and audit log management processes quarterly
Implement automated log inspection to ensure rotation is functioning correctly
Create runbooks for common log-related incidents
Enforce consistent log format standards across container deployments

Security Considerations

Apply appropriate permissions to log files and rotation configurations
Verify log integrity when logs might be used for security auditing
Consider encrypted compression for sensitive log content
Implement secure deletion for logs containing sensitive information
Rotate credentials or tokens that might appear in log files
Establish retention policies that comply with relevant regulations
Use secure transport when forwarding logs to external systems

💡

For better logging strategies, check out these best practices: Logging Best Practices.

Step-by-Step Tutorial for Docker Logrotate

This section provides a detailed, production-ready implementation that incorporates best practices for log rotation in containerized environments.

Step 1: Design Your Log Rotation Strategy

Before implementation, define your log management requirements:

Requirement	Consideration	Example Decision
Retention Period	Regulatory requirements, debugging needs	30 days for application logs, 90 days for audit logs
Rotation Frequency	Log volume, access patterns	Size-based (100MB) with daily backup check
Compression	Storage constraints vs. CPU usage	Compress after 1 day, using zstd
Log Types	Different services, different needs	Separate configs for access logs, error logs, audit logs
Notification	Log consumers need to know about rotation	Signal processes and monitoring system

Step 2: Create Comprehensive Logrotate Configuration

Create a file named app-logrotate.conf with advanced settings:

# Application Logs
/var/log/app/application-*.log {
    daily
    size 100M
    rotate 30
    compress
    compresscmd /usr/bin/zstd
    compressext .zst
    compressoptions -19
    delaycompress
    missingok
    notifempty
    create 0640 app-user app-group
    dateext
    dateformat -%Y%m%d-%H%M%S
    extension .log
    maxage 30
    olddir /var/log/archive/application
    createolddir 750 app-user app-group
    sharedscripts
    postrotate
        # Signal application to reopen log files
        if [ -f /var/run/app.pid ]; then
            kill -USR1 $(cat /var/run/app.pid)
        fi
        # Update prometheus metrics
        echo "logrotate_application_last_run $(date +%s)" > /var/lib/metrics/logrotate.prom
    endscript
}

# Access Logs (rotated hourly due to high volume)
/var/log/app/access-*.log {
    hourly
    size 200M
    rotate 168 # 7 days worth of hourly logs
    compress
    delaycompress
    missingok
    notifempty
    create 0640 app-user app-group
    dateext
    dateformat -%Y%m%d-%H%M%S
    olddir /var/log/archive/access
    createolddir 750 app-user app-group
    sharedscripts
    postrotate
        if [ -f /var/run/app.pid ]; then
            kill -USR1 $(cat /var/run/app.pid)
        fi
    endscript
}

# Audit Logs (strict retention requirements)
/var/log/app/audit-*.log {
    daily
    size 50M
    rotate 90
    compress
    compresscmd /usr/bin/zstd
    compressext .zst
    compressoptions -19
    delaycompress
    missingok
    notifempty
    create 0640 app-user app-group
    dateext
    dateformat -%Y%m%d-%H%M%S
    olddir /var/log/archive/audit
    createolddir 750 app-user app-group
    sharedscripts
    postrotate
        # Create verification hash for compliance
        for f in $(find /var/log/archive/audit -name "audit-*.log.%Y%m%d-*" -mtime -1); do
            sha256sum "$f" >> /var/log/archive/audit/checksums.txt
        done
    endscript
}

Step 3: Create a Robust Dockerfile with Logrotate Integration

# Use multi-stage build for a lean final image
FROM ubuntu:22.04 AS builder

# Install required packages for building
RUN apt-get update && apt-get install -y \
    gcc \
    make \
    wget \
    zstd \
    && rm -rf /var/lib/apt/lists/*

# Build logrotate from source for latest features
WORKDIR /build
RUN wget https://github.com/logrotate/logrotate/releases/download/3.21.0/logrotate-3.21.0.tar.gz \
    && tar xf logrotate-3.21.0.tar.gz \
    && cd logrotate-3.21.0 \
    && ./configure \
    && make \
    && make install

# Create final image
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y \
    cron \
    zstd \
    curl \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

# Copy built logrotate from builder stage
COPY --from=builder /usr/local/sbin/logrotate /usr/local/sbin/logrotate

# Create necessary users and groups
RUN groupadd -r app-group && useradd -r -g app-group app-user

# Set up directories with proper permissions
RUN mkdir -p /var/log/app /var/log/archive/application /var/log/archive/access /var/log/archive/audit /var/lib/metrics \
    && touch /var/log/app/application.log /var/log/app/access.log /var/log/app/audit.log \
    && chown -R app-user:app-group /var/log/app /var/log/archive \
    && chmod 755 /var/log/app /var/log/archive \
    && chmod 644 /var/log/app/*.log

# Copy logrotate configuration
COPY app-logrotate.conf /etc/logrotate.d/
RUN chmod 644 /etc/logrotate.d/app-logrotate.conf

# Set up state file
RUN touch /var/lib/logrotate/status \
    && chmod 644 /var/lib/logrotate/status

# Create cron jobs for different rotation schedules
RUN echo "0 * * * * /usr/local/sbin/logrotate -f /etc/logrotate.d/app-logrotate.conf -s /var/lib/logrotate/status" > /etc/cron.d/hourly-logrotate \
    && echo "0 0 * * * /usr/local/sbin/logrotate /etc/logrotate.conf -s /var/lib/logrotate/status" > /etc/cron.d/daily-logrotate \
    && chmod 0644 /etc/cron.d/hourly-logrotate /etc/cron.d/daily-logrotate \
    && crontab /etc/cron.d/hourly-logrotate

# Copy application files
COPY --chown=app-user:app-group app/ /app/

# Create startup script
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

# Switch to non-root user for application
USER app-user

# Set up volume for persistent logs
VOLUME ["/var/log/app", "/var/log/archive"]

ENTRYPOINT ["/entrypoint.sh"]

Step 4: Create a Robust Entrypoint Script

Create an entrypoint.sh file:

#!/bin/bash
set -e

# Start cron as root (requires sudo configuration)
sudo service cron start

# Initialize log files if they don't exist
for logtype in application access audit; do
    if [ ! -f "/var/log/app/$logtype.log" ]; then
        touch "/var/log/app/$logtype.log"
        chmod 0640 "/var/log/app/$logtype.log"
    fi
done

# Record container start in logs
echo "$(date --iso-8601=seconds) - Container started" >> /var/log/app/application.log

# Store PID for logrotate to use
echo $ > /var/run/app.pid

# Execute the application
exec "$@"

Step 5: Create a Docker Compose Configuration

version: "3.8"

services:
  app:
    build: .
    command: ["node", "/app/server.js"]
    volumes:
      - app_logs:/var/log/app
      - log_archives:/var/log/archive
    ports:
      - "8080:8080"
    environment:
      - NODE_ENV=production
      - LOG_LEVEL=info
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"

volumes:
  app_logs:
    driver: local
  log_archives:
    driver: local

Step 6: Testing and Verification

After deploying your container, verify the log rotation setup:

# Create test logs of various sizes
docker exec -it <container_id> bash
dd if=/dev/urandom bs=1M count=110 | base64 > /var/log/app/application-test.log

# Force logrotate to run with verbose output
sudo /usr/local/sbin/logrotate -vf /etc/logrotate.d/app-logrotate.conf

# Verify rotation occurred properly
ls -la /var/log/app/
ls -la /var/log/archive/application/

# Check logrotate status file
cat /var/lib/logrotate/status

# Verify cron is scheduled properly
crontab -l

# Test signal handling
ps aux | grep server.js
kill -USR1 <pid>
# Check application logs to ensure it reopened log files

Step 7: Monitoring Integration

For production environments, integrate with your monitoring system:

# Sample script to push metrics to Prometheus Pushgateway
cat << 'EOF' > /app/scripts/log_metrics.sh
#!/bin/bash

# Calculate log statistics
APP_LOG_SIZE=$(du -sb /var/log/app | cut -f1)
ARCHIVE_SIZE=$(du -sb /var/log/archive | cut -f1)
LOG_COUNT=$(find /var/log/app -type f | wc -l)
ROTATION_SUCCESS=$(grep -c "Rotation successful" /var/log/logrotate.log)

# Create temporary file with metrics
cat << EOT > /tmp/logrotate_metrics.txt
# TYPE log_directory_bytes gauge
log_directory_bytes{type="app"} $APP_LOG_SIZE
log_directory_bytes{type="archive"} $ARCHIVE_SIZE
# TYPE log_file_count gauge
log_file_count $LOG_COUNT
# TYPE logrotate_success_total counter
logrotate_success_total $ROTATION_SUCCESS
EOT

# Push to Prometheus Pushgateway
curl -s --data-binary @/tmp/logrotate_metrics.txt http://pushgateway:9091/metrics/job/logrotate/instance/$(hostname)
EOF

chmod +x /app/scripts/log_metrics.sh
echo "*/10 * * * * /app/scripts/log_metrics.sh" >> /etc/cron.d/metrics

This comprehensive implementation provides a robust, production-ready log rotation solution that addresses the needs of containerized applications with different log types, retention requirements, and monitoring integration.

Conclusion

The comprehensive approaches detailed in this guide address the unique challenges of containerized architectures while providing concrete implementation patterns that can be adapted to various deployment scenarios.

Key takeaways from this guide include:

Strategic Importance: Log management directly impacts system stability, performance, and compliance, making it a foundational aspect of container operations rather than an afterthought.
Implementation Options: From embedded logrotate configurations to sidecar containers and host-based solutions, multiple architectural patterns exist to address different operational requirements.
Advanced Capabilities: Beyond basic rotation, logrotate offers sophisticated features including customized naming schemes, tiered retention policies, and integration with monitoring systems.
Operational Integration: Effective log rotation must be part of a broader operational strategy that includes monitoring, alerting, and automated remediation.
Standardization Benefits: Implementing consistent log management practices across container fleets reduces operational complexity and improves debugging capabilities.

💡

And if you ever want to talk more, join our Discord community! We have a dedicated channel where you can discuss your use case with other developers.

How to Master Log Management with Logrotate in Docker Containers

Contents

What is Logrotate and Why Use it with Docker?

Setting Up Logrotate in Docker: Implementation Approaches

Method 1: Embedding Logrotate in the Application Container

Method 2: Sidecar Container Pattern

Method 3: Host-Based Rotation

Troubleshooting Logrotate in Docker Environments

Problem: Logrotate Not Executing on Schedule

Problem: Permission Denied Errors During Rotation

Problem: Managing Docker Engine Logs (JSON Log Files)

Crafting the Perfect Logrotate Config for Docker

Advanced Logrotate Techniques for Docker Environments

Dynamic Log Path Discovery and Pattern Matching

Implementing Customized Log Naming and Retention Strategies

Integration with Monitoring and Observability Systems

Best Practices for Docker Logrotate Implementation

Configuration Management

Resource Optimization

Operational Excellence

Security Considerations

Step-by-Step Tutorial for Docker Logrotate

Step 1: Design Your Log Rotation Strategy

Step 2: Create Comprehensive Logrotate Configuration

Step 3: Create a Robust Dockerfile with Logrotate Integration

Step 4: Create a Robust Entrypoint Script

Step 5: Create a Docker Compose Configuration

Step 6: Testing and Verification

Step 7: Monitoring Integration

Conclusion

Contents

Start observing for free. No lock-in.

OpenTelemetry · Prometheus

Datadog · New Relic · Others

Built on Open Standards