How to Master Log Management with Logrotate in Docker Containers

Docker containers continuously generate logs during operation, and without proper management, these logs can consume significant disk space, impact system performance, and create operational issues. Logrotate offers an effective solution for managing these logs in containerized environments.

This guide covers the implementation of logrotate in Docker containers – from initial setup through advanced configurations that ensure stable, maintainable container deployments. The techniques provided address common challenges faced in production environments and offer concrete solutions for effective log management.

What is Logrotate and Why Use it with Docker?

Docker containers generate substantial log data during normal operation. These logs come from multiple sources:

1. Container stdout/stderr: Captured by Docker's logging drivers
2. Application-specific logs: Written to files inside the container
3. Service logs: Generated by components running within the container

Without a structured log rotation strategy, these files grow continuously, potentially causing:

• Disk space exhaustion leading to service outages
• Degraded I/O performance affecting container operations
• Increased backup duration and storage costs
• Challenges in log analysis and troubleshooting

Logrotate provides a robust solution by handling log rotation, compression, removal, and mailing of log files. When implemented in Docker environments, it offers:

• Configurable retention policies based on size, age, or both
• Automated compression to reduce storage requirements
• Customizable naming conventions for archived logs
• Rotation triggering that can integrate with application signal handling
• Scriptable pre/post-processing for logs during rotation events

While Docker does provide basic logging controls through logging drivers (JSON-file, local, syslog, etc.), these mechanisms often lack the granularity and flexibility required for production environments. Logrotate delivers precise control over log lifecycle management with minimal overhead.

Setting Up Logrotate in Docker: Implementation Approaches

Implementing logrotate in Docker environments can be accomplished through several architectural patterns, each with specific advantages depending on your infrastructure requirements. The following methods provide a foundation for integrating log rotation capabilities with containerized applications:

Method 1: Embedding Logrotate in the Application Container

This approach integrates logrotate directly into your application container, creating a self-contained solution where each container manages its logs. This method is particularly useful for stateful applications that maintain persistent logs.

FROM ubuntu:20.04

# Install logrotate and dependencies
RUN apt-get update && apt-get install -y \
    logrotate \
    cron \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

# Configure logrotate with specific settings
COPY my-logrotate.conf /etc/logrotate.d/

# Set up cron to run logrotate hourly
RUN echo '0 * * * * /usr/sbin/logrotate -f /etc/logrotate.conf >/proc/1/fd/1 2>/proc/1/fd/2' > /etc/cron.d/logrotate-cron \
    && chmod 0644 /etc/cron.d/logrotate-cron \
    && crontab /etc/cron.d/logrotate-cron

# Create log directories with appropriate permissions
RUN mkdir -p /var/log/app \
    && touch /var/log/logrotate.status \
    && chmod 644 /var/log/logrotate.status

# Start both cron daemon and application
ENTRYPOINT ["/bin/bash", "-c", "service cron start && exec $@"]
CMD ["cron", "-f"]

This configuration ensures that:

• Logrotate runs on an hourly schedule
• Rotation status is properly tracked
• Cron output is directed to container stdout for visibility
• The container has appropriate permissions for log management

Method 2: Sidecar Container Pattern

The sidecar pattern employs a dedicated container that runs alongside your application container, with both sharing access to the same log volume. This approach adheres to the single-responsibility principle, keeping your application container focused on its core function while delegating log management to a specialized container.

version: '3.8'
services:
  app:
    image: your-application:latest
    volumes:
      - logs:/var/log/app
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
  
  logrotate:
    image: blacklabelops/logrotate:latest
    volumes:
      - logs:/var/log/app
    environment:
      - LOGS_DIRECTORIES=/var/log/app
      - LOGROTATE_STATUSFILE=/var/log/logrotate/logrotate.status
      - LOGROTATE_INTERVAL=daily
      - LOGROTATE_COPIES=7
      - LOGROTATE_SIZE=50M
      - LOGROTATE_COMPRESSION=compress
      - LOG_FILE_ENDINGS=log
      - LOGROTATE_DATEFORMAT=-%Y%m%d
    restart: unless-stopped
    depends_on:
      - app

volumes:
  logs:
    driver: local

This configuration provides:

• Complete separation of concerns between application and log management
• Fine-grained control over rotation parameters via environment variables
• Automatic restart capability if the logrotate container fails
• Shared access to log volumes with proper sequence guarantees

Method 3: Host-Based Rotation

This method leverages the host system's existing logrotate infrastructure by mounting container log directories to specified host paths. The approach simplifies container configuration while centralizing log management at the host level.

# Run container with logs mounted to host directory
docker run \
  --name app-container \
  -v /var/log/containers/app1:/var/log \
  -d your-container

# Alternative using docker-compose
# docker-compose.yml excerpt:
# services:
#   app:
#     image: your-container
#     volumes:
#       - /var/log/containers/app1:/var/log

On the host system, create a dedicated logrotate configuration:

# /etc/logrotate.d/docker-containers
/var/log/containers/*/*.log {
    daily
    rotate 7
    size 100M
    compress
    delaycompress
    missingok
    notifempty
    dateext
    dateformat -%Y%m%d
    create 0644 root root
    sharedscripts
    postrotate
        # Optional: notify containers about rotation
        for container in $(docker ps --format '{{.Names}}'); do
            docker kill --signal=USR1 $container >/dev/null 2>&1 || true
        done
    endscript
}

Benefits of this approach:

• Centralizes log rotation policy across multiple containers
• Utilizes existing host system maintenance workflows
• Reduces container complexity and image size
• Provides unified log access for host-based monitoring tools

This method works particularly well in environments where:

• Host systems are long-lived compared to containers
• Operations teams prefer monitoring logs from the host's perspective
• Container images need to be kept minimal

Troubleshooting Logrotate in Docker Environments

When implementing logrotate in containerized environments, several common issues can arise. This section addresses these challenges with specific diagnostic steps and solutions.

Problem: Logrotate Not Executing on Schedule

This issue typically occurs when the container lacks a proper scheduler mechanism or when process management is misconfigured.

Diagnostic Steps:

Examine cron logs for execution attempts:

docker exec -it container-name grep cron /var/log/syslog

Verify crontab configuration:

docker exec -it container-name crontab -l

Check if cron is installed and running in the container:

docker exec -it container-name ps aux | grep cron

Solution: Implement a robust scheduling mechanism with proper logging:

# Add to your Dockerfile
RUN apt-get update && apt-get install -y cron rsyslog

# Configure cron with logging enabled
RUN touch /var/log/cron.log
COPY crontab /etc/cron.d/logrotate-cron
RUN chmod 0644 /etc/cron.d/logrotate-cron
RUN crontab /etc/cron.d/logrotate-cron

# Startup script to ensure services run
COPY start.sh /start.sh
RUN chmod +x /start.sh
CMD ["/start.sh"]

Your crontab file should include verbose logging:

0 0 * * * /usr/sbin/logrotate -v /etc/logrotate.conf >> /var/log/cron-logrotate.log 2>&1

The start.sh script ensures both services run:

#!/bin/bash
service rsyslog start
service cron start
# Keep container running
tail -f /var/log/cron.log

Problem: Permission Denied Errors During Rotation

Logrotate requires specific permissions to read, write, and manage log files. In Docker environments, permission issues commonly arise due to user mismatches between the application generating logs and the process rotating them.

Diagnostic Steps:

Examine logrotate error messages:

docker exec container-name logrotate -d -v /etc/logrotate.conf

Check existing log file permissions:

docker exec container-name ls -la /var/log/

Identify the user running your application:

docker exec container-name ps aux

Solution: Implement comprehensive permission management in your Dockerfile:

# Add to your Dockerfile - adjust user/group as needed
RUN groupadd -r appgroup && useradd -r -g appgroup appuser

# Create log directory with appropriate ownership
RUN mkdir -p /var/log/app \
    && touch /var/log/app/app.log \
    && chown -R appuser:appgroup /var/log/app \
    && chmod 755 /var/log/app \
    && chmod 644 /var/log/app/*.log

# Ensure logrotate has access
RUN chown root:root /etc/logrotate.d/* \
    && chmod 644 /etc/logrotate.d/*

# Configure logrotate to create new logs with correct permissions
# In your logrotate config file:
# create 0644 appuser appgroup

For scenarios where the application user differs from the logrotate user, use the su directive in your logrotate configuration:

/var/log/app/*.log {
    daily
    rotate 7
    # Run as specific user
    su appuser appgroup
    # Other directives...
}

Problem: Managing Docker Engine Logs (JSON Log Files)

While application logs inside containers can be managed with logrotate, Docker's own JSON log files (typically stored in /var/lib/docker/containers/[container-id]/[container-id]-json.log) require a different approach since they're generated outside the container context.

Diagnostic Steps:

Verify current Docker daemon configuration:

docker info | grep "Logging Driver"
cat /etc/docker/daemon.json

Check current Docker log usage:

sudo find /var/lib/docker/containers -name "*-json.log" -exec ls -sh {} \;

Solution: Implement a dual-layer approach:

1. Configure container-level log limits in your docker-compose.yml:

services:
  app:
    image: your-application
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "5"
        compress: "true"
        labels: "production_app"
        env: "staging,production"

2. For system-wide defaults, configure the Docker daemon by editing /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "3",
    "labels": "production",
    "env": "os,customer"
  }
}

After updating this file, restart the Docker daemon:

sudo systemctl restart docker

3. For existing containers with excessive logs, use:

sudo sh -c 'truncate -s 0 /var/lib/docker/containers/*/*-json.log'
docker restart $(docker ps -q)

This comprehensive approach ensures both internal application logs and Docker's container logs are properly managed. For production environments, consider advanced solutions like forwarding logs to centralized logging systems (Last9, ELK Stack, Graylog, etc.) using the appropriate log drivers (fluentd, syslog, etc.).

Crafting the Perfect Logrotate Config for Docker

A well-tuned logrotate configuration makes all the difference. Here's a solid starting point:

/var/log/app/*.log {
    daily
    rotate 7
    size 10M
    missingok
    compress
    delaycompress
    notifempty
    create 0644 root root
    sharedscripts
    postrotate
        kill -USR1 $(cat /var/run/app.pid 2>/dev/null) 2>/dev/null || true
    endscript
}

Let's break down the key options:

Advanced Logrotate Techniques for Docker Environments

Beyond basic configuration, logrotate offers sophisticated capabilities that address complex logging requirements in containerized infrastructures. The following techniques provide enhanced control and efficiency:

Dynamic Log Path Discovery and Pattern Matching

Containerized applications often generate logs with dynamic names including timestamps, UUIDs, or runtime-generated identifiers. Logrotate can handle these scenarios using pattern matching combined with script execution controls.

Implementation:

/var/log/app/*.log /var/log/app/*/*.log {
    daily
    rotate 7
    maxage 30
    sharedscripts
    olddir /var/log/archive
    createolddir 750 appuser appgroup
    dateext
    dateformat -%Y%m%d-%s
    extension .gz
    sharedscripts
    postrotate
        # Send USR1 signal to application to reopen log files
        pid=$(cat /var/run/app.pid 2>/dev/null)
        if [ -n "$pid" ]; then
            kill -USR1 "$pid"
        else
            # Alternative: find process by name and signal it
            pkill -USR1 -f "app_process" || true
        fi
    endscript
}

This configuration:

• Matches logs in both the root directory and subdirectories
• Archives rotated logs to a separate directory
• Creates the archive directory if it doesn't exist
• Uses both date and timestamp in rotated filenames for uniqueness
• Sends the USR1 signal to the application after rotating all matched logs (once only)

For microservice architectures where each service might have its logging pattern, you can create separate configuration files:

# /etc/logrotate.d/auth-service
/var/log/app/auth-*.log {
    hourly
    rotate 24
    # Service-specific settings...
}

# /etc/logrotate.d/api-service
/var/log/app/api-*.log {
    daily
    rotate 14
    # Service-specific settings...
}

Implementing Customized Log Naming and Retention Strategies

Structured log file naming is crucial for effective log management in container environments, particularly for automated log analysis and retention compliance.

Implementation Options:

/var/log/app/server.log {
    daily
    dateext
    # ISO 8601 date format for international standardization
    dateformat _%Y-%m-%d
    # Keep 7 daily logs, 4 weekly logs, and 3 monthly logs
    rotate 7
    # After daily rotation complete, perform weekly rotation
    prerotate
        if [ $(date +%u) -eq 7 ]; then
            # Sunday - move the newest rotated log to weekly archive
            if [ -f /var/log/app/server.log_$(date +%Y-%m-%d -d "yesterday") ]; then
                cp /var/log/app/server.log_$(date +%Y-%m-%d -d "yesterday") /var/log/app/weekly/server.log_week$(date +%V)
            fi
        fi
        
        # First day of month - archive previous month's log
        if [ $(date +%d) -eq 1 ]; then
            lastmonth=$(date +%Y-%m -d "last month")
            if [ -f /var/log/app/server.log_${lastmonth}-* ]; then
                # Find most recent log from last month and archive it
                newest=$(ls -1t /var/log/app/server.log_${lastmonth}-* | head -1)
                cp $newest /var/log/app/monthly/server.log_${lastmonth}
            fi
        fi
    endscript
    # Additional options...
    compress
    compresscmd /usr/bin/xz
    compressext .xz
    compressoptions -9
    delaycompress
}

This advanced configuration:

• Uses ISO 8601 date format for standardized naming
• Implements a tiered retention strategy (daily, weekly, monthly)
• Uses highly efficient XZ compression for long-term storage
• Automatically manages multiple retention periods from a single configuration

For containers that need to maintain logs according to regulatory requirements (e.g., PCI-DSS, HIPAA), you can extend this with an audit trail:

/var/log/app/financial-*.log {
    daily
    dateext
    dateformat _%Y-%m-%d
    rotate 365  # 1 year retention for financial data
    compress
    notifempty
    # Record all rotation events for compliance
    prerotate
        echo "$(date --iso-8601=seconds) - Beginning rotation of financial logs" >> /var/log/rotation-audit.log
    endscript
    postrotate
        echo "$(date --iso-8601=seconds) - Completed rotation of financial logs" >> /var/log/rotation-audit.log
        # Optional: Generate hash of rotated logs for integrity verification
        find /var/log/app/ -name "financial-*.log_*" -type f -mtime -1 -exec sha256sum {} \; >> /var/log/rotation-audit.log
    endscript
}

Integration with Monitoring and Observability Systems

Effective log rotation should complement your overall observability strategy. Logrotate can be configured to interact with monitoring systems, providing visibility into the rotation process and maintaining consistent log analysis.

Implementation:

/var/log/app/*.log {
    daily
    rotate 7
    missingok
    compress
    dateext
    sharedscripts
    postrotate
        # Notify monitoring system about rotation event
        EVENT_DATA=$(cat <<EOF
        {
          "service": "app-container",
          "event": "log_rotation",
          "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
          "details": {
            "files_rotated": "$(find /var/log/app -name '*.log.*' -mmin -10 | wc -l)",
            "total_size_bytes": "$(du -sb /var/log/app | cut -f1)",
            "host": "$(hostname)"
          }
        }
EOF
        )
        
        # Send to monitoring endpoint (with retry logic)
        for i in 1 2 3; do
            if curl -s -X POST \
                -H "Content-Type: application/json" \
                -H "Authorization: Bearer ${MONITORING_API_KEY}" \
                -d "$EVENT_DATA" \
                https://your-monitoring-service.com/api/events; then
                break
            else
                sleep $((2**$i))
            fi
        done
        
        # Update Prometheus metrics file for node_exporter textfile collector
        METRICS_DIR="/var/lib/node_exporter/textfile"
        mkdir -p $METRICS_DIR
        cat > $METRICS_DIR/logrotate.prom <<EOF
        # HELP logrotate_last_run_timestamp_seconds Unix timestamp of last logrotate execution
        # TYPE logrotate_last_run_timestamp_seconds gauge
        logrotate_last_run_timestamp_seconds $(date +%s)
        # HELP logrotate_rotated_files_total Number of files rotated in last execution
        # TYPE logrotate_rotated_files_total gauge
        logrotate_rotated_files_total $(find /var/log/app -name '*.log.*' -mmin -10 | wc -l)
        # HELP logrotate_total_log_size_bytes Total size of log directory in bytes
        # TYPE logrotate_total_log_size_bytes gauge
        logrotate_total_log_size_bytes $(du -sb /var/log/app | cut -f1)
EOF
    endscript
}

This configuration:

• Provides detailed event information to external monitoring systems
• Includes retry logic to handle temporary network issues
• Exposes Prometheus metrics for integration with Grafana dashboards
• Tracks both rotation events and overall log volume trends

For containerized environments using centralized logging, you can also notify log shippers to reload:

postrotate
    # Signal Filebeat to check for rotated files
    pkill -SIGHUP -f filebeat || true
    
    # Update fluentd configuration if using tail input
    if [ -f /var/run/td-agent/td-agent.pid ]; then
        systemctl reload td-agent
    fi
endscript

Best Practices for Docker Logrotate Implementation

Implementing logrotate in containerized environments requires careful planning and adherence to operational best practices. The following guidelines help ensure reliable, efficient log management:

Configuration Management

• Version control your logrotate configurations alongside application code
• Parameterize configurations using environment variables for different deployment environments
• Validate configurations before deployment using logrotate -d (debug mode)
• Test rotation impact on application performance in staging environments
• Use separate configuration files for different application components or log types
• Implement configuration templating for consistency across multiple environments

Resource Optimization

• Schedule rotations during low-traffic periods to minimize the impact
• Balance rotation frequency and log sizes based on I/O performance characteristics
• Implement size-based triggers alongside time-based rotation to prevent disk space issues
• Use copytruncate for applications that cannot reopen log files when rotated
• Optimize IO patterns by using delay-compress to defer CPU-intensive operations
• Consider log volume growth rates when setting rotation thresholds

Consider compression algorithm tradeoffs:

Operational Excellence

• Include log rotation validation in your CI/CD pipeline testing
• Monitor rotation events through application logs or dedicated metrics
• Set up alerts for rotation failures or skipped rotation cycles
• Document your rotation strategy in infrastructure as code repositories
• Review and audit log management processes quarterly
• Implement automated log inspection to ensure rotation is functioning correctly
• Create runbooks for common log-related incidents
• Enforce consistent log format standards across container deployments

Security Considerations

• Apply appropriate permissions to log files and rotation configurations
• Verify log integrity when logs might be used for security auditing
• Consider encrypted compression for sensitive log content
• Implement secure deletion for logs containing sensitive information
• Rotate credentials or tokens that might appear in log files
• Establish retention policies that comply with relevant regulations
• Use secure transport when forwarding logs to external systems

Step-by-Step Tutorial for Docker Logrotate

This section provides a detailed, production-ready implementation that incorporates best practices for log rotation in containerized environments.

Step 1: Design Your Log Rotation Strategy

Before implementation, define your log management requirements:

Step 2: Create Comprehensive Logrotate Configuration

Create a file named app-logrotate.conf with advanced settings:

# Application Logs
/var/log/app/application-*.log {
    daily
    size 100M
    rotate 30
    compress
    compresscmd /usr/bin/zstd
    compressext .zst
    compressoptions -19
    delaycompress
    missingok
    notifempty
    create 0640 app-user app-group
    dateext
    dateformat -%Y%m%d-%H%M%S
    extension .log
    maxage 30
    olddir /var/log/archive/application
    createolddir 750 app-user app-group
    sharedscripts
    postrotate
        # Signal application to reopen log files
        if [ -f /var/run/app.pid ]; then
            kill -USR1 $(cat /var/run/app.pid)
        fi
        # Update prometheus metrics
        echo "logrotate_application_last_run $(date +%s)" > /var/lib/metrics/logrotate.prom
    endscript
}

# Access Logs (rotated hourly due to high volume)
/var/log/app/access-*.log {
    hourly
    size 200M
    rotate 168 # 7 days worth of hourly logs
    compress
    delaycompress
    missingok
    notifempty
    create 0640 app-user app-group
    dateext
    dateformat -%Y%m%d-%H%M%S
    olddir /var/log/archive/access
    createolddir 750 app-user app-group
    sharedscripts
    postrotate
        if [ -f /var/run/app.pid ]; then
            kill -USR1 $(cat /var/run/app.pid)
        fi
    endscript
}

# Audit Logs (strict retention requirements)
/var/log/app/audit-*.log {
    daily
    size 50M
    rotate 90
    compress
    compresscmd /usr/bin/zstd
    compressext .zst
    compressoptions -19
    delaycompress
    missingok
    notifempty
    create 0640 app-user app-group
    dateext
    dateformat -%Y%m%d-%H%M%S
    olddir /var/log/archive/audit
    createolddir 750 app-user app-group
    sharedscripts
    postrotate
        # Create verification hash for compliance
        for f in $(find /var/log/archive/audit -name "audit-*.log.%Y%m%d-*" -mtime -1); do
            sha256sum "$f" >> /var/log/archive/audit/checksums.txt
        done
    endscript
}

Step 3: Create a Robust Dockerfile with Logrotate Integration

# Use multi-stage build for a lean final image
FROM ubuntu:22.04 AS builder

# Install required packages for building
RUN apt-get update && apt-get install -y \
    gcc \
    make \
    wget \
    zstd \
    && rm -rf /var/lib/apt/lists/*

# Build logrotate from source for latest features
WORKDIR /build
RUN wget https://github.com/logrotate/logrotate/releases/download/3.21.0/logrotate-3.21.0.tar.gz \
    && tar xf logrotate-3.21.0.tar.gz \
    && cd logrotate-3.21.0 \
    && ./configure \
    && make \
    && make install

# Create final image
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y \
    cron \
    zstd \
    curl \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

# Copy built logrotate from builder stage
COPY --from=builder /usr/local/sbin/logrotate /usr/local/sbin/logrotate

# Create necessary users and groups
RUN groupadd -r app-group && useradd -r -g app-group app-user

# Set up directories with proper permissions
RUN mkdir -p /var/log/app /var/log/archive/application /var/log/archive/access /var/log/archive/audit /var/lib/metrics \
    && touch /var/log/app/application.log /var/log/app/access.log /var/log/app/audit.log \
    && chown -R app-user:app-group /var/log/app /var/log/archive \
    && chmod 755 /var/log/app /var/log/archive \
    && chmod 644 /var/log/app/*.log

# Copy logrotate configuration
COPY app-logrotate.conf /etc/logrotate.d/
RUN chmod 644 /etc/logrotate.d/app-logrotate.conf

# Set up state file
RUN touch /var/lib/logrotate/status \
    && chmod 644 /var/lib/logrotate/status

# Create cron jobs for different rotation schedules
RUN echo "0 * * * * /usr/local/sbin/logrotate -f /etc/logrotate.d/app-logrotate.conf -s /var/lib/logrotate/status" > /etc/cron.d/hourly-logrotate \
    && echo "0 0 * * * /usr/local/sbin/logrotate /etc/logrotate.conf -s /var/lib/logrotate/status" > /etc/cron.d/daily-logrotate \
    && chmod 0644 /etc/cron.d/hourly-logrotate /etc/cron.d/daily-logrotate \
    && crontab /etc/cron.d/hourly-logrotate

# Copy application files
COPY --chown=app-user:app-group app/ /app/

# Create startup script
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

# Switch to non-root user for application
USER app-user

# Set up volume for persistent logs
VOLUME ["/var/log/app", "/var/log/archive"]

ENTRYPOINT ["/entrypoint.sh"]

Step 4: Create a Robust Entrypoint Script

Create an entrypoint.sh file:

#!/bin/bash
set -e

# Start cron as root (requires sudo configuration)
sudo service cron start

# Initialize log files if they don't exist
for logtype in application access audit; do
    if [ ! -f "/var/log/app/$logtype.log" ]; then
        touch "/var/log/app/$logtype.log"
        chmod 0640 "/var/log/app/$logtype.log"
    fi
done

# Record container start in logs
echo "$(date --iso-8601=seconds) - Container started" >> /var/log/app/application.log

# Store PID for logrotate to use
echo $ > /var/run/app.pid

# Execute the application
exec "$@"

Step 5: Create a Docker Compose Configuration

version: '3.8'

services:
  app:
    build: .
    command: ["node", "/app/server.js"]
    volumes:
      - app_logs:/var/log/app
      - log_archives:/var/log/archive
    ports:
      - "8080:8080"
    environment:
      - NODE_ENV=production
      - LOG_LEVEL=info
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"

volumes:
  app_logs:
    driver: local
  log_archives:
    driver: local

Step 6: Testing and Verification

After deploying your container, verify the log rotation setup:

# Create test logs of various sizes
docker exec -it <container_id> bash
dd if=/dev/urandom bs=1M count=110 | base64 > /var/log/app/application-test.log

# Force logrotate to run with verbose output
sudo /usr/local/sbin/logrotate -vf /etc/logrotate.d/app-logrotate.conf

# Verify rotation occurred properly
ls -la /var/log/app/
ls -la /var/log/archive/application/

# Check logrotate status file
cat /var/lib/logrotate/status

# Verify cron is scheduled properly
crontab -l

# Test signal handling
ps aux | grep server.js
kill -USR1 <pid>
# Check application logs to ensure it reopened log files

Step 7: Monitoring Integration

For production environments, integrate with your monitoring system:

# Sample script to push metrics to Prometheus Pushgateway
cat << 'EOF' > /app/scripts/log_metrics.sh
#!/bin/bash

# Calculate log statistics
APP_LOG_SIZE=$(du -sb /var/log/app | cut -f1)
ARCHIVE_SIZE=$(du -sb /var/log/archive | cut -f1)
LOG_COUNT=$(find /var/log/app -type f | wc -l)
ROTATION_SUCCESS=$(grep -c "Rotation successful" /var/log/logrotate.log)

# Create temporary file with metrics
cat << EOT > /tmp/logrotate_metrics.txt
# TYPE log_directory_bytes gauge
log_directory_bytes{type="app"} $APP_LOG_SIZE
log_directory_bytes{type="archive"} $ARCHIVE_SIZE
# TYPE log_file_count gauge
log_file_count $LOG_COUNT
# TYPE logrotate_success_total counter
logrotate_success_total $ROTATION_SUCCESS
EOT

# Push to Prometheus Pushgateway
curl -s --data-binary @/tmp/logrotate_metrics.txt http://pushgateway:9091/metrics/job/logrotate/instance/$(hostname)
EOF

chmod +x /app/scripts/log_metrics.sh
echo "*/10 * * * * /app/scripts/log_metrics.sh" >> /etc/cron.d/metrics

This comprehensive implementation provides a robust, production-ready log rotation solution that addresses the needs of containerized applications with different log types, retention requirements, and monitoring integration.

Conclusion

The comprehensive approaches detailed in this guide address the unique challenges of containerized architectures while providing concrete implementation patterns that can be adapted to various deployment scenarios.

Key takeaways from this guide include:

1. Strategic Importance: Log management directly impacts system stability, performance, and compliance, making it a foundational aspect of container operations rather than an afterthought.
2. Implementation Options: From embedded logrotate configurations to sidecar containers and host-based solutions, multiple architectural patterns exist to address different operational requirements.
3. Advanced Capabilities: Beyond basic rotation, logrotate offers sophisticated features including customized naming schemes, tiered retention policies, and integration with monitoring systems.
4. Operational Integration: Effective log rotation must be part of a broader operational strategy that includes monitoring, alerting, and automated remediation.
5. Standardization Benefits: Implementing consistent log management practices across container fleets reduces operational complexity and improves debugging capabilities.