# Advanced Container Resource Monitoring with docker stats

> Go beyond basics with docker stats! Learn how to monitor container CPU, memory, and I/O like a pro for peak performance. 

Source: https://last9.io/blog/container-resource-monitoring-with-docker-stats/

If you’ve ever needed to check how much CPU or memory a Docker container is using, `docker stats` is the command for the job. It provides real-time resource usage metrics, helping you monitor and troubleshoot containers efficiently.

This guide covers everything you need to know about `docker stats`: how to use it, what each metric means, and how to integrate it into a larger monitoring setup. We'll also explore advanced techniques for optimizing your workflow and using `docker stats` enterprise-grade monitoring solutions.

## What Is `docker stats` and Why Should You Care?

### The Basics

The [`docker stats` command](https://docs.docker.com/reference/cli/docker/container/stats/) provides live resource usage metrics for your running containers. Think of it as the Docker equivalent of Linux's `top` command, but specifically designed for containers.

It shows critical metrics including:

- **CPU usage**: Both percentage and absolute values
- **Memory consumption**: Current usage, limits, and cache statistics
- **Network activity**: Incoming and outgoing traffic rates
- **Disk I/O**: Read/write operations and throughput
- **Process information**: Number of processes running inside the container

If you're monitoring containers, you’ll probably need to filter logs efficiently—here’s [how you can do that using `grep`](https://last9.io/blog/how-to-filter-docker-logs-with-grep/).

### Why Resource Monitoring Is Critical

Unchecked containers can lead to several serious problems:

- **System instability**: Resource-hungry containers can starve other services
- **Performance degradation**: Slow applications due to resource contention
- **Increased infrastructure costs**: Inefficient resource usage leads to over-provisioning
- **Application crashes**: Out-of-Memory (OOM) kills when containers exceed memory limits
- **Troubleshooting complexity**: Difficulty pinpointing issues without visibility

Understanding `docker stats` ensures that you can:

- **Optimize resource allocation**: Right-size your containers based on actual usage
- **Prevent container crashes**: Detect resource issues before they cause failures
- **Identify bottlenecks**: Spot performance issues early in your deployment
- **Plan capacity**: Make informed decisions about infrastructure scaling
- **Validate container specifications**: Ensure your resource limits are appropriate

## How to Use `docker stats` for Real-Time Container Monitoring

### Basic Usage

To get started with basic monitoring, simply run:

```bash
docker stats
```

This will display a continuously updating table of all running containers along with their resource usage metrics in real time.

### Monitoring Specific Containers

To focus on a specific container, use:

```bash
docker stats <container_id>
```

Or for monitoring multiple specific containers:

```bash
docker stats <container_id1> <container_id2>
```

You can use either container IDs or container names in these commands.

### Using Container Name Patterns

You can also use name patterns with the `--filter` option:

```bash
docker stats --filter "name=api-*"
```

This will show stats only for containers whose names start with "api-".

### One-Time Snapshot vs. Continuous Monitoring

By default, `docker stats` provides continuous updates. For a single snapshot:

```bash
docker stats --no-stream
```

This is particularly useful for scripts or when you want to capture a point-in-time measurement.

Managing container resources also means keeping logs in check. [Here’s how you can clear Docker logs](https://last9.io/blog/docker-clear-logs/) to free up space and avoid clutter.

## Breaking Down `docker stats` Output: What Each Metric Means

When you run `docker stats`, you'll see output similar to this:

```
CONTAINER ID   NAME        CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
b3d5d9a8e123   my_app      2.55%     150MiB / 500MiB       30.0%     1.2MB / 800kB     0B / 0B           12
f7d6e9c8b123   database    35.20%    1.2GiB / 2GiB         60.0%     5.5MB / 1.2MB     250MB / 15MB      45
```

Let's examine each column in detail:

### CONTAINER ID and NAME

- **CONTAINER ID**: The unique identifier for the container
- **NAME**: The human-readable name assigned to the container

### CPU %

The percentage of host CPU resources used by the container. This value can exceed 100% on multi-core systems, as it represents the usage across all available cores.

- **\<10%**: Generally indicates low CPU utilization
- **10-50%**: Moderate utilization
- **50-80%**: High utilization, may need attention
- **\>80%**: Very high utilization, potential performance bottleneck

For multi-core systems, if your container is allowed to use all cores, a value of 200% would mean it's using the equivalent of 2 full CPU cores.

### MEM USAGE / LIMIT

This shows current memory consumption and the maximum limit (if set):

- **Usage**: The actual memory being used by the container, including application memory and any buffers/cache
- **Limit**: The maximum memory allocation for the container (will show host total if no limit is set)

Memory values are displayed in binary units:

- KiB (kibibytes) = 1,024 bytes
- MiB (mebibytes) = 1,024 KiB
- GiB (gibibytes) = 1,024 MiB

### MEM %

Memory usage as a percentage of the total limit available to the container. Critical thresholds:

- **\<50%**: Healthy utilization
- **50-80%**: Moderate utilization, monitor for trends
- **\>80%**: High utilization, risk of OOM kills if usage spikes

### NET I/O

Network traffic metrics shown as:

```
[data received] / [data transmitted]
```

This helps detect:

- Unexpected network traffic spikes
- Data exfiltration concerns
- Network bottlenecks
- Communication patterns between containers

### BLOCK I/O

Disk read/write activity shown as:

```
[data read] / [data written]
```

High values might indicate:

- Excessive logging
- Inefficient storage operations
- Database write pressure
- File system issues

### PIDS

Number of processes running inside the container. This is critical for detecting:

- Process leaks
- Fork bombs
- Inefficient process management
- Container complexity

Typical ranges:

- **1-10**: Simple applications
- **10-50**: Moderate complexity
- **\>50**: Complex applications or potential issues

Understanding resource usage is key, whether you're running a single container or orchestrating with Swarm. [Here’s how Docker and Docker Swarm compare](https://last9.io/blog/docker-vs-docker-swarm-key-differences-explained/).

## Advanced Filtering and Custom Output Formatting for `docker stats`

### Customize Output Fields

The `--format` flag allows you to specify exactly what information you want to see:

```bash
docker stats --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}"
```

Available format placeholders include:

- `.Container`: Container ID
- `.Name`: Container name
- `.ID`: Container ID
- `.CPUPerc`: CPU percentage
- `.MemUsage`: Memory usage
- `.MemPerc`: Memory percentage
- `.NetIO`: Network I/O
- `.BlockIO`: Block I/O
- `.PIDs`: Number of processes

### Sort Containers by Memory or CPU

`docker stats` has no sort flag. To rank containers, take a single snapshot
with `--no-stream` and pipe the output to `sort`:

```bash
# Highest memory percentage first
docker stats --no-stream --format "{{.Name}}\t{{.MemPerc}}" | sort -k2 -g -r

# Highest CPU percentage first
docker stats --no-stream --format "{{.Name}}\t{{.CPUPerc}}" | sort -k2 -g -r
```

Sort on the percentage fields rather than `.MemUsage`. Docker prints usage in
whatever unit fits, so a list can mix `512MiB` and `1.2GiB`, and a plain
numeric sort puts the 512MiB container on top. Percentages share one unit, so
they always rank correctly.

Omitting `table` from the format string drops the header row, which would
otherwise be sorted along with the data.

### JSON Output for Automation and Integration

For programmatic use or feeding into other tools:

```bash
docker stats --format "{{json .}}"
```

This produces structured JSON output for each container:

```json
{
  "BlockIO": "0B / 0B",
  "CPUPerc": "0.00%",
  "Container": "b3d5d9a8e123",
  "ID": "b3d5d9a8e123",
  "MemPerc": "0.26%",
  "MemUsage": "21.05MiB / 7.772GiB",
  "Name": "my_app",
  "NetIO": "726B / 0B",
  "PIDs": "1"
}
```

### Create Custom Monitoring Scripts

You can combine `docker stats` with `jq` for powerful filtering:

```bash
docker stats --no-stream --format "{{json .}}" | jq 'select(.CPUPerc | rtrimstr("%") | tonumber > 10)'
```

This example shows only containers using more than 10% CPU.

### Periodic Sampling with Timestamps

Combine with `watch` and `date` for timestamped samples:

```bash
watch -n 10 'date; docker stats --no-stream'
```

This captures stats every 10 seconds with timestamps.

## How to Interpret Resource Usage Patterns

Understanding resource usage patterns can help you diagnose performance issues and optimize your containers. Here’s a quick breakdown of what different trends might indicate:

### **CPU Usage Patterns**

- **Steady high CPU** → A compute-heavy task or an infinite loop.
- **CPU spikes** → Likely batch jobs, cron tasks, or periodic processing.
- **Low but constant CPU** → An idle service or a background process ticking along.
- **Oscillating CPU** → Possible resource contention or CPU throttling.

### **Memory Usage Patterns**

- **Steady increase** → Could be a memory leak—watch out!
- **Sawtooth pattern** → Expected behavior from garbage collection.
- **Sudden drops** → The process might be restarting, or the system killed it (OOM event).
- **Constant high memory** → Inefficient caching or handling large datasets.

### **Network I/O Patterns**

- **Constant small transfers** → Likely health checks or heartbeat signals.
- **Large sudden spikes** → Bulk data transfers happening.
- **Consistently high bandwidth** → This container is probably handling data streaming.
- **Asymmetric I/O** → Heavy uploads or downloads in progress.

Real-time monitoring is crucial for container performance. [Here’s a guide on using `docker logs --tail`](https://last9.io/blog/docker-logs-tail-a-guide/) to keep an eye on recent logs efficiently.

### **Disk I/O Patterns**

- **High writes, low reads** → Common in logging or data collection processes.
- **High reads, low writes** → Likely caching or content delivery workloads.
- **Periodic spikes** → Could be scheduled backups, cleanup tasks, or maintenance jobs.
- **Constant high I/O** → A database or file-intensive application is at work.

## Comparing `docker stats` vs. Other Monitoring Tools

While `docker stats` is excellent for quick inspections, it has several limitations for production environments:

| Feature           | docker stats   | cAdvisor | Prometheus+Grafana  | Last9               |
| ----------------- | -------------- | -------- | ------------------- | ------------------- |
| Historical data   | No             | Limited  | Yes                 | Yes                 |
| Alerting          | No             | No       | Yes                 | Yes                 |
| Visualization     | Basic terminal | Basic UI | Advanced dashboards | Advanced dashboards |
| API access        | Limited        | Yes      | Yes                 | Yes                 |
| Resource overhead | Very low       | Low      | Medium              | Low                 |
| Learning curve    | Minimal        | Low      | Medium              | Minimal             |

### When to Upgrade Your Monitoring Solution

Consider more robust monitoring when:

1.  **You need historical data**: For trend analysis and capacity planning
2.  **You're running multiple hosts**: For cluster-wide visibility
3.  **You need alerting**: For proactive issue detection
4.  **You require custom dashboards**: For executive/team visibility
5.  **You need to correlate with application metrics**: For end-to-end monitoring

### Enterprise Monitoring Options

- **cAdvisor**: Google's container advisor provides deep insights into container resource usage
- **Prometheus + Grafana**: The open-source standard for metric collection and visualization
- **Last9**: An observability platform with structured container monitoring at scale
- **Datadog**: Commercial solution with extensive container and Kubernetes monitoring
- **New Relic**: Full-stack observability with container insights
- **Sysdig**: Container-native monitoring and security

## Best Practices for Efficient Container Resource Management

### Setting Resource Limits

Always define resource constraints when running containers:

```bash
docker run --memory=512m --memory-swap=1g --cpus=1.5 --pids-limit=50 my_image
```

Key limit types:

- **\--memory**: Hard memory limit
- **\--memory-swap**: Combined memory and swap limit
- **\--cpus**: Number of CPU cores to allocate
- **\--pids-limit**: Maximum number of processes

### Logging Best Practices

- **Use log rotation**: Configure the Docker logging driver with size limits
- **Consider volume-mounted logs**: For high-throughput applications
- **Set log level appropriately**: Debug in development, info or warning in production
- **Monitor log volume**: Excessive logging impacts disk I/O

Example log configuration:

```bash
docker run --log-driver=json-file --log-opt max-size=10m --log-opt max-file=3 my_image
```

### Dynamic Resource Management

Use `docker update` to adjust resources without restarting containers:

```bash
docker update --memory 1G --cpus 2 my_container
```

This allows for:

- Addressing resource bottlenecks in real-time
- Scaling resources during high-traffic periods
- Reducing resource allocation for idle containers
- Testing resource configurations without downtime

### Resource Reservation vs. Limits

Understand the difference between reservations and hard limits:

- **Reservations**: Guaranteed resources (--memory-reservation, --cpu-shares)
- **Limits**: Maximum allowed resources (--memory, --cpus)

For critical services, set both to ensure consistent performance.

Keeping track of container performance means keeping an eye on logs. [Here’s a detailed guide on Docker logs](https://last9.io/blog/docker-logs/) to help you understand and manage them effectively.

## Advanced `docker stats` Techniques

### Monitoring Container Groups

To monitor containers by label:

```bash
docker stats $(docker ps --filter "label=environment=production" -q)
```

### Monitoring Based on Resource Usage

Combine with other commands to find containers based on resource usage:

```bash
# Find containers using more than 10% CPU
for id in $(docker stats --no-stream --format "{{if gt (trimSuffix \"%\" .CPUPerc) 10.0}}{{.Container}}{{end}}"); do
    docker inspect --format '{{.Name}}' $id
done
```

### Custom Monitoring Metrics

Extract advanced metrics not directly shown in `docker stats`:

```bash
# Get detailed memory information
docker inspect $(docker ps -q) --format '{{.Name}}: {{.HostConfig.Memory}}'
```

### Scheduled Resource Reports

Create a cron job for regular resource snapshots:

```bash
# Add to crontab
# */10 * * * * /usr/local/bin/docker-stats-report.sh >> /var/log/container-stats.log 2>&1
```

With a script like:

```bash
#!/bin/bash
echo "=== Container Stats $(date) ==="
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"
```

## Troubleshooting Container Performance Issues

### Identifying Memory Leaks

If memory consistently increases without returning to baseline after GC:

```bash
# Get container process details
docker top <container_id>

# Examine memory use by process within container
docker exec <container_id> ps -o pid,rss,command ax
```

Monitor memory usage over time:

```bash
watch -n 60 'docker stats --no-stream'
```

### Dealing with CPU Throttling

Examine detailed CPU stats:

```bash
docker exec <container_id> cat /sys/fs/cgroup/cpu/cpu.stat
```

Check if containers are being throttled:

```bash
docker stats --no-stream | grep -E '[8-9][0-9]\.[0-9]%|100.0%'
```

Monitoring container resources is just as important in Kubernetes. [Here’s how CPU throttling affects your workloads](https://last9.io/blog/kubernetes-cpu-throttling/) and what you can do about it.

### Network Bottlenecks

For detailed network inspection:

```bash
docker exec <container_id> netstat -tuln
```

Check network I/O:

```bash
docker stats --format "{{.Name}}: {{.NetIO}}"
```

### Disk I/O Issues

Check detailed block I/O stats:

```bash
docker exec <container_id> cat /sys/fs/cgroup/blkio/blkio.throttle.io_service_bytes
```

Identify containers with heavy disk activity:

```bash
docker stats --format "table {{.Name}}\t{{.BlockIO}}" | sort -k 2 -r
```

## Integrating `docker stats` with CI/CD Pipelines

### Performance Testing

Include resource monitoring in your CI/CD pipelines:

```yaml
# Example GitLab CI job
performance_test:
  stage: test
  script:
    - docker-compose up -d
    - sleep 10 # Wait for services to initialize
    - docker stats --no-stream > baseline_stats.txt
    - ./run-load-tests.sh
    - docker stats --no-stream > load_stats.txt
    - python3 compare_stats.py baseline_stats.txt load_stats.txt
  artifacts:
    paths:
      - baseline_stats.txt
      - load_stats.txt
      - stats_comparison.html
```

### Fail Builds on Resource Thresholds

Create a script to fail builds if containers exceed resource thresholds:

```bash
#!/bin/bash
# check_resource_usage.sh
MAX_CPU=80
MAX_MEM=75

docker stats --no-stream --format "{{.CPUPerc}},{{.MemPerc}},{{.Name}}" | while IFS=, read cpu mem name; do
  cpu_val=$(echo $cpu | sed 's/%//')
  mem_val=$(echo $mem | sed 's/%//')

  if (( $(echo "$cpu_val > $MAX_CPU" | bc -l) )); then
    echo "ERROR: Container $name exceeded CPU threshold: $cpu_val%"
    exit 1
  fi

  if (( $(echo "$mem_val > $MAX_MEM" | bc -l) )); then
    echo "ERROR: Container $name exceeded memory threshold: $mem_val%"
    exit 1
  fi
done
```

## Conclusion

`docker stats` is an invaluable tool for quick, real-time insights into container performance. While it serves as an excellent starting point for resource monitoring, production environments will benefit from more comprehensive solutions.

With the metrics provided by `docker stats` you can:

1.  **Optimize your container deployments** for maximum efficiency
2.  **Prevent outages** caused by resource exhaustion
3.  **Troubleshoot performance issues** with data-driven decisions
4.  **Plan capacity** based on actual resource requirements
5.  **Develop a monitoring strategy** that scales with your container deployment

## FAQs

### 1\. How often does `docker stats` update?

By default, `docker stats` refreshes every second. You can control the refresh rate using the `watch` command in Linux:

```bash
watch -n 5 docker stats
```

This would update every 5 seconds.

### 2\. Does `docker stats` affect container performance?

`docker stats` has minimal overhead since it reads metrics from the cgroups filesystem, which Docker already uses for resource control. The impact is negligible even in production environments.

### 3\. How can I monitor stopped containers?

`docker stats` only works on running containers. For historical data on stopped containers:

- Use `docker inspect <container_id>` for configuration details
- Check logs using `docker logs <container_id>`
- Implement persistent monitoring with tools like Prometheus

### 4\. How do I calculate the actual CPU cores used?

To convert from percentage to cores:

```
Cores used = (CPU percentage / 100) × Total available cores
```

For example, if a container shows 250% CPU usage on an 8-core system, it's using:

```
(250 / 100) = 2.5 cores
```

### 5\. Why is my container using more memory than specified?

Several reasons might explain this:

- **Kernel memory**: Used by the container but not included in user limits
- **Cache**: File system cache that can be reclaimed if needed
- **Buffer**: Memory used for I/O operations
- **Shared memory**: Memory shared between processes

To see detailed memory breakdown:

```bash
docker exec <container_id> cat /sys/fs/cgroup/memory/memory.stat
```

### 6\. How do I limit CPU or memory usage for a container?

Use resource constraints when running a container:

```bash
docker run --memory=512m --memory-reservation=256m --cpus=1 --cpu-shares=1024 my_image
```

For existing containers:

```bash
docker update --cpus=1.5 --memory=1G <container_id>
```

### 7\. Why do I see high memory usage even after stopping a container?

Docker might not release memory immediately due to:

- Cached data in the host's page cache
- Kernel memory not being fully reclaimed
- Docker daemon holding references

Try restarting the Docker daemon:

```bash
systemctl restart docker
```

### 8\. How can I export `docker stats` data to CSV?

Use a combination of formatting and redirection:

```bash
docker stats --no-stream --format "{{.Name}},{{.CPUPerc}},{{.MemUsage}},{{.MemPerc}}" > container_stats.csv
```

Add headers with:

```bash
echo "Container,CPU %,Memory Usage,Memory %" > container_stats.csv && docker stats --no-stream --format "{{.Name}},{{.CPUPerc}},{{.MemUsage}},{{.MemPerc}}" >> container_stats.csv
```