Using Kubectl Logs: Guide to Viewing Kubernetes Pod Logs

As a developer working with Kubernetes, mastering kubectl logs is crucial for debugging and monitoring your applications. This comprehensive guide will walk you through everything you need about this powerful command.

The kubectl logs command retrieves container logs from pods in your Kubernetes cluster. It's your go-to tool for peeking into what's happening inside your containers without SSH access.

What is kubectl?

Kubectl is the command-line interface for interacting with Kubernetes clusters. It allows developers and administrators to deploy applications, inspect and manage cluster resources, and view logs.

Kubernetes Architecture and Logging Internals Explained

Before diving into the specifics of kubectl logs, it's crucial to understand the basics of Kubernetes architecture and how logging fits into this ecosystem.

Kubernetes Architecture: Key Components and Their Roles

Kubernetes follows a distributed architecture with several key components:

1. Master Node(s):
   • API Server: The central management entity that receives all REST requests.
   • etcd: A distributed key-value store that stores all cluster data.
   • Scheduler: Assigns pods to nodes.
   • Controller Manager: Manages various controllers that regulate the state of the cluster.
2. Worker Nodes:
   • Kubelet: Ensures containers are running in a pod.
   • Container Runtime: Software responsible for running containers (e.g., Docker, containerd).
   • Kube-proxy: Maintains network rules on nodes.
3. Pods: The smallest deployable units in Kubernetes, containing one or more containers.

How Logging Works in Kubernetes

Kubernetes doesn't provide a native, cluster-wide logging solution. Instead, it relies on the underlying container runtime to capture stdout and stderr streams from each container. Here's how logging works:

1. Container Logs:
   • Applications inside containers write logs to stdout and stderr.
   • The container runtime captures these streams and writes them to files on the node's disk.
2. Node-level Logging:
   • Kubelet on each node is responsible for exposing these logs to the Kubernetes API.
   • Log files are typically stored /var/log/pods/ on the node.
3. Cluster-level Access:
   • The Kubernetes API server provides endpoints to access these logs.
   • kubectl logs command uses these endpoints to retrieve logs from specific pods or containers.
4. Log Lifecycle:
   • Logs are tied to the lifecycle of the container. When a container is deleted, its logs are typically deleted as well.
   • Some container runtimes implement log rotation to manage disk space.
5. Advanced Logging Solutions:
   • For production environments, cluster-wide logging solutions (like Levitate, ElasticSearch, and Kibana stack) are often implemented to aggregate logs from all nodes and provide advanced search and analysis capabilities.

Understanding this architecture is crucial for effective log management and troubleshooting in Kubernetes environments. With this context, let's explore how kubectl logs interface with this system to provide access to container logs.

Getting Started with kubectl logs

The simplest form of the command is:

kubectl logs <pod-name>

For multi-container pods, specify the container:

kubectl logs <pod-name> -c <container-name>

Expected output:

2023-08-24 10:15:30 INFO  Application started
2023-08-24 10:15:31 DEBUG Connecting to database
2023-08-24 10:15:32 INFO  Database connection established

Advanced Techniques

Tailing logs in real-time

To stream logs continuously (similar to tail -f):

kubectl logs <pod-name> -f

This is particularly useful for monitoring during deployments or troubleshooting live issues.

Viewing logs from previous instances

For pods that have restarted:

kubectl logs <pod-name> --previous

Limiting output

Limit the number of lines:

kubectl logs <pod-name> --tail=100

Viewing logs from multiple pods

Use label selectors:

kubectl logs -l app=myapp

Including timestamps

kubectl logs <pod-name> --timestamps

Time-based log retrieval

View logs from a specific time:

kubectl logs <pod-name> --since=1h

View logs up to a specific time:

kubectl logs <pod-name> --until=2023-08-24T10:00:00Z

Understanding stdout vs stderr

Kubernetes captures both stdout and stderr streams. To see only stderr logs:

kubectl logs <pod-name> --container <container-name> --stderr

Where Does kubectl logs Retrieve Logs From?

Understanding where kubectl logs retrieve their information is crucial for effective log management and troubleshooting in Kubernetes. Let's dive deeper into this process:

Container Runtime Log Files

1. Local Node Storage: When you run kubectl logs, it reads log files directly from the local storage of the node where the pod is running. These log files are typically stored in a directory on the node's filesystem.
2. Container Runtime: The exact location and format of these log files depending on the container runtime being used (e.g., Docker, containerd, CRI-O). For example:
   • Docker: /var/lib/docker/containers/<container-id>/<container-id>-json.log
   • containerd: /var/log/pods/<pod-uid>/<container-name>/0.log
   • CRI-O: /var/log/pods/<pod-namespace>_<pod-name>_<pod-uid>/<container-name>/0.log
3. Log Rotation: The container runtime usually handles log rotation to prevent log files from consuming too much disk space. The rotation policy can affect how far back you can retrieve logs using kubectl logs.

Kubernetes API Server

1. API Request: When you run kubectl logs, it sends a request to the Kubernetes API server.
2. Kubelet Communication: The API server then communicates with the kubelet on the specific node where the pod is running.
3. Kubelet Retrieval: The kubelet accesses the container runtime to fetch the requested logs.
4. Stream Back: The logs are then streamed back through the API server to your kubectl client.

stdout and stderr Streams

1. Standard Streams: Kubernetes captures logs that are written to the stdout (standard output) and stderr (standard error) streams of the container processes.
2. Application Logging: This means that your application needs to write its logs to stdout/stderr for them to be accessible via kubectl logs. Logging to files inside the container won't be captured unless you've set up a logging sidecar.

Logging Drivers

1. Container Runtime Logging: The container runtime uses logging drivers to capture the stdout/stderr streams and write them to files.
2. Driver Configuration: The logging driver can be configured to affect how logs are stored and rotated. For example, Docker's JSON file driver stores logs in JSON format with metadata.

Important Considerations for Log Management in Kubernetes

1. Log Persistence: Logs are only available for the lifetime of the pod's containers. Once a container is terminated and removed, its logs are no longer accessible through kubectl logs.
2. Previous Containers: The --previous flag allows you to access logs from a previous instance of a container, but only if the pod is still present on the node.
3. Node Failures: If a node fails, logs stored locally on that node become inaccessible. This is one reason why centralized logging solutions are often used in production environments.
4. Performance Impact: Excessive logging or frequent log queries can impact the performance of the node and the Kubernetes API server.
5. Security: Access to logs is controlled by Kubernetes RBAC (Role-Based Access Control). Users need appropriate permissions to view logs from pods.

Additional Topics to Explore

• Kubernetes Pods: Understanding pod concepts for effective log management.
• Log Rotation: Learn about Kubernetes log rotation policies.
• Exposing Logs: Methods to expose logs for external monitoring systems.
• Open Telemetry Logs: Using Open Telemetry send your logs to a central system like Levitate

Quick Reference (Cheat Sheet)

# Basic log viewing
kubectl logs <pod-name>

# View logs from a specific container
kubectl logs <pod-name> -c <container-name>

# Tail logs in real-time
kubectl logs <pod-name> -f

# View previous instance logs
kubectl logs <pod-name> --previous

# Limit log output
kubectl logs <pod-name> --tail=100

# View logs from multiple pods
kubectl logs -l app=myapp

# Include timestamps
kubectl logs <pod-name> --timestamps

# View logs since a specific time
kubectl logs <pod-name> --since=1h

# View logs until a specific time
kubectl logs <pod-name> --until=2023-08-24T10:00:00Z

# View only stderr logs
kubectl logs <pod-name> --container <container-name> --stderr

Enhance Your Kubernetes Logging with Levitate Logs

While kubectl logs are powerful, managing logs at scale can be challenging. Levitate Logs offers a robust solution for centralized log management in Kubernetes environments.

Key features include:

1. Aggregation: Collect logs from all pods and containers.
2. Search and Filtering: Advanced options for log analysis.
3. Real-time Streaming/ Live Tail: View logs in real-time across your cluster. Just like your local tail -f you can use tail your logs through levitate directly in the browser.
4. Retention: Store logs for extended periods.
5. Alerts: Set up alerts based on log patterns including Scheduled Search.
6. Visualization: Create dashboards for log data trends through Grafana.
7. Integration: Seamless integration with Kubernetes infrastructure through helm charts and open telemetry support.

Levitate Logs addresses many limitations of kubectl logs, especially in complex, distributed systems.

Conclusion

Mastering kubectl logs is essential for effective Kubernetes development and operations. As your applications grow, consider complementing them with advanced logging solutions like Levitate Logs.

By understanding the various options and use cases for kubectl logs, you'll be better equipped to diagnose issues, monitor applications, and ensure smooth Kubernetes deployments.

Explore related topics like pod management, log rotation, and advanced Kubernetes commands to enhance your overall expertise including log aggregation.

Happy logging!

FAQs related to kubectl