# Kubernetes Sidecar for Logs

> Collect custom application logs using OpenTelemetry sidecar containers for applications logging to files

Source: https://last9.io/docs/integrations/containers-and-k8s/kubernetes-sidecar-for-logs/

Ingest application logs via Kubernetes Sidecar Container using OpenTelemetry. This setup is ideal when your application logs to custom files apart from STDOUT/STDERR, providing dedicated log collection for each application pod.

For applications logging to standard STDOUT and STDERR, refer to our [Kubernetes Cluster Monitoring](/docs/integrations/containers-and-k8s/kubernetes-cluster-monitoring/) integration instead.

## Prerequisites

Before setting up sidecar log collection, ensure you have:

- **Kubernetes Cluster**: A running Kubernetes cluster (v1.19+)
- **kubectl**: Configured and connected to your cluster
- **Application Logging to Files**: Your application must write logs to files (not just STDOUT/STDERR)
- **Last9 Account**: With log ingestion credentials

## When to Use Sidecar Log Collection

Use sidecar log collection when:

- ✅ **File-based Logging**: Your application writes logs to specific files
- ✅ **Custom Log Formats**: Need specialized parsing or transformation
- ✅ **Per-Application Configuration**: Different log collection settings per service
- ✅ **Legacy Applications**: Applications that cannot be modified to log to STDOUT
- ✅ **Multi-file Logging**: Applications writing to multiple log files

**Don't use sidecars when:**

- ❌ Applications only log to STDOUT/STDERR (use cluster-wide collection instead)
- ❌ Simple logging needs (adds complexity and resource overhead)

1. **Understand the Sidecar Architecture**

   The sidecar pattern involves:

   - **Shared Volume**: Both app and sidecar containers mount the same volume
   - **Log File Sharing**: Application writes logs, sidecar reads and forwards them
   - **Independent Processing**: Each pod has its own log collector
   - **Resource Isolation**: Collector issues don't affect other applications

   ```
   ┌─────────────────────────────────────┐
   │              Pod                    │
   │  ┌─────────────┐  ┌─────────────┐    │
   │  │     App     │  │   Sidecar   │    │
   │  │ Container   │  │  Collector  │    │
   │  │             │  │             │    │
   │  │   writes    │  │    reads    │    │
   │  │     ↓       │  │     ↓       │    │
   │  └─────────────┘  └─────────────┘    │
   │         │               │            │
   │         └───────────────┘            │
   │            Shared Volume             │
   │              (/log)                  │
   └─────────────────────────────────────┘
   ```

2. **Create OpenTelemetry Collector Configuration**

   Create a file named `last9-otelcol-config.yaml` with the collector configuration:

   ```yaml
   receivers:
     filelog:
       # File path pattern to read logs from. Update this to match your log files
       include: [/log/*.log]
       include_file_path: true
       # Optional: Add static attributes to all log records
       # attributes:
       #   log_type: "application"
       #   service_version: "1.0.0"
       # Optional: Add static resource attributes
       # resource:
       #   deployment.environment: "production"
       retry_on_failure:
         enabled: true

   processors:
     batch:
       timeout: 10s
       send_batch_size: 10000
       send_batch_max_size: 10000
     transform/logs:
       error_mode: ignore
       log_statements:
         - context: resource
           statements:
             - set(attributes["service.name"], "my-app") # Service name
             - set(attributes["deployment.environment"], "staging") # Deployment Environment
     resourcedetection/system:
       detectors: ["system"]
       system:
         hostname_sources: ["os"]

   exporters:
     otlp/last9:
       endpoint: "{{ .Logs.WriteURL }}"
       headers:
         "Authorization": "{{ .Logs.AuthValue }}"

   service:
     pipelines:
       logs:
         receivers: [filelog]
         processors: [batch, transform/logs, resourcedetection/system]
         exporters: [otlp/last9]
   ```

   Replace the placeholder values with your actual Last9 credentials from the Last9 Integrations page.

3. **Create the ConfigMap**

   Create a Kubernetes ConfigMap containing the collector configuration:

   ```bash
   kubectl create configmap last9-otelcol-config \
     --from-file=last9-otelcol-config.yaml \
     -n <your-namespace>
   ```

   Verify the ConfigMap was created correctly:

   ```bash
   kubectl get configmap last9-otelcol-config -n <your-namespace> -o yaml
   ```

4. **Update Your Application Deployment**

   Modify your application deployment to include the sidecar collector. Here's an example deployment:

   ```yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-app-with-sidecar
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: my-app
     template:
       metadata:
         labels:
           app: my-app
       spec:
         containers:
           # Your main application container
           - name: my-app
             image: my-application:latest
             ports:
               - containerPort: 8080
             volumeMounts:
               - name: log-volume
                 mountPath: /log
             env:
               - name: LOG_PATH
                 value: "/log/application.log"

           # OpenTelemetry Collector sidecar
           - name: last9-otel-collector
             image: otel/opentelemetry-collector-contrib:0.126.0
             command: ["/otelcol-contrib"]
             args: ["--config=/etc/otelcol-config.yaml"]
             volumeMounts:
               - name: log-volume
                 mountPath: /log
               - name: last9-otelcol-config
                 mountPath: /etc/otelcol-config.yaml
                 subPath: last9-otelcol-config.yaml
             resources:
               limits:
                 cpu: 200m
                 memory: 256Mi
               requests:
                 cpu: 100m
                 memory: 128Mi

         volumes:
           # Shared volume for log files
           - name: log-volume
             emptyDir: {}
           # ConfigMap containing collector configuration
           - name: last9-otelcol-config
             configMap:
               name: last9-otelcol-config
   ```

   **Important Volume Considerations:**

   - **emptyDir Volume**: Creates a temporary directory shared between containers
   - **Path Selection**: Mount to dedicated paths like `/log` or `/tmp/shared`
   - **Avoid Code Paths**: Never mount emptyDir to paths containing application code
   - **File Permissions**: Ensure both containers can read/write to the shared volume

5. **Deploy Your Application**

   Apply the deployment with the sidecar configuration:

   ```bash
   kubectl apply -f my-app-with-sidecar.yaml -n <your-namespace>
   ```

   Verify both containers are running:

   ```bash
   kubectl get pods -n <your-namespace>
   kubectl describe pod <pod-name> -n <your-namespace>
   ```

## Configuration Customization

### Multiple Log Files

To collect from multiple log files or patterns:

```yaml
receivers:
  filelog:
    include:
      - /log/*.log
      - /log/errors/*.log
      - /log/access/*.log
    exclude:
      - /log/debug/*.log # Exclude debug logs
    include_file_path: true
```

### Custom Log Parsing

Add parsing for structured logs:

```yaml
receivers:
  filelog:
    include: [/log/*.log]
    operators:
      - type: json_parser
        id: json_parser
        if: 'body matches "^\\{"'
      - type: timestamp
        id: time_parser
        parse_from: attributes.timestamp
        layout: "2006-01-02T15:04:05.000Z"
```

### Environment-Specific Configuration

Use different configurations per environment:

```yaml
transform/logs:
  log_statements:
    - context: resource
      statements:
        - set(attributes["service.name"], "my-app")
        - set(attributes["deployment.environment"], env("DEPLOYMENT_ENV"))
        - set(attributes["service.version"], env("APP_VERSION"))
```

### Resource Attributes

Add comprehensive metadata to logs:

```yaml
transform/logs:
  log_statements:
    - context: resource
      statements:
        - set(attributes["service.name"], "user-service")
        - set(attributes["service.version"], "2.1.0")
        - set(attributes["deployment.environment"], "production")
        - set(attributes["team"], "backend")
        - set(attributes["region"], "us-east-1")
```

## Verification

1. **Check Pod Status**

   Verify both containers are running correctly:

   ```bash
   kubectl get pods -n <your-namespace>
   kubectl logs <pod-name> -c my-app -n <your-namespace>
   kubectl logs <pod-name> -c last9-otel-collector -n <your-namespace>
   ```

2. **Generate Test Logs**

   Generate some test log entries from your application:

   ```bash
   # If your app exposes an endpoint that generates logs
   kubectl exec -it <pod-name> -c my-app -n <your-namespace> -- curl localhost:8080/test

   # Or write directly to log file for testing
   kubectl exec -it <pod-name> -c my-app -n <your-namespace> -- sh -c "echo 'Test log entry' >> /log/application.log"
   ```

3. **Verify File Sharing**

   Check that log files are visible to both containers:

   ```bash
   # Check from app container
   kubectl exec -it <pod-name> -c my-app -n <your-namespace> -- ls -la /log/

   # Check from collector container
   kubectl exec -it <pod-name> -c last9-otel-collector -n <your-namespace> -- ls -la /log/
   ```

4. **Monitor Collector Logs**

   Check collector logs for successful log processing:

   ```bash
   kubectl logs <pod-name> -c last9-otel-collector -n <your-namespace> | grep -i "logs sent"
   ```

5. **Verify Logs in Last9**

   Log into your Last9 account and check that logs are being received in the [Logs Explorer](https://app.last9.io/logs).

## Best Practices

### Resource Management

- **Set Resource Limits**: Always set CPU and memory limits for sidecar containers
- **Right-sizing**: Monitor actual usage and adjust limits accordingly
- **Log Rotation**: Implement log rotation to prevent disk space issues

```yaml
resources:
  limits:
    cpu: 200m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 128Mi
```

### Volume Configuration

- **Use emptyDir for Temporary Logs**: Perfect for application logs that don't need persistence
- **Use PersistentVolumes for Important Logs**: When logs need to survive pod restarts
- **Avoid Mounting to App Directories**: Never mount volumes to paths containing application code

### Application Configuration

```yaml
env:
  - name: LOG_LEVEL
    value: "INFO"
  - name: LOG_PATH
    value: "/log/app.log"
  - name: LOG_FORMAT
    value: "json" # Use structured logging when possible
```

---

## Troubleshooting

- **Sidecar Container Not Starting**

  Check collector configuration and image availability:

  ```bash
  kubectl describe pod <pod-name> -n <your-namespace>
  kubectl logs <pod-name> -c last9-otel-collector -n <your-namespace>
  ```

- **No Logs Being Collected**

  Verify file paths and permissions:

  ```bash
  # Check if log files exist and are readable
  kubectl exec -it <pod-name> -c last9-otel-collector -n <your-namespace> -- ls -la /log/

  # Check collector configuration
  kubectl exec -it <pod-name> -c last9-otel-collector -n <your-namespace> -- cat /etc/otelcol-config.yaml
  ```

- **High Resource Usage**

  Monitor and optimize resource consumption:

  ```bash
  kubectl top pods -n <your-namespace>
  kubectl describe pod <pod-name> -n <your-namespace>
  ```

- **ConfigMap Updates Not Reflected**

  Restart pods after ConfigMap changes:

  ```bash
  kubectl rollout restart deployment/my-app-with-sidecar -n <your-namespace>
  ```

- **Alternative Approaches**

  Consider these based on your needs:

  - **DaemonSet Collectors**: For cluster-wide log collection from node filesystems
  - **Fluent Bit Sidecar**: If you prefer Fluent Bit over OpenTelemetry
  - **Init Container + Volume**: For one-time log file setup scenarios
  - **External Log Agents**: For environments with existing log infrastructure

Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:support@last9.io) if you have any questions.