Sidecar Log

Ingest application logs via Kubernetes Sidecar Container using OpenTelemetry. This setup is ideal when your application logs to custom files apart from STDOUT/STDERR, providing dedicated log collection for each application pod.

For applications logging to standard STDOUT and STDERR, refer to our Kubernetes Cluster Monitoring integration instead.

Prerequisites

Before setting up sidecar log collection, ensure you have:

• Kubernetes Cluster: A running Kubernetes cluster (v1.19+)
• kubectl: Configured and connected to your cluster
• Application Logging to Files: Your application must write logs to files (not just STDOUT/STDERR)
• Last9 Account: With log ingestion credentials

When to Use Sidecar Log Collection

Use sidecar log collection when:

• ✅ File-based Logging: Your application writes logs to specific files
• ✅ Custom Log Formats: Need specialized parsing or transformation
• ✅ Per-Application Configuration: Different log collection settings per service
• ✅ Legacy Applications: Applications that cannot be modified to log to STDOUT
• ✅ Multi-file Logging: Applications writing to multiple log files

Don’t use sidecars when:

• ❌ Applications only log to STDOUT/STDERR (use cluster-wide collection instead)
• ❌ Simple logging needs (adds complexity and resource overhead)

1. Understand the Sidecar Architecture

   The sidecar pattern involves:

   • Shared Volume: Both app and sidecar containers mount the same volume
   • Log File Sharing: Application writes logs, sidecar reads and forwards them
   • Independent Processing: Each pod has its own log collector
   • Resource Isolation: Collector issues don’t affect other applications

   ┌─────────────────────────────────────┐
   │              Pod                    │
   │  ┌─────────────┐  ┌─────────────┐    │
   │  │     App     │  │   Sidecar   │    │
   │  │ Container   │  │  Collector  │    │
   │  │             │  │             │    │
   │  │   writes    │  │    reads    │    │
   │  │     ↓       │  │     ↓       │    │
   │  └─────────────┘  └─────────────┘    │
   │         │               │            │
   │         └───────────────┘            │
   │            Shared Volume             │
   │              (/log)                  │
   └─────────────────────────────────────┘

2. Create OpenTelemetry Collector Configuration

   Create a file named last9-otelcol-config.yaml with the collector configuration:

   receivers:
     filelog:
       # File path pattern to read logs from. Update this to match your log files
       include: [/log/*.log]
       include_file_path: true
       # Optional: Add static attributes to all log records
       # attributes:
       #   log_type: "application"
       #   service_version: "1.0.0"
       # Optional: Add static resource attributes
       # resource:
       #   deployment.environment: "production"
       retry_on_failure:
         enabled: true
   
   processors:
     batch:
       timeout: 10s
       send_batch_size: 10000
       send_batch_max_size: 10000
     transform/logs:
       error_mode: ignore
       log_statements:
         - context: resource
           statements:
             - set(attributes["service.name"], "my-app") # Service name
             - set(attributes["deployment.environment"], "staging") # Deployment Environment
     resourcedetection/system:
       detectors: ["system"]
       system:
         hostname_sources: ["os"]
   
   exporters:
     otlp/last9:
       endpoint: "{{ .Logs.WriteURL }}"
       headers:
         "Authorization": "{{ .Logs.AuthValue }}"
   
   service:
     pipelines:
       logs:
         receivers: [filelog]
         processors: [batch, transform/logs, resourcedetection/system]
         exporters: [otlp/last9]

   Replace the placeholder values with your actual Last9 credentials from the Last9 Integrations page.

3. Create the ConfigMap

   Create a Kubernetes ConfigMap containing the collector configuration:

   kubectl create configmap last9-otelcol-config \
     --from-file=last9-otelcol-config.yaml \
     -n <your-namespace>

   Verify the ConfigMap was created correctly:

   kubectl get configmap last9-otelcol-config -n <your-namespace> -o yaml

4. Update Your Application Deployment

   Modify your application deployment to include the sidecar collector. Here’s an example deployment:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-app-with-sidecar
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: my-app
     template:
       metadata:
         labels:
           app: my-app
       spec:
         containers:
           # Your main application container
           - name: my-app
             image: my-application:latest
             ports:
               - containerPort: 8080
             volumeMounts:
               - name: log-volume
                 mountPath: /log
             env:
               - name: LOG_PATH
                 value: "/log/application.log"
   
           # OpenTelemetry Collector sidecar
           - name: last9-otel-collector
             image: otel/opentelemetry-collector-contrib:0.126.0
             command: ["/otelcol-contrib"]
             args: ["--config=/etc/otelcol-config.yaml"]
             volumeMounts:
               - name: log-volume
                 mountPath: /log
               - name: last9-otelcol-config
                 mountPath: /etc/otelcol-config.yaml
                 subPath: last9-otelcol-config.yaml
             resources:
               limits:
                 cpu: 200m
                 memory: 256Mi
               requests:
                 cpu: 100m
                 memory: 128Mi
   
         volumes:
           # Shared volume for log files
           - name: log-volume
             emptyDir: {}
           # ConfigMap containing collector configuration
           - name: last9-otelcol-config
             configMap:
               name: last9-otelcol-config

   Important Volume Considerations:

   • emptyDir Volume: Creates a temporary directory shared between containers
   • Path Selection: Mount to dedicated paths like /log or /tmp/shared
   • Avoid Code Paths: Never mount emptyDir to paths containing application code
   • File Permissions: Ensure both containers can read/write to the shared volume

5. Deploy Your Application

   Apply the deployment with the sidecar configuration:

   kubectl apply -f my-app-with-sidecar.yaml -n <your-namespace>

   Verify both containers are running:

   kubectl get pods -n <your-namespace>
   kubectl describe pod <pod-name> -n <your-namespace>

Configuration Customization

Multiple Log Files

To collect from multiple log files or patterns:

receivers:
  filelog:
    include:
      - /log/*.log
      - /log/errors/*.log
      - /log/access/*.log
    exclude:
      - /log/debug/*.log # Exclude debug logs
    include_file_path: true

Custom Log Parsing

Add parsing for structured logs:

receivers:
  filelog:
    include: [/log/*.log]
    operators:
      - type: json_parser
        id: json_parser
        if: 'body matches "^\\{"'
      - type: timestamp
        id: time_parser
        parse_from: attributes.timestamp
        layout: "2006-01-02T15:04:05.000Z"

Environment-Specific Configuration

Use different configurations per environment:

transform/logs:
  log_statements:
    - context: resource
      statements:
        - set(attributes["service.name"], "my-app")
        - set(attributes["deployment.environment"], env("DEPLOYMENT_ENV"))
        - set(attributes["service.version"], env("APP_VERSION"))

Resource Attributes

Add comprehensive metadata to logs:

transform/logs:
  log_statements:
    - context: resource
      statements:
        - set(attributes["service.name"], "user-service")
        - set(attributes["service.version"], "2.1.0")
        - set(attributes["deployment.environment"], "production")
        - set(attributes["team"], "backend")
        - set(attributes["region"], "us-east-1")

Verification

1. Check Pod Status

   Verify both containers are running correctly:

   kubectl get pods -n <your-namespace>
   kubectl logs <pod-name> -c my-app -n <your-namespace>
   kubectl logs <pod-name> -c last9-otel-collector -n <your-namespace>

2. Generate Test Logs

   Generate some test log entries from your application:

   # If your app exposes an endpoint that generates logs
   kubectl exec -it <pod-name> -c my-app -n <your-namespace> -- curl localhost:8080/test
   
   # Or write directly to log file for testing
   kubectl exec -it <pod-name> -c my-app -n <your-namespace> -- sh -c "echo 'Test log entry' >> /log/application.log"

3. Verify File Sharing

   Check that log files are visible to both containers:

   # Check from app container
   kubectl exec -it <pod-name> -c my-app -n <your-namespace> -- ls -la /log/
   
   # Check from collector container
   kubectl exec -it <pod-name> -c last9-otel-collector -n <your-namespace> -- ls -la /log/

4. Monitor Collector Logs

   Check collector logs for successful log processing:

   kubectl logs <pod-name> -c last9-otel-collector -n <your-namespace> | grep -i "logs sent"

5. Verify Logs in Last9

   Log into your Last9 account and check that logs are being received in the Logs dashboard.

Best Practices

Resource Management

• Set Resource Limits: Always set CPU and memory limits for sidecar containers
• Right-sizing: Monitor actual usage and adjust limits accordingly
• Log Rotation: Implement log rotation to prevent disk space issues

resources:
  limits:
    cpu: 200m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 128Mi

Volume Configuration

• Use emptyDir for Temporary Logs: Perfect for application logs that don’t need persistence
• Use PersistentVolumes for Important Logs: When logs need to survive pod restarts
• Avoid Mounting to App Directories: Never mount volumes to paths containing application code

Application Configuration

env:
  - name: LOG_LEVEL
    value: "INFO"
  - name: LOG_PATH
    value: "/log/app.log"
  - name: LOG_FORMAT
    value: "json" # Use structured logging when possible

Troubleshooting

Sidecar Container Not Starting

Check collector configuration and image availability:

kubectl describe pod <pod-name> -n <your-namespace>
kubectl logs <pod-name> -c last9-otel-collector -n <your-namespace>

No Logs Being Collected

Verify file paths and permissions:

# Check if log files exist and are readable
kubectl exec -it <pod-name> -c last9-otel-collector -n <your-namespace> -- ls -la /log/

# Check collector configuration
kubectl exec -it <pod-name> -c last9-otel-collector -n <your-namespace> -- cat /etc/otelcol-config.yaml

High Resource Usage

Monitor and optimize resource consumption:

kubectl top pods -n <your-namespace>
kubectl describe pod <pod-name> -n <your-namespace>

ConfigMap Updates Not Reflected

Restart pods after ConfigMap changes:

kubectl rollout restart deployment/my-app-with-sidecar -n <your-namespace>

Alternative Approaches

Consider these alternatives based on your needs:

• DaemonSet Collectors: For cluster-wide log collection from node filesystems
• Fluent Bit Sidecar: If you prefer Fluent Bit over OpenTelemetry
• Init Container + Volume: For one-time log file setup scenarios
• External Log Agents: For environments with existing log infrastructure

Need Help?

If you encounter any issues or have questions:

• Join our Discord community for real-time support
• Contact our support team at support@last9.io