Kubernetes Events

Ingest Kubernetes events — scheduling, rollouts, failures, state changes — into Last9.

Prerequisites

• A running Kubernetes cluster (v1.19+)
• helm v3.9+ installed
• Cluster admin access
• Last9 OTLP credentials from the Integrations page

Installation

curl -O https://raw.githubusercontent.com/last9/last9-k8s-observability/main/last9-otel-setup.sh
chmod +x last9-otel-setup.sh

./last9-otel-setup.sh events-only \
  endpoint="$last9_otlp_endpoint" \
  token="$last9_otlp_auth_header" \
  monitoring-endpoint="$last9_remote_write_url"

Verification

kubectl get pods -n last9
kubectl logs -n last9 deployment/last9-kube-events-agent

Visit the Logs Explorer to see events flowing in.

Filter by the event service name or Kubernetes attributes to narrow the stream:

• service.name = "kubernetes-events"
• k8s.namespace.name = "default"
• k8s.pod.name = "<pod-name>"
• k8s.node.name = "<node-name>"

Open an event log line and check Resource Info and event attributes. The most useful fields for correlation are the involved object name, namespace, node, reason, message, event type, and event timestamps.

Understanding Event Fields

Kubernetes events are operational context. They explain what the control plane observed, but they do not automatically prove application root cause.

Common Investigation Workflows

Pod Restart or OOM

1. Open the affected pod in Discover Kubernetes and note the restart time window.
2. Filter Kubernetes events by k8s.pod.name and the same time range.
3. Look for reasons such as Killing, BackOff, OOMKilling, or probe failures.
4. Correlate with container memory, CPU, and application logs before classifying the issue.

Node Scale-Down or Drain

1. Filter events by k8s.node.name and the scale-down time window.
2. Look for node cordon, drain, eviction, and pod rescheduling messages.
3. Follow the affected pod names to confirm where replacements were scheduled.
4. Compare application error rate or latency during the same window to decide whether user traffic was affected.

HPA or Autoscaler Activity

1. Filter by namespace and deployment during the scaling window.
2. Look for replica changes, scheduling events, and resource-pressure warnings.
3. Compare the event timeline with CPU, memory, throughput, and latency charts.
4. Treat the event as scaling context, then confirm service impact through metrics and traces.

Failed Scheduling

1. Filter for reason = "FailedScheduling" or Warning events in the namespace.
2. Check the message for resource, taint, affinity, node selector, or quota constraints.
3. Compare the pod’s requested CPU and memory with node capacity and namespace limits.

Uninstall

./last9-otel-setup.sh uninstall function="uninstall_events_agent"

Troubleshooting

No Events in Logs Explorer

• Confirm the agent is running: kubectl get pods -n last9.
• Check agent logs: kubectl logs -n last9 deployment/last9-kube-events-agent.
• Generate a fresh event after installation. Historical Kubernetes events are not replayed indefinitely.
• Verify the Last9 OTLP endpoint and token values used during setup.

Timestamp Parsing Warnings

Some Kubernetes event payloads can include timestamp fields in formats that differ across Kubernetes versions and event sources. If the agent logs timestamp parsing warnings, first collect diagnostics instead of applying a workaround blindly:

kubectl logs -n last9 deployment/last9-kube-events-agent
kubectl get events --all-namespaces --sort-by=.lastTimestamp
kubectl get events --all-namespaces -o yaml

Compare the warning text with the event’s eventTime, firstTimestamp, lastTimestamp, and metadata timestamps. If events still appear in Last9, the warning may affect only some event records. If no events appear, share the warning text and a redacted event sample with Last9 support.

Please get in touch with us on Discord or Email if you have any questions.