# kube-state-metrics: Your Guide to Kubernetes Observability

> This guide provides an in-depth look at its setup and usage, helping you monitor and manage your Kubernetes clusters more efficiently.

Source: https://last9.io/blog/kube-state-metrics/

As a DevOps engineer with years of experience optimizing Kubernetes clusters, I've come to appreciate the critical role that robust monitoring solutions play in maintaining healthy, efficient systems.

Among the myriad tools available, kube-state-metrics stands out as an indispensable component of any comprehensive Kubernetes observability stack. In this in-depth guide, I'll share my experiences and insights on leveraging this powerful exporter to enhance your Kubernetes monitoring capabilities.

## **Understanding kube-state-metrics**

### **What is kube-state-metrics?**

[kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) is an open-source add-on for Kubernetes that listens to the Kubernetes API server and generates metrics about the state of various Kubernetes objects.

Unlike [metrics-server](https://github.com/kubernetes-sigs/metrics-server), which provides resource usage data (CPU, memory), kube-state-metrics focuses on the health and status of Kubernetes resources.

### **Key Features of kube-state-metrics**

1.  **Object-focused metrics**: Provides metrics for a wide range of Kubernetes objects, including Pods, Deployments, StatefulSets, and more.
2.  **Read-only access**: Only requires read-only access to the Kubernetes API, enhancing security.
3.  **Prometheus compatibility**: Exposes metrics in Prometheus format, making it easy to integrate with existing monitoring stacks.
4.  **Custom resource support**: Can be extended to monitor custom resources (CRDs).

Also read: [How to Use Kubectl Logs for Viewing Kubernetes Pod Logs](https://last9.io/blog/using-kubectl-logs-guide-to-viewing-kubernetes-pod-logs/)

### **Why kube-state-metrics Matters**

In my years working with Kubernetes clusters, I've found that while tools like kubelet and metrics-server provide crucial CPU and memory usage data, they don't give the full picture.

kube-state-metrics fills this gap by offering insights into the state of deployments, pods, nodes, and other Kubernetes objects.

_For example,_ while metrics-server might tell you that a node is using 80% of its CPU, kube-state-metrics can tell you how many pods are running on that node, how many are in a failed state, or whether there are any pending pods due to resource constraints.

## **Setting Up kube-state-metrics**

Let's understand the process of setting up kube-state-metrics in a Kubernetes cluster:

### **Installation Options**

- [Using Helm](#using-helm-recommended)
- [Manual Installation using YAML Manifests](#manual-installation-using-yaml-manifests)
- [Building from Source](#building-from-source)

I prefer using Helm, as it simplifies the process and manages updates efficiently.

### Using Helm (Recommended)

**First, add the Prometheus community repo:**

```bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update
```

**Then, install kube-state-metrics:**

```bash
helm install kube-state-metrics prometheus-community/kube-state-metrics
```

**For more customization options, you can create values.yaml file:**

```yaml
replicas: 2
autosharding:
  enabled: true
podSecurityPolicy:
  enabled: true
```

**Then install with:**

```bash
helm install kube-state-metrics prometheus-community/kube-state-metrics -f values.yaml
```

Also read: [Monitoring Kubernetes with Prometheus and Grafana](https://last9.io/blog/kubernetes-monitoring-with-prometheus-and-grafana/)

### Manual Installation using YAML Manifests

If you prefer manual installation or need more control over the deployment, you can find the YAML manifests in the official [kube-state-metrics GitHub repo](https://github.com/kubernetes/kube-state-metrics/tree/master/examples/standard).

**Clone the repository:**

```bash
git clone https://github.com/kubernetes/kube-state-metrics.git
```

1.  cd kube-state-metrics
2.  Apply the manifests:

```bash
kubectl apply -f examples/standard
```

### **Configuration and RBAC**

After installation, kube-state-metrics will create a ServiceAccount, ClusterRole, and ClusterRoleBinding to ensure it has the necessary permissions to access the Kubernetes API server.

Here's a snippet of the ClusterRole YAML:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
  - apiGroups: [""]
    resources:
      - configmaps
      - secrets
      - nodes
      - pods
      - services
      - resourcequotas
      - replicationcontrollers
      - limitranges
      - persistentvolumeclaims
      - persistentvolumes
      - namespaces
      - endpoints
    verbs: ["list", "watch"]
  - apiGroups: ["apps"]
    resources:
      - statefulsets
      - daemonsets
      - deployments
      - replicasets
    verbs: ["list", "watch"]
# ... more rules ...
```

This RBAC configuration ensures that kube-state-metrics has read-only access to the necessary Kubernetes resources.

**To verify the installation, run:**

```bash
kubectl get pods -n kube-system | grep kube-state-metrics
```

You should see the kube-state-metrics pod(s) running.

### Building from Source

Building kube-state-metrics from the source allows you to customize the application or use the latest features not yet released in the official binaries. Follow these steps to build kube-state-metrics from the source:

#### Prerequisites

Before you begin, ensure you have the following installed:

- **Go**: Make sure you have Go installed (version 1.18 or later is recommended). You can download it from the [official Go website](https://golang.org/dl/).
- **Git**: You’ll need Git to clone the kube-state-metrics repository. Install Git from the [official site](https://git-scm.com/downloads) if you don't have it already.

#### Steps to Build kube-state-metrics

1.  **Clone the Repository**  
    Open a terminal and run the following command to clone the kube-state-metrics repository:

```bash
git clone https://github.com/kubernetes/kube-state-metrics.git
```

Change into the cloned directory:

```bash
cd kube-state-metrics
```

2.  **Checkout the Desired Version**  
    If you want to build a specific version, use the following command to check out that version:

```bash
git checkout <version-tag>
```

Replace `<version-tag>` with the desired version (e.g., `v2.0.0`).

3.  **Build the Project**  
    Use the following command to build the kube-state-metrics binary:

```bash
make build
```

- This command compiles the code and generates the `kube-state-metrics` binary in the `./bin` directory.

4.  **Run kube-state-metrics**  
    You can run kube-state-metrics directly from the terminal using the built binary:

```bash
./bin/kube-state-metrics
```

By default, it listens on the port `8080`. You can configure additional options as needed.

6.  **Verify the Installation**  
    Open your web browser or use `curl` to access the kube-state-metrics metrics endpoint:

```bash
curl http://localhost:8080/metrics
```

## **Integrating with Prometheus**

Now that we have kube-state-metrics running, let's integrate it with Prometheus to start collecting and visualizing our metrics.

### **Configuring Prometheus**

Add the following job to your Prometheus config:

```yaml
- job_name: "kube-state-metrics"
  kubernetes_sd_configs:
    - role: endpoints
  relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      regex: kube-state-metrics
      action: keep
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
```

1.  target_label: **kubernetes_name**
2.  Restart Prometheus to apply the changes.

### **Querying kube-state-metrics in Prometheus**

Once Prometheus is scraping kube-state-metrics, you can start querying the metrics. Here are some useful queries:

1.  **Number of pods by namespace:**

```
sum(kube_pod_info) by (namespace)
```

2.  **Pods not in Running state:**

```
sum(kube_pod_status_phase{phase!="Running"}) by (namespace, phase)
```

3.  **Deployments not at the desired number of replicas:**

```
kube_deployment_spec_replicas != kube_deployment_status_replicas_available

```

4.  **Persistent volumes by phase:**

```
sum(kube_persistentvolume_status_phase) by (phase)
```

Also read: [Converting OpenTelemetry Traces to Metrics with the SpanMetrics Connector](https://last9.io/blog/convert-opentelemetry-traces-to-metrics-using-spanconnector/)

## **kube-state-metrics Use Cases**

Let's understand some practical use cases where I've found kube-state-metrics to be invaluable:

### **1\. Monitoring Pod Health**

One of the most common issues I've encountered is pods stuck in a crash loop. kube-state-metrics makes it easy to track pod restarts:

```
sum(kube_pod_container_status_restarts_total) by (pod)
```

This query helps identify problematic pods quickly, saving precious debugging time. I once used this to identify a memory leak in a Java application that was causing frequent restarts.

### **2\. Resource Quota Management**

In multi-tenant clusters, managing resource quotas is crucial. kube-state-metrics provides metrics like:

```
kube_resourcequota
```

This allows us to track resource usage against quotas, helping prevent resource starvation issues. I've seen this bring down entire namespaces when not properly monitored.

Example query to check CPU usage against quota:

```
sum(kube_pod_container_resource_requests_cpu_cores) by (namespace) /
sum(kube_resourcequota{resourcequota!="", resource="requests.cpu"}) by (namespace)

```

### **3\. Persistent Volume Monitoring**

I once dealt with a critical outage caused by running out of persistent volume space. kube-state-metrics could have helped prevent this with metrics like:

```
kube_persistentvolume_status_phase

```

This metric allows you to track the status of your persistent volumes and set up alerts for when they're nearing capacity.

### **4\. Deployment Tracking**

Tracking the success of rolling updates is crucial. kube-state-metrics provides detailed deployment metrics:

```
kube_deployment_status_replicas_available
kube_deployment_status_replicas_unavailable

```

These metrics have helped me quickly identify failed deployments and roll back when necessary.

For example, you can set up an alert for when available replicas don't match the desired state:

```
kube_deployment_status_replicas_available != kube_deployment_spec_replicas

```

### **5\. Node Capacity Planning**

kube-state-metrics provides valuable data for capacity planning:

```
sum(kube_node_status_capacity_cpu_cores) - sum(kube_node_status_allocatable_cpu_cores)
```

This query shows the difference between total CPU capacity and allocatable CPU, helping identify overhead and plan for cluster scaling.

Know more about [collecting logs from Kubernetes with the OpenTelemetry Filelog Receiver](https://last9.io/blog/opentelemetry-filelog-receiver-kubernetes-log-collection/) in our blog!

## **Advanced Topics and Best Practices**

As you become more comfortable with kube-state-metrics, consider these advanced topics and best practices:

### **1\. Custom Resources**

kube-state-metrics supports custom resources. I've used this to monitor application-specific CRDs, providing valuable business metrics.

To enable custom resource metrics, you need to build kube-state-metrics from the source with the custom resource enabled. Here's an example Dockerfile:

```dockerfile
FROM golang:1.17 as builder
WORKDIR /go/src/k8s.io/kube-state-metrics
COPY . .
RUN make build-local

FROM gcr.io/distroless/static:nonroot
COPY --from=builder /go/src/k8s.io/kube-state-metrics/kube-state-metrics .
USER nonroot:nonroot
ENTRYPOINT ["./kube-state-metrics", "--custom-resource-state-config=/path/to/cr-config.yaml"]
```

The cr-config.yaml file would look something like this:

```yaml
spec:
  resources:
    - groupVersionKind:
        group: "myapp.com"
        kind: "MyCustomResource"
        version: "v1"
      metrics:
        - name: "mycustomresource_status_phase"
          help: "The current phase of the custom resource"
          each:
            type: Gauge
            gauge:
              valueFrom:
                path: "{.status.phase}"
```

### **2\. High Availability**

For critical environments, consider running multiple replicas of kube-state-metrics using a StatefulSet for high availability. Here's an example configuration:

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kube-state-metrics
spec:
  serviceName: "kube-state-metrics"
  replicas: 2
  selector:
    matchLabels:
      app: kube-state-metrics
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      containers:
        - name: kube-state-metrics
          image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.1.0
          ports:
            - containerPort: 8080
              name: http-metrics
            - containerPort: 8081
              name: telemetry
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            timeoutSeconds: 5
```

### **3\. Resource Limits**

Be sure to set appropriate CPU and memory limits for kube-state-metrics. I've seen it consume significant resources in large clusters. Here's an example of setting resource limits:

```yaml
resources:
  limits:
    cpu: 100m
    memory: 150Mi
  requests:
    cpu: 100m
    memory: 150Mi
```

### **4\. Metric Relabeling**

Use Prometheus' relabeling features to add useful metadata to your metrics, making them more informative and easier to query. Here's an example relabeling configuration:

```yaml
metric_relabel_configs:
  - source_labels: [__name__]
    regex: "kube_pod_container_status_running"
    action: keep
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
```

### **5\. Grafana Dashboards**

Create comprehensive Grafana dashboards using kube-state-metrics. The [kube-state-metrics GitHub repo](https://github.com/kubernetes/kube-state-metrics) has some great examples to get you started.

Here's a simple Grafana dashboard query to show the number of pods per namespace:

```
sum(kube_pod_info) by (namespace)
```

For more complex dashboards, you can combine kube-state-metrics with other data sources. For example, to show CPU usage vs. requests:

```
sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace) /
sum(kube_pod_container_resource_requests_cpu_cores) by (namespace)

```

Explore the [Top Splunk Alternatives for 2024](https://last9.io/blog/top-splunk-alternatives-for-2024-a-comprehensive-guide/) in this comprehensive guide!

### **6\. Alerting**

Set up alerting rules in Prometheus to proactively notify you of potential issues. Here's an example alert for pods in a non-running state:

```yaml
groups:
  - name: kubernetes-apps
    rules:
      - alert: KubePodNotReady
        expr: sum by (namespace, pod) (kube_pod_status_phase{phase=~"Pending|Unknown"}) > 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} is not ready"
          description: "Pod {{ $labels.pod }} has been in a non-ready state for more than 15 minutes."
```

### **7\. Performance Tuning**

For large clusters, you might need to tune kube-state-metrics for better performance.

Some options include:

1.  Enabling metric caching:

```
--metric-cache-size=1000
```

2.  Adjusting the metrics update interval:

```
--metric-resync-interval=30s
```

3.  Using metric allow-lists to reduce the number of metrics collected:

```
--metric-allowlist=kube_pod_status_phase,kube_deployment_status_replicas
```

These optimizations can significantly improve the runtime performance of kube-state-metrics, especially in large Kubernetes environments.

### **8\. Integration with Cloud Providers**

When running Kubernetes on cloud providers like AWS, you can leverage kube-state-metrics to monitor cloud-specific resources. For example, you can track the number of load balancers created by your Ingress controllers:

```
sum(kube_service_spec_type{type="LoadBalancer"}) by (namespace)
```

This query helps you keep track of your AWS resources and associated costs.

### **9\. Container Image Management**

kube-state-metrics can help you track the usage of container images across your cluster:

```
sum(kube_pod_container_info) by (image)
```

This is particularly useful for ensuring that all pods are running the expected Docker images and versions.

## Troubleshooting Common Issues

Even with careful setup, you might encounter some issues. Here are some common problems and their solutions:

1.  **High CPU Usage**: If kube-state-metrics is consuming too much CPU, consider using metric allow-lists or increasing the metric resync interval.
2.  **Memory Leaks**: Ensure you're using the latest version of kube-state-metrics. Earlier versions had memory leak issues that have since been resolved.
3.  **Missing Metrics**: Check the RBAC permissions. kube-state-metrics might not have access to all namespaces or resources.
4.  **Inconsistent Metrics**: This can happen in large clusters due to the eventually consistent nature of the Kubernetes API. Consider increasing the metric resync interval.
5.  **Ingress Metrics Missing**: If you're not seeing metrics for Ingress resources, make sure you're using a compatible Ingress controller and that kube-state-metrics has permission to access Ingress resources.
6.  **Authentication Issues**: If you're experiencing authentication problems, ensure that the ServiceAccount associated with kube-state-metrics has the correct RBAC permissions. You may need to adjust the ClusterRole or Role bindings.

Monitoring Kubernetes at scale?Kube-state-metrics generates a lot of series: especially with dynamic workloads. Last9 ingests it all without cardinality limits, so you keep full visibility as clusters grow.

[Try Last9 free →](https://app.last9.io/signin)

## **Conclusion**

kube-state-metrics has become an essential tool in my Kubernetes observability stack. Its ability to provide deep insights into the state of Kubernetes objects, combined with the power of Prometheus and Grafana, has significantly improved my ability to maintain and troubleshoot Kubernetes clusters.

As you implement kube-state-metrics in your own clusters, keep in mind that every environment is unique.

Happy monitoring!

> We'd love to hear your experiences with reliability, observability, or monitoring! Join the conversation and share your insights with us in the [SRE Discord community](https://discord.com/invite/W8gMppQC4b).

## FAQs

### What are kube-state-metrics?

kube-state-metrics listens to the Kubernetes API server and generates metrics about Kubernetes object states. It provides a metrics endpoint for Prometheus, offering insights into the health and status of various resources.

### What is the difference between node exporter and kube-state-metrics?

Node exporter collects system-level metrics from nodes (CPU, memory, disk usage), while kube-state-metrics generates metrics about Kubernetes objects (pods, deployments, services). They work together to give a complete view of cluster health.

### How do you deploy kube-state-metrics on Kubernetes?

You can deploy kube-state-metrics using Helm charts or YAML manifests. For Helm, use: `helm install kube-state-metrics prometheus-community/kube-state-metrics`

### How can we expose metrics to Prometheus?

kube-state-metrics exposes a metrics endpoint for Prometheus to scrape. Add this job to your Prometheus config:

```yaml
- job_name: "kube-state-metrics"
  kubernetes_sd_configs:
    - role: endpoints
  relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      regex: kube-state-metrics
      action: keep
```

### What metrics are provided by kube-state-metrics in Kubernetes?

kube-state-metrics provides metrics such as pod status, deployment status, node capacity, PersistentVolume and PersistentVolumeClaim status, and service details.

### How do I monitor pod status using kube-state-metrics in Kubernetes?

Use Prometheus queries like `kube_pod_status_phase{phase="Running"}` for running pods and `kube_pod_container_status_restarts_total` for container restart counts.

### Can you monitor Kubernetes cluster status using kube-state-metrics?

Yes, kube-state-metrics provides metrics on cluster status, such as node conditions and resource usage. For example, use `kube_node_status_condition{condition="Ready", status="true"}` to find ready nodes.

### How many load balancer services do we have and what are their IPs?

Use the query `kube_service_spec_type{type="LoadBalancer"}` and `kube_service_status_load_balancer_ingress` to get the count and IPs of LoadBalancer services.