How to Run Elasticsearch on Kubernetes

Elasticsearch stands as one of the most robust open-source search engines available today. Built on Apache Lucene, it handles complex search operations, real-time analytics, and large-scale data processing with impressive speed and accuracy.

Kubernetes has transformed how we deploy and manage containerized applications. This orchestration platform automates deployment, scaling, and operations of application containers across clusters of hosts.

When you combine Elasticsearch with Kubernetes, you get a powerful foundation for scalable search infrastructure.

This blog covers how to deploy Elasticsearch on Kubernetes using native Kubernetes constructs and Helm.

Should You Run Elasticsearch on Kubernetes?

Running Elasticsearch on Kubernetes can simplify operations if you're already managing stateful services in the same environment. But it comes with trade-offs that need careful consideration.

When Kubernetes Is a Good Fit

Kubernetes works well when your Elasticsearch workloads are dynamic, multi-tenant, or need to scale automatically. Some common patterns include:

• Log aggregation
  Ingest logs from multiple services using agents like Fluent Bit or Fluentd. Autoscaling helps handle spikes in log volume.
• Search applications
  Serve user-facing search queries—product catalogs, documentation, or structured content. Kubernetes allows you to scale read nodes based on demand.
• Metrics and observability
  Use Elasticsearch as a backend for storing metrics or trace data. Integrate with Kibana or Grafana for dashboards and alerts.
• Analytics and aggregations
  Run large-scale aggregation queries over time-series or semi-structured data. Horizontal scaling improves query performance under load.

Features That Support Elasticsearch

Several Kubernetes primitives help improve Elasticsearch resilience and uptime:

• Pod anti-affinity
  Spread Elasticsearch pods across nodes to avoid single points of failure.
• Rolling updates
  Apply configuration changes or upgrade Elasticsearch without taking the cluster offline.
• Liveness and readiness probes
  Detect and recover failed pods automatically using health checks.
• Load balancing
  Kubernetes Services route traffic evenly across healthy pods to improve request distribution.

Operational Trade-Offs

• Increased complexity
  You'll need working knowledge of both Elasticsearch internals and Kubernetes resource management.
• Storage configuration
  Elasticsearch requires stable, high-performance storage. Use StatefulSets with persistent volume claims and test your StorageClass for IOPS and latency.
• Network performance
  Kubernetes networking adds some latency compared to bare metal. Monitor tail latency if your workload is sensitive to response times.

Elasticsearch Internals on Kubernetes

When you deploy Elasticsearch on Kubernetes, each node becomes a pod. These pods are typically managed by a StatefulSet, which gives them stable DNS identities and persistent volumes, both required for reliable cluster formation and data durability.

Cluster Formation and Node Communication

Elasticsearch uses internal discovery to form a cluster. With StatefulSets, each pod gets a predictable hostname (e.g., elasticsearch-0.elasticsearch) and its persistent disk. This allows Elasticsearch to:

• Elect master nodes
• Distribute shards across nodes
• Maintain replicas for fault tolerance

Kubernetes manages pod restarts using liveness and readiness probes. If a pod crashes or becomes unhealthy, it’s automatically rescheduled. Elasticsearch rebalances data as needed based on shard allocation rules.

Shards, Replicas, and Node Awareness

Each Elasticsearch index is split into shards, with replicas for redundancy. In Kubernetes, these shards are distributed across pods, which may land on different worker nodes.

To avoid having both a primary and its replica on the same physical node (and risking data loss), use shard allocation awareness:

cluster.routing.allocation.awareness.attributes: k8s_node_name
node.attr.k8s_node_name: ${HOSTNAME}

This ensures Kubernetes schedules replicas across distinct nodes.

Kibana Integration

Kibana provides a UI to query, visualize, and debug your Elasticsearch data. You can deploy it in the same namespace using a standard Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: elastic-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:8.11.0
        ports:
        - containerPort: 5601
        env:
        - name: ELASTICSEARCH_HOSTS
          value: "http://elasticsearch:9200"

Once deployed, Kibana connects to the cluster and gives you dashboards, log views, and access to query tools.

Kubernetes Requirements for Running Elasticsearch

A stable and performant Elasticsearch deployment on Kubernetes depends on the correct use of StatefulSets, storage, namespaces, and resource constraints.

Use StatefulSets and Persistent Volumes

Elasticsearch pods must retain their identity and storage across restarts. Use a StatefulSet to ensure each pod gets:

• A stable network name (elasticsearch-0, elasticsearch-1, etc.)
• A dedicated volume that persists when the pod is rescheduled

Provision volumes using a StorageClass with high IOPS and support for dynamic provisioning:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Each Elasticsearch pod will claim its own PersistentVolumeClaim based on this class.

Isolate Resources with Namespaces and RBAC

Use a dedicated namespace (e.g., elastic-system) to group all Elasticsearch resources. This makes it easier to manage access, apply limits, and avoid naming collisions.

Example namespace and minimal RBAC:

apiVersion: v1
kind: Namespace
metadata:
  name: elastic-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elasticsearch
rules:
- apiGroups: [""]
  resources: ["pods", "services", "endpoints"]
  verbs: ["get", "list", "watch"]

Bind the role to a service account as needed based on your deployment model (Helm, ECK, or custom controllers).

Resource Sizing and Performance Tuning

Elasticsearch is sensitive to CPU, memory, and disk throughput. Start with conservative resource limits and tune based on actual workload characteristics.

CPU

• Start with 1–2 cores per data node.
• Increase if you see slow query execution or heavy indexing load.

Memory

• Allocate 50% of the pod memory to the JVM heap.
• Leave the rest for the file system cache (used by Lucene).

A typical small node starts with:

resources:
  requests:
    memory: "4Gi"
    cpu: "1"
  limits:
    memory: "4Gi"
    cpu: "2"

Storage

• Use SSD-backed volumes for all data nodes.
• Plan for 1.5–2× your expected data size to account for indexing overhead, replicas, and future growth.

Ongoing Monitoring

Track these metrics continuously:

• JVM heap usage
• Disk I/O saturation
• Pod restarts or evictions
• Indexing and query latency

Use these signals to tune vertical limits or scale out horizontally.

How to Deploy Elasticsearch on Kubernetes

There are several ways to deploy Elasticsearch on Kubernetes, depending on your operational model, control requirements, and experience level.

1. Raw YAML using StatefulSets
2. Helm charts (Elastic’s official)
3. ECK (Elastic Cloud on Kubernetes)

Method 1: Deploy Using Raw YAML (StatefulSet)

This method gives you the most control. It’s ideal for teams that want to tune everything from node layout to volume provisioning, without relying on a package manager.

Here’s a basic 3-node Elasticsearch cluster using a StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: elastic-system
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
        ports:
        - containerPort: 9200
          name: rest
        - containerPort: 9300
          name: inter-node
        env:
        - name: cluster.name
          value: k8s-logs
        - name: node.name
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: discovery.seed_hosts
          value: "elasticsearch-0.elasticsearch,elasticsearch-1.elasticsearch,elasticsearch-2.elasticsearch"
        - name: cluster.initial_master_nodes
          value: "elasticsearch-0,elasticsearch-1,elasticsearch-2"
        - name: ES_JAVA_OPTS
          value: "-Xms1g -Xmx1g"
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "fast-ssd"
      resources:
        requests:
          storage: 100Gi

This configuration:

• Creates 3 Elasticsearch pods with stable hostnames
• Persists data using dynamically provisioned volumes (StorageClass = fast-ssd)
• Sets up inter-node communication on port 9300

Apply it with:

kubectl apply -f elasticsearch-statefulset.yaml

Expose the cluster with a headless service:

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: elastic-system
spec:
  selector:
    app: elasticsearch
  ports:
  - port: 9200
    targetPort: 9200
    name: rest
  - port: 9300
    targetPort: 9300
    name: inter-node
  clusterIP: None

This setup provides stable DNS for each pod (e.g., elasticsearch-0.elasticsearch), which is required for cluster discovery.

Method 2: Deploy Using Helm

Helm simplifies the deployment process using prebuilt charts and sane defaults. Use this method if you want to get started quickly or integrate into an existing Helm-based workflow.

1. Add the official Elastic Helm repo:

helm repo add elastic https://helm.elastic.co
helm repo update

2. Install Elasticsearch:

helm install elasticsearch elastic/elasticsearch \
  --namespace elastic-system \
  --create-namespace \
  --set replicas=3 \
  --set volumeClaimTemplate.resources.requests.storage=100Gi \
  --set resources.requests.memory=2Gi \
  --set resources.limits.memory=2Gi

This installs a 3-node cluster with PVCs and memory limits set. You can override more values using a values.yaml file or with --set flags.

The Helm chart handles:

• StatefulSet creation
• Headless services for internal communication
• Volume provisioning
• Basic resource configs

Use helm upgrade for config changes and helm uninstall to tear it down cleanly.

Method 3: Deploy Using ECK (Elastic Cloud on Kubernetes)

Elastic Cloud on Kubernetes (ECK) is the most automated option. It runs an operator that watches for custom Elasticsearch resources and manages the cluster lifecycle.

Install the ECK operator:

kubectl apply -f https://download.elastic.co/downloads/eck/2.10.0/crds.yaml
kubectl apply -f https://download.elastic.co/downloads/eck/2.10.0/operator.yaml

Then create an Elasticsearch cluster using the custom resource:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
  namespace: elastic-system
spec:
  version: 8.11.0
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            limits:
              memory: 2Gi
              cpu: 1
            requests:
              memory: 2Gi
              cpu: 1
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 100Gi

ECK automates:

• Cluster discovery
• TLS and cert management
• Upgrades
• Node scaling
• Monitoring setup (if using the rest of the Elastic Stack)

This is the best choice if you're standardizing around the Elastic Stack or want full lifecycle automation.

Choosing the Right Deployment Method

Set Up Elasticsearch on Kubernetes with AWS

Running Elasticsearch on Kubernetes is production-ready on AWS, provided the infrastructure is tuned for stateful workloads. Amazon EKS simplifies Kubernetes operations, while services like EBS and NLB offer the underlying performance and availability Elasticsearch needs.

Use Amazon EKS for Kubernetes Management

EKS (Elastic Kubernetes Service) provides a managed control plane, reducing operational overhead for Kubernetes clusters. This is particularly helpful when running stateful workloads like Elasticsearch that require high availability and predictable node behavior.

Key advantages for Elasticsearch:

• Managed control plane
  AWS handles upgrades, failover, and scaling for Kubernetes masters. You only manage worker nodes.
• Native integrations
  Out-of-the-box support for IAM roles, ALB/NLB provisioning, EBS volumes, and CloudWatch metrics.
• Infrastructure as code support
  Use eksctl to automate EKS provisioning with consistent configurations:

eksctl create cluster \
  --name elasticsearch-cluster \
  --version 1.28 \
  --region us-west-2 \
  --nodegroup-name elasticsearch-nodes \
  --node-type m5.xlarge \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 10 \
  --managed

Use instance types that support enhanced networking (e.g., m5n, c5n) for low-latency inter-node communication.

Configure Load Balancers for External Access

Elasticsearch is often queried externally, either by apps, dashboards, or observability systems. Use a Kubernetes Service backed by an AWS Network Load Balancer (NLB) for external traffic routing:

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-external
  namespace: elastic-system
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
  type: LoadBalancer
  selector:
    app: elasticsearch
  ports:
  - port: 9200
    targetPort: 9200
    protocol: TCP

Restrict access to known IP ranges using AWS Security Groups:

aws ec2 authorize-security-group-ingress \
  --group-id sg-xxxxxxx \
  --protocol tcp \
  --port 9200 \
  --cidr 203.0.113.0/24

Use IAM roles and access policies if exposing via an API Gateway or other secured proxy.

Storage Options for Elasticsearch Pods

Elasticsearch requires fast, consistent storage with high IOPS and throughput. AWS provides several volume types suited to different workload intensities:

• EBS gp3
  Good default option. Set custom baseline IOPS and throughput for predictable performance.
• EBS io2
  Best for large-scale indexing or search clusters with heavy disk activity. Offers higher durability and guaranteed performance.
• Instance store (ephemeral)
  Very fast but non-persistent. Use only when you can rebuild lost data from upstream systems (e.g., log ingestion pipelines).

Provision volumes using a dedicated StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Network and Node Considerations

Elasticsearch is a chatty service. Inter-node communication (for replication, cluster state updates, and shard allocation) is frequent. Use instance types that support:

• Enhanced networking (ENA) for lower latency
• High network throughput (e.g., 10 Gbps or more on m5n, r5n, c5n instances)

Pin critical Elasticsearch pods using nodeSelectors or affinity rules to ensure they land on appropriate nodes.

Final Thoughts

Elasticsearch runs well on Kubernetes if storage, networking, and resource configs are tuned correctly. Start with a small StatefulSet or Helm deployment. Use managed services like EKS to reduce setup overhead.

If you’re using Elasticsearch to handle observability data — logs, metrics, traces, you're likely dealing with high cardinality, ingestion spikes, and noisy alerts. Kubernetes adds even more variability: autoscaling, ephemeral pods, and dynamic labels can quickly overwhelm your Elasticsearch cluster.

This is where Last9 helps: it gives you a Control Plane to decide which metrics to drop, pre-aggregate, or forward, reducing telemetry load before it reaches Elasticsearch. That means fewer noisy alerts, better control over ingestion, and visibility that reflects actual system pressure, not just surface-level spikes.

Get started with us today or book a time with us to know more!

FAQs

1. Should You Run Elasticsearch on Kubernetes?
Yes, if you're already using Kubernetes and comfortable managing stateful workloads. Elasticsearch benefits from Kubernetes features like rolling updates, horizontal scaling, and consistent environments, but it also requires careful tuning around storage, networking, and resource limits.

2. How Do I Deploy Elasticsearch on Kubernetes?
You can deploy Elasticsearch using:

• StatefulSets with custom YAML configs
• Helm charts from the Elastic Helm repo
• Elastic Cloud on Kubernetes (ECK), an operator-based approach by Elastic

Each method has tradeoffs based on control, automation, and operational complexity.

3. How Do I Set Up Elasticsearch for High Availability in Kubernetes?
Key configs for HA include:

• At least three master-eligible nodes
• Use of Persistent Volumes via StatefulSets
• Pod anti-affinity rules to avoid co-locating critical pods
• Liveness/readiness probes for automatic recovery
• Enabling replicas and proper shard allocation awareness

4. How Do You Scale an Elasticsearch Cluster on Kubernetes?
Use kubectl edit or Helm to increase the replica count of your data or ingest nodes. Ensure the underlying infrastructure (CPU, memory, disk IOPS) can support the additional load. Horizontal scaling works well if your workloads are balanced across shards.

5. What Is Elasticsearch in Kubernetes?
It’s a deployment of Elasticsearch where each node runs as a pod, typically managed by a StatefulSet. Kubernetes handles orchestration, scheduling, and restarts, while Elasticsearch handles clustering and data distribution.

6. What Are the Requirements for Running Elasticsearch on Kubernetes?
You’ll need:

• StatefulSets for pod identity and storage
• SSD-backed Persistent Volumes (like AWS gp3 or io2)
• Proper CPU/memory tuning (e.g., 4Gi RAM per node, 50% heap)
• A dedicated namespace and RBAC configuration
• StorageClasses and resource requests tuned for Elasticsearch

7. What Is the Elastic Stack?
The Elastic Stack—formerly ELK Stack—includes:

• Elasticsearch (search engine)
• Logstash (data pipeline)
• Kibana (visualization layer)
• Beats (lightweight data shippers)

It's commonly used for centralized logging, metrics, and observability.

8. What Is the ELK Stack?
The ELK Stack refers to Elasticsearch, Logstash, and Kibana—the original trio before Beats and other tools were added. It's used to collect, store, and visualize log data.

9. What Is Kubernetes on AWS?
It’s typically run via Amazon EKS (Elastic Kubernetes Service), which provides a managed Kubernetes control plane. You’re responsible for node groups and workloads, but AWS handles upgrades and availability of the master components.

10. Is the Discovery Config Still Valid in Elasticsearch?
Yes. When deploying in Kubernetes, discovery.seed_hosts and cluster.initial_master_nodes are still required in your Elasticsearch pod environment variables or config. For StatefulSets, this usually includes static pod names like elasticsearch-0, elasticsearch-1, etc.

11. What Does the curl Command Do?

curl http://10.102.46.55:9200/twitter/_search

This sends a query to the Elasticsearch node at 10.102.46.55 on the twitter index. If the index exists and the node is reachable, it returns documents. Useful to verify if data is still queryable.