Skip to content

Deploy Elasticsearch Cluster & Kibaba On Kubernetes

Preface

Nowadays maybe the most advanced and widly used log management and analizis system is the ELK stack. I have to mention Graylog and Grafana Loki which are also great and advanced tools for montioring your environments and collect log files from them.

There is another enterprise ready and feature rich log management system which based on Elasticsearch and Kibana: OpenSearch. If you are looking for a free alternaive to Elasticsearch you may want to give OpenSearch a try. I'm going to post about OpenSearch as well, but at this time I want to show you a method to install Elasticsearch & Kibana on your Kubernetes cluster.

Requirements

  • A working Kubernetes cluster. The current version of my cluster: v1.24.4
  • Kubectl cli tool
  • Installed and ready to use Persistent Volume solution (Example Longhorn, OpenEBS, rook, etc)
  • At least 2GB of free memory for Elasticsearch instances.

Set vm.max_map_count To At Least 262144

This is a strict requirements of Elasticsearch. You have to set this value on each node you are planning to run Elasticsearch. You can select the nodes where to run Elasticsearch with nodeselectors and node labels.

Add the following line to /etc/sysctl.conf file:

vm.max_map_count=262144

To apply the setting on a live system, run:

sysctl -w vm.max_map_count=262144

Prepareing

The first and most important thing is to choose a names of your Elasticsearch cluster and Instances. We will deploy Elasticsearch cluster as StatefulSet, so the name of instances will be sequential.

Create Certificates

  • Create a directory for your certificates:
mkdir /tmp/es-certs
chown 1000:1000 /tmp/es-certs
  • Create the instances.yml file.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
cat <<EOF>/tmp/es-certs/instances.yml
instances:
- name: elastic-0
  dns:
    - elastic-0.es-cluster
    - localhost
    - es-cluster
  ip:
    - 127.0.0.1
- name: elastic-1
  dns:
    - elastic-1.es-cluster
    - localhost
    - es-cluster
  ip:
    - 127.0.0.1
- name: elastic-2
  dns:
    - elastic-3.es-cluster
    - localhost
    - es-cluster
  ip:
    - 127.0.0.1
EOF

Important

The - name: elastic-0 is must mach the StatefulSet name plus the sequence number appended by dash. The DNS (- name: elastic-1 ... elastic-n) name must mach the name of StatfulSet: metadata.name: elastic and the headless service name. [STATFULSET_NAME]-[NUMBER].[STATEFUL_SERVICE_NAME] The third DNS record is the neme of the Kubernetes (headless) Service. This will be used for Kubernetes internal use, for example for Kibana.

  • Generate the certificates

Run a temporary contianer to work in it:

docker run -v /tmp/es-certs:/usr/share/elasticsearch/config/certs -it --rm docker.elastic.co/elasticsearch/elasticsearch:8.5.1 /bin/sh

Run the following commands inside the container:

1
2
3
4
5
6
7
# Generate CA certificates
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip
unzip config/certs/ca.zip -d config/certs

# Generate Elasticsearch Certificates
bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key
unzip config/certs/certs.zip -d config/certs

Exit from the container.

After the certificate generation your folder and file should look like that:

/tmp/es-certs/
/tmp/es-certs/certs.zip
/tmp/es-certs/elastic-2
/tmp/es-certs/elastic-2/elastic-2.key
/tmp/es-certs/elastic-2/elastic-2.crt
/tmp/es-certs/ca.zip
/tmp/es-certs/elastic-0
/tmp/es-certs/elastic-0/elastic-0.key
/tmp/es-certs/elastic-0/elastic-0.crt
/tmp/es-certs/instances.yml
/tmp/es-certs/elastic-1
/tmp/es-certs/elastic-1/elastic-1.crt
/tmp/es-certs/elastic-1/elastic-1.key
/tmp/es-certs/ca
/tmp/es-certs/ca/ca.key
/tmp/es-certs/ca/ca.crt
  • Move all files to the /tmp/es-certs/
1
2
3
cd /tmp/es-certs
find . -mindepth 2 -maxdepth 2 -type f -ls -exec mv "{}" . \;
find . -mindepth 1 -maxdepth 1 -type d -ls -exec rmdir "{}" \;

Now your folder should be similar to this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
total 56
drwxr-xr-x  2 vinyo vinyo 4096 Nov 18 14:07 .
drwxrwxrwt 25 root  root  4096 Nov 18 14:08 ..
-rw-rw-r--  1 vinyo root  1200 Nov 18 14:02 ca.crt
-rw-rw-r--  1 vinyo root  1679 Nov 18 14:02 ca.key
-rw-------  1 vinyo root  2519 Nov 18 14:02 ca.zip
-rw-------  1 vinyo root  7851 Nov 18 14:04 certs.zip
-rw-rw-r--  1 vinyo root  1220 Nov 18 14:04 elastic-0.crt
-rw-rw-r--  1 vinyo root  1679 Nov 18 14:04 elastic-0.key
-rw-rw-r--  1 vinyo root  1220 Nov 18 14:04 elastic-1.crt
-rw-rw-r--  1 vinyo root  1679 Nov 18 14:04 elastic-1.key
-rw-rw-r--  1 vinyo root  1220 Nov 18 14:04 elastic-2.crt
-rw-rw-r--  1 vinyo root  1675 Nov 18 14:04 elastic-2.key
-rw-r--r--  1 vinyo vinyo  299 Nov 18 14:04 instances.yml

Create Kubernetes Secrets & Namespace

  • Certificates
1
2
3
4
5
6
# Create the Namespace
kubectl create ns logging

# Delete the secret if it is already exists:
# kubectl -n logging delete secret es-certs
kubectl -n logging create secret generic es-certs --from-file=/tmp/es-certs
  • Elastic Password

kubectl -n logging create secret generic elastic-password --from-literal=elastic=Admin1234
You shoud replace Admin1234 (of course).

You will use this username/password to login to Kiabana.

ElasticSearch StatefulSet & Service

StatefulSet

docs/files/es-statefulset.yaml
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: elastic
  namespace: logging
spec:
  replicas: 3
  selector:
    matchLabels:
      k8s-app: elastic
  template:
    metadata:
      name: elastic
      creationTimestamp: null
      labels:
        k8s-app: elastic
    spec:
      volumes:
        - name: es-certs
          secret:
            secretName: es-certs
            defaultMode: 420
      containers:
        - name: elastic
          image: docker.elastic.co/elasticsearch/elasticsearch:8.5.1
          env:
            - name: NODENAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: SERVICENAME
              value: es-cluster
            - name: cluster.name
              value: $(SERVICENAME)              
            - name: node.name
              value: $(NODENAME).$(SERVICENAME)
            - name: discovery.seed_hosts
              value: elastic-0.es-cluster,elastic-1.es-cluster,elastic-3.es-cluster
            - name: cluster.initial_master_nodes
              value: elastic-0.es-cluster,elastic-1.es-cluster,elastic-3.es-cluster
            - name: ES_JAVA_OPTS
              value: '-Xms2g -Xmx2g'
            - name: xpack.security.enabled
              value: 'true'
            - name: xpack.security.http.ssl.enabled
              value: 'true'
            - name: xpack.security.http.ssl.key
              value: certs/$(NODENAME).key
            - name: xpack.security.http.ssl.certificate
              value: certs/$(NODENAME).crt
            - name: xpack.security.http.ssl.certificate_authorities
              value: certs/ca.crt
            - name: xpack.security.http.ssl.verification_mode
              value: certificate
            - name: xpack.security.transport.ssl.enabled
              value: 'true'
            - name: xpack.security.transport.ssl.key
              value: certs/$(NODENAME).key
            - name: xpack.security.transport.ssl.certificate
              value: certs/$(NODENAME).crt
            - name: xpack.security.transport.ssl.certificate_authorities
              value: certs/ca.crt
            - name: xpack.security.transport.ssl.verification_mode
              value: certificate
            - name: ELASTIC_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: elastic-password
                  key: elastic
          resources:
            limits:
              cpu: 1500m
              memory: 3Gi
            requests:
              cpu: 250m
              memory: 2Gi
          volumeMounts:
            - name: es-data
              mountPath: /usr/share/elasticsearch/data
              subPath: data
            - name: es-certs
              readOnly: true
              mountPath: /usr/share/elasticsearch/config/certs
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext:
        fsGroup: 1000
      schedulerName: default-scheduler
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: es-data
        creationTimestamp: null
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: local-hostpath
        volumeMode: Filesystem
  serviceName: es-cluster
  podManagementPolicy: OrderedReady
  updateStrategy:
    type: RollingUpdate
  revisionHistoryLimit: 10
  minReadySeconds: 10

Important Parts Of The Manifests

PersistentVolumeClaim

  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: es-data
        creationTimestamp: null
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: local-hostpath
        volumeMode: Filesystem

I really recommend to use some kind of hostpath volume, for example OpenEBS, since Elasticsearch operations can be IO heavy. If you decide to use OpenEBS hostpath all the POD will be scheduled to the same host all the time.

Environment Variables

            - name: NODENAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name

This variable is not used direrctly by the pod itself. It is just for this manifest. The value is the name of the StatefulSet. It's purpose to use in other variables. (metadata.name could not be nested)

            - name: SERVICENAME
              value: es-cluster

This must mach with the serviceName: es-cluster in this manifest, and the neme of the headless Service.

            - name: node.name
              value: $(NODENAME).$(SERVICENAME)

Each Elasticsearch instance created by the StatefulSet get the node name like elastic-0.es-clsuster, elastic-1.es-clsuster, etc. This is really important for the next parameters:

            - name: discovery.seed_hosts
              value: elastic-0.es-cluster,elastic-1.es-cluster,elastic-3.es-cluster
            - name: cluster.initial_master_nodes
              value: elastic-0.es-cluster,elastic-1.es-cluster,elastic-3.es-cluster

Important

Now you can see that how important to decide the names of each component. As I wrote above the DNS names in the instances.yml must mach these names. elastic-0.es-cluster means the [POD_NAME].[HEADLESS_SERVICE:metadata.name]. In our case the pod name is always the name of the StatefulSet + sequence number (because of the StatefulSet). This way the elastic-[n].es-cluster always points to the actual IP address of the pods create by the StatefulSet.

Note

You can increase or decrease the number of Elasticsearch instances, but keep in mind to modify these values:

  • Certificate generation: Modify the instances.yml, and regenerate the certificates, but only certs.zip not the CA! Don't forget to update the Kubernetes secret.
  • StatefulSet:
    • spec.replicas
    • Variables: discovery.seed_hosts & cluster.initial_master_nodes According to the instances.yml file.
            - name: xpack.security.http.ssl.key
              value: certs/$(NODENAME).key
            - name: xpack.security.http.ssl.certificate
              value: certs/$(NODENAME).crt

Every node has its own certificate. That's why we need the $(NODENAME) variable. This way the certs/$(NODENAME).crt will be certs/elastic-0.crt for the first pod and certs/elastic-1.crt for the second one, etc.

Note

You can create a single certificate which holds all of the DNS record for all nodes, but it is antipattern and not recommended for security reason.

            - name: ELASTIC_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: elastic-password
                  key: elastic

This is the password for the built-in elastic superuser.

  • Volumes
          volumeMounts:
            ...
            ...
            - name: es-certs
              readOnly: true
              mountPath: /usr/share/elasticsearch/config/certs

Here we mount the previously created Kubernetes secret which contains all of the necessary certificates.

Checks

Get into the elastic-0 pod:

kubectl -n logging exec -it elastic-0 -- /bin/sh

And run the following commands:

curl -i -k  -XGET https://localhost:9200/_cat/nodes?v -u 'elastic:Admin1234'
ouptut
HTTP/1.1 200 OK
X-elastic-product: Elasticsearch
content-type: text/plain; charset=UTF-8
content-length: 302

ip          heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
10.26.6.107           14          83   1    1.57    1.83     1.59 cdfhilmrstw -      elastic-0.es-cluster
10.26.4.230           37          83   2    0.58    0.76     0.62 cdfhilmrstw *      elastic-1.es-cluster

curl -i -k  -XGET https://localhost:9200/_cat/allocation?v -u 'elastic:Admin1234'
output
HTTP/1.1 200 OK
X-elastic-product: Elasticsearch
content-type: text/plain; charset=UTF-8
content-length: 314

shards disk.indices disk.used disk.avail disk.total disk.percent host        ip          node
     4       39.9mb    19.2gb     89.3gb    108.5gb           17 10.26.4.230 10.26.4.230 elastic-1.es-cluster
     4       39.8mb    65.8gb     50.2gb    116.1gb           56 10.26.6.107 10.26.6.107 elastic-0.es-cluster

Hint

As you can see I have only two nodes at the moment. But everything looks fine.

Deploy Kibana

Prepare

First prepare the kibana_system built-in user password:

Important

Run the following command inside one of your elastic pod!!!

curl -k -i -X POST -u "elastic:Admin1234" -H "Content-Type: application/json" https://localhost:9200/_security/user/kibana_system/_password -d "{\"password\":\"Admin123\"}"
output
HTTP/1.1 200 OK
X-elastic-product: Elasticsearch
content-type: application/json
content-length: 2

{}

Warning

Do not use the cli tools (/usr/share/elasticsearch/bin/elasticsearch-*) to update/reseet paswword. . This will create a file inside the /usr/share/elasticsearch/config directory, and after the pod restart this file will be gone.

Note

Please note that the password (elastic:Admin1234) comes from the ELASTIC_PASSWORD environment variable (pre-created secret).

Create a Kuernetes secret:

kubectl -n logging create secret generic kibanasystem --from-literal=kibana_system=Admin123

Manifest

docs/files/kibana-deployment.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
kind: Deployment
apiVersion: apps/v1
metadata:
  name: kibana
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana
  template:
    metadata:
      name: kibana
      creationTimestamp: null
      labels:
        k8s-app: kibana
    spec:
      volumes:
        - name: es-certs
          secret:
            secretName: es-certs
            defaultMode: 420
      containers:
        - name: kibana
          image: kibana:8.5.1
          env:
            - name: ELASTICSEARCH_HOSTS
              value: https://es-cluster:9200
            - name: ELASTICSEARCH_USERNAME
              value: kibana_system
            - name: ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: kibanasystem
                  key: kibana_system
            - name: SERVER_PUBLICBASEURL
              value: https://kibana.vincze.work
            - name: ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES
              value: config/certs/ca.crt
          resources: {}
          volumeMounts:
            - name: es-certs
              readOnly: true
              mountPath: /usr/share/kibana/config/certs
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: false
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: Recreate
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

Noticeable parts:

  • Kibana use the same secret to mount the certificate as Elasticsearch. (volumeMounts: es-certs), but different mountPath: /usr/share/kibana/config/certs
  • Set SERVER_PUBLICBASEURL to the hostname that you will use in your ingress. If you miss this step Kibana will warn you to correct this.
  • ELASTICSEARCH_HOSTS: This value points to the headless service. That's why we need to add es-cluster as DNS record in instances.yml.
  • ELASTICSEARCH_USERNAME: Do NOT modify this value. Older versions of Elasticsearch used kibana, but it is deprecated. The username should be kibana_system.

Service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
kind: Service
apiVersion: v1
metadata:
  name: kibana
  namespace: logging
spec:
  ports:
    - name: web
      protocol: TCP
      port: 5601
      targetPort: 5601
  selector:
    k8s-app: kibana
  type: ClusterIP
  sessionAffinity: None
  ipFamilies:
    - IPv4
  internalTrafficPolicy: Cluster

Ingress

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  name: kibana
  namespace: logging
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 100m
spec:
  tls:
    - hosts:
        - kibana.******.com
      secretName: kibana-https
  rules:
    - host: kibana.vincze.work
      http:
        paths:
          - pathType: ImplementationSpecific
            backend:
              service:
                name: kibana
                port:
                  name: web

This is only an example ingress, so modify according to your needs.

Send Logs To The Elasticsearch Cluster

From inside the Kubernetes cluster it is really simple, just create a headless service:

kind: Service
apiVersion: v1
metadata:
  name: es-cluster
  namespace: logging
spec:
  ports:
    - name: rest
      protocol: TCP
      port: 9200
      targetPort: 9200
    - name: inter-node
      protocol: TCP
      port: 9300
      targetPort: 9300
  selector:
    k8s-app: elastic
  clusterIP: None
  type: ClusterIP
  sessionAffinity: None
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  internalTrafficPolicy: Cluster

Now you can use the es-cluster.logging.svc.cluster.local address.

Accessing Elasticsearch from outside the Kubernetes cluster a bit more complicated and highly depends on your environment. I have never tried, but you may create an Ingress, since the port 9200 for API calls over HTTP. https://discuss.elastic.co/t/what-are-ports-9200-and-9300-used-for/238578

Caution

This way your Elasticsearch cluster may be exposed to the public Internet.

Another way can be using NodePort serivce, or MetalLB LoadBalancer serivce. Example MetalLB service:

kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
  namespace: logging
  annotations:
    metallb.universe.tf/address-pool: default
spec:
  ports:
    - name: tcp-9200
      protocol: TCP
      port: 9200
      targetPort: 9200
  selector:
    k8s-app: elastic
  type: LoadBalancer
  sessionAffinity: None
  externalTrafficPolicy: Cluster
  ipFamilyPolicy: SingleStack
  allocateLoadBalancerNodePorts: true
  internalTrafficPolicy: Cluster

Info

MetalLB will create nodeport(s), as well.

Important

Remember the DNS config in instances.yaml! When you accessing your Elasticsearch cluster the DNS or IP address must mach the entries in the instances.yaml. So if you create a DNS entry with es.example.com domain, this must present in the DNS entries. Or if you accessing the ES cluster over MetalLB service, the ip address of the service must be added to the IP sections. Because of the Kubernetes service you don't know which pod will get the request that's why all node certificate should contain all possible domain name and/or IP address.

Bonus - Single Node Deployment

If you want to test Elasticsarch or you don't need multinode environment you can deploy Elasticearch as a single node environment.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: elastic
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: elastic
  template:
    metadata:
      name: elastic
      creationTimestamp: null
      labels:
        k8s-app: elastic
    spec:
      volumes:
        - name: es-data
          persistentVolumeClaim:
            claimName: es-data
      containers:
        - name: elastic
          image: docker.elastic.co/elasticsearch/elasticsearch:8.5.1
          env:
            - name: discovery.type
              value: single-node
            - name: cluster.name
              value: es-single
            - name: node.name
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: ES_JAVA_OPTS
              value: '-Xms2g -Xmx2g'
            - name: xpack.security.enabled
              value: 'true'
            - name: xpack.security.http.ssl.enabled
              value: 'false'
            - name: xpack.security.transport.ssl.enabled
              value: 'false'
            - name: ELASTIC_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: elastic-password
                  key: elastic
          resources:
            limits:
              cpu: 1500m
              memory: 3Gi
            requests:
              cpu: 250m
              memory: 2Gi
          volumeMounts:
            - name: es-data
              mountPath: /usr/share/elasticsearch/data
              subPath: data
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext:
        fsGroup: 1000
      schedulerName: default-scheduler
  strategy:
    type: Recreate
  minReadySeconds: 10
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

This is very similar to the StatefulSet, but notice the following parameters:

  • discovery.type: single-node --> This indicates that only one ES node will be present.
  • xpack.security.enabled: true --> Without this you won't be able to create users, and must find another way to protect Kibana. (Example: Ingress basic auth)
  • xpack.security.*.ssl.enabled: false --> Use plain HTTP. If you set them true, you have to generate certificates and set up them as in the StatefulSet.
  • The PersistentVolumeClaim must be pre-created.

References