Vm.max_map_count problems on GKE ElasticSearch StatefulSet

elasticsearchgcloudgoogle-kubernetes-enginekubernetes

A problem appeared on working ElasticSearch cluster on GKE.
Nodes with "data" roles began to crash unexpectedly with an error:

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

bootstrap checks failed

Of course, there is an init container in this StatefulSet controller which sets vm.max_map_count to 262144

Even more, this init container seem to complete successfully:

kubectl descrive pod elastic-data-0


Init Containers:
  init-sysctl:
    Container ID:  docker://23d3b3d11198510aa01aef340b92e1603785804fbf75e963fdbd61acfe458318
    Image:         busybox:latest
    Image ID:      docker-pullable://busybox@sha256:5e8e0509e829bb8f990249135a36e81a3ecbe94294e7a185cc14616e5fad96bd
    Port:          <none>
    Command:
      sysctl
      -w
      vm.max_map_count=262144
    State:          Terminated
      Reason:       Completed

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: elastic-data
  labels:
    component: elasticsearch
    role: data
spec:
  serviceName: elasticsearch-data
  updateStrategy:
    type: RollingUpdate
  replicas: 3
  template:
    metadata:
      labels:
        component: elasticsearch
        role: data
    spec:
      initContainers:
      - name: init-sysctl
        image: busybox:latest
        imagePullPolicy: Always
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        securityContext:
          privileged: true
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: role
                  operator: In
                  values:
                  - data
              topologyKey: kubernetes.io/hostname
      containers:
      - name: es-data
        image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
        env:
        - name: node.name
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: cluster.name
          value: "elastic"
        - name: node.master
          value: "false"
        - name: node.data
          value: "true"
        - name: node.ingest
          value: "false"
        - name: http.enabled
          value: "true"
        - name: bootstrap.memory_lock
          value: "false"
        - name: path.data
          value: "/data/data"
        - name: path.logs
          value: "/data/log"
        - name: discovery.zen.ping.unicast.hosts
          value: "elasticsearch-discovery"
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        - name: processors
          valueFrom:
            resourceFieldRef:
              resource: limits.cpu
        resources:
          limits:
            cpu: 2
          requests:
            cpu: 300m
        ports:
        - containerPort: 9200
          name: http
        - containerPort: 9300
          name: transport
        livenessProbe:
          tcpSocket:
            port: transport
          initialDelaySeconds: 20
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /_cluster/health
            port: http
          initialDelaySeconds: 20
          timeoutSeconds: 5
        volumeMounts:
        - name: storage-volume
          mountPath: /data
      securityContext:
        runAsUser: 1000
        fsGroup: 100
  volumeClaimTemplates:
  - metadata:
      name: storage-volume
    spec:
      storageClassName: manual
      accessModes: [ ReadWriteOnce ]
      resources:
        requests:
          storage: 300Gi

Logs:

kubectl logs elastic-data-0 
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2018-08-16T15:40:33,998][INFO ][o.e.n.Node               ] [elastic-data-0] initializing ...
[2018-08-16T15:40:34,163][INFO ][o.e.e.NodeEnvironment    ] [elastic-data-0] using [1] data paths, mounts [[/data (/dev/sdb)]], net usable_space [231.2gb], net total_space [245gb], types [ext4]
[2018-08-16T15:40:34,165][INFO ][o.e.e.NodeEnvironment    ] [elastic-data-0] heap size [503.6mb], compressed ordinary object pointers [true]
[2018-08-16T15:40:34,544][INFO ][o.e.n.Node               ] [elastic-data-0] node name [elastic-data-0], node ID [C2vCCIpHS3mpiDHduimS0g]
[2018-08-16T15:40:34,545][INFO ][o.e.n.Node               ] [elastic-data-0] version[6.3.2], pid[1], build[default/tar/053779d/2018-07-20T05:20:23.451332Z], OS[Linux/4.14.22+/amd64], JVM["Oracle Corporation"/OpenJDK 64-Bit Server VM/10.0.2/10.0.2+13]
[2018-08-16T15:40:34,545][INFO ][o.e.n.Node               ] [elastic-data-0] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.zCR3bQNp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2018-08-16T15:40:36,612][WARN ][o.e.d.c.s.Settings       ] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2018-08-16T15:40:38,484][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [aggs-matrix-stats]
[2018-08-16T15:40:38,485][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [analysis-common]
[2018-08-16T15:40:38,485][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [ingest-common]
[2018-08-16T15:40:38,485][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [lang-expression]
[2018-08-16T15:40:38,485][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [lang-mustache]
[2018-08-16T15:40:38,485][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [lang-painless]
[2018-08-16T15:40:38,486][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [mapper-extras]
[2018-08-16T15:40:38,486][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [parent-join]
[2018-08-16T15:40:38,486][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [percolator]
[2018-08-16T15:40:38,486][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [rank-eval]
[2018-08-16T15:40:38,486][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [reindex]
[2018-08-16T15:40:38,486][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [repository-url]
[2018-08-16T15:40:38,486][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [transport-netty4]
[2018-08-16T15:40:38,486][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [tribe]
[2018-08-16T15:40:38,486][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [x-pack-core]
[2018-08-16T15:40:38,486][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [x-pack-deprecation]
[2018-08-16T15:40:38,486][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [x-pack-graph]
[2018-08-16T15:40:38,487][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [x-pack-logstash]
[2018-08-16T15:40:38,487][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [x-pack-ml]
[2018-08-16T15:40:38,487][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [x-pack-monitoring]
[2018-08-16T15:40:38,487][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [x-pack-rollup]
[2018-08-16T15:40:38,487][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [x-pack-security]
[2018-08-16T15:40:38,487][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [x-pack-sql]
[2018-08-16T15:40:38,488][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [x-pack-upgrade]
[2018-08-16T15:40:38,488][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded module [x-pack-watcher]
[2018-08-16T15:40:38,488][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded plugin [ingest-geoip]
[2018-08-16T15:40:38,489][INFO ][o.e.p.PluginsService     ] [elastic-data-0] loaded plugin [ingest-user-agent]
[2018-08-16T15:40:44,991][INFO ][o.e.x.s.a.s.FileRolesStore] [elastic-data-0] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[2018-08-16T15:40:45,793][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/61] [Main.cc@109] controller (64 bit): Version 6.3.2 (Build 903094f295d249) Copyright (c) 2018 Elasticsearch BV
[2018-08-16T15:40:47,003][INFO ][o.e.d.DiscoveryModule    ] [elastic-data-0] using discovery type [zen]
[2018-08-16T15:40:48,139][INFO ][o.e.n.Node               ] [elastic-data-0] initialized
[2018-08-16T15:40:48,140][INFO ][o.e.n.Node               ] [elastic-data-0] starting ...
[2018-08-16T15:40:48,337][INFO ][o.e.t.TransportService   ] [elastic-data-0] publish_address {10.0.1.11:9300}, bound_addresses {[::]:9300}
[2018-08-16T15:40:48,452][INFO ][o.e.b.BootstrapChecks    ] [elastic-data-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-08-16T15:40:48,477][INFO ][o.e.n.Node               ] [elastic-data-0] stopping ...
[2018-08-16T15:40:48,503][INFO ][o.e.n.Node               ] [elastic-data-0] stopped
[2018-08-16T15:40:48,504][INFO ][o.e.n.Node               ] [elastic-data-0] closing ...
[2018-08-16T15:40:48,525][INFO ][o.e.n.Node               ] [elastic-data-0] closed
[2018-08-16T15:40:48,529][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started

Kubernetes version is 1.10.5-gke.4 (yes, it's on GKE)

Any ideas are appreciated.

Best Answer

The problem is that init-containers is executed only when pod is created to node, I Think you restarted kubernetes nodes behind this and you did get error:

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Normally to fix this, you should run the command below for each node you have when you restarted kubernetes:

sudo sysctl -w vm.max_map_count=262144

I suggest you to use DeamonSet to your cluster. Seems to do the trick. Using google's startup-script container. you will find the solution provided below.

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    k8s-app: sysctl-conf
  name: sysctl-conf
spec:
  template:
    metadata:
      labels:
        k8s-app: sysctl-conf
    spec:
      containers:
      - command:
        - sh
        - -c
        - sysctl -w vm.max_map_count=262166 && while true; do sleep 86400; done
        image: busybox:1.26.2
        name: sysctl-conf
        resources:
          limits:
            cpu: 10m
            memory: 50Mi
          requests:
            cpu: 10m
            memory: 50Mi
        securityContext:
          privileged: true
      terminationGracePeriodSeconds: 1

to validate your update take one node, ssh to it node and run the command to list how much VM max:

sudo sysctl -a | grep vm.max_map_count

Related Solutions

Java – Unable to change vm.max_map_count for elasticsearch

you are almost there, It does not matter if is a virtual machine or a physical machine, those settings are always changeable.

I'll show 3 methods.

Some pre-information:

1) It's better to execute as root, if possible.

2) /proc on unix is not a real filesystem, it's a in-memory kernel file system, but it appears to be like a normal disk file system. You can call it 'fake filesystem' or 'special filesystem', you cannot edit those fake-files with vi or any other editor, because they are not files, they just look like files. I stuck with the same problem years ago.

But it's simple to change their values, just require another kind of 'mechanics' to edit them.

I'll will explain: First, need to be root: (sudo does work in some distros, but does not on some other distros like you tried, this first method is universal and works on any Linux, macOS, or any Unix-based. Hope you have access to root password.

Proceed at prompt:

    $ su root

Enter root password.

Now you are root, let's check the current value of: /proc/sys/vm/max_map_count

    $ cat /proc/sys/vm/max_map_count  
    65536

Let's change it:

    echo 262144 > /proc/sys/vm/max_map_count

Let's verify:

    cat /proc/sys/vm/max_map_count
    262144

It's done! And it's already applied and functional. By changing values of any pseudo-file under /proc the settings becomes active instantly. But they don't persist after a reboot. You can play with values and measure performance changes at elasticsearh or any other application or system metrics. Go tunning your system, writing the values on some paper, keep the best values. On any mistake, reboot and they will all be back to original values, and start again until all wished values are optimal. There's a lot of disk and memory tunnable parameters under /proc. And they make a huge difference and performance gain if you to tune them well (and have time for it). You are on the right way.

When satisfied, let's make them permanent:

First method:

using /etc/rc.local

    vi /etc/rc.local

put all parameters inside rc.local file, example:

    echo 220000000 > /proc/sys/vm/dirty_background_bytes
    echo 320000000 > /proc/sys/vm/dirty_bytes
    echo 0 > /proc/sys/vm/dirty_background_ratio
    echo 0 > /proc/sys/vm/dirty_ratio
    echo 500 > /proc/sys/vm/dirty_writeback_centisecs
    echo 4500 > /proc/sys/vm/dirty_expire_centisecs
    echo 1 > /proc/sys/net/ipv4/tcp_rfc1337
    echo 10 > /proc/sys/vm/swappiness
    echo never > /sys/kernel/mm/transparent_hugepage/enabled
    echo never > /sys/kernel/mm/transparent_hugepage/defrag
    echo 120 > /proc/sys/net/ipv4/tcp_keepalive_time
    echo 0 > /proc/sys/vm/zone_reclaim_mode
    echo deadline > /sys/block/sda/queue/scheduler
    echo 8 > /sys/class/block/sda/queue/read_ahead_kb
    echo 1048575 > /proc/sys/vm/max_map_count

quit vi editor saving the file.

Those parameters will be set on every reboot, AFTER all init services have started, just before the login prompt shows.

(/etc/rc.local file is executed after all startup linux services, it may not work if elasticsearch starts before it as a service, but this method can be useful on another setup if you need in future, or you can use like this by putting them inside your elasticsearch init script, because init script run as root, so it's the same syntax above to use inside init scripts)

You can also copy them now and paste them for instant changes. The parameters above are valid, tuned and running on my apache cassandra server. If you wish, try them as a start point to tune yours.

Second method to make them permanent:

Parameters now will be set BEFORE any startup service on linux.

Edit /etc/sysctl.conf, put parameters inside

 vm.max_map_count=1048575
 vm.zone_reclaim_mode=0
 vm.dirty_background_bytes=220000000
 vm.dirty_background_ratio=0
 vm.dirty_bytes=320000000
 vm.dirty_ratio=0
 vm.swappiness=10

keep going with the others, save /etc/sysctl.conf, reboot your server to apply changes, or execute: sysctl -p to apply the changes without reboot. They will be permanent across reboots.

Two methods above are the most commom. There's another one, and it may work for you, it's by using sudo, almost like you were doing:

instead of:

  sudo sysctl -w vm.max_map_count=262144

try:

  echo 262144 | sudo tee /proc/sys/vm/max_map_count

It works on ubuntu.

Verify:

   user@naos:~$ cat /proc/sys/vm/max_map_count
   262144

Hope I have helped someway, at least by giving the 3 different options to deal with the problem, since it's almost a year old your question ;)

Regards, Rafael Prado

Can’t see pod logs in Stackdriver UI for cluster deployed on GKE

Need to make sure the cluster is set up with Stackdriver logging/monitoring enabled.
Following the mentioned documentation, you should have a pod named counter. If you run:

$ kubectl logs counter # Are you able to see the output for that pod?

If you intend to customize Stackdriver Logs for Kubernetes Engine with Fluentd, this tutorial suggests to first create the cluster service account and assign required roles. Next, the cluster is created with "no-enable-cloud-logging"; and, deploying the Fluentd daemonset.
Trying the advanced filter like below should help:

resource.type="container"

resource.labels.pod_id="es-data-0"

Best Answer

Related Solutions

Java – Unable to change vm.max_map_count for elasticsearch

Can’t see pod logs in Stackdriver UI for cluster deployed on GKE

Related Topic