How to increase disk size in a stateful set

amazon-ebselasticsearchkubernetes

I'm managing an Elasticsearch deployment in Kubernetes. I see that the disk storage is getting close to full, so I would like to increase the size of the persistent volumes.

I want to change this value in the Stateful Set:

spec.
volumeClaimTemplates[0].spec.resources.requests.storage : "10Gi"

But when I do this using the Kubernetes-dashboard, I get the following message:

Internal server error

StatefulSet.apps "es-cluster" is invalid: spec: Forbidden: updates to statefulset > spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.

This makes me think that I will have to delete my existing Stateful Set and deploy a new one.

Is it possible to increase the per-pod disk storage without a service interruption or loss of data?

Additional detail

I have multiple Elasticsearch data pods and am using replica count=1, so if I am able to take them down and upgrade the disk storage one pod at a time, there shouldn't be a problem. I just don't see how to do this given the restriction indicated above.

Best Answer

Pieced this procedure together based on other comments on https://github.com/kubernetes/kubernetes/issues/68737. I tested this on kubernetes 1.14:

  1. kubectl edit pvc <name> for each PVC in the StatefulSet, to increase its capacity.
  2. kubectl delete sts --cascade=orphan <name> to delete the StatefulSet and leave its pods.
  3. kubectl apply -f <name> to recreate the StatefulSet.
  4. kubectl rollout restart sts <name> to restart the pods, one at a time. During restart, the pod's PVC will be resized.

If you want to monitor what's happening, run two more shell windows with these commands, before any of the commands above:

  • kubectl get pod -w
  • kubectl get pvc -w
Related Topic