I'm managing an Elasticsearch deployment in Kubernetes. I see that the disk storage is getting close to full, so I would like to increase the size of the persistent volumes.
I want to change this value in the Stateful Set:
spec.
volumeClaimTemplates[0].spec.resources.requests.storage : "10Gi"
But when I do this using the Kubernetes-dashboard, I get the following message:
Internal server error
StatefulSet.apps "es-cluster" is invalid: spec: Forbidden: updates to statefulset > spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.
This makes me think that I will have to delete my existing Stateful Set and deploy a new one.
Is it possible to increase the per-pod disk storage without a service interruption or loss of data?
Additional detail
I have multiple Elasticsearch data pods and am using replica count=1, so if I am able to take them down and upgrade the disk storage one pod at a time, there shouldn't be a problem. I just don't see how to do this given the restriction indicated above.
Best Answer
Pieced this procedure together based on other comments on https://github.com/kubernetes/kubernetes/issues/68737. I tested this on kubernetes 1.14:
kubectl edit pvc <name>
for each PVC in the StatefulSet, to increase its capacity.kubectl delete sts --cascade=orphan <name>
to delete the StatefulSet and leave its pods.kubectl apply -f <name>
to recreate the StatefulSet.kubectl rollout restart sts <name>
to restart the pods, one at a time. During restart, the pod's PVC will be resized.If you want to monitor what's happening, run two more shell windows with these commands, before any of the commands above:
kubectl get pod -w
kubectl get pvc -w