Fixing etcdDatabaseHighFragmentationRatio prometheus alert

If you receive a following error in alertmanager

1etcd cluster "kube-etcd": database size in use on instance xxx.xxx.xxx.xxx:2381
2is 49.55% of the actual allocated disk space, please run defragmentation 
3(e.g. etcdctl defrag) to retrieve the unused fragmented disk space.

it can be easily solved by running a following command:

1kubectl exec $(kubectl get pods --selector=component=etcd -A -o name \
2| head -n 1) -n kube-system -- etcdctl defrag --cluster \
3--cacert /etc/kubernetes/pki/etcd/ca.crt \
4--key /etc/kubernetes/pki/etcd/server.key \
5--cert /etc/kubernetes/pki/etcd/server.crt

Note, the above would work if you are running a cluster installed by kubeadm, otherwise your mileage may vary

Copyright

Comments