Recovery of etcd failing node

One of my etcd nodes in my home k8s cluster has been failing with following message:

12021-01-14 11:16:09.233458 I | embed: listening for peers on
2raft2021/01/14 11:16:09 tocommit(29492601) is out of range [lastIndex(29492469)]. Was the raft log corrupted, truncated, or lost?
3panic: tocommit(29492601) is out of range [lastIndex(29492469)]. Was the raft log corrupted, truncated, or lost?

These are steps I took to fix it:

Connect to one of the healthy nodes and prepare env variables for the connections

1export ETCDCTL_CACERT='/etc/kubernetes/pki/etcd/ca.crt'
2export ETCDCTL_CERT='/etc/kubernetes/pki/etcd/server.crt'
3export ETCDCTL_KEY='/etc/kubernetes/pki/etcd/server.key'
4export ETCDCTL_ENDPOINTS='https://[]:2379'

For the operations I used official documentation .

Verify current cluster members

1# etcdctl  member list
31afbd87f4cc07a99, started, nas,,
44de56726b08ede88, started, xps-server,,
57ad397dcfdcca303, started, cooler-master,,

The failing node is named nas, so we are going to remove it from the quorum

1etcdctl member remove 1afbd87f4cc07a99

verify that it has been removed

1# etcdctl  member list
24de56726b08ede88, started, xps-server,,
37ad397dcfdcca303, started, cooler-master,,

remove failing pod

1kubectl delete pod etcd-nas

on the failing node delete the corrupted data folder

1rm -rf /var/lib/etcd/member

Add the failing node back to the etcd cluster

1# etcdctl member add nas --peer-urls=""
2Member 693136f4829284e2 added to cluster d965929a4c4424e9

Verify the member list

1# etcdctl member list
243fc832867481d8c, unstarted, ,,,
34de56726b08ede88, started, xps-server,,
47ad397dcfdcca303, started, cooler-master,,

after restart of the node nas, I could see that the third member has joined and pod is up and running