https://blog.alekc.org/tags/recovery/Recent content in recovery on Alekc's BlogHugo -- gohugo.io©{year}, All Rights ReservedThu, 14 Jan 2021 16:40:59 +0000https://blog.alekc.org/posts/2021/recovery-of-etcd-failing-node/Thu, 14 Jan 2021 16:40:59 +0000https://blog.alekc.org/posts/2021/recovery-of-etcd-failing-node/<p>One of my etcd nodes in my home k8s cluster has been failing with following message:</p> <div class="highlight"><pre class="chroma"><code class="language-fallback" data-lang="fallback"><span class="ln">1</span>2021-01-14 11:16:09.233458 I | embed: listening for peers on 192.168.0.33:2380 <span class="ln">2</span>raft2021/01/14 11:16:09 tocommit(29492601) is out of range [lastIndex(29492469)]. Was the raft log corrupted, truncated, or lost? <span class="ln">3</span>panic: tocommit(29492601) is out of range [lastIndex(29492469)]. Was the raft log corrupted, truncated, or lost? </code></pre></div><p>These are steps I took to fix it:</p>