k8s on Alekc's Blog

Useful debugging tools for kubernetes

Sat, 03 Dec 2022 00:00:00 +0000

Sometimes its useful to be able to run some ephemeral containers on kubernetes cluster in order to perform some debugging (i.e. dns resolution, pinging nodes, etc). Sadly, most of the times, it can be tricky to remember all overrides so here is a small list Run an ephemeral shell on a random node 1kubectl run -ti --restart=Never --rm debug --image=alekcander/network-diag -- sh Run a container with specific service account 1kubectl run -ti --restart=Never --rm debug --image=alekcander/network-diag --overrides='{ "spec": { "serviceAccount": "terraform-runner" } }' -- sh Run on a specific node 1kubectl run -ti --restart=Never --rm debug --image=alekcander/network-diag --overrides='{"spec": { "nodeName": "alekc-worker-01"}}' -- sh Run based on specific selector (ie.

Fixing etcdDatabaseHighFragmentationRatio prometheus alert

Sat, 03 Sep 2022 23:00:00 +0000

If you receive a following error in alertmanager 1etcd cluster "kube-etcd": database size in use on instance xxx.xxx.xxx.xxx:2381 2is 49.55% of the actual allocated disk space, please run defragmentation 3(e.g. etcdctl defrag) to retrieve the unused fragmented disk space. it can be easily solved by running a following command: 1kubectl exec $(kubectl get pods --selector=component=etcd -A -o name \ 2| head -n 1) -n kube-system -- etcdctl defrag --cluster \ 3--cacert /etc/kubernetes/pki/etcd/ca.

Kubernetes networking

Wed, 17 Aug 2022 08:00:00 +0000

Kubernetes networking Pause container In Kubernetes, the pause container serves as the “parent container” for all of the containers in your pod, and it has two main responsibilities: it serves as the basis of Linux namespace sharing in the pod with PID (process ID) namespace sharing enabled, it serves as PID 1 for each pod and reaps zombie processes. In Linux, each running process communicates within a network namespace that provides a logical networking stack with its own routes, firewall rules, and network devices.

Longhorn disaster recovery jobs

Mon, 15 Aug 2022 02:00:00 +0000

Longhorn is a brilliant piece of software (especially for those who cannot use block volumes on their Kubernetes cluster), but it can (and does) happen to have some volumes getting degraded and thus requiring to recover from a backup.

How to create cross cloud self managed kubernetes cluster

Sun, 14 Aug 2022 23:00:00 +0000

While it’s fairly “trivial” to install a stacked kubernetes cluster with kubeadm on any cloud provider or managed bare metal (where you have a certain degree of management over the networking which permits you to use bgp for example), it’s not so trivial when your nodes are situated in different network segments (clouds) and/or behind NAT.

With this guide I will try to alleviate a pain related to this kind of setup.

Prometheus - kube-proxy endpoint connection refused

Mon, 30 May 2022 23:00:00 +0000

When using kube-prometheus-stack and kubernetes cluster provisioned by kubeadm you will likely have an issue of prometheus not being able to connect to the kube-proxy metrics It can be easily fixed by editing it’s config map kubectl edit cm/kube-proxy -n kube-system Change the metricsBindAddress property from 127.0.0.1:10249 (or empty value) to metricsBindAddress: 0.0.0.0:10249 Delete and redeploy kube-proxy pods kubectl delete pod -l k8s-app=kube-proxy -n kube-system

How to mass remove finalizers from argocd applications

Mon, 24 Jan 2022 11:00:00 +0000

If you are using finalizers in your argo-cd applications, you may find that it’s impossible to delete them if argo-cd installation is broken. In order to fix it, you can either update them one by one with kubectl edit and remove the finalizer tag, or you can run following command which will patch all argocd entities in argocd namespace kubectl get Application -A -o name | xargs kubectl patch -p '{"metadata":{"finalizers":null}}' --type=merge -n argocd

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation

Wed, 29 Sep 2021 05:40:59 +0000

When working with EKS under AWS, it’s possible that at some point you wanted to run a pod under a certain role, and you’ve encountered a following error:

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

What’s frustrating, is that by default AWS doesn’t provide you a lot of feedback of why that error happened.

So I’ve written down some debug steps for further reference:

Recovery of etcd failing node

Thu, 14 Jan 2021 16:40:59 +0000

One of my etcd nodes in my home k8s cluster has been failing with following message:

12021-01-14 11:16:09.233458 I | embed: listening for peers on 192.168.0.33:2380
2raft2021/01/14 11:16:09 tocommit(29492601) is out of range [lastIndex(29492469)]. Was the raft log corrupted, truncated, or lost?
3panic: tocommit(29492601) is out of range [lastIndex(29492469)]. Was the raft log corrupted, truncated, or lost?

These are steps I took to fix it: