Longhorn is a brilliant piece of software (especially for those who cannot use block volumes on their Kubernetes cluster), but it can (and does) happen to have some volumes getting degraded and thus requiring to recover from a backup.
Usually I use one of two approaches / yaml files to achieve recovery
Case 1 - Manual recovery
Let’s assume that we have a volume (pvc
my-volume) which is fully functional, but we want to recover some files from the previous backup. To do so, mount the old backup as a new volume, and create a new pvc for it (
If you can support a bit of a downtime, then the fastest solution would be to delete the workload using
my-volume so that it can be used by another pod (unless your volumes do support ReadWriteMany), then mount old and new volume through a following job
1apiVersion: batch/v1 2kind: Job 3metadata: 4 name: migration 5spec: 6 completions: 1 7 parallelism: 1 8 backoffLimit: 3 9 template: 10 metadata: 11 name: volume-migration 12 labels: 13 name: volume-migration 14 spec: 15 restartPolicy: Never 16 containers: 17 - name: volume-migration 18 image: alpine 19 tty: true 20 command: [ "/bin/sh" ] 21 args: [ "-c", "sleep 10000" ] 22 volumeMounts: 23 - name: new-vol 24 mountPath: /mnt/new 25 - name: old-vol 26 mountPath: /mnt/old 27 volumes: 28 - name: old-vol 29 persistentVolumeClaim: 30 claimName: my-volume-old # change to data source pvc 31 - name: new-vol 32 persistentVolumeClaim: 33 claimName: my-volume # change to data target pvc
it will create a new pod to which you can attach a shell and perform copy/paste operation.
Once you have recovered files you need, simply delete the job and recreate your preexisting workload.
Let’s assume that you have a volume which you want to recover to a previous backup (for the semplicity I am assuming that it’s empty, if not you probably want to amend the code and append rm -rf as the first command).
As in previous case, mount the backup as a volume, and create a pvc for it. Then you can fire off the job which will automatically copy over files from the old volume to the new one while maintaining the owner and all file masks.
1apiVersion: batch/v1 2kind: Job 3metadata: 4 name: volume-migration 5spec: 6 completions: 1 7 parallelism: 1 8 backoffLimit: 3 9 template: 10 metadata: 11 name: volume-migration 12 labels: 13 name: volume-migration 14 spec: 15 restartPolicy: Never 16 containers: 17 - name: volume-migration 18 image: ubuntu:xenial 19 tty: true 20 command: [ "/bin/sh" ] 21 args: [ "-c", "apk add rsync && rsync -arv /mnt/old/ /mnt/new/" ] 22 volumeMounts: 23 - name: old-vol 24 mountPath: /mnt/old 25 - name: new-vol 26 mountPath: /mnt/new 27 volumes: 28 - name: old-vol 29 persistentVolumeClaim: 30 claimName: qbittorrent-backup # change to data source pvc 31 - name: new-vol 32 persistentVolumeClaim: 33 claimName: qbittorrent # change to data target pvc
Manual mount of an existing volume
In casae you need to mount an existing volume (ie in order to investigate the file structure, you can mount it via
apiVersion: batch/v1 kind: Job metadata: name: migration spec: completions: 1 parallelism: 1 backoffLimit: 3 template: metadata: name: volume-migration labels: name: volume-migration spec: restartPolicy: Never containers: - name: volume-migration image: alpine tty: true command: [ “/bin/sh” ] args: [ “-c”, “sleep 10000” ] volumeMounts: - name: new-vol mountPath: /mnt/new volumes: - name: new-vol persistentVolumeClaim: claimName: home-assistant