Restoring with the Borg

So, you've got all of your backups working and you decide that you want to do a full restore using your borg backups? It's not as simple as simply restoring everything while the VM/computer is running, but it's not much more complicated.

A quick shopping list before we get started:

  • Your latest borg backup that has all of the files backed up.
  • The archlinux install DVD/ISO.

As a quick "heads-up": if you don't have a full backup, then this will not work. For example, if you're only backing up the /home directory, then you cannot restore the entire system from backup. You can restore that directory via the borg extract command without going through the steps in this guide.

Procedure

In the following procedure, the backup server is located at backup.server.local. The server I'm restoring is dns.server.local. The backup user that I used for the backups is borguser on the backup server. The backups I'm restoring are in the /mnt/backup/dns folder on the backup server. Remember, these are examples so make sure that you change them to your values.

Also, if your backups are encrypted via keyfiles, you will need them to decrypt the backups before you restore them. These are located in /root/.config/borg. To use them, you will have to copy them to /root/.config/borg on the server you're restoring (e.g. just copy the /root/.config/borg directory).

  1. First, boot off of the Arch Linux install media. I will be using a VM because I'm writing this on my laptop on my couch. I will not apologize for comfort.
  2. Get to the console, this is where the magic will happen. I've decided to access the ISO via SSH, but everything should work out if you do it directly as well.

  3. If you haven't already or if your network info is set via DHCP, configure the networking so that you can access the backup server's backups. Since we'll be restoring from it, it will require network access.

    Also, make sure that if you're hand-jamming your networking information that you add your DNS information to the /etc/resolv.conf file.

    If your backup is located on physical media such as a USB hard drive, you won't really need the networking part.

    ip addr add 10.0.0.2/24 dev eth0
    route add default gw 10.0.0.1
  4. Now, let's go ahead and install the borg backup software. We'll need this to restore the files from the backup server.

    pacman -Sy borg
    root@archiso ~ # pacman -Sy borg
    :: Synchronizing package databases...
     core                     123.0 KiB   225K/s 00:01 [######################] 100%
     extra                   1685.9 KiB  2.06M/s 00:01 [######################] 100%
     community                  3.7 MiB  3.18M/s 00:01 [######################] 100%
    resolving dependencies...
    looking for conflicting packages...
    

    Packages (8) python-3.6.0-2 python-appdirs-1.4.0-5 python-msgpack-0.4.8-2
    python-packaging-16.8-2 python-pyparsing-2.1.10-2
    python-setuptools-1:34.1.1-1 python-six-1.10.0-3 borg-1.0.9-2

    Total Download Size: 31.80 MiB
    Total Installed Size: 130.12 MiB

    :: Proceed with installation? [Y/n] y

  5. After borg has finished installing, go ahead and verify that all of the partitioning is correct. If it isn't, go ahead and fix it. Unfortunately, there's not much I can do to explain those steps.

    For this guide, I'm going to pretend that the partition table is blank and that I'm going to set up three partitions (/, swap, and /boot). Also, this VM is tiny.

    fdisk -l /dev/xvda
    Disk /dev/xvda: 10 GiB, 10737418240 bytes, 20971520 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: dos
    Disk identifier: 0x837bbbe7
    

    Device Boot Start End Sectors Size Id Type
    /dev/xvda1 2048 206847 204800 100M 83 Linux
    /dev/xvda2 206848 2303999 2097152 1G 83 Linux
    /dev/xvda3 2304000 20971519 18667520 8.9G 83 Linux

  6. Next, I've (re)formatted those partitions using the appropriate mkfs and mkswap commands. This is critical, as we'll be writing the backup to those partitions once they are properly mounted.

    Also, ensure that you run the swapon command for any of your swap partitions. Again, these are examples for my tiny VM.

    mkfs.ext4 /dev/xvda1
    mkswap /dev/xvda2 && swapon /dev/xvda2
    mkfs.btrfs /dev/xvda3
    mount /dev/xvda3 /mnt
    mkdir /mnt/boot
    mount /dev/xvda1
  7. Since everything is mounted on /mnt, we're almost ready to begin. We'll need to install the initial Arch Linux filesystem to /mnt which is as easy as just installing the filesystem package.

    Also, this is actually optional. The restore (if done properly) will have most of these directories. This step just keeps us from having to constantly chroot into /mnt to figure out what directories are missing.

    pacstrap /mnt filesystem
    ==> Creating install root at /mnt
    ==> Installing packages to /mnt
    :: Synchronizing package databases...
     core is up to date
     extra is up to date
     community                  3.7 MiB  3.83M/s 00:01 [######################] 100%
    resolving dependencies...
    looking for conflicting packages...
    

    Packages (2) iana-etc-20161101-1 filesystem-2016.12-2

    Total Download Size: 0.36 MiB
    Total Installed Size: 3.79 MiB

    :: Proceed with installation? [Y/n]
    :: Retrieving packages...
    iana-etc-20161101-1-any 355.6 KiB 8.08M/s 00:00 [######################] 100%
    filesystem-2016.12-... 9.7 KiB 0.00B/s 00:00 [######################] 100%
    (2/2) checking keys in keyring [######################] 100%
    (2/2) checking package integrity [######################] 100%
    (2/2) loading package files [######################] 100%
    (2/2) checking for file conflicts [######################] 100%
    (2/2) checking available disk space [######################] 100%
    :: Processing package changes...
    (1/2) installing iana-etc [######################] 100%
    (2/2) installing filesystem [######################] 100%

  8. Now, navigate to the /mnt directory. This is where the borg magic will happen. Since borg is supposed to be run in the root of the filesystem, and since we're now in the root of the new filesystem, let's fire it up and get it restoring.

    Grab a list of the backups currently available. We will restore the latest backup on the backup server.

    cd /mnt
    SRV=borguser@backup.server.local:/mnt/backup/dns
    borg list $SRV
    borguser@backup.server.local's password:
    dns.server.local-2017-01-29       Sun, 2017-01-29 07:21:47
    dns.server.local-2017-01-30       Mon, 2017-01-30 01:50:06
    dns.server.local-2017-01-31       Tue, 2017-01-31 01:50:03
    dns.server.local-2017-02-01       Wed, 2017-02-01 01:50:02
    dns.server.local-2017-02-02       Thu, 2017-02-02 01:50:03
    dns.server.local-2017-02-03       Fri, 2017-02-03 01:50:04
    dns.server.local-2017-02-04       Sat, 2017-02-04 01:50:03
    dns.server.local-2017-02-05       Sun, 2017-02-05 01:50:03
  9. From the output above, the latest backup is the dns.server.local-2017-02-05 backup. We'll use that backup and feed it into borg.

    Ensure that you are in the /mnt directory. Also, this command will take some time to execute as it is putting all of the files back where they are supposed to be.

    REPO=dns.server.local-2017-02-05
    borg extract $SRV::$REPO
    borguser@backup.server.local's password:
    borg extract $SRV::$REPO  144.96s user 56.57s system 51% cpu 6:34.59 total
  10. So, what do we need to get this newly-restored server booting? Well, let's arch-chroot into it and run some basic commands.

    arch-chroot /mnt
  11. Let's fix the networking as there's a chance that the information may have changed. For this VM, it's been moved from ESXi to XenServer which means the ethernet interface changed from ens33 to eth0. Since I keep my ethernet configuration files directly in /etc/netctl in the ethernet-static file, I can run the following commands to fix it.

    If you're using the same server, or the same VM, you probably won't have to do this as the interface name shouldn't change.

    cd /etc/netctl
    sed -i 's/ens33/eth0/g' ethernet-static
    netctl reenable ethernet-static
  12. The next command is to re-roll the initrd for the kernel. Since the media could have changed, it's a pretty good idea. If you are using a custom kernel, or an LTS kernel, then change the following command to the correct kernel.

    mkinitcpio -p linux
  13. Next, let's get grub installed to the MBR, and get a new /boot/grub/grub.cfg generated to boot.

    grub-mkconfig -o /boot/grub/grub.cfg
    Generating grub configuration file ...
    Found linux image: /boot/vmlinuz-linux
    Found initrd image(s) in /boot: initramfs-linux.img
    Found fallback initrd image(s) in /boot: initramfs-linux-fallback.img
    done
    grub-install --target=i386-pc /dev/xvda
    Installing for i386-pc platform.
    Installation finished. No error reported.
  14. Go ahead and exit the chroot environment. We have one thing left to do.

    exit
  15. Before everything is finished, let's update the /mnt/etc/fstab file with the new filesystem layout—assuming that the hard drive layout has changed.

    Use the genfstab command to append the new configuration, and then remove the old configuration from the /mnt/etc/fstab file.

    genfstab /mnt >> /mnt/etc/fstab
  16. Go ahead and unmount everything, power-off the newly created server, and then reboot. You should have a fully restored machine that boots off the borg backup you created.

Conclusion

If everything went correctly, you should have a fully restored server from a borg backup. As always, if you see any issues, please let me know!

Changelog

2017-02-08
  • Fixed archiso/chroot mix-up in the networking section.
2017-02-07
  • Added networking fix into guide.
2017-02-06
  • Initial release.
  • Forgot to add a step to regenerate the /etc/fstab file using genfstab, fixed.