Restoring with the Borg

So, you've got all of your backups working and you decide that you want to do a full restore using your borg backups? It's not as simple as simply restoring everything while the VM/computer is running, but it's not much more complicated.

A quick shopping list before we get started:

Your latest borg backup that has all of the files backed up.
The archlinux install DVD/ISO.

As a quick "heads-up": if you don't have a full backup, then this will not work. For example, if you're only backing up the /home directory, then you cannot restore the entire system from backup. You can restore that directory via the borg extract command without going through the steps in this guide.

Procedure

In the following procedure, the backup server is located at backup.server.local. The server I'm restoring is dns.server.local. The backup user that I used for the backups is borguser on the backup server. The backups I'm restoring are in the /mnt/backup/dns folder on the backup server. Remember, these are examples so make sure that you change them to your values.

Also, if your backups are encrypted via keyfiles, you will need them to decrypt the backups before you restore them. These are located in /root/.config/borg. To use them, you will have to copy them to /root/.config/borg on the server you're restoring (e.g. just copy the /root/.config/borg directory).

First, boot off of the Arch Linux install media. I will be using a VM because I'm writing this on my laptop on my couch. I will not apologize for comfort.
Get to the console, this is where the magic will happen. I've decided to access the ISO via SSH, but everything should work out if you do it directly as well.
If you haven't already or if your network info is set via DHCP, configure the networking so that you can access the backup server's backups. Since we'll be restoring from it, it will require network access.

Also, make sure that if you're hand-jamming your networking information that you add your DNS information to the /etc/resolv.conf file.

If your backup is located on physical media such as a USB hard drive, you won't really need the networking part.
```
ip addr add 10.0.0.2/24 dev eth0
route add default gw 10.0.0.1
```

Now, let's go ahead and install the borg backup software. We'll need this to restore the files from the backup server.

pacman -Sy borg
root@archiso ~ # pacman -Sy borg
:: Synchronizing package databases...
 core                     123.0 KiB   225K/s 00:01 [######################] 100%
 extra                   1685.9 KiB  2.06M/s 00:01 [######################] 100%
 community                  3.7 MiB  3.18M/s 00:01 [######################] 100%
resolving dependencies...
looking for conflicting packages...
Packages (8) python-3.6.0-2  python-appdirs-1.4.0-5  python-msgpack-0.4.8-2

python-packaging-16.8-2  python-pyparsing-2.1.10-2

python-setuptools-1:34.1.1-1  python-six-1.10.0-3  borg-1.0.9-2
Total Download Size:    31.80 MiB

Total Installed Size:  130.12 MiB
:: Proceed with installation? [Y/n] y

After borg has finished installing, go ahead and verify that all of the partitioning is correct. If it isn't, go ahead and fix it. Unfortunately, there's not much I can do to explain those steps.

For this guide, I'm going to pretend that the partition table is blank and that I'm going to set up three partitions (/, swap, and /boot). Also, this VM is tiny.

fdisk -l /dev/xvda
Disk /dev/xvda: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x837bbbe7
Device     Boot   Start      End  Sectors  Size Id Type

/dev/xvda1         2048   206847   204800  100M 83 Linux

/dev/xvda2       206848  2303999  2097152    1G 83 Linux

/dev/xvda3      2304000 20971519 18667520  8.9G 83 Linux

Next, I've (re)formatted those partitions using the appropriate mkfs and mkswap commands. This is critical, as we'll be writing the backup to those partitions once they are properly mounted.

Also, ensure that you run the swapon command for any of your swap partitions. Again, these are examples for my tiny VM.
```
mkfs.ext4 /dev/xvda1
mkswap /dev/xvda2 && swapon /dev/xvda2
mkfs.btrfs /dev/xvda3
mount /dev/xvda3 /mnt
mkdir /mnt/boot
mount /dev/xvda1
```

Since everything is mounted on /mnt, we're almost ready to begin. We'll need to install the initial Arch Linux filesystem to /mnt which is as easy as just installing the filesystem package.

Also, this is actually optional. The restore (if done properly) will have most of these directories. This step just keeps us from having to constantly chroot into /mnt to figure out what directories are missing.

pacstrap /mnt filesystem
==> Creating install root at /mnt
==> Installing packages to /mnt
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community                  3.7 MiB  3.83M/s 00:01 [######################] 100%
resolving dependencies...
looking for conflicting packages...
Packages (2) iana-etc-20161101-1  filesystem-2016.12-2
Total Download Size:   0.36 MiB

Total Installed Size:  3.79 MiB
:: Proceed with installation? [Y/n]

:: Retrieving packages...

iana-etc-20161101-1-any  355.6 KiB  8.08M/s 00:00 [######################] 100%

filesystem-2016.12-...     9.7 KiB  0.00B/s 00:00 [######################] 100%

(2/2) checking keys in keyring                     [######################] 100%

(2/2) checking package integrity                   [######################] 100%

(2/2) loading package files                        [######################] 100%

(2/2) checking for file conflicts                  [######################] 100%

(2/2) checking available disk space                [######################] 100%

:: Processing package changes...

(1/2) installing iana-etc                          [######################] 100%

(2/2) installing filesystem                        [######################] 100%

Now, navigate to the /mnt directory. This is where the borg magic will happen. Since borg is supposed to be run in the root of the filesystem, and since we're now in the root of the new filesystem, let's fire it up and get it restoring.

Grab a list of the backups currently available. We will restore the latest backup on the backup server.

cd /mnt
SRV=borguser@backup.server.local:/mnt/backup/dns
borg list $SRV
borguser@backup.server.local's password:
dns.server.local-2017-01-29       Sun, 2017-01-29 07:21:47
dns.server.local-2017-01-30       Mon, 2017-01-30 01:50:06
dns.server.local-2017-01-31       Tue, 2017-01-31 01:50:03
dns.server.local-2017-02-01       Wed, 2017-02-01 01:50:02
dns.server.local-2017-02-02       Thu, 2017-02-02 01:50:03
dns.server.local-2017-02-03       Fri, 2017-02-03 01:50:04
dns.server.local-2017-02-04       Sat, 2017-02-04 01:50:03
dns.server.local-2017-02-05       Sun, 2017-02-05 01:50:03

From the output above, the latest backup is the dns.server.local-2017-02-05 backup. We'll use that backup and feed it into borg.

Ensure that you are in the /mnt directory. Also, this command will take some time to execute as it is putting all of the files back where they are supposed to be.
```
REPO=dns.server.local-2017-02-05
borg extract $SRV::$REPO
borguser@backup.server.local's password:
borg extract $SRV::$REPO  144.96s user 56.57s system 51% cpu 6:34.59 total
```
So, what do we need to get this newly-restored server booting? Well, let's arch-chroot into it and run some basic commands.
```
arch-chroot /mnt
```
Let's fix the networking as there's a chance that the information may have changed. For this VM, it's been moved from ESXi to XenServer which means the ethernet interface changed from ens33 to eth0. Since I keep my ethernet configuration files directly in /etc/netctl in the ethernet-static file, I can run the following commands to fix it.

If you're using the same server, or the same VM, you probably won't have to do this as the interface name shouldn't change.
```
cd /etc/netctl
sed -i 's/ens33/eth0/g' ethernet-static
netctl reenable ethernet-static
```
The next command is to re-roll the initrd for the kernel. Since the media could have changed, it's a pretty good idea. If you are using a custom kernel, or an LTS kernel, then change the following command to the correct kernel.
```
mkinitcpio -p linux
```

Next, let's get grub installed to the MBR, and get a new /boot/grub/grub.cfg generated to boot.

grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-linux
Found initrd image(s) in /boot: initramfs-linux.img
Found fallback initrd image(s) in /boot: initramfs-linux-fallback.img
done
grub-install --target=i386-pc /dev/xvda
Installing for i386-pc platform.
Installation finished. No error reported.

Go ahead and exit the chroot environment. We have one thing left to do.
```
exit
```
Before everything is finished, let's update the /mnt/etc/fstab file with the new filesystem layout—assuming that the hard drive layout has changed.

Use the genfstab command to append the new configuration, and then remove the old configuration from the /mnt/etc/fstab file.
```
genfstab /mnt >> /mnt/etc/fstab
```
Go ahead and unmount everything, power-off the newly created server, and then reboot. You should have a fully restored machine that boots off the borg backup you created.

Conclusion

If everything went correctly, you should have a fully restored server from a borg backup. As always, if you see any issues, please let me know!

Changelog

2017-02-08

Fixed archiso/chroot mix-up in the networking section.

2017-02-07

Added networking fix into guide.

2017-02-06

Initial release.
Forgot to add a step to regenerate the /etc/fstab file using genfstab, fixed.