Introduction

So, after playing with Bareos for a bit, I realized that I had chewed off a bit more than I could swallow. It isn't that it's a bad product, it's just more nuclear weapon and I only need a grenade.

I looked around for a bit and stumbled upon Attic which seemed to be a pretty great bit of software for what I needed, and then I was informed that it had lagged a bit behind in development.

So, I looked at its fork: Borg. It looked like everything that I could want, only needed port 22 since I would be using a central server to house the backups, and was an easy install since it was available via pip.

Note: I know that most of my readers are more Arch Linux than Ubuntu. Knowing this, a forewarning: there will be some Ubuntu stuff in here as well.

Getting the Backup Server

This is an easy one: build a server with lots of storage for all of your backups. We'll use Arch Linux for the backup server, so you will need a few things:

  • Decent sized HDD
  • borg installed
  • openssh installed
  • A target user that servers can SSH into to save backups. (My example user is jweatherly, I would recommend something more like borguser).

So, in short:

Procedure

I'm assuming that the essential things are done such as acquiring a server (physical or VM) and installing Arch Linux on it, and that an account has been created with sudo privileges.

  1. Log into the server.

  2. Install borg on the server, and ensure that there's an ssh server installed.

    sudo pacman -S borg openssh

  3. Start and enable the newly installed ssh server.

    sudo systemctl enable sshd
sudo systemctl start sshd

  4. Make a directory on the server to house all of these backups. I chose /mnt/backups as my root, so we'll use that. Notice that the /mnt/backups directory is owned by my backup user.

    sudo mkdir -p /mnt/backups
sudo chown jweatherly:jweatherly /mnt/backups
chmod 0750 /mnt/backups

Installing the "Client"

We'll be pushing backups to the backup server via cron jobs eventually, but there's a bit of work to do before then.

Installing the Prerequisites

So, for the clients on the servers we'll need most of the same things that the backup server had. You don't strictly need openssh installed, but I'm guessing/assuming that it already is.

Procedure for Arch Linux

  1. First, install borg.

    sudo pacman -S borg cronie

  2. Ensure that cronie is enabled and running.

    sudo systemctl enable cronie
sudo systemctl start cronie

Procedure for Ubuntu

  1. Install the borg backup software.

    sudo apt install borgbackup

Getting Things Set Up

This procedure should work for both Ubuntu and Arch Linux. We'll initialize the new backup, then conduct an initial full backup to the backup server.

Then, at the end, we'll set up a cron job to make everything automatic.

Procedure

  1. First, let's generated some SSH keys for the root user. root will use these keys to connect to the backup server. We will use empty passphrases so that these keys can be used without user intervention.

    ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/root/.ssh/id_ed25519): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_ed25519.
Your public key has been saved in /root/.ssh/id_ed25519.pub.

  2. Next, we're going to log into the backup server using our user account that we set up in the very first section. We don't want to SSH as root. I'm going to use the jweatherly account from above.

    ssh-copy-id [email protected]
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

    Number of key(s) added: 1
    

    Now try logging into the machine, with:   "ssh '[email protected]'"
    
and check to make sure that only the key(s) you wanted were added.

  3. Now, SSH into the backup server and make a directory for the server's backups to live in. I name them off the first part of their FQDN (i.e. server1.localserver.com would be server1). Since the name of this server is server1, we'll use that as the directory name.

    hostname --short
server1
ssh [email protected]
Last login: Mon Jan 30 00:15:25 2017 from 10.11.12.13
mkdir /mnt/backups/server1
chmod 0750 /mnt/backup/server1
exit

  4. Next, let's initialize the /mnt/backups/server1 directory on the backup server via the borg init command. This is required before we start backup blasting to the remote server.

    We won't assign a passphrase to the encryption key (thought you may if you want to, just remember that you'll have to add it somewhere to automate everything).

    borg init --encryption=keyfile [email protected]:/mnt/backups/server1 
Enter new passphrase: 
Enter same passphrase again: 
Do you want your passphrase to be displayed for verification? [yN]:

  5. Okay, so let's start our first backup since everything is in place. I exclude several directories by default as they're not something you should try to backup: /dev, /proc, /sys, /var/run, /run, /lost+found, and /var/lib/lxcfs.

    The following commands look a bit messy, but we define three variables to clean it up slightly. The REPO variable just holds the location of our backups that we set up earlier. The HOST variable gives us good information about our host (notice that it's the same as the REPO directory) and the DATE just sets the date of the backup.

    [email protected]:/mnt/backups/server1
HOST=$(hostname)
DATE=$(date +%Y-%m-%d)
borg create -v --stats $REPO::$HOST-$DATE / --exclude /dev --exclude /proc --exclude /sys --exclude /var/run --exclude /run --exclude /lost+found --exclude /mnt --exclude /var/lib/lxcfs

  6. Now that the backup is complete, we will create a script that will backup our server and prune any backups that our outside of our requirements.

    For pruning, we'll set it up so that we have 7 daily backups, 4 weekly backups, and 6 monthly backups, and save it in the /etc/backup/backup_script.sh file.

    mkdir /etc/backup
chmod 755 /etc/backup
vim /etc/backup/backup_script.sh
    LOCAL=$(hostname --short)
[email protected]:/mnt/backups/$LOCAL
HOST=$(hostname)
DATE=$(date +Y%-%m-%d)

    borg create -v --stats $REPO::$HOST-$DATE / --exclude /dev --exclude /proc --exclude /sys --exclude /var/run --exclude /run --exclude /lost+found --exclude /mnt --exclude /var/lib/lxcfs
    

    Keep 7 daily backups, 4 weekly backups, and 6 monthly ones
    

    borg prune -v --list $REPO --keep-daily=7 --keep-weekly=4 --keep-monthly=6

  7. The script should be in place, and ready to go. We'll set it so that it's executable, then create the cronjob that will allow it to run everyday at 0100.

    chmod 0755 /etc/backup/backup_script.sh
crontab -e
    0 1 * * * /etc/backup/backup_script.sh
  8. Basically, enjoy your backups!

Conclusion

This should show a decent way of configuring backups on servers without too much issue. I'm almost positive that there's a mistake somewhere in this document, but this procedure mirror's my own when getting everything up and running.

I would recommend using things like ansible and puppet to help out with setting up these backups across multiple servers, or using some nfs-type links to manage the backup_scripts.sh file across the servers. Also, staggering the backup times in your cronjobs will help out greatly depending on the number of servers you're backing up.

Changelog

Hopefully, there won't be too many entries here. I'm probably wrong.

2017-01-29
  • Initial release.