Centralised Backup With Borg¶
TL;DR¶
Why I Abandoned Duplicati?
I use to use Duplicati as backup solution for ages. It's a really good backup solution with a handy user interface. Everything can be done on it's interface.
My requirements toward a backup software:
- Differential backups on daily basis
- Support linux operation system
- Google Drive support for storing backup remotly
- Encryption, of course
- Be as lightweight as possible
I know Google Drive support a bit unusall, but I have 2TB storage and I don't want to pay for an other service. Duplicati fulfills all my requirments. There are only two weakness because of why I looked for another solution: resource consumption and speed:
- Duplicati can be really slow on restore from Google Drive if you store a lot of files.
- Duplicati uses [Mono](https://www.mono-project.com. (Cross platform, open source .NET framework). Running .NET on linux is not my taste, and sometimes consumes too much resource, especially on a Raspberry PI3.
After some hours of Googling and trying some softwares I found Borg Backup.
The only missing feature is the Google Drive, but it can be achieved with rclone
.
I don't want to write pages about my choice, features, advantages, disadvantages, etc. If you are reading this article you probably want to try or use Borg.
Installing Borg¶
I will create three Virtual Machine for demonstration the installation and usage.
Since the borg install procedure is always the same, I've done once and cloned the VM.
Install On Debian 11¶
OS version:
Update apt:
Install Borg Backup
Check installed version:
So now I have a Proxmox template with ID 102. I'm goning to create 3 clone:
qm clone 102 501 -full false -name borg-01
qm clone 102 502 -full false -name borg-02
qm clone 102 503 -full false -name borg-03
# Start
qm start 501
qm start 502
qm start 503
IP Addresses:
- borg-01:
172.16.1.236/22
- borg-02:
172.16.1.218/22
- borg-03:
172.16.1.219/22
Configure SSH And Users¶
The borg-01 will be the server, it will store the backups.
-
Create User
Command -
Create SSH keys
-
Distribute the private key accross the other servers.
Test:
Linux borg-01 5.13.19-1-pve #1 SMP PVE 5.13.19-2 (Tue, 09 Nov 2021 12:59:38 +0100) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Dec 11 15:51:46 2021 from 172.16.1.216
borg@borg-01:~$
Ok. It works well.
Warning
Please keep your private key safe. Since this is a demo environment I don't care much the security. I storngly recommend to use individual keys for each host. This post is not about ssh key management, but Borg backup.
Info
I'm using the root
user on the clients, bacuse only the root
account has access to the directories I want to backup.
Create & Mange Backups¶
Initialize & Passphare¶
-
Create very strong passphare
-
Initialize
Command OutputBy default repositories initialized with this version will produce security errors if written to with an older version (up to and including Borg 1.0.8). If you want to use these older versions, you can disable the check by running: borg upgrade --disable-tam ssh://borg@172.16.1.236/home/borg/borg-02 See https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability for details about the security implications. IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo! Use "borg key export" to export the key, optionally in printable format. Write down the passphrase. Store both at safe place(s).
Caution
If you lose the passphrase (~/.borg-passphrase
) you lose all of the backups, as well. So I do really recommend to keep it in a safe place, not just in the home directory!
Repeat this step on all nodes you want to create backup.
Important
There are two really important things to be kept in safe, the passphrase and the key. I strongly recommend to save them to you password manager or keep them somewhere in really safe.
Manage Your Key And Passphrase¶
I assume you are using password manager in any way, and hope this manager is not a plain text file. :)
Save your passphrase to your password manager is simple. Just save the conent of this file: ~/.borg-passphrase
BORG_KEY f968c50460d1ca7aaec9a8e2347a61fd286b26fb84adcaa6de7808966026b51e
hqlhbGdvcml0aG2mc2hhMjU2pGRhdGHaAZ7vhs+Yzwxg9VgXxo95S5+ScE8RT3yY6elK5J
KJKhfwz/YYJrGO6ZlDSpr9i+fnUI7qz6BfIxBLA6yILdcFVpOUuy99cDp79Uyc7wrIDnTV
sk0oiQWBt3710yM3hJQC84Q69grriPrF0jdzgSCvDKn+FNfQQgLTgnYMavxxnXZESTStng
zfxMtcJZMghEm1Mfd8ZwRTDXPgpF5z03bXy+7DrQ/btxgiW8G+h6DEccBDKvf0oAfDOPvH
sGgC2aBq+lqpUcxdIhpd+CZ0BzFkWCMrQkr3QOhlMbGtkqi7a78/rIYeJWevyWwODM7RvZ
i01qqbrDoflkRAg/LiY76p0wi46ls8Annygw9RY7YzOq7+xEvImGRYXX5joJ9Lb1GQ3Eh1
7MSFFdVRfAXbcAUlyQXZ+k/TzxZIFw7ZsXvQL33AFD1MwuXVJdXCJZFtWNUD97Cd5cTwEq
f6T5AofjK6WAIF5qD4RGVEoH0X8+7MJ6IHCu8aPbjnqVLvjR9Ubii7mS5gC9IRdaN5T61i
PjNC3Lm8TjqL8WlSjqfqvu2BczikaGFzaNoAIL0XSOEvCdQ46MtJO5/Q98J1mEDsC9tLVv
OBZZy+emXAqml0ZXJhdGlvbnPOAAGGoKRzYWx02gAg/tXk5wRp5YZlOHdzm+Gk+8f5Qi/f
s2VHKZJPL8BfecWndmVyc2lvbgE=
If you can save this file as attachment you are done. But not all password manager supports attachments, and the line brakes can be broken. In this case I recommend to use base64
:
Qk9SR19LRVkgZjk2OGM1MDQ2MGQxY2E3YWFlYzlhOGUyMzQ3YTYxZmQyODZiMjZmYjg0YWRjYWE2ZGU3ODA4OTY2MDI2YjUxZQpocWxoYkdkdmNtbDBhRzJtYzJoaE1qVTJwR1JoZEdIYUFaN3ZocytZend4ZzlWZ1h4bzk1UzUrU2NFOFJUM3lZNmVsSzVKCktKS2hmd3ovWVlKckdPNlpsRFNwcjlpK2ZuVUk3cXo2QmZJeEJMQTZ5SUxkY0ZWcE9VdXk5OWNEcDc5VXljN3dySURuVFYKc2swb2lRV0J0MzcxMHlNM2hKUUM4NFE2OWdycmlQckYwamR6Z1NDdkRLbitGTmZRUWdMVGduWU1hdnh4blhaRVNUU3RuZwp6ZnhNdGNKWk1naEVtMU1mZDhad1JURFhQZ3BGNXowM2JYeSs3RHJRL2J0eGdpVzhHK2g2REVjY0JES3ZmMG9BZkRPUHZICnNHZ0MyYUJxK2xxcFVjeGRJaHBkK0NaMEJ6RmtXQ01yUWtyM1FPaGxNYkd0a3FpN2E3OC9ySVllSldldnlXd09ETTdSdloKaTAxcXFickRvZmxrUkFnL0xpWTc2cDB3aTQ2bHM4QW5ueWd3OVJZN1l6T3E3K3hFdkltR1JZWFg1am9KOUxiMUdRM0VoMQo3TVNGRmRWUmZBWGJjQVVseVFYWitrL1R6eFpJRnc3WnNYdlFMMzNBRkQxTXd1WFZKZFhDSlpGdFdOVUQ5N0NkNWNUd0VxCmY2VDVBb2ZqSzZXQUlGNXFENFJHVkVvSDBYOCs3TUo2SUhDdThhUGJqbnFWTHZqUjlVYmlpN21TNWdDOUlSZGFONVQ2MWkKUGpOQzNMbThUanFMOFdsU2pxZnF2dTJCY3ppa2FHRnphTm9BSUwwWFNPRXZDZFE0Nk10Sk81L1E5OEoxbUVEc0M5dExWdgpPQlpaeStlbVhBcW1sMFpYSmhkR2x2Ym5QT0FBR0dvS1J6WVd4MDJnQWcvdFhrNXdScDVZWmxPSGR6bStHays4ZjVRaS9mCnMyVkhLWkpQTDhCZmVjV25kbVZ5YzJsdmJnRT0K
Restore command: echo -n [BASE64_STRING] | base64 -d
Tip
If you want to make the base64
string smaller you can use gzip
.
Encode: cat /tmp/key | gzip -c | base64
Decode: echo -n [BASE64_STRING] | base64 -d | gunzip -c
Create Backups¶
Important
Don't forget to export BORG_PASSCOMMAND
before you use borg
command!
export BORG_PASSCOMMAND="cat $HOME/.borg-passphrase"
------------------------------------------------------------------------------
Archive name: firstBackup
Archive fingerprint: 882ed726a7115928149aa438af4b78f09d322a34c17dd65f0bf7ce537092ee1b
Time (start): Sat, 2021-12-11 17:43:09
Time (end): Sat, 2021-12-11 17:43:11
Duration: 2.40 seconds
Number of files: 443
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 1.61 MB 654.41 kB 649.83 kB
All archives: 1.61 MB 654.41 kB 649.83 kB
Unique chunks Total chunks
Chunk index: 424 434
------------------------------------------------------------------------------
Create another backup:
borg create --stats borg@172.16.1.236:/home/borg/borg-02::secondBackup /root /etc /home /opt /var/log/
------------------------------------------------------------------------------
Archive name: secondBackup
Archive fingerprint: f61a6d2ce46fc6433a6cfa9cdb1f146933f897c6bce769d071644e7117684cd9
Time (start): Sat, 2021-12-11 17:44:53
Time (end): Sat, 2021-12-11 17:44:55
Duration: 1.56 seconds
Number of files: 470
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 51.16 MB 8.95 MB 8.35 MB
All archives: 52.77 MB 9.60 MB 9.00 MB
Unique chunks Total chunks
Chunk index: 458 898
------------------------------------------------------------------------------
Check & List Backups¶
-
List Backups
-
Check The Conent Of A Backup
Outputdrwx------ root root 0 Sat, 2021-12-11 17:36:41 root -rw-r--r-- root root 161 Tue, 2019-07-09 12:05:50 root/.profile -rw-r--r-- root root 571 Sat, 2021-04-10 22:00:00 root/.bashrc -rw------- root root 273 Sat, 2021-12-11 17:21:55 root/.bash_history drwx------ root root 0 Sat, 2021-12-11 17:26:32 root/.ssh -rw------- root root 2602 Sat, 2021-12-11 17:25:59 root/.ssh/id_rsa -rw-r--r-- root root 222 Sat, 2021-12-11 17:26:32 root/.ssh/known_hosts ... ...
Extact Content¶
For example we want to restore the home direcrory from the secondBackup
.
cd /tmp
mkdir restore
cd restore
borg extract borg@172.16.1.236:/home/borg/borg-02::secondBackup home
find
.
./home
./home/user
./home/user/.bash_history
./home/user/.bash_logout
./home/user/.bashrc
./home/user/.profile
Borg has a really lovely feature: you can mount any of your backup.
Mount A Backup¶
Command:
Check:
total 4
drwxr-xr-x 1 root root 0 Dec 11 18:01 .
drwxr-xr-x 18 root root 4096 Nov 21 13:38 ..
drwxr-xr-x 1 root root 0 Dec 11 17:43 etc
drwxr-xr-x 1 root root 0 Nov 21 13:42 home
drwxr-xr-x 1 root root 0 Nov 21 13:35 opt
drwx------ 1 root root 0 Dec 11 17:36 root
drwxr-xr-x 1 root root 0 Dec 11 18:01 var
You can browse inside the backup and restore any file you want.
If you don't need the mount anymore, don't forget to unmount:
Prune¶
Assume that you create backups every day. Probably you don't need every backup forever. You can prue your repository and keep only certain amount of backup. For example I use the following parameters to prune the repository:
This will keep 3 daily backup, 2 weekly and 5 monthly. What does it mean? Example:
Weekly 2: 2021-11-28T02:00:01 Sun, 2021-11-28 02:00:03 [c1c349e361dc5f..... ]
Monthly 1: 2021-11-30T02:00:01 Tue, 2021-11-30 02:00:03 [a83f0b4d9d686f..... ]
Weekly 1: 2021-12-05T02:00:01 Sun, 2021-12-05 02:00:04 [39980ab7451c33..... ]
Daily 3: 2021-12-09T02:00:01 Thu, 2021-12-09 02:00:02 [daf1c1ea020b16..... ]
Daily 2: 2021-12-10T02:00:01 Fri, 2021-12-10 02:00:03 [dd6fee702f0593..... ]
Daily 1: 2021-12-11T02:00:01 Sat, 2021-12-11 02:00:03 [e0385e46a1e968..... ]
Info
My script creates backup on every day at 02:00am.
keep-daily
--> keeps the last backup of each day.keep-weekly
--> keeps the last backup from the lastn
Sundaykeep-monthly
--> keeps the last backup from the last day of the month
Read more: https://borgbackup.readthedocs.io/en/stable/usage/prune.html
Backup With Crontab¶
I have a little shell script to automatize my daily backup:
#!/bin/bash
exec > >(logger -i --tag borgbackup) 2>&1
export BORG_PASSCOMMAND="cat $HOME/.borg-passphrase"
REPO_PATH="borg@172.16.1.236:/home/borg/borg-02"
REPO_BCK_DATE="$(date +%FT%T)"
BACKUPS="/etc/ /opt/ /home/ /root"
echo "==================== Creating Backup ===================="
borg create --stats ${REPO_PATH}::${REPO_BCK_DATE} ${BACKUPS}
echo "==================== Prune ===================="
borg prune --stats --keep-daily 3 --keep-weekly 2 --keep-monthly 5 ${REPO_PATH}
echo "==================== List ===================="
borg list ${REPO_PATH}
This script write logs to the syslog
:
Dec 11 18:23:56 borg-02 borgbackup[4557]: ==================== Creating Backup ====================
Dec 11 18:23:58 borg-02 borgbackup[4557]: ------------------------------------------------------------------------------
Dec 11 18:23:58 borg-02 borgbackup[4557]: Archive name: 2021-12-11T18:23:56
Dec 11 18:23:58 borg-02 borgbackup[4557]: Archive fingerprint: d8be120187e55f630cd4816b3879b2aae83f262a00336c3d20bd62da96dce7fa
Dec 11 18:23:58 borg-02 borgbackup[4557]: Time (start): Sat, 2021-12-11 18:23:56
Dec 11 18:23:58 borg-02 borgbackup[4557]: Time (end): Sat, 2021-12-11 18:23:57
Dec 11 18:23:58 borg-02 borgbackup[4557]: Duration: 0.37 seconds
Dec 11 18:23:58 borg-02 borgbackup[4557]: Number of files: 444
Dec 11 18:23:58 borg-02 borgbackup[4557]: Utilization of max. archive size: 0%
Dec 11 18:23:58 borg-02 borgbackup[4557]: ------------------------------------------------------------------------------
Dec 11 18:23:58 borg-02 borgbackup[4557]: Original size Compressed size Deduplicated size
Dec 11 18:23:58 borg-02 borgbackup[4557]: This archive: 1.61 MB 654.31 kB 53.47 kB
Dec 11 18:23:58 borg-02 borgbackup[4557]: All archives: 3.22 MB 1.31 MB 703.21 kB
Dec 11 18:23:58 borg-02 borgbackup[4557]:
Dec 11 18:23:58 borg-02 borgbackup[4557]: Unique chunks Total chunks
Dec 11 18:23:58 borg-02 borgbackup[4557]: Chunk index: 430 870
Dec 11 18:23:58 borg-02 borgbackup[4557]: ------------------------------------------------------------------------------
Dec 11 18:23:58 borg-02 borgbackup[4557]: ==================== Prune ====================
Dec 11 18:24:00 borg-02 borgbackup[4557]: ------------------------------------------------------------------------------
Dec 11 18:24:00 borg-02 borgbackup[4557]: Original size Compressed size Deduplicated size
Dec 11 18:24:00 borg-02 borgbackup[4557]: Deleted data: -1.61 MB -654.32 kB -53.48 kB
Dec 11 18:24:00 borg-02 borgbackup[4557]: All archives: 1.61 MB 654.31 kB 649.73 kB
Dec 11 18:24:00 borg-02 borgbackup[4557]:
Dec 11 18:24:00 borg-02 borgbackup[4557]: Unique chunks Total chunks
Dec 11 18:24:00 borg-02 borgbackup[4557]: Chunk index: 425 435
Dec 11 18:24:00 borg-02 borgbackup[4557]: ------------------------------------------------------------------------------
Dec 11 18:24:00 borg-02 borgbackup[4557]: ==================== List ====================
Dec 11 18:24:01 borg-02 borgbackup[4557]: 2021-12-11T18:23:56 Sat, 2021-12-11 18:23:56 [d8be120187e55f630cd4816b3879b2aae83f262a00336c3d20bd62da96dce7fa]
You can schedule this script using crontab
:
Save Backup To Google Drive¶
As I mentioned early in this article, one of my important goal is to save my backups to Google Drive. I did not bother too much with this. There is an excellent tool for linux: Rclone
After configuring rclone
you can upload the backups to Google Dirve from your central Borg server. Example command:
Quote
Sync the source to the destination, changing the destination only. Doesn't transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary.
Info
Since all of the backups are encrypted we don't need to bother with extra encryption or password protection.
There are two disadvantages over Duplicati from my point of view:
- This way we store the backup twice. (On the Borg central server and Google Drive)
- We need a running Borg Server.
- This can be avoided. You can create local backups and upload them individually from the hosts.
Thank you for reading! I don't know if this post was useful for you or not, but I think giving more and more examples is always helpful.