7. Backup¶

Caution

Data backup is not performed automatically by Agora. It is the responsibility of the local Agora administrator. Data backup at regular intervals is essential for data loss prevention in case of hardware failure or other adverse events. The scripts and commands in this section are a suggestion for implementing a backup procedure. They may or may not fit a particular environment and will require customization.

Caution

Backups should be performed preferably during hours with low user-traffic on Agora. Remember to check regularly if your backup procedure is working and reliable. Make sure to test the complete recovery and restoring process as well.

Backups should include the following files and locations:

Configuration files (config.json, typically in /etc/gyrotools/agora)
Data files (content of the agora_data_1 volume)
The postgres database
The mongo database

7.1. Backup Configuration¶

The agora configuration is important for sytem recovery.

sudo rsync -aP /etc/gyrotools/agora /backup/agora_etc/

7.2. Backup Data Files¶

Data files can be backed up with a simple rsync script:

sudo rsync -aP /path/to/the/agora_data_1/volume /backup

Tip

If your are unsure in wich directory the data is stored use the docker inspect command:

docker volume inspect agora_data_1

[
    {
        "CreatedAt": "2021-05-04T20:00:25+02:00",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/agora_data_1/_data",
        "Name": "agora_data_1",
        "Options": {
            "device": "/agora_data/disk_01",
                    ^^^^^^^^^^^^^^^^^^^^^^-------------------- the relevant path

            "o": "bind",
            "type": "none"
        },
        "Scope": "local"
    }
]

Remember that you may have multiple data volumes configured in Agora. Make sure to backup all agora_data_X volumes. The following script is an example for an efficient file backup to another server using rsync.

#!/bin/bash

BACKUP_USER=user
BACKUP_SRV=IP
BACKUP_DIR=/home/gyrotools/backup/agora_data_01
SRC_DIR=/data/disk_01/agora_data_1

#########################################################################

# get the latest backup
LINK_DEST=$(ssh $BACKUP_USER@$BACKUP_SRV ls -r $BACKUP_DIR | head -n1)

# name of the current backup
CURRENT_BACKUP=`date "+%Y%m%d-%H%M%S"`

echo "Run backup $SRC_DIR => $BACKUP_DIR/$CURRENT_BACKUP with LINK_DEST=$BACKUP_DIR/$LINK_DEST"

rsync -a --delete --out-format="%t %f %b" -e ssh --link-dest=$BACKUP_DIR/$LINK_DEST $SRC_DIR  $BACKUP_USER@$BACKUP_SRV:$BACKUP_DIR/$CURRENT_BACKUP

echo "Done"

Note

This script creates incremental backups by using rsync’s --link-dest feature!

7.3. Backup Databases¶

As usual with databases, these cannot simply be backed up using rsync (unless Agora is in a halted state, but this would not be practical for a daily backup). The databases must be backed up with the appropriate ‘dump’ command.

The gtagora tool provides a simple command to back up all databases:

sudo gtagora db dump --backup-name agora-monday /backup/agora/

The databases will be dumped with the appropriate database tool (pg_dump and mongodump) and compressed using gzip.

Example of a script that creates a separate database backup for each day of the week:

#!/bin/sh

set -e  # immediately exits upon encountering an error

DAY_OF_WEEK=$(date +%u)
BACKUP_NAME=agora_$DAY_OF_WEEK
echo "Run Agora Databse Backup: $BACKUP_NAME"

gtagora db dump --backup-name $BACKUP_NAME /backup

# Send a email via the Mail Server configured in Agora
docker exec -t agora_app python /code/manage.py send_test_mail -s "[agora] DB Backup done" -c "`ls -lha /backup/${BACKUP_NAME}*`" -t admin@example.com

Database restore:

cat your_dump.sql | docker exec -i agora_app psql -U agora

Instead of using the gtagora tool, the database can also be dumped using docker commands directly:

#!/bin/sh

set -e
set -u

DAY_OF_WEEK=$(date +%u)
BACKUP_NAME=agora_dump_all_$DAY_OF_WEEK.sql
BACKUP_PATH=/agora_data/disk_01/backup/postgresql/
BACKUP_FILENAME=${BACKUP_PATH}/${BACKUP_NAME}
MAIL_RECEPIENT=xyz@gyrotools.com
echo "Run Agora postgresql backup: $BACKUP_NAME"

# Run pg_dumpall and store dump file inside container
# Note: It's also possible to use a pipe to stream the dump file to the host directly.
#       But the approach with storing the dump file inside the container seems to be
#       more stable.
docker exec -t agora_db pg_dumpall -c -U agora -f $BACKUP_NAME
# Copy the dump file out of the container to the final destination
docker cp agora_db:$BACKUP_NAME ${BACKUP_PATH}
# Delete the dump file inside the container
docker exec agora_db rm $BACKUP_NAME
# Send a report email
docker exec -t agora_app python /app/srv/manage.py send_test_mail -s "[gauss4] DB Backup done" -c "`ls -lha ${BACKUP_FILENAME}`" -t ${MAIL_RECEPIENT}

Caution

Do not just copy the database files in the agora_db volume. This would not be a reliable database backup and would likely result in data loss or a corrupted database. Always use the pg_dumpall tool to dump the database to a file.