7. Backup¶
Caution
Data backup is not performed automatically by Agora. It is the responsibility of the local Agora administrator. Data backup at regular intervals is essential for data loss prevention in case of hardware failure or other adverse events. The scripts and commands in this section are a suggestion for implementing a backup procedure. They may or may not fit a particular environment and will require customization.
Caution
Backups should be performed preferably during hours with low user-traffic on Agora. Remember to check regularly if your backup procedure is working and reliable. Make sure to test the complete recovery and restoring process as well.
Backups should include the following files and locations:
Configuration files (config.json, typically in /etc/gyrotools/agora)
Data files (content of the agora_data_1 volume)
The postgres database
The mongo database
7.1. Backup Configuration¶
The agora configuration is important for sytem recovery.
sudo rsync -aP /etc/gyrotools/agora /backup/agora_etc/
7.2. Backup Data Files¶
Data files can be backed up with a simple rsync script:
sudo rsync -aP /path/to/the/agora_data_1/volume /backup
Tip
If your are unsure in wich directory the data is stored use the docker inspect command:
docker volume inspect agora_data_1
[
{
"CreatedAt": "2021-05-04T20:00:25+02:00",
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/agora_data_1/_data",
"Name": "agora_data_1",
"Options": {
"device": "/agora_data/disk_01",
^^^^^^^^^^^^^^^^^^^^^^-------------------- the relevant path
"o": "bind",
"type": "none"
},
"Scope": "local"
}
]
Remember that you may have multiple data volumes configured in Agora. Make sure to backup all agora_data_X volumes. The following script
is an example for an efficient file backup to another server using rsync.
#!/bin/bash
BACKUP_USER=user
BACKUP_SRV=IP
BACKUP_DIR=/home/gyrotools/backup/agora_data_01
SRC_DIR=/data/disk_01/agora_data_1
#########################################################################
# get the latest backup
LINK_DEST=$(ssh $BACKUP_USER@$BACKUP_SRV ls -r $BACKUP_DIR | head -n1)
# name of the current backup
CURRENT_BACKUP=`date "+%Y%m%d-%H%M%S"`
echo "Run backup $SRC_DIR => $BACKUP_DIR/$CURRENT_BACKUP with LINK_DEST=$BACKUP_DIR/$LINK_DEST"
rsync -a --delete --out-format="%t %f %b" -e ssh --link-dest=$BACKUP_DIR/$LINK_DEST $SRC_DIR $BACKUP_USER@$BACKUP_SRV:$BACKUP_DIR/$CURRENT_BACKUP
echo "Done"
Note
This script creates incremental backups by using rsync’s --link-dest feature!
7.3. Backup Databases¶
As usual with databases, these cannot simply be backed up using rsync (unless Agora is in a halted state, but this would not be practical for a daily backup). The databases must be backed up with the appropriate ‘dump’ command.
The gtagora tool provides a simple command to back up all databases:
sudo gtagora db dump --backup-name agora-monday /backup/agora/
The databases will be dumped with the appropriate database tool (pg_dump and mongodump) and compressed using gzip.
Example of a script that creates a separate database backup for each day of the week:
#!/bin/sh
set -e # immediately exits upon encountering an error
DAY_OF_WEEK=$(date +%u)
BACKUP_NAME=agora_$DAY_OF_WEEK
echo "Run Agora Databse Backup: $BACKUP_NAME"
gtagora db dump --backup-name $BACKUP_NAME /backup
# Send a email via the Mail Server configured in Agora
docker exec -t agora_app python /code/manage.py send_test_mail -s "[agora] DB Backup done" -c "`ls -lha /backup/${BACKUP_NAME}*`" -t admin@example.com
Instead of using the gtagora tool, the database can also be dumped using docker commands directly:
#!/bin/sh
set -e
set -u
DAY_OF_WEEK=$(date +%u)
BACKUP_NAME=agora_dump_all_$DAY_OF_WEEK.sql
BACKUP_PATH=/agora_data/disk_01/backup/postgresql/
BACKUP_FILENAME=${BACKUP_PATH}/${BACKUP_NAME}
MAIL_RECEPIENT=xyz@gyrotools.com
echo "Run Agora postgresql backup: $BACKUP_NAME"
# Run pg_dumpall and store dump file inside container
# Note: It's also possible to use a pipe to stream the dump file to the host directly.
# But the approach with storing the dump file inside the container seems to be
# more stable.
docker exec -t agora_db pg_dumpall -c -U agora -f $BACKUP_NAME
# Copy the dump file out of the container to the final destination
docker cp agora_db:$BACKUP_NAME ${BACKUP_PATH}
# Delete the dump file inside the container
docker exec agora_db rm $BACKUP_NAME
# Send a report email
docker exec -t agora_app python /app/srv/manage.py send_test_mail -s "[gauss4] DB Backup done" -c "`ls -lha ${BACKUP_FILENAME}`" -t ${MAIL_RECEPIENT}
Caution
Do not just copy the database files in the agora_db volume. This would not be a reliable database backup and would likely result in data loss or a corrupted database. Always use the pg_dumpall tool to dump the database to a file.
7.4. Restore¶
To restore the Agora databases and the underlying data files, you can use the procedures described below. This documentation assumes a clean Linux installation (with Docker) and a backup of the Agora data files, databases and configuration as described in the sections above.
7.4.1. Download the Agora Installer¶
Download the Agora installer as described in the Installation Manual:
sudo curl -L https://portal.gyrotools.com/portal/download/gtagora/ -o /usr/bin/gtagora
sudo chmod +x /usr/bin/gtagora
7.4.2. Restore Agora Configuration¶
Copy the Agora configuration file to /etc/gyrotools/agora/:
sudo mkdir -p /etc/gyrotools/agora
sudo cp /path/to/config/backup/config.json /etc/gyrotools/agora/config.json
7.4.3. Create the Agora Volumes¶
Create all necessary docker volumes (see Data storage). You can either create them all at once in a common root directory or create them one by one and adapt the paths to your needs:
Create all volumes at once in a common root path (adapt the root path to your needs):
sudo gtagora volume create-all /agora_data
OR
Create each volume individually (adapt the paths to your needs):
sudo gtagora volume create agora_db /agora_data/db # Agora database (Postgres)
sudo gtagora volume create agora_mongo /agora_data/mongo # Parameter database (MongoDB)
sudo gtagora volume create agora_data_1 /agora_data/disk_01 # Agora data files
sudo gtagora volume create agora_log /agora_data/log # Agora log files
sudo gtagora volume create agora_nginx /agora_data/nginx # Nginx configuration files
7.4.4. Rebuild the docker-compose file and pull the Agora images:¶
sudo gtagora rebuild
sudo gtagora pull
7.4.5. Restore the Data Files¶
Copy the Agora data files to the appropriate volume. This is typically done with rsync or cp and might take some time depending on the size of the data files.
sudo cp -r /path/to/backup/agora_data_1/* $(docker volume inspect agora_data_1 --format '{{ .Options.device }}')/
7.4.6. Restore the Agora Database (Postgres)¶
Start the Agora database container:
sudo docker compose -f /etc/gyrotools/agora/docker-compose.yml up -d db
sudo docker ps # check if the database container is running and write down the container name
Hint
On older systems you might need to replace the “docker compose” command with “docker-compose”.
extract the zipped database dump into the database volume:
sudo gunzip -c /path/to/agora/database_dump.gz | sudo tee $(docker volume inspect agora_db --format '{{ .Options.device }}')/agora_db_dump.sql > /dev/null
Restore the database from the dump file:
docker exec -i agora_db bash -c '
cd /var/lib/postgresql/data/ # change to the Postgres data directory
chown postgres:postgres agora_db_dump.sql # make sure the dump file is owned by the postgres user
dropdb -U agora agora # delete the existing database
createdb -U agora agora # create a new database
psql -U agora -d agora -f agora_db_dump.sql # restore the database from the dump file
'
Warning
This will delete all existing data in the current database! This is irrelevant if you started with a fresh installation, but if you have an existing Agora installation, make sure to backup the current database first.
7.4.7. Restore the Parameter Database (MongoDB)¶
Start the MongoDB container:
sudo docker compose -f /etc/gyrotools/agora/docker-compose.yml up -d mongo
sudo docker ps # check if the MongoDB container is running and write down the container name
Hint
On older systems you might need to replace the “docker compose” command with “docker-compose”.
extract the zipped database dump into the mongo volume:
sudo gunzip -c /path/to/agora/mongo_database_dump.gz | sudo tee $(docker volume inspect agora_mongo --format '{{ .Options.device }}')/agora_db_dump.sql > /dev/null
Restore the database from the dump file:
docker exec -i agora_mongo mongorestore --drop --uri="mongodb://agora:agora@mongo:27017" --archive=/data/db/agora_mongo_dump.sql
Warning
This will delete all existing data in the current database! This is irrelevant if you started with a fresh installation, but if you have an existing Agora installation, make sure to backup the current database first.
7.4.8. Start Agora¶
Start Agora and check if everything is running correctly and all data is available:
sudo gtagora start