Backup

For Postgres backups WAL-G is used. “WAL-G is an archival restoration tool for Postgres”.

For more information please refer to the official docs of WAL-G and the official docs of Postgres on write ahead log (WAL). Current setup took inspiration from this blog post.

Backup setup

The WAL-G backup setup is integrated into the Postgres Docker image. It will do a baseline backup of the database to Google Cloud Storage every day utilizing a cron job and wal-g backup-push. After that the Postgres will push WAL (Write-Ahead Log) files to Google Cloud Storage regularly using wal-g wal-push. For exact commands please take a look at following script:

  • postgres/wal-g/make_basebackup.sh

  • postgres/wal-g/archive_command.sh

WAL-G is installed alongside Postgres. See the Dockerfile of Postgres (postgres/Dockerfile) for details. In the docker-compose postgres command (docker-compose.yml) archive parameter of postgres are set needed to make archives.

Configuration

To store backups in Google Cloud Storage, WAL-G requires that this variable is set: WALG_GS_PREFIX to specify where to store backups (eg. gs://x4m-test-bucket/walg-folder). Please add this to the .env file at the root of MapSwipe Back-end (See .example-env for environment variables which have to be set)

WAL-G determines Google Cloud credentials using application-default credentials like other GCP tools. Get a Service Account Key file (serviceAccountKey.json) for your Google Cloud Storage (See Google Cloud Docs) and save it to postgres/serviceAccountKey.json.

Restore setup

The WAL-G restore setup is realized in a dedicated Docker image (postgres/recovery/Dockerfile). The entrypoint is the ini_recovery.sh srcipt. This script will create a new Postgres database cluster, fetch latest backup using wal-g backup-fetch and create a recovery.conf file in the new cluster. recovery.conf is used by Postgres during first start to get the restore_command. Again the exact commands are to be found in postgres/recovery/restore_command.sh. During first start Postgres will get WALs from backup server and restore the database.

Configuration

The same configuration as for the backup setup is requiered. Except the Service Account Key has to be stored at postgres/recovery/serviceAccountKey.