Development Setup¶
In this document some tips and workflows for development are loosely collected. Those are independent of the production setup using Docker-Compose. A working Firebase Project (Including Firebase Functions and Database Rules) is presupposed. Get in touch with the MapSwipe team (e.g. in Slack) to get access to an existing Firebase Instance for development purposes.
Check list:
Github: Clone repo from GitHub.
Requirements: Install GDAL/OGR and GDAL for Python on your machine.
Configuration: Set environment variables and get a Service Account Key File.
Database: Set up local Postgres database using Docker.
Python-Package: Install MapSwipe Workers Python package.
Run MapSwipe Workers.
Installation¶
Clone from GitHub¶
… and switch to development branch.
git clone https://github.com/mapswipe/python-mapswipe-workers.git
cd python-mapswipe-workers
git checkout dev
Requirements¶
MapSwipe Workers requires GDAL/OGR (gdal-bin
) and GDAL for Python (libgdal-dev
, python-gdal
) to be installed.
Furthermore, we rely on Docker to set up Postgres.
Configuration¶
All configurations values are stored in environment variables. Please refer to the documentation on Configuration for further details.
Service Account Key¶
The MapSwipe Workers requires a Service Account Key (serviceAccountKey.json
) to access Firebase database.
Request yours from the MapSwipe working group.
The path to the Service Account Key is defined in the GOOGLE_APPLICATION_CREDENTIALS
environment variable.
You could also set up your own Firebase instance. However, this is not recommended. If you still want to do it, get your Service Account Key from Firebase from Google Cloud Service Accounts.
Directories¶
MapSwipe Workers needs access to a data directory for logs and data for the API:
To create this directories run:
mkdir --parents ~/.local/share/mapswipe_workers
Note: XDG Base Directory Specification is respected
Database¶
Setup a local Postgres instance for MapSwipe Workers using the Dockerfile provided for development purposes (postgres/Dockerfile-dev
).
The Dockerfile for production (postgres/Dockerfile
) does need additional setup for build-in backup to Google Cloud Storage, which is not needed for local development. That is why a simplified Dockerfile for development is provided.
Make sure that the specified port is not in use already. If so, adjust the port (also in the .env
file).
docker build -t mapswipe_postgres -f ./postgres/Dockerfile-dev ./postgres
docker run -d -p 5432:5432 --name mapswipe_postgres -e POSTGRES_DB="$POSTGRES_DB" -e POSTGRES_USER="$POSTGRES_USER" -e POSTGRES_PASSWORD="$POSTGRES_PASSWORD" mapswipe_postgres
Or set up Postgres using the initdb.sql
file in the postgres/
folder.
Mapswipe-Workers Python Package¶
Export environment variables to current shell.
Create a Python virtual environment with
system-site-packages
option enabled to get access to GDAL/OGR Python packagesActivate the virtual environment.
Install MapSwipe Workers using pip.
Run it.
export $(cat .env | xargs)
cd mapswipe_workers
python -m venv --system-site-packages venv
source venv/bin/activate
pip install --editable mapswipe_workers/
mapswipe_workers --help
Yeah! If you reached this point, you are ready to get into coding. Below you find some more information on Logging, Firebase Functions and Database Backup. However, you don’t need this to get started for now.
Further Information¶
Logging¶
Mapswipe workers logs are generated using the Python logging module of the standard library (See Official docs or this Tutorial.
To use the logger object import it from the definitions
module:
from mapswipe_workers.definitions import logger
logger.info('information messages')
logger.waring('warning messages')
# Include stack trace in the log
try:
do_something()
except Exception:
logger.exception('Additional information.')
Default logging level is Info. To change the logging level edit the logging configuration which is found in the module definitions.py
.
Logs are written to STDOUT and ~/.local/share/mapswipe_workers/mapswipe_workers.log
.
Per default logging of third-party packages is disabled. To change this edit the definition module (mapswipe_workers/defintions.py
).
Set the disable_existing_loggers
parameter of the logging.config.fileConfig()
function to False
.
Firebase Functions¶
Firebase functions are used to increment/decrement or calculate various attribute values which are used by the MapSwipe App. This includes:
project.progress
project.numberOfTasks
project.resultCount
project.contributorCount
group.progress
group.finishedCount
group.requiredCount
user.projectContributionCount
user.groupContribtionCount
user.taskContributionCount
user.timeSpentMapping
user.contibutions{.projectId.groupId.{timestamp, startTime, endTime}}
Those functions will be directly or indirectly triggered by incoming results from the MapSwipe App.
By using Firebase functions those attributes can be calculated in real-time and be accessed by users immediately. The use of those functions also reduces the data-transfer between the Firebase Realtime Database and MapSwipe Workers.
On how to setup the development environment and how to deploy functions to the Firebase instance please refer to the official Guide on Cloud Function for Firebase. For more information refer to the official Reference on Cloud Function for Firebase. For example function take a look at this GitHub repository.
OSM OAuth 2¶
Firebase functions are also used to allow users to login to MapSwipe with their OpenStreetMap account. Refer to the notes in the app repository for more information.
Database Backup¶
Firebase¶
Manual Backup
curl https://
.firebaseio.com/.json?format=export ref: https://stackoverflow.com/questions/27910784/is-it-possible-to-backup-firebase-db
Postgres¶
Manual Backup
Backup database in compressed splited files of specified size:
pg_dump -U mapswipe -d mapswipe -h localhost -p 5432 | gzip | split -b 100m - mapswipe.pgsql.gz
Copy the backup to your local machine when logged into your local machine:
scp username@ipadress:mapswipe.pgsql.gz* /path/to/destination
Restore database backup from multiple compressed files
cat mapswipe.pgsql.gz* | gunzip | psql -U mapswipe -d mapswipe -h localhost -p 5432