Processes

Overview

The MapSwipe workers fulfill several tasks, which we call processes from here on. All processes can be executed using the run_mapswipe_worker.py script.

  • Import: Get a new project into the MapSwipe App.
  • Transfer Results: Get results from Firebase and transfer to a postgresql database for long term storage.
  • Update: Update the number of contributors and the progress for MapSwipe projects.
  • Export: Export aggregated results and statistics in json Format.
  • Archive: Archive a finished MapSwipe project.
  • Delete: Delete a MapSwipe project and all corresponding results in Firebase and postgresql.

You can easily run a process using the run_mapswipe_worker.py script using the following parameters:

  • -p or --process: choose the process you want to run (choices: ‘import’, ‘transfer_results’, ‘update’, ‘export’, ‘archive’, ‘delete’)
  • -mo or --modus: choose which mapswipe instance to use. (choices: ‘development’ (default), ‘production’)
  • -f or --filter: choose which projects to process (choices: ‘all’, ‘not_finished’, ‘active’, ‘list’)
  • -l or --list: project id of the project(s) to process. You can add multiple project ids. will only be considered if --filter=='list'
  • -o or --output: output path. Provide a location for storing files.
  • -s or --sleep_time: the time in seconds for which the script will pause in between two runs
  • -m or --max_iterations: the maximum number of runs that should be performed

Import

The import module uses data uploaded to the firebase imports table and creates the respective project information and information regarding groups and tasks.

python run_mapswipe_worker.py --process=import --modus=development

The import module gets data from firebase and creates the following tables/files:

  • data/project_{id}.kml: the geometry of the project as provided in the import
  • data/groups_{}.json: the groups, containing information on tasks, that are uploaded to firebase
  • data/groups_{id}.geojson: the groups, without further task information, and geometry and extent for each group
  • insert a column for each new project in mysql projects table

Transfer Results

The transfer results module downloads results from firebase results table and inserts them into the postgresql results table. All results that have been inserted successfully will be removed from firebase. Results that have already been imported (duplicates) will be removed from firebase.

python run_mapswipe_worker.py --process=transfer_results --modus=development

Update

The update module updates the progress and number of contributors for MapSwipe projects.

python run_mapswipe_worker.py --process=update --filter=active
python run_mapswipe_worker.py --process=update --filter=list --list=13505 --list=13507

The number of contributors is generated by counting the unique users from all results per project. The progress is generated using complementedCount and verificationCount for all groups of a project.

Export

The export module gets data from the postgresql database and saves it as json files.

python run_mapswipe_worker.py --process=export --filter=active --output=data
python run_mapswipe_worker.py --process=export --filter=list --list=13505 --list=13507 --output=data

In the given output directory, the following files will be created.

  • {outdir}/projects.json: contains information on all MapSwipe projects
  • {outdir}/stats.json: contains information on total number of users, total distance mapped, total contributions for all projects
  • {outdir}/users.json: contains information on name, mapped distance and number of contributions for all users
  • {outdir}/results/results_124.json, {outdir}/results/303.json, …: contains aggregated project results
  • {outdir}/progress/progress_124.txt, {outdir}/progress/progress_303.txt, …: contains information on number of contributors and progress (percent finished) per project

Archive

To be implemented and documented.

Delete

To be documented.