Long term maintenance plan: raw-data-api & export tool

[spwoodcock]

The current setup

Data export on demand

• Export tool is a Django backend containing:
  • Metadata for export jobs.
  • Static templates containing frontend code, that have variables injected on load.
• The actual data export for Export tool is done by raw-data-api underneath.
• raw-data-api doesn't store much metadata - only jobs and related info in the temporary redis queue until complete.

Note

The key distinction between Export Tool and raw-data-api, is that raw-data-api metadata is ephermeral. The data is extracted and stored, but the job history is not retained. Export tool is a platform where people make 'exports' and the metadata persists, so the user can visit their dashboard and view all of their exports over time.

HDX data exports

• Managed via Github workflow crons.
• Send a specific request to raw-data-api to export to HDX too.
• raw-data-api mostly handles this.

Alternatives to raw-data-api

• Overpass: difficult to work with & not very viable to do country-level exports (works on diffs).
• SliceOSM: run by OSMUS, can export OSM XML only. Could be an option to handle certain use cases. Would be good to do a comparison of raw-data-api / sliceosm capabilities, but I have a hunch SliceOSM won't offer everything raw-data-api does.

The problem

• raw-data-api also needs maintenance long term
  • infrastructure --> kubernetes
  • dependency updates
  • need to update database indexes
  • also needs new features, such as adding pcodes
• Export Tool is starting to age. We don't want another OAM experience, where it is not maintained for years and eventually becomes too difficult to fix.
  • The backend needs a Django version upgrade. Dependencies need upgrading.
  • The code needs a cleanout, removing all the old code related to data exporting (all handled by raw-data-api now).
  • The frontend is good! But unfortunately it's in Django templates, so not transferable.
• The HDX exports need to be handled by someone. Likely @spwoodcock. We need a good catalogue of what is running & docs for how to increase frequencies (during an 'activation') / add new exports over time.

Important

Export Tool aging is something we need to keep on top of - it needs to be addressed in 1-2yr time period - but it's not a burning high priority item right now. Still, the earlier we start addressing it in parts, the easier it will be.

Strategy

1. First check how many people return to same export multiple times? The metadata in Export Tool serves this purpose, but do we actually need it?

   • If possible, just replace the backend of export tool entirely with raw-data-api directly?
   • This would mean we don't keep extra metadata, such as which username the export is linked to.
   • The same export also couldn't be updated (via Export Tool 'linking' capability in the db).
   • With this approach, the metadata from export tool would be deleted in time (give a banner warning for half a year).
   • However, if it's a critical feature for users, then we need another approach: upgrade and maintain the Django API.

2. Next steps (mid-term)

   • Migrate raw-data-api --> Kubernetes
   • Possibly replace parts with SliceOSM, or other tools? Do an assessment?

3. Long term goal: additional integrations for raw-data-api.

   • We already have export tool frontend for exports.
   • A QGIS plugin to do this would be a bonus. Should be simple as we have https://github.com/hotosm/raw-data-api-py
   • Look at options for creating exports directly in uMap?

@kshitijrajsharma I can't find my notes I took at FOSS4G 😬 I'll keep looking!
In the meantime, which repo were the github workflows / cron schedules for the HDX exports stored?

[kshitijrajsharma]

In the meantime, which repo were the github workflows / cron schedules for the HDX exports stored?

Here : https://github.com/hotosm/osm-country-exports

[kshitijrajsharma]

One more option that I think worth trying out would be : if we dump latest osm xml to geoparquet then the country exports can be replaced with something like this https://github.com/osgeonepal/overture2hdx that would simplify the exports which doesn't need per minute update and daily update is good and would be performant ! something to lookinto , i am sure there will be lot of complications , but at some point i was considering this based on how geoparquet evolved !

[kshitijrajsharma]

if we can figure out the migration , raw-data-api does have a user table , if we manage the migration properly this could be another option something similar like this : https://github.com/kshitijrajsharma/raw-data-export but much closer to export tool UI so people won't notice too much difference

[spwoodcock]

Love these ideas!

Into the future I would love to trial out:

Large scale scheduled exports via GeoParquet (HDX)

• Large scale (country level) data exports are difficult to manage without raw-data-api (one of it's primary advantages).
• As you mention, we could have a daily schedule to export each country in our priority regions --> geoparquet on S3. https://github.com/osgeonepal/overture2hdx
  • A huge advantage of this approach is that we can then leverage all of the tooling built around geoparquet that the open-source community currently uses for Overture data in geoparquet.
• On daily / weekly / monthly schedules for specific exports required in HDX, we can extract excerpts from the country geoparquet --> geojson and other useful formats in HDX (replacing the raw-data-api requirement for HDX).

Export tool --> raw-data-api migration

• Migrate the user data from export tool --> raw-data-api db (might need some smart matching based on usernames?)
  • (Optional) migrate the export tool entries that are newer than 90 days (by default raw-data-api only keeps exports for 90 days).
  • Communicate to users that we will be deleting all exports older than 90 days old into the future (and prompt them to download any exports they wish to keep).
• Remove the export tool backend and replace with raw-data-api entirely.
• Create a new export tool frontend, based on the existing one or https://github.com/kshitijrajsharma/raw-data-export (shouldn't be a big task with a dedicated dev or two for a few weeks).

Reducing raw-data-api costs (ideas)

• Migrate raw-data-api database from AWS RDS --> Kubernetes cluster (RDS is overpriced, this would save a fair bit probably).
• Perhaps we only allow exports in our priority regions, and ask users to defer to other services such as SliceOSM if they need exports elsewhere (reducing the db size - would this also reduce costs? maybe not so much if we do the migration mentioned above)
• Queue more aggressively, with reduce autoscaling. We could simply make users wait a bit longer for their exports, and to autoscale so much during high demand. Just append to the queue and tick off the exports slowly as they are done (we have the separate queues in place for tools that need fast exports in <10s, such as TM, FieldTM, etc).

To really dig into this part, we need to better assess where the costs for raw-data-api actually come from. Is it the server costs, database storage cost?

[spwoodcock]

Adding notes from a slack convo, for future ref.

A rough summary of what I have been thinking over the past months:

1. Do thorough testing of Postpass or SliceOSM as replacements for most general exports in raw-data-api, and hook up as the new backend to Export Tool.
2. Revamp the export tool UI / API as suggested by Kshitij.
3. https://github.com/hotosm/osm-country-exports can actually run using geoparquet OSM exports, as mentioned by @kshitijrajsharma
4. ~~Significantly simplify raw-data-api, to no longer be an exposed API. Instead it's just an OSM database replica hosted in our Kubernetes cluster (it's approx ~1TB, ~$80/month). Run the HDX exports as scheduled Kubernetes jobs, monthly~~ replace by geoparquet idea above - we can shut down the db, the main expense.