Add daily OSM postbox import workflow

.github/workflows/osm-import.yml

@@ -0,0 +1,166 @@
+name: OSM postbox import
+
+# Daily OSM → Firestore sync: runs osm_import.sh, which downloads every UK
+# amenity=post_box node from Overpass and hands it to
+# functions/import_postboxes.js (incremental upsert of postbox/osm_* docs plus
+# meta/stats.totalPostboxes). The refreshed postboxes.json is NOT committed
+# back — Firestore is the source of truth and the download is discarded.
+#
+# Credentials: keyless Workload Identity Federation. The importer uses
+# Application Default Credentials, which google-github-actions/auth provides
+# via GOOGLE_APPLICATION_CREDENTIALS. One-time GCP setup:
+#
+#   PROJECT_ID=the-postbox-game
+#   PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')
+#   SA=osm-importer@$PROJECT_ID.iam.gserviceaccount.com
+#   gcloud services enable iamcredentials.googleapis.com sts.googleapis.com --project $PROJECT_ID
+#   gcloud iam service-accounts create osm-importer --project $PROJECT_ID \
+#     --display-name "OSM postbox importer (GitHub Actions)"
+#   gcloud projects add-iam-policy-binding $PROJECT_ID \
+#     --member serviceAccount:$SA --role roles/datastore.user
+#   gcloud iam workload-identity-pools create github --project $PROJECT_ID \
+#     --location global --display-name "GitHub Actions"
+#   gcloud iam workload-identity-pools providers create-oidc postbox-game \
+#     --project $PROJECT_ID --location global --workload-identity-pool github \
+#     --issuer-uri https://token.actions.githubusercontent.com \
+#     --attribute-mapping "google.subject=assertion.sub,attribute.repository=assertion.repository,attribute.ref=assertion.ref" \
+#     --attribute-condition "assertion.repository == 'code418/postbox_game' && assertion.ref == 'refs/heads/master'"
+#   gcloud iam service-accounts add-iam-policy-binding $SA --project $PROJECT_ID \
+#     --role roles/iam.workloadIdentityUser \
+#     --member "principalSet://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/github/attribute.repository/code418/postbox_game"
+#
+# then set these repository VARIABLES (not secrets — neither is sensitive):
+#   GCP_WORKLOAD_IDENTITY_PROVIDER =
+#     projects/<PROJECT_NUMBER>/locations/global/workloadIdentityPools/github/providers/postbox-game
+#   GCP_IMPORT_SERVICE_ACCOUNT = osm-importer@the-postbox-game.iam.gserviceaccount.com
+#
+# The ref == refs/heads/master condition means only a workflow running from
+# master can mint Firestore credentials: a branch dispatch or a PR workflow
+# cannot write production data. Scheduled runs only ever fire from the default
+# branch anyway, so nothing runs until this file is on master.
+
+on:
+  schedule:
+    # 03:17 UTC daily: off the hour (top-of-hour crons queue longest) and well
+    # clear of newDayScoreboard's London-midnight rollover.
+    - cron: '17 3 * * *'
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: 'Report only (--dry-run): no Firestore writes, manifest untouched'
+        type: boolean
+        default: false
+      prune:
+        description: 'Soft-mark osm_* docs whose node has left OSM (--prune)'
+        type: boolean
+        default: true
+      full_reimport:
+        description: 'Ignore the cached manifest and rewrite every postbox (--no-manifest)'
+        type: boolean
+        default: false
+
+permissions:
+  contents: read
+  # OIDC token for Workload Identity Federation.
+  id-token: write
+
+# Never two imports at once: they would race on Firestore and on the manifest
+# cache. A manual dispatch during the scheduled run queues behind it.
+concurrency:
+  group: osm-import
+  cancel-in-progress: false
+
+jobs:
+  import:
+    runs-on: ubuntu-latest
+    # curl may retry the 600 s Overpass query five times; a full import of
+    # ~115k docs takes several minutes on top.
+    timeout-minutes: 90
+
+    steps:
+      - uses: actions/checkout@v6
+
+      - uses: actions/setup-node@v4
+        with:
+          # Same runtime as functions/package.json and functions-ci.yml.
+          node-version: '22'
+          cache: 'npm'
+          cache-dependency-path: functions/package-lock.json
+
+      - name: Install importer dependencies
+        working-directory: functions
+        # The importer needs only firebase-admin and ngeohash.
+        run: npm ci --omit=dev
+
+      - name: Check GCP configuration
+        if: vars.GCP_WORKLOAD_IDENTITY_PROVIDER == '' || vars.GCP_IMPORT_SERVICE_ACCOUNT == ''
+        run: |
+          echo "::error::Repository variables GCP_WORKLOAD_IDENTITY_PROVIDER and GCP_IMPORT_SERVICE_ACCOUNT must be set (see the setup notes at the top of .github/workflows/osm-import.yml)."
+          exit 1
+
+      # After checkout, which would otherwise clean away the credentials file
+      # this writes into the workspace.
+      - uses: google-github-actions/auth@v3
+        with:
+          workload_identity_provider: ${{ vars.GCP_WORKLOAD_IDENTITY_PROVIDER }}
+          service_account: ${{ vars.GCP_IMPORT_SERVICE_ACCOUNT }}
+
+      # The importer only writes nodes whose hash changed since the previous
+      # run, tracked in a local (gitignored) manifest. A fresh runner has none,
+      # so persist it in the Actions cache: a unique key per run, restored by
+      # prefix (newest wins). Superseded entries are never read again and
+      # expire after 7 days unused; losing the cache just costs one full
+      # re-import, which is still correct.
+      - name: Restore import manifest
+        uses: actions/cache/restore@v6
+        with:
+          path: functions/.last_import_manifest.json
+          key: osm-import-manifest-${{ github.run_id }}-${{ github.run_attempt }}
+          restore-keys: osm-import-manifest-
+
+      - name: Fetch from Overpass and import
+        env:
+          DRY_RUN: ${{ inputs.dry_run }}
+          PRUNE: ${{ inputs.prune }}
+          FULL_REIMPORT: ${{ inputs.full_reimport }}
+        run: |
+          args=()
+          if [ "$DRY_RUN" = "true" ]; then args+=(--dry-run); fi
+          # Scheduled runs have no inputs (empty string), so they prune. The
+          # importer refuses (and fails the run) if a truncated export would
+          # soft-mark more than 5% of the postboxes.
+          if [ "$PRUNE" != "false" ]; then args+=(--prune); fi
+          if [ "$FULL_REIMPORT" = "true" ]; then args+=(--no-manifest); fi
+          # The default shell sets pipefail, so tee keeps the script's exit code.
+          ./osm_import.sh "${args[@]}" 2>&1 | tee "$RUNNER_TEMP/import.log"
+
+      # Also after a failure: the importer leaves failed batches out of the
+      # manifest it writes, so saving it still makes the next run retry them,
+      # and a run that died before writing one just re-saves the restored copy.
+      - name: Save import manifest
+        if: always() && inputs.dry_run != true && hashFiles('functions/.last_import_manifest.json') != ''
+        uses: actions/cache/save@v6
+        with:
+          path: functions/.last_import_manifest.json
+          key: osm-import-manifest-${{ github.run_id }}-${{ github.run_attempt }}
+
+      - name: Job summary
+        if: always()
+        run: |
+          [ -f "$RUNNER_TEMP/import.log" ] || exit 0
+          {
+            echo '### OSM postbox import'
+            echo '```'
+            sed -n '/^Import summary:/,$p' "$RUNNER_TEMP/import.log"
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"
+
+      # osm_import.sh leaves the download as postboxes.json.tmp when it fails
+      # its sanity check (Overpass error page, throttle response, truncation).
+      - name: Upload rejected Overpass response
+        if: failure() && hashFiles('postboxes.json.tmp') != ''
+        uses: actions/upload-artifact@v7
+        with:
+          name: overpass-response
+          path: postboxes.json.tmp
+          retention-days: 7

CLAUDE.md

@@ -68,7 +68,7 @@ Fully implemented end-to-end.
 - **App entry**: `lib/main.dart` → **if unauthenticated** → `_UnauthGate` → `Intro` (first run) or `LoginScreen`; **if authenticated** → `Home`. `Home` (`lib/home.dart`) is a `NavigationBar` + `IndexedStack` shell: tabs are **Nearby** (index 0), **Claim** (index 1), **Leaderboard** (index 2), **Friends** (index 3), **History** (index 4 — `ClaimHistoryScreen`, map/list `ViewToggle`). The AppBar `PopupMenuButton` has My reports, Admin · Reports (admins only), Settings, How to play. Named routes `/nearby`, `/claim`, `/friends`, `/leaderboard`, `/history`, `/settings` are retained for deep-link use (each wrapped in an auth guard).
 - **Backend**: `functions/src/index.ts` exports `nearbyPostboxes`, `startScoring`, `updateDisplayName`, `onUserCreated`, `newDayScoreboard`, `streakReminder`, `registerFcmToken`, `onFriendAdded`, `userClaimHistory`, `submitReport`, `reviewReport`, `routePostboxes`. Helper modules: `_lookupPostboxes.ts` (ngeohash + Firestore geohash prefix queries), `_getPoints.ts` (monarch → points: EIIR=2, GR/GVR/GVIR/SCOTTISH_CROWN=4, VR=7, EVIIR/CIIIR=9, EVIIIR=12; also `KNOWN_MONARCHS`, `pointsForMonarch`), `_leaderboardUtils.ts` (period key staleness, merge/sort helpers), `_nearbyUtils.ts` (`applyUserClaims` for per-user claim state), `_streakUtils.ts` (`computeNewStreak`), `_dateUtils.ts` (`getTodayLondon`, `previousDay`, `getLondonHourMinute`), `_notifications.ts` (FCM send via the now-exported `sendToUser`, notification eligibility helpers), `_recomputeScores.ts` (retroactive re-scoring after a cypher correction), `_routePlanner.ts` (pure orienteering + corridor filters + beam search + `routeOverlap`/`selectAlternatives` for alternative routes; reused by `routePostboxes` and the `scripts/plan_route.ts` CLI). `reports.ts` holds the two report callables + the pure helpers `buildOsmChange`, `parsePhotos`, `nextQuotaState`. `functions/set_admin.js` is a CLI to grant/revoke the `admin` custom claim. Friends list in `users/{uid}/friends` array; leaderboards updated by Cloud Functions in `leaderboards/{daily|weekly|monthly|lifetime}` documents. `reports/{id}` holds user-submitted data-problem reports (server-write only); `reportQuotas/{uid}` is the per-user daily submit-rate counter (server-only, never client-read). `fcmTokens/{uid}` stores FCM tokens (separate collection — not exposed via world-readable `users/{uid}` rules). `newDayScoreboard` scheduled at midnight London time; resets daily scores, rebuilds weekly/monthly from claims. `streakReminder` (`functions/src/streakReminder.ts`) is a scheduled gameplay notification: it polls every 15 min from 17:00–20:45 Europe/London and fires once per day at a random slot within that window, pushing a "you'll lose your streak" reminder to users whose `lastClaimDate == yesterday` (claimed yesterday but not today), gated on the `streakReminder` notification pref. Dispatch bookkeeping (random target slot + sent marker) lives in the server-only `notificationState/streakReminder` doc.
 - **Postbox data source and storage**: Postbox data is **sourced from OpenStreetMap (OSM)**—e.g. Overpass API (`amenity=post_box`, UK area). **test.json** in the repo is a sample of the OSM/Overpass response: nodes with `type`, `id`, `lat`, `lon`, and `tags` (e.g. `amenity`, `ref`, `royal_cypher`, `post_box:type`, `collection_times`, `postal_code`). This data is **not** queried from OSM at app runtime; it is **ingested and stored in the cloud database** (Firestore). The app and existing Cloud Functions read from Firestore only.
-- **OSM→Firestore import pipeline**: Implemented in `functions/import_postboxes.js` (the single canonical importer — no `scripts/` duplicate). Run from the `functions/` directory: `node import_postboxes.js <overpass-export.json> --project the-postbox-game`. Stores each postbox as `{ geohash (precision 9), geopoint, overpass_id, monarch?, reference?, county? }` in `postbox/{osm_<id>}` with batch writes of 400. **Incremental by default**: subsequent runs only write nodes whose post-validation fields (geohash, lat/lon, monarch, reference, county) have changed since the previous run, tracked via a SHA-256 manifest at `functions/.last_import_manifest.json` (gitignored). Flags: `--dry-run` (scan & report counts without writing), `--prune` (soft-mark `osm_*` docs whose OSM node disappeared via `removedFromOsm: true` + `removedFromOsmAt` — never hard-deletes; re-appearance auto-clears because every normal write delete()s the flag), `--manifest <path>` (override), `--no-manifest` (force full re-import), `--overwrite-corrections` (re-import over `correctedBy` docs). GEOHASH_PRECISION must remain 9 (maximum) so stored hashes match precision-8 prefix queries used by the 30 m claim scan. Postboxes added from accepted "missing postbox" reports live at `postbox/manual_{reportId}` with `source: 'user_report'`, `reportId`, and the same geohash/geopoint schema; an accepted cypher correction also adds `correctedBy`/`correctedAt` (and a future OSM re-import of the now-added node would need dedup against `manual_*` docs — not yet built). New Flutter deps for reporting: `firebase_storage`, `image_picker`, `exif`, `url_launcher`.
+- **OSM→Firestore import pipeline**: Implemented in `functions/import_postboxes.js` (the single canonical importer — no `scripts/` duplicate). Run from the `functions/` directory: `node import_postboxes.js <overpass-export.json> --project the-postbox-game`. Stores each postbox as `{ geohash (precision 9), geopoint, overpass_id, monarch?, reference?, county? }` in `postbox/{osm_<id>}` with batch writes of 400. **Incremental by default**: subsequent runs only write nodes whose post-validation fields (geohash, lat/lon, monarch, reference, county) have changed since the previous run, tracked via a SHA-256 manifest at `functions/.last_import_manifest.json` (gitignored). Flags: `--dry-run` (scan & report counts without writing), `--prune` (soft-mark `osm_*` docs whose OSM node disappeared via `removedFromOsm: true` + `removedFromOsmAt` — never hard-deletes; re-appearance auto-clears because every normal write delete()s the flag), `--manifest <path>` (override), `--no-manifest` (force full re-import), `--overwrite-corrections` (re-import over `correctedBy` docs). `osm_import.sh` (repo root) wraps it: downloads the UK set from Overpass into `postboxes.json`, sanity-checks it, then runs the importer with any extra flags forwarded; it runs **daily in CI** via `osm-import.yml` (see CI below). GEOHASH_PRECISION must remain 9 (maximum) so stored hashes match precision-8 prefix queries used by the 30 m claim scan. Postboxes added from accepted "missing postbox" reports live at `postbox/manual_{reportId}` with `source: 'user_report'`, `reportId`, and the same geohash/geopoint schema; an accepted cypher correction also adds `correctedBy`/`correctedAt` (and a future OSM re-import of the now-added node would need dedup against `manual_*` docs — not yet built). New Flutter deps for reporting: `firebase_storage`, `image_picker`, `exif`, `url_launcher`.
 - **Auth**: `UserRepository` + `AuthenticationBloc`; Google Sign-In + Email/Password + Sign in with Apple (iOS/Android/web); `firebase_options.dart` has Android, iOS, macOS, Web, and Windows configurations (generated by FlutterFire CLI).
 
 ## Critical issues
@@ -98,6 +98,8 @@ Fully implemented end-to-end.
 
 **Both workflows trigger on `pull_request` AND `push` to master** — work lands on master directly here as often as via a PR, and those pushes previously got no CI at all.
 
+`osm-import.yml` is not a check: it runs `osm_import.sh` against **production Firestore** daily at 03:17 UTC (plus `workflow_dispatch` with `dry_run` / `prune` / `full_reimport` inputs). Scheduled runs pass `--prune`. It authenticates keylessly via Workload Identity Federation: the repo **variables** `GCP_WORKLOAD_IDENTITY_PROVIDER` and `GCP_IMPORT_SERVICE_ACCOUNT` (an `osm-importer` SA with `roles/datastore.user`), and the provider's attribute condition only admits `code418/postbox_game` on `refs/heads/master`. The one-time gcloud setup is in the workflow's header comment. The importer's incremental manifest lives in the Actions cache (a fresh key per run, restored by prefix), so losing it costs only one full re-import. The refreshed `postboxes.json` is discarded, never committed back.
+
 ## Security / release
 
 `firebase_options.dart` and test file reference project ID **the-postbox-game** and service account path; ensure no secrets in repo for store release. Use environment/config for CI and store builds.