Skip to content

refactor: in-process catalog sync + concurrent provider opens (MAPCO-11560) - #57

Open
shimoncohen wants to merge 7 commits into
feat/geotiff-heights-migrationfrom
refactor/catalog-sync-inprocess
Open

shimoncohen wants to merge 7 commits into
feat/geotiff-heights-migrationfrom
refactor/catalog-sync-inprocess

Conversation

@shimoncohen

Copy link
Copy Markdown
Contributor

Summary

Stacked on #52. Replaces the CSW catalog-sync worker_threads Worker with an in-process, self-rescheduling setTimeout poll loop (CatalogSyncManager), and opens GeoTIFF providers concurrently instead of sequentially.

  • src/heights/models/catalogSyncManager.ts (new) — owns the CSW client, fetch filter, isSame diff, and scheduling. Injects only CONFIG+LOGGER; receives the CatalogRecords + DEMTerrainCacheManager singletons via start(...) to avoid a circular import with containerConfig. Fetch errors are caught, logged, and the loop reschedules — self-healing where a dead worker used to freeze the cache silently.
  • src/containerConfig.ts — worker removed; manager registered as a singleton, started after DI registration, and stopped in onSignal (graceful shutdown the worker never had).
  • src/heights/models/DEMTerrainCacheManager.ts — initProviders sequential for loop → bounded PromisePool (concurrency = samplingConcurrency), preserving per-record error isolation.
  • src/workerCatalogRecords.ts — deleted.

Why

Worker threads are for CPU-bound work; this job is pure network I/O (CSW fetch + GeotiffHeightProvider.fromUrl). The worker bought nothing, structured-cloned the record set across the thread boundary every cycle, and its exit/error handlers only logged — a dead worker silently froze the cache.

Behavior parity

/points API, response shape, and openapi3.yaml unchanged. Same CSW filter (GEOTIFF / PUBLISHED), same 1..1000 bounds, same isSame-gated update-only-on-change, same non-blocking startup.

Test Plan

Notes / accepted trade-offs

🤖 Generated with Claude Code

https://claude.ai/code/session_01ATQerkHAV3Vur82HVVFack

Comment thread src/heights/models/catalogSyncManager.ts Outdated
Comment thread src/heights/models/catalogSyncManager.ts Outdated
@shimoncohen shimoncohen self-assigned this Sep 24, 2026
shimoncohen added a commit that referenced this pull request Sep 24, 2026
… order

Addresses PR #57 review:
- Swap so initProviders runs before catalogRecords.setValue, so a reader never
  sees the new catalog paired with stale providers (a removed record would resolve
  to an undefined catalog entry mid-refresh).
- Sort records by id before the isSame diff so CSW returning the same set in a
  different order no longer triggers a needless provider rebuild.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATQerkHAV3Vur82HVVFack
shimoncohen added a commit that referenced this pull request Sep 24, 2026
… order

Addresses PR #57 review:
- Swap so initProviders runs before catalogRecords.setValue, so a reader never
  sees the new catalog paired with stale providers (a removed record would resolve
  to an undefined catalog entry mid-refresh).
- Sort records by id before the isSame diff so CSW returning the same set in a
  different order no longer triggers a needless provider rebuild.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATQerkHAV3Vur82HVVFack
@shimoncohen
shimoncohen force-pushed the refactor/catalog-sync-inprocess branch from 02cd4a2 to 97ed0de Compare September 24, 2026 08:57
shimoncohen and others added 7 commits September 28, 2026 14:14
require-array-sort-compare flags a bare .sort() on the provider-key
assertion added for the concurrent initProviders test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATQerkHAV3Vur82HVVFack
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATQerkHAV3Vur82HVVFack
… order

Addresses PR #57 review:
- Swap so initProviders runs before catalogRecords.setValue, so a reader never
  sees the new catalog paired with stale providers (a removed record would resolve
  to an undefined catalog entry mid-refresh).
- Sort records by id before the isSame diff so CSW returning the same set in a
  different order no longer triggers a needless provider rebuild.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATQerkHAV3Vur82HVVFack
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@shimoncohen
shimoncohen force-pushed the feat/geotiff-heights-migration branch from 9ad9fbf to 620d66c Compare September 28, 2026 11:16
@shimoncohen
shimoncohen force-pushed the refactor/catalog-sync-inprocess branch from 5f16baa to c8f70a3 Compare September 28, 2026 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants