refactor: in-process catalog sync + concurrent provider opens (MAPCO-11560) - #57
Open
shimoncohen wants to merge 7 commits into
Open
shimoncohen wants to merge 7 commits into
shimoncohen wants to merge 7 commits into
Conversation
lirantul123
reviewed
Sep 23, 2026
shimoncohen
added a commit
that referenced
this pull request
Sep 24, 2026
… order Addresses PR #57 review: - Swap so initProviders runs before catalogRecords.setValue, so a reader never sees the new catalog paired with stale providers (a removed record would resolve to an undefined catalog entry mid-refresh). - Sort records by id before the isSame diff so CSW returning the same set in a different order no longer triggers a needless provider rebuild. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ATQerkHAV3Vur82HVVFack
shimoncohen
added a commit
that referenced
this pull request
Sep 24, 2026
… order Addresses PR #57 review: - Swap so initProviders runs before catalogRecords.setValue, so a reader never sees the new catalog paired with stale providers (a removed record would resolve to an undefined catalog entry mid-refresh). - Sort records by id before the isSame diff so CSW returning the same set in a different order no longer triggers a needless provider rebuild. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ATQerkHAV3Vur82HVVFack
shimoncohen
force-pushed
the
refactor/catalog-sync-inprocess
branch
from
September 24, 2026 08:57
02cd4a2 to
97ed0de
Compare
lirantul123
approved these changes
Sep 28, 2026
require-array-sort-compare flags a bare .sort() on the provider-key assertion added for the concurrent initProviders test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ATQerkHAV3Vur82HVVFack
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ATQerkHAV3Vur82HVVFack
… order Addresses PR #57 review: - Swap so initProviders runs before catalogRecords.setValue, so a reader never sees the new catalog paired with stale providers (a removed record would resolve to an undefined catalog entry mid-refresh). - Sort records by id before the isSame diff so CSW returning the same set in a different order no longer triggers a needless provider rebuild. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ATQerkHAV3Vur82HVVFack
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
shimoncohen
force-pushed
the
feat/geotiff-heights-migration
branch
from
September 28, 2026 11:16
9ad9fbf to
620d66c
Compare
shimoncohen
force-pushed
the
refactor/catalog-sync-inprocess
branch
from
September 28, 2026 11:16
5f16baa to
c8f70a3
Compare
lirantul123
approved these changes
Sep 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #52. Replaces the CSW catalog-sync
worker_threadsWorker with an in-process, self-reschedulingsetTimeoutpoll loop (CatalogSyncManager), and opens GeoTIFF providers concurrently instead of sequentially.src/heights/models/catalogSyncManager.ts(new) — owns the CSW client, fetch filter,isSamediff, and scheduling. Injects only CONFIG+LOGGER; receives theCatalogRecords+DEMTerrainCacheManagersingletons viastart(...)to avoid a circular import withcontainerConfig. Fetch errors are caught, logged, and the loop reschedules — self-healing where a dead worker used to freeze the cache silently.src/containerConfig.ts— worker removed; manager registered as a singleton, started after DI registration, and stopped inonSignal(graceful shutdown the worker never had).src/heights/models/DEMTerrainCacheManager.ts—initProviderssequentialforloop → boundedPromisePool(concurrency =samplingConcurrency), preserving per-record error isolation.src/workerCatalogRecords.ts— deleted.Why
Worker threads are for CPU-bound work; this job is pure network I/O (CSW fetch +
GeotiffHeightProvider.fromUrl). The worker bought nothing, structured-cloned the record set across the thread boundary every cycle, and itsexit/errorhandlers only logged — a dead worker silently froze the cache.Behavior parity
/pointsAPI, response shape, andopenapi3.yamlunchanged. Same CSW filter (GEOTIFF/PUBLISHED), same1..1000bounds, sameisSame-gated update-only-on-change, same non-blocking startup.Test Plan
npm run test:unit— 25/25 (addscatalogSyncManager+ concurrent-initProviderscoverage)npm run test:integration— 8/8 (manager stubbed in the DI override so the real poll loop never runs in tests — no leaked timers, clean Jest exit)npm run buildcleanNotes / accepted trade-offs
feat/geotiff-heights-migration(feat: replace QMesh terrain engine with direct GeoTIFF/COG sampling (MAPCO-11560) #52), notmaster. Rebase onto master before merge if feat: replace QMesh terrain engine with direct GeoTIFF/COG sampling (MAPCO-11560) #52 lands first.DEMTerrainCacheManager.tsline 53, integration spec lines 127/158, other feat: replace QMesh terrain engine with direct GeoTIFF/COG sampling (MAPCO-11560) #52 test files) are untouched — this branch adds none. CI (wearerequired/lint-action) already reflects the base state.samplingConcurrencyis reused to bound provider-open concurrency (records are few; avoids a new config key across default/env/helm).onSignalresolves the manager from the global tsyringe container, matching the prior worker-handler pattern (useChildis never set true).PromisePool.withConcurrencythrows on a NaN/0samplingConcurrency; a bad value already breaks the query path on feat: replace QMesh terrain engine with direct GeoTIFF/COG sampling (MAPCO-11560) #52, so config validation is the right home for that — out of scope here.🤖 Generated with Claude Code
https://claude.ai/code/session_01ATQerkHAV3Vur82HVVFack