Repository navigation
perf(api): serve planet search from a committed catalog instead of a 4 s cold download - #49
Merged
Merged
Conversation
Name matching, ranking, browse filters and keyset paging lived inside the API, so the website could not answer the same questions without duplicating them. They now live in a new @exora/planet-catalog workspace package that the API imports. The ADQL-only helpers stay in the API, and the catalog tests move with the code and cover browse paging and limit clamping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The catalog needs a committed file that both the API and the website can read, and full profile JSON repeats every key for about six thousand planets. The shared package now encodes profiles as one array of values per planet under a single list of field paths, stores the source metadata once, keeps missing values as null, and decodes the rows back into profiles. Decoding refuses unknown format versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A snapshot generator has to fetch the catalog with exactly the columns, row validation and normalization the API uses, but that code was private to the repository class. The TAP request, row schema check and profile normalization now live in an exported loadNasaPlanets function, and the full-catalog query and endpoint are exported. The repository calls the same function, so its behavior is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every cold function instance downloaded the full pscomppars table, although the catalog holds about 6,500 rows and changes at most daily. A generator script fetches the catalog through the API's own loader, sorts it by name and writes it in the compact snapshot format, one planet per line, with the generation time and source query. When only the timestamp would change, it keeps the old file, so an unchanged archive produces no diff. It refuses to write a suspiciously small catalog. The first snapshot (6,451 planets, 1.77 MB raw, 510 kB gzip, 408 kB brotli) is committed, and the formatter skips the generated file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each cold function instance downloaded the full pscomppars table before browse could answer, and search and exact-name reads issued live TAP queries until that download finished. The API now bundles the committed snapshot, decodes it on first use and answers search, browse and exact-name reads from it. Live TAP queries remain only as fallbacks: an exact name the snapshot lacks (a newer planet), a search with no snapshot match, or a repository created without a snapshot. The per-instance catalog loader, its waitUntil hook and the tests for its refresh and cooldown behavior are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The API registered background catalog loads with waitUntil from @vercel/functions. With the catalog bundled there is nothing left to keep alive, so the package was unused. Remove it from the API, the workspace catalog and the zod 4.1 advisory override that only its transitive @vercel/cli-config needed. The production audit still reports no known vulnerabilities. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every planet search keystroke, browse page and atlas name lookup was an API round trip, although the whole catalog fits in one cacheable file. The website now emits the committed snapshot as a hashed asset and can load it, at low fetch priority, in a lazy chunk that uses the shared search, browse and lookup code. Once loaded, api-client answers those reads locally. Before then, or if loading fails, it calls the API as before, and a failed load is retried after a minute. Exact names and searches the snapshot lacks still go to the API, so planets confirmed since the snapshot are found. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The local catalog only helps if it is loaded before the reader starts typing, but it must not compete with the first 3D scene. The website starts the download after the page load event, once the browser is idle, unless the connection asks to save data or is 2G. Opening the planet catalog or the command palette starts it immediately. Both paths share the single load in api-client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The service worker already keeps hashed assets for offline use, so it caches the catalog snapshot. But each weekly refresh gets a new hash, and old 1.7 MB copies would pile up. When the worker stores a snapshot, it now deletes older snapshots from the asset cache. A test covers offline reuse and pruning, and the cache fake now returns Request keys like the real Cache API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A committed snapshot goes stale unless someone regenerates it, and NASA adds planets every week. A scheduled workflow (Mondays, and on demand) installs Vite+ the same way as the quality workflow, with actions pinned by SHA, regenerates the snapshot, validates it with the catalog package tests, and opens or force-updates one refresh pull request when the data changed. Only that job can write contents and pull requests. The header notes that the "Allow GitHub Actions to create and approve pull requests" setting must be enabled, and that pull requests opened with GITHUB_TOKEN do not start the Quality workflow on their own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decoding walked and sliced every field path for every planet. For 6,451 rows that took about 20 ms in Node and 55 ms on a 4x CPU-throttled browser, which is a long task. The decoder now resolves each field to its containing object once, so each row only creates its objects and assigns values. The median Node decode drops from 19 ms to 10 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a 4x CPU-throttled run, JSON parsing and decoding the snapshot ran back to back as one 76 ms main-thread task, long enough to delay input that arrives meanwhile. The loader now yields to the event loop between parsing and decoding, so each step runs as its own shorter task. A worker would still have to copy the decoded profiles back to the main thread, so it was not used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Starting the snapshot download at the page load event can compete with the first scene. On a Fast 3G-like network with a 4x CPU-throttled run, serving the 1.77 MB snapshot uncompressed made the scene settle about 0.85 s later than with the download blocked. With brotli, as Vercel serves it, the difference was within noise, but slower links and heavier scenes leave less margin. The idle prefetch now starts from the shared destination shell once its scene has rendered (or failed), still waits for an idle callback, and still skips save-data and 2G connections. Opening search or browse starts the download immediately, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The caching document and README still described a six-hour per-instance catalog download kept alive with waitUntil, which no longer exists. They now describe the committed snapshot and its generator, the weekly refresh pull request and the repository setting it needs, the shared search package, how the API bundles the snapshot, and how the website loads it, falls back to the API and keeps it offline. The local measurements of sizes, cold start, parse, search and scene impact replace the old ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each cold Vercel Function instance downloaded the full NASA Exoplanet Archive table (~6.4k planets) before it could serve a planet browse. That took about 3.9 s on a cold start. Recent fixes (
waitUntil, waiting for the catalog before browsing) only worked around the problem.The catalog is now a committed snapshot that the API and the website share:
packages/planet-catalogholds the snapshot and its search code. The data has the same columns, validation and normalization as before, and fields NASA left out staynull. It's 1.77 MB raw, 408 kB with brotli, one planet per line so diffs are readable.@vercel/functionsare removed.Proof:
vp check,vp run -r test(582 tests across 4 projects),quality:assets,vp run -r build(JS budget and deployment-bundle check),vp pm audit --prod, and the desktop browser suite (56 tests) all pass. A Playwright run against the production builds showed:/apirequests.Follow-up, not in this PR:
/api/planets/populationstill downloads its own full table per instance and could read this snapshot.Not reviewed by Sol: the Codex account is at its usage limit. I reviewed the diff myself instead.
Created with Claude Opus 5.5 in Claude Code.
🤖 Generated with Claude Code