Skip to content

fix(search): keep one locale index and read search text from build output (#393) - #394

Merged
zhuangjianguo merged 7 commits into
mainfrom
claude/project-thread-u7eoxi
Oct 29, 2026
Merged

zhuangjianguo merged 7 commits into
mainfrom
claude/project-thread-u7eoxi

Conversation

@zhuangjianguo

Copy link
Copy Markdown
Collaborator

Requested by Steedos · project thread
Fixes #393
Clause-②: no
Metadata-protocol: none

维护者速读

  • 改了什么:/api/search 每个 isolate 只保留最近一个语言的索引;页面的搜索文本改为构建时生成(apps/docs/.search/,gzip),不再通过 page.data.load() 导入整页模块;新增 CI 步骤 Search fits the Worker's memory,超过 100 MB 或第二轮仍在增长即变红。
  • 为什么改:docs(i18n): translate build/automation/ and build/interface/ into ja (#388, PR 2 of 7) #390 部署后 /api/search 返回 503 并两次回滚。原因是路由把每个语言的索引(各 30–40 MB)和导入过的页面模块都留在内存里,8 个语言后堆达到 283 MB,而 Worker 只有 128 MB。
  • 风险与回滚:搜索结果与 main 逐字节一致(21,992 个查询,固定页面顺序后比较)。代价是切换语言的第一次搜索要重建该语言索引(本地 Worker 预览约 0.6–1.9 秒)。Worker 上传体积增加约 1.2 MB,仍在预算内。回滚即 revert 本 PR。
  • 先看:apps/docs/app/api/search/route.ts 的 texts()、indexes() 和 KEEP,然后看 apps/docs/scripts/gen-search-data.mjs。
  • 两分钟验证:看本 PR 的 CI step summary 中 Search fits the Worker's memory 表格,8 个语言问两轮,堆稳定在约 53 MB。
  • 需要你做的:无,席位落地。另有一个发现:日文全量覆盖会让 Worker 上传超过 Cloudflare 64 MiB 上限(与搜索无关),见下文。

What changes

File Change
apps/docs/scripts/gen-search-data.mjs (new) Before next build, computes each page's structuredData with fumadocs-mdx's own remark chain (applyMdxPreset from the compiled source.config plus remarkInclude, with rehype skipped because remarkStructure runs in the remark phase). It writes one module per locale to apps/docs/.search/ as gzipped JSON in base64. The directory carries its own .gitignore.
apps/docs/app/api/search/route.ts Reads the locale's text from .search/ (plus the default language's for fallback pages) instead of page.data.load(). Keeps only the most recently used locale's index (KEEP = 1). A page missing from .search/ throws, except under next dev, where it falls back to load() for pages added since dev started.
apps/docs/package.json build, type-check and dev run fumadocs-mdx and the generator first.
.github/scripts/check-search-heap.mjs (new) Loads the built route, asks every locale twice, reads heapUsed after a forced GC, and fails on threw, status, over-budget (100 MB) or still-growing (round two more than 8 MB above round one's peak). --self-test has 6 cases, and every rule fires.
.github/workflows/ci.yml New step Search fits the Worker's memory after Search answers every locale.
tools/ci-scripts/run-self-tests.mjs check-search-heap.mjs added to SELF_TESTED.

No content files. The branch carries one -s ours merge of the landed #390 branch head so the push fast-forwards; the diff against main is only these 6 files.

Tests

Heap after a forced GC, measured on the built route (node --expose-gc .github/scripts/check-search-heap.mjs):

after main 09784e7 this PR this PR + full ja (98 ja pages)
route loaded 17.8 MB 18.1 MB 18.4 MB
en 58.3 MB 47.5 MB 47.8 MB
zh-Hans 97.9 MB 51.8 MB 52.0 MB
zh-Hant 137.1 MB 53.2 MB 53.6 MB
ja 171.6 MB 52.7 MB 54.5 MB
de 200.9 MB 51.8 MB 52.6 MB
es 229.7 MB 52.0 MB 52.9 MB
fr 254.3 MB 51.2 MB 51.9 MB
ko 283.4 MB 52.5 MB 53.4 MB
second round, highest ~283.7 MB 54.1 MB 55.1 MB

The full-ja column used #390's 11 pages, the 54 translated pages held for PRs 3–7, and English copies for the 2 not yet translated. They were removed after measuring.

  • Same text. On the same build, the structuredData the old route got from page.data.load() was compared with what the generator writes: 98 pages × en, zh-Hans, zh-Hant and ja, 0 differences.
  • Same answers. Every locale × (every page title, every ##/### heading, and 9 probes) = 21,992 requests, with each response hashed. On the first pass, 2,070 responses differed, all with the same set of hits in a different order further down the list. The cause is upstream of this PR: fumadocs-mdx writes .source/server.ts in a different file order on each run (three runs gave three hashes), which changes insertion order and so the order of tied scores between any two builds of main. With the page order fixed to the same sort in both the old and new route (a test-only patch), 21,992 of 21,992 responses were byte-identical.
  • check-search-locales.mjs: all 8 locales answer 200, and every line except ja matches the earlier run on main. The ja line differs only because docs(i18n): translate build/automation/ and build/interface/ into ja (#388, PR 2 of 7) #390's pages were reverted. The Docs polish from the 2026-10-06 audit: muted-text contrast, the License table on phones, FAQ formatting, search ranking, glossary order, licence/license spelling #301 ranking holds: own pages are found by title within the first 3, and en has 93 of 98 first.
  • Worker runtime. opennextjs-cloudflare preview (workerd) answered /api/search in all 8 locales twice with 200. smoke-docs.mjs --base against it passed.
  • Worker upload. Built on the same machine: main 55,398.33 KiB; this PR 56,596.10 KiB (+1,198 KiB, budget 61,440). Unzipped JSON would have added about 3.5 MB, so the text is gzipped.
  • pnpm turbo run type-check, pnpm turbo run test (13 self-tests), check-locale-surface.mjs, run-self-tests.mjs and check-search-heap.mjs --self-test all pass.

Acceptance notes


Generated by Claude Code

zhuangjianguo and others added 7 commits October 29, 2026 10:17
…ions, use)

Phase 1 of the other-locale plan. Adds ja translations of the 19
entry-path pages (index, why, start/*, editions/*, use/*) and the ja
sidebar labels for start/ and editions/. Every heading keeps its
English id, fences match the English byte for byte, terms follow the
glossary's ja column, and each page carries a --stamp.

check-translations: ja 0 stale, 79 missing (was 98).

Fixes #386

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
Phase 2 of the other-locale plan, first batch: ja for the build/ top
level (index, agents, ai-builder, ai-skills, extend-existing-systems,
marketplace, packages, reports, templates) and build/data/ (index,
relationships, formulas, validation-rules). Same rules as #386: English
heading ids, fences byte for byte, glossary ja column, UI labels in
English, one line per paragraph, each page stamped.

check-translations: ja 0 stale, 66 missing (was 79).

Refs #388

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
…388, PR 2 of 7)

Adds ja translations for the 11 automation and interface pages:
build/automation/{index,flows,workflows,approvals} and
build/interface/{index,apps,views,forms,dashboards,pages,actions}.
Both sections' meta.ja.json already match the English pages list.

Refs #388

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
…tput (#393)

/api/search kept every locale's Orama index (30-40 MB each) and every
page module page.data.load() imported, so one Worker isolate grew from
58 MB after its first locale to 283 MB after all eight, and #390's ja
pages tipped it past 128 MB into 503s.

- scripts/gen-search-data.mjs computes each page's structuredData before
  next build with fumadocs-mdx's own remark chain and writes one gzipped
  module per locale to apps/docs/.search/; the route reads that instead
  of importing page modules.
- The route keeps the most recently used locale's index only.
- check-search-heap.mjs asks the built route in every locale twice and
  fails above 100 MB or when the second pass keeps growing; it runs in
  ci.yml after check-search-locales and its --self-test is in SELF_TESTED.

Measured on the built route: levels at 50-54 MB across both passes.

Fixes #393

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
The tree is unchanged: the diff against main is the #393 commit only.

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
@objectstack-fleet

Copy link
Copy Markdown
Contributor

席位核对 · ACCEPT · 2026-10-29T12:18Z · session_01QsR2jTaZ2GeyfjANMwstiw · seat repo:docs.objectos.ai#1

  1. Shape. Draft PR against main 09784e7, head 6434c3f. 6 files, +401/−15, no content files. The body says Fixes #393. Claim 6495418401, branch claude/project-thread-u7eoxi. The changes stay within the claimed surface: route.ts, a new scripts/gen-search-data.mjs (build, dev and type-check scripts in apps/docs/package.json), a new check-search-heap.mjs with its SELF_TESTED entry and ci.yml step. The branch history carries merges of already-landed heads, including docs(i18n): translate build/automation/ and build/interface/ into ja (#388, PR 2 of 7) #390's. The tree diff against main is exactly these 6 files, and the queue squashes it, so docs(i18n): translate build/automation/ and build/interface/ into ja (#388, PR 2 of 7) #390's reverted pages do not come back with it.
  2. Design. The route keeps one locale's index, as an LRU with KEEP = 1. Search text is read from .search/<locale>.ts, which gen-search-data.mjs writes before every build with fumadocs-mdx's own remark chain. It used to come from page.data.load(), whose page modules stayed resident. The output is gitignored. A missing page throws outside next dev, so a stale .search/ cannot ship silently.
  3. Seat measurements, at 6434c3f. The seat built the PR, then ran its own harness: the built route.js, routeModule.userland.GET, --expose-gc.
    • Heap. 47.5 MB after en, and level at 51–53 MB through all eight locales. main at 3bbc710 reached 285.9 MB.
    • check-search-heap. Every locale was asked twice. Heap stayed under the 100 MB budget and stopped growing.
    • check-search-locales. All 8 locales answer 200. Each locale's own pages are found by title: en 98/98, zh-Hans 98/98, zh-Hant 98/98, ja 32/32. The nonce finds nothing.
    • Self-tests. Every script passes except check-doc-samples, which is NOT MEASURED locally because its npm prefix is not installed. CI installs it.
    • Check-runs. build success · Node floor success · Ownership & freshness success.
  4. The dev's evidence.
    • structuredData matches load() exactly for 98 pages × 4 locales.
    • 21,992 responses are byte-identical to main once page order is fixed.
    • The workerd preview answers all 8 locales.
    • The Worker upload is 56,596 KiB against 55,398 KiB on main, under the 61,440 KiB budget.
  5. Follow-up. The dev also measured that each translated page adds about 172 KiB to the Worker, so full ja coverage would pass Cloudflare's 64 MiB limit. That predates this PR and is filed as Each translated page adds about 172 KiB to the Worker upload, so full ja coverage (#388) would exceed Cloudflare's 64 MiB Worker limit #395.
  6. Landing. Tier none. It lands through the merge queue under the standing go-ahead, serially. After smoke is green on main, Other locales, phase 2: ja to full coverage (the remaining 79 pages, seven serial PRs) #388 resumes by re-landing docs(i18n): translate build/automation/ and build/interface/ into ja (#388, PR 2 of 7) #390's pages.

@zhuangjianguo
zhuangjianguo marked this pull request as ready for review October 29, 2026 12:18
@zhuangjianguo
zhuangjianguo added this pull request to the merge queue Oct 29, 2026
Merged via the queue into main with commit 9e51646 Oct 29, 2026
4 checks passed
zhuangjianguo added a commit that referenced this pull request Oct 29, 2026
…388, PR 2 of 7)

Reverts #392, which reverted #390 because /api/search ran out of Worker
memory. #394 (#393) fixed that, so the same 11 ja pages come back
unchanged.

Refs #388

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
zhuangjianguo added a commit that referenced this pull request Oct 29, 2026
The tree is unchanged: the diff against main is the re-landed ja pages only.

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant