fix(topic-memory): add durable block inspection and recovery

[knqiufan]

Which issue or RFC does this PR close?

Closes #1885.

Design RFC: #2018. This implementation remains draft pending agreement on the durable recovery model.

Rationale for this change

An exhausted Topic frontier prevents later Sources in the same Scope from progressing, but previously had no supported inspection or recovery API. Administrators need durable diagnostics and an explicit bounded retry that preserves the processing evidence. Worker timeout diagnostics must also remain consistent between live Supervisor state and retained block records.

What changes are included in this PR?

• Persist content-free attempt and block metadata: exact identities, each attempt's selected Source range, retained target, reserved usage, failure categories, first blocked observation and accepted retry receipt. Shrinking a later window preserves its exact range independently of the budget's cumulative maximum. Legacy budgets without attempt snapshots retain their recorded cumulative coverage; unavailable historical facts are not reconstructed.
• Add HTTP, typed Python Client SDK and topic-memory status|block|retry CLI operations. Inspection requires scope.read; retry requires scope.admin and the observed block_id.
• Atomically grant one fresh bounded allowance and enqueue processing while retaining Sources, Cursor and target. Duplicate IDs replay the original acceptance after progress or disabled processing; stale unretried IDs conflict, and cross-Scope or unknown IDs are not found.
• Keep an active final attempt pending. Share the Supervisor's failure classification between live and retained diagnostics, preserving worker_timeout and model_timeout as distinct controlled categories.
• Regenerate API models, desktop contract types, host operation metadata and the DSH bundle. Document recovery and backup requirements in English and Chinese.
• Provision and verify sufficient OceanBase datafile capacity for acceptance tests, with test cleanup.

Are there any user-facing changes?

Administrators can inspect and recover blocked processing without editing the database. Retry acceptance schedules work; it does not guarantee generation success. Each accepted retry retains the three-attempt, 512-request and 64,000,000-token ceilings. Ordinary flushes, status reads, restarts and model changes grant no extra allowance.

The management APIs and audit table are additive. Existing block identities and diagnostics remain readable after recovery. Historical processing_failed records are retained without guessing whether they represented timeouts.

How was this change tested?

All checks on commit 4cdf119bd434321ebf2c18e98834f0190557bbd7 completed: 38 passed, with the conditional installed-package job skipped by its workflow gate. No failed or pending checks remain.

• Main CI: Python 3.11–3.14 unit/end-to-end suites, quality and contract checks, Windows portability, native SeekDB, website and integration packages passed.
• E2E harness: SQLite/OceanBase acceptance, clean migration and recall regressions passed.
• Desktop validation: Windows build, native SQLite/CLI checks and installed-package smoke tests passed. ZCode/Kimi Code acceptance, native personal services, Skill guidance and license workflows also completed successfully.

Local validation:

• uv sync --locked.
• uv run --no-sync python -X utf8 -m pytest -q tests/builtin/runtime/test_topic_memory_recovery.py tests/builtin/runtime/test_topic_memory_processing.py tests/builtin/runtime/test_topic_memory_scope.py tests/builtin/runtime/test_topic_memory_terminal_frontier.py tests/test_topic_memory_recovery_cli.py tests/e2e/test_topic_memory_recovery_access.py tests/e2e/test_topic_memory_terminal_frontier.py tests/test_api_contract.py tests/test_js_operations.py: 132 passed.
• Updated recovery tests: 12 passed, including a deliberate timeout before Topic attempt admission. Parallel validation with pytest -n 2 tests/builtin/runtime/test_topic_memory_recovery.py -k 'timeout_diagnostics or attempt_range or generic_failed_workers': 5 passed. The timeout regression waits for the persisted terminal block and still verifies three admitted attempts, controlled categories and reopening.
• The selected-range regression failed before the fix: actual windows were 2, 1, 1, but the final block reported end 2. It now verifies end 1, cumulative budget end 2, retained target 3, exact block readback and file-backed SQLite reopening.
• Python and JavaScript API generation checks and node desktop/scripts/generate.mjs --check passed. The two desktop manifests match the combined contract hash 5e0de479c92d57c2286fe04f3ee2c8e3321798f35412fc237920c105df48ed2d.
• uv run --no-sync prek run -a --skip ty-check: all formatting and lint hooks passed.
• uv run --no-sync ty check --python-platform linux --force-exclude .github desktop e2e evaluation examples integrations scripts src tests: passed for project directories, excluding local untracked research files. Changed persistence and recovery tests passed Linux- and Windows-targeted type checks.

The range regression runs the actual ScopeProcessor and WindowSelector with file-backed SQLite. Timeout regressions use a test launcher with the actual Supervisor deadline. Model responses are fixtures; no live external-model or local OceanBase qualification is claimed.

AI usage statement

OpenAI Codex assisted implementation, regression testing and review. Open Code Review delegation supplied file selection and review rules: all 24 selected files were reviewed, with 15 excluded documentation, test and generated-map files checked separately. No additional blocking findings remained. The configured OCR model invocation timed out, so that incomplete invocation is not claimed as a completed independent model review.

[hidb4ai]

One reproducible issue remains in the persisted recovery diagnostics.

[hidb4ai]

[P2] Preserve the Worker timeout category in retained diagnostics

When asyncio.timeout(binding.worker_timeout_seconds) expires, this branch passes TimeoutError to processing_error_code, which maps it to processing_failed. The existing _record_failure path recognizes the same exception as worker_timeout, so the live Supervisor state and the retained block disagree. I reproduced three timed-out Topic attempts with SQLite and the actual Supervisor timeout control: the Supervisor reported timeouts=3 and last_error=worker_timeout, while processing_status.block.last_error_code was processing_failed. The launcher used a deliberately hanging probe; this was not a native child-process timeout test. Once the process restarts, the retained status/block APIs lose the specific timeout diagnosis. Please persist a controlled Worker-timeout category and verify that it survives block readback after restart.

[knqiufan]

Thanks for the clear report. Fixed in d4fe051: live and retained diagnostics now share the same classification, keeping worker_timeout and model_timeout distinct. Added SQLite regressions for status/block readback after reopening the database, using the real Supervisor deadline with a test launcher.

[hidb4ai]

One correctness issue remains in the retained attempt diagnostics. This independently confirms the range issue noted in the existing PR comment.

[hidb4ai]

[P2] Persist the actual range selected for this attempt

current["source_through"] is the frontier budget's cumulative maximum, so it can exceed the end selected for this attempt when a later window becomes smaller, for example after reducing the model context limit. I independently reproduced three attempts with ends 2, 1, 1 using real async SQLite: processing_status and exact block readback associate the third attempt ID with source_through=2, although that attempt covered only (0, 1]. This makes the durable diagnostics attribute an unattempted Source to the final failure and violates the documented attempted-range contract. Please persist this attempt's selected end in its audit row, keeping the cumulative budget range and retained target separate. This reproduction did not use a live model or OceanBase.

[knqiufan]

Thanks for the clear reproduction. Fixed in 4e7b77f: each audit row now records the selected range, independently of the cumulative budget and target. A SQLite regression covers windows 2, 1, 1 and verifies status/block readback after reopening.

[hidb4ai]

please update & resolve conflicts

[AlexStocks]

[P2] 与当前 master 的冲突解决指引（DIRTY 状态）

本 PR 相对 base 666b043a 的桌面契约清单是一致的：head openapi/powercontext.yaml（LF 归一化）sha256 = 5e0de479...，与签入的两处清单一致，已独立复算。master 自 merge-base 以来的变化里与本 PR 相交的只有三处：operations.json/operations.ts 的 SHA 行（#1988 已把 master 侧刷成 c4571e4b）和 desktop/scripts/generate.mjs——后者只是把漂移报错文案改成带修复命令，wanted 操作集没有变化。

因此：

• openapi/powercontext.yaml 是纯增量（+158），与 master 合并后摘要仍为 5e0de479...；
• 两处清单的 SHA 冲突应保留本 PR 侧的 5e0de479...（master 侧的 c4571e4b 是旧契约），无需重新生成；
• api.d.ts 与 _generated/*.py master 侧未动，会自动合并；
• 合并后建议跑一次 node desktop/scripts/generate.mjs --check 终验。

其余核对结论：topic_memory_recovery.py 的锁序、租约甄别（ MySQL repeatable-read 下加锁重读）、retried_at 幂等回放、stale 冲突拒绝均成立；e2e-harness 的 OB datafile 配置带 30 次自校验门控，配错会显式失败，无静默风险。本机 71 个相关测试通过（SQLite 路径）。

[knqiufan]

感谢详细核对。已 rebase 到最新 master，保留合并后契约的 5e0de479… 摘要；两处清单一致，node desktop/scripts/generate.mjs --check 已通过。

[hidb4ai]

Before merging, I would like a short RFC/design review of the new pc_topic_memory_work_attempts table. We should be conservative about adding persistent tables and explicitly assess reuse. The latest correctness fixes look addressed; this is a separate storage-design decision.

The table currently combines three responsibilities: per-attempt execution history, blocked-frontier diagnostics, and durable retry acceptance receipts. Persisting the receipt is necessary to prevent duplicate requests from granting another allowance after restart, but that alone does not establish the need for a Topic-specific history of every attempt, including successful attempts.

Please add a concise design supplement covering:

• Existing storage and Artifact reuse: Explain why the existing processing/budget records or Artifact storage cannot support the required lifecycle. Current pending/intents and leases are mutable operational state, and the budget is deleted on retry or success, so none is a drop-in history store. For Artifact reuse, assess atomicity with the budget reset and processing intent, access control, and cleanup behavior that preserves retry idempotency.
• Minimum retained evidence: Does the recovery workflow require only a durable block snapshot and retry receipt, or a complete execution history? Justify retaining successful attempts, and define retention, cleanup, and expected storage growth. Cleanup must not allow an old retry request to grant fresh allowance.
• Ownership and reuse: Execution identity, timestamps and controlled failure categories may belong to the shared processing layer; Source windows, frontier allowances and Topic token/request ceilings are domain-specific. Explain why this table belongs to Topic Memory, or validate a shared design against a second concrete processing workflow. Renaming the table alone would not establish reusable semantics, and a speculative generic framework is not required.

The blocked-processing bug is worth fixing. My earlier correctness assessment should not be taken as approval of the new table's architectural boundary. I would hold the merge until we agree on the smallest durable model and its reuse/retention rationale; this is a design gate, not an additional reproduced correctness defect.

[knqiufan]

Thanks for clarifying the design gate. This implementation is now draft; the focused storage-design RFC is #2018.

The RFC proposes reusing existing budget rows for the current attempt and retaining only terminal-block snapshots with retry receipts, without successful-attempt history. It assesses Artifact and operational-store reuse, atomicity, ownership, indefinite block/receipt retention, legacy evidence, explicit offline migration, and expected growth with isolated SQLite sizing.

I will align this implementation with the accepted design before requesting implementation review again.