Skip to content

perf(archive): batch target compatibility checks - #2204

Draft
knqiufan wants to merge 2 commits into
oceanbase:masterfrom
knqiufan:codex/batch-archive-compatibility-2169
Draft

knqiufan wants to merge 2 commits into
oceanbase:masterfrom
knqiufan:codex/batch-archive-compatibility-2169

Conversation

@knqiufan

@knqiufan knqiufan commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Which issue or RFC does this PR close?

Closes #2169.

Rationale for this change

Archive validation performs one target lookup per record. Batching reduces compatibility queries for 5,000 Sources, one Scope, and one journal head from 5,002 to 52 while retaining per-record decoding, dependency validation, and digest comparisons.

What changes are included in this PR?

  • Group target lookups by record type into bounded UNION ALL statements: up to 100 branches per statement, 900 pending parameters, and 1 MiB of encoded pending keys/digests.
  • Carry archive ordinals through SQL to preserve database collation and alternate unique-key semantics. Any differing match counts as one conflict; nullable alternate keys retain SQL NULL behavior.
  • Retain keys/digests instead of archive bodies and stream target results one row at a time.
  • Preserve fresh restore-time validation, write-time conflicts and rollback, publication/tag integrity checks, and projection receipts.
  • Emit target progress after every completed batch so counts that skip multiples of 1,000 still produce intermediate updates.
  • Cover classification, query budgets, wide keys, collation, nullable keys, cancellation/retry, large payloads, changes after dry-run, and intermediate progress for absent, empty, and identical targets.
  • Include a reproducible SQLite benchmark and English/Chinese operator documentation.

Are there any user-facing changes?

Archive dry-run, restore validation, and idempotent retry perform fewer database executions. Target-check progress is reported after each completed batch. Public APIs, schemas, archive formats, and conflict classification are unchanged.

Recorded file-backed SQLite measurements from 2026-10-23 show 5,002-record validation decreasing from 16.3445 to 9.1713 seconds for an empty target and from 16.9955 to 10.7552 seconds for an identical target. These are medians of three runs on Windows 11, CPython 3.12.13, and SQLite 3.50.4 with 256-byte Source content and no injected network delay. Environment, baseline, query counts, restore timing, and qualification limits are documented in both locales; these measurements are not a general backend throughput guarantee.

How was this change tested?

  • .venv/Scripts/python.exe -m pytest -n 4 tests/builtin/portability/test_bundle.py tests/test_archive_cli.py tests/e2e/test_portable_archive.py -q: 203 passed, 63 skipped.
  • .venv/Scripts/python.exe -m pytest tests/builtin/portability/test_bundle.py -k intermediate_target_progress -q: 3 passed. Before the progress fix, the empty and identical target cases failed; the absent-table control passed.
  • Ruff check and format for the four changed Python files: passed.
  • Native Windows ty check for the four changed Python files: passed.
  • Repository-wide ty check --python-platform linux --exclude docs/my --exclude evaluation/web: passed. The exclusions are existing local, untracked directories.
  • UV_NO_SYNC=true SKIP=ty-check uv run --locked prek run -a: passed; type checks were run separately as above.
  • uv lock --locked and git diff --check: passed.
  • Local review using the Open Code Review delegation rules: 6/6 changed files reviewed, no files skipped, no remaining findings. Tests and Markdown were included manually because the CLI's default filter excludes them.

SQLite was executed locally. Native SeekDB is unavailable in this Windows environment, and no OceanBase test URL was configured. The real-backend cases remain unqualified for the changed query path; parameterized tests do not establish acceptance. Keep this PR in draft until actual SeekDB/OceanBase runs qualify it.

AI usage statement

OpenAI Codex assisted with implementation, regression tests, documentation, and local review using the Open Code Review plugin's delegation rules. The original optimization and benchmark also used the plugin's configured DeepSeek Flash provider for automated review.

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One non-blocking progress-reporting regression in the batching change.

Comment thread src/powercontext/builtin/portability/bundle.py Outdated
@AlexStocks AlexStocks added the 1.4.1 v1.4.1 release tracking label Oct 24, 2026
@hidb4ai hidb4ai added 1.5.0 v1.5.0 release tracking and removed 1.4.1 v1.4.1 release tracking labels Oct 25, 2026
@knqiufan
knqiufan force-pushed the codex/batch-archive-compatibility-2169 branch from 9c2cd4a to 4b5c8a7 Compare October 25, 2026 06:00

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.5.0 v1.5.0 release tracking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(archive): batch target compatibility checks to reduce per-record SQL queries

3 participants