Skip to content

ci: shard OceanBase clean migration checks - #2243

Open
zamaoxiaoji wants to merge 5 commits into
oceanbase:masterfrom
zamaoxiaoji:ci/2233-shard-oceanbase-migrations
Open

ci: shard OceanBase clean migration checks#2243
zamaoxiaoji wants to merge 5 commits into
oceanbase:masterfrom
zamaoxiaoji:ci/2233-shard-oceanbase-migrations

Conversation

@zamaoxiaoji

@zamaoxiaoji zamaoxiaoji commented Oct 26, 2026 •
edited
Loading

Copy link
Copy Markdown

Which issue or RFC does this PR close?

Closes #2233.

Rationale for this change

OceanBase clean migration acceptance sets the E2E harness execution critical path.
Three isolated runners divide its existing cases while preserving per-case process
and database isolation, following the maintainer discussion.

What changes are included in this PR?

  • Add zero-based --shard INDEX/COUNT. Validate the full collection before
    selecting nodes, and allow legitimate empty shards.
  • Record full collections, successfully executed nodes, shard identities, and
    completion status. Collection-only and interrupted runs cannot claim completion.
  • Split all five OceanBase acceptance groups, including managed startup, across
    three runners, each with its
    own container and artifact, and preserve the existing aggregate check name.
  • Require every shard to succeed and verify complete, exactly-once execution in
    each group. Reject missing, duplicate, inconsistent, or incomplete reports.
  • Add CLI behavior tests, isolate their generated pytest suites from inherited
    repository options, update CI path selection, and document reproduction and
    artifact validation.

Are there any user-facing changes?

This affects contributor CI and adds an optional internal runner argument. Public
SDK/HTTP APIs and persisted formats are unchanged. Omitting --shard preserves
full-suite execution, including the existing SeekDB command.

How was this change tested?

Current head 2d5f202e includes upstream master at e273457d plus the
observability documentation correction and the upgrade-gate fixture isolation.
The previous E2E harness run
passed, including OceanBase/SQLite acceptance, Recall effort, and the three migration
shards plus aggregate coverage verification. License Check and Kimi Code also passed
on the current head; the remaining current-head workflows are still running.

The current Main workflow
has 11 failed check entries, which collapse into two upstream baseline failures:

  • quality, the upgrade preparation job, four Python unit jobs, and the downstream
    Upgrade release gate fail because make upgrade-policy-check reports
    Persistence changed (file set or content; CRLF/LF normalized): update the compatibility review. The policy records 9b67d639..., while the current
    upstream persistence inputs produce a different digest.
  • The four Python E2E jobs each report four incompatible_schema failures because
    the actual persistence schema does not match the recorded migration fingerprint.

The official master Main run
reproduces the same quality failure,
unit baseline failures,
and E2E schema failures.
This PR does not change the persistence inputs, migration fingerprints, or policy
file; make upgrade-policy-check passes locally on this branch. The upstream
compatibility review and migration fingerprints must be synchronized before Main
can be green. These failures are not reported as successful Main validation.

  • In the earlier successful run, all Python 3.11, 3.12, 3.13, and 3.14 unit jobs
    passed. The Python 3.14 unit log reports 6,298 passed and 248 skipped.

  • All three real OceanBase migration shards passed. Their downloaded artifacts
    passed independent verification: 2 Scope search, 1 background listener,
    47 clean migration, 3 managed-startup, and 8 long-query cases each passed exactly
    once without skips. The GitHub aggregate coverage job also passed.

    python3 scripts/check_atomic_memory_migration_shards.py \
  --reports .powercontext/issue-2233/ci/artifacts-40300916664 --shards 3

  • make install, make check, and actionlint .github/workflows/e2e-harness.yml
    passed. This includes lint, formatting, type checks, lock consistency,
    integration guidance, immutable Action references, and merge-marker checks.

  • The focused command below passed with 95 tests, including rejection of a missing
    managed-startup report. The partition regression explicitly sets the CI pytest
    option to protect generated-suite isolation.

    PYTEST_ADDOPTS=--require-dsh-config-runtime uv run python -m pytest -q \
  tests/test_atomic_memory_migration_shards.py \
  tests/test_check_atomic_memory_migration_shards.py \
  tests/test_ci_changes.py tests/test_workflow_actions.py

  • Local make unit-test PYTEST_WORKERS=4: 6,249 passed, 295 skipped, and 2 failed.
    One failure is the Kimi history diagnostic (KeyError: 'runtime') previously
    reproduced on unchanged upstream 31ecdd61. The other was a 60s npm pack timeout;
    its isolated rerun passed (1 passed in 20.56s). All four real CI unit jobs passed
    as reported above.

Measured CI timing

Current base e273457d

Serial master run 40298253463
versus PR run 40300916664, head 8ac0c797.
The downloaded baseline nodes.txt files and the shard reports contain identical
full node lists in all five groups: 61 cases in total.

Measurement Serial baseline PR
Slowest migration runner execution, including setup 71m 43s 22m 40s
Slowest clean migration test step 48m 07s 15m 28s
Migration gate, first runner start to final check completion 71m 43s 73m 11s
Entire E2E harness elapsed time, including runner waits 135m 56s 118m 01s
Shard Actual passed cases Runner execution Runner queue Clean migration step
0/3 22 17m 32s 23m 49s 10m 02s
1/3 21 22m 40s 23m 36s 15m 16s
2/3 18 22m 29s 22m 35s 15m 28s

The serial runner queued for 21m 45s. The
aggregate
executed for 9s after a 49m 21s runner wait. That wait makes this run's migration
gate elapsed time slightly longer than the serial job, despite the shorter shard
execution. Both E2E workflows completed successfully. Their entire-workflow times
include runner waits and other acceptance jobs, so the elapsed difference cannot
be attributed solely to migration execution savings. The PR time starts at the
approved attempt's run_started_at, excluding the earlier approval wait.

Recorded runs on base 31ecdd61

Compare the serial master baseline
with two measured PR executions:
run 40281181143
and run 40285080688, 915cd58f.
The two measured PR runs use identical runner, verifier, and workflow code.
These measurements and the serial baseline predate integration of upstream
e273457d; the current-base comparison above includes managed-startup acceptance.

Durations use Actions API timestamps. Entire-workflow elapsed time starts at the
approved attempt's run_started_at, excluding the earlier approval wait.

Measurement Serial baseline PR run 40281181143 PR run 40285080688
Slowest migration runner execution, including setup 35m 19s 11m 10s 13m 22s
Slowest clean migration test step 27m 31s 8m 13s 9m 04s
Migration gate, first runner start to final check completion 35m 19s 22m 24s 44m 48s
Entire E2E harness elapsed time, including runner waits 44m 05s 27m 47s 80m 04s

Shard details for run 40285080688 (915cd58f):

Shard Actual passed cases Runner execution Runner queue Clean migration step
0/3 21 13m 22s 22m 25s 8m 53s
1/3 20 12m 27s 22m 46s 8m 52s
2/3 17 12m 07s 20m 09s 9m 04s

The serial runner queued for 6m 35s. In PR run 40281181143 the aggregate executed
for 9s after a 6m 41s runner wait. In run 40285080688 the
aggregate
executed for 7s after a 29m 02s
runner wait. That attempt also spent 35m 15s before its first shard started.
These waits dominate its 80m 04s E2E elapsed time; its slowest migration
runner still finished below Acceptance (oceanbase) at 17m 59s of execution.

Queue durations are started_at - created_at; the migration gate includes
staggered shard starts and the aggregate wait. The two runs show substantial queue
variability, so execution savings and end-to-end elapsed time are reported
separately.

AI usage statement

OpenAI Codex (GPT-6) was used for repository investigation, implementation,
regression tests, local validation, CI investigation, and PR preparation.

AlexStocks
AlexStocks reviewed Oct 26, 2026
View reviewed changes
Comment thread .github/workflows/e2e-harness.yml

atomic-memory-clean-oceanbase:
name: Atomic Memory clean migration (OceanBase)
atomic-memory-clean-oceanbase-shards:

@AlexStocks AlexStocks Oct 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 保留旧 job 名后，两份 observability 文档仍把 test_background_metrics.py 的执行归给该 job

本 PR 把执行体移到 atomic-memory-clean-oceanbase-shards（矩阵 0/1/2），而沿用旧 id 的 atomic-memory-clean-oceanbase 现在只做「下载三个 artifact + 跑 check_atomic_memory_migration_shards.py」。但 docs/en/development/observability.md:136 与 docs/zh/development/observability.md:138 仍写「E2E harness workflow 的 atomic-memory-clean-oceanbase job 通过 clean migration acceptance runner 执行 tests/e2e/test_background_metrics.py，不允许测试 skip」。

影响：按 job 名排查 background listener 失败时会落到聚合 job，那里没有 RUN <node> 行、也没有 background-metrics/background.log；这些证据现在位于 atomic-memory-clean-oceanbase-<sha>-shard-0 artifact 的 background-metrics/ 下（该组只有 1 个 node，因此只有 shard 0 有 background.log）。文档描述的「执行者」与本 head 的实现不一致。

证据：git grep -n "atomic-memory-clean-oceanbase" -- docs/ 只命中上述两行；本 head 的聚合 job 121522025190 日志（Check out → Download shard reports → Verify complete execution coverage）只有 5 行 ... cases passed exactly once without skips. 汇总，没有任何 RUN/pytest 输出。仓库内不存在其它引用旧 job 名的脚本或 workflow（.github/ci-paths.toml、master.yml 的 seekdb 路径都用 --shard 默认值，未受影响）。

建议：同一 PR 内把两份文档改成「...-shards（三个 runner）执行、atomic-memory-clean-oceanbase 负责聚合校验」，并说明 background.log 在 shard 0 的 artifact 内。聚合 job 本身已 success（scope-search 2 / background-metrics 1 / clean-migration 47 / startup-snapshot 3 / long-query 8），所以这条不影响门禁，只影响按文档排障。

@zamaoxiaoji

zamaoxiaoji commented Oct 26, 2026 via email

Copy link
Copy Markdown
Author

hireview001
hireview001 reviewed Oct 26, 2026
View reviewed changes

@hireview001 hireview001 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@AlexStocks AlexStocks added the 1.5.0 v1.5.0 release tracking label Oct 27, 2026
hireview001
hireview001 reviewed Oct 27, 2026
View reviewed changes

@hireview001 hireview001 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hidb4ai
hidb4ai approved these changes Oct 28, 2026
View reviewed changes

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2d5f202. No blocking issues found in the sharding changes or fixture isolation. Full collection validation, per-case isolation, and complete aggregate coverage are preserved. No public API or database schema changes. The remaining Main failures also occur on master 927bb6e; the current OceanBase shards and aggregate check passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AlexStocks AlexStocks AlexStocks left review comments
@hireview001 hireview001 hireview001 left review comments
@hidb4ai hidb4ai hidb4ai approved these changes
@PsiACE PsiACE Awaiting requested review from PsiACE PsiACE is a code owner
@frostming frostming Awaiting requested review from frostming frostming is a code owner
@zhanghuidinah zhanghuidinah Awaiting requested review from zhanghuidinah zhanghuidinah is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

1.5.0 v1.5.0 release tracking

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

ci: shard the OceanBase Atomic Memory clean migration job

4 participants

@zamaoxiaoji @AlexStocks @hireview001 @hidb4ai