ci: shard the OceanBase Atomic Memory clean migration job #2233

Feature description

Shard the E2E harness / Atomic Memory clean migration (OceanBase) job so its cases run on several runners at once, each against its own OceanBase container, without changing what each case verifies.

Problem and proposed solution

The problem. This job is the slowest check on a typical PR, and any src/** change selects it through full in .github/ci-paths.toml.

• Across the last 19 successful E2E harness runs it took 22.7–38.8 min, median 34.1 min. That includes master (22.7, 30.7, 32.3, 33.6 min).
• On fix(context): bound a preparation's query embedding waits by one budget #2231 the next slowest jobs were Acceptance (oceanbase) at 17.9 min and Atomic Memory clean migration (SeekDB) at 11.3 min; everything else was 8 min or less. So this one job sets the time from a push to a complete CI result, about twice the next one.
• The workflow uses cancel-in-progress for pull requests, so every new push restarts the full half hour.

Where the time goes. These are the steps of the master run 40160706229:

scripts/run_atomic_memory_clean_migration.py runs each case in its own pytest process, one after another. The OceanBase fixture in tests/e2e/conftest.py creates a fresh pc_atomic_migrate_* database for every case, and most cases seed a legacy schema and run the full migration. Every clean-migration case takes 25–42 s; test_atomic_memory_migration_evidence.py alone takes 15.9 min.

That per-case isolation is deliberate (the script notes that OceanBase cases share one tenant), and this proposal keeps it.

Proposed solution.

1. Add --shard INDEX/COUNT to scripts/run_atomic_memory_clean_migration.py. The script keeps collecting every node, so its existing check that each suite collects a nonempty, unique set of nodes still applies. It then runs only the nodes whose position modulo COUNT equals INDEX.
2. Turn the job into a matrix over the shards, each with its own OceanBase container and its own report directory/artifact name. The short steps and the long-query step could run in one shard, or get a shard of their own.
3. Add a small aggregate job that keeps the current check name and passes only when every shard passed without skips. Branch protection rules and readers then still have one result to look at.

With 3–4 shards, I expect the job to take about 10–12 min instead of about 35 min. That is an estimate from the timings above and has not been measured yet; I would measure it on the PR. After this change Acceptance (oceanbase) (≈18 min) would become the slowest check.

The SeekDB clean-migration job (≈11 min) uses the same script and could take the same flag later if that helps.

Cost. This uses more runners at once and roughly 1–2 extra runner-minutes per shard for container start and environment setup. The PR queues are already busy at times, so whether 3 or 4 shards is worth it depends on the repository's runner capacity. I'd like a maintainer's view on that before choosing the number.

Alternatives considered

• Share one migrated database among the read-only cases. Several evidence cases only read the migrated data and could reuse one migration. Others merge, undo or restore, so each would need checking, and it weakens the guarantee that every case starts from a clean migration. Sharding gets most of the time back without that risk.
• Select the job more narrowly. For example, skip it for src/** changes outside persistence and migration code. That saves the most, but ci: scope PR validation to changed areas #2072 deliberately made src/** run everything, so it is a policy decision for maintainers rather than part of this change.
• Run cases concurrently against one container. The script deliberately keeps OceanBase schema creation and cleanup apart, and concurrent DDL on one tenant risks new flakiness. Separate containers avoid that.

Additional context

• The job was added in feat(memory): implement independent atomic memory #1857 and extended in fix(search): avoid OceanBase long-query expression overflow #2024, it was scoped by ci: scope PR validation to changed areas #2072, and fix: restore isolated installs and stabilize migration CI #2118 stabilized migration CI.
• cc @frf12 @hidb4ai: does sharding fit how you want this job to work, and is there a runner budget I should stay within?

Are you willing to contribute to this feature?

• Yes, I am willing to contribute code, docs, or design feedback.