Skip to content

ci: consolidate validation and reduce pull request runner demand - #2285

Open
hidb4ai wants to merge 6 commits into
oceanbase:masterfrom
hidb4ai:codex/ci-failure-analysis-20261028
Open

hidb4ai wants to merge 6 commits into
oceanbase:masterfrom
hidb4ai:codex/ci-failure-analysis-20261028

Conversation

@hidb4ai

@hidb4ai hidb4ai commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Which issue or RFC does this PR close?

Related CI incident: Main run 40513436397.

Rationale for this change

Each PR currently starts multiple independent workflows and scope detectors, expands expensive work before basic validation succeeds, and runs eight Python/version combinations for routine changes. A bounded read-only sample on 2026-10-29 at 11:16 UTC found 160 queued jobs and 20 running jobs across 32 unfinished workflows. Several old Main runs had completed 50 jobs but were still waiting for their final upgrade gate. Workflow counts alone obscure this distinction.

What changes are included in this PR?

  • Make Main the single automatic entry point for PRs targeting master and pushes to master. License and nine existing host/platform/backend/diagnostic workflows become reusable workflows. Child workflows read the supplied scope directly and skip their detector job; manual diagnostics and Desktop preview remain available.
  • Gate tests on License and quality/API-contract success. Gate expensive backend, upgrade, packaging and host acceptance on the selected Python tests as well. Draft PRs allocate no automatic runners; converting to draft cancels the previous run.
  • Run Python 3.11/3.14 unit tests and 3.12 E2E for ordinary PRs. Preserve the complete 3.11–3.14 × unit/E2E matrix for trunk pushes, daily validation, manual Main, ci:full PR events, dependency/CI-policy/shared-fixture changes and unclassified inputs. This reduces ordinary PR Python combinations from eight to three; it does not claim a measured 62.5% reduction in total CI cost.
  • Select scopes from complete Git diffs, including removals/moves and changes beyond native path-filter limits. Trunk comparison retains failed/cancelled changes until a successful Main ancestor verifies them. Unavailable baselines or APIs fail conservatively to full scope and full Python coverage.
  • Preserve complete selected real-backend, native-platform, installation-role and installed-release upgrade acceptance. Bound per-matrix bursts: Python 3, isolated roles 2, upgrade cases 3, OceanBase migration shards 2, replay/capture 2. Upgrade and migration evidence still requires all cases/shards.
  • Add a stable CI result backed by a scope-to-required-jobs contract. Selected jobs that fail, cancel, unexpectedly skip or disappear cannot pass. Cancelled workflows do not allocate new summary-gate runners. Release publication keeps independent complete upgrade/evidence checks.
  • Add daily full ordinary validation at 20:17 UTC, excluding historical schema generation and lock experiments. Keep long lock observation manual. Separate push/schedule/manual cancellation groups.
  • Add a bounded, read-only job queue reporter and self-contained English/Chinese CI architecture, operations, capacity, rollout and rollback documentation. Preserve Cindy consumers, migration sharding and Desktop changes from the current trunk.

Are there any user-facing changes?

Routine PRs receive earlier basic failure feedback and a smaller Python matrix. Intermediate Python-version regressions may first surface on trunk/daily validation; ci:full on the next PR update/reopen and manual Main retain full validation. Selected backends/platforms and release publication are not replaced by simulations or skipped to obtain green CI.

Reusable checks acquire Main caller prefixes. Organization runner pools, spending/concurrency settings and required-status-check rules are not changed by workflow code. After rollout, administrators should require CI result while retaining existing review requirements and verify organization-wide capacity/OS quotas. Per-matrix max-parallel is not a global resource semaphore. Existing backlog cleanup requires confirming each run against its current PR head; this PR does not claim historical runs have already been cleared.

How was this change tested?

  • Focused workflow, selector, gate, publication, schema-equivalence, plugin-contract and migration-shard checks: 326 passed, 1 skipped on Python 3.12. The automatic-entry admission check was additionally replayed after its repository-wide assertion was added.
  • Selection/profile/gate suite on Python 3.11 and 3.14: 93 passed on each.
  • make integration-manifest-check integration-guidance-check: 61 tests passed, manifest and generated guidance checks passed.
  • Official actionlint v1.7.12 (-shellcheck= -pyflakes=): passed.
  • Workflow action pin checker: 218 references checked.
  • Ruff, staged formatting/lint hooks, modified-file ty check and git diff --check: passed.
  • Full make check reports two local global ty unresolved imports: task in tests/e2e/cross_host_work_continuity/fixtures/verify.py and scripts.upgrade_database_probe in tests/test_upgrade_schema_equivalence.py. Both reproduce in an untouched current-trunk bf2ac1fc checkout using the same environment. Their CI checks remain enabled.
  • The queue reporter executed against live upstream Actions metadata with complete coverage of the sampled unfinished workflows and job pages. This is a sampled API report, not an atomic measurement of organization-wide capacity or job queue latency.
  • Local validation does not establish actual GitHub scheduling, native OS or external backend acceptance. Throughput and runner-minute improvement require measurement after rollout. No green full-runtime/tox matrix is claimed from the focused checks.

AI usage statement

Implemented and validated with OpenAI Codex. AI assistance covered Git merge/conflict resolution, CI diagnosis, architecture, implementation, tests, bilingual operations documentation and this PR description.

Comment thread .github/ci-paths.toml
Comment thread .github/workflows/master.yml Outdated
jobs:
seekdb-managed-startup:
needs: changes
if: ${{ !cancelled() && (needs.changes.result != 'success' || fromJSON(needs.changes.outputs.suites || '{}').upgrade) }}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Select startup acceptance when its runner script changes

A change limited to scripts/run_atomic_memory_clean_migration.py selects only tests and database, so this condition skips the dedicated SeekDB managed-startup regression even though it executes that script. Add the script to the upgrade ownership scope so changes to the acceptance runner exercise this consumer.

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI scope map still omits consumers of changed inputs. The startup-runner omission is already tracked in the existing thread; the additional action-consumer omission is detailed below.

Comment thread .github/ci-paths.toml

[shared.node_cache]
patterns = [".github/actions/cache-pnpm/**"]
suites = ["desktop", "website", "dsh", "pi", "opencode", "openclaw"]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Include the Python test matrix among pnpm action consumers

Changing only .github/actions/cache-pnpm/action.yml selects desktop, website, dsh, pi, opencode, and openclaw, but leaves tests=false (reproduced with the checked-in CLI against real Git commits). Main’s tests job also uses this action to prepare the native DSH configuration dependencies before running the unit suite, so action changes now skip that consumer across all four supported Python versions. The package jobs do not exercise this matrix setup. Include tests in this shared scope and add a regression for an action-only change so its consuming matrix remains covered.

@hidb4ai hidb4ai changed the title ci: scope expensive validation to affected changes ci: consolidate validation and reduce pull request runner demand Oct 29, 2026
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Require successful selected checks

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] ci-result 是本次重构引入的唯一汇总门禁(needs 覆盖 master.yml 全部 job,tests/test_workflow_actions.py 也用 set(main["ci-result"]["needs"]) == set(main) - {"ci-result"} 把这个约束固化了),但仓库 ruleset 9930477 里 required_status_checks 是空的(updated_at=2026-10-12T06:39:58+08:00,阻塞项只有 required_approving_review_count=2、non_fast_forward、copilot_code_review,外加 bypass_actors: OrganizationAdmin/always)。也就是说合并阻塞目前只来自 approval 数量,不来自任何 check run。

后果:在 ruleset 侧把 CI result 补成 required check 之前,这个汇总门禁没有任何阻塞力——quality/tests 红、upgrade-release-gate 红都不影响 merge。本次对 upgrade-release-gate 的语义修复(区分 not-required 与 missing acceptance,见 needs: [changes, upgrade-acceptance] 与 UPGRADE_SELECTED 分支)同样只有在该 job 成为 required 之后才对外生效。

建议:合并后在 ruleset 增加 CI result(required check;strict 建议保持 false 以避免频繁 rebase),并在 PR 描述里留一条 follow-up,否则这轮重构的实际收益只剩 runner 数量。


附本机验证范围与两处环境性红色(非缺陷,供复核时排除):

  • exact head b84f6681(base bf2ac1fc,24 文件),跑 pytest tests/test_ci_changes.py tests/test_ci_pipeline.py tests/test_upgrade_release_gate.py tests/test_e2e_workflow_contract.py tests/test_workflow_actions.py → 219 passed, 6 skipped, 2 failed。
  • 两个失败都在 tests/test_upgrade_release_gate.py:489 test_upgrade_gate_distinguishes_not_required_from_missing_acceptance 的参数化分支(success-false-skipped-True、success-true-success-True)。根因是 Windows + PortableGit:用例以 subprocess.run(["bash", "-e", "-c", ...], env={**os.environ, ...}) 传环境变量,实测子进程读到的 SCOPE_RESULT/UPGRADE_SELECTED/UPGRADE_RESULT 全为空(打印为 [][][]),脚本于是走 else 分支并 test "" = success 失败;把同样三个变量直接交给 shell 的 bash -e -c 时脚本 rc=0、行为符合预期。Linux CI 不受影响,不作为本 PR 的缺陷记录。
  • 本 PR 不触碰 server_resources/、server_sources/、release/upgrade-support.json,因此与 R-PC51/R-PC54 的 managed_digest 基线滞后类红色无关;upgrade-release.yml 只加了 strategy.max-parallel: 3(配合 test_compatibility_and_upgrade_matrices_have_bounded_runner_bursts 的 <= 3 约束)。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants