Skip to content

ci(release): require positive upgrades from 1.3.0 on all backends - #2181

Open
Zxf-xufeng wants to merge 10 commits into
masterfrom
codex/issue-2161-upgrade-release-gate
Open

Zxf-xufeng wants to merge 10 commits into
masterfrom
codex/issue-2161-upgrade-release-gate

Conversation

@Zxf-xufeng

@Zxf-xufeng Zxf-xufeng commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Which issue or RFC does this PR close?

Closes #2161. Related: #2154, #2160, and RFC #1771.

Rationale for this change

A release must not pass its upgrade gate while a complete database created by PowerContext 1.3.0 still cannot be migrated. Require a successful upgrade from 1.3.0 to the actual wheel built by CI, on SQLite, embedded seekDB, and container OceanBase. Correctly rejecting the old database does not satisfy this requirement.

What changes are included in this PR?

  • Reject unimplemented acceptance configurations before opening a database. Record the executed configuration in backend evidence and require it to match the declared case; this suite implements only full-server-no-vector.
  • Run installation and acceptance commands under an isolated Linux child subreaper. Clean up detached Server/engine descendants on success, failure, timeout and cancellation before deleting the workspace. Full installed-release acceptance requires Linux; policy and publication regressions remain available on other hosts.
  • Resolve the fixed 1.3.0 source wheel from public PyPI with its SHA-256 and install source/target artifacts in isolated environments. Future 1.4.1/1.5.0 builds keep the same source automatically. Extra source/configuration cases require reviewed adapters; this baseline does not establish compatibility with every intermediate release's data shapes.
  • Add an executable full Server acceptance suite. The actual source Server creates the database and persists memories, source references, Candidate versions, a published Artifact, an access grant and a pagination cursor. Its installed SDK seeds a pending processing intent.
  • Run the target wheel's public db-migrate status/plan/apply/verify commands. Require state=ready and server_ready=true, followed by target startup, preserved data, new writes, FTS retrieval, authorization, Candidate approval/CAS, history/cursors, and fresh/upgraded schema comparison. The installed target SDK must load and idempotently acknowledge the old processing intent.
  • Preserve SQL literals, quoted identifiers, comments and identifier boundaries during schema comparison. Ignore only unquoted formatting whitespace and table-level next-id counters; retain column auto-increment attributes and meaningful DEFAULT/CHECK differences.
  • Seed a grant for an ordinary viewer with the source release. Both installed releases must allow reads in that scope and reject cross-scope reads and writes. The native SDK authentication provider supplies only the fixed viewer identity; real HTTP handlers enforce persisted builtin authorization. Stop the CLI before each SDK probe to preserve exclusive embedded-engine ownership.
  • Require all three backends in both upgrade-required and supported-upgrades modes. Reject empty matrices, absent suites, skips, xfail, partial readiness, command failures and missing evidence. Continue other backend cases after one fails and archive per-case logs, installed versions, schema inventories and results.
  • Reject fresh-install-only as an upgrade policy. Historical rejection regressions remain standalone diagnostics and are not invoked by the positive CI gate.
  • Start embedded seekDB directly on the runner. Provision OceanBase using the existing E2E CE image and MySQL tenant configuration, with dedicated empty test databases, readiness/capacity checks, diagnostics and cleanup on failure.
  • Bind compatibility review to managed persistence inputs, normalizing CRLF/LF checkout line endings while retaining byte-exact distribution SHA-256s. File-set and content changes still require review. Synchronize bilingual installation/release documentation.
  • Validate the complete publication set: exactly one wheel and one matching source distribution, including metadata and an actual sdist-to-wheel rebuild with equal member contents (excluding only derived RECORD). Reject old-version and stale same-version source archives. Acceptance tests a private snapshot; local make publish uploads its exact accepted paths, and CI uploads only validated-dist/. Later writes to the original dist/ cannot join publication.

Are there any user-facing changes?

This PR intentionally has a failing Upgrade release gate until the full migration implementation tracked by #2154 and the relevant schema-probe defects are fixed. Do not weaken the assertions to merge it. Validate the combined migration and CI changes before merging/delivery.

The current upgrade-required declaration keeps upgrade support unverified while always executing positive acceptance. Publication additionally requires the reviewed supported-upgrades declaration after the tests pass. No runtime migration implementation is added here.

This profile covers explicit writes and FTS, not optional vector storage or model-generated output. The processing check covers persisted intent compatibility and acknowledgment, not execution of a model-driven Worker. Post-migration phases remain unverified while migration is blocked.

Administrators must require Upgrade release gate in branch protection. The release.published workflow blocks packages/assets, not the earlier creation of the Release page; use the documented prepublication command before that action.

How was this change tested?

  • Linux gate regressions for 52bebcf2: 89 passed, 1 failed, 1 skipped. All six descendant-cleanup regressions passed, including detached grandchildren, forced timeout, parent termination and repeated interruption, while preserving an unrelated process. The non-Linux rejection test was skipped on Linux. The sole failure is the existing merged-tree persistence digest mismatch in the repository policy check; installed database upgrades did not run. This is not positive upgrade evidence.
  • uv run pytest -q tests/test_upgrade_release_gate.py tests/test_upgrade_publication.py tests/test_upgrade_schema_equivalence.py tests/test_release_version.py tests/test_ci_release_verification.py — 121 passed, 6 Linux-specific tests skipped on macOS. Includes mandatory three-backend coverage, rejection of policy bypasses, continued execution after a backend fails, fixed-source selection for future target versions, and release blocking.
  • Added 12 schema regressions, including actual SQLite defaults/CHECK constraints, quoted values/escapes, table counters and non-ASCII identifier boundaries.
  • Verified ordinary-user authorization on separate actual installed 1.3.0 and target SQLite fixtures, including a negative check: revoking the viewer grant makes the probe fail. These fixture checks do not establish successful migration or three-backend acceptance.
  • Publication regressions invoke an actual offline PEP 517 source build and intercept uploads. They cover old-version, metadata-mismatched and same-version stale source archives, extra/missing artifacts, failed acceptance, and concurrent changes to the original distribution directory. CRLF fixtures verify normalized policy hashes while artifact hashes remain byte-exact.
  • Built the actual project wheel and sdist with uv build, then rebuilt the sdist through the new validator: wheel member contents matched. No package was published during validation.
  • make check and uv run prek run -a — passed, including lint, formatting, type checks, lock/release/action checks and 50 integration manifest tests.
  • Local installed 1.3.0 SQLite fixture preparation passed. The target wheel's fresh-database API operations also passed as a fixture compatibility check; this is not upgrade evidence. Actual local SQLite upgrade acceptance fails at db-migrate status with unknown_baseline.
  • Three-backend acceptance on PR head 10efc32f — failed as required, with all three source-database phases passing. The PR merge wheel was 1.4.1.dev96+gfd76973bd; all cases installed source 1.3.0. Source-distribution rebuilding and wheel-content comparison passed before acceptance. OceanBase provisioning, diagnostic upload and cleanup passed. No validated-dist/ publication set was produced after the failed gate. The remote quality job also passed.
Backend Source database Actual migration failure
SQLite 81 tables; source fixture passed unknown_baseline: schema is not registered
Embedded seekDB 56 tables; source fixture passed incompatible_schema: CHECK expression reported as unbalanced parentheses
OceanBase CE 56 tables; source fixture passed incompatible_schema: unsupported schema option

Each case fails at the public migration status command. Dependent startup/data/schema phases therefore cannot pass; no rejection is counted as a successful upgrade. The run's upgrade-release-evidence-* artifact contains the complete evidence.

AI usage statement

Implemented and validated with OpenAI Codex (GPT-6 family), under maintainer direction.

Comment thread scripts/upgrade_release_gate.py Outdated
"""
inputs = [root / relative for relative in MANAGED_INPUTS]
files = {
path.relative_to(root).as_posix(): sha256(path)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] managed_digest 对工作树字节取哈希,Windows 检出上必然误报持久化变更

managed_digest() 直接 sha256(path.read_bytes()),比较对象是提交在 release/upgrade-support.json 里的常量。.gitattributes 没有为 src/powercontext/builtin/persistence/**、builtin/code/**、server/authz/** 等受管输入声明 eol=lf,因此 core.autocrlf=true 的 Windows 检出会把它们全部转成 CRLF。

实测(同一 commit d7f2d897,同一棵树、只有行尾不同):

  • CRLF 工作树:6b533d6e15f1de1a50d2885f1fce4406c64cc16f6b495eb02cd293f5daad93d7
  • 逐文件把 \r\n 归一为 \n 后:ca3b475f2d0c883e19291a8d8b1dadbe50f93ebd93aad550a9112a3497737927 = 声明值

即唯一差异就是行尾,却会让已并入 make check / make release-check 的 upgrade-policy-check 失败,报错是 Persistence changed: update the compatibility review。Windows 贡献者若照提示改那个常量,写进去的是 CRLF 派生值,Linux CI 上又会红——错误信息把行尾噪声说成了兼容性变更,代价是两边来回改。

建议:

  1. 哈希前归一化行尾(path.read_bytes().replace(b"\r\n", b"\n")),或对 MANAGED_INPUTS 全覆盖声明 text eol=lf,让检出与 blob 一致;
  2. 失败信息区分「文件内容集合变化」与「仅行尾差异」,避免把人引向改常量。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 新 head 8391cf98 未改变本条口径:make check 在 Windows 检出上仍必然报 Persistence changed

scripts/upgrade_release_gate.py::managed_digest() 在本 head 仍是 sha256(path) 直接读工作树字节(第 82 行未改),
而 release/upgrade-support.json 声明的 ca3b475f…37927 仍是 LF 口径:

  • 同一 tree(8391cf98)按 core.autocrlf=true 原样检出时,MANAGED_INPUTS 内 138 个 .py/.sql/.json 为 CRLF,
    managed_digest() 得 6b533d6e…93d7,与声明值不等;
  • 仅把每个文件 \r\n 归一为 \n 后重算得 ca3b475f…37927,与声明值完全一致 ⇒ 差异全部来自行尾,不含真实 schema 语义变化。

upgrade-policy-check 已在 make check 与 make release-check 的依赖里(Makefile 第 22、238 行),
所以 Windows 贡献者未改动任何文件执行 make check 就会收到 "Persistence changed: update the compatibility review";
照提示改常量又会在 Linux CI 上失败。修复方向二选一:哈希前按文件归一化行尾,或为这些路径在 .gitattributes
声明 text eol=lf(与仓库既有的字节敏感资产处理方式一致)。

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 10efc32. managed_digest() now normalizes CRLF to LF before hashing managed text inputs. File additions, removals and content changes still invalidate the compatibility declaration; wheel and sdist hashes remain byte-exact. The diagnostic explicitly states that CRLF/LF is normalized.

Regression tests cover LF/CRLF equivalence across the managed input locations and confirm that artifact hashes still distinguish different bytes. The fix was also confirmed in the re-review.

The current merge-tree digest mismatch is a separate compatibility-review requirement caused by changed persistence inputs. This line-ending fix does not resolve that blocker or establish successful database upgrade acceptance.

@AlexStocks AlexStocks added the 1.4.1 v1.4.1 release tracking label Oct 21, 2026

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One additional P2 in the local publication path. The CRLF digest issue is already covered by the existing review thread.

Comment thread Makefile Outdated
.PHONY: publish
publish: ## Publish a release to PyPI.
publish: ## Publish after release upgrade acceptance; UPGRADE_RELEASE_NOTES must name the reviewed release notes file.
@uv run python scripts/run_upgrade_release_gate.py --release --release-notes "$$UPGRADE_RELEASE_NOTES"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Validate the complete set of distributions before publishing

The runner checks only dist/*.whl, but this recipe subsequently uploads dist/*. In a controlled fixture with a policy-matching 1.4.0 wheel and a leftover 1.3.0 sdist, the gate passed and both files were selected for upload (the upload was intercepted). The sdist therefore bypasses the target-version and acceptance checks; an older build of a same-version sdist would also remain unchecked. Validate the complete publication set and restrict the upload to those validated artifacts.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 10efc32. Publication now requires exactly one wheel and one matching sdist. The validator checks filenames and package metadata, rebuilds the sdist into a wheel, and compares member contents, excluding only the derived RECORD listing. Old-version and stale same-version source archives are rejected.

Acceptance uses a private snapshot of the distribution inputs. Local publication uploads only the exact validated paths, and CI publishes only validated-dist/; subsequent changes to the original dist/ cannot add or replace publication inputs.

Regression tests exercise real offline PEP 517 builds and intercept uploads, covering stale archives, metadata mismatches, extra/missing artifacts, failed acceptance and concurrent changes to dist/. The fix was also confirmed in the re-review. Full three-backend upgrade acceptance remains a separate outstanding requirement.

@Zxf-xufeng Zxf-xufeng changed the title ci(release): gate supported database upgrades on installed artifacts ci(release): require positive upgrades from 1.3.0 on all backends Oct 22, 2026

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 10efc32f and replayed both counterexamples. The CRLF digest and publication-set P2s are fixed. Publication now validates the wheel/sdist pair and uploads only the accepted snapshot. No new confirmed defects found in this pass.

Merge remains blocked by the Upgrade release gate. I inspected its downloaded evidence: all three 1.3.0 source fixtures completed, but the public migration status command failed on every backend:

  • SQLite: unknown_baseline — the schema is not registered.
  • Embedded seekDB: incompatible_schema — a CHECK expression is reported as having unbalanced parentheses.
  • OceanBase: incompatible_schema — an unsupported schema option is reported.

Post-migration startup, read/write, authorization, history/cursors, and schema equivalence therefore remain unverified. Complete the migration integration and rerun positive acceptance on all three backends before merging.

@Zxf-xufeng
Zxf-xufeng marked this pull request as draft October 22, 2026 08:56
@Zxf-xufeng
Zxf-xufeng marked this pull request as ready for review October 23, 2026 10:07

@hireview001 hireview001 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • CI failed: Upgrade release gate (failure).
  • CI failed: Main (failure). Failed steps: Upgrade release gate / Run declared installed-release acceptance.

@Zxf-xufeng

Copy link
Copy Markdown
Member Author

The upgrade CI failure is expected at this stage. This PR requires successful end-to-end upgrades from PowerContext 1.3.0 on SQLite, embedded seekDB, and OceanBase, but the complete migration framework integration is still pending in PR #2168.

PR #2168 needs to address the migration framework blockers before this gate can pass. Once those changes are integrated, we will rerun acceptance on all three backends and verify the remaining migration and post-upgrade checks. The gate should remain failing until those checks actually succeed; expected failure here is not grounds to bypass the gate.

@Zxf-xufeng

Copy link
Copy Markdown
Member Author

Addressed the remaining local review findings in commit 9f78868:

  • Schema equivalence (P2): DDL comparison now preserves literals, quoted identifiers, comments, and identifier boundaries. It ignores only unquoted formatting whitespace and table-level auto-increment counters. Different DEFAULT/CHECK values such as 'a b' and 'a b' no longer compare equal. Added regression coverage using actual SQLite behavior, plus quoting, escaping, counter, and identifier-boundary cases.
  • Authorization coverage: The source fixture now grants scope.viewer to an ordinary user. Both installed releases must allow that user's reads in the granted scope and reject cross-scope reads and writes. The probe uses the installed Server's real HTTP handlers and persisted builtin authorization; only the fixed authentication identity is supplied. CLI startup, invalid-token rejection, and persisted-grant checks remain mandatory.

Validation: 113 focused tests passed; make check and uv run prek run -a passed. Separate installed 1.3.0 and target SQLite fixtures passed the viewer checks, and revoking the grant correctly caused the probe to fail. These are fixture validations, not evidence of a completed upgrade.

The existing positive upgrade gate remains mandatory and unchanged in its success requirements. The migration framework blockers tracked in #2168 and the related schema-probe defects still need to be resolved before full acceptance can pass on SQLite, embedded seekDB, and OceanBase. No skips or expected-failure bypasses were added.

@hidb4ai

hidb4ai commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

test failed

@hireview001 hireview001 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 9f7886894489. The branch snapshot passes the focused publication, gate and schema-equivalence checks.

The current Upgrade release gate stops before installed-release acceptance: CI tests merge 53b134d412c8 (this head into 1da997476651), whose compatibility declaration is ca3b475f...37927 while the actual managed-input digest is 59e34e86...1ab95. I reproduced the mismatch from that exact CI tree; the branch's own digest matches. The quality and unit-test failures have the same preflight cause.

Please integrate the migration changes, review/update the declaration for the resulting tree, and rerun positive installed 1.3.0-to-target acceptance on SQLite, seekDB and OceanBase. This CI run has not reached those upgrade checks; its failure should not be treated as an exercised migration rejection.

@hidb4ai

hidb4ai commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

@Zxf-xufeng Please update this PR together with #2168. The current CI failure is a compatibility-declaration mismatch in the merge tree, so the gate stops before installed-release upgrade acceptance.

Please integrate the migration changes, review the resulting persistence changes and refresh the declaration, then rerun positive installed 1.3.0-to-target upgrades on SQLite, seekDB and OceanBase. Please share an ETA and reply with the updated commit and passing CI evidence; an expected gate failure is not sufficient for merge readiness.

@Zxf-xufeng

Copy link
Copy Markdown
Member Author

@hidb4ai Following up on your integration and acceptance request: agreed. The current failure is a compatibility-declaration mismatch in the merge tree, before installed-release upgrade acceptance. My earlier description of the failure as expected was too broad; this run does not demonstrate an exercised migration rejection or merge readiness.

Progress on #2181: commit 52bebcf adds rejection of unimplemented acceptance configurations, verifies configuration evidence, and cleans up detached acceptance processes on timeout, failure and cancellation. All six Linux process-cleanup regressions passed in the latest gate job. The job still failed on the existing persistence-digest mismatch, so full database upgrade acceptance has not run.

#2168 remains open and is the dependency for the complete migration implementation. Once those changes are ready for integration, we will review the resulting persistence changes, refresh the compatibility declaration, and rerun positive installed 1.3.0-to-target acceptance on SQLite, embedded seekDB and OceanBase against the same final build. I will follow up here with the integrated commit and passing CI evidence; this request remains outstanding until then.

I do not yet have a reliable ETA because completion depends on #2168 and the combined three-backend acceptance results. We will not bypass the gate or treat an expected failure as sufficient for merging.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.4.1 v1.4.1 release tracking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci(release): gate supported upgrades on complete databases from actual prior releases

4 participants