Skip to content

feat(artifact): add managed plugin framework and some example - #2246

Merged
hidb4ai merged 16 commits into
oceanbase:masterfrom
Junkrat77:codex/artifact-plugin-framework
Oct 31, 2026
Merged

hidb4ai merged 16 commits into
oceanbase:masterfrom
Junkrat77:codex/artifact-plugin-framework

Conversation

@Junkrat77

Copy link
Copy Markdown
Contributor

Which issue or RFC does this PR close?

Related to RFC #2175. RFC changes are maintained separately.

Rationale for this change

Allow applications to generate custom Artifact families through plugins. PowerContext manages generation, persistence, revisions, authorization, retrieval and exact evidence reads.

What changes are included in this PR?

  • Plugin declarations, configuration validation, Worker generation, evidence validation and transactional publication.
  • Discovery, loading, enable/disable/unload, code and configuration upgrades through reload, and restart recovery.
  • Shared listing, text/vector/hybrid retrieval and exact revision reads for custom families.
  • Initial index construction, incremental maintenance and failure recovery.
  • Frozen registered index definitions and searchable schema identity. Query-policy updates do not rebuild indexes, interrupt search or re-encode vectors.
  • Shared database initialization for search tables and native vector spaces.
  • HTTP and generated client contracts, minimal plugin examples, usage documentation and regression tests.

Eight logical tables store family declarations, index definitions, vector spaces, build state, coverage, concurrent changes and text/vector projections, with backend-specific FTS and native vector storage support.

Are there any user-facing changes?

  • Administrators configure a plugin directory and manage plugins over HTTP.
  • Applications discover custom families and read, search or cite exact revisions. Existing content remains readable after disablement or unloading.
  • Read responses expose optional schema_version; ArtifactReadFamily accepts strings instead of a closed enum.
  • Reload validates the candidate, reaps old Workers and installs the new definition while preserving a healthy registration's enabled/disabled state. Candidate validation failure preserves the accepted definition and state.
  • Plugins must be trusted local code; subprocess isolation is not a security sandbox.
  • Reload cannot change registered index definitions. Incompatible existing search table layouts are rejected.

How was this change tested?

  • Core index/read/maintenance/declaration tests: 26 passed across focused runs.
  • Real Server/HTTP/Worker regressions: 15 passed across initial and targeted runs; 2 external-backend cases skipped.
  • make contract-test: 56 passed.
  • uv run prek run -a: passed.
  • Desktop generate:check, TypeScript checks and isolated generated Rust wire compilation: passed.

Validation limits: real OceanBase/seekDB execution was not verified. The full Tauri build was not run because GTK/WebKit development dependencies were unavailable.

AI usage statement

OpenAI Codex assisted with implementation, extracting the Artifact framework changes, mainline compatibility, regression checks and local review.

@hireview001 hireview001 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Resolve merge conflicts with the target branch.

if self.access is not None:
await self.access.for_read(connection).require(
self.principal,
AccessAction.ARTIFACT_READ,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve Topic Memory's scope authorization for plugin evidence

When a plugin declares artifact_families=("topic-memory",) and reads an existing revision under enforced authorization, this unconditional ARTIFACT_READ check raises AccessInvalidRequestError because Topic Memory has no artifact access profile, preventing generation from completing. Apply its existing scope-read authorization consistently to evidence reads, listing, and commit-time reference checks.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. Fixed in 2ac6c60.

Plugin evidence access now follows the existing Scope authorization rules, including for Topic Memory. Redundant per-Artifact read checks are skipped where the built-in provider already grants access through the Scope. Custom-provider filtering and output write authorization remain intact.

"schemas/sqlite-s1402.json",
"schemas/mysql-s1402.json"
],
"v1_5_0_001": [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 注册 v1_5_0_001 后没有更新 release/upgrade-support.json,quality 与 4 个单测 job 全红

head 71d5b44a 的失败签名是 Upgrade release gate failed: Persistence changed (file set or content; CRLF/LF normalized): update the compatibility review(quality job 122044641036,make check → upgrade-policy-check)。

在 exact head 副本上重算 upgrade_release_gate.managed_digest() 得 2046f6687ee3522b3e3ef2eb97b0b7b07d2cc0993042c2fb2965f772027fe28b,而 release/upgrade-support.json 的 schema_review.persistence_sha256 仍是 master 的 9b67d639cf57b3ae7cf37dcd1845f50561af36bb19edd9a7b979408f31e9a6de;本 PR 的 102 个文件里没有该文件。

同一 head 的 tests (3.11/3.12/3.13/3.14, unit) 各 4 条失败全部是同一签名 tests/test_upgrade_release_gate.py::test_repository_baseline_covers_all_three_databases[*],upgrade-acceptance / Prepare upgrade artifacts and matrix 也是同一个 GateError。同日 base 为 cdb8da29 的 #2258 在该 job 绿,可排除基线。

建议在注册新 revision 的同一次改动里更新 schema_review:重算 digest,并在 rationale 中说明 v1_5_0_001 新增的存储面(pc_artifact_search_*、pc_artifact_vector_*、pc_artifact_index_*、pc_artifact_family_schemas 等)与保留对象不变,然后重跑 make upgrade-policy-check。

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. Fixed in 9fe2e68.

Updated release/upgrade-support.json with the persistence digest and compatibility rationale for the Artifact plugin storage and migration changes. The digest was refreshed again after the latest mainline merge. The upgrade policy check and related regression tests pass. The formal release version and supported upgrade scope remain unchanged.


def upgrade():
resource = Path(__file__).parents[1] / "schemas/sqlite-v1_5_0_001.json"
for statement in json.loads(resource.read_text())["statements"]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 这 21 条 DDL 进入每次 bootstrap,Acceptance (oceanbase) 已超 40 分钟步预算

sqlite-v1_5_0_001.json 含 21 条语句(ALTER TABLE ... ADD COLUMN 与新建 pc_artifact_search_*、pc_artifact_vector_*、pc_artifact_index_* 等表),本行逐条 exec_driver_sql。每个 e2e 用例都会新建数据库,因此这 21 条 DDL 进入每一次 bootstrap。

head 71d5b44a 的 Acceptance (oceanbase)(job 122044916138)在 21 条用例跑到 19 条时被 The action 'Verify OceanBase Atomic Memory and HTTP regressions' has timed out after 40 minutes 终止;单条用例耗时 140–160 s(06:29:17→06:31:45→06:34:13)。同日 #2258 的同一批 21 条用例单条 90–100 s,整批 26 分钟跑完。量级上 21×150 s≈52 min,已超出 40 min 步预算。

建议:把新列合并进更少的 ALTER(OceanBase 支持单条多列),或按 R-PC26 的做法把新增存储面的初始化从每个用例的 bootstrap 里移除/单独 job 化,并在 PR 正文说明 CI 预算如何吸收这批 DDL。

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for flagging the initialization cost. Addressed in 9fe2e68.

Combined the seven registration-column additions into one ALTER TABLE, reducing the Artifact migration’s native DDL statements from 16 to 10. Added a regression test for interruption and recovery after the combined ALTER. The E2E harness run for that commit, including OceanBase acceptance, passed without increasing the timeout.

@AlexStocks AlexStocks added the 1.5.0 v1.5.0 release tracking label Oct 27, 2026

@hireview001 hireview001 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_bind_evidence_access(runtime, active_access_control, resolved.access.mode)
if (
config.runtime.artifact_processing_supervisor_mode == "global"
and config.runtime.artifact_processing_role != "api"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Restore custom-family authorization independently of worker startup

When a server starts with processing role api or supervisor mode dedicated, this condition skips the manager that restores persisted custom-family access profiles, causing authorized reads of retained custom artifacts to fail with AccessInvalidRequestError. Restore those profiles independently of plugin generation and management so existing artifacts remain readable in these server modes.

@Junkrat77 Junkrat77 Oct 28, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. Fixed in b2af2aa.

The authorization profiles for persisted custom Artifact families are now restored during Server startup, even when the plugin manager is not started. This allows authorized users to read existing artifacts and their historical revisions while preserving permission checks. Regression tests also verify that access to unauthorized Scopes remains denied.

@Junkrat77

Copy link
Copy Markdown
Contributor Author

Migration versioning follow-up:
PR #2187 introduced durable owner-write recovery, and PR #2212 registered its storage as v1_4_1_004. This migration only creates pc_access_owner_recovery. It is merged into master but is not included in the published 1.4.1 release, whose migration chain ends at s1402.
For this mainline merge, we are retaining the chain s1402 → v1_4_1_004 → v1_5_0_001 to preserve both the upstream owner-recovery migration and this PR’s Artifact plugin migration.
Before the planned 1.5.0 release, the unreleased migrations’ version names should be aligned with their actual release. Whether to retain separate sequential revisions or consolidate them can be decided during that review.
@hidb4ai

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One reproducible capability-discovery issue remains in server modes without the plugin manager.

Comment thread src/powercontext/server/app.py Outdated
return request.app.state.capabilities
value = capability_provider() if capability_provider is not None else request.app.state.capabilities
manager = request.app.state.artifact_plugin_manager
if manager is None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Discover retained custom families without the plugin manager

This early return drops persisted custom families from /v1/capabilities whenever the Server runs in dedicated supervisor mode or with processing role api, because those modes intentionally do not create ArtifactPluginManager. Reproduced with file-backed SQLite, artifact_processing_supervisor_mode="dedicated", enforced authorization and an administrator: after storing startup-report and restarting, both GET of the artifact and GET of /revisions/1 return 200, while capabilities returns 200 with custom_artifact_schemas: [] and omits startup-report from artifact_families. Applications using discovery therefore cannot find data that this Server can read. Read the persisted declarations independently of the generation/management lifecycle, as the startup authorization restoration now does.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. Capabilities discovery now reads persisted custom-family declarations independently of the plugin manager, so retained families remain discoverable in API-only and dedicated-supervisor deployments. Added regression coverage for capabilities discovery after Server restart.

embedding = None
if spec.inference_config is not None:
embedding, _ = await _embedding_models(spec.inference_config, resources, None)
config = BuiltinConfig(database=spec.database, runtime=spec.runtime_config or RuntimeConfig())

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve the Server inference configuration in plugin Workers

When generation_model is configured and artifact_processing_families is unset, this BuiltinConfig defaults inference to an empty configuration, causing open_builtin_contexts to reject every plugin Worker because its capability manifest differs from the Server's persisted processing manifest. Include spec.inference_config in the Worker configuration so plugins can generate alongside configured built-in families.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. Plugin Workers now preserve the Server’s inference configuration when constructing BuiltinConfig, so inferred processing capabilities match the persisted manifest when generation_model is configured and artifact_processing_families is unset. Added regression coverage verifying plugin generation both with and without inference configuration.

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Junkrat77 Please sync with current master, regenerate the checked-in Cindy worker, and rerun cindy-package. The shared operation additions leave its bundle stale in the tested merge result; details are inline.

Comment on lines +34 to +40
list_artifact_plugins: { method: 'GET', path: '/v1/admin/artifact-plugins', location: null, scopeMode: 'none', pathParameters: [], queryParams: [], headerParams: [], successStatuses: [200], emptyStatuses: [] },
get_artifact_plugin: { method: 'GET', path: '/v1/admin/artifact-plugins/{binding_name}', location: null, scopeMode: 'none', pathParameters: ['binding_name'], queryParams: [], headerParams: [], successStatuses: [200], emptyStatuses: [] },
enable_all_artifact_plugins: { method: 'POST', path: '/v1/admin/artifact-plugins/enable-all', location: null, scopeMode: 'none', pathParameters: [], queryParams: [], headerParams: [], successStatuses: [200], emptyStatuses: [] },
reload_artifact_plugin: { method: 'POST', path: '/v1/admin/artifact-plugins/{binding_name}/reload', location: "body", scopeMode: 'none', pathParameters: ['binding_name'], queryParams: [], headerParams: [], successStatuses: [202], emptyStatuses: [] },
enable_artifact_plugin: { method: 'POST', path: '/v1/admin/artifact-plugins/{binding_name}/enable', location: "body", scopeMode: 'none', pathParameters: ['binding_name'], queryParams: [], headerParams: [], successStatuses: [202], emptyStatuses: [] },
disable_artifact_plugin: { method: 'POST', path: '/v1/admin/artifact-plugins/{binding_name}/disable', location: "body", scopeMode: 'none', pathParameters: ['binding_name'], queryParams: [], headerParams: [], successStatuses: [202], emptyStatuses: [] },
unload_artifact_plugin: { method: 'POST', path: '/v1/admin/artifact-plugins/{binding_name}/unload', location: "body", scopeMode: 'none', pathParameters: ['binding_name'], queryParams: [], headerParams: [], successStatuses: [202], emptyStatuses: [] },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Rebuild the Cindy bundle for the new shared operations

Cindy now imports the shared DSH client, so these seven additions also change its checked-in integrations/cindy/plugins/powercontext-hooks/node/worker.cjs. The tested merge f958fcfe fails make cindy-test at the bundle git diff --exit-code check (job 123867735891). Confirmed against mainline bf2ac1fc: the unmodified base rebuilds cleanly, while replacing only this operation table with the PR version produces exactly 77 added bundle lines, matching CI. Refresh against current master and commit the regenerated Cindy bundle alongside the DSH bundle; rerunning the same code cannot clear this deterministic packaging failure.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Junkrat77 Follow-up on the current head 0d78bcb1: the Cindy bundle mismatch is still unresolved, cindy-package is still failing, and GitHub reports merge conflicts with master. Please sync with current master, resolve the conflicts, regenerate and commit integrations/cindy/plugins/powercontext-hooks/node/worker.cjs, then rerun the affected checks. The base-versus-PR reproduction above still applies because the head has not changed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 079de10. Synced with master and regenerated the Cindy worker bundle alongside the DSH bundle. The Cindy bundle now includes the seven added plugin management operations. make cindy-test passes locally, including the generated-bundle drift check and packaging. The updated CI runs have been triggered and are pending.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

在当前 head 079de103 上复核,该线程根因已闭合,但证据只到字符串覆盖层,不能替代 make cindy-test 的字节比对。

核对方法:取 integrations/dsh/plugins/powercontext/src/operations.generated.ts 的 operation 名集合,与本 PR 的 base(master 90355b8e)做差集,得到 7 个新增 operation:disable_artifact_plugin、enable_all_artifact_plugins、enable_artifact_plugin、get_artifact_plugin、list_artifact_plugins、reload_artifact_plugin、unload_artifact_plugin。这 7 个名字全部出现在 079de103 的 integrations/cindy/plugins/powercontext-hooks/node/worker.cjs 中。

上一轮 0d78bcb1 的树里根本没有 worker.cjs(该文件由 master 的 cindy 集成引入,本 PR 的旧 base 早于它),所以当时的比对是在缺文件的前提下做的;079de103 的 merge 已经把该文件带进来并重建(blob 0d9e03a0,master 为 dd9f77e1)。

剩余风险:字符串命中说明描述符被写入 bundle,但不足以证明 bundle 与生成器输出逐字节一致——cindy-package 的 git diff --exit-code 判据在 079de103 上仍是 pending(该 head 22 个 check 中 20 个 pending)。合并前需要该 job 变绿,或本地跑 make cindy-test 复核。

@hireview001 hireview001 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/powercontext/__init__.py Outdated

"""Stable public exports for PowerContext clients."""

from powercontext import artifact_plugins

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep optional dependencies out of the base package import

With only the declared core dependencies installed, import powercontext now fails because artifact_plugins imports artifacts.indexes, which initializes the Memory package and imports optional httpx through jev_reranking. Make the plugin export lazy or remove this eager dependency chain so core-only installations remain importable.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Junkrat77 Independently confirmed on f062a1d2 using built wheels installed without extras into two fresh Python 3.14 environments. Both environments have only the declared core dependencies and no httpx: the base 0c1dfc28 wheel imports successfully, while the PR wheel fails on import powercontext with ModuleNotFoundError: No module named 'httpx' through __init__ -> artifact_plugins -> artifacts.indexes -> builtin.artifacts.memory -> jev_reranking. This is a PR-introduced core-installation regression, not a development-environment dependency issue. Please restore imports with the declared core dependencies before merging and cover the installed wheel without extras in regression validation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6d567f4. The top-level artifact_plugins export is now loaded lazily, so importing powercontext no longer pulls in optional runtime dependencies.

Added an installed-wheel regression without extras to the Python 3.11 and 3.14 CI matrix. Local validation on Python 3.12 passed in a fresh environment without httpx; explicit plugin imports and the existing storage/search smoke test also passed with the builtin extra installed.

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Junkrat77 Re-reviewed exact head 322b7a956ae58230b9305cd943ce66a7d23ba868. The core-only import regression is fixed: a freshly built wheel imports successfully without extras and without httpx. The new Source-to-Artifact test also passed locally through a real Server, MCP client and spawned Worker; cindy-package is green.

The remaining merge blocker is CI: quality, all four unit-test jobs, and Recall OceanBase are still failing. The failures are in inherited test fixtures: test_code_publication.py omits the required SeekDBGraphStore(database=...) argument, and the MySQL vector-admission test mock does not accept allow_missing_database. The affected code is unchanged from base 0c1dfc28; the missing-database failures also reproduce on that exact base. Recall OceanBase stops at the same type-check errors before acceptance runs. These are baseline failures, not new correctness findings against this PR. Please coordinate the baseline fixture fixes and rerun the affected checks before merging.

@hidb4ai
hidb4ai merged commit 176b3f9 into oceanbase:master Oct 31, 2026
81 of 87 checks passed
jackie-cqz added a commit to jackie-cqz/powercontext that referenced this pull request Nov 1, 2026
Resolve conflicts with the managed plugin framework (oceanbase#2246) and the 36 other
commits that landed since the previous merge, including PreparedContext
receipts (oceanbase#2258) and the consolidated CI (oceanbase#2285).

- `server/authz/profiles.py` and `service.py` were edited upstream
  (`register_custom_artifact_access_family`, `prepare_resource`) after this
  branch moved the package. The edits apply to `builtin/authz` unchanged apart
  from the docstring and import-order edits already made here.
- `server/app.py`, `server/factory.py`: keep the relocated `builtin.authz`
  imports and add only the names the new upstream code references.
- Restructured upstream tests (`harness.py`, `test_lexical_admission.py`,
  `test_local_dashboard_access.py`, `test_unified_lexical_search.py`): take the
  upstream version and drop `scheduler_path` again.
- `release/upgrade-support.json`: keep the rationale added upstream, append the
  sentence about this branch's code-organization changes, and recompute the
  persistence digest. The generated table and index DDL for all 72 tables is
  identical to master on the SQLite and MySQL dialects.

Follow the moves into code that arrived with the merge:

- The plugin framework's `builtin/runtime` modules import access control from
  `powercontext.builtin.authz` and the worker-side security from
  `powercontext.builtin.authz.worker`, so no new `builtin` -> `server` import
  is introduced.
- Restore the `perf_counter` import in `builtin/runtime/application.py`; the
  receipt code uses it.
- Drop the extra `scheduler` argument from the two harness helpers' new callers
  (`test_native_code_fixture.py`, `test_real_experience_skill_baseline.py`).
- Point the CI change-selection test at `builtin/authz/owner_recovery.py`,
  which classifies the same way as the previous path.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.5.0 v1.5.0 release tracking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants