Skip to content

feat(media): add opt-in image evidence with Artifact processing - #1956

Open
knqiufan wants to merge 23 commits into
oceanbase:masterfrom
knqiufan:codex/experiment-image-source
Open

knqiufan wants to merge 23 commits into
oceanbase:masterfrom
knqiufan:codex/experiment-image-source

Conversation

@knqiufan

@knqiufan knqiufan commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Which issue or RFC does this PR close?

Refs #1932 and RFC #1972, pinned at 919d6b30. The RFC remains open; this implementation does not imply merge approval.

Integrated with master 6b976859. Images remain disabled by default.

Rationale for this change

Preserve submitted screenshots and diagrams as durable, inspectable evidence. Machine interpretations use existing Artifact revisions and authorization, while deterministic text publications feed existing Source-only Memory processing without treating a late interpretation as a new fact.

What changes are included in this PR?

  • image.original Source → exact image-analysis Artifact revision → deterministic image.interpretation Source → existing Atomic Memory, Topic Memory and Handoff flows. Generic Source and Artifact reads expose exact history; image-analysis is excluded from generic mutation and default recall.
  • Three shared tables: pc_media_assets for scoped BLOBs, pc_media_processing_requests for pinned inputs, immutable initial receipts and bounded progress, and pc_media_capacity for transactional derived counters. Admission commits bytes, Source, request, receipt, capacity and intent together. Result publication is also atomic. No private image schema-version table, file store, lease table or separate scheduler remains.
  • Existing ArtifactProcessingSupervisor binding with independent image budgets, Vision-only child processes, fencing, permission checks before dispatch/publication, committed attempts, bounded recovery and stale-response rejection. Retry preserves the failed profile; reparse checks the exact Artifact revision, profile digest and any replaced request/state version.
  • Exactly five new operations: capture_image, get_media_asset, get_media_request, list_media_requests and request_image_analysis. Existing capabilities/Source/Artifact reads supply diagnostics and content. Python, JS, Desktop, CLI, Dashboard and Codex delivery use the native contract.
  • Versioned, bounded Memory support tied to exact kind/text and individual statements. Origin order is separate from the consumption cursor. Explicit fact dates take precedence; older or unknown conflicting evidence preserves the current identity, while independent facts can progress. Exact associations survive revisions, restoration and ordinary logical copy; corrupt support is rejected.
  • Shared migration plans, maintenance locking, native SQLite backups and frozen media DDL. Active-request indexing avoids scheduling through terminal history. A reproducible capacity comparison records admission/lock costs, query plans and full startup integrity costs at 1 GiB and the 1024-request ceiling.

The existing collector recovery and builtin installation findings remain fixed: temporary capacity is recoverable, disabled workspaces cannot monopolize delivery, and builtin/images include their direct platformdirs dependency. Review threads are left for their reviewers.

Are there any user-facing changes?

Static PNG/JPEG intake initially supports durable file SQLite. Decode support is optional through powercontext[server,images]; ordinary Server installation works without Pillow. Configuration and normal Codex setup preserve explicit workspace consent and managed collector lifecycle.

Compatibility and migration: the five native operations replace the seven experimental HTTP operations; clients and Server must be upgraded together. Existing text databases require the registered media plan/apply maintenance workflow, and processing ownership changes use the existing processing migration. Legacy experimental Source decoders remain available. Seven-table/file deployments are rejected before migration writes with experimental_media_import_unsupported; this PR does not provide a qualified converter or silently rewrite historical identities.

A complete native SQLite backup includes media bytes and requests. Portable logical archive v1 rejects scopes containing media before publishing or restoring an incomplete archive; ordinary Memory ordering metadata supports logical copy.

Artifact sharing permits reading the interpretation without granting access to the original Scope, bytes or request. Receipt, interpretation, Memory publication and fresh-session recall remain separate outcomes.

How was this change tested?

Executed on Windows 11 with Python 3.14.5; overlapping collections are not summed:

  • Broad Memory/archive/HTTP/storage/Evidence/Dashboard set: 362 passed, 66 skipped. After master integration and statement-support changes, Memory/order/unrelated-candidate set: 108 passed, 3 skipped. Final media storage/migration checks: 11 passed.

  • Actual Supervisor spawn, HTTP provider, OS termination/recovery, retry budgets and stale-result rejection: 5 passed in a fixed checkout.

  • API/JS/Desktop/image contracts: 70 passed. Authorization, records, HTTP admission and integration manifest: 103 passed, 4 skipped.

  • Codex, image CLI, wizard and archive CLI: 393 passed, 6 skipped; two strict deadline cases passed separately (2 passed). Parallel Windows runs showed timing variability; assertions and production timeout logic were preserved.

  • Built wheel at 4894785a: six independent environments, Python 3.11/3.14 × builtin/images/server+cli, passed isolated runtime smoke checks. Fresh Server environments explicitly verified Pillow was absent.

  • All 578 packaged runtime files matched the recorded wheel source commit. Native Desktop protocol passed anonymous, Bearer, HTTPS CA/base-path and enforced-provider modes. The product qualification regression passed; qualification identities and reproduction preserve historical records.

  • Desktop lint, 98 UI tests, build and generated contract checks passed. DSH build and 427 passed, 1 skipped. Website lint/tests/build verified 980 public pages and internal links.

  • Official Python/JS generation checks, both uv 0.10.12 lock checks, integration docs, Linux-targeted ty, non-type pre-commit hooks and diff checks passed. Native Windows ty retains 34 existing diagnostics; whole-platform type checking is not claimed green.

  • CI correction at 6b86fcd4: exact capability assertions passed 100 tests (4 backend configuration skips); Desktop IPC export check, typecheck and 98 UI tests passed; Dify pinned host/daemon compatibility passed 82 tests and real SDK/SQLite/HTTP passed 3 tests; Skill guidance passed 28 regressions and checksum-verified skill-up 0.12.0 native validation/dry-run. All eight GitHub workflows on 6b86fcd4 passed: Main (36/36 jobs, including Python 3.11–3.14 unit/E2E and Dify), Desktop Windows/macOS arm64/macOS Intel, actual SQLite/OceanBase acceptance and clean migration, Skill guidance, Native personal service, Kimi, ZCode and License. These are the new-head results, not reused earlier checks.

The 14 failed jobs on 74dada26 came from four omissions in this submission: capability expectation lists, Desktop IPC export, Dify contract generation, and the neutral Skill tool-argument reference. The OceanBase evidence check was downstream of the stale readiness assertion, without a demonstrated storage failure. An additional Dify Windows path-separator bug also existed on base 6b976859; it is fixed here and covered by the existing generator regressions. A separate, non-blocking aiosqlite worker-thread Event loop is closed warning occurs in the unchanged test_processing_maintenance_plan_leaves_the_stored_journal_mode_unchanged on both base and this head. Base Python 3.11 CI passed with the same warning. A follow-up issue should investigate connection/loop shutdown ownership; the warning alone does not establish a production failure or an image regression. Dependency deprecations are also separate from the failed-job causes.

Open Code Review delegate selection/rules and direct inspection covered 74 files in this architecture-change scope, with no skipped files. Final follow-ups fixed bounded integrity recovery, exact support validation, image read/share authorization, Topic notifications and active scheduling indexes. This is not a claim to have freshly audited all unchanged earlier collector code.

Acceptance limits: loopback Vision and injected Memory models validate local behavior, not external model quality. Real Codex Desktop drag/paste, fresh-chat injection/recall and native macOS/Linux collector lifecycle were not requalified. Capacity results include concurrent build load and are observations, not a general latency guarantee. New-head GitHub CI is separate from local validation.

Details: contracts, migration/backup, capacity and acceptance, implementation log.

AI usage statement

OpenAI Codex assisted with implementation, integration, documentation and validation. Open Code Review's delegate workflow provided file selection and review rules; the host agent performed the review and fixes. The author remains responsible for implementation and acceptance before merge.

@knqiufan
knqiufan force-pushed the codex/experiment-image-source branch from df4c766 to 078cf6f Compare October 13, 2026 01:23

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two reproducible collector recovery issues remain: temporary outbox capacity pressure permanently stops reading the affected turn, and pending attachments from a disabled workspace can block delivery for other enabled workspaces. Details are inline.

outbox.finish(row["id"], state="done", error="image_consent_changed")
except (ValueError, TypeError, KeyError, OSError, AttributeError, RecursionError) as error:
code = str(error) if str(error).startswith("image_") and len(str(error)) < 80 else "image_read_unavailable"
outbox.finish(row["id"], state="error", error=code)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep temporary outbox capacity pressure recoverable

When the shared 128 MiB staging budget fills, Outbox.advance() raises image_outbox_full, and this handler permanently marks the turn's reader as error. Outbox.jobs() subsequently selects only watching jobs, while reset_delivery() resets attachments but never restores the reader. Thus an offline backlog can cause all remaining images in that turn to be omitted even after uploads free space or the user explicitly retries. I reproduced this with a scaled capacity limit: the first attachment was staged and then uploaded, but the second remained unread after capacity was freed and delivery reset. Keep the saved cursor recoverable for capacity pressure so reading can resume without changing its consent boundary.

for row in outbox.pending():
try:
if not _permitted(json.loads(row["job"]), settings):
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Allow enabled workspaces to progress past disabled deliveries

Outbox.pending() selects the oldest eight due attachments globally before this consent check. If workspace A accumulates eight pending attachments while offline and capture is then disabled for A, every pass selects and skips those same rows; they remain pending and due forever. Images from an enabled workspace B are consequently never selected. I reproduced this with eight staged attachments for A and one for B: after disabling only A, three delivery passes made zero uploads and B remained pending. Preserve A's consent and staged data, but ensure ineligible rows cannot monopolize the delivery page, for example by suspending them or advancing through a bounded eligible selection.

@knqiufan
knqiufan force-pushed the codex/experiment-image-source branch from 9090294 to f71748a Compare October 13, 2026 03:31
@hidb4ai

hidb4ai commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Before moving this capability toward merge, I would like us to review the design through an RFC:

  1. RFC first. This introduces image ingestion, durable assets, background vision processing, host capture, and new public operations. Please prepare an RFC and have the design reviewed before proceeding with implementation approval. It should describe the user workflow, architecture boundaries, API and storage changes, and acceptance criteria.

  2. Evaluate extending the existing Artifact framework. Could the user-facing image capability be implemented by extending existing Artifact families and their evidence/provenance flows? Please explain which responsibilities fit the existing Source/Artifact contracts and which require a separate image service. We should reuse existing versioning, authorization, retrieval, and evidence inspection where their contracts fit.

  3. Be conservative about new database tables. The project already has many tables, and adding seven more deserves careful justification. Please evaluate reuse of existing storage and consolidation of the proposed tables first. For each table that remains necessary, explain its distinct responsibility and why existing structures cannot meet the requirement. Reuse should preserve idempotency, recovery, historical evidence, and concurrency guarantees; it should not simply mix unrelated business state into an existing table.

The feature has a useful direction, but I would prefer to settle these architectural choices in the RFC before merging it.

knqiufan commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

Thank you for the architectural feedback. I agree that this capability should go through RFC review first. I’ve opened RFC #1972, linked to #1932, for design review.

It compares the Source/Artifact boundaries, evaluates existing binary storage and shared processing contracts, and gives a table-by-table analysis with a proposed three-table design. The proposal keeps idempotency, recovery, historical evidence and concurrency guarantees explicit.

I’ll keep #1956 as a draft prototype and wait for the RFC to be reviewed and accepted before proceeding with further implementation changes. After that, I’ll align the implementation with the agreed design and split it into focused PRs where appropriate.

@hidb4ai

hidb4ai commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

please update & resolve conflicts

@hidb4ai hidb4ai removed the 1.4.0 v1.4.0 release tracking label Oct 16, 2026
@AlexStocks AlexStocks added the 1.4.1 v1.4.1 release tracking label Oct 17, 2026
@knqiufan
knqiufan force-pushed the codex/experiment-image-source branch 2 times, most recently from eca666c to 072bcfc Compare October 21, 2026 01:49
@knqiufan
knqiufan marked this pull request as ready for review October 21, 2026 01:49
@knqiufan
knqiufan force-pushed the codex/experiment-image-source branch from 072bcfc to 3fde6d5 Compare October 21, 2026 15:00
ExperimentalImageTaskPage,
ExperimentalListImageTasksRequest,
)
from powercontext.paths import powercontext_data_dir

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] powercontext[builtin] 安装后无法导入内置运行时

builtin/runtime/application.py:151 在模块层导入 powercontext.builtin.images.service,该文件第 64 行导入 powercontext.paths,而 src/powercontext/paths.py 在模块层执行 from platformdirs import user_data_path。pyproject.toml 的 builtin extra 没有声明 platformdirs(只有 server 与 cli 声明),images extra 又只组合 powercontext[builtin] + pillow。

后果:pip install "powercontext[builtin]" 之后任何 import powercontext.builtin.runtime 都抛 ModuleNotFoundError: No module named 'platformdirs',powercontext[images] 同样不可用。

证据:

  • CI 同签名失败:isolated-installations (3.11, builtin) job 119297345299 与 isolated-installations (3.14, builtin) job 119297345844,栈为 ci_role_smoke.py:31 → builtin/runtime/__init__.py → builtin/runtime/application.py:151 → builtin/images/service.py:64 → paths.py:22 → ModuleNotFoundError: platformdirs。
  • 同一 Main 工作流在 base 2d11abd6 上 success(run 39602208746),即该 job 在 base 是绿的。
  • 静态核对模块级导入闭包:base 2d11abd6 的 powercontext.builtin.runtime 闭包不含 powercontext.paths,本 head 含(唯一新增路径就是 builtin/images/service.py)。所以缺失依赖是本 PR 引入,不是既有形态。

修复方向(二选一):

  1. 在 builtin extra 增加 platformdirs>=4,<5(与 server/cli 一致);
  2. 或者让 images 不依赖 paths:把数据目录通过 ImageConfig/BuiltinConfig 注入,或在 service.py 内延迟导入。

无论选哪条,请保留 isolated-installations (x, builtin) 的覆盖——它正是这次唯一抓到该问题的门禁;tests (x, unit) 与 quality 都因为工作区已装 server 依赖而看不到它。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复核新 head 9abb3628:本 P1 已闭合。pyproject.toml 的 [project.optional-dependencies].builtin 现声明 platformdirs>=4,<5(images 仍为 powercontext[builtin] + pillow),src/powercontext/builtin/images/service.py:64 的模块级 from powercontext.paths import powercontext_data_dir 保持不变,但 builtin 隔离安装不再缺该依赖,原先的 ModuleNotFoundError: platformdirs 形态不可达。

补充两点范围说明:head 9abb3628 的 isolated-installations (3.11/3.14, builtin) 在本轮复核时仍为 pending,本条结论来自依赖声明与导入闭包的静态核对,不等同于 CI 已绿;tests/e2e/test_image_* 未在本机执行(工作区 venv 缺 pillow),该 PR 的图片链路行为证据仍应以上述 CI job 为准。

@knqiufan
knqiufan force-pushed the codex/experiment-image-source branch from 3fde6d5 to bada505 Compare October 22, 2026 10:06
@hidb4ai hidb4ai added 1.5.0 v1.5.0 release tracking and removed 1.4.1 v1.4.1 release tracking labels Oct 23, 2026

@hireview001 hireview001 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Resolve merge conflicts with the target branch.

@knqiufan
knqiufan force-pushed the codex/experiment-image-source branch from 85d0898 to 9abb362 Compare October 25, 2026 05:53
@knqiufan knqiufan changed the title feat(sources): add opt-in image evidence and Codex capture feat(media): add opt-in image evidence with Artifact processing Oct 25, 2026

@hireview001 hireview001 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Resolve merge conflicts with the target branch.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.5.0 v1.5.0 release tracking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants