Skip to content

feat(auto_resource): interpret image resources into daily notes - #500

Open
wang-qisen wants to merge 11 commits into
agentscope-ai:mainfrom
wang-qisen:feat/resource-image-caption
Open

feat(auto_resource): interpret image resources into daily notes#500
wang-qisen wants to merge 11 commits into
agentscope-ai:mainfrom
wang-qisen:feat/resource-image-caption

Conversation

@wang-qisen

@wang-qisen wang-qisen commented Aug 27, 2026

Copy link
Copy Markdown

Summary

Before this PR, Auto Resource interpreted only text-based resources (md/txt/json/jsonl/csv/yaml/html). This PR extends the same resource pipeline to images: an image placed under resource/ is captioned by a vision model into a source-linked daily note, so its visible content becomes retrievable through ReMe's existing text-search and chat flow while the original file remains untouched.

  • One public resource entry point: the watcher and manual job expose only auto_resource_step. AutoResourceStep groups each batch by processor, dispatches images to AutoImageResourceStep, and leaves AutoTextResourceStep as the unique final fallback. A future modality can register another matcher before that fallback without changing router code.
  • Real processor Steps: text and image processors retain independent YAML prompts, MRO prompt loading, PromptHandler, standard Ref dependency resolution, invocation-scoped RuntimeContext, and normal run_job() / dispatch_steps() lifecycle behavior.
  • Safe result aggregation: the router snapshots each child Step result immediately, restores the original changes, and emits one final response/hook in the caller's original item order, avoiding shared-Response overwrites between modality batches.
  • Shared note lifecycle: the unregistered BaseAutoResourceStep owns path/date validation, exact source_resource ownership, collision-free initial paths, conservative deletion, frontmatter repair, rename/deduplication, modified-state reporting, day-index refresh, per-resource failure isolation, and common response metadata.
  • Workspace containment: relative traversal, outside absolute paths, paths outside the configured resource root, and escaping symlinks are rejected through one helper reused by text and image processors. Deleted paths receive lexical containment checks even when the source no longer exists.
  • Image interpretation: one captioning operation per image. Structured output (name/description/caption) is attempted first and may be followed by one plain-call retry when unavailable or unusable. If both results are blank, the item fails before any note is created or overwritten.
  • Request-only preprocessing: images over 2048 px are downscaled in memory with LANCZOS/JPEG quality 85; BMP/TIFF/HEIC are decoded and re-encoded to provider-friendly PNG/JPEG bytes for the request only. Source files are never rewritten.
  • Dependency boundary: Pillow is lazy-loaded from the core extra. pillow-heif remains in the optional image-heif extra and is imported/registered only for .heic processing. Missing dependencies, decode failures, and resize/transcode failures produce distinct per-resource errors.
  • Filename context: filename and stem are passed to both image prompts only as weak naming/disambiguation hints. The prompt requires visible content to win on conflict and forbids treating the filename as an image fact.
  • Watcher compatibility: default image suffixes are watched case-insensitively for both initial scans and live changes, including common uppercase camera extensions such as .JPG, .PNG, and .HEIC.
  • Model configuration: an optional as_llm.vision instance is resolved through the standard Ref mechanism and falls back to as_llm.default, so a multimodal default model requires no additional configuration.

Related issue

Partially addresses #345 (feat(multimodal): build multimodal capabilities for ReMe). This PR covers image-resource caption notes under resource/, keeping the implementation local-first and file-native. Audio/video ingestion, pluggable preprocessing/embedding pipelines, and native cross-modal retrieval remain follow-up work.

Contract and data impact

  • No public configuration, schema, CLI, endpoint, streaming, or workspace-layout contract changes
  • No user-owned memory files are deleted or rewritten
  • Derived indexes, catalogs, graphs, caches, and metadata remain rebuildable

The first item is intentionally unchecked:

  • resource_watch_loop.watch_suffixes gains image suffixes by default (png/jpg/jpeg/webp/gif/bmp/tiff/heic).
  • auto_resource_step gains configured image/text processor dispatch.
  • Pillow is added to core; pillow-heif is available through the optional image-heif extra.

No existing configuration key from main is removed. The temporary public auto_image job introduced earlier in this PR has been removed; manual processing uses the unified reme auto_resource interface. Legacy auto_resource_step configurations without explicit processor dispatch remain supported through unique-fallback discovery.

The .heif filename suffix remains outside this PR's scope. .heic is supported when reme-ai[image-heif] is installed.

Validation

  • Focused tests pass

  • Unit tests pass, or omitted tests are explained below

  • pre-commit run --all-files passes, or omitted checks are explained below

  • Frontend checks were run when reme_studio/ changed

  • Focused resource suite: 137 passed across image processing, existing text-resource behavior, and review regressions.

  • Full unit suite: pytest tests/unit1215 passed.

  • Pre-commit: every hook passed for all 25 files changed by this PR (AST/YAML/TOML/private-key/whitespace/trailing-commas/Black/Flake8/Pylint/Pyroma).

  • Frontend checks: N/A — no reme_studio/ changes.

  • Credentialed external-provider integration tests were not run. Both end-to-end validations used a deterministic localhost OpenAI-compatible model boundary while exercising the real ReMe application, job, CLI, file, indexing, search, chat, SSE, and session layers.

End-to-end validation:

  1. Mixed-resource one-shot CLI: reme start job=auto_resource processed one interleaved batch of 12 resources (7 images across PNG/JPEG/WebP/BMP/TIFF/HEIC and 5 text files including Chinese Markdown). All 12 results succeeded in original input order, all notes contained exact source_resource links, and all source SHA-256 hashes remained unchanged. A deliberately misleading filename (sunset-beach-vacation.png) produced a database-maintenance note based on the supplied visible-content result rather than filename facts.
  2. Service + retrieval + new chat session: a running ReMe service processed five complex fashion/product images through the public reme auto_resource CLI, indexed the generated notes, and served fresh /chat sessions. The semantic query 一件棕色外套 retrieved a caption note whose source filename was the non-semantic catalog_frame_057.png, then the agent followed search → read and returned the coat's length, double-breasted six-button front, self belt, pocket design, daily-note path, and source-image path. Four positive turns and an independent negative-control session passed; the negative control called only search, did not call read, and invented no paths.

Checklist

  • I reviewed the diff for unrelated changes and sensitive data
  • Tests cover intentional behavior changes
  • Defaults, schemas, and concise documentation were updated together when required
  • Long-lived clients, tasks, services, and executors follow the application lifecycle

Screenshots or additional notes

Example caption note:

---
description: 一件棕色外套的无文字模特图:深巧克力棕色长款羊毛大衣采用宽翻领、双排六粒深色角扣、自系同料腰带和两侧斜插袋。
kind: image
media_type: image/png
name: dark-brown-belted-wool-overcoat
source_resource: '[[resource/2026-09-01/catalog_frame_057.png]]'
---

![[resource/2026-09-01/catalog_frame_057.png]]

## Caption

一张没有标题、标签、Logo 或其他可见文字的全身时装照片。成年女性模特站在暖灰色极简摄影棚内,身穿深巧克力棕色、长度到小腿中部的羊毛外套。外套有宽大的缺口翻领,前襟为双排六粒深色角扣,腰间系同面料腰带,两侧各有一个斜插式嵌线口袋……

@CLAassistant

CLAassistant commented Aug 27, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@wang-qisen
wang-qisen force-pushed the feat/resource-image-caption branch from f3c0d24 to 3ddd06c Compare August 27, 2026 09:38
@wang-qisen wang-qisen changed the title Feat/resource image caption feat(auto_image): interpret image resources into daily notes Aug 28, 2026
@jinliyl
jinliyl requested review from imrewce and jinliyl August 31, 2026 02:20
@jinliyl

jinliyl commented Aug 31, 2026

Copy link
Copy Markdown
Member

感谢补充图片资源自动理解能力。通过 VLM 将图片转换成带 source_resource 的 daily caption note,可以让现有文本检索体系直接检索图片内容,同时不修改用户原图,整体也符合 ReMe local-first、file-native 的原则。

看完实现后,有两个结构和依赖方面的建议,想请考虑一下。

1. 建议重构 auto_resource / auto_image 的分发方式

目前 watcher 中依次执行:

- backend: auto_resource_step
- backend: auto_image_step

每次资源变化都会进入两个 Step,其中一个实际处理,另一个返回 skipped。除了职责重复外,顺序 Step 共享同一个 RuntimeContextResponse:例如文本资源先由 auto_resource_step 成功处理,随后 auto_image_step 又写入 non_image_file/skipped,最终 response、metadata 或宿主 hook 看到的结果可能被后一个 Step 覆盖。批次同时包含文本和图片时,结果聚合也会更难保持清晰。

建议 watcher 对外只保留一个统一入口:

- backend: auto_resource_step

内部根据资源类型选择具体 Step:

AutoResourceStep              # 统一路由与最终结果聚合
├── AutoTextResourceStep      # 文本资源处理
└── AutoImageResourceStep     # 图片资源处理

这里建议 text/image processor 仍然是真正的 Step,而不是普通 handler,以继续保留:

  • 根据类及 MRO 自动加载对应 YAML prompt;
  • PromptHandlerprompt_format() 和语言选择;
  • Ref 依赖解析;
  • invocation-scoped RuntimeContext
  • run_job() / dispatch_steps()
  • Step 输入输出映射和生命周期。

公共资源笔记逻辑可以抽到一个不注册的 BaseAutoResourceStep,集中维护路径和日期解析、source_resource 查找、删除关联、frontmatter、改名与冲突处理、day index 刷新及批次结果结构。文本和图片 Step 各自实现 interpretation/upsert,并保留独立 YAML prompt。

Router 可以按 suffix 对 changes 分组,只 dispatch 对应的 Step。由于 dispatch_steps() 会继续传递当前 RuntimeContext,router 应及时保存每个子 Step 的结果,最后统一聚合 response,避免不同批次再次互相覆盖。

如果本 PR 暂时不希望进行较大重构,至少建议增加一个轻量 router,避免 watcher 对同一批 changes 无条件顺序执行两个 processor。

2. Pillow 建议 lazy import;Pillow 放 core,pillow-heif 放 extra

当前 auto_image.py 在模块顶层直接 from PIL import Image,而 reme.steps.evolve.__init__ 会导入 auto_image。Pillow 缺失或安装异常时,即使用户不使用图片能力,也可能导致整个 evolve Step 包导入失败。建议在真正探测、缩放或转码时再加载 Pillow:

def _load_pillow():
    try:
        from PIL import Image
    except ImportError as exc:
        raise RuntimeError(
            "Image preprocessing requires Pillow; install the ReMe core dependencies.",
        ) from exc
    return Image

HEIC/HEIF 插件也建议只在实际处理对应格式时加载和注册:

def _register_heif_opener() -> None:
    try:
        from pillow_heif import register_heif_opener
    except ImportError as exc:
        raise RuntimeError(
            'HEIC/HEIF support requires: pip install "reme[image-heif]"',
        ) from exc
    register_heif_opener()

另外,当前 _normalize_image_bytes() 用同一个 None 同时表示“不需要转换”和“解码失败”。建议至少通过返回状态、日志或 metadata 区分:

  • 图片不需要转换;
  • Pillow 不可用;
  • pillow-heif 未安装;
  • 图片损坏或无法解码;
  • 缩放/转码失败。

这样 provider 拒绝图片时,用户能看到明确原因,而不只是模型调用错误。

依赖建议采用:

  • Pillow 放 core:提供通用尺寸检查、2048px 缩放、BMP/TIFF 转码和颜色模式处理,是默认图片能力的基础,体积也可接受。
  • pillow-heif 放可选 extra:只服务 HEIC/HEIF,并携带原生编解码组件,没有必要让所有 core 用户强制安装。

例如:

[project.optional-dependencies]
core = [
    "pillow>=10.0.0",
]

image-heif = [
    "pillow-heif>=0.13.0",
]

pillow-heif 自身依赖 Pillow,因此 extra 中可以不重复声明。文档可注明:默认 core 支持常见图片缩放和转码;HEIC/HEIF 需要安装 reme[image-heif]

即使 Pillow 已放入 core,我仍建议 lazy import:core 表达默认安装能力,lazy import 用于降低模块导入耦合、保证注册及非图片路径更稳健,两者并不冲突。

再次感谢这个 PR。图片 caption note、原图不变、结构化输出 fallback、单条失败隔离以及完整测试覆盖这些设计都很好。上述建议主要希望统一 resource 入口、避免共享 Response 被 skipped 结果覆盖,并让原生图像依赖边界更清楚。

@jinliyl

jinliyl commented Aug 31, 2026

Copy link
Copy Markdown
Member

再补充一点:当前 prompt 已传入 {file_path},模型能看到图片文件名。建议进一步把 filenamestem 明确作为上下文传入,这对语义化文件名(如 q3-budget-dashboard.png)生成更准确的 namedescription 会有帮助。

同时建议在 prompt 中注明:文件名只能作为命名和消歧的弱提示,不能当作图片中的可见事实;若文件名与图片内容冲突,应以图片实际内容为准。caption 仍应主要依据可见内容,避免文件名引入幻觉。

@wang-qisen
wang-qisen force-pushed the feat/resource-image-caption branch from 3ddd06c to fe4098c Compare September 1, 2026 04:04
@wang-qisen

Copy link
Copy Markdown
Author

Thank you @jinliyl for the detailed review. The latest commits address the suggestions as follows.

Review item 1 — Unified auto-resource routing: ADDRESSED

  • The watcher now exposes only auto_resource_step; it no longer invokes the text and image processors sequentially.
  • AutoResourceStep groups changes by processor and dispatches only the matching processor.
  • AutoTextResourceStep and AutoImageResourceStep remain real Steps with independent YAML prompts, MRO-based prompt loading, PromptHandler, standard Ref resolution, invocation-scoped RuntimeContext, run_job() / dispatch_steps(), and the normal Step lifecycle.
  • The unregistered BaseAutoResourceStep now owns the shared path/date handling, source_resource lookup and deletion, frontmatter, rename/collision handling, day-index refresh, per-resource failure isolation, and batch result construction.
  • The router snapshots each child Step result immediately, restores the original changes, and produces one final aggregated response/hook. This prevents later processors from overwriting earlier results through the shared RuntimeContext and Response.
  • AutoImageResourceStep has an explicit suffix matcher. AutoTextResourceStep is the unique final fallback, preserving the previous auto_resource behavior for direct calls, custom suffixes, and other non-image resources.
  • A new modality can be inserted before the text fallback by registering another processor and matcher, without changing the router implementation.

Review item 2 — Pillow and HEIF dependency boundaries: ADDRESSED

  • Pillow is now a lazy-loaded core dependency, so importing and registering evolve Steps does not require importing Pillow on non-image paths.
  • pillow-heif is provided through the optional image-heif extra and is loaded and registered only when processing .heic.
  • “No conversion required” remains a normal preprocessing result. Missing Pillow, missing HEIC support, image decode failures, and resize/transcode failures are surfaced as distinct per-resource errors.
  • Image resizing and transcoding remain request-only operations; the original resource file is never modified.

Additional suggestion — Filename/stem prompt context: ADDRESSED

  • filename and stem are now passed explicitly to both the English and Chinese image prompts.
  • The prompts define them as weak naming and disambiguation hints, not as visible facts.
  • When the filename conflicts with visible image content, the image content must take precedence, and the caption must remain grounded in what is visible.

Backward compatibility: ADDRESSED

  • Legacy auto_resource_step configurations without explicit processor dispatch discover and use the unique registered fallback.
  • The direct auto_image job remains available for manual processing, while the default watcher uses only the unified router.

Scope note — .heif filename suffix: DEFERRED

  • This PR currently supports .heic through the optional reme-ai[image-heif] extra.
  • Adding .heif as another watched filename suffix is intentionally deferred to a broader image-format detection follow-up.

Validation

  • Focused resource tests: 98 passed
  • Full unit suite: 1176 passed
  • Pre-commit checks for every file changed by this PR: passed

Thank you again for the structural and dependency-boundary suggestions. Please take another look when convenient.

@imrewce

imrewce commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

LGTM in terms of the overall structure and basic test cases, to be further decided by @jinliyl

@jinliyl

jinliyl commented Sep 1, 2026

Copy link
Copy Markdown
Member

感谢根据上一轮建议完成统一 router、图片/文本 processor 拆分、Pillow/HEIF lazy import,以及 filename/stem 弱提示约束。重新 checkout 最新提交并结合实际运行路径又检查了一轮:上一轮提到的问题已经处理,但目前还发现 4 个边界问题,其中前两个涉及 workspace/file-native 安全,建议合并前修复。

1. [Blocking] resource 相对路径可通过 .. 或 symlink 逃出 workspace

BaseAutoResourceStep._handle_change() 目前只对绝对路径调用 to_workspace_relative();相对路径直接进入 _parse_resource_path()。这个解析函数只检查前两个 lexical path segment 是否为 resource/<date>,不会拒绝后续的 ..

之后 text/image processor 都直接拼接和读取:

abs_path = self.workspace_path / file_path

因此下面的输入会被接受:

resource/2026-01-01/../../../outside.png

本地复现结果:

_parse_resource_path(...) == ("2026-01-01", "../../../outside.png")
resolved path             == workspace 外的 outside.png
_read_image(...)          == 成功

如果该路径是有效图片,内容随后会进入 VLM 请求;text fallback 也有同类读取路径。workspace 内指向外部文件的 symlink 同样可以绕过 lexical prefix 检查。由于 auto_resource 和新增 auto_image 都是 service-exposed job,这不仅是异常输入问题,也可能成为 agent 读取并向模型发送 workspace 外文件的边界绕过。

建议:

  • added/modified 在 stat/read 前统一调用现有的 resolve_path()
  • 对 resolve 后的实际路径再次确认其位于 resolve 后的 resource_dir 内,而不只是 workspace 内;
  • 拒绝 ..、workspace 外绝对路径以及逃逸 symlink;
  • deleted 场景中文件可能已不存在,仍应做规范化 lexical containment 校验;
  • 增加 traversal、absolute outside path、symlink outside 三类测试;
  • text/image 必须复用同一个 base helper,避免其中一个 processor 修好、另一个仍可绕过。

2. [Blocking] fallback note lookup 会覆盖或删除无关用户笔记

BaseAutoResourceStep._find_resource_note() 先匹配 source_resource,这是正确的;但如果没有匹配项,第二轮只要 note 的路径等于:

daily/<date>/<source_stem>.md

就会把它当成该资源的旧版 note,即使它已经有一个不同的、明确的 source_resource

图片 upsert 随后会先把 caption 写到这个 fallback path,再进行 name collision/rename。因此当前的 collision handling 只保护“模型生成的最终 name 冲突”,没有保护“初始 fallback path 冲突”。

本地复现场景:

daily/2026-01-01/img.md
  source_resource: [[conversation/keep-me]]
  body: USER CONTENT MUST SURVIVE

resource/2026-01-01/img.png

执行后结果:

response.success = True
原 USER CONTENT MUST SURVIVE 不再存在
daily/2026-01-01/img.md 被覆盖后移动为 new-caption.md

deleted 流程也会通过相同 fallback lookup 删除同名但不属于该 resource 的 note。这与 local-first/file-native 原则冲突:派生资源卡不应覆盖或删除用户已有 memory。

建议:

  • fallback candidate 如果存在非空、且与当前资源不同的 source_resource,必须视为冲突,不能认领;
  • 新建 note 时应在第一次 write 之前就分配 collision-free path,不能先 overwrite fallback 再 rename;
  • deletion 最好只删除明确匹配当前 source_resource 的 note;
  • 若必须兼容非常旧、没有 source_resource 的资源卡,建议采用保守识别或显式 migration,不要默认认领任意同 stem 用户笔记;
  • 增加“fallback path 已存在且属于其他 source”“无 source_resource 的普通笔记”“删除同名资源不能删除无关 note”的测试。

3. plain fallback 也为空时仍写空 caption,并报告成功

_caption_with_retry() 会在 structured 输出为空时调用 plain fallback,但返回 plain 解析结果后没有再次检查 caption / description 是否可用。

后续 _handle_upsert() 无条件构造并写入:

![[resource/.../img.png]]

## Caption

本地使用 structured {}、plain "" 复现:

response.success = True
创建了只有 embed 和空 Caption 标题的 note

如果这是 modified image,还会把先前有效 caption 覆盖为空。目前测试只覆盖“structured 为空、plain 成功”,没有覆盖“两次都为空”。

建议 plain fallback 后再次验证:

parsed = _parse_caption_json(...)
if not parsed["caption"] and not parsed["description"]:
    raise RuntimeError("Vision model returned no usable caption")

这样会由已有 per-resource isolation 转换成明确失败,并且不会写文件。

4. 默认 watcher 大小写敏感,会漏掉常见的 .JPG / .PNG / .HEIC

router 的 matches_change() 会 lower suffix,但资源在到达 router 前已经经过 watch rule。当前 _match_rule() 使用大小写敏感的:

p.name.endswith("." + suffix)

默认配置又只列出小写后缀。实测:

IMG.PNG -> False
IMG.png -> True

相机和手机生成大写 .JPG.HEIC 很常见,这些文件会被 watcher 静默忽略。

建议 watch rule 两侧统一 casefold() 或比较 p.suffix.lower(),并增加 initial scan 和 live watch 对大写图片扩展名的测试。这样也能统一现有文本扩展名的行为。


可以顺便精简的代码

下面这些不是 blocking,但在修复上述问题时可以减少重复和后续维护成本。

A. 删除 AutoImageResourceStep 的重复异常捕获

AutoImageResourceStep._handle_change() 后半段再次 except Exception 并组装 failure result;但 BaseAutoResourceStep.execute() 已经通过 _failed_change_result() 实现逐资源异常隔离、metadata 和 logging。

图片 Step 可以只保留 direct call 时“非图片返回 skipped”的兼容分支,然后:

return await super()._handle_change(file_path, raw_change)

让异常交给 base execute 统一处理。这样可以删除约 20–30 行,并避免 text/image 两套 failure metadata 和错误文案发生漂移。如果确实需要图片专属错误前缀,可以给 base 增加一个很小的 error-message hook,而不是重复整个 catch/result 逻辑。

B. 抽取 text/image 共有的 note finalize 流程

AutoTextResourceStep._handle_upsert()AutoImageResourceStep._handle_upsert() 仍重复以下生命周期:

  1. 查找已有 note;
  2. 保存 before path/bytes;
  3. 写入后重新查找;
  4. source_resource
  5. 根据 frontmatter name 重命名和去重;
  6. 计算 modified;
  7. refresh day index;
  8. 组装公共 response metadata。

建议不要再引入一个很重的 template-method 框架,只抽两个窄 helper,例如:

async def _resolve_written_note(...)
async def _finalize_resource_note(...)

text processor 只负责读取文本和调用 agent;image processor 只负责读取图片、调用 VLM 和生成 caption body;note ownership、collision、rename、index 与公共 metadata 都由 base 维护。这样也能保证上面的 fallback ownership 修复只写一次。

C. 如果没有必须保留的公开接口,可以移除独立 auto_image job

统一 auto_resource router 已经可以手动处理图片,独立 auto_image job 又暴露了一套几乎相同的 parameters/schema,并促使 image processor 保留 direct non-image compatibility 分支。

如果该 job 尚未成为需要兼容的公开接口,可以统一为:

reme auto_resource changes=...

删除 jobs.auto_image 及相应特殊逻辑/测试。若产品上确实需要一个明确的图片工具名,则保留即可,不建议仅为减少代码破坏接口。

D. 参数化图片测试,减少重复 workspace boilerplate

tests/unit/test_auto_image_steps.py 中大量用例重复:

  • TemporaryDirectory + temp_chdir
  • 创建 app context/file store;
  • 安装 file jobs;
  • try/finally close
  • asyncio.run(run())

可以抽一个 async fixture/context manager,并对以下场景参数化:

  • PNG/JPEG/WebP/BMP/TIFF;
  • decode/resize/dependency errors;
  • structured/plain fallback;
  • added/modified/deleted。

这样能减少较多测试样板,同时保留现有覆盖。

建议处理顺序

  1. 先修 workspace containment;
  2. 再修 fallback note ownership 和首次写入 collision;
  3. 补 blank caption 与 uppercase watcher;
  4. 最后做重复异常层和 finalize helper 的小范围精简。

我在最新 feat/resource-image-caption 上完成以上复现;worktree 保持干净,没有修改代码。上一轮提出的统一 router、Response 聚合、lazy dependencies 和 filename weak hint 均确认已处理。

@wang-qisen wang-qisen changed the title feat(auto_image): interpret image resources into daily notes feat(auto_resource): interpret image resources into daily notes Sep 1, 2026
@wang-qisen

Copy link
Copy Markdown
Author

Thank you @jinliyl for the thorough second review. I have pushed a follow-up that addresses all four edge cases and the cleanup suggestions.

1. Resource path containment — addressed

  • Added one shared base helper for both text and image processors.
  • Relative traversal, workspace-external absolute paths, paths outside the configured resource root, and escaping symlinks are rejected before stat/read/model calls.
  • Deleted paths are still checked lexically when the source no longer exists.
  • Regression coverage includes direct image processing and the unified router, plus the text fallback path.

2. Note ownership and initial-path collisions — addressed

  • Existing notes are now owned only through an exact source_resource match; the same-stem fallback has been removed.
  • A collision-free path is allocated before the first write, so an unrelated note cannot be temporarily overwritten and then renamed.
  • Final note resolution refuses to claim a note with another explicit owner.
  • Deletion removes only the exact source-owned note and preserves same-stem notes with another source or no source_resource.

3. Blank structured + plain caption fallback — addressed

  • The plain fallback is validated again after parsing.
  • If both caption and description are blank, the item fails before writing, so a new empty note is not created and an existing caption is not overwritten.

4. Uppercase watcher suffixes — addressed

  • Watch-rule matching now case-folds both the filename and configured suffix.
  • Tests cover uppercase image extensions in both the initial scan and live watcher path.

Cleanup suggestions — addressed

  • Removed the duplicate image-level _handle_change() exception layer; per-resource isolation now lives only in BaseAutoResourceStep.
  • Moved prepare/resolve/finalize ownership, collision, rename, modified-state, day-index refresh, and common metadata into narrow shared base helpers used by both processors.
  • Removed the public auto_image job and its direct non-image compatibility branch. Manual compensation now uses the unified reme auto_resource entry point.
  • Reworked the image tests around a scoped async fixture and parameterized format, dependency, decode, resize, conversion, structured/plain fallback, and added/modified/deleted cases. The resize-failure case explicitly injects a thumbnail() failure.

The previous unified router, immediate child-result snapshots, standard named-model Ref resolution, lazy Pillow/HEIF imports, and filename/stem weak-hint behavior remain covered.

Validation

  • Focused resource suite: 137 passed
  • Full unit suite: 1215 passed
  • Pre-commit for every PR-changed file: passed
  • Mixed-resource CLI validation: 12/12 resources succeeded
  • Service/chat validation: 5/5 image notes were indexed; a fresh chat session retrieved the non-semantic catalog_frame_057.png from the query 一件棕色外套 through search → read; the independent negative control found nothing and did not invent a path

The branch has also been merged with the latest upstream/main and updated with a regular fast-forward push. Please take another look when convenient. Thank you again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants