feat(auto_resource): interpret image resources into daily notes - #500
feat(auto_resource): interpret image resources into daily notes#500wang-qisen wants to merge 11 commits into
Conversation
f3c0d24 to
3ddd06c
Compare
|
感谢补充图片资源自动理解能力。通过 VLM 将图片转换成带 看完实现后,有两个结构和依赖方面的建议,想请考虑一下。 1. 建议重构 auto_resource / auto_image 的分发方式目前 watcher 中依次执行: - backend: auto_resource_step
- backend: auto_image_step每次资源变化都会进入两个 Step,其中一个实际处理,另一个返回 skipped。除了职责重复外,顺序 Step 共享同一个 建议 watcher 对外只保留一个统一入口: - backend: auto_resource_step内部根据资源类型选择具体 Step: 这里建议 text/image processor 仍然是真正的 Step,而不是普通 handler,以继续保留:
公共资源笔记逻辑可以抽到一个不注册的 Router 可以按 suffix 对 changes 分组,只 dispatch 对应的 Step。由于 如果本 PR 暂时不希望进行较大重构,至少建议增加一个轻量 router,避免 watcher 对同一批 changes 无条件顺序执行两个 processor。 2. Pillow 建议 lazy import;Pillow 放 core,pillow-heif 放 extra当前 def _load_pillow():
try:
from PIL import Image
except ImportError as exc:
raise RuntimeError(
"Image preprocessing requires Pillow; install the ReMe core dependencies.",
) from exc
return ImageHEIC/HEIF 插件也建议只在实际处理对应格式时加载和注册: def _register_heif_opener() -> None:
try:
from pillow_heif import register_heif_opener
except ImportError as exc:
raise RuntimeError(
'HEIC/HEIF support requires: pip install "reme[image-heif]"',
) from exc
register_heif_opener()另外,当前
这样 provider 拒绝图片时,用户能看到明确原因,而不只是模型调用错误。 依赖建议采用:
例如: [project.optional-dependencies]
core = [
"pillow>=10.0.0",
]
image-heif = [
"pillow-heif>=0.13.0",
]
即使 Pillow 已放入 core,我仍建议 lazy import:core 表达默认安装能力,lazy import 用于降低模块导入耦合、保证注册及非图片路径更稳健,两者并不冲突。 再次感谢这个 PR。图片 caption note、原图不变、结构化输出 fallback、单条失败隔离以及完整测试覆盖这些设计都很好。上述建议主要希望统一 resource 入口、避免共享 Response 被 skipped 结果覆盖,并让原生图像依赖边界更清楚。 |
|
再补充一点:当前 prompt 已传入 同时建议在 prompt 中注明:文件名只能作为命名和消歧的弱提示,不能当作图片中的可见事实;若文件名与图片内容冲突,应以图片实际内容为准。 |
3ddd06c to
fe4098c
Compare
|
Thank you @jinliyl for the detailed review. The latest commits address the suggestions as follows. Review item 1 — Unified auto-resource routing: ADDRESSED
Review item 2 — Pillow and HEIF dependency boundaries: ADDRESSED
Additional suggestion — Filename/stem prompt context: ADDRESSED
Backward compatibility: ADDRESSED
Scope note —
Validation
Thank you again for the structural and dependency-boundary suggestions. Please take another look when convenient. |
|
LGTM in terms of the overall structure and basic test cases, to be further decided by @jinliyl |
|
感谢根据上一轮建议完成统一 router、图片/文本 processor 拆分、Pillow/HEIF lazy import,以及 filename/stem 弱提示约束。重新 checkout 最新提交并结合实际运行路径又检查了一轮:上一轮提到的问题已经处理,但目前还发现 4 个边界问题,其中前两个涉及 workspace/file-native 安全,建议合并前修复。 1. [Blocking] resource 相对路径可通过
|
|
Thank you @jinliyl for the thorough second review. I have pushed a follow-up that addresses all four edge cases and the cleanup suggestions. 1. Resource path containment — addressed
2. Note ownership and initial-path collisions — addressed
3. Blank structured + plain caption fallback — addressed
4. Uppercase watcher suffixes — addressed
Cleanup suggestions — addressed
The previous unified router, immediate child-result snapshots, standard named-model Validation
The branch has also been merged with the latest |
Summary
Before this PR, Auto Resource interpreted only text-based resources (
md/txt/json/jsonl/csv/yaml/html). This PR extends the same resource pipeline to images: an image placed underresource/is captioned by a vision model into a source-linked daily note, so its visible content becomes retrievable through ReMe's existing text-search and chat flow while the original file remains untouched.auto_resource_step.AutoResourceStepgroups each batch by processor, dispatches images toAutoImageResourceStep, and leavesAutoTextResourceStepas the unique final fallback. A future modality can register another matcher before that fallback without changing router code.PromptHandler, standardRefdependency resolution, invocation-scopedRuntimeContext, and normalrun_job()/dispatch_steps()lifecycle behavior.changes, and emits one final response/hook in the caller's original item order, avoiding shared-Responseoverwrites between modality batches.BaseAutoResourceStepowns path/date validation, exactsource_resourceownership, collision-free initial paths, conservative deletion, frontmatter repair, rename/deduplication, modified-state reporting, day-index refresh, per-resource failure isolation, and common response metadata.name/description/caption) is attempted first and may be followed by one plain-call retry when unavailable or unusable. If both results are blank, the item fails before any note is created or overwritten.coreextra.pillow-heifremains in the optionalimage-heifextra and is imported/registered only for.heicprocessing. Missing dependencies, decode failures, and resize/transcode failures produce distinct per-resource errors.filenameandstemare passed to both image prompts only as weak naming/disambiguation hints. The prompt requires visible content to win on conflict and forbids treating the filename as an image fact..JPG,.PNG, and.HEIC.as_llm.visioninstance is resolved through the standardRefmechanism and falls back toas_llm.default, so a multimodal default model requires no additional configuration.Related issue
Partially addresses #345 (
feat(multimodal): build multimodal capabilities for ReMe). This PR covers image-resource caption notes underresource/, keeping the implementation local-first and file-native. Audio/video ingestion, pluggable preprocessing/embedding pipelines, and native cross-modal retrieval remain follow-up work.Contract and data impact
The first item is intentionally unchecked:
resource_watch_loop.watch_suffixesgains image suffixes by default (png/jpg/jpeg/webp/gif/bmp/tiff/heic).auto_resource_stepgains configured image/text processor dispatch.core;pillow-heifis available through the optionalimage-heifextra.No existing configuration key from
mainis removed. The temporary publicauto_imagejob introduced earlier in this PR has been removed; manual processing uses the unifiedreme auto_resourceinterface. Legacyauto_resource_stepconfigurations without explicit processor dispatch remain supported through unique-fallback discovery.The
.heiffilename suffix remains outside this PR's scope..heicis supported whenreme-ai[image-heif]is installed.Validation
Focused tests pass
Unit tests pass, or omitted tests are explained below
pre-commit run --all-filespasses, or omitted checks are explained belowFrontend checks were run when
reme_studio/changedFocused resource suite:
137 passedacross image processing, existing text-resource behavior, and review regressions.Full unit suite:
pytest tests/unit→1215 passed.Pre-commit: every hook passed for all 25 files changed by this PR (AST/YAML/TOML/private-key/whitespace/trailing-commas/Black/Flake8/Pylint/Pyroma).
Frontend checks: N/A — no
reme_studio/changes.Credentialed external-provider integration tests were not run. Both end-to-end validations used a deterministic localhost OpenAI-compatible model boundary while exercising the real ReMe application, job, CLI, file, indexing, search, chat, SSE, and session layers.
End-to-end validation:
reme start job=auto_resourceprocessed one interleaved batch of 12 resources (7 images across PNG/JPEG/WebP/BMP/TIFF/HEIC and 5 text files including Chinese Markdown). All 12 results succeeded in original input order, all notes contained exactsource_resourcelinks, and all source SHA-256 hashes remained unchanged. A deliberately misleading filename (sunset-beach-vacation.png) produced a database-maintenance note based on the supplied visible-content result rather than filename facts.reme auto_resourceCLI, indexed the generated notes, and served fresh/chatsessions. The semantic query一件棕色外套retrieved a caption note whose source filename was the non-semanticcatalog_frame_057.png, then the agent followedsearch → readand returned the coat's length, double-breasted six-button front, self belt, pocket design, daily-note path, and source-image path. Four positive turns and an independent negative-control session passed; the negative control called onlysearch, did not callread, and invented no paths.Checklist
Screenshots or additional notes
Example caption note: