Repository navigation
feat(sync): put --watch one-way directory watch (minimal tier, carved from Phase 2 sync drive) #110
Description
schemaVersion: requirement-record.v1
revision: R1
status: needs-design
priority: P2
productOwner: "@PeterGuy326"
technicalOwner: "unassigned"
userOutcome: "A user points mem at a local directory and new or changed files flow into the netdisk automatically via a one-way watch with a per-cycle import report — while the full sync drive stays a recorded Phase 2 item."
requirements:
- REQ-001
- REQ-002
- REQ-003
- REQ-004
acceptanceCriteria:
- AC-001
- AC-002
- AC-003
- AC-004
parent: null
dependencies: []
supersedes: []
lastDecisionAt: nullUser problem and observable outcome
Founder query "does mem include scanning?" surfaced a baseline expectation: users expect to point at a folder and have content flow in automatically (netdisk baseline; GOAL promises "like an ordinary netdisk"). Current state (product-verified 2026-08-29): one-time recursive import exists (mem put -r); the one-way daemon mem put <path> --watch is designed in SPEC.md:604 but unimplemented (no watch anywhere in code, no --watch flag on put); the full sync drive is explicitly Phase 2 (SPEC.md:91/1045/1140, GOAL.md:123 unchecked). No tracking issue existed — this record is it. Implementation truth belongs to Tech P9.
Observable outcome: a watched directory yields automatic one-way ingestion of new/changed files with a per-cycle import report; the full sync drive remains recorded as Phase 2 with revival preconditions.
Requirements
- REQ-001: One-way watch daemon — implement the SPEC.md:604 shape:
mem put <path> --watchwatches the designated directory and ingests new/changed files into mem; strictly local→mem one-way; flag semantics align with the SPEC design (if implementation must diverge, tech raises a SPEC-revision decision). - REQ-002: Per-cycle import report — each watch cycle emits an import report (added/unchanged/failed counts plus item locators); individual failures fail-closed with stable codes while the daemon continues; report content follows evidence vocabulary.
- REQ-003: Boundary hard constraints — no write-back to the local disk, no deletion propagation (local deletions do not remove already-ingested mem content), no bidirectional sync, conflict merging, or remote sync — all explicitly Phase 2 (SPEC.md:91/1045/1140, GOAL.md:123).
- REQ-004: Canon consolidation — this record is the formal tracking item for the scan/auto-sync family: the Phase 2 full sync drive stays deferred with recorded revival preconditions (rclone-lib route decision, adjacent to bundle v2 / incremental packages per GOAL.md:95); SPEC/GOAL/README scattered wording cross-references this record.
Acceptance criteria
- AC-001: Watch fixture — new/changed files under a watched directory are ingested; report counts correct.
- AC-002: Failure fixture — a failing file yields a stable code in the report; the daemon continues; no crash, no silent drop.
- AC-003: Boundary fixture — local deletion does not propagate; no disk write-back observable; no bidirectional behavior.
- AC-004: SPEC alignment — flag semantics match SPEC.md:604, or a tech-raised SPEC-revision decision is recorded.
Non-goals and forbidden shortcuts
- No bidirectional sync, conflict resolution, deletion propagation, or remote sync (Phase 2, rclone route).
- No mobile/share-link/preview scope creep (SPEC Phase 1 cut list holds).
- No re-indexing behavior changes beyond what ingestion already triggers.
Lifecycle, status, priority, blockers, and open decisions
- status=needs-design; priority P2; technicalOwner unassigned (assign at implementation kickoff; scheduling adjacent to bundle v2 / incremental packages, GOAL.md:95 migration-completeness cluster).
- Blockers: none.
- Open decisions: (1) daemon lifecycle (foreground vs background/service mode, restart behavior); (2) report shape and persistence (stdout only vs ledgered); (3) change detection semantics (mtime/hash) and scope of "changed"; (4) interaction with existing put path semantics (shared ingestion path vs watch-specific path) — tech adjudication.
Evidence plan
- Fixtures first (watch/failure/boundary), deterministic; daemon behavior evidenced per three-axis discipline.
- Evidence input: ops observation memo (2026-08-29) plus product-side anchor verification (SPEC.md:604/91/1045/1140, GOAL.md:123, cmds_file.go:91).
Decisions
- DEC-MEM-110-001 (2026-08-29T03:54:00Z): product ruling recorded per founder-approved intake (2026-08-29): minimal tier promoted as P2 (one-way
put --watch, carved from the full sync drive); full sync drive remains Phase 2 with revival preconditions recorded here; ops observation as evidence input; implementation truth belongs to Tech P9.
Revision history
- R1 (2026-08-29T03:54:00Z): initial record created per founder-approved ruling (scan/auto-sync family consolidated).
Activity
waterbro-8 commentedon Aug 30, 2026
裁决摘要(技术 P9)
owner 已定;#110 的 4 条 open decisions 全部出解;与 #103 的关系裁定为共用摄取内核、不共用写入面("一个内核两个 sink")。取证基线:
main @ 419a420,另含在途 PR #108。REQ-001 的"changed"口径收窄另立 R3(本条只记录不弱化既有要求的 additive 部分)。Owner
technicalOwner: "@waterbro-8"— owns design and implementation truth for this record, and is theimplementationOwnerof the bounded PR.AGENTS.mdstep 6 requires an approval from someone other than the author, and.github/CODEOWNERSnames@PeterGuy326and@Bindy-lbbas the only review authorities. Named reviewer for this issue's PRs: either code owner, requested after required CI is green and the head is stable.productOwnerstays@PeterGuy326; the R3 narrowing of REQ-001 needs product concurrence (see that comment).D-1 Daemon lifecycle — foreground-only, no supervision in this tier
mem put <path> --watchruns in the calling terminal, blocks, and exits when signalled.--daemon, no shipped unit/service file. A detached writer into a vault with no owner is an unowned process; restart policy, log routing, upgrade behaviour and single-instance arbitration are exactly the "完整本地同步盘" semantics GOAL.md:95/123 defers to Phase 2. Users who need supervision wrap the foreground mode themselves (systemdType=exec, launchd, or a composecommand:), which costs mem zero code.server/cmd/mem/main.go:25already installssignal.NotifyContext(..., os.Interrupt)at the CLI root, so Ctrl-C is free;SIGTERMis handled only incmd/memd/main.go:78and must be added to the watch loop's own context sodocker stop/systemd-stop terminates between cycles rather than mid-upload. Clean signal shutdown exits 0 — for a watcher, being stopped is the expected terminal state, and any non-zero code makes every wrapper think it failed.--interval <duration>, default30s, poll loop. No fsnotify in this tier: it is absent fromserver/go.mod, and event APIs bring per-platform edge semantics (inotify watch limits, FSEvents coalescing, Windows) that the current CI has no coverage for. A poll loop is deterministic and satisfies AC-001; an event-driven follow-up needs its own per-platform acceptance criteria.(user_id, sha256)(server/internal/file/file.go:196-201), so a replayed cycle is a 秒传 no-op, not a duplicate object.auth/plan_quota/provider_timeoutclass, exit with the SPEC.md:582 mapped code (3 / 4 / 5) instead of spinning silently. A supervisor then restarts a process that reported a real error.D-2 Report shape and persistence — per-cycle report, local log, no server ledger
--format json(SPEC.md:579 already defines this convention) makes it machine-readable, text mode keeps today's human summary. Vocabulary is closed:scanned,ingested,deduped,unchanged,local_gone,failed, plusitems[]carrying locators (local absolute path, returned file id, returned virtual path, sha256 prefix, failure code).dedupedthe report MUST carry the path the server returned, not the requested--tofolder.Service.Putanswers identical content with the pre-existing row (server/internal/file/file.go:196-201), whose virtual path can be anywhere in the workspace;uploadDirtoday printsput <local> -> <folder>before the call and so already implies a placement the server never confirmed (server/cmd/mem/cmds_file.go:161). A watch report that copies that habit would claim a location it did not cause.auth,plan_quota,provider_timeout,network,read_denied,upload_rejected,root_missing,state_corrupt. Per-item failure never aborts the cycle —uploadDiralready degrades per file and continues (server/cmd/mem/cmds_file.go:162-164), and PR feat(ingest): add mem ingest qoder for AI-agent conversation transcripts (#103) #108 established the same discipline for 409s atserver/cmd/mem/cmds_ingest.go:220-232.<state root>/watch/reports/<sha1(abs root)>.jsonl, reusing thecliStateRoot()convention (server/cmd/mem/cmds_ingest.go:104-112, the same$HOME/.memthe qoder connector uses), capped at the last 200 cycles with rotation. A long-running watcher must not be able to fill the user's disk.workspace_importsfor bundle imports (docs/maintainers precedent, PR feat(transfer): add import history ledger endpoint and TransferPage view #102); routing watch cycles into it needs a migration, an API surface and a UI view — that is Phase 2 scope and would make a P2 minimal tier unreviewable as one PR.D-4 Ingestion path — one shared core, two sinks, extracted before it is used
Answer to "does #110 share an ingestion engine with #103": yes for the engine, no for the sink, and neither is done inside PR #108.
server/cmd/mem/qoder_checkpoint.go:37-59 and 63-79), stat-then-hash change gate, cycle aggregation, failure classification, state-root layout, and the--dry-run/--limitsemantics that PR feat(ingest): add mem ingest qoder for AI-agent conversation transcripts (#103) #108 just had to learn the hard way (--dry-runmust not advance a cursor).put --watchwrites the file plane via multipart upload, whose idempotency is content-addressed (uniq_files_user_sha,server/internal/db/migrations/0001_init.sql:61);ingest qoderwrites the memory plane viaPOST /v1/memorieswith a per-lineIdempotency-Key(server/cmd/mem/cmds_ingest.go:342). These are two different data planes — forcing one write API would either lose file bytes or lose per-turn provenance. Sharing the engine is right; sharing the endpoint is wrong.https://github.com/fullstack-ai-infra/mem/issues/111and recorded here asdependencies: [#111 at R1], with the constraint that chore(ingest): extract the shared local ingestion core (walk, cursor, change gate, report) from the qoder connector #111 must not start before feat(ingest): add mem ingest qoder for AI-agent conversation transcripts (#103) #108 merges, so no in-flight PR is disturbed and no diff spans two issues.ACs added by this revision
not_found) without entering the loop.--format jsonemits exactly the closed count set and closed failure-code set; adedupeditem carries the server-returned id and path.需要 maintainer 执行(本账号无写权限)
waterbro-8对 upstream issue 无 body/label 写权限(PATCH/labels 均 403,需 admin)。请 @PeterGuy326 按本条 + R3 条依次落 R2/R3 body,并补 triage 要求的标签:area:cli(现缺 area)、evidence:e2-source(本条已给源码级取证)、status 按 R3 结论决定(R2 本身不解锁status:ready,REQ-001 口径仍待产品确认)。本条不改动任何旧 decision comment。—— 技术 P9
waterbro-8 commentedon Aug 30, 2026
裁决摘要(技术 P9)
D-3(变更检测语义)出解,并据此把 REQ-001 的 "new/changed" 口径收窄为"新文件 + 变更只上报不重摄"。收窄依据是 SPEC 锚点本身只承诺新文件,以及文件面没有版本/替换模型这两条硬事实。此条需要产品确认。
D-3 Change detection — stat gate, hash authority, one quiet interval, append-only
Cursor row per absolute path, under the shared state root:
{abs, size, mtime, sha256, fileId, ingestedAt}.(size, mtime)identical across two consecutive scansingested(size, mtime)unchangedunchangedunchangedchangedlocal_goneThree rulings inside that table matter:
(size, mtime)is a cost gate, sha256 is the only authority. mtime alone must never trigger an upload —rsync -t,touch, backup restores and cloud-sync placeholders all move mtime without moving bytes. The stat gate exists so a 10k-file directory does not get 10k hashes per cycle.(size, mtime)is unchanged between two consecutive scans.server/internal/file/file.go:196-201). That is a success, and it must be counted asdeduped, never asingested, with the returned id/path as the locator (AC-006). "Your content is in the vault" is true there; "it landed in the folder you passed to--to" is not necessarily.Why
changedis reported rather than re-ingested.Service.Putalways inserts a fresh row unless the content hash already exists, and nothing infilesmakes one path versioned — so a re-put of/Docs/report.pdfproduces two rows at the same path, both visible in Drive, both searchable, indistinguishable except by id. With a watcher on an edited directory that is a new entry per save. Linking them is impossible on the file plane (memory_relationsis memory-scoped), and choosing which onemem get/mem catshould return is a product decision, not an implementation detail. That design is the Phase 2 sync-drive work GOAL.md:95/123 and SPEC.md:1140 already defer; carryingchangedas a visible, counted state is the honest interim, and it keeps this issue one reviewable PR.AC changes by this revision
scanned/ingested/deduped/unchanged.local_gonein the next cycle and issues zero delete or write-back requests — asserted at the HTTP-contract level with a stub transport that fails the test on anyDELETEor non-upload call, following thecmds_ingest_test.goprecedent.changedwith the local path and the previously ingested file id, and leaves the vault contents unchanged across two consecutive cycles.Non-goals added
No file-revision or replacement model, no in-place update, no overwrite-by-path, no delete propagation, no ignore/exclude rule language (
--excludeis Phase 2 with the sync drive — the directory the user picks is the boundary in this tier), and no event-API (fsnotify) dependency.给产品的确认点
收窄需要 @PeterGuy326 一句书面确认(R3 已按
requirement-decision.v1记录)。若产品选择"变更也要入",正确动作是先立"文件版本/替换语义"设计项(哪个 id 是现值、路径冲突怎么展示、mem get返回谁),而不是在 #110 里放开重摄开关。—— 技术 P9
docs(cli): cite bytefolk#110's real requirement ids in the watch comm…
waterbro-8 commentedon Sep 3, 2026
Coverage read-back after today's fork closes: nothing is in flight for this issue
Both of this issue's implementation candidates were fork pull requests, and both were closed on 2026-09-03 under the decision that fork workflow permissions stay off and the work is redone in-repo:
mem put --watchone-way directory watch daemon #150feat(cli): add mem put --watch one-way directory watch(heada7aa85b5) -- closed, not imported.refactor(ingest): extract the shared local ingestion core …(head785de61e, 3 commits) -- closed, not imported. It listed feat(sync): put --watch one-way directory watch (minimal tier, carved from Phase 2 sync drive) #110 among what it covered.Why nothing was transplanted rather than cherry-picked:
cherry-pickpreserves the author, and these heads are authored byliyuanyang <liyuanyang@users.noreply.github.com>(and in #147 also by a personal@qq.comaddress). Importing them would put exactly the attribution shape today's decision exists to prevent ontomain. Preserving the original署名 and satisfying that rule are mutually exclusive for these heads, which is why the rework has to be written in-repo. Nothing was deleted -- the fork branches are untouched, so the closed heads stay inspectable.The gap is real, not a paperwork artifact: on
main@1332bf46there is no watch path at all -- a grep forwatchacrossserver/cmd/mem/*.goreturns zero matches.So that nobody reads the history the wrong way: #110 is now both unassigned and uncovered. Anyone who saw #150 open earlier should not assume the minimal watch tier is in progress.
Two conditions worth knowing before someone starts the in-repo version:
Web → Audit dependenciesis red onmainitself today (browserslist <=4.28.6, read-back on #138), so a new PR inherits one failing required check until #159 lands; and every PR still needs @PeterGuy326 or @Bindy-lbb, since the baseCODEOWNERSnames only those two.No label, status, assignment, or body change made here by me.
sun-970 commentedon Sep 8, 2026
Draft Implementation Proposal
Based on the existing decisions (R2/R3) from the technical P9, here is a concrete implementation sketch for team discussion. This is a draft — flag anything that looks off before any code is written.
Sequencing
Two phases, matching the dependency chain:
Phase A — #111 first: Extract shared ingestion core
Before writing watch code, extract the shared local-walk + cursor + change-detection logic into a reusable package. Two call sites: existing
ingest qoderand the newput --watch.Phase B — #110: Implement
put --watchon top of the corePhase A: Shared core extraction (#111)
Proposed package layout:
Key interfaces:
The
ingest qoderpath andput --watchpath each provide their ownSinkimplementation. The scanner is shared; the write surface is not.Cursor storage reuses the existing atomic-rename pattern from
qoder_checkpoint.go:37-59.Phase B:
put --watchimplementationNew file:
server/cmd/mem/cmds_watch.goFlag surface (aligned with SPEC.md:604):
Runtime loop:
Signal handling: hook into the existing
signal.NotifyContextatmain.go:25, add SIGTERM alongside SIGINT for the watch subcommand.Advisory lock: use
syscall.Flock(Linux/macOS) — no new dependency.Open questions for discussion
Package name
localscan— or wouldingestcore/dirwalkbe clearer? The name needs to signal "shared between watch and qoder-ingest."Lock file location —
<state root>/watch/locks/is proposed for consistency with the report log location. Alternative: lock file next to the watched root (more discoverable but pollutes user directories).Give-up bound — 10 consecutive all-fail cycles is from D-1. Should this be configurable via flag, or is a hardcoded constant sufficient for P2 tier?
JSONL rotation — 200-cycle cap from D-2. Should we also add a max-file-size cap (e.g., 1MB) as a safety net?
Test strategy — following the fixtures-first approach from the issue body:
watch_test.go: new/changed files ingested, report counts correct (AC-001)watch_failure_test.go: failing file yields stable code, daemon continues (AC-002)watch_boundary_test.go: local deletion ->local_gone, zero delete/write-back calls (AC-003)watch_lifecycle_test.go: signal handling, lock contention, missing root (AC-005)CI gate — the latest comment notes
Web -> Audit dependenciesis red on main (browserslist). Should this PR wait for build(deps-dev): bump browserslist from 4.28.2 to 4.28.8 in /web #159 to land first, or proceed in parallel?What this proposal deliberately does NOT include
--exclude/ ignore rules (Phase 2 per R3 non-goals)wadrzl commentedon Sep 8, 2026
Issue #110 Verification (2026-09-08)
Baseline: origin/main read today. R3 scope (9 ACs). No open PR targets this issue or its dependency #111. Fork PRs #150 (watch) and #147 (shared core) closed without merge 2026-09-03; nothing imported.
Requirements (R3 scope)
Acceptance criteria (R3 scope)
Dependency
Closed PRs
mem put --watchone-way directory watch daemon #150 (watch impl, head a7aa85b): closed not imported, attribution constraint.Summary
0/4 requirements met. 0/9 ACs met. Dependency #111 open with no PR. Zero implementation candidates in-repo. The design record (R1/R2/R3) is complete and detailed; implementation has not started.
waterbro-8 commentedon Sep 17, 2026
Still blocked on #111.
mem put --watchis absent onmain. Fork candidate #150 was closed because a long-lived watcher writing through the ingest path is unsafe until #139/#111 land.I am not implementing
--watchon top of the unextracted qoder connector. After #191 and a re-landed #111, this can be an organization-branch PR.waterbro-8 commentedon Sep 17, 2026
Predecessor extraction is now on an organization branch: #217 (issue #111).
put --watchshould wait for that PR to land so it can consumeserver/internal/ingestinstead of copying cursor/report code.I am not opening a
--watchPR until #217 is reviewed.refactor(ingest): extract the shared local ingestion core
refactor(ingest): extract the shared local ingestion core
refactor(ingest): extract the shared local ingestion core (#111) (#217)
wadrzl commentedon Sep 18, 2026
#110 依赖拆解与实现计划
前置 #111 已于 2026-09-18 合入 main(PR #217,commit
f72e294)。server/internal/ingest包已可用,提供:Walk)CodeAuth/CodePlanQuota/CodeProviderTimeout/CodeNetwork/CodeReadDenied/CodeUploadRejected/CodeRootMissing/CodeStateCorrupt)Report/Run)CanonicalRoot/CursorPath)依赖关系
server/internal/ingest直接在 main 上无剩余阻塞依赖。
实现拆解
一个 PR,约 5 个新文件 + 2 个改动文件:
1. File-plane sink(新文件:
server/internal/ingest/file_sink.go)put --watch和ingest qoder的区别在 sink 层:/v1/memorieswatch 的 ParseFunc 需要返回一个 Unit,其 data 是文件内容。UploadFunc 调用现有的
Service.Put路径(multipart upload + content-addressed dedup)。deduped判定:server 返回已有行的 id/path(server/internal/file/file.go:196-201),report 中 item 携带 server 返回的 path,不是请求的--to路径。2. Watch 命令(新文件:
server/cmd/mem/cmds_watch.go)运行循环:
CanonicalRoot解析 watched path<state root>/watch/locks/<sha1(abs root)>),已持有则 exit 1a. 调用
ingest.Run执行一次 scan + ingestb. 聚合 cycle report
c. 输出 report 到 stdout
d. 追加 JSONL 到
<state root>/watch/reports/<sha1(abs root)>.jsonle. 超 200 条则 rotate
f. sleep
--interval3. Advisory lock(新文件:
server/cmd/mem/cmds_watch_lock.go)复用
server/internal/ingest/cursor_lock_unix.go的 flock 模式。每个 watched root 一个 lock 文件,syscall.Flock(LOCK_EX | LOCK_NB)实现 fail-fast。4. JSONL report 持久化(新文件:
server/cmd/mem/cmds_watch_report.go)<cliStateRoot>/watch/reports/<sha1(abs root)>.jsonl5. Signal handling
复用
server/cmd/mem/main.go:25的signal.NotifyContext,watch 子命令额外注册 SIGTERM。Signal 到达时完成当前 cycle 后 exit 0。6. 测试
cmds_watch_test.gocmds_watch_failure_test.gocmds_watch_boundary_test.gocmds_watch_lifecycle_test.gocmds_watch_report_test.go不做的事
--exclude/ ignore rules(Phase 2)预计时间
@waterbro-8 这是你作为 technicalOwner 的实现。如需我协助 review 或其他前置操作,请在群内说明。
waterbro-8 commentedon Sep 20, 2026
平台组分工补派(冯浩然 / waterbro-8)
sun-970,仓库暂不能指派,以钉钉+本评论为准)waterbro-8 commentedon Sep 20, 2026
平台组群成员派活(冯浩然 / waterbro-8)
sun-970 commentedon Sep 20, 2026
Implemented in-repo
mem put <path> --watchper R2/R3. Consumesserver/internal/ingest(#111 / PR #217). No fsnotify, no bidirectional sync, no deletion propagation.PR: #225 (
sun-970:feat/110-put-watch-20260920)@waterbro-8 please review.
feat(cli): add one-way mem put --watch (bytefolk#110)
feat(cli): add one-way mem put --watch (#110) (#225)
waterbro-8 commentedon Sep 23, 2026
关闭验收:PR #225 已合入 main(merge ca6ed33;final head 3bc3d58),实现 R2/R3 最小 one-way mem put --watch,并复用 #111 的 shared ingest core;不包含 fsnotify、双向同步或删除传播,符合非目标。final head 已正式 APPROVED,全部 hosted checks 通过;合入后的 main 四套 workflow 全绿。