Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,13 @@ The project publishes 0.x prerelease versions; a stable release line is not yet

### Added

- Workspace-scoped Markdown Vault sync HTTP API (`#234`): discover Vaults, pull bounded
snapshots and atomically commit logical-note deltas with whole-Vault CAS.
Independent empty notes, immutable revisions and deletion tombstones preserve
identity and concurrent edits; workspace, read/write and token-path boundaries
reuse canonical authorization. Additive migration 0027 leaves existing files
unchanged. See `docs/VAULT_SYNC.md` for limits, downgrade data loss and the
explicit boundary from current workspace-bundle export/import formats.
- Per-agent memory namespace filter on lexical recall (`#227`): `POST /v1/context`
and `mem_context` accept `agent_id` plus optional `extra_agent_ids`. When set,
structured-memory recall hard-filters `producer_agent` before ranking so
Expand All @@ -17,8 +24,20 @@ The project publishes 0.x prerelease versions; a stable release line is not yet
Design: `docs/agent-memory-namespace.md`.
- `mem put <path> --watch` one-way foreground directory watch: poll `--interval` (default 30s), ingest new files after one quiet interval, report changed files without re-ingesting, and never propagate local deletes. Refs #110.

### Fixed

- Restore Linux/Windows Web dependency audit gates by removing the unpatched
braces build-tool chain with official Tailwind/PostCSS 4 and typescript-eslint
8, while retaining existing theme utility behavior and browser acceptance.
Apply compatible transitive security updates without relaxing audit thresholds.
Validation and public advisory references: `docs/acceptance/VAULT_SYNC_VALIDATION.md`.
- Build isolated lifecycle-test MinIO/mc images from checksum-pinned official
sources when former public images are no longer anonymously pullable. Keep
production assets and all HTTP/CLI/MCP assertions unchanged. Refs #234.

### Changed

- The live recall producer (`python3 -m benchmarks.recall produce`) reads engine, profile, provider and embedding dimension from the running memd instead of CLI labels, scores the multilingual v1 set (English and Chinese, including image-description queries), and stays on-demand rather than a CI gate. Refs #175.
- Rewrite the root README around a concise product promise, a verified Compose first-run path, source-built MCP setup, current capability and trust boundaries, and links to authoritative reference docs. Remove the unsupported unpublished `@bytefolk/mem-mcp@0.1.2` install path and keep its migration explicitly tracked in #153. Refs #229.
- Index generation HTTP create/activate/rollback stay `503 execution_unavailable` with `execution_wired: false` on that body, and the same flag is now on events as well as list/status/cancel/resume/discard. Create still returns `400` for malformed JSON or an empty `profile_id` before the 503. Empty-workspace activate is refused by the same 503. Refs #174.
- Ingest cursor locks try non-blocking exclusive locks and give up after 5s so a wedged peer becomes a warning instead of a silent hang. Refs #139.
Expand Down
22 changes: 22 additions & 0 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,28 @@ workspace import 的数据库 commit 若无法确认结果,服务端不得删

---

### F9A · Workspace Markdown Vault 同步(HTTP v1)

需求:[mem #234](https://github.com/bytefolk/mem/issues/234)。

`GET /v1/vault/list`、`GET /v1/vault/snapshot?vaultId=<UUID>` 与
`POST /v1/vault/commit` 提供工作区内的逻辑笔记同步底座。每份笔记使用独立
`noteId`,相同正文和空笔记不会因 blob 去重而合并。commit 传递
`{vaultId,baseRevision,title?,entries:[{noteId,path,content,deleted?,properties?}]}`
增量;整个 Vault 的 head 在单个事务内执行 CAS,过期 head 返回 `409`
及 `currentRevision`,不覆盖其他客户端。删除保留 tombstone 与不可原地更新的
正文 revision;重命名保留逻辑 ID。只读需 `read`,提交需 `read` + `write`,
复用现有 workspace membership 和 `/Vaults/<vaultId>` token path 边界。
properties 仅为有界数据,不能授予工具或身份权限。图片等原件继续使用 files API。

`vault-snapshot.v1` 是独立的当前状态可迁移合同,详见
[API、限制与错误码](docs/VAULT_SYNC.md) 和
[JSON Schema](docs/schemas/vault-snapshot.v1.schema.json)。CLI/MCP/Web 本次不新增
Vault 命令或独立笔记编辑器。现有 workspace bundle v1/v2 不包含 Vault 表和历史;
完整历史仍需数据库备份。服务不运行桌面文件监听或自动合并客户端冲突;客户端必须
拉取 head、三方合并并保留冲突副本。迁移 0027 不改变 files/auth/memory 表;降级
到 0026 会丢弃 Vault 数据,操作前需备份。

## 4. 非功能需求

| 维度 | 目标 |
Expand Down
60 changes: 36 additions & 24 deletions benchmarks/recall/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@

> **Evidence boundary: producer checks do not complete the live benchmark.**
>
> The producer corrections tracked in #196 do not complete #175's acceptance.
> The original #184 live-run requirement still needs a populated real memd,
> actual embedding-provider output and saved ranking artifacts. Unit or
> loopback HTTP fixtures establish transport behavior only. A real model-free
> lexical run also cannot establish vector/provider quality. See
> The producer can emit `mem.recall-rankings.v1` from a running memd and copies
> engine/profile identity from that process. It still needs a populated real
> memd, actual embedding-provider output, and a saved ranking artifact before
> a real-memd baseline can land in `baselines/`. Unit or loopback HTTP fixtures
> establish transport behavior only. See
> [Exact remaining live prerequisites](#exact-remaining-live-prerequisites).

This directory provides a small, repeatable retrieval benchmark. It is a
Expand Down Expand Up @@ -184,21 +184,32 @@ emits a `mem.recall-rankings.v1` file that the existing `run --rankings` path
consumes. Latency is measured client-side per request; the `0 ms` sentinel
warning above applies only to the offline lexical lane.

Engine, provider, model/profile identity and embedding dimension are read from
the running process (`GET /v1/version` and
`GET /v1/workspaces/current/ai-profile`). They are not CLI flags. The artifact
cannot be labeled `lexical-reference`. ANN identity is not advertised by those
endpoints, so `configuration.index.kind` is recorded as `not-advertised`
rather than guessed.

The default dataset is the multilingual v1 fixture. Scoring the same run
reports Recall@1/5/10 and nDCG@10 under `metrics.by_language` (`en`, `zh`)
and `metrics.by_source_kind` (`text`, `image_caption`, `structured`). The
Chinese set includes image-description queries (`q-zh-image-paraphrase`).
Structured-memory queries cannot be served by `POST /v1/search`; they are
recorded as `unsupported_source_kind` and do not fail the producer.

```bash
python3 -m benchmarks.recall produce \
--memd-url http://localhost:8080 \
--token "$MEM_TOKEN" \
--dataset benchmarks/recall/data/profile-text-v1 \
--output /tmp/live-rankings.json \
--dimension 768 --provider "$MEM_PROVIDER_LABEL" --model "$MEM_MODEL_LABEL" \
--mode vector
```

Then score the saved rankings (the default v1 baseline uses a different corpus):
Then score the saved rankings:

```bash
python3 -m benchmarks.recall run \
--dataset benchmarks/recall/data/profile-text-v1 \
--rankings /tmp/live-rankings.json \
--output /tmp/live-artifact.json
```
Expand All @@ -224,11 +235,9 @@ silently ignored.
Vector mode sends `route=text`; it does not claim a hybrid lexical/vector or
multimodal experiment. Lexical mode sends `route=lexical` and requires the
server capability from #183. It emits null provider/model/dimension as required
by the ranking schema. Provider, model, dimension and index configuration are
operator declarations, not discovered or verified server metadata. The
`hardware.host` value deliberately records only the producer client's
OS/architecture, never its hostname. It is not the server's hardware inventory
and cannot establish comparable performance conditions.
by the ranking schema. The `hardware.host` value deliberately records only the
producer client's OS/architecture, never its hostname. It is not the server's
hardware inventory and cannot establish comparable performance conditions.

The full v1 corpus also contains structured-memory queries. `/v1/search` cannot
serve these; the producer records `unsupported_source_kind` and exits 2. Any
Expand All @@ -248,17 +257,20 @@ acceptance. Those remain NOT VERIFIED until a real populated memd run is saved.
interpreting the producer's output. Direct database seeding can establish
retrieval/transport behavior but does not verify ingestion or Worker indexing.
3. For vector acceptance, select the same actual embedding model for corpus and
queries, record its dimension and profile, and independently inspect the
active generation/index and deployment revision. Producer labels alone do
not verify any of those properties.
queries. The producer now copies provider/model/dimension from the active
workspace AI profile; independently inspect the ANN index and deployment
revision. `configuration.index.kind=not-advertised` means those endpoints
do not expose ANN identity, not that no index exists.
4. Run the documented producer command, retain its output, score the resulting
rankings and record errors and measured client latencies. An empty/error run
or a fake embedding provider cannot establish vector quality. A lexical run
requires the separate #194 server capability and remains a distinct result.
5. The bounded file-only experiment does not cover #175's bilingual image-query
acceptance or the full v1 structured-memory corpus. The original issue and
live quality acceptance must not be described as complete on this evidence.

The producer remains opt-in; the normal recall CI gate runs deterministic unit
and fixture checks only. No real-model baseline is checked in until its actual
configuration and saved run are available for review.
5. Score the v1 artifact's `metrics.by_language.zh` and
`metrics.by_source_kind.image_caption` for GOAL.md §6. Structured-memory
rows remain `unsupported_source_kind` because `/v1/search` is file search.

The producer remains **on-demand**. `make test-recall` and CI run deterministic
unit and fixture checks only; they never start memd or call a provider. No
real-memd baseline is checked into `benchmarks/recall/baselines/` until a
populated deployment's saved rankings are reviewed. That remaining #175
acceptance bullet is still open.
42 changes: 30 additions & 12 deletions benchmarks/recall/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

from .dataset import load_dataset
from .errors import BenchmarkError
from .live_producer import produce_rankings
from .live_producer import discover_live_configuration, produce_rankings
from .runner import (
compare_artifacts,
comparison_summary,
Expand All @@ -24,6 +24,15 @@
PACKAGE_ROOT = Path(__file__).resolve().parent
DEFAULT_DATASET = PACKAGE_ROOT / "data" / "v1"
DEFAULT_BASELINE = PACKAGE_ROOT / "baselines" / "lexical-reference.v1.json"
_PRODUCE_EXPECTED_GAPS = frozenset({"unsupported_source_kind", "unsupported_filter"})


def produce_exit_code(rankings: dict) -> int:
"""Exit 2 only for transport/mapping failures, not dataset surface gaps."""
for query in rankings.get("queries", []):
if query.get("status") == "error" and query.get("error_code") not in _PRODUCE_EXPECTED_GAPS:
return 2
return 0


def _parser() -> argparse.ArgumentParser:
Expand Down Expand Up @@ -79,13 +88,9 @@ def _parser() -> argparse.ArgumentParser:
produce.add_argument("--output", type=Path, required=True)
produce.add_argument("--limit", type=int, default=10)
produce.add_argument("--timeout", type=float, default=30.0)
produce.add_argument("--engine", default="live-memd")
produce.add_argument("--dimension", type=int, default=768)
produce.add_argument(
"--mode", default="vector", choices=["lexical", "vector"]
)
produce.add_argument("--provider", default="operator-unspecified")
produce.add_argument("--model", default="operator-unspecified")
return parser


Expand Down Expand Up @@ -122,24 +127,37 @@ def main(argv: list[str] | None = None) -> int:

if args.command == "produce":
dataset = load_dataset(args.dataset)
live_config = discover_live_configuration(
args.memd_url,
args.token,
mode=args.mode,
timeout=args.timeout,
)
rankings = produce_rankings(
dataset,
base_url=args.memd_url,
token=args.token,
limit=args.limit,
timeout=args.timeout,
engine_label=args.engine,
dimension=args.dimension,
mode=args.mode,
provider=args.provider,
model=args.model,
live_config=live_config,
)
write_json(args.output, rankings)
ok_count = sum(1 for q in rankings["queries"] if q["status"] == "ok")
err_count = sum(1 for q in rankings["queries"] if q["status"] == "error")
print(f"produced rankings: {ok_count} ok, {err_count} error")
real_errors = [
q
for q in rankings["queries"]
if q.get("status") == "error" and q.get("error_code") not in _PRODUCE_EXPECTED_GAPS
]
gap_count = sum(
1
for q in rankings["queries"]
if q.get("error_code") in _PRODUCE_EXPECTED_GAPS
)
print(f"produced rankings: {ok_count} ok, {len(real_errors)} error, {gap_count} unsupported")
print(f"engine: {rankings['engine']}")
print(f"rankings artifact: {args.output}")
return 2 if err_count else 0
return produce_exit_code(rankings)

first = run_benchmark(
dataset_dir=args.dataset,
Expand Down
Loading
Loading