diff --git a/CHANGELOG.md b/CHANGELOG.md index 7eb1a5a..c3b3334 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ The project publishes 0.x prerelease versions; a stable release line is not yet ### Changed +- The live recall producer (`python3 -m benchmarks.recall produce`) reads engine, profile, provider and embedding dimension from the running memd instead of CLI labels, scores the multilingual v1 set (English and Chinese, including image-description queries), and stays on-demand rather than a CI gate. Refs #175. - Rewrite the root README around a concise product promise, a verified Compose first-run path, source-built MCP setup, current capability and trust boundaries, and links to authoritative reference docs. Remove the unsupported unpublished `@bytefolk/mem-mcp@0.1.2` install path and keep its migration explicitly tracked in #153. Refs #229. - Index generation HTTP create/activate/rollback stay `503 execution_unavailable` with `execution_wired: false` on that body, and the same flag is now on events as well as list/status/cancel/resume/discard. Create still returns `400` for malformed JSON or an empty `profile_id` before the 503. Empty-workspace activate is refused by the same 503. Refs #174. - Ingest cursor locks try non-blocking exclusive locks and give up after 5s so a wedged peer becomes a warning instead of a silent hang. Refs #139. diff --git a/benchmarks/recall/README.md b/benchmarks/recall/README.md index 69a1e10..6141a17 100644 --- a/benchmarks/recall/README.md +++ b/benchmarks/recall/README.md @@ -2,11 +2,11 @@ > **Evidence boundary: producer checks do not complete the live benchmark.** > -> The producer corrections tracked in #196 do not complete #175's acceptance. -> The original #184 live-run requirement still needs a populated real memd, -> actual embedding-provider output and saved ranking artifacts. Unit or -> loopback HTTP fixtures establish transport behavior only. A real model-free -> lexical run also cannot establish vector/provider quality. See +> The producer can emit `mem.recall-rankings.v1` from a running memd and copies +> engine/profile identity from that process. It still needs a populated real +> memd, actual embedding-provider output, and a saved ranking artifact before +> a real-memd baseline can land in `baselines/`. Unit or loopback HTTP fixtures +> establish transport behavior only. See > [Exact remaining live prerequisites](#exact-remaining-live-prerequisites). This directory provides a small, repeatable retrieval benchmark. It is a @@ -184,21 +184,32 @@ emits a `mem.recall-rankings.v1` file that the existing `run --rankings` path consumes. Latency is measured client-side per request; the `0 ms` sentinel warning above applies only to the offline lexical lane. +Engine, provider, model/profile identity and embedding dimension are read from +the running process (`GET /v1/version` and +`GET /v1/workspaces/current/ai-profile`). They are not CLI flags. The artifact +cannot be labeled `lexical-reference`. ANN identity is not advertised by those +endpoints, so `configuration.index.kind` is recorded as `not-advertised` +rather than guessed. + +The default dataset is the multilingual v1 fixture. Scoring the same run +reports Recall@1/5/10 and nDCG@10 under `metrics.by_language` (`en`, `zh`) +and `metrics.by_source_kind` (`text`, `image_caption`, `structured`). The +Chinese set includes image-description queries (`q-zh-image-paraphrase`). +Structured-memory queries cannot be served by `POST /v1/search`; they are +recorded as `unsupported_source_kind` and do not fail the producer. + ```bash python3 -m benchmarks.recall produce \ --memd-url http://localhost:8080 \ --token "$MEM_TOKEN" \ - --dataset benchmarks/recall/data/profile-text-v1 \ --output /tmp/live-rankings.json \ - --dimension 768 --provider "$MEM_PROVIDER_LABEL" --model "$MEM_MODEL_LABEL" \ --mode vector ``` -Then score the saved rankings (the default v1 baseline uses a different corpus): +Then score the saved rankings: ```bash python3 -m benchmarks.recall run \ - --dataset benchmarks/recall/data/profile-text-v1 \ --rankings /tmp/live-rankings.json \ --output /tmp/live-artifact.json ``` @@ -224,11 +235,9 @@ silently ignored. Vector mode sends `route=text`; it does not claim a hybrid lexical/vector or multimodal experiment. Lexical mode sends `route=lexical` and requires the server capability from #183. It emits null provider/model/dimension as required -by the ranking schema. Provider, model, dimension and index configuration are -operator declarations, not discovered or verified server metadata. The -`hardware.host` value deliberately records only the producer client's -OS/architecture, never its hostname. It is not the server's hardware inventory -and cannot establish comparable performance conditions. +by the ranking schema. The `hardware.host` value deliberately records only the +producer client's OS/architecture, never its hostname. It is not the server's +hardware inventory and cannot establish comparable performance conditions. The full v1 corpus also contains structured-memory queries. `/v1/search` cannot serve these; the producer records `unsupported_source_kind` and exits 2. Any @@ -248,17 +257,20 @@ acceptance. Those remain NOT VERIFIED until a real populated memd run is saved. interpreting the producer's output. Direct database seeding can establish retrieval/transport behavior but does not verify ingestion or Worker indexing. 3. For vector acceptance, select the same actual embedding model for corpus and - queries, record its dimension and profile, and independently inspect the - active generation/index and deployment revision. Producer labels alone do - not verify any of those properties. + queries. The producer now copies provider/model/dimension from the active + workspace AI profile; independently inspect the ANN index and deployment + revision. `configuration.index.kind=not-advertised` means those endpoints + do not expose ANN identity, not that no index exists. 4. Run the documented producer command, retain its output, score the resulting rankings and record errors and measured client latencies. An empty/error run or a fake embedding provider cannot establish vector quality. A lexical run requires the separate #194 server capability and remains a distinct result. -5. The bounded file-only experiment does not cover #175's bilingual image-query - acceptance or the full v1 structured-memory corpus. The original issue and - live quality acceptance must not be described as complete on this evidence. - -The producer remains opt-in; the normal recall CI gate runs deterministic unit -and fixture checks only. No real-model baseline is checked in until its actual -configuration and saved run are available for review. +5. Score the v1 artifact's `metrics.by_language.zh` and + `metrics.by_source_kind.image_caption` for GOAL.md ยง6. Structured-memory + rows remain `unsupported_source_kind` because `/v1/search` is file search. + +The producer remains **on-demand**. `make test-recall` and CI run deterministic +unit and fixture checks only; they never start memd or call a provider. No +real-memd baseline is checked into `benchmarks/recall/baselines/` until a +populated deployment's saved rankings are reviewed. That remaining #175 +acceptance bullet is still open. diff --git a/benchmarks/recall/__main__.py b/benchmarks/recall/__main__.py index fe15870..0e2ed09 100644 --- a/benchmarks/recall/__main__.py +++ b/benchmarks/recall/__main__.py @@ -9,7 +9,7 @@ from .dataset import load_dataset from .errors import BenchmarkError -from .live_producer import produce_rankings +from .live_producer import discover_live_configuration, produce_rankings from .runner import ( compare_artifacts, comparison_summary, @@ -24,6 +24,15 @@ PACKAGE_ROOT = Path(__file__).resolve().parent DEFAULT_DATASET = PACKAGE_ROOT / "data" / "v1" DEFAULT_BASELINE = PACKAGE_ROOT / "baselines" / "lexical-reference.v1.json" +_PRODUCE_EXPECTED_GAPS = frozenset({"unsupported_source_kind", "unsupported_filter"}) + + +def produce_exit_code(rankings: dict) -> int: + """Exit 2 only for transport/mapping failures, not dataset surface gaps.""" + for query in rankings.get("queries", []): + if query.get("status") == "error" and query.get("error_code") not in _PRODUCE_EXPECTED_GAPS: + return 2 + return 0 def _parser() -> argparse.ArgumentParser: @@ -79,13 +88,9 @@ def _parser() -> argparse.ArgumentParser: produce.add_argument("--output", type=Path, required=True) produce.add_argument("--limit", type=int, default=10) produce.add_argument("--timeout", type=float, default=30.0) - produce.add_argument("--engine", default="live-memd") - produce.add_argument("--dimension", type=int, default=768) produce.add_argument( "--mode", default="vector", choices=["lexical", "vector"] ) - produce.add_argument("--provider", default="operator-unspecified") - produce.add_argument("--model", default="operator-unspecified") return parser @@ -122,24 +127,37 @@ def main(argv: list[str] | None = None) -> int: if args.command == "produce": dataset = load_dataset(args.dataset) + live_config = discover_live_configuration( + args.memd_url, + args.token, + mode=args.mode, + timeout=args.timeout, + ) rankings = produce_rankings( dataset, base_url=args.memd_url, token=args.token, limit=args.limit, timeout=args.timeout, - engine_label=args.engine, - dimension=args.dimension, mode=args.mode, - provider=args.provider, - model=args.model, + live_config=live_config, ) write_json(args.output, rankings) ok_count = sum(1 for q in rankings["queries"] if q["status"] == "ok") - err_count = sum(1 for q in rankings["queries"] if q["status"] == "error") - print(f"produced rankings: {ok_count} ok, {err_count} error") + real_errors = [ + q + for q in rankings["queries"] + if q.get("status") == "error" and q.get("error_code") not in _PRODUCE_EXPECTED_GAPS + ] + gap_count = sum( + 1 + for q in rankings["queries"] + if q.get("error_code") in _PRODUCE_EXPECTED_GAPS + ) + print(f"produced rankings: {ok_count} ok, {len(real_errors)} error, {gap_count} unsupported") + print(f"engine: {rankings['engine']}") print(f"rankings artifact: {args.output}") - return 2 if err_count else 0 + return produce_exit_code(rankings) first = run_benchmark( dataset_dir=args.dataset, diff --git a/benchmarks/recall/live_producer.py b/benchmarks/recall/live_producer.py index 9888b36..2ea2541 100644 --- a/benchmarks/recall/live_producer.py +++ b/benchmarks/recall/live_producer.py @@ -72,6 +72,113 @@ def _coarse_host() -> str: return "unknown" +def _json_get( + base_url: str, + path: str, + token: str, + *, + timeout: float, +) -> dict[str, Any]: + url = base_url.rstrip("/") + path + req = Request(url, method="GET") + req.add_header("Authorization", f"Bearer {token}") + try: + with urlopen(req, timeout=timeout) as resp: + payload = json.loads(resp.read().decode("utf-8")) + except HTTPError as exc: + raise BenchmarkError(f"GET {path} failed with HTTP {exc.code}") from exc + except URLError as exc: + raise BenchmarkError(f"GET {path} failed: {exc.reason}") from exc + except Exception as exc: + raise BenchmarkError(f"GET {path} returned unreadable JSON") from exc + if not isinstance(payload, dict): + raise BenchmarkError(f"GET {path} must return a JSON object") + return payload + + +def discover_live_configuration( + base_url: str, + token: str, + *, + mode: str, + timeout: float = 30.0, +) -> dict[str, Any]: + """Read engine/profile identity from a running memd, not from CLI flags.""" + if mode not in {"lexical", "vector"}: + raise BenchmarkError("memd /v1/search does not expose a hybrid lexical/vector route") + version = _json_get(base_url, "/v1/version", token, timeout=timeout) + version_label = version.get("version") + revision = version.get("revision") + contract = version.get("contract") + if not isinstance(version_label, str) or not version_label.strip(): + raise BenchmarkError("GET /v1/version did not return a version string") + engine = f"memd/{version_label.strip()}" + if engine == "lexical-reference": + raise BenchmarkError("engine must identify live memd, not lexical-reference") + + profile_payload = _json_get( + base_url, "/v1/workspaces/current/ai-profile", token, timeout=timeout + ) + active = profile_payload.get("active") + server: dict[str, Any] = { + "version": version_label.strip(), + "revision": revision if isinstance(revision, str) else None, + "contract": contract if isinstance(contract, str) else None, + } + index = { + "kind": "not-advertised", + "evidence": ( + "GET /v1/version and GET /v1/workspaces/current/ai-profile do not " + "expose ANN identity" + ), + } + evidence = ( + "discovered from GET /v1/version and GET /v1/workspaces/current/ai-profile" + ) + if mode == "lexical": + return { + "engine": engine, + "provider": None, + "model": None, + "dimension": None, + "index": index, + "server": server, + "evidence": evidence, + } + if not isinstance(active, dict): + raise BenchmarkError( + "vector produce requires an active workspace AI profile from the running memd" + ) + embedding = active.get("embedding") + if not isinstance(embedding, dict): + raise BenchmarkError("active AI profile is missing embedding stage") + provider = embedding.get("provider") + dimension = embedding.get("dimensions") + profile_id = active.get("profile_id") + profile_revision = active.get("profile_revision") + if not isinstance(provider, str) or not provider.strip(): + raise BenchmarkError("active AI profile embedding.provider is empty") + if not isinstance(dimension, int) or isinstance(dimension, bool) or dimension <= 0: + raise BenchmarkError("active AI profile embedding.dimensions must be a positive integer") + if not isinstance(profile_id, str) or not profile_id.strip(): + raise BenchmarkError("active AI profile_id is empty") + model = profile_id.strip() + if isinstance(profile_revision, str) and profile_revision.strip(): + model = f"{model}@{profile_revision.strip()}" + server["profile_id"] = profile_id.strip() + if isinstance(profile_revision, str) and profile_revision.strip(): + server["profile_revision"] = profile_revision.strip() + return { + "engine": engine, + "provider": provider.strip(), + "model": model, + "dimension": dimension, + "index": index, + "server": server, + "evidence": evidence, + } + + def _query_memd( base_url: str, token: str, @@ -131,11 +238,23 @@ def produce_rankings( mode: str = "vector", provider: str = "operator-unspecified", model: str = "operator-unspecified", + live_config: dict[str, Any] | None = None, ) -> dict[str, Any]: if mode not in {"lexical", "vector"}: raise BenchmarkError("memd /v1/search does not expose a hybrid lexical/vector route") if not 1 <= limit <= 100 or not math.isfinite(timeout) or timeout <= 0: raise BenchmarkError("limit must be 1..100 and timeout must be positive and finite") + index_config: dict[str, Any] = {"kind": "operator-unspecified"} + evidence = "operator-declared configuration; model and index not verified by producer" + server_identity: dict[str, Any] | None = None + if live_config is not None: + engine_label = live_config["engine"] + provider = live_config["provider"] + model = live_config["model"] + dimension = live_config["dimension"] + index_config = dict(live_config["index"]) + evidence = live_config["evidence"] + server_identity = live_config.get("server") if not isinstance(engine_label, str) or not engine_label.strip() or engine_label == "lexical-reference": raise BenchmarkError("engine must identify live memd, not lexical-reference") if mode == "vector" and ( @@ -236,24 +355,25 @@ def produce_rankings( row["error_code"] = mapping_error query_rows.append(row) + configuration: dict[str, Any] = { + "mode": mode, + "provider": None if mode == "lexical" else provider, + "model": None if mode == "lexical" else model, + "dimension": None if mode == "lexical" else dimension, + "evidence": evidence, + "index": index_config, + "search": { + "top_k": limit, + "route": "lexical" if mode == "lexical" else "text", + "workspace": "bound by the supplied token; not inferred from dataset labels", + }, + } + if isinstance(server_identity, dict) and server_identity: + configuration["server"] = server_identity return { "schema_version": "mem.recall-rankings.v1", "engine": engine_label, - "configuration": { - "mode": mode, - "provider": None if mode == "lexical" else provider, - "model": None if mode == "lexical" else model, - "dimension": None if mode == "lexical" else dimension, - "evidence": "operator-declared configuration; model and index not verified by producer", - "index": { - "kind": "operator-unspecified", - }, - "search": { - "top_k": limit, - "route": "lexical" if mode == "lexical" else "text", - "workspace": "bound by the supplied token; not inferred from dataset labels", - }, - }, + "configuration": configuration, "hardware": { "host": _coarse_host(), }, diff --git a/benchmarks/recall/tests/test_live_producer.py b/benchmarks/recall/tests/test_live_producer.py index 6985e7e..e4ac243 100644 --- a/benchmarks/recall/tests/test_live_producer.py +++ b/benchmarks/recall/tests/test_live_producer.py @@ -8,12 +8,13 @@ import unittest from unittest.mock import patch -from benchmarks.recall.__main__ import main +from benchmarks.recall.__main__ import main, produce_exit_code from benchmarks.recall.adapters import load_external_rankings from benchmarks.recall.errors import BenchmarkError from benchmarks.recall.live_producer import ( _build_path_index, _match_doc_by_path, + discover_live_configuration, produce_rankings, ) from benchmarks.recall.dataset import Document, load_dataset @@ -118,6 +119,35 @@ def test_http_fixture_runs_transport_and_emits_loadable_artifact(self) -> None: requests = [] class Handler(BaseHTTPRequestHandler): + def do_GET(self): + if self.path == "/v1/version": + payload = json.dumps({ + "version": "0.1.2", + "revision": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "contract": "durable-context.v1", + }).encode() + elif self.path == "/v1/workspaces/current/ai-profile": + payload = json.dumps({ + "active": { + "profile_id": "local-fast", + "profile_revision": "2026-01-01.1", + "embedding": { + "enabled": True, + "provider": "ollama", + "dimensions": 768, + }, + }, + "available": [], + }).encode() + else: + self.send_error(404) + return + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(payload))) + self.end_headers() + self.wfile.write(payload) + def do_POST(self): requests.append((self.path, json.loads(self.rfile.read(int(self.headers["Content-Length"]))))) payload = json.dumps({"results": [{"path": "/research", "name": "saturn.md", "score": 0.9}]}).encode() @@ -137,7 +167,7 @@ def log_message(self, *args): try: code = main(["produce", "--memd-url", f"http://127.0.0.1:{server.server_port}", "--token", "fixture", "--dataset", str(self.dataset), - "--output", str(output), "--provider", "fixture", "--model", "fixture"]) + "--output", str(output)]) finally: server.shutdown() thread.join() @@ -146,6 +176,12 @@ def log_message(self, *args): self.assertEqual(requests[0][0], "/v1/search") self.assertEqual(requests[0][1]["route"], "text") artifact = load_external_rankings(output, load_dataset(self.dataset)) + self.assertEqual(artifact["engine"], "memd/0.1.2") + self.assertNotEqual(artifact["engine"], "lexical-reference") + self.assertEqual(artifact["configuration"]["provider"], "ollama") + self.assertEqual(artifact["configuration"]["model"], "local-fast@2026-01-01.1") + self.assertEqual(artifact["configuration"]["dimension"], 768) + self.assertIn("discovered", artifact["configuration"]["evidence"]) self.assertEqual(artifact["queries"][0]["results"][0]["doc_id"], "file-en-cassini") self.assertGreater(artifact["queries"][0]["latency_ms"], 0) self.assertNotIn("client", artifact["hardware"]) @@ -206,16 +242,36 @@ def test_lexical_artifact_is_loadable_and_route_is_sent(self, mock_query) -> Non load_external_rankings(output, dataset) self.assertEqual(mock_query.call_args.kwargs["route"], "lexical") + @patch("benchmarks.recall.__main__.discover_live_configuration") @patch("benchmarks.recall.live_producer._query_memd") - def test_produce_command_fails_when_requests_fail(self, mock_query) -> None: + def test_produce_command_fails_when_requests_fail(self, mock_query, mock_discover) -> None: + mock_discover.return_value = { + "engine": "memd/0.1.2", + "provider": "ollama", + "model": "local-fast@2026-01-01.1", + "dimension": 768, + "index": {"kind": "not-advertised"}, + "evidence": "discovered from GET /v1/version", + "server": {"version": "0.1.2"}, + } mock_query.return_value = ([], 1, "http_503") code = main(["produce", "--memd-url", "http://localhost", "--token", "test", "--dataset", str(self.dataset), "--output", str(self.root / "failed.json")]) self.assertEqual(code, 2) + @patch("benchmarks.recall.__main__.discover_live_configuration") @patch("benchmarks.recall.live_producer._query_memd") - def test_malformed_hits_retain_error_artifact(self, mock_query) -> None: + def test_malformed_hits_retain_error_artifact(self, mock_query, mock_discover) -> None: + mock_discover.return_value = { + "engine": "memd/0.1.2", + "provider": "ollama", + "model": "local-fast@2026-01-01.1", + "dimension": 768, + "index": {"kind": "not-advertised"}, + "evidence": "discovered from GET /v1/version", + "server": {"version": "0.1.2"}, + } good = {"path": "/research", "name": "saturn.md", "score": 0.9} malformed = [ {"path": None}, {"path": []}, {"path": {}}, {"path": 42}, @@ -292,6 +348,103 @@ def test_equal_same_workspace_candidates_fail_closed(self) -> None: other = replace(document, id="other") self.assertIsNone(_match_doc_by_path(document.path, document.text, [document, other])) + def test_discover_live_configuration_reads_running_memd(self) -> None: + class Handler(BaseHTTPRequestHandler): + def do_GET(self): + if self.path == "/v1/version": + payload = json.dumps({ + "version": "0.1.2", + "revision": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "contract": "durable-context.v1", + }).encode() + elif self.path == "/v1/workspaces/current/ai-profile": + payload = json.dumps({ + "active": { + "profile_id": "local-fast", + "profile_revision": "r1", + "embedding": {"provider": "ollama", "dimensions": 768}, + }, + }).encode() + else: + self.send_error(404) + return + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(payload))) + self.end_headers() + self.wfile.write(payload) + + def log_message(self, *args): + pass + + server = HTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + cfg = discover_live_configuration( + f"http://127.0.0.1:{server.server_port}", + "tok", + mode="vector", + ) + finally: + server.shutdown() + thread.join() + server.server_close() + self.assertEqual(cfg["engine"], "memd/0.1.2") + self.assertEqual(cfg["provider"], "ollama") + self.assertEqual(cfg["model"], "local-fast@r1") + self.assertEqual(cfg["dimension"], 768) + self.assertNotEqual(cfg["engine"], "lexical-reference") + + def test_discover_live_configuration_fails_closed_without_profile(self) -> None: + class Handler(BaseHTTPRequestHandler): + def do_GET(self): + if self.path == "/v1/version": + payload = json.dumps({"version": "0.1.2"}).encode() + else: + payload = json.dumps({"active": None, "available": []}).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(payload))) + self.end_headers() + self.wfile.write(payload) + + def log_message(self, *args): + pass + + server = HTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + with self.assertRaises(BenchmarkError): + discover_live_configuration( + f"http://127.0.0.1:{server.server_port}", + "tok", + mode="vector", + ) + finally: + server.shutdown() + thread.join() + server.server_close() + + def test_unsupported_source_kind_is_not_a_produce_failure(self) -> None: + self.assertEqual( + produce_exit_code({ + "queries": [ + {"status": "ok"}, + {"status": "error", "error_code": "unsupported_source_kind"}, + {"status": "error", "error_code": "unsupported_filter"}, + ] + }), + 0, + ) + self.assertEqual( + produce_exit_code({ + "queries": [{"status": "error", "error_code": "http_503"}] + }), + 2, + ) + if __name__ == "__main__": unittest.main()