fix(recall): discover live memd identity in the rankings producer (#175)

CHANGELOG.md

@@ -19,6 +19,7 @@ The project publishes 0.x prerelease versions; a stable release line is not yet
 
 ### Changed
 
+- The live recall producer (`python3 -m benchmarks.recall produce`) reads engine, profile, provider and embedding dimension from the running memd instead of CLI labels, scores the multilingual v1 set (English and Chinese, including image-description queries), and stays on-demand rather than a CI gate. Refs #175.
 - Rewrite the root README around a concise product promise, a verified Compose first-run path, source-built MCP setup, current capability and trust boundaries, and links to authoritative reference docs. Remove the unsupported unpublished `@bytefolk/mem-mcp@0.1.2` install path and keep its migration explicitly tracked in #153. Refs #229.
 - Index generation HTTP create/activate/rollback stay `503 execution_unavailable` with `execution_wired: false` on that body, and the same flag is now on events as well as list/status/cancel/resume/discard. Create still returns `400` for malformed JSON or an empty `profile_id` before the 503. Empty-workspace activate is refused by the same 503. Refs #174.
 - Ingest cursor locks try non-blocking exclusive locks and give up after 5s so a wedged peer becomes a warning instead of a silent hang. Refs #139.

benchmarks/recall/README.md

@@ -2,11 +2,11 @@
 
 > **Evidence boundary: producer checks do not complete the live benchmark.**
 >
-> The producer corrections tracked in #196 do not complete #175's acceptance.
-> The original #184 live-run requirement still needs a populated real memd,
-> actual embedding-provider output and saved ranking artifacts. Unit or
-> loopback HTTP fixtures establish transport behavior only. A real model-free
-> lexical run also cannot establish vector/provider quality. See
+> The producer can emit `mem.recall-rankings.v1` from a running memd and copies
+> engine/profile identity from that process. It still needs a populated real
+> memd, actual embedding-provider output, and a saved ranking artifact before
+> a real-memd baseline can land in `baselines/`. Unit or loopback HTTP fixtures
+> establish transport behavior only. See
 > [Exact remaining live prerequisites](#exact-remaining-live-prerequisites).
 
 This directory provides a small, repeatable retrieval benchmark. It is a
@@ -184,21 +184,32 @@ emits a `mem.recall-rankings.v1` file that the existing `run --rankings` path
 consumes. Latency is measured client-side per request; the `0 ms` sentinel
 warning above applies only to the offline lexical lane.
 
+Engine, provider, model/profile identity and embedding dimension are read from
+the running process (`GET /v1/version` and
+`GET /v1/workspaces/current/ai-profile`). They are not CLI flags. The artifact
+cannot be labeled `lexical-reference`. ANN identity is not advertised by those
+endpoints, so `configuration.index.kind` is recorded as `not-advertised`
+rather than guessed.
+
+The default dataset is the multilingual v1 fixture. Scoring the same run
+reports Recall@1/5/10 and nDCG@10 under `metrics.by_language` (`en`, `zh`)
+and `metrics.by_source_kind` (`text`, `image_caption`, `structured`). The
+Chinese set includes image-description queries (`q-zh-image-paraphrase`).
+Structured-memory queries cannot be served by `POST /v1/search`; they are
+recorded as `unsupported_source_kind` and do not fail the producer.
+
 ```bash
 python3 -m benchmarks.recall produce \
   --memd-url http://localhost:8080 \
   --token "$MEM_TOKEN" \
-  --dataset benchmarks/recall/data/profile-text-v1 \
   --output /tmp/live-rankings.json \
-  --dimension 768 --provider "$MEM_PROVIDER_LABEL" --model "$MEM_MODEL_LABEL" \
   --mode vector
 ```
 
-Then score the saved rankings (the default v1 baseline uses a different corpus):
+Then score the saved rankings:
 
 ```bash
 python3 -m benchmarks.recall run \
-  --dataset benchmarks/recall/data/profile-text-v1 \
   --rankings /tmp/live-rankings.json \
   --output /tmp/live-artifact.json
 ```
@@ -224,11 +235,9 @@ silently ignored.
 Vector mode sends `route=text`; it does not claim a hybrid lexical/vector or
 multimodal experiment. Lexical mode sends `route=lexical` and requires the
 server capability from #183. It emits null provider/model/dimension as required
-by the ranking schema. Provider, model, dimension and index configuration are
-operator declarations, not discovered or verified server metadata. The
-`hardware.host` value deliberately records only the producer client's
-OS/architecture, never its hostname. It is not the server's hardware inventory
-and cannot establish comparable performance conditions.
+by the ranking schema. The `hardware.host` value deliberately records only the
+producer client's OS/architecture, never its hostname. It is not the server's
+hardware inventory and cannot establish comparable performance conditions.
 
 The full v1 corpus also contains structured-memory queries. `/v1/search` cannot
 serve these; the producer records `unsupported_source_kind` and exits 2. Any
@@ -248,17 +257,20 @@ acceptance. Those remain NOT VERIFIED until a real populated memd run is saved.
    interpreting the producer's output. Direct database seeding can establish
    retrieval/transport behavior but does not verify ingestion or Worker indexing.
 3. For vector acceptance, select the same actual embedding model for corpus and
-   queries, record its dimension and profile, and independently inspect the
-   active generation/index and deployment revision. Producer labels alone do
-   not verify any of those properties.
+   queries. The producer now copies provider/model/dimension from the active
+   workspace AI profile; independently inspect the ANN index and deployment
+   revision. `configuration.index.kind=not-advertised` means those endpoints
+   do not expose ANN identity, not that no index exists.
 4. Run the documented producer command, retain its output, score the resulting
    rankings and record errors and measured client latencies. An empty/error run
    or a fake embedding provider cannot establish vector quality. A lexical run
    requires the separate #194 server capability and remains a distinct result.
-5. The bounded file-only experiment does not cover #175's bilingual image-query
-   acceptance or the full v1 structured-memory corpus. The original issue and
-   live quality acceptance must not be described as complete on this evidence.
-
-The producer remains opt-in; the normal recall CI gate runs deterministic unit
-and fixture checks only. No real-model baseline is checked in until its actual
-configuration and saved run are available for review.
+5. Score the v1 artifact's `metrics.by_language.zh` and
+   `metrics.by_source_kind.image_caption` for GOAL.md §6. Structured-memory
+   rows remain `unsupported_source_kind` because `/v1/search` is file search.
+
+The producer remains **on-demand**. `make test-recall` and CI run deterministic
+unit and fixture checks only; they never start memd or call a provider. No
+real-memd baseline is checked into `benchmarks/recall/baselines/` until a
+populated deployment's saved rankings are reviewed. That remaining #175
+acceptance bullet is still open.

benchmarks/recall/__main__.py

@@ -9,7 +9,7 @@
 
 from .dataset import load_dataset
 from .errors import BenchmarkError
-from .live_producer import produce_rankings
+from .live_producer import discover_live_configuration, produce_rankings
 from .runner import (
     compare_artifacts,
     comparison_summary,
@@ -24,6 +24,15 @@
 PACKAGE_ROOT = Path(__file__).resolve().parent
 DEFAULT_DATASET = PACKAGE_ROOT / "data" / "v1"
 DEFAULT_BASELINE = PACKAGE_ROOT / "baselines" / "lexical-reference.v1.json"
+_PRODUCE_EXPECTED_GAPS = frozenset({"unsupported_source_kind", "unsupported_filter"})
+
+
+def produce_exit_code(rankings: dict) -> int:
+    """Exit 2 only for transport/mapping failures, not dataset surface gaps."""
+    for query in rankings.get("queries", []):
+        if query.get("status") == "error" and query.get("error_code") not in _PRODUCE_EXPECTED_GAPS:
+            return 2
+    return 0
 
 
 def _parser() -> argparse.ArgumentParser:
@@ -79,13 +88,9 @@ def _parser() -> argparse.ArgumentParser:
     produce.add_argument("--output", type=Path, required=True)
     produce.add_argument("--limit", type=int, default=10)
     produce.add_argument("--timeout", type=float, default=30.0)
-    produce.add_argument("--engine", default="live-memd")
-    produce.add_argument("--dimension", type=int, default=768)
     produce.add_argument(
         "--mode", default="vector", choices=["lexical", "vector"]
     )
-    produce.add_argument("--provider", default="operator-unspecified")
-    produce.add_argument("--model", default="operator-unspecified")
     return parser
 
 
@@ -122,24 +127,37 @@ def main(argv: list[str] | None = None) -> int:
 
         if args.command == "produce":
             dataset = load_dataset(args.dataset)
+            live_config = discover_live_configuration(
+                args.memd_url,
+                args.token,
+                mode=args.mode,
+                timeout=args.timeout,
+            )
             rankings = produce_rankings(
                 dataset,
                 base_url=args.memd_url,
                 token=args.token,
                 limit=args.limit,
                 timeout=args.timeout,
-                engine_label=args.engine,
-                dimension=args.dimension,
                 mode=args.mode,
-                provider=args.provider,
-                model=args.model,
+                live_config=live_config,
             )
             write_json(args.output, rankings)
             ok_count = sum(1 for q in rankings["queries"] if q["status"] == "ok")
-            err_count = sum(1 for q in rankings["queries"] if q["status"] == "error")
-            print(f"produced rankings: {ok_count} ok, {err_count} error")
+            real_errors = [
+                q
+                for q in rankings["queries"]
+                if q.get("status") == "error" and q.get("error_code") not in _PRODUCE_EXPECTED_GAPS
+            ]
+            gap_count = sum(
+                1
+                for q in rankings["queries"]
+                if q.get("error_code") in _PRODUCE_EXPECTED_GAPS
+            )
+            print(f"produced rankings: {ok_count} ok, {len(real_errors)} error, {gap_count} unsupported")
+            print(f"engine: {rankings['engine']}")
             print(f"rankings artifact: {args.output}")
-            return 2 if err_count else 0
+            return produce_exit_code(rankings)
 
         first = run_benchmark(
             dataset_dir=args.dataset,

benchmarks/recall/live_producer.py

@@ -72,6 +72,113 @@ def _coarse_host() -> str:
         return "unknown"
 
 
+def _json_get(
+    base_url: str,
+    path: str,
+    token: str,
+    *,
+    timeout: float,
+) -> dict[str, Any]:
+    url = base_url.rstrip("/") + path
+    req = Request(url, method="GET")
+    req.add_header("Authorization", f"Bearer {token}")
+    try:
+        with urlopen(req, timeout=timeout) as resp:
+            payload = json.loads(resp.read().decode("utf-8"))
+    except HTTPError as exc:
+        raise BenchmarkError(f"GET {path} failed with HTTP {exc.code}") from exc
+    except URLError as exc:
+        raise BenchmarkError(f"GET {path} failed: {exc.reason}") from exc
+    except Exception as exc:
+        raise BenchmarkError(f"GET {path} returned unreadable JSON") from exc
+    if not isinstance(payload, dict):
+        raise BenchmarkError(f"GET {path} must return a JSON object")
+    return payload
+
+
+def discover_live_configuration(
+    base_url: str,
+    token: str,
+    *,
+    mode: str,
+    timeout: float = 30.0,
+) -> dict[str, Any]:
+    """Read engine/profile identity from a running memd, not from CLI flags."""
+    if mode not in {"lexical", "vector"}:
+        raise BenchmarkError("memd /v1/search does not expose a hybrid lexical/vector route")
+    version = _json_get(base_url, "/v1/version", token, timeout=timeout)
+    version_label = version.get("version")
+    revision = version.get("revision")
+    contract = version.get("contract")
+    if not isinstance(version_label, str) or not version_label.strip():
+        raise BenchmarkError("GET /v1/version did not return a version string")
+    engine = f"memd/{version_label.strip()}"
+    if engine == "lexical-reference":
+        raise BenchmarkError("engine must identify live memd, not lexical-reference")
+
+    profile_payload = _json_get(
+        base_url, "/v1/workspaces/current/ai-profile", token, timeout=timeout
+    )
+    active = profile_payload.get("active")
+    server: dict[str, Any] = {
+        "version": version_label.strip(),
+        "revision": revision if isinstance(revision, str) else None,
+        "contract": contract if isinstance(contract, str) else None,
+    }
+    index = {
+        "kind": "not-advertised",
+        "evidence": (
+            "GET /v1/version and GET /v1/workspaces/current/ai-profile do not "
+            "expose ANN identity"
+        ),
+    }
+    evidence = (
+        "discovered from GET /v1/version and GET /v1/workspaces/current/ai-profile"
+    )
+    if mode == "lexical":
+        return {
+            "engine": engine,
+            "provider": None,
+            "model": None,
+            "dimension": None,
+            "index": index,
+            "server": server,
+            "evidence": evidence,
+        }
+    if not isinstance(active, dict):
+        raise BenchmarkError(
+            "vector produce requires an active workspace AI profile from the running memd"
+        )
+    embedding = active.get("embedding")
+    if not isinstance(embedding, dict):
+        raise BenchmarkError("active AI profile is missing embedding stage")
+    provider = embedding.get("provider")
+    dimension = embedding.get("dimensions")
+    profile_id = active.get("profile_id")
+    profile_revision = active.get("profile_revision")
+    if not isinstance(provider, str) or not provider.strip():
+        raise BenchmarkError("active AI profile embedding.provider is empty")
+    if not isinstance(dimension, int) or isinstance(dimension, bool) or dimension <= 0:
+        raise BenchmarkError("active AI profile embedding.dimensions must be a positive integer")
+    if not isinstance(profile_id, str) or not profile_id.strip():
+        raise BenchmarkError("active AI profile_id is empty")
+    model = profile_id.strip()
+    if isinstance(profile_revision, str) and profile_revision.strip():
+        model = f"{model}@{profile_revision.strip()}"
+    server["profile_id"] = profile_id.strip()
+    if isinstance(profile_revision, str) and profile_revision.strip():
+        server["profile_revision"] = profile_revision.strip()
+    return {
+        "engine": engine,
+        "provider": provider.strip(),
+        "model": model,
+        "dimension": dimension,
+        "index": index,
+        "server": server,
+        "evidence": evidence,
+    }
+
+
 def _query_memd(
     base_url: str,
     token: str,
@@ -131,11 +238,23 @@ def produce_rankings(
     mode: str = "vector",
     provider: str = "operator-unspecified",
     model: str = "operator-unspecified",
+    live_config: dict[str, Any] | None = None,
 ) -> dict[str, Any]:
     if mode not in {"lexical", "vector"}:
         raise BenchmarkError("memd /v1/search does not expose a hybrid lexical/vector route")
     if not 1 <= limit <= 100 or not math.isfinite(timeout) or timeout <= 0:
         raise BenchmarkError("limit must be 1..100 and timeout must be positive and finite")
+    index_config: dict[str, Any] = {"kind": "operator-unspecified"}
+    evidence = "operator-declared configuration; model and index not verified by producer"
+    server_identity: dict[str, Any] | None = None
+    if live_config is not None:
+        engine_label = live_config["engine"]
+        provider = live_config["provider"]
+        model = live_config["model"]
+        dimension = live_config["dimension"]
+        index_config = dict(live_config["index"])
+        evidence = live_config["evidence"]
+        server_identity = live_config.get("server")
     if not isinstance(engine_label, str) or not engine_label.strip() or engine_label == "lexical-reference":
         raise BenchmarkError("engine must identify live memd, not lexical-reference")
     if mode == "vector" and (
@@ -236,24 +355,25 @@ def produce_rankings(
             row["error_code"] = mapping_error
         query_rows.append(row)
 
+    configuration: dict[str, Any] = {
+        "mode": mode,
+        "provider": None if mode == "lexical" else provider,
+        "model": None if mode == "lexical" else model,
+        "dimension": None if mode == "lexical" else dimension,
+        "evidence": evidence,
+        "index": index_config,
+        "search": {
+            "top_k": limit,
+            "route": "lexical" if mode == "lexical" else "text",
+            "workspace": "bound by the supplied token; not inferred from dataset labels",
+        },
+    }
+    if isinstance(server_identity, dict) and server_identity:
+        configuration["server"] = server_identity
     return {
         "schema_version": "mem.recall-rankings.v1",
         "engine": engine_label,
-        "configuration": {
-            "mode": mode,
-            "provider": None if mode == "lexical" else provider,
-            "model": None if mode == "lexical" else model,
-            "dimension": None if mode == "lexical" else dimension,
-            "evidence": "operator-declared configuration; model and index not verified by producer",
-            "index": {
-                "kind": "operator-unspecified",
-            },
-            "search": {
-                "top_k": limit,
-                "route": "lexical" if mode == "lexical" else "text",
-                "workspace": "bound by the supplied token; not inferred from dataset labels",
-            },
-        },
+        "configuration": configuration,
         "hardware": {
             "host": _coarse_host(),
         },