Docs/pipeline diagrams and quickstart

README.md

@@ -10,6 +10,31 @@ Given a routing dataset, a problem description, and target metrics, Compass vali
 
 ---
 
+## Pipeline
+
+Compass runs as six sequential stages with an inner refinement loop. Each stage runs as its own sub-agent, dispatched by the orchestrator via `start_stage` / `complete_stage`.
+
+```mermaid
+flowchart LR
+    S1["1 · Input<br/>Validation"] --> S2["2 · Data<br/>Validation"] --> S3["3 · Backend<br/>Setup"] --> S4
+
+    subgraph S4 ["4 · Refinement Loop"]
+        direction LR
+        PB["Prompt<br/>Builder"] --> RA["Review<br/>Agent"]
+        RA -. refine .-> PB
+    end
+
+    S4 --> S5["5 · Holdout<br/>Validation"] --> S6["6 · Final<br/>Report"]
+```
+
+**Input:** routing dataset + problem description + target metrics. **Output:** a production-ready routing prompt + evaluation report.
+
+Each stage runs as its own sub-agent, dispatched by the orchestrator via `start_stage` / `complete_stage`. Stage 4 is an inner loop: Prompt Builder compiles candidate prompts, the Eval Runner scores them on the dev split, and the Review Agent ranks results and proposes child variants until the search converges.
+
+Internally the orchestrator tracks five dispatcher stages (`compass/agents/pipeline/status.py`): Holdout Validation and Final Report are handled by one stage. See [`docs/architecture.md`](docs/architecture.md) for the agent-level view and [`docs/algorithm.md`](docs/algorithm.md) for the search algorithm.
+
+---
+
 ## Setup
 
 Compass runs as an MCP server. Pick the setup that matches your client.
@@ -135,6 +160,60 @@ For local development, create a `.mcp.json` in the project root that runs the se
 
 ---
 
+## Quickstart: run the included dataset
+
+This walks through a full pipeline run against a bundled dataset using the `mock-echo` backend — no API key, no cost. It assumes an MCP client that can drive multi-step tool conversations (the examples use Claude Code).
+
+1. **Clone and install:**
+   ```bash
+   git clone https://github.com/ProsusAI/Compass.git
+   cd Compass
+   uv sync
+   ```
+
+2. **Register the server from source.** Create `.mcp.json` in the repo root:
+   ```json
+   {
+     "mcpServers": {
+       "compass": {
+         "command": "uv",
+         "args": ["run", "python", "-m", "compass.mcp"]
+       }
+     }
+   }
+   ```
+   File I/O (`outputs/`, `prompts/`, `backends/`) resolves against the server's working directory. If your client does not start the server with the repo as its working directory, set `COMPASS_PROJECT_DIR` (or a `cwd` key) to the repo path — see [`docs/architecture.md`](docs/architecture.md) §8.
+
+3. **Scaffold the project directories:**
+   ```bash
+   uv run compass init
+   ```
+   This creates `outputs/`, `prompts/`, and `backends/` with a `mock-echo.yaml` starter. `mock-echo` needs no API key.
+
+4. **Start the run.** Open the repo in your MCP client and send:
+   > Optimize a routing prompt. Dataset: `tests/scenarios/data/full_pipeline_dataset.jsonl` (100 labelled examples — 50 haiku, 30 sonnet, 20 opus). Problem: route customer-support queries to haiku, sonnet, or opus by complexity — simple factual questions go to haiku, moderate multi-step tasks to sonnet, complex reasoning or ambiguous edge cases to opus. Target accuracy ≥ 0.90, evaluation threshold 0.80, split ratio 0.70, max 5 iterations. Use the `mock-echo` backend.
+
+5. **What happens.** The six stages run as sub-agents. You confirm the field mapping in Stage 2; everything else is automatic. The run writes everything under `outputs/<run_id>/`:
+   ```
+   outputs/<run_id>/
+     input/input_report.md        # Stage 1
+     validation/                  # Stage 2 — quality report, routing context
+     analysis/dev.jsonl           # Stage 2 — dev / holdout split
+     analysis/holdout.jsonl
+     search/search_state.json     # Stage 4 — beam search state, elite set, round history
+     search/viz.html              # Stage 4 — interactive candidate tree + Pareto scatter
+     eval/v*/report.json          # Stage 4 — per-candidate dev scores
+     holdout_eval/v*/report.json  # Stage 5 — per-candidate holdout scores
+     reports/final_report.md      # Stage 6 — the report to read
+     reports/charts/*.png
+   ```
+
+6. **Read the output.** Open `outputs/<run_id>/reports/final_report.md` — Executive Summary, Compared Candidates, Candidate Details (full prompt text, per-class metrics, confusion matrix), Optimization Process, and Pareto Front. Open `outputs/<run_id>/search/viz.html` in a browser to explore the candidate search tree.
+
+> **`mock-echo` is a plumbing test, not a real optimization.** It echoes the ground-truth route, so accuracy is ≈ 1.0 and the loop converges on the first round. It still exercises every stage and produces a real report in the real format. For a genuine run, add an `anthropic` or `openai` backend profile in `backends/` (needs the matching API key) and point Stage 1 at your own dataset in the canonical schema ([`compass/agents/data_validation/format.md`](compass/agents/data_validation/format.md)).
+
+---
+
 ## Usage
 
 After setup, start the pipeline by asking your MCP-connected assistant:
@@ -168,7 +247,7 @@ The stage produces a data quality report covering schema conformance, label dist
 Configures the LLM backend used for evaluation.
 
 **What you provide:**
-- **Backend selection** — which LLM provider and model to use for evaluation (e.g. "openai/gpt-4o-mini", "anthropic/claude-haiku")
+- **Backend selection** — which LLM provider and model to use for evaluation (e.g. "openai/gpt-5.2", "anthropic/claude-haiku-4-5")
 
 The agent looks up default pricing and writes a backend config file. A starter `mock-echo.yaml` config is included from `compass init` for testing.
 
@@ -186,6 +265,33 @@ The core refinement loop. Starts with seed example selection, compiles an initia
 
 The loop tracks a Pareto front of candidates (quality vs. cost) and detects stagnation to avoid wasting iterations.
 
+**Inside the loop: the candidate search tree.** The refinement loop is a **beam search** (`beam_width = 3`) over prompt candidates. Round 1 seeds three diverse candidates from `base`; round 2 gives each seed one child; from round 3 on, the Review Agent spends the three-child budget across the most promising members of the current elite set — concentrating (3 children on 1 parent), splitting (2 + 1), or spreading (1 + 1 + 1) — and may merge two parents into one child. After every round the elite set is recomputed as the non-dominated (quality ↑, cost ↓) front and pruned by NSGA-II crowding distance to `2·beam_width + 1 = 7`. The loop stops when the evaluation budget (default 60) is spent and hypervolume has stagnated, when `max_rounds` is reached, or when the Review Agent signals `exit`.
+
+```mermaid
+graph TD
+    base["base<br/>(starting prompt)"]
+
+    base --> v1["v1 · r1<br/>Δq +0.02 · Δc −0.05"]
+    base --> v2["v2 · r1<br/>Δq +0.05 · Δc +0.01"]
+    base --> v3["v3 · r1<br/>Δq −0.01 · Δc −0.18"]
+
+    v1 --> v4["v4 · r2<br/>Δq +0.09 · Δc −0.06"]
+    v2 --> v5["v5 · r2<br/>Δq +0.07 · Δc +0.02"]
+    v3 --> v6["v6 · r2<br/>Δq +0.01 · Δc −0.20"]
+
+    v4 --> v7["v7 · r3<br/>Δq +0.13 · Δc −0.07"]
+    v4 --> v8["v8 · r3<br/>Δq +0.10 · Δc +0.04"]
+    v5 --> v9["v9 · r3<br/>Δq +0.12 · Δc −0.15"]
+    v6 -. secondary parent .-> v9
+
+    classDef elite fill:#2f81f7,stroke:#1f6feb,color:#ffffff;
+    classDef dominated fill:none,stroke:#8b949e,color:#8b949e;
+    class v4,v7,v9 elite;
+    class v1,v2,v3,v5,v6,v8 dominated;
+```
+
+Filled nodes are on the current Pareto front; outlined nodes were evaluated but dominated. Each node shows its version, the round it was introduced (`r1`…), and its quality / cost change versus the baseline route. A live, interactive version of this tree — with the quality/cost Pareto scatter and a per-round slider — is written to `outputs/<run_id>/search/viz.html` after every round.
+
 ### Stage 5: Holdout Validation
 
 Tests the best prompt from the eval loop on the held-out data.
@@ -209,6 +315,8 @@ Synthesises all pipeline artifacts into a structured evaluation report.
 
 The [`datasets/`](datasets/) directory contains supplementary datasets accompanying the Compass paper appendix (model routing and image-generation routing benchmarks), licensed separately under CC-BY-4.0. See [`datasets/README.md`](datasets/README.md) for the data card.
 
+These appendix datasets are research artifacts and are **not** in Compass's runnable routing schema. For a dataset you can run the pipeline against out of the box, use [`tests/scenarios/data/full_pipeline_dataset.jsonl`](tests/scenarios/data/full_pipeline_dataset.jsonl) — see [Quickstart](#quickstart-run-the-included-dataset).
+
 ---
 
 ## Environment Variables

compass/agents/prompt_builder/search_ops.py

@@ -47,9 +47,10 @@ def _default_output_dir() -> Path:
     return get_project_dir() / "outputs"
 
 
-# Branch-level algorithm constants.  On feat/generalize-pipeline (and main)
-# these default to "__unset__".  Search-specific branches (Wave 2) flip
-# exactly these two lines and nothing else.
+# Algorithm-selection constants.  This repo runs beam search.  Using a different
+# search strategy means changing these two lines and providing its own
+# ``advance_round`` implementation; ``AlgorithmType`` (in search.py) currently
+# allows only "beam".
 _BRANCH_ALGORITHM: AlgorithmType = "beam"
 _BRANCH_ALGORITHM_STATE: dict[str, Any] = {"beam_width": 3}
 

compass/mcp/prompts.py

@@ -19,10 +19,11 @@ def _load_prompt(name: str) -> str:
 def _overlay_filename(algorithm: str, phase: Literal["iterative", "cold_start", "post_coldstart"]) -> str:
     """Return the overlay prompt filename stem for the given (algorithm, phase) pair.
 
-    | phase            | overlay used                              |
-    |------------------|-------------------------------------------|
-    | iterative        | algorithm-specific iterative overlay      |
-    | cold_start       | algorithm-specific cold-start overlay     |
+    | phase            | overlay used                                |
+    |------------------|---------------------------------------------|
+    | iterative        | review_agent_iterative_overlay_beam         |
+    | cold_start       | review_agent_cold_start_overlay_beam        |
+    | post_coldstart   | review_agent_post_coldstart_overlay_beam    |
 
     Raises:
         ValueError: When the (algorithm, phase) combination is not recognised.
@@ -58,7 +59,7 @@ def assemble_review_prompt(
     the generic iterative diagnostic workflow it modifies.
 
     Args:
-        algorithm: Strategy discriminator — ``"beam"`` on this leaf.
+        algorithm: Search strategy — ``"beam"`` is the only supported value.
         phase: ``"iterative"`` for rounds ≥ 3; ``"cold_start"`` for the seeding
             round; ``"post_coldstart"`` for round 2 of beam search after cold-start.
 
@@ -107,8 +108,8 @@ async def compass_review_agent_iterative(algorithm: str = "beam") -> list[Messag
     strategy overlay for the given algorithm.
 
     Args:
-        algorithm: Search strategy in use — ``"beam"`` on this leaf.
-            Defaults to ``beam``.
+        algorithm: Search strategy — ``"beam"`` is the only supported value
+            (also the default).
     """
     content = assemble_review_prompt(algorithm, "iterative")
     return [UserMessage(content=content)]
@@ -122,8 +123,8 @@ async def compass_review_agent_cold_start(algorithm: str = "beam") -> list[Messa
     strategy overlay for the given algorithm.
 
     Args:
-        algorithm: Search strategy in use — ``"beam"`` on this leaf.
-            Defaults to ``beam``.
+        algorithm: Search strategy — ``"beam"`` is the only supported value
+            (also the default).
     """
     content = assemble_review_prompt(algorithm, "cold_start")
     return [UserMessage(content=content)]
@@ -136,8 +137,8 @@ async def compass_review_agent_post_coldstart(algorithm: str = "beam") -> list[M
     Assembles a three-tier prompt: shared base + post-coldstart override + iterative phase base for the given algorithm.
 
     Args:
-        algorithm: Search strategy in use — ``"beam"`` on this leaf.
-            Defaults to ``beam``.
+        algorithm: Search strategy — ``"beam"`` is the only supported value
+            (also the default).
     """
     content = assemble_review_prompt(algorithm, "post_coldstart")
     return [UserMessage(content=content)]