diff --git a/README.md b/README.md index 77eaccf..7f0d7aa 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,31 @@ Given a routing dataset, a problem description, and target metrics, Compass vali --- +## Pipeline + +Compass runs as six sequential stages with an inner refinement loop. Each stage runs as its own sub-agent, dispatched by the orchestrator via `start_stage` / `complete_stage`. + +```mermaid +flowchart LR + S1["1 · Input
Validation"] --> S2["2 · Data
Validation"] --> S3["3 · Backend
Setup"] --> S4 + + subgraph S4 ["4 · Refinement Loop"] + direction LR + PB["Prompt
Builder"] --> RA["Review
Agent"] + RA -. refine .-> PB + end + + S4 --> S5["5 · Holdout
Validation"] --> S6["6 · Final
Report"] +``` + +**Input:** routing dataset + problem description + target metrics. **Output:** a production-ready routing prompt + evaluation report. + +Each stage runs as its own sub-agent, dispatched by the orchestrator via `start_stage` / `complete_stage`. Stage 4 is an inner loop: Prompt Builder compiles candidate prompts, the Eval Runner scores them on the dev split, and the Review Agent ranks results and proposes child variants until the search converges. + +Internally the orchestrator tracks five dispatcher stages (`compass/agents/pipeline/status.py`): Holdout Validation and Final Report are handled by one stage. See [`docs/architecture.md`](docs/architecture.md) for the agent-level view and [`docs/algorithm.md`](docs/algorithm.md) for the search algorithm. + +--- + ## Setup Compass runs as an MCP server. Pick the setup that matches your client. @@ -135,6 +160,60 @@ For local development, create a `.mcp.json` in the project root that runs the se --- +## Quickstart: run the included dataset + +This walks through a full pipeline run against a bundled dataset using the `mock-echo` backend — no API key, no cost. It assumes an MCP client that can drive multi-step tool conversations (the examples use Claude Code). + +1. **Clone and install:** + ```bash + git clone https://github.com/ProsusAI/Compass.git + cd Compass + uv sync + ``` + +2. **Register the server from source.** Create `.mcp.json` in the repo root: + ```json + { + "mcpServers": { + "compass": { + "command": "uv", + "args": ["run", "python", "-m", "compass.mcp"] + } + } + } + ``` + File I/O (`outputs/`, `prompts/`, `backends/`) resolves against the server's working directory. If your client does not start the server with the repo as its working directory, set `COMPASS_PROJECT_DIR` (or a `cwd` key) to the repo path — see [`docs/architecture.md`](docs/architecture.md) §8. + +3. **Scaffold the project directories:** + ```bash + uv run compass init + ``` + This creates `outputs/`, `prompts/`, and `backends/` with a `mock-echo.yaml` starter. `mock-echo` needs no API key. + +4. **Start the run.** Open the repo in your MCP client and send: + > Optimize a routing prompt. Dataset: `tests/scenarios/data/full_pipeline_dataset.jsonl` (100 labelled examples — 50 haiku, 30 sonnet, 20 opus). Problem: route customer-support queries to haiku, sonnet, or opus by complexity — simple factual questions go to haiku, moderate multi-step tasks to sonnet, complex reasoning or ambiguous edge cases to opus. Target accuracy ≥ 0.90, evaluation threshold 0.80, split ratio 0.70, max 5 iterations. Use the `mock-echo` backend. + +5. **What happens.** The six stages run as sub-agents. You confirm the field mapping in Stage 2; everything else is automatic. The run writes everything under `outputs//`: + ``` + outputs// + input/input_report.md # Stage 1 + validation/ # Stage 2 — quality report, routing context + analysis/dev.jsonl # Stage 2 — dev / holdout split + analysis/holdout.jsonl + search/search_state.json # Stage 4 — beam search state, elite set, round history + search/viz.html # Stage 4 — interactive candidate tree + Pareto scatter + eval/v*/report.json # Stage 4 — per-candidate dev scores + holdout_eval/v*/report.json # Stage 5 — per-candidate holdout scores + reports/final_report.md # Stage 6 — the report to read + reports/charts/*.png + ``` + +6. **Read the output.** Open `outputs//reports/final_report.md` — Executive Summary, Compared Candidates, Candidate Details (full prompt text, per-class metrics, confusion matrix), Optimization Process, and Pareto Front. Open `outputs//search/viz.html` in a browser to explore the candidate search tree. + +> **`mock-echo` is a plumbing test, not a real optimization.** It echoes the ground-truth route, so accuracy is ≈ 1.0 and the loop converges on the first round. It still exercises every stage and produces a real report in the real format. For a genuine run, add an `anthropic` or `openai` backend profile in `backends/` (needs the matching API key) and point Stage 1 at your own dataset in the canonical schema ([`compass/agents/data_validation/format.md`](compass/agents/data_validation/format.md)). + +--- + ## Usage After setup, start the pipeline by asking your MCP-connected assistant: @@ -168,7 +247,7 @@ The stage produces a data quality report covering schema conformance, label dist Configures the LLM backend used for evaluation. **What you provide:** -- **Backend selection** — which LLM provider and model to use for evaluation (e.g. "openai/gpt-4o-mini", "anthropic/claude-haiku") +- **Backend selection** — which LLM provider and model to use for evaluation (e.g. "openai/gpt-5.2", "anthropic/claude-haiku-4-5") The agent looks up default pricing and writes a backend config file. A starter `mock-echo.yaml` config is included from `compass init` for testing. @@ -186,6 +265,33 @@ The core refinement loop. Starts with seed example selection, compiles an initia The loop tracks a Pareto front of candidates (quality vs. cost) and detects stagnation to avoid wasting iterations. +**Inside the loop: the candidate search tree.** The refinement loop is a **beam search** (`beam_width = 3`) over prompt candidates. Round 1 seeds three diverse candidates from `base`; round 2 gives each seed one child; from round 3 on, the Review Agent spends the three-child budget across the most promising members of the current elite set — concentrating (3 children on 1 parent), splitting (2 + 1), or spreading (1 + 1 + 1) — and may merge two parents into one child. After every round the elite set is recomputed as the non-dominated (quality ↑, cost ↓) front and pruned by NSGA-II crowding distance to `2·beam_width + 1 = 7`. The loop stops when the evaluation budget (default 60) is spent and hypervolume has stagnated, when `max_rounds` is reached, or when the Review Agent signals `exit`. + +```mermaid +graph TD + base["base
(starting prompt)"] + + base --> v1["v1 · r1
Δq +0.02 · Δc −0.05"] + base --> v2["v2 · r1
Δq +0.05 · Δc +0.01"] + base --> v3["v3 · r1
Δq −0.01 · Δc −0.18"] + + v1 --> v4["v4 · r2
Δq +0.09 · Δc −0.06"] + v2 --> v5["v5 · r2
Δq +0.07 · Δc +0.02"] + v3 --> v6["v6 · r2
Δq +0.01 · Δc −0.20"] + + v4 --> v7["v7 · r3
Δq +0.13 · Δc −0.07"] + v4 --> v8["v8 · r3
Δq +0.10 · Δc +0.04"] + v5 --> v9["v9 · r3
Δq +0.12 · Δc −0.15"] + v6 -. secondary parent .-> v9 + + classDef elite fill:#2f81f7,stroke:#1f6feb,color:#ffffff; + classDef dominated fill:none,stroke:#8b949e,color:#8b949e; + class v4,v7,v9 elite; + class v1,v2,v3,v5,v6,v8 dominated; +``` + +Filled nodes are on the current Pareto front; outlined nodes were evaluated but dominated. Each node shows its version, the round it was introduced (`r1`…), and its quality / cost change versus the baseline route. A live, interactive version of this tree — with the quality/cost Pareto scatter and a per-round slider — is written to `outputs//search/viz.html` after every round. + ### Stage 5: Holdout Validation Tests the best prompt from the eval loop on the held-out data. @@ -209,6 +315,8 @@ Synthesises all pipeline artifacts into a structured evaluation report. The [`datasets/`](datasets/) directory contains supplementary datasets accompanying the Compass paper appendix (model routing and image-generation routing benchmarks), licensed separately under CC-BY-4.0. See [`datasets/README.md`](datasets/README.md) for the data card. +These appendix datasets are research artifacts and are **not** in Compass's runnable routing schema. For a dataset you can run the pipeline against out of the box, use [`tests/scenarios/data/full_pipeline_dataset.jsonl`](tests/scenarios/data/full_pipeline_dataset.jsonl) — see [Quickstart](#quickstart-run-the-included-dataset). + --- ## Environment Variables diff --git a/compass/agents/prompt_builder/search_ops.py b/compass/agents/prompt_builder/search_ops.py index 639f9b9..bf74599 100644 --- a/compass/agents/prompt_builder/search_ops.py +++ b/compass/agents/prompt_builder/search_ops.py @@ -47,9 +47,10 @@ def _default_output_dir() -> Path: return get_project_dir() / "outputs" -# Branch-level algorithm constants. On feat/generalize-pipeline (and main) -# these default to "__unset__". Search-specific branches (Wave 2) flip -# exactly these two lines and nothing else. +# Algorithm-selection constants. This repo runs beam search. Using a different +# search strategy means changing these two lines and providing its own +# ``advance_round`` implementation; ``AlgorithmType`` (in search.py) currently +# allows only "beam". _BRANCH_ALGORITHM: AlgorithmType = "beam" _BRANCH_ALGORITHM_STATE: dict[str, Any] = {"beam_width": 3} diff --git a/compass/mcp/prompts.py b/compass/mcp/prompts.py index bd52fe0..09555e3 100644 --- a/compass/mcp/prompts.py +++ b/compass/mcp/prompts.py @@ -19,10 +19,11 @@ def _load_prompt(name: str) -> str: def _overlay_filename(algorithm: str, phase: Literal["iterative", "cold_start", "post_coldstart"]) -> str: """Return the overlay prompt filename stem for the given (algorithm, phase) pair. - | phase | overlay used | - |------------------|-------------------------------------------| - | iterative | algorithm-specific iterative overlay | - | cold_start | algorithm-specific cold-start overlay | + | phase | overlay used | + |------------------|---------------------------------------------| + | iterative | review_agent_iterative_overlay_beam | + | cold_start | review_agent_cold_start_overlay_beam | + | post_coldstart | review_agent_post_coldstart_overlay_beam | Raises: ValueError: When the (algorithm, phase) combination is not recognised. @@ -58,7 +59,7 @@ def assemble_review_prompt( the generic iterative diagnostic workflow it modifies. Args: - algorithm: Strategy discriminator — ``"beam"`` on this leaf. + algorithm: Search strategy — ``"beam"`` is the only supported value. phase: ``"iterative"`` for rounds ≥ 3; ``"cold_start"`` for the seeding round; ``"post_coldstart"`` for round 2 of beam search after cold-start. @@ -107,8 +108,8 @@ async def compass_review_agent_iterative(algorithm: str = "beam") -> list[Messag strategy overlay for the given algorithm. Args: - algorithm: Search strategy in use — ``"beam"`` on this leaf. - Defaults to ``beam``. + algorithm: Search strategy — ``"beam"`` is the only supported value + (also the default). """ content = assemble_review_prompt(algorithm, "iterative") return [UserMessage(content=content)] @@ -122,8 +123,8 @@ async def compass_review_agent_cold_start(algorithm: str = "beam") -> list[Messa strategy overlay for the given algorithm. Args: - algorithm: Search strategy in use — ``"beam"`` on this leaf. - Defaults to ``beam``. + algorithm: Search strategy — ``"beam"`` is the only supported value + (also the default). """ content = assemble_review_prompt(algorithm, "cold_start") return [UserMessage(content=content)] @@ -136,8 +137,8 @@ async def compass_review_agent_post_coldstart(algorithm: str = "beam") -> list[M Assembles a three-tier prompt: shared base + post-coldstart override + iterative phase base for the given algorithm. Args: - algorithm: Search strategy in use — ``"beam"`` on this leaf. - Defaults to ``beam``. + algorithm: Search strategy — ``"beam"`` is the only supported value + (also the default). """ content = assemble_review_prompt(algorithm, "post_coldstart") return [UserMessage(content=content)] diff --git a/docs/algorithm.md b/docs/algorithm.md index af20a4a..cec7fe2 100644 --- a/docs/algorithm.md +++ b/docs/algorithm.md @@ -1,205 +1,273 @@ -# EMOSA: Decomposition-Based Multi-Objective Simulated Annealing +# Beam Search over Prompt Candidates -This file is the source of truth for anyone touching the search engine. The algorithm is a clean adaptation of **EMOSA** (Li & Landa-Silva 2011) for LLM-driven prompt search: decomposition-based multi-objective SA with Tchebycheff scalarization, K parallel trajectories, per-trajectory Metropolis acceptance, EMOSA's neighborhood replacement, and a plain non-dominated archive. When reading or modifying `compass/agents/prompt_builder/annealing.py`, `search_ops.py`, `compass/agents/review/preprocessor.py`, or the Review Agent prompts, consult this document first. +This file is the source of truth for anyone touching the Stage-4 search engine. +The algorithm is a **multi-objective beam search**: each round expands the current elite set +into `beam_width` new prompt candidates, evaluates them on the dev split, and keeps the +non-dominated (quality, cost) front — pruned for spread by NSGA-II crowding distance. The +mutation operator is the LLM Review Agent + Prompt Builder pipeline, not a random perturbation. ---- - -## Algorithm at a glance +When reading or modifying [`compass/agents/prompt_builder/search.py`](../compass/agents/prompt_builder/search.py), +[`compass/agents/prompt_builder/search_ops.py`](../compass/agents/prompt_builder/search_ops.py), +[`compass/agents/review/preprocessor.py`](../compass/agents/review/preprocessor.py), or the +Review Agent prompts, consult this document first. -| Component | Technique | Primary reference | -|-----------|-----------|-------------------| -| Sub-problem decomposition | Tchebycheff scalarization with K weight vectors | Zhang & Li 2007 (MOEA/D); Li & Landa-Silva 2011 (EMOSA) | -| Per-sub-problem state | K parallel current-solutions with independent energies | Li & Landa-Silva 2011 | -| Acceptance criterion | Per-trajectory Metropolis on Tchebycheff energy delta | Kirkpatrick et al. 1983; Li & Landa-Silva 2011 | -| Neighborhood replacement | Every generated child (accepted or not) replaces any neighbor whose current is scalarized-worse; no Metropolis gate | Li & Landa-Silva 2011 | -| Archive | Plain non-dominated set; dominance filter only; no size limits | Li & Landa-Silva 2011 | -| Mutation operator | LLM Review Agent + Prompt Builder pipeline (per-sub-problem-aware) | This codebase | -| Convergence | Temperature floor, eval budget, Review Agent LoopSignal exit | — | +> The search strategy is chosen by two constants in `search_ops.py` (see +> [Configuration](#configuration)); this repo sets them to beam search, which is what this +> document describes. Using a different strategy means changing those constants and adding +> its own `advance_round` implementation. --- -## Tchebycheff decomposition +## Algorithm at a glance -The scalar energy for a solution `x` under weight vector `λ = (λ_q, λ_c)` is: +| Component | Technique | Where | +|-----------|-----------|-------| +| Expansion | `beam_width` children per round, allocated across elite parents by the Review Agent | `review_agent_iterative_overlay_beam.md` | +| Mutation operator | LLM Review Agent (identify failure → hypothesise → directive) + Prompt Builder compile | `compass/agents/review/`, `compass/agents/prompts/prompt_builder_system.md` | +| Selection | Pareto dominance on (quality ↑, cost ↓) | `dominates`, `compute_pareto_front` in `search.py` | +| Diversity / pruning | NSGA-II crowding distance, prune to `2·beam_width + 1`, endpoints protected | `crowding_distance`, `prune_to_size` in `search.py` | +| Progress metric | 2-D hypervolume of the front vs a worst-seen reference point | `compute_hypervolume` in `search.py` | +| Stagnation | relative hypervolume improvement ≤ `epsilon` | `advance_round_beam` in `search_ops.py` | +| Convergence | eval budget spent **and** stagnation ≥ `convergence_limit`, or `max_rounds`, or Review Agent `LoopSignal(action="exit")` | `advance_round_beam` | -``` -E(x; λ) = max( - λ_q · norm_q(x), - λ_c · norm_c(x) -) -``` +--- -where: -- `norm_q(x) = (nadir_q − quality(x)) / (nadir_q − ideal_q)` — 0 at the ideal, 1 at the nadir -- `norm_c(x) = (cost(x) − ideal_c) / (nadir_c − ideal_c)` — 0 at the ideal, 1 at the nadir -- `ideal_q`, `ideal_c` are the best quality and lowest cost seen across all evaluations -- `nadir_q`, `nadir_c` are the worst quality and highest cost seen +## Configuration -Energy `E = 0` is the ideal corner; higher is worse. A solution's **binding axis** is the index `i` where `λ_i × norm_i` is largest — the term that dominates the `max`. Equivalently: +Two module-level constants in `search_ops.py` select the strategy. In this repo: -``` -binding_axis = argmax_i ( λ_i · (f_i − ideal_i) / (nadir_i − ideal_i) ) +```python +_BRANCH_ALGORITHM: AlgorithmType = "beam" +_BRANCH_ALGORITHM_STATE: dict[str, Any] = {"beam_width": 3} ``` -**Why Tchebycheff instead of weighted sum:** A weighted-sum scalarization `E = λ_q · g_q + λ_c · g_c` cannot produce solutions on concave regions of the Pareto front regardless of the weight choice. The Tchebycheff aggregation `E = max(λ_q · g_q, λ_c · g_c)` can reach any point on the Pareto front — convex or concave — by varying `λ`. This property is critical when the quality/cost tradeoff surface is not convex. +`init_search_state` copies `_BRANCH_ALGORITHM_STATE` into `SearchState.algorithm_state`. +`advance_step` (MCP tool) dispatches to `advance_round` → `advance_round_beam`. -**Normalization:** `normalize_objectives` (in `annealing.py`) maps raw quality and cost into `[0, 1]` relative to `ideal_point` and `nadir_point`. After normalization, `norm_q = 0` means the candidate equals the best-ever quality; `norm_q = 1` means it equals the worst-ever quality. Cost is analogous. +`SearchState` fields that govern the loop (defaults from `search.py`): -**Concrete example:** Suppose `ideal = (0.90 quality, 0.02 cost)`, `nadir = (0.70, 0.10)`. A candidate with quality 0.80 and cost 0.06 gives `norm_q = (0.90 − 0.80)/(0.90 − 0.70) = 0.50`, `norm_c = (0.06 − 0.02)/(0.10 − 0.02) = 0.50`. For trajectory 0 (`λ = (0.9, 0.1)`): `E = max(0.9 × 0.50, 0.1 × 0.50) = max(0.45, 0.05) = 0.45`; binding axis = quality. For trajectory 4 (`λ = (0.1, 0.9)`): `E = max(0.1 × 0.50, 0.9 × 0.50) = max(0.05, 0.45) = 0.45`; binding axis = cost. For a cost-efficient candidate (quality 0.72, cost 0.03): `norm_q = 0.90`, `norm_c = 0.125`; trajectory 0 energy = `max(0.81, 0.013) = 0.81`; trajectory 4 energy = `max(0.09, 0.113) = 0.113`. Trajectory 4 sees it as much better. +| Field | Default | Meaning | +|---|---|---| +| `algorithm_state["beam_width"]` | `3` | children generated per round | +| `evaluation_budget` | `60` | total candidate evaluations before convergence is allowed | +| `max_rounds` | `50` | hard round cap | +| `stagnation_limit` | `3` | rounds without hypervolume progress before `mutation_mode` may flip / backtrack cues fire | +| `convergence_limit` | `5` | consecutive stagnant rounds required to converge (must be `> stagnation_limit`) | +| `epsilon` | `0.001` | minimum *relative* hypervolume improvement that counts as progress | +| `algorithm_state["epsilon_min"]` | `0.0005` | floor for `epsilon` after tightening | +| `algorithm_state["backtrack_threshold"]` | `2` | stagnation count at which the round summary sets `backtracking = True` | +| `mutation_mode` | `"targeted"` | `targeted` (faithful edits) vs `exploratory` (structural rewrite) | -**Implementation:** `compute_tchebycheff_energy` in `compass/agents/prompt_builder/annealing.py`. The function normalizes objectives via `normalize_objectives`, then applies the max aggregation using `ideal_point` and `nadir_point` from `AnnealingState`. - -**Citations:** Zhang, Q. & Li, H. (2007). *MOEA/D*. IEEE TEC 11(6):712–731. Li, H. & Landa-Silva, D. (2011). *EMOSA*. Evolutionary Computation 19(4):561–595. +`search_state.json` is auto-created at Stage 4 entry by `_ensure_stage4_search_state` +(called from `_next_action_for_stage_4` in `status.py`), so cold-start sub-agents always find +a real `SearchState` on disk. --- -## Weight vectors and trajectories - -`compute_weight_vectors(num_trajectories)` in `annealing.py` generates K evenly-spaced weight vectors spanning `λ_q ∈ [0.1, 0.9]` with `λ_c = 1 − λ_q`. Quality-focused trajectories (`λ_q` close to 0.9) are listed first. For `num_trajectories = 1`, the single vector is `(0.5, 0.5)`; for `num_trajectories = 2`, vectors are `(0.9, 0.1)` and `(0.1, 0.9)`. +## The candidate tree -For the default `num_trajectories = 5`, the generated weight vectors are: +Every candidate is a `Candidate` record (`search.py`): -| Trajectory ID | λ_q | λ_c | Sub-problem emphasis | -|---------------|-----|-----|----------------------| -| 0 | 0.9 | 0.1 | Heavy quality focus | -| 1 | 0.7 | 0.3 | Quality-leaning | -| 2 | 0.5 | 0.5 | Balanced (knee region) | -| 3 | 0.3 | 0.7 | Cost-leaning | -| 4 | 0.1 | 0.9 | Heavy cost focus | - -Each trajectory maintains its own `current_solution` (the `prompt_version` of the candidate it currently holds) and `current_energy` (the Tchebycheff energy of that candidate under its weight vector). These are stored in `TrajectoryState` inside `AnnealingState`. This design runs K independent SA chains in parallel, one per sub-problem — characteristic of EMOSA and MOEA/D extended with per-sub-problem SA. +```python +prompt_version: str # "v1", "v2", … (monotonic, from SearchState.next_variant_seq) +parent_version: str | None # "base" for cold-start seeds, else a prior vN +secondary_parent_version: str | None # set only for two-parent merges (round ≥ 3) +quality_score: float # = metrics "quality_change": signed fraction vs the baseline route +cost: float # = metrics "cost_change_with_overhead": signed fraction vs baseline +round_introduced: int +example_ids: list[str] # few-shot ids embedded in this prompt +eval_status: "pending" | "running" | "complete" | "failed" | None +``` -The `acceptance_history` field in `TrajectoryState` records the last 5 accept/reject decisions for that trajectory, enabling the Review Agent to detect when a trajectory is stuck (consistently rejecting moves). +The root is the sentinel string `"base"` (`INITIAL_PARENT_VERSION` in +`compass/agents/review/models.py`). `parent_version` / `secondary_parent_version` edges form +the search tree (a DAG once merges appear). Lineage of dominated / evicted candidates is +preserved in the append-only `outputs//search/candidate_archive.json`; the live tree is +rendered to `outputs//search/viz.html` after every state mutation. -The number of trajectories (`num_trajectories`) is configured in `init_annealing_state` (called by `init_search_state_tool`). The default is 5; higher values increase Pareto front coverage at the cost of more evaluations per round. +**Scores are deltas, not accuracy.** `quality_score` and `cost` are signed fractions relative +to the dataset's baseline route, produced by `compute_cost_quality_change` in +[`compass/eval/metrics.py`](../compass/eval/metrics.py). Classifier accuracy is reported in the +score report and shown in the viz tooltip, but it is not the optimization objective. --- -## Per-sub-problem SA acceptance +## Round structure -Each trajectory applies the standard SA acceptance rule to its own Tchebycheff energy: +Each round follows the fixed Prompt Builder tool sequence +(`compass/agents/prompts/prompt_builder_system.md`): -``` -if Δ_E ≤ 0: - accept # improvement always accepted -else: - accept with probability exp(−Δ_E / T) -``` +> `register_candidate` (per candidate) → `run_batch_eval` (once) → `record_eval_result` +> (per succeeded entry) → `advance_step`. Never reorder. Never reuse a version number. -**Implementation:** `metropolis_accept(delta_e, temperature)` in `annealing.py`. Applied inside `_advance_emosa_search` in `search_ops.py` for each trajectory independently against that trajectory's own `temperature`. When a trajectory generates M children (`children_per_trajectory > 1`), the acceptance rule is applied to each child separately and the accepted child with the lowest energy is kept ("Metropolis-then-best-of-accepted" semantics). +`run_batch_eval` evaluates all of a round's candidates concurrently under one shared rate +limiter (`compass/eval/batch_eval.py`). -**Per-trajectory adaptive cooling:** each trajectory holds its own `temperature` and `alpha` on `TrajectoryState`. After Metropolis (and neighborhood replacement) each round, every trajectory that attempted a step adjusts its temperature via `adaptive_cool` based on its recent acceptance rate against a target band: +### Round 1 — cold start -- if rate > `target_acceptance_high` (default 0.6) → `T ← T × α^cooling_exp_fast` (cool faster, default exp 1.5) -- if rate < `target_acceptance_low` (default 0.4) → `T ← T × α^cooling_exp_slow` (cool slower, default exp 0.5) -- otherwise → `T ← T × α` (default geometric step) +Overlay: `review_agent_cold_start_overlay_beam.md`. -`α` is fixed per trajectory at calibration via `compute_cooling_rate(t_initial, t_min, max_steps)`. The adaptive rule modifies the *exponent* applied each round, not `α` itself. This matches Li & Landa-Silva 2011 §3.4. Convergence on `temperature_floor` requires ALL trajectories to be below `t_min`. +The Review Agent produces **K = `beam_width`** diverse seed candidates with no eval data yet. +Seeds must span confusion cells *and* cost regions. All seeds have `parent_version = "base"`. -**Default `t_initial = 0.2`**: chosen so that for the empirically observed median worsening Δ_E ≈ 0.07 (across the run at `outputs/d92011e7/`), P(accept) at the start of search is ≈ 0.7 — exploration without random-walk. The previous default of 1.0 left SA in random-walk regime for the first 60% of the budget, with acceptance histories of 5/5 True confirming the gate did nothing useful. +`advance_round_beam` special-cases round 1: `update_elite_set(..., is_cold_start_round=True)` +**bypasses Pareto filtering and crowding-distance pruning** — every scored seed is retained so +each initial strategy gets a second data point in round 2. `validate_elite_set` is skipped for +the same reason. Stagnation count is forced to 0. -**Citations:** -- Kirkpatrick, S., Gelatt, C. D. & Vecchi, M. P. (1983). *Optimization by simulated annealing*. Science, 220(4598):671–680. -- Li, H. & Landa-Silva, D. (2011). *EMOSA*. Evolutionary Computation 19(4):561–595. +### Round 2 — post-cold-start ---- +Overlay: `review_agent_post_coldstart_overlay_beam.md`. -## Neighborhood replacement +The elite set this round holds every scored round-1 seed as a **protected parent**. The Review +Agent must emit **exactly one `ChildVariant` per scored elite member**, using that member as +`parent_version`: -Every generated child is offered to its originating trajectory's neighborhood, regardless of whether the originating trajectory's Metropolis accepted it. The originators themselves are excluded from the replacement target set — Metropolis owns that decision. For each neighbor trajectory `j` (in `B(i)` and not in the originating set): +- one child per protected parent — no doubling up; +- `secondary_parent_version` must be `null` (no merges yet); +- failed cold-start seeds (`eval_status != "complete"`) are skipped; +- `LoopSignal.continue_search = true` unconditionally — round 2 is a structured exploration + step, not a convergence check. -``` -if compute_tchebycheff_energy(child, λ_j) < neighbor_j.current_energy: - neighbor_j.current_solution ← child - neighbor_j.current_energy ← energy under λ_j -``` - -This replacement is **unconditional** — there is no Metropolis random gate on the neighbor step. If the child is better than the neighbor's current under the neighbor's weight, the neighbor adopts it. This is the core EMOSA mechanism that prevents over-specialization: a child generated for one sub-problem can strengthen adjacent sub-problems without any explicit cross-trajectory logic in the Review Agent. Critically, this includes children that the originating trajectory's Metropolis rejected — they are still offered to neighbors, matching the canonical algorithm. - -**Default:** B = 4 for K = 5 (each trajectory sees every other trajectory's accepted children). B is a config field in `AnnealingState`. - -**Citations:** Li & Landa-Silva 2011 (EMOSA neighborhood replacement); Zhang & Li 2007 (MOEA/D neighborhood structure as precedent). +Standard Pareto competition begins in round 3. ---- +### Round ≥ 3 — steady-state iterative -## External archive +Overlay: `review_agent_iterative_overlay_beam.md`. -The global non-dominated archive (`elite_set` in `SearchState`) accumulates Pareto-optimal candidates across all trajectories. Archive management is a plain dominance filter: +The Review Agent emits `beam_width` children total, **allocated at its discretion** across 1–3 +members of the current elite set: -- **Dominance filter:** `update_archive` in `annealing.py` rejects any new candidate dominated by an existing archive member and removes existing members dominated by the new candidate. -- **No size limits:** the archive grows monotonically, bounded only by the total evaluation budget. There is no soft limit, no hard limit, and no pruning step. +- **Concentrate** (3 → 1): one elite is clearly most promising toward the threshold (or, once + the threshold is met, toward the oracle point); +- **Split** (2 + 1): two elites look comparably promising; +- **Spread** (1 + 1 + 1): a child from each of three elites. -The archive is shared across all trajectories: any trajectory can add a Pareto-improving candidate regardless of which trajectory generated it. +Per-child workflow (from `review_agent_iterative_base_system.md`): **identify failure mode → +hypothesise from data → create directive(s)**. The hypothesis is grounded in specific example +ids or metric patterns. Confusion cells are ranked by **threshold gap** while the user +threshold is unmet, and by **oracle gap** (`oracle_quality_change` / `oracle_cost_change` per +cell) once it is met. Multiple children off one parent must target *different* cells. Two-parent +merges (`secondary_parent_version`) are allowed from round 3 on. -**Citation:** Li & Landa-Silva 2011. +The Prompt Builder compiles each `ChildVariant`'s `EditDirective`s (`block_type` ∈ `rule`, +`example`, `output_schema`, `vocabulary`, `contrast_pair`) into a concrete prompt in the fixed +section order Objective → Categories → Decision Logic → Examples → Output Format. --- -## Ideal and nadir point updates - -The Tchebycheff normalization depends on `ideal_point` and `nadir_point`. These are updated incrementally in `_advance_emosa_search`: after collecting scored candidates, the ideal and nadir are expanded to include any new extremes discovered this round. The ideal point never regresses (quality only increases, cost only decreases); the nadir can expand in either direction. +## Selection: elite set update -This incremental update means early rounds operate with a narrow reference interval (small difference between ideal and nadir), which compresses the energy values toward zero. As more of the tradeoff surface is explored, the normalization spreads out and energy differences become more informative. +`advance_round_beam` calls `update_elite_set(current_elite, scored_pending, max_size = 2*beam_width + 1)`: -To keep the Metropolis Δ_E comparison range-consistent, each trajectory caches the raw `(current_quality, current_cost)` of its `current_solution`. At the top of `advance_round`, after the new ideal/nadir are computed, every trajectory's `current_energy` is **recomputed** under the new normalization before the Metropolis gate runs. Without this refresh, stored energies from rounds with narrower normalization are systematically lower than newly-computed energies under expanded normalization — the Metropolis Δ_E is positive even when the child strictly dominates under the current weight, biasing the gate toward keeping the held current and stranding trajectories on early picks. +1. Combine the current elite with the newly scored candidates; drop placeholder `(0.0, 0.0)` + entries; dedupe by `prompt_version`. +2. `compute_pareto_front` — keep only non-dominated candidates. `dominates(a, b)` is true when + `a.quality_score >= b.quality_score` **and** `a.cost <= b.cost` with at least one strict + inequality. +3. `prune_to_size(front, 2*beam_width + 1)` — while the front exceeds the cap, repeatedly + remove the non-endpoint candidate with the smallest **NSGA-II crowding distance**. The two + endpoints (highest quality, lowest cost) are protected every iteration. ---- +`crowding_distance` gives endpoints `inf` and each interior point the sum over the quality and +cost axes of the normalized gap between its two neighbours — the standard NSGA-II diversity +measure. With `beam_width = 3` the elite set holds at most **7** candidates. -## Initial seeding / Calibration phase +`validate_elite_set` recomputes the front defensively after every round except round 1 and logs +if it had to drop a dominated member. -Before the main search loop begins, the algorithm runs a **calibration phase** to seed each trajectory with an initial current solution and energy. +--- -**Steps (`calibration_complete` in `search_ops.py`):** -1. The Review Agent cold-start produces K diverse hypotheses (one per trajectory) without axis pre-commitment. See `review_agent_cold_start_system.md`. -2. The Prompt Builder compiles and evaluates one candidate per hypothesis. -3. The ideal and nadir points are initialized from the calibration candidates: `ideal_q = max quality seen`, `ideal_c = min cost seen`, `nadir_q = min quality seen`, `nadir_c = max cost seen`. -4. Each trajectory's `current_solution` and `current_energy` are seeded 1:1 in generation order: variant 0 → trajectory 0, variant 1 → trajectory 1, and so on. The assignment is arbitrary by design. -5. The archive is initialized with all non-dominated calibration candidates. -6. `AnnealingState.phase` transitions from `"calibration"` to `"search"`. +## Progress and stagnation + +After updating the elite set, `advance_round_beam`: + +1. Builds a **worst-seen reference point** from all elite + scored candidates this round: + `ref = (worst_quality * 0.9 or -0.1, worst_cost * 1.1 or 0.1)` — a lower-left corner below + every point seen. +2. Computes `new_hypervolume = compute_hypervolume(new_elite, ref)` — the 2-D area the front + dominates, via a sweepline over quality. +3. Stagnation (skipped on round 1): + ``` + relative_improvement = (new_hypervolume - hypervolume_prev) / hypervolume_prev # or new_hypervolume if prev == 0 + new_stagnation_count = 0 if relative_improvement > state.epsilon else state.stagnation_count + 1 + ``` +4. **Epsilon tightening (one-time):** when all user targets are first met by the elite set, + `epsilon ← max(epsilon / 2, epsilon_min)`, the stagnation count resets to 0, and a flag is + set so this never repeats. This raises the bar for "progress" once the loop is in + refinement territory. +5. **Backtracking flag:** `backtracking = new_stagnation_count >= backtrack_threshold` (2). The + iterative overlay surfaces `stagnation_signal.hypervolume_delta` to the Review Agent as a + cue to change which elite(s) it expands and pick a different confusion cell. + +Hypervolume, previous hypervolume, and the reference point are stored back into +`algorithm_state` for the next round; the per-round values are also recorded on the +`RoundSummary` (`hypervolume`, `reference_point`, `backtracking`, `target_improvement`, +`front_quality_spread`, `stagnation_count`). -**Alignment via neighborhood replacement:** The initial 1:1 assignment makes no attempt to match hypotheses to weight-vector axes. Alignment between each trajectory's current solution and its weight vector emerges naturally during the first several rounds via neighborhood replacement — trajectories that receive a well-aligned child quickly converge on it; misaligned starting seeds are superseded. This is consistent with EMOSA's design and is not a defect. +--- -The calibration phase corresponds to what the cold-start prompt calls "round 0". The `amosa_calibration_step_tool` wraps `calibration_complete` for MCP-level invocation. +## Convergence ---- +``` +total_evaluated = sum(len(r.candidates_evaluated) for r in round_history) + len(this_round) +budget_reached = total_evaluated >= evaluation_budget # default 60 +converged = (budget_reached and new_stagnation_count >= convergence_limit) # default 5 + or new_round >= max_rounds # default 50 +``` -## LLM Review Agent as mutation operator +`convergence_reason` is `"max_rounds"` when the round cap is hit, otherwise `"stagnation"`. +On the terminal round `RoundSummary.converged` is `True`, `SearchState.converged` is set, and +`loop_phase` flips to `"build"` (the Prompt Builder closes the run out instead of dispatching +another review). -EMOSA is operator-agnostic: it specifies the acceptance criterion and neighborhood replacement, but not how candidate mutations are generated. In this codebase, the Review Agent + Prompt Builder pipeline realizes the per-sub-problem-aware mutation slot. Each round, the Review Agent for trajectory `i` proposes directives targeted at reducing the term dominating the Tchebycheff max on that trajectory's binding axis — a semantically richer mutation than random perturbation. The Prompt Builder compiles the directives into a concrete candidate prompt for evaluation. +The Review Agent can also end the loop early by emitting `LoopSignal(action="exit")`. Per +`review_agent_iterative_base_system.md`, this is only safe when `single_candidate_meets_all` is +true — one candidate meets *every* declared user target. -Neighborhood replacement prevents over-specialization: a strong cross-axis child generated under one trajectory's mandate can propagate to neighboring trajectories if it scalarizes better there. The Review Agent does not need to anticipate this; it focuses on its own trajectory's binding axis. +After `advance_round_beam` returns, scored candidates are appended to +`candidate_archive.json`, `pending_candidates.json` is cleared, and `viz.html` is regenerated. --- -## Convergence +## Mutation modes -The search converges when any of the following conditions is met: +`SearchState.mutation_mode` toggles between: -| Condition | `convergence_reason` value | -|-----------|---------------------------| -| Temperature falls below `t_min` | `"temperature_floor"` | -| `total_evals >= max_evals` | `"eval_budget"` | -| Review Agent emits `LoopSignal(action="exit")` | `"review_exit"` | +- **`targeted`** — faithful paraphrase / reorder / example swap against the parent prompt; +- **`exploratory`** — structural rewrite with different example sets. -All convergence checks happen inside `advance_round`. The round summary's `converged` field is `True` on the terminal round; `AnnealingState.phase` transitions to `"converged"`. The Prompt Builder Agent reads phase to decide whether to call `advance_round_tool` again or close out the run. +The mode flips toward `exploratory` when the search stalls (stagnation), giving the Review +Agent licence to propose larger changes. See `prompt_builder_system.md` for how each mode +constrains the compiled prompt. --- ## Pointers for future changes -- **Changing the scalarization** (e.g., to PBI — Penalty-Boundary Intersection, per Zhang & Li 2007): edit `compute_tchebycheff_energy` in `compass/agents/prompt_builder/annealing.py` and update the "Tchebycheff decomposition" section above. PBI is a natural next extension; EMOSA's original paper discusses it as an alternative. -- **Adding an axis** (e.g., latency): update `classify_user_target` in `preprocessor.py`, add weight-vector logic in `compute_weight_vectors`, extend `compute_tchebycheff_energy` to K-objective form, and extend this document's "Tchebycheff decomposition" and "Weight vectors and trajectories" sections. -- **Changing neighborhood size B**: edit the `neighborhood_size` field in `AnnealingState` (default 4, set when constructing the `algorithm_state` pocket on `init_search_state`) and update the "Neighborhood replacement" section above. -- **Changing archive behavior** (e.g., adding crowding-distance-based pruning if archive grows too large): edit `update_archive` in `annealing.py` and update the "External archive" section above. -- **Changing convergence detection**: edit the convergence checks in `_advance_emosa_search` in `search_ops.py` and update the "Convergence" section above. -- **Tuning the adaptive cooling band**: adjust `target_acceptance_low`/`target_acceptance_high`/`cooling_exp_fast`/`cooling_exp_slow` defaults on `AnnealingState` in `annealing.py`. The defaults (0.4 / 0.6 / 1.5 / 0.5) follow Li & Landa-Silva 2011 §3.4. Lower bands push trajectories to cool faster overall. +- **Change the beam width:** edit `_BRANCH_ALGORITHM_STATE` in `search_ops.py`. The elite-set + cap (`2*beam_width + 1`) and per-round child count both follow from it; the iterative overlay + reads `briefing.beam_width`. +- **Change the selection objective** (e.g. add latency as a third axis): extend `Candidate`, + `dominates`, `compute_pareto_front`, `crowding_distance`, and `compute_hypervolume` in + `search.py` to K objectives, and update `classify_user_target` in + `compass/agents/review/preprocessor.py`. +- **Change pruning** (e.g. reference-point-based instead of crowding distance): edit + `prune_to_size` / `crowding_distance` in `search.py`. +- **Change convergence detection:** edit the `converged` / `budget_reached` logic in + `advance_round_beam` in `search_ops.py` and update the "Convergence" section above. +- **Tune stagnation sensitivity:** `epsilon`, `epsilon_min`, `stagnation_limit`, + `convergence_limit`, `backtrack_threshold`. Note the `convergence_limit > stagnation_limit` + invariant enforced by a `model_validator` on `SearchState`. --- ## References -Kirkpatrick, S., Gelatt, C. D. & Vecchi, M. P. (1983). Optimization by simulated annealing. *Science*, 220(4598):671–680. - -Li, H. & Landa-Silva, D. (2011). An adaptive evolutionary multi-objective approach based on simulated annealing. *Evolutionary Computation*, 19(4):561–595. *(Primary reference — EMOSA.)* +Deb, K., Pratap, A., Agarwal, S. & Meyarivan, T. (2002). *A fast and elitist multiobjective +genetic algorithm: NSGA-II.* IEEE Transactions on Evolutionary Computation, 6(2):182–197. +*(Crowding-distance diversity operator used in `prune_to_size`.)* -Zhang, Q. & Li, H. (2007). MOEA/D: A multiobjective evolutionary algorithm based on decomposition. *IEEE Transactions on Evolutionary Computation*, 11(6):712–731. *(Tchebycheff decomposition and neighborhood structure.)* +Zitzler, E. & Thiele, L. (1999). *Multiobjective evolutionary algorithms: a comparative case +study and the strength Pareto approach.* IEEE Transactions on Evolutionary Computation, +3(4):257–271. *(Hypervolume indicator used as the progress metric.)* diff --git a/docs/architecture.md b/docs/architecture.md index db3c5f6..5d246a9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -6,16 +6,19 @@ Quick re-orientation guide for the Compass multi-agent routing optimizer. ```mermaid graph TD - U["User"] -->|problem + dataset| A1["User Input Agent
LLM-driven
Status: done"] - A1 -->|validated_input_report_path| A2["Data Validation Agent
LLM-driven
Phase 1: ingest & map → Phase 2: validate + split
Status: done"] - A2 -->|RoutingContext + dev/holdout splits| A3["Prompt Builder Agent
LLM-driven
Status: done"] - A3 -->|prompt version| A4["Eval Runner Agent
code-driven
Status: done"] - A4 -->|eval_score_report| A5["Review Agent
LLM-driven
Status: done"] - A5 -->|iterate| A3 - A5 -->|accept| A6["Final Report Agent
Hybrid (code + LLM)
Holdout eval + report
Status: done"] + U["User"] -->|problem + dataset| A1["Stage 1 · User Input Agent
LLM-driven"] + A1 -->|validated_input_report_path| A2["Stage 2 · Data Validation Agent
LLM-driven
Phase 1: ingest & map → Phase 2: validate + split"] + A2 -->|RoutingContext + dev/holdout splits| A2b["Stage 3 · Backend Setup Agent
LLM-driven
write backend profile"] + A2b -->|backend profile| A3["Stage 4 · Prompt Builder Agent
LLM-driven"] + A3 -->|prompt version| A4["Stage 4 · Eval Runner Agent
code-driven"] + A4 -->|eval_score_report| A5["Stage 4 · Review Agent
LLM-driven"] + A5 -->|loop_signal = refine| A3 + A5 -->|loop_signal = exit / converged| A6["Stage 5–6 · Final Report Agent
Hybrid (code + LLM)
Holdout eval + report"] A6 -->|final report| U ``` +**Stage numbering.** The six agent roles above map to five internal dispatcher stages in [`compass/agents/pipeline/status.py`](../compass/agents/pipeline/status.py) (`_STAGES`): Stage 4 covers the Prompt Builder ↔ Eval Runner ↔ Review Agent loop, and internal Stage 5 ("Final Report") runs both holdout evaluation and report generation. The README's `## Usage` section presents the same flow as six user-facing stages. + **Rerun mode:** When Stage 4 has converged, the orchestrator can call `initiate_rerun` to re-enter the pipeline at Stage 3 for a different backend. The rerun flow is: Stage 3 (new backend) → Stage 4 (Prompt Builder Rerun: format restructure + single eval) → Stage 5 (final report). The original search state is preserved as `search_state_original.json`; `rerun_config.json` drives rerun-mode behavior throughout `status.py`. ## 2. Agent Registry @@ -26,9 +29,9 @@ graph TD | Data Validation | LLM-driven | [`compass/agents/prompts/data_validation_system.md`](../compass/agents/prompts/data_validation_system.md), [`compass/agents/data_validation/checks.py`](../compass/agents/data_validation/checks.py), [`compass/agents/data_validation/split.py`](../compass/agents/data_validation/split.py) | Done | `validated_input_report_path` | `data_quality_report`, `routing_context`, `dataset_path`, `original_dataset_path`, `dev_jsonl_path`, `holdout_jsonl_path`, `split_report_path` (debug-only) | | Eval Runner | Code-driven | [`compass/agents/eval_runner.py`](../compass/agents/eval_runner.py) | Done | `prompt_version`, `data_source`, `backend`, `run_config` or `config_path` | `eval_score_report` | | Backend Setup | LLM-driven | [`compass/agents/prompts/backend_setup_system.md`](../compass/agents/prompts/backend_setup_system.md) | Done | (user conversation) | `backend` (new YAML file written to `backends/`) | -| Prompt Builder | LLM-driven | (planned) | Planned | `routing_context`, `dev_jsonl_path` | `prompt_version` | +| Prompt Builder | LLM-driven | [`compass/agents/prompts/prompt_builder_system.md`](../compass/agents/prompts/prompt_builder_system.md), [`compass/agents/prompt_builder/search.py`](../compass/agents/prompt_builder/search.py), [`search_ops.py`](../compass/agents/prompt_builder/search_ops.py) | Done | `routing_context`, `dev_jsonl_path`, child variants | `prompt_version` (one file per candidate) | | Prompt Builder Rerun | LLM-driven | [`compass/agents/prompts/prompt_builder_rerun_system.md`](../compass/agents/prompts/prompt_builder_rerun_system.md) | Done | `run_id`, `source_prompt_version`, `new_backend` (from subagent instruction) | `prompt_version` (restructured) | -| Review | Hybrid (code + LLM) | [`compass/agents/review/models.py`](../compass/agents/review/models.py), [`compass/agents/review/preprocessor.py`](../compass/agents/review/preprocessor.py), [`compass/agents/review/ops.py`](../compass/agents/review/ops.py), three-tier prompt: `review_agent_base_system.md` + phase base + strategy overlay (see Prompts table) | Done | `eval_score_report`, `review_briefing` | `review_result` (debug-only) | +| Review | Hybrid (code + LLM) | [`compass/agents/review/models.py`](../compass/agents/review/models.py), [`compass/agents/review/preprocessor.py`](../compass/agents/review/preprocessor.py), [`compass/agents/review/ops.py`](../compass/agents/review/ops.py), layered prompt: `review_agent_base_system.md` + phase base + beam overlay (see Prompts table) | Done | `eval_score_report`, `review_briefing` | `review_result` (debug-only) | | Final Report | Hybrid (code + LLM) | [`compass/agents/final_report/models.py`](../compass/agents/final_report/models.py), [`compass/agents/final_report/preprocessor.py`](../compass/agents/final_report/preprocessor.py), [`compass/agents/prompts/final_report_system.md`](../compass/agents/prompts/final_report_system.md), [`compass/agents/prompts/final_report_template.md`](../compass/agents/prompts/final_report_template.md) | Done | holdout dataset, search state, all eval reports | `final_report.md`, per-version `baseline_comparison.json`, optimization charts | ## 3. Context Dict Reference @@ -56,29 +59,28 @@ graph TD ## 4. Shared Models **`Candidate` / `SearchState` / `RoundSummary`** ([`compass/agents/prompt_builder/search.py`](../compass/agents/prompt_builder/search.py)) -`Candidate` is the canonical prompt-candidate record. Core fields: `prompt_version`, `parent_version`, `quality_score`, `cost`, `round_introduced`, `example_ids`. Optional fields (all default `None`): `secondary_parent_version`, `eval_status` (parallel eval tracking), `mutation_strategy`, `route_metrics`, `trajectory_id`. Accepts `iteration_introduced` as an alias for `round_introduced` (back-compat). Old state files carrying `dominated` load without error (`extra="ignore"`). +`Candidate` is the canonical prompt-candidate record. Core fields: `prompt_version`, `parent_version`, `quality_score`, `cost`, `round_introduced`, `example_ids`. Optional fields (all default `None`): `secondary_parent_version` (two-parent merges), `eval_status` (parallel eval tracking), `mutation_strategy`, `route_metrics`. `trajectory_id` is present for other search strategies and unused by beam. Accepts `iteration_introduced` as an alias for `round_introduced` (back-compat). Old state files carrying `dominated` load without error (`extra="ignore"`). `SearchState` holds the mutable search loop state. Key fields: | Field | Type | Description | |---|---|---| | `elite_set` | `list[Candidate]` | Current non-dominated candidate set (formerly `pareto_front`; old files with `pareto_front` key are migrated on load) | -| `algorithm` | `str` | Discriminator set from `_BRANCH_ALGORITHM` at `init_search_state` time; trunk default is `"__unset__"`, leaf branches set a concrete value (e.g. `"hill_climb"`) | -| `algorithm_state` | `dict[str, Any]` | Strategy-specific sub-state pocket; hardcoded per branch via `_BRANCH_ALGORITHM_STATE` in `search_ops.py` (empty `{}` on trunk) | -| `round`, `stagnation_count`, `mutation_mode` | — | Bookkeeping fields (semantics may vary by algorithm) | -| `converged`, `loop_phase` | — | Convergence flag and current sub-phase; base phases on trunk: `"build"`, `"review"`, `"build_recovering"`; leaf branches extend with algorithm-specific phases (e.g. `"calibration"`, `"warmup_seed"`, `"warmup_build"`, `"warmup_reduce"`) | +| `algorithm` | `str` | Discriminator set from `_BRANCH_ALGORITHM` at `init_search_state` time; this repo sets `"beam"` | +| `algorithm_state` | `dict[str, Any]` | Strategy-specific sub-state pocket, set from `_BRANCH_ALGORITHM_STATE` in `search_ops.py` (`{"beam_width": 3}`) | +| `round`, `stagnation_count`, `mutation_mode` | — | Search-loop bookkeeping | +| `converged`, `loop_phase` | — | Convergence flag and current sub-phase. Beam uses `"build"`, `"review"`, `"build_recovering"`; the enum also carries reserved values (`"calibration"`, `"warmup_seed"`, `"warmup_build"`, `"warmup_reduce"`) for other search strategies | | `active_evals` | `list[str]` | Prompt versions currently being evaluated (pending or running). Invariant: non-empty iff `loop_phase == "build"` and a concurrent batch eval was interrupted. `_detect_stage_4_phase` returns `"build_recovering"` when this field is non-empty. | -`RoundSummary` is the per-round progress record. Field names follow the unified cross-branch schema: +`RoundSummary` is the per-round progress record. A few fields were renamed from earlier names, which still load via a `model_validator(mode="before")`: -| Canonical field | Renamed from | Strategy | -|---|---|---| -| `new_elite_entries` | `new_pareto_points` (main) | all | -| `elite_size` | `front_size` (main) | all | -| `target_improvement` | `front_improvement` (main) | all | -| `mutation_mode`, `stagnation_count` | — (main only) | optional (hill-climb) | +| Field | Legacy alias | +|---|---| +| `new_elite_entries` | `new_pareto_points` | +| `elite_size` | `front_size` | +| `target_improvement` | `front_improvement` | -Old state files with `new_pareto_points` / `front_size` / `front_improvement` are migrated on load via a `model_validator(mode="before")`. +`mutation_mode` and `stagnation_count` are optional (`None` when unset). **`DataQualityReport`** ([`compass/agents/data_validation/checks.py`](../compass/agents/data_validation/checks.py)) Top-level report from the Data Validation agent containing `SchemaFinding` list, `LabelDistribution`, `VolumeAssessment`, and optional `QueryLengthDistribution`. The LLM agent writes the narrative `summary`; the Python checks populate the structured sections. @@ -91,18 +93,15 @@ Domain-agnostic routing configuration holding a `domain` description, `RouteDefi `build_review_briefing` now returns a **markdown progressive-disclosure summary** (rendered by [`compass/agents/review/render.py`](../compass/agents/review/render.py)) rather than raw JSON. Companion detail-fetch tools (`get_score_report`, `get_confusion_cell`, `get_round_child_variants`, `query_dev_examples`, `query_holdout_examples`, `get_dataset_oracle_distribution`, `get_per_class_recall`) provide on-demand drill-down without reloading the full briefing on every call. Dataset rows remain query-only: row tools page results with `offset`/`limit` and never inline an entire dataset. `get_directive_history` and `get_batch_outcomes` are retained only as deprecated MCP placeholders because the legacy `round_reports` persistence they depended on is no longer written by production code. `ReviewResult` is the LLM output: `candidate_ranking`, `child_variants`, `promotion_decisions`, `loop_signal`, and `regression_guards`. Review-side persistence and legacy artifact loading live in [`review/ops.py`](../compass/agents/review/ops.py). Child variants are persisted to `child_variants.json` via `record_directive_outcomes` and retrieved by the Prompt Builder via `get_child_variants`. `get_edit_directives` is a back-compat helper that flattens all directives across variants into a single list. -Strategy-specific optional fields pre-provisioned as `None`-default slots in the model; each leaf branch's preprocessor function populates the slots relevant to its algorithm: +`ReviewBriefing` also carries optional `None`-default slots, populated by the round-advance preprocessor: -| Field | Type | Algorithm | Populated by | -|---|---|---|---| -| `parent_a_version`, `parent_b_version` | `str \| None` | all | `build_review_briefing` | -| `trajectory_id` | `int \| None` | leaf branch | `_populate_*_review_fields` (leaf branch) | -| `weight_vector` | `tuple[float, float] \| None` | leaf branch | `_populate_*_review_fields` (leaf branch) | -| `binding_axis` | `"quality" \| "cost" \| None` | leaf branch | `_populate_*_review_fields` (leaf branch) | -| `acceptance_history` | `list[bool] \| None` | leaf branch | `_populate_*_review_fields` (leaf branch) | -| `stagnation_signal` | `dict \| None` | all | algorithm-specific `_populate_*_review_fields` on the leaf branch | +| Field | Type | Used by | +|---|---|---| +| `parent_a_version`, `parent_b_version` | `str \| None` | beam — set by `build_review_briefing` for two-parent merges | +| `stagnation_signal` | `dict \| None` | beam — hypervolume-delta cue for the iterative overlay | +| `trajectory_id`, `weight_vector`, `binding_axis`, `acceptance_history` | — | other search strategies only; `None` under beam | -Algorithm-specific advance logic (e.g. hill-climb step, beam step) lives entirely on the leaf branches' `search_ops.py`. +The round-advance logic for beam search (`advance_round_beam`) lives in [`search_ops.py`](../compass/agents/prompt_builder/search_ops.py). **`ScoreReport` / `RunReport`** ([`compass/eval/models.py`](../compass/eval/models.py)) `RunReport` is the full evaluation output (config, metrics, results, summary). `ScoreReport` is the inter-agent contract (context key `eval_score_report`) containing metrics, summary, error breakdown, run-over-run `RunDiff`, and output file paths. @@ -155,10 +154,10 @@ Pydantic model representing a validated backend configuration loaded from a YAML | `save_final_report` | Implemented | Save the final report markdown to disk | [`compass/mcp/final_report_tools.py`](../compass/mcp/final_report_tools.py) | | `get_pipeline_status` | Implemented | Read-only inspector for pipeline progress. Not part of the dispatch loop — orchestrators use `start_stage` directly. | [`compass/agents/pipeline/status.py`](../compass/agents/pipeline/status.py) | | `get_default_pricing` | Implemented | Look up default pricing for a (provider, model) pair; used by the backend setup agent | [`compass/eval/pricing.py`](../compass/eval/pricing.py) | -| `init_search_state` | Implemented | Initialize prompt-builder search state for a run; algorithm is hardcoded per branch — no `algorithm`/`algorithm_state` params | [`compass/agents/prompt_builder/search_ops.py`](../compass/agents/prompt_builder/search_ops.py) | +| `init_search_state` | Implemented | Initialize prompt-builder search state for a run; the algorithm is set from the module constants in `search_ops.py` — no `algorithm`/`algorithm_state` params | [`compass/agents/prompt_builder/search_ops.py`](../compass/agents/prompt_builder/search_ops.py) | | `register_candidate` | Implemented | Register a new prompt candidate for evaluation | [`compass/agents/prompt_builder/search_ops.py`](../compass/agents/prompt_builder/search_ops.py) | | `record_eval_result` | Implemented | Record evaluation results for Pareto tracking | [`compass/agents/prompt_builder/search_ops.py`](../compass/agents/prompt_builder/search_ops.py) | -| `advance_step` | Implemented | Strategy-dispatched step advance; implementation provided by the leaf branch's `advance_round` in `search_ops.py` | [`compass/mcp/prompt_building_tools.py`](../compass/mcp/prompt_building_tools.py) | +| `advance_step` | Implemented | Advance the search by one round; dispatches to `advance_round` → `advance_round_beam` in `search_ops.py` | [`compass/mcp/prompt_building_tools.py`](../compass/mcp/prompt_building_tools.py) | | `get_search_state` | Implemented | Return a markdown summary of the current search state via the shared renderer, with round history capped to the last 3 rounds | [`compass/mcp/prompt_building_tools.py`](../compass/mcp/prompt_building_tools.py) | | `filter_holdout_dataset` | Implemented | Remove few-shot examples from holdout before final eval | [`compass/agents/prompt_builder/holdout_filter.py`](../compass/agents/prompt_builder/holdout_filter.py) | | `get_child_variants` | Implemented | Retrieve the current round's child variants (grouped directives per child prompt) for the Prompt Builder | [`compass/mcp/prompt_building_tools.py`](../compass/mcp/prompt_building_tools.py) | @@ -181,7 +180,7 @@ The orchestrator calls `start_stage(run_id)` before spawning a sub-agent (no `st | `prompt_building` | `init_search_state`, `register_candidate`, `run_batch_eval`, `record_eval_result`, `advance_step`, `get_search_state`, `get_routing_context`, `get_edit_directives`, `get_child_variants`, `get_prompt_text`, `get_score_report`, `save_prompt`, `signal_eval_complete`, `get_pipeline_status` | | `review_cold` | `build_review_briefing`, `record_directive_outcomes`, `query_dev_examples`, `get_search_state`, `get_score_report`, `get_confusion_cell`, `get_round_child_variants`, `get_dataset_oracle_distribution`, `get_per_class_recall`, `get_pipeline_status` | | `review` | `build_review_briefing`, `record_directive_outcomes`, `query_dev_examples`, `query_holdout_examples`, `get_prompt_text`, `get_search_state`, `get_score_report`, `get_confusion_cell`, `get_round_child_variants`, `get_dataset_oracle_distribution`, `get_per_class_recall`, `get_pipeline_status` | -| `calibration` | Algorithm-specific phase — tool set defined by the leaf branch | +| `calibration` | Reserved phase for other search strategies; not used by beam | | `final_report` | `filter_holdout_dataset`, `run_holdout_eval`, `build_final_report_briefing`, `save_final_report`, `get_pipeline_status` | #### Model Routing @@ -226,12 +225,12 @@ The shared guard layer lives in [`compass/agents/pipeline/dispatch.py`](../compa |---|---| | `"review"` | all strategies (default) | | `"build"` | all strategies | -| `"warmup_seed"`, `"warmup_build"`, `"warmup_reduce"` | reserved — strategy-branch warmup phases | -| `"calibration"` | reserved — strategy-branch cold-start calibration phase | -| `"build_recovering"` | All strategies — entered when `active_evals` is non-empty (interrupted batch eval). Recovery sub-agent calls `run_batch_eval(candidates=[])` to resume. | -| `"review_post_coldstart"` | Leaf-branch-specific derived phase (not stored on disk); leaf branch `_detect_stage_4_phase` logic returns this for algorithm-specific post-cold-start handling. | +| `"warmup_seed"`, `"warmup_build"`, `"warmup_reduce"` | reserved for other search strategies — warmup phases | +| `"calibration"` | reserved for other search strategies — cold-start calibration phase | +| `"build_recovering"` | entered when `active_evals` is non-empty (interrupted batch eval). Recovery sub-agent calls `run_batch_eval(candidates=[])` to resume. | +| `"review_post_coldstart"` | derived phase (not stored on disk); `_detect_stage_4_phase` returns it for the round-2 post-cold-start review. | -The trunk pipeline uses only `"build"`, `"review"`, and `"build_recovering"`. Other values are reserved for leaf branches; unknown values encountered in legacy JSON are silently remapped to `"review"` by a `model_validator(mode="before")`. +Beam uses `"build"`, `"review"`, `"build_recovering"`, and the derived `"review_post_coldstart"`. The remaining values are reserved for other search strategies; unknown values in legacy JSON are silently remapped to `"review"` by a `model_validator(mode="before")`. **Dispatch markers** @@ -242,11 +241,11 @@ Two JSON sentinel files signal that a sub-agent is in-flight for the current rou | `search/build_dispatched.json` | `register_candidate` (first builder action) | `advance_step` (round complete); `run_batch_eval_impl` (when `active_evals` drains after batch eval) | `complete_stage("prompt_building")` rejects while present | | `search/review_dispatched.json` | `build_review_briefing` (reviewer dispatch) | `record_directive_outcomes` (directives saved) | `complete_stage("review")` checks fanout | -`build_dispatched.json` contains `{"round": N}`. `review_dispatched.json` is `{"round": N}` for single-slot algorithms; multi-slot leaf branches may extend this to include additional tracking fields. +`build_dispatched.json` contains `{"round": N}`. `review_dispatched.json` is `{"round": N}`. (A search strategy that dispatches several parallel reviewers per round would extend this; beam uses one reviewer.) -**Multi-trajectory review fanout (leaf-branch feature)** +**Multi-trajectory review fanout (extension point)** -Algorithms with multiple parallel trajectories extend `_next_action_for_stage_4` on their leaf branch to dispatch N Review Agent sub-agents in parallel — one per trajectory ID. The trunk pipeline uses a single-slot (`expected=1`) fanout; `complete_stage("review")` calls `review_fanout_status(run_id, expected=1)` and checks `is_complete`. +`_next_action_for_stage_4` can be extended to dispatch N Review Agent sub-agents in parallel — one per trajectory — for a search strategy that needs it. Beam uses a single-slot (`expected=1`) fanout; `complete_stage("review")` calls `review_fanout_status(run_id, expected=1)` and checks `is_complete`. **`DispatchFanout` and `review_fanout_status`** @@ -261,11 +260,11 @@ Algorithms with multiple parallel trajectories extend `_next_action_for_stage_4` | `is_complete` | `bool` | `len(completed) >= expected` | | `missing` | `list[int]` | `in_flight + not_dispatched` | -For `expected=1` (single-slot — the trunk default), fanout is complete when `search/child_variants.json` exists. Multi-slot dispatch is a leaf-branch concern; leaf branches extend `review_fanout_status` as needed. +For `expected=1` (single-slot — what beam uses), fanout is complete when `search/child_variants.json` exists. Multi-slot dispatch is an extension point: `review_fanout_status` would be extended for a strategy that needs it. **`child_variants.json`** -Written by `record_directive_outcomes`; acts as the canonical review-completion sentinel. Multi-slot per-trajectory variants (`child_variants_t.json`) are a leaf-branch extension. +Written by `record_directive_outcomes`; acts as the canonical review-completion sentinel. (Per-trajectory variants `child_variants_t.json` are an extension point for a multi-trajectory strategy.) **Defense-in-depth phase flip** @@ -281,27 +280,28 @@ Retained as a back-compat shim for runs paused before automated marker clearing |---|---|---| | `compass_routing_input` | Activate the User Input agent conversation | [`compass/agents/prompts/user_input_system.md`](../compass/agents/prompts/user_input_system.md) | | `compass_data_validation` | Activate the Data Validation agent conversation | [`compass/agents/prompts/data_validation_system.md`](../compass/agents/prompts/data_validation_system.md) | -| `compass_review_agent_iterative(algorithm)` | Review Agent — iterative phase (round ≥ 2); assembled from three-tier prompt: base + iterative phase base + strategy overlay | see Review Agent prompt files below | -| `compass_review_agent_cold_start(algorithm)` | Review Agent — cold-start / seeding phase; assembled from three-tier prompt: base + cold-start phase base + strategy overlay | see Review Agent prompt files below | -| `compass_review_agent_post_coldstart(algorithm)` | Review Agent — round-2 post-cold-start phase (leaf-branch-specific); assembled from four-tier prompt: base + iterative phase base + post-coldstart override + strategy overlay | see Review Agent prompt files below | +| `compass_prompt_builder` | Activate the Prompt Builder agent (Stage 4 build phase) | [`compass/agents/prompts/prompt_builder_system.md`](../compass/agents/prompts/prompt_builder_system.md) | +| `compass_review_agent_iterative(algorithm)` | Review Agent — iterative phase (round ≥ 3); assembled from three layers: base + iterative phase base + beam overlay | see Review Agent prompt files below | +| `compass_review_agent_cold_start(algorithm)` | Review Agent — cold-start / seeding phase (round 1); assembled from three layers: base + cold-start phase base + beam overlay | see Review Agent prompt files below | +| `compass_review_agent_post_coldstart(algorithm)` | Review Agent — round-2 post-cold-start phase; assembled from three layers: base + post-coldstart override + iterative phase base | see Review Agent prompt files below | | `compass_backend_setup` | Backend setup agent — select or create backend | [`compass/agents/prompts/backend_setup_system.md`](../compass/agents/prompts/backend_setup_system.md) | | `compass_final_report` | Final Report agent — holdout eval + report generation | [`compass/agents/prompts/final_report_system.md`](../compass/agents/prompts/final_report_system.md) | | `compass_prompt_builder_rerun` | Prompt Builder Rerun agent — format-only restructure for a different backend (single eval round) | [`compass/agents/prompts/prompt_builder_rerun_system.md`](../compass/agents/prompts/prompt_builder_rerun_system.md) | -**Review Agent prompt files — three-tier structure (trunk)** +**Review Agent prompt files — layered structure** -The Review Agent prompt is assembled at dispatch time from three layers (iterative / cold-start): a shared base, a phase-specific base, and a strategy overlay. The `algorithm` argument on the MCP prompt selects the overlay. Leaf branches may extend this to four layers by adding a post-coldstart override and their own overlay files. +The Review Agent prompt is assembled at dispatch time (see [`assemble_review_prompt`](../compass/mcp/prompts.py)) from a shared base, a phase-specific base, and a beam overlay — three layers for the iterative and cold-start phases. Round 2 uses base + post-cold-start override + iterative base. The `algorithm` argument on the MCP prompt selects the overlay (`beam` in this repo). -**Dispatch protocol.** The Stage-4 dispatcher (any session driving the MCP) MUST fetch the canonical system prompt via `compass_review_agent_cold_start(algorithm)` when `SearchState.warm_up_complete == False` and via `compass_review_agent_iterative(algorithm)` otherwise, and pass the returned text verbatim as the sub-agent system prompt. Hand-rolling the dispatch prompt is forbidden: the `ChildVariant` / `EditDirective` schema is declared with `extra="forbid"` ([`compass/agents/review/models.py:272-310`](../compass/agents/review/models.py)) and cannot be safely re-stated, and the canonical prompt owns the no-Bash invariant ([`review_agent_base_system.md:13-15`](../compass/agents/prompts/review_agent_base_system.md)) needed to keep the sub-agent on the MCP surface. See [`.claude/rules/compass-stage4-dispatch.md`](../.claude/rules/compass-stage4-dispatch.md). +**Dispatch protocol.** The Stage-4 dispatcher (any session driving the MCP) MUST fetch the canonical system prompt via `compass_review_agent_cold_start(algorithm)` for the seeding round (round 1), `compass_review_agent_post_coldstart(algorithm)` for round 2, and `compass_review_agent_iterative(algorithm)` for rounds ≥ 3, and pass the returned text verbatim as the sub-agent system prompt. Hand-rolling the dispatch prompt is forbidden: the `ChildVariant` / `EditDirective` schema is declared with `extra="forbid"` ([`compass/agents/review/models.py:272-310`](../compass/agents/review/models.py)) and cannot be safely re-stated, and the canonical prompt owns the no-Bash invariant ([`review_agent_base_system.md`](../compass/agents/prompts/review_agent_base_system.md)) needed to keep the sub-agent on the MCP surface. | File | Role | |---|---| -| [`compass/agents/prompts/review_agent_base_system.md`](../compass/agents/prompts/review_agent_base_system.md) | Shared base — entry verification, briefing schema, directive types, output schema, self-check rules | -| [`compass/agents/prompts/review_agent_iterative_base_system.md`](../compass/agents/prompts/review_agent_iterative_base_system.md) | Iterative phase base — "identify failure mode → hypothesise → create directive" flow | -| [`compass/agents/prompts/review_agent_cold_start_base_system.md`](../compass/agents/prompts/review_agent_cold_start_base_system.md) | Cold-start phase base — "formulate diverse strategies" flow | -| [`compass/agents/prompts/review_agent_post_coldstart_base_system.md`](../compass/agents/prompts/review_agent_post_coldstart_base_system.md) | Post-cold-start override (leaf-branch-specific) — present on trunk for leaf branches to compose against | -| `review_agent_iterative_overlay_.md` | Algorithm-specific iterative overlay — lives on the leaf branch, not the trunk | -| `review_agent_cold_start_overlay_.md` | Algorithm-specific cold-start overlay — lives on the leaf branch, not the trunk | +| [`review_agent_base_system.md`](../compass/agents/prompts/review_agent_base_system.md) | Shared base — entry verification, briefing schema, directive types, output schema, self-check rules | +| [`review_agent_iterative_base_system.md`](../compass/agents/prompts/review_agent_iterative_base_system.md) | Iterative phase base — "identify failure mode → hypothesise → create directive" flow | +| [`review_agent_cold_start_base_system.md`](../compass/agents/prompts/review_agent_cold_start_base_system.md) | Cold-start phase base — "formulate diverse strategies" flow | +| [`review_agent_cold_start_overlay_beam.md`](../compass/agents/prompts/review_agent_cold_start_overlay_beam.md) | Cold-start overlay — K = `beam_width` seeds, diversity across confusion cells and cost regions | +| [`review_agent_iterative_overlay_beam.md`](../compass/agents/prompts/review_agent_iterative_overlay_beam.md) | Iterative overlay (round ≥ 3) — child allocation modes, confusion-cell ranking, stagnation cue | +| [`review_agent_post_coldstart_overlay_beam.md`](../compass/agents/prompts/review_agent_post_coldstart_overlay_beam.md) | Round-2 override — one protected child per cold-start seed, no merges | ### Resources @@ -322,15 +322,25 @@ The Review Agent prompt is assembled at dispatch time from three layers (iterati | `compass://agents/backend-setup/defaults` | Backend defaults and pricing resolution | [`compass/agents/backend_setup_defaults.md`](../compass/agents/backend_setup_defaults.md) | | `compass://agents/final-report/template` | Markdown skeleton for the final report — section order and placeholders | [`compass/agents/prompts/final_report_template.md`](../compass/agents/prompts/final_report_template.md) | -## 6. Strategy Extension Points +## 6. Search Algorithm -`feat/generalize-pipeline` is the algorithm-agnostic trunk. Strategy-specific implementations live on dedicated leaf branches cut from this branch. Leaf branches override exactly two module-level constants in `search_ops.py` to activate their algorithm. +The Stage-4 search strategy is selected by two module-level constants in [`compass/agents/prompt_builder/search_ops.py`](../compass/agents/prompt_builder/search_ops.py). This repo runs **beam search**: -**Algorithm is hardcoded per branch.** `search_ops.py` exposes two module-level constants (`_BRANCH_ALGORITHM`, `_BRANCH_ALGORITHM_STATE`) that strategy branches flip. On the trunk `_BRANCH_ALGORITHM = "__unset__"` — `init_search_state` raises `RuntimeError` if called on trunk. `search_state.json` is **auto-created at Stage 4 entry** by `_ensure_stage4_search_state` (called from `_next_action_for_stage_4` in `status.py`) before `_detect_stage_4_phase` runs, so cold-start sub-agents always find a real `SearchState` on disk. +```python +_BRANCH_ALGORITHM: AlgorithmType = "beam" +_BRANCH_ALGORITHM_STATE: dict[str, Any] = {"beam_width": 3} +``` -| Strategy | Branch | Algorithm module | Dispatcher arm | Preprocessor populate fn | Prompt overlays | -|---|---|---|---|---|---| -| `hill_climb` | `feat/generalize-hill_climb` | `prompt_builder/search.py`, `search_ops.py` | `_advance_hill_climb` | (none — default fields) | `review_agent_iterative_overlay_hillclimb.md`, `review_agent_cold_start_overlay_hillclimb.md` | +`init_search_state` reads them when it creates `search_state.json`. That file is **auto-created at Stage 4 entry** by `_ensure_stage4_search_state` (called from `_next_action_for_stage_4` in `status.py`) before `_detect_stage_4_phase` runs, so cold-start sub-agents always find a real `SearchState` on disk. `advance_step` dispatches to `advance_round` → `advance_round_beam`. + +| Component | Where | +|---|---| +| Models, Pareto dominance, crowding distance, hypervolume | [`compass/agents/prompt_builder/search.py`](../compass/agents/prompt_builder/search.py) | +| Round advance (`advance_round_beam`), elite-set update, convergence | [`compass/agents/prompt_builder/search_ops.py`](../compass/agents/prompt_builder/search_ops.py) | +| Review Agent overlays (cold-start / iterative / post-cold-start) | `compass/agents/prompts/review_agent_*_overlay_beam.md` | +| Full algorithm write-up | [`docs/algorithm.md`](algorithm.md) | + +These two constants are the seam for a different strategy: change them and supply the matching `advance_round` implementation. `AlgorithmType` in `search.py` currently allows only `"beam"`. ## 7. Directory Guide @@ -351,8 +361,8 @@ The Review Agent prompt is assembled at dispatch time from three layers (iterati | `outputs//validation/` | Pipeline run: transformed dataset, quality report, routing context | | `outputs//analysis/` | Pipeline run: dev/holdout splits | | `outputs//prompts/` | Pipeline run: versioned routing prompts (v1.txt, v2.txt, ...) | -| `outputs//search/` | Pipeline run: search state, candidates, round reports, directive history | -| `outputs//search/viz.html` | Self-contained interactive visualization (tree + scatter + round slider); regenerated after each state mutation by `_try_write_viz` in [`search_ops.py`](../compass/agents/prompt_builder/search_ops.py). Node coloring reflects `on_front` (Pareto elite-set) membership; algorithm-specific overlays can be added on leaf branches. | +| `outputs//search/` | Pipeline run: search state, pending candidates, candidate archive, `viz.html` | +| `outputs//search/viz.html` | Self-contained interactive visualization (candidate tree + Pareto scatter + round slider); regenerated after each state mutation by `_try_write_viz` in [`search_ops.py`](../compass/agents/prompt_builder/search_ops.py). Nodes are coloured by Pareto elite-set membership. | | `outputs//search/child_variants.json` | `ChildVariant[]` — Review Agent output: grouped directives with parent preferences and hypotheses (canonical directive storage; retrieved via `get_child_variants`) | | `outputs//rerun_config.json` | Rerun mode marker: `mode`, `source_prompt_version`, `original_backend`, `new_backend` (null until Stage 3 completes) | | `outputs//search/search_state_original.json` | Preserved original search state from before rerun initiation | diff --git a/docs/project-overview.md b/docs/project-overview.md index 62c9843..eaab563 100644 --- a/docs/project-overview.md +++ b/docs/project-overview.md @@ -29,11 +29,11 @@ Doing this manually is slow, inconsistent, and hard to validate. This pipeline a ## 3. Pipeline Architecture -The pipeline is structured as six sequential stages, with an inner refinement loop. The diagram below maps to the Excalidraw design file (`Agentic-Workflow-Prompt-Routing.excalidraw`). +The pipeline is structured as six sequential stages, with an inner refinement loop. ``` ┌─────────────────────────────────────────────────────────┐ -│ Stage 1: User Input │ +│ Stage 1: Input Validation │ │ Collect routing dataset, problem description, metrics │ │ Check for blocking gaps; request clarification if any │ └────────────────────────────┬────────────────────────────┘ @@ -71,6 +71,10 @@ The pipeline is structured as six sequential stages, with an inner refinement lo └─────────────────────────────────────────────────────────┘ ``` +> Internally the orchestrator tracks **five** dispatcher stages +> (`compass/agents/pipeline/status.py`, `_STAGES`): Stages 5 and 6 above (Holdout Validation +> and Final Report) are one internal stage. See [docs/architecture.md](architecture.md). + --- ## 4. Agents @@ -272,7 +276,7 @@ compass/ agents/ # Agent implementations prompts/ # Agent system prompts eval/ # Evaluation engine (complete) - backends/ # Backend registry + LiteLLM client + backends/ # Backend registry (Anthropic, OpenAI, Bedrock, mock-echo) controller.py # Run Controller dataset.py # Dataset Manager metrics.py # Metrics Engine @@ -283,10 +287,10 @@ compass/ models.py # Data models docs/ # Eval engine documentation prompts/ # Prompt Manager -data/ # Dataset files (JSONL) -outputs/ # Run outputs and reports -prompts/ # Routing prompt store (versioned prompts) -configs/ # Run configuration YAML -tests/ # Full test suite +datasets/ # Paper appendix datasets (CC-BY-4.0) +outputs/ # Run outputs, reports, run_config.yaml (runtime, gitignored) +prompts/ # Routing prompt store — versioned prompts (runtime, gitignored) +backends/ # Backend profile YAML (runtime, gitignored) +tests/ # Full test suite (unit + tests/scenarios/ MCP runbooks) docs/ # Design specs and plans ```