Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .html-kit/00_index.html
Original file line number Diff line number Diff line change
Expand Up @@ -848,6 +848,11 @@ <h4><a href="16_session-backlog-board.html">Three days of work, laid flat</a></h
<h4><a href="17_laminar-coverage-audit.html">Laminar coverage — 2 of 7</a></h4>
<p style="font-size:13px;color:var(--g700);max-width:none">Opus audit of the T17 Laminar wiring. Phase wall-times + artifacts captured; everything inside the 535s omp black box (per-turn LLM calls, ~30 DuckDB queries, persona switches) is opaque. <code>lmnr-claude-code-proxy</code> verdict, 1-hour quick wins, 1-week session-JSON post-parse path.</p>
</div>
<div class="card" style="border-top:3px solid var(--clay)">
<div class="tag">18 · T25 debate trigger spec</div>
<h4><a href="18_t25-debate-trigger-spec.md">When to escalate to opus + N-builder debate</a></h4>
<p style="font-size:13px;color:var(--g700);max-width:none">Cost-gated escalation trigger for multi-agent debate. Five primitives OR-composed: score floor (&lt;0.7), critic-severity cluster (≥2 HIGH/CRIT), 3-run stagnation (Δ&lt;0.05 &amp; score&lt;0.85), cite-graph contradiction, manual override. Asymmetric bias: false-positive (~5x cost) cheaper than false-negative (bad claim into RL signal).</p>
</div>
</div>

<footer>
Expand Down
20 changes: 9 additions & 11 deletions .html-kit/16_session-backlog-board.html
Original file line number Diff line number Diff line change
Expand Up @@ -366,24 +366,22 @@ <h3>Board state — paste into next session</h3>
summary:'Validate Hamilton DAG aspects of generated code via omp --no-lsp flip — gives us a typed editing surface for the analyst sub-agent.' },
{ id:'T25', col:'next', tag:'orch', size:'l',
title:'Multi-agent debate orchestration (opus deep + sonnet builders)',
summary:'Opus runs the deep critique, N sonnet builders argue. Needs a concrete decision-fork trigger before this earns its run cost.',
blocks:'needs trigger spec' },
summary:'Opus runs the deep critique, N sonnet builders argue. Trigger spec landed — see 18_t25-debate-trigger-spec.md (score < 0.7 OR ≥2 HIGH/CRIT critic claims OR 2 flat/regressing runs OR cite-graph contradiction OR --debate override).',
link:'18_t25-debate-trigger-spec.md', linkLabel:'trigger spec →' },

// ---------- BLOCKED / FUTURE ----------
{ id:'T21', col:'blocked', tag:'telemetry', size:'l',
{ id:'T21', col:'done', tag:'telemetry', size:'l',
title:'Wire e2e trace OOB (omp sessions + parser + dashboard)',
summary:'Stand up omp session export, write the parser, ship a dashboard. Blocked by the telemetry decision (Phoenix vs Langfuse vs roll-your-own).',
blocks:'blocked by telemetry stack decision (see #19)' },
{ id:'T24', col:'blocked', tag:'eval', size:'l',
summary:'Telemetry stack decision closed: Laminar (self-host on lan.ds.ke for gRPC + cloud fallback). Parser shipped (insight_kit.integrations.omp.session_parser, B1). Back-emit helpers shipped (insight_kit.platform.observability.lmnr.emit_session_spans, B2). Dashboard = Laminar UI; per-run sessions keyed by git SHA pending wire-up in growth_insights lmnr_harness.main().' },
{ id:'T24', col:'next', tag:'eval', size:'l',
title:'Agent-as-judge + Harbor connection',
summary:'Harness the run-level judge into the Harbor RL signal loop. Blocked by score.py semantic hardening (D01) — judging on structural completeness alone produces a weak gradient.',
blocks:'blocked by D01 (score.py semantic hardening)' },
summary:'Harness the run-level judge into the Harbor RL signal loop. Unblocked: D01 shipped (semantic gradient now meaningful). Next: wire score.py → Harbor reward function, decide signal shape (raw 0..1 vs delta-vs-baseline).' },

// ---------- DISCOVERED (this session) ----------
{ id:'D01', col:'discovered', tag:'discovered', size:'l',
{ id:'D01', col:'done', tag:'eval', size:'l',
title:'score.py semantic hardening',
summary:'Add claim-id regex check, citation cross-validation, length-distribution priors. Today\'s score.py is structural-completeness only. Blocks T24 — RL on structural-only signal will overfit to format.',
link:'15_t17-smoke-1pt0-critic.html', linkLabel:'critic § on this →' },
summary:'SHIPPED via insight-kit B1 — insight_kit.platform.eval.score: 9-check gate (3 structural, 3 file-existence, claim_id regex w/ T-suffix accept, query_path citation cross-check, length prior 2-50KB). Unblocks T24.',
link:'../eval/README.md', linkLabel:'README §score →' },
{ id:'D02', col:'discovered', tag:'discovered', size:'l',
title:'24-run isolation factorial (8 cells × 3 trials)',
summary:'Attribute the 0.83 → 1.00 jump to agent / channel / model. Three variables changed at once in T18 — n=1 cannot tell us which one mattered. Blocks any honest RL signal.',
Expand Down
95 changes: 95 additions & 0 deletions .html-kit/18_t25-debate-trigger-spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# T25 — Multi-agent debate orchestration: trigger spec

**Status:** spec landed 2026-06-09. Cards: `[[16_session-backlog-board]]` § T25.
**Scope:** *when* to spawn Opus-deep + N-sonnet-builder debate vs single-agent analyst/critic. Cost gate. Not architecture.

## Why a trigger gate

Debate run = O(5x) cost of single analyst/critic loop (1 opus deep + ~4 sonnet builders + synthesis). Default-on burns budget on the 80% of runs single-pass handles cleanly. Default-off leaves the 20% hard cases under-resolved.

Trigger = automatic escalation signal. Cheap to evaluate. False-positive cost = wasted ~5x run. False-negative cost = bad claim ships.

## Trigger primitives (OR-composed)

A run escalates to debate if **any** primitive fires.

### TR1 — Score floor
**Condition:** `insight_kit.platform.eval.score(run_dir) < 0.7` after analyst pass.

**Why:** below 0.7 means ≥3 of 9 gate checks failed. Structural floor breached → single-agent retry rarely recovers. Empirical anchor: T17 May 25 real run = 0.8889. 0.7 = ~3σ below the working-baseline ceiling.

**Tuning hook:** `T25_SCORE_FLOOR` env. Default 0.7.

### TR2 — Critic severity cluster
**Condition:** critic emits ≥2 claims with `severity ∈ {HIGH, CRITICAL}` against the analyst's published claims in a single pass.

**Why:** one HIGH/CRIT can be a single-issue patch; two+ signals systemic disagreement that debate is designed to resolve. Single critic is one perspective; debate adds adversarial diversity.

**Tuning hook:** `T25_CRITIC_THRESHOLD` env. Default 2.

### TR3 — Stagnation across runs
**Condition:** rolling 3-run score window shows `Δ_max < 0.05` AND latest score `< 0.85`.

**Why:** flat-and-mediocre = single-agent loop has converged on a local minimum. Debate forces re-attack from independent angles. Excludes flat-at-the-top (≥0.85) since flat-good is the goal state, not stagnation.

**Tuning hook:** `T25_STAGNATION_WINDOW` (default 3), `T25_STAGNATION_DELTA` (default 0.05), `T25_STAGNATION_CEILING` (default 0.85).

### TR4 — Cite-graph contradiction
**Condition:** `query_cites` over `record.json` surfaces a published claim A whose `cites` includes a claim B where B is itself `refuted` by a sibling critic claim (T29 supports/refutes edges).

**Why:** publishing on a refuted foundation is a data-integrity bug, not a style issue. Single-pass critic missed it (or the refutation came later). Debate forces re-derivation. Catches RT5 (adversarially wrong claims publish) at the cite layer.

**Tuning hook:** none — boolean trigger.

### TR5 — Manual override
**Condition:** `--debate` flag passed to eval runner, OR `T25_FORCE_DEBATE=1` env set.

**Why:** human-in-the-loop override for known-hard goals (founder explicitly says "go deep on this one"). No automatic equivalent — judgment call lives with the operator.

## Composition rule

Escalate if `TR1 OR TR2 OR TR3 OR TR4 OR TR5`. No AND-gating — any single fire is sufficient. Rationale: false-positive (debate on an easy run) costs ~5x; false-negative (single-pass on a hard run) ships a bad claim into the Harbor RL signal. Asymmetric → bias toward over-escalation.

## What does NOT trigger

- **Single LOW/MED critic claim.** Single-pass critic-then-revise handles this without debate.
- **Score ≥0.7 but <0.85 on first pass.** Normal working band. Let the critic loop retry once before debating.
- **Long wall-time.** Slow ≠ hard. Cred/network latency is not a content signal.
- **Tool-call count.** High tool-call count = analyst used the tools they were given. Not a quality signal.

## Implementation surface (forward-pointer)

When wiring lands, expected shape (NOT spec'd here):

```python
from insight_kit.platform.orchestration.debate import (
should_escalate, DebateTrigger,
)

triggers: list[DebateTrigger] = should_escalate(
run_dir=run_dir,
rolling_window=load_recent_scores(n=3),
critic_severity_threshold=int(os.getenv("T25_CRITIC_THRESHOLD", "2")),
score_floor=float(os.getenv("T25_SCORE_FLOOR", "0.7")),
)
if triggers or args.debate:
run_debate(run_dir, reason=triggers or "manual")
else:
finalize_single_pass(run_dir)
```

Module `insight_kit.platform.orchestration.debate` does not exist yet. Card moves to `next` when implementation is claimed.

## Open questions (not spec'd here — defer to impl session)

- Builder count N — 2, 3, 4? Cost-quality elbow not measured.
- Builder model — sonnet 4.6 vs sonnet 4.8? Pre-empts the same question for council.
- Synthesis step — single opus pass or a 2nd debate round? Defaulting to single pass; revisit if synth quality is the bottleneck.
- Cite-graph TR4 — requires T29 critic edges to be live in the consumer repo. `growth_insights` status: T29 shipped on `insight-kit` gate, consumer wiring TBC.

## Cross-refs

- Cost model: `[[14_agentic-coding-observability]]` § §6 single-pass economics
- Critic edges schema: `SPEC.md` § §T T29
- Score gate: `eval/README.md` § `insight_kit.platform.eval.score`
- Board: `[[16_session-backlog-board]]` § next-up T25
Loading