Skip to content

chore(html-kit): close D04, promote D02 to next w/ design spec 19_ - #22

Merged
nyashkn merged 1 commit into
mainfrom
chore/d02-spec-d04-close
Jun 9, 2026
Merged

nyashkn merged 1 commit into
mainfrom
chore/d02-spec-d04-close

Conversation

@nyashkn

@nyashkn nyashkn commented Jun 9, 2026

Copy link
Copy Markdown
Owner

Summary

  • D02 (24-run isolation factorial) — `discovered` → `next`. Operational spec landed at `.html-kit/19_d02-factorial-design.md`. Implementation gated on `Dockerfile.factorial-pi` variant + openrouter cred re-add to Infisical `mi-eval-harness` scope.
  • D04 (score.py docstring vs framing alignment) — `discovered` → `done`. RESOLVED post-B1: docstring now enumerates 6 structural + 3 semantic checks; "structural-only" framing is gone.

D02 spec — 2×2×2 full factorial, k=3 trials per cell

Factor Levels
AGENT pi-0.75.4, omp-15.2.4
CHANNEL openrouter, opencode-go
MODEL claude-sonnet-4.6, deepseek-v4-flash
  • Block-randomised: 3 blocks of 8 cells, Latin-square shuffle within block.
  • Response variables: D01 score, per-check pass count, `socket_dropped` grep on agent stderr, `critic_md_bytes`, token counts (omp arm via `insight_kit.integrations.omp.session_parser`).
  • 3-way ANOVA + partial η² + 2-way interactions + socket-drop & verbosity marginals + cell-variance repeatability floor.
  • Pass/fail: GO on T24 RL coupling iff floor < 0.05 AND one factor explains ≥ 50% variance. HARD STOP if any cell hits 0.0 across all 3 trials.
  • Cost: ~4h sequential wall, ~$5 openrouter spend, ~30 min cred re-wire.

Test plan

  • Open `.html-kit/00_index.html` — verify card 19 appears.
  • Open `.html-kit/16_session-backlog-board.html` — verify D02 in Next-up (with link to 19_), D04 in Done.
  • Read `.html-kit/19_d02-factorial-design.md` end-to-end for statistical & operational consistency.

🤖 Generated with Claude Code

Board moves (16_session-backlog-board.html):
- D02 (24-run isolation factorial) — discovered → next. Operational spec
  landed; runner skeleton spec included. Now needs Dockerfile.factorial-pi
  variant + openrouter cred re-add to Infisical mi-eval-harness scope.
- D04 (score.py docstring vs framing) — discovered → done. RESOLVED post-B1:
  insight_kit.platform.eval.score docstring now enumerates "structural +
  semantic" with 3 semantic checks (claim_id regex, query_path citation
  cross-check, length prior 2-50KB). "Structural-only" framing is gone;
  stderr also prints per-check PASS/FAIL diagnostics.

New: 19_d02-factorial-design.md
- 2×2×2 full factorial: AGENT(pi/omp) × CHANNEL(openrouter/opencode-go) ×
  MODEL(sonnet-4.6/deepseek-v4-flash). k=3 trials per cell = 24 runs.
- Block randomisation: 3 blocks of 8, Latin-square shuffle within block.
- Response variables: D01 score, per-check pass count, socket_dropped grep
  on agent stderr, critic_md_bytes, token counts (omp arm via
  insight_kit.integrations.omp.session_parser).
- Analysis: 3-way ANOVA, partial η² per factor, 2-way interactions, socket-
  drop and verbosity marginals, cell-variance repeatability floor.
- Pass/fail: GO on T24 RL coupling iff floor <0.05 AND one factor explains
  ≥50% variance. HARD STOP if any cell hits 0.0 across all 3 trials.
- Runner contract: growth_insights/deploy/eval/factorial_runner.py
  (not yet implemented; shape sketched).
- Cost recap: ~4h sequential wall, ~$5 openrouter, ~30min cred re-wire.
- Cross-refs 15_ §4 (causal sketch source), eval/README.md (score gate +
  session parser), T24 (blocked on this experiment shipping a floor).

Index (00_index.html): card 19 added.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@nyashkn
nyashkn merged commit c064e8b into main Jun 9, 2026
@nyashkn
nyashkn deleted the chore/d02-spec-d04-close branch June 9, 2026 01:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant