A/B: does the /rlm skill (Opus root + Haiku leaf) beat plain Opus & Haiku agents on OOLONG?

[john-adeojo]

Goal

Does the /rlm skill — an Opus 4.8 root orchestrating a cheap Haiku 4.5 leaf — beat plain
Claude Code agents on the OOLONG trec_coarse eval (10 samples)? Compare that one RLM configuration
against the same two models run as standard agents (RLM off): Opus 4.8 and Haiku 4.5.

The three arms

Arm	Setup	Role
Opus agent (RLM off)	claude-opus-4-8, standard agent, tools on	control — capable but expensive
Haiku agent (RLM off)	claude-haiku-4-5, standard agent, tools on	control — cheap but collapses
RLM (RLM on)	claude-opus-4-8 root + claude-haiku-4-5 leaf, via /rlm	treatment

The RLM is a fixed configuration — Opus orchestrates, Haiku does the per-item labour. There is no
"Haiku-root RLM": the point of the scaffold is exactly opus-brain + cheap-labour.

What we're testing

• RLM vs Opus agent — can the scaffold match the frontier agent's accuracy at much lower cost?
  (The bulk is the cheap Haiku leaf; the Opus root only writes orchestration code over metadata.)
• RLM vs Haiku agent — does structuring the cheap work through the scaffold beat the plain Haiku
  agent's 20% collapse?

i.e. does the RLM break the accuracy/cost frontier the two agents define?

Matched harness — the only difference is the skill

All three arms run through one harness (run_rlm_skill_eval.py, --mode rlm|agent), identical in
task-prompt framing, single-shot headless claude -p, cold per-task state, disallowed
background/delegation tools, usage accounting, and score.py output format. The control arms run agent
mode (no Skill tool, plain prompt); the RLM arm runs skill mode. Do not reuse the old agent_*
baselines — re-run the controls through the matched harness.

Where artifacts go (keep the tree clean)

Every run writes only under a timestamped folder —
rlm_vs_agent_experiment/runs/<YYYYMMDD_HHMMSS>/<arm>/ — never loose in the experiment folder. Pass the
same --run-id to all three arms so one experiment groups together:

TS=$(date +%Y%m%d_%H%M%S)
python rlm_vs_agent_experiment/run_rlm_skill_eval.py --run-id $TS --mode agent --root opus
python rlm_vs_agent_experiment/run_rlm_skill_eval.py --run-id $TS --mode agent --root haiku
python rlm_vs_agent_experiment/run_rlm_skill_eval.py --run-id $TS --mode rlm   --root opus
python rlm_vs_agent_experiment/score.py --predictions rlm_vs_agent_experiment/runs/$TS/rlm_skill_opus/preds_rlm_skill.jsonl

runs/ is gitignored (local scratch); the committed deliverable is REPORT.md. This is a driver
convention only — the /rlm skill is unchanged (its transient .claude/rlm_state/ is wiped per task
and never touches the experiment folder).

What "cold" means (RLM arm)

The Opus root gets only context=<file> + query=<question> and discovers the strategy itself —
probe, decompose, sub-query the Haiku 4.5 leaf, aggregate in Python, set FINAL. No TREC labels, no
classify-then-count recipe, no OOLONG parsing; no reuse of the archived harness.

Done when

• All three arms run through the matched harness over all 10 samples (into one runs/<ts>/ folder),
  scored with score.py (default manifest path).
• Cost captured per arm — for the RLM arm root and leaf separately; for the controls, agent tokens.
• For the RLM arm, also report the orchestration-failure rate (tasks where the root couldn't drive
  the skill: no FINAL set, a forbidden/deferral tool, or a direct context read).
• All three arms written into REPORT.md as the RLM-vs-agents comparison.