Skip to content

Commit 5ca702e

Browse files
committed
docs: record the RLVR training-loop insight in the LLM section
Add a "The deploy loop is a training loop (RLVR)" subsection to research-landscape-2026.md: the fix-loop's proposal->verdict pairs are free labelled examples, so deployment is data generation. Captures why a sound deterministic verifier is a cleaner reward than RLHF's learned reward model; the free bootstrap from deterministic fixes + rejection-sampling SFT (no RL infra needed); and the load-bearing trap — reward hacking / Goodhart (checker-green != behaviour-preserved), so the metamorphic harness becomes part of the reward function, with the verifier setting both the ceiling and the floor. Re-confirms fix > spec from the training side (a fix has a verifiable reward; a mined spec does not). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rg8kSk1YT14x7A1vo5zgED
1 parent bd6e3e7 commit 5ca702e

1 file changed

Lines changed: 39 additions & 0 deletions

File tree

‎docs/notes/research-landscape-2026.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,45 @@ accept/reject. Honest caveat: **green ≠ behavior-preserving**, so the loop nee
119119
a **metamorphic conformance harness** (the doc's StaAgent / Statfier line) as the
120120
check, not human trust.
121121

122+
### The deploy loop is a training loop (RLVR) — and its one trap
123+
124+
The fix-loop has a second payoff that falls out for free: **every
125+
`proposal → verdict` pair is a labelled training example**, so the deployment
126+
loop *is* a data-generation loop. This is **RL from Verifiable Rewards (RLVR)** —
127+
the regime that carried the recent frontier math/code gains — and it fits here
128+
unusually well:
129+
130+
- The reward comes from a **deterministic, sound verifier**, not a learned reward
131+
model. RLHF's reward model is itself a fallible LLM (noisy, gameable); our
132+
checker + compile + tests is ground truth, so the signal is clean — precisely
133+
the verifiable-domain regime where RLVR actually works rather than just markets.
134+
- **No human-labelled dataset** — the checker manufactures the labels on real code.
135+
- **Bootstrap is free**: the deterministic fixes (the mechanical ~80%) are
136+
*guaranteed-correct demonstrations* to SFT on; the LLM takes the non-local ~20%,
137+
the verifier filters. Flywheel: deploy → propose → verify → keep winners → train
138+
→ better proposals.
139+
- Cheapest stable form needs no RL infra — **rejection sampling**: draw N fixes,
140+
keep the ones that pass checker + compile + tests, fine-tune on the winners.
141+
GRPO/PPO with the verifier as reward is a later, sample-efficiency move.
142+
143+
**The trap — and it is the same caveat as everywhere above.** "Checker went
144+
quiet" ≠ "bug fixed correctly". Optimise literally for that and the model learns
145+
**reward hacking / Goodhart**: delete the resource, suppress the diagnostic,
146+
weaken the code, remove the failing test — Own.NET green, behaviour broken. This
147+
is RLVR-for-code's documented failure mode (the model hacks the test suite, not
148+
the task). So the **metamorphic harness stops being only a deployment check and
149+
becomes part of the reward function**: the reward must be *checker-green AND
150+
behaviour-preserved* (metamorphic invariants / behaviour-diff / held-out tests),
151+
or you train a checker-gamer. The verifier sets both the **ceiling** (the model
152+
cannot exceed what the verifier can check — it learns the blind spots as exploits)
153+
and the **floor** (without behaviour checks it degrades).
154+
155+
This re-confirms **fix > spec** from the training side too: a **fix** has a clean
156+
verifiable reward (re-run checker + tests = the label); a **mined spec** has none
157+
— its correctness is not locally checkable (trusted base), so there is nothing
158+
sound to reward beyond consistency. The role worth training is the one we already
159+
chose — the **refactor assistant, not the truth generator**.
160+
122161
## Concrete backlog this implies (recorded, NOT scheduled)
123162

124163
Ordered by leverage; each annotated with where it attaches and its real cost.

0 commit comments

Comments
 (0)