Repository navigation
fix-arm: thread full attempt history into AI revise loop - #16
Conversation
…MPILOT lessons
Mined the COMPILOT paper (LLM-in-a-loop with a compiler) for transferable agent-loop
mechanics. It strongly validates the fix-arm design (delegate correctness to a
deterministic verifier — their direct-codegen variant had 17.6% false positives at
5.3x tokens; our re-audit + no-new-findings reject is the safe path) and surfaces one
real, cheap gap:
- The LLM client is stateless (complete(system,user), no chat history), but _revise
only injected the LAST rejection reason — so the model could re-propose an identical
failing fix. COMPILOT's loop works because every prior outcome stays in the dialogue
(illegal-rate drops over the dialogue as negative feedback accumulates). Now _revise
threads the full history of rejected candidates + specific reasons ("do NOT repeat
them; try a DIFFERENT fix"), and "introduced new findings" now lists which
(rule@file:line). +1 test asserting round-3 carries both prior attempts.
- docs/agent-loop-lessons.md: the full mapping (validated / adopted-now / menu-for-later:
best-of-K restarts, reasoning-first, cheap pre-filter before re-audit, measured budget).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LLHz4cevQyXHKZnjzCqhoa
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 827855a539
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if introduced else "the finding is still reported") | ||
| reason = (f"it introduced new findings: {_fmt_findings(introduced)}" | ||
| if introduced else f"the finding [{f.rule}] is still reported") | ||
| history.append(("".join(cand), reason)) # accumulate — episodic memory |
There was a problem hiding this comment.
Store only the rejected replacement in revise history
When a revise round is rejected, cand is the entire spliced file because _propose() returns cur[:a] + repl + cur[b:], so this appends a full-file snapshot to every later prompt rather than just the attempted replacement/window. On large C# files, two rejected rounds can add thousands of unrelated lines to a local model's context, causing truncation or context-limit/API failures and making the revise loop less reliable than the original last-reason feedback. Capture the replacement/diff for the requested range instead of "".join(cand).
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch, fixed in 543d390. _propose() returns the full spliced file, so "".join(cand) was storing a whole-file snapshot per rejected round — exactly the context bloat you flagged. Now storing only the proposed replacement window (cand[a:len(cand)-(len(cur)-b)]). Added a regression assertion that the outer file lines (namespace …) never leak into a later prompt.
Generated by Claude Code
_propose() returns the whole spliced file (cur[:a] + repl + cur[b:]), so appending "".join(cand) to the revise history put a full-file snapshot into every later prompt. On a large .cs file two rejected rounds add thousands of unrelated lines to the local model's context — truncation / context-limit failures, exactly the reliability hit the history was meant to avoid. Store only the proposed replacement window (cand[a:len(cand)-(len(cur)-b)]). Test strengthened to assert the outer file lines never leak into the prompt. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLHz4cevQyXHKZnjzCqhoa
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe PR adds agent-loop lessons documentation, updates the fix-arm revise loop to thread rejected-candidate history and richer feedback through retries, and adds a test that checks multi-round prompt history accumulation. ChangesAgent-loop lessons and revise history
Sequence Diagram(s)sequenceDiagram
participant R as "AiFixApplier._revise"
participant H as "_history_block"
participant B as "build_user"
participant V as "reaudit"
loop each revise round
R->>H: render rejected attempts
R->>B: pass feedback text into the prompt
B-->>R: prompt content
R->>V: audit candidate
V-->>R: findings or clear result
alt candidate rejected
R->>R: append rejected window and reason to history
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@fix/fixarm/ai_fix.py`:
- Around line 185-189: The rejection reason in ai_fix.py currently only reports
introduced findings when diff_findings returns non-empty, even if the original
finding is still present; update the reason construction in the verification
block around introduced/_still_present to preserve both rejection causes when
both checks fail. Keep the early verified return unchanged, but when building
reason for cand, combine the “introduced new findings” message with the “finding
is still reported” message instead of choosing only one, so the feedback
reflects both the remaining issue and any regressions.
In `@fix/tests/test_ai_fix.py`:
- Around line 172-175: The round-3 prompt assertion in test_ai_fix should verify
that the rejection reason is attached to each prior attempt, not just present
somewhere in the full prompt. Update the assertions around the
client.calls[2][1] prompt so the reason text is checked per attempt header or
counted for both “attempt 1” and “attempt 2”, using the existing third variable
and FINDING.rule to locate the relevant prompt content.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 4e1a835c-ef0e-4200-a9f7-46d01e46f92a
📒 Files selected for processing (3)
docs/agent-loop-lessons.mdfix/fixarm/ai_fix.pyfix/tests/test_ai_fix.py
…checks When a revise candidate BOTH leaves the original finding present AND introduces a new one, the reason only reported the introduced findings — dropping the "still reported" signal and steering the next prompt away from the unfixed target. Combine both causes with "; ". Test tightened to assert the rejection reason is attached to each prior attempt (count == 2), not merely present once. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLHz4cevQyXHKZnjzCqhoa
What
Closes the highest-value gap the COMPILOT paper illuminated in our AI fix-arm: the local LLM client is stateless (
complete(system, user)— no chat history), yet_reviseonly fed back the most recent rejection. Across revise rounds the model could re-propose an identical failing fix because it never saw its earlier rejected attempts.Changes
fix/fixarm/ai_fix.py—_revisenow accumulates ahistoryof(rejected candidate, specific reason)and threads all of it into each prompt via a new_history_block("do NOT repeat them; try a DIFFERENT fix"). This is the agent's only episodic memory given the stateless client._fmt_findingsturns "it introduced new findings" into the actual list (rule@file:line), giving the model something concrete to act on.docs/agent-loop-lessons.md— maps COMPILOT's findings onto our loop: what it validates (deterministic-verifier-not-LLM, the re-audit gate, give-up tracking), what we adopt now (this change), and a menu for later (best-of-K restarts, reasoning-first CoT, cheap pre-filter before the expensive re-audit, measured iteration budget).fix/tests/test_ai_fix.py—test_ai_revise_accumulates_historyasserts the round-3 prompt carries both prior rejected candidates and the specific reason.Why it's safe
No change to the trust model: the LLM still only proposes, the real analyzers re-audit, any fix that introduces a new finding is rejected and reverted, and the gate stays REVIEW (never auto-commit). This only improves how a weak local model converges inside that envelope.
Tests
All suites pass under bare
python3and-O(test_ai_fix10/10 including the new test).🤖 Generated with Claude Code
Generated by Claude Code
Summary by CodeRabbit