Skip to content

fix-arm: thread full attempt history into AI revise loop - #16

Merged
PhysShell merged 3 commits into
mainfrom
claude/sts-runtime-analysis-2mo4z9
Jun 26, 2026
Merged

PhysShell merged 3 commits into
mainfrom
claude/sts-runtime-analysis-2mo4z9

Conversation

@PhysShell

@PhysShell PhysShell commented Jun 26, 2026 •

Copy link
Copy Markdown
Owner

What

Closes the highest-value gap the COMPILOT paper illuminated in our AI fix-arm: the local LLM client is stateless (complete(system, user) — no chat history), yet _revise only fed back the most recent rejection. Across revise rounds the model could re-propose an identical failing fix because it never saw its earlier rejected attempts.

Changes

  • fix/fixarm/ai_fix.py — _revise now accumulates a history of (rejected candidate, specific reason) and threads all of it into each prompt via a new _history_block ("do NOT repeat them; try a DIFFERENT fix"). This is the agent's only episodic memory given the stateless client.
  • Specific typed feedback — _fmt_findings turns "it introduced new findings" into the actual list (rule@file:line), giving the model something concrete to act on.
  • docs/agent-loop-lessons.md — maps COMPILOT's findings onto our loop: what it validates (deterministic-verifier-not-LLM, the re-audit gate, give-up tracking), what we adopt now (this change), and a menu for later (best-of-K restarts, reasoning-first CoT, cheap pre-filter before the expensive re-audit, measured iteration budget).
  • fix/tests/test_ai_fix.py — test_ai_revise_accumulates_history asserts the round-3 prompt carries both prior rejected candidates and the specific reason.

Why it's safe

No change to the trust model: the LLM still only proposes, the real analyzers re-audit, any fix that introduces a new finding is rejected and reverted, and the gate stays REVIEW (never auto-commit). This only improves how a weak local model converges inside that envelope.

Tests

All suites pass under bare python3 and -O (test_ai_fix 10/10 including the new test).

🤖 Generated with Claude Code


Generated by Claude Code

Summary by CodeRabbit

  • New Features
    • Improved AI-assisted fix retries by carrying forward prior rejected attempts and their reasons into later prompts.
    • Added clearer, round-by-round context for each revise attempt.
  • Bug Fixes
    • Reduced repeated failing revisions by making retries aware of earlier rejected candidates; fixes now stop when the issue is cleared without introducing new problems.
  • Tests
    • Added coverage for multi-round fix attempts to verify rejected-history is preserved in prompts.
  • Documentation
    • Added “Agent-loop lessons” documentation explaining the AI loop behavior and iteration stopping/give-up guidance.

…MPILOT lessons

Mined the COMPILOT paper (LLM-in-a-loop with a compiler) for transferable agent-loop
mechanics. It strongly validates the fix-arm design (delegate correctness to a
deterministic verifier — their direct-codegen variant had 17.6% false positives at
5.3x tokens; our re-audit + no-new-findings reject is the safe path) and surfaces one
real, cheap gap:

- The LLM client is stateless (complete(system,user), no chat history), but _revise
  only injected the LAST rejection reason — so the model could re-propose an identical
  failing fix. COMPILOT's loop works because every prior outcome stays in the dialogue
  (illegal-rate drops over the dialogue as negative feedback accumulates). Now _revise
  threads the full history of rejected candidates + specific reasons ("do NOT repeat
  them; try a DIFFERENT fix"), and "introduced new findings" now lists which
  (rule@file:line). +1 test asserting round-3 carries both prior attempts.

- docs/agent-loop-lessons.md: the full mapping (validated / adopted-now / menu-for-later:
  best-of-K restarts, reasoning-first, cheap pre-filter before re-audit, measured budget).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LLHz4cevQyXHKZnjzCqhoa

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 827855a539

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread fix/fixarm/ai_fix.py Outdated
if introduced else "the finding is still reported")
reason = (f"it introduced new findings: {_fmt_findings(introduced)}"
if introduced else f"the finding [{f.rule}] is still reported")
history.append(("".join(cand), reason)) # accumulate — episodic memory

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Store only the rejected replacement in revise history

When a revise round is rejected, cand is the entire spliced file because _propose() returns cur[:a] + repl + cur[b:], so this appends a full-file snapshot to every later prompt rather than just the attempted replacement/window. On large C# files, two rejected rounds can add thousands of unrelated lines to a local model's context, causing truncation or context-limit/API failures and making the revise loop less reliable than the original last-reason feedback. Capture the replacement/diff for the requested range instead of "".join(cand).

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, fixed in 543d390. _propose() returns the full spliced file, so "".join(cand) was storing a whole-file snapshot per rejected round — exactly the context bloat you flagged. Now storing only the proposed replacement window (cand[a:len(cand)-(len(cur)-b)]). Added a regression assertion that the outer file lines (namespace …) never leak into a later prompt.


Generated by Claude Code

_propose() returns the whole spliced file (cur[:a] + repl + cur[b:]), so
appending "".join(cand) to the revise history put a full-file snapshot into
every later prompt. On a large .cs file two rejected rounds add thousands of
unrelated lines to the local model's context — truncation / context-limit
failures, exactly the reliability hit the history was meant to avoid. Store
only the proposed replacement window (cand[a:len(cand)-(len(cur)-b)]). Test
strengthened to assert the outer file lines never leak into the prompt.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LLHz4cevQyXHKZnjzCqhoa
@coderabbitai

coderabbitai Bot commented Jun 26, 2026 •

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 951b4dc9-3b94-4352-95ac-2e080c591455

📥 Commits

Reviewing files that changed from the base of the PR and between 543d390 and f630b05.

📒 Files selected for processing (2)
  • fix/fixarm/ai_fix.py
  • fix/tests/test_ai_fix.py

📝 Walkthrough

Walkthrough

The PR adds agent-loop lessons documentation, updates the fix-arm revise loop to thread rejected-candidate history and richer feedback through retries, and adds a test that checks multi-round prompt history accumulation.

Changes

Agent-loop lessons and revise history

Layer / File(s) Summary
Lessons doc
docs/agent-loop-lessons.md
Documents the agent loop, verification behavior, iteration stopping, adopted history and reason threading, and follow-on ideas.
Prompt feedback formatting
fix/fixarm/ai_fix.py
build_user now inserts provided feedback directly, and new helpers format finding lists and prior rejected attempts.
Revise history loop
fix/fixarm/ai_fix.py, fix/tests/test_ai_fix.py
_revise keeps rejected windows and reasons across rounds, reuses them on each proposal, and the new test checks the third-round prompt and final acceptance.

Sequence Diagram(s)

sequenceDiagram
  participant R as "AiFixApplier._revise"
  participant H as "_history_block"
  participant B as "build_user"
  participant V as "reaudit"

  loop each revise round
    R->>H: render rejected attempts
    R->>B: pass feedback text into the prompt
    B-->>R: prompt content
    R->>V: audit candidate
    V-->>R: findings or clear result
    alt candidate rejected
      R->>R: append rejected window and reason to history
    end
  end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

I hopped through prompts with twitchy nose,
And tucked the rejects in tidy rows.
Old failures thumped, then softly spun,
New fixes bloom—one, two, then done.
🐇✨ The loop goes hop, audit, hop.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: threading full attempt history into the AI revise loop.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/sts-runtime-analysis-2mo4z9

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@fix/fixarm/ai_fix.py`:
- Around line 185-189: The rejection reason in ai_fix.py currently only reports
introduced findings when diff_findings returns non-empty, even if the original
finding is still present; update the reason construction in the verification
block around introduced/_still_present to preserve both rejection causes when
both checks fail. Keep the early verified return unchanged, but when building
reason for cand, combine the “introduced new findings” message with the “finding
is still reported” message instead of choosing only one, so the feedback
reflects both the remaining issue and any regressions.

In `@fix/tests/test_ai_fix.py`:
- Around line 172-175: The round-3 prompt assertion in test_ai_fix should verify
that the rejection reason is attached to each prior attempt, not just present
somewhere in the full prompt. Update the assertions around the
client.calls[2][1] prompt so the reason text is checked per attempt header or
counted for both “attempt 1” and “attempt 2”, using the existing third variable
and FINDING.rule to locate the relevant prompt content.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 4e1a835c-ef0e-4200-a9f7-46d01e46f92a

📥 Commits

Reviewing files that changed from the base of the PR and between 18b10b5 and 543d390.

📒 Files selected for processing (3)
  • docs/agent-loop-lessons.md
  • fix/fixarm/ai_fix.py
  • fix/tests/test_ai_fix.py

Comment thread fix/fixarm/ai_fix.py Resolved
Comment thread fix/tests/test_ai_fix.py Resolved
…checks

When a revise candidate BOTH leaves the original finding present AND introduces
a new one, the reason only reported the introduced findings — dropping the
"still reported" signal and steering the next prompt away from the unfixed
target. Combine both causes with "; ". Test tightened to assert the rejection
reason is attached to each prior attempt (count == 2), not merely present once.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LLHz4cevQyXHKZnjzCqhoa
@PhysShell
PhysShell merged commit 1708aae into main Jun 26, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants