Repository navigation
Phase 4: Repairer (repair + validate + routing + repair eval) - #4
Merged
Merged
Conversation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…hten prompt test Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…/style Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…report Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… test case Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… test Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rden rubric flag Pass original_content into build_validation_user_message so the LLM has both ORIGINAL and REWRITTEN sections when judging style_consistent. OR the model's needs_human_review flag with a deterministic min-score<=3 check in judge_repair so low scores are always flagged regardless of what the model returns. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e block preservation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 4 of DocPulse v1 — the repairer: given a confirmed-stale doc section + verdict + code change, produce a style-preserving corrected section, validate it, route it by confidence, and measure repair quality in the eval harness.
repair_modelconfig — optional override (Config.resolve_repair_model()), falls back tomodel.src/docpulse/repair/repairer.py— single-shot structured-outputrepair()(submit_repairtool schema), one retry on malformed output, fail-safe to unchanged content on any LLM/parse failure.validator.py— deterministicpreservation_ratio()(fraction of original paragraph blocks surviving byte-identical) +validate()= preservation gate (≥0.5) × LLM accuracy-vs-new-code × style-vs-original.routing.py— pureroute(verdict, repair, config) -> auto_fix | draft | skip.prompts.py— repairer + validator prompts.src/docpulse/eval/repair_harness.py— eval-only: synthesizes a stale verdict from eachlabel.yml(isolates repair quality from verifier noise), runs repair→validate→route, LLM rubric judge scoring vsreference_correction(accuracy/completeness/style 1–5 + human-review flag), aggregates mean preservation, % of stale cases ≥0.95 preserved (exit-gate metric), rubric means, flagged count.docpulse eval --repair— runs the repair eval after the verify table; prints per-case preservation/tier/rubric + aggregates + unified diffs.Deferred to Phase 5 (consistent with Phase 3 keeping CLI orchestration out): the standalone git-diff-driven
docpulse repair --basepipeline. The validator is product-ready; only its CLI entrypoint waits.Not in this PR — keyed measurement: the exit-gate run (
docpulse eval --repair --model anthropic/claude-haiku-4-5) needsANTHROPIC_API_KEY, like Phase 3's baseline. Running it yields the ≥95%-preservation number + rubric scores; STATUS.md / 03-plan.md get updated after.Built subagent-driven (TDD, per-task spec + quality review, plus a final integrated review that caught and fixed a real bug: the validator's style check wasn't being shown the original section).
Test Plan
uv run pytest -q— 132 passeduv run ruff check— clean (repo-wide)--repairagainst the 6 seed stale cases) — requiresANTHROPIC_API_KEY🤖 Generated with Claude Code