Repository navigation
Description
Problem
Mined 232 reject findings across 68 PRs with review comments from the last 2 weeks (2026-04-01 to 2026-04-15). These are reviewer comments that resolve-review classified as factually wrong or flagging intentional design patterns.
| Category | Count | % |
|---|---|---|
documented_behavior |
73 | 31.5% |
guard_scope |
43 | 18.5% |
stale_comment |
25 | 10.8% |
assert_boundary |
15 | 6.5% |
api_misunderstanding |
14 | 6.0% |
except_handler |
13 | 5.6% |
stdlib_isolation |
9 | 3.9% |
asyncio_pattern |
8 | 3.4% |
never_raises |
8 | 3.4% |
deferred_import |
5 | 2.2% |
platform_specific |
5 | 2.2% |
other |
14 | 6.0% |
Data Methodology Caveats
The mining methodology has five known weaknesses that should be addressed before full deployment:
- Selection bias: Rejects identified by keyword search ("Investigated", "intentional") in reply comments. Silent dismissals (no reply posted) are missed. Recommendation: audit 50 no-reply PRs to bound the silent-reject rate.
- Survivorship bias: Only PRs where resolve-review ran are included, pre-selecting FP-dense PRs. The true FP rate across all review-pr invocations is likely lower.
- Unknown base rate: 232 rejects found but total finding count across the 68 PRs is not computed. Per-category accept/reject ratios are needed to confirm suppression rules target the right patterns. Recommendation: pull total finding counts from session logs before deployment.
- Circular confounding: resolve-review uses the same Sonnet model family to classify rejects. Systematic model blind spots (asyncio semantics, control flow analysis) may produce unreliable reject classifications. The
asyncio_patternandguard_scopecategories carry the highest confounding risk. - Temporal bias: 2-week window during active convention establishment.
documented_behaviordominance may be a transient artifact of convention churn, not a permanent signal.
Minimum data strengthening before deployment: Compute total finding counts, human-validate 30 stratified findings (2 per category weighted toward Tier 1), re-run analysis on PRs from 4-8 weeks ago to check temporal stability.
Architecture: Hybrid Approach (Not Pure Prompt Rules)
Red-team validation and gap analysis identified that pure prompt-rule injection is suboptimal. The false-positive landscape has three distinct clusters requiring different interventions:
| Cluster | Rules | % of Rejects | Best Treatment | FN Risk |
|---|---|---|---|---|
| Deterministic patterns | G, J, K, L | 11% | Pre-filter in diff_annotator.py |
Zero |
| Verification failures | C, E | 17% | Two-pass self-critique (8th subagent) or prompt rules | Low |
| Contextual reasoning | A, B, D, F, H, I | 72% | Prompt rules with tightening | Medium |
Key architectural insight: Stage 2 (resolve-review) has full codebase access + git history and is fundamentally more reliable for context-dependent decisions. A false negative at stage 1 costs one extra REJECT classification in stage 2 -- not a missed production bug. This asymmetry justifies accepting medium FN risk at stage 1 to reduce stage 2's ~58% noise rate.
Component 1: Deterministic Pre-Filtering (diff_annotator.py)
Add a classify_diff_files() function that returns per-file tags injected into diff headers. No LLM reasoning needed.
Rule G (stdlib-isolation): If file path starts with hooks/ and first 30 lines contain "stdlib-only", tag as [STDLIB_ONLY]. Suppress cohesion/arch findings on these files.
Rule K (platform-scoped): If file contains module-level pytestmark = pytest.mark.skipif(sys.platform, tag as [PLATFORM_GUARDED].
Rule L (convention-separators): Strip # ----- separator comments from annotated diff before sending to slop subagent, or tag lines as [CONVENTION].
Rule J (deferred-import): Tag in-function imports with [DEFERRED_IMPORT] marker so arch subagent can skip circular-risk flags.
Implementation: ~50-80 lines in diff_annotator.py. Zero false-negative risk. Eliminates ~25 rejects per 2-week window.
Component 2: Prompt Rules (Subagent Template)
Inject into the subagent prompt template in review-pr/SKILL.md between the severity instructions and the diff content. Maximum 5 rules to avoid attention dilution.
Rule A: documented-behavior-annotation (73 rejects)
Dimensions: all | Validated FN risk: Medium
Tightened after red-team validation:
Before downgrading a finding, documentation must: (a) be within 2 lines of the flagged code, (b) affirm the behavior as intentional (TODO/FIXME/should language does NOT qualify), (c) be on the enclosing function/class not an ancestor, (d) for
# noqathe annotation must name the relevant rule category, (e) not be contradicted by the implementation. When all conditions are met, downgrade toinfoseverity.
Rule B: guard-scope-misread (43 rejects)
Dimensions: defense, bugs | Validated FN risk: Medium
Tightened:
Before flagging a missing guard, verify no enclosing scope provides equivalent protection. The guard must specifically cover the flagged failure mode: except clauses must catch the relevant exception type (or be documented as intentionally broad with context-preserving logging),
|| truemust be on provably idempotent cleanup (kill, unlink) with no downstream dependency, module-level markers must be the specific marker providing the protection.
Rule D: except-exception-with-logging (13 rejects)
Dimensions: defense | Validated FN risk: Medium
Tightened:
Do NOT flag
except Exceptionas swallowed when the handler logs withexc_info=True(orlogger.exception()) at WARNING level or above AND either re-raises or returns a documented sentinel. Retain the finding when: log level is DEBUG only, handler returns a value causing silent correctness failure in callers, logging is inside a nested try/except, or an explicit exception tuple has a barepassbody.
Rule F: assert-boundary-misread (15 rejects)
Dimensions: tests | Validated FN risk: Low-Medium
Only suppress "weak assertion" flags when the test name/docstring explicitly states smoke/presence/at-least-one semantics AND the function under test is not safety-critical AND at least one other test in the same file covers the exact-count case.
Rule I: never-raises-contract (8 rejects)
Dimensions: defense, bugs | Validated FN risk: Medium
Tightened:
Only suppress when: (1) the never-raises contract is visible in the diff, (2) the callee contains NO
raisein the same diff, (3) the callee does not make unwrapped I/O calls, (4) the callee is not abstract/protocol. If the diff modifies the callee to add araise, this rule is disabled entirely for that callee.
Component 3: Self-Critique Pass (Verification Failures)
Instead of prompt rules for C and E, add a sequential 8th subagent after the 7 parallel audit subagents that verifies each finding against the diff content.
Rule C (stale-findings, 25 rejects) + Rule E (hallucination-verification, 14 rejects):
Critique subagent prompt: "For each finding below, verify that the claimed issue is supported by evidence visible in the provided diff. Check: (1) if the finding claims an element is missing, scan all + lines across all files to confirm it is not present, (2) if the finding claims a syntax error, verify the syntax is actually wrong. Mark each finding as CONFIRMED or RETRACT with a one-line reason."
Latency cost: ~30-90 seconds additional per review. Covers 17% of false positives with low FN risk.
Component 4: Cross-Rule Interaction Clause (MANDATORY)
When two or more suppression rules would apply to the same finding, downgrade to
infoseverity rather than suppressing entirely. Note which rules applied.
Specific high-risk compounding interactions identified by red-team validation:
- A + B: Documented behavior + enclosing guard -- both suppress same finding
- D + stale-fix: Fix commit adds
exc_info=True+ final-state verification = compounded - G + I: stdlib isolation + never-raises = unhandled exceptions in hooks invisible
Component 5: Feedback Loop (Phase 2)
Long-term, resolve-review's reject_patterns files should be automatically fed back to review-pr as a dynamic prompt appendix. The infrastructure (reject pattern persistence in resolve-review Step 6.6) already exists. The missing piece is a recipe step that reads the accumulated corpus and generates the appendix.
This makes the static rules eventually unnecessary as the system learns from its own reject history. The static rules are a sound interim position.
Generalizability Assessment
| Universality | Rules | % of Rejects |
|---|---|---|
| Fully universal (any Python project) | A, C, D, E, F, J | 62.5% |
| Universal with language rewrite | B, I, K | 24% |
| Narrowly project-specific | G, H, L | 9% |
Rules G, H, L should be documented as project-specific. If this suppression framework is intended to generalize, it should distinguish universal rules, language-specific modules, and project-specific overrides.
Missing categories not present in this project: generated/scaffolded files (protobuf, ORM migrations), CSS/frontend patterns, API versioning duplication, database migration conventions. These would need separate rules for projects that have those concerns.
Measurement Framework (Required Before Active Deployment)
The proposal currently has no way to measure whether rules are working. Required:
- Structured suppression logging: Each suppression event logged to
{temp}/review-pr/suppression_log_{pr}_{ts}.jsonwith rule_id, file, line, severity, dimension. - Post-activation monitoring: Track verdict distribution shifts (approved/changes_requested/needs_human) across the first 20 post-activation PRs. If any rule's false-negative rate exceeds 10%, tighten the rule text and file a sub-issue.
- Progressive rollout: Enable rules in phases starting with highest-confidence (deterministic pre-filters first, then prompt rules). Adjust based on monitoring data.
- Per-rule decay budget: Any rule that has not triggered in 10 consecutive PRs should be reviewed for removal.
- Rule-to-test linkage: Each rule should cite the enforcing test or contract. When the test changes, the rule is reviewed.
Implementation Priority
| Phase | Action | Rules | Impact |
|---|---|---|---|
| 1a | Deterministic pre-filters in diff_annotator.py |
G, J, K, L | 11% FP reduction, zero FN risk |
| 1b | Add suppression logging infrastructure | all | Measurement prerequisite |
| 2 | Activate prompt rules | A, B, D, F, I | ~72% FP reduction target |
| 3 | Self-critique pass (8th subagent) | C, E | 17% FP reduction, low FN risk |
| 4 | Feedback loop from reject_patterns | all | Long-term self-improving system |
Dependency: resolve-review thread lifecycle fixes
Prerequisite: #1037 must be completed before or alongside this issue. That issue fixes two resolve-review bugs:
- Info-severity silent drop — resolve-review filters out
[info]findings without posting any reply. This issue's suppression rules (Rule A) downgrade findings toinfoseverity; if resolve-review still silently dropsinfofindings, the downgrade path produces zero response. - REJECT thread resolution gap — REJECT-classified threads are never marked resolved, causing review-pr to re-raise identical findings as duplicates on re-review. This inflates the reject count that this issue aims to reduce.
Both bugs affect the accuracy of the reject mining data used in this issue and the effectiveness of the proposed suppression rules.
Data
- Full mining report:
.autoskillit/temp/resolve-review/fp_mining_report.md - Red-team validation report:
.autoskillit/temp/resolve-review/fp_validation_report.md - Reject patterns:
.autoskillit/temp/resolve-review/reject_patterns_925_*.json(4 files, 87 entries from PR Promote integration to main (198 PRs, 182 issues, 102 fixes, 107 features, 1 infra, 7 tests, 4 docs) #925) - Mining covered 68 PRs with 7 parallel sub-agent batches, 3 red-team validation agents, 4 gap analysis agents
Activity