Skip to content

fix: require evidence before rejecting a chosen approach - #266

Merged
lukemaj merged 1 commit into
mainfrom
agentsmd-265-evidence
Oct 26, 2026
Merged

lukemaj merged 1 commit into
mainfrom
agentsmd-265-evidence

Conversation

@lukemaj

@lukemaj lukemaj commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What: Require evidence before any agent rejects a user-selected approach, and retain that approach until evidence rules it out.
Why: The existing negative-result gate covered relaying delegated work but missed a planner's own conversational claim.
So what: All required checks and independent exact-head review pass; Lukasz must approve the final merge.

Closes #265.

Edit the two existing paragraphs only: Judgment requires the smallest authorized experiment or a primary source, reported in the same message, and preserves the chosen route until that evidence exists. Bounded delegation covers the agent's own conversational claims as well as delegated results. Existing expected-text assertions follow this intentional contract change; all prior source/attempt/challenger requirements remain.

evals/chosen-route-evidence.json freezes a stdlib re task where chained fixed-width negative lookbehinds preserve the user's route even though the original alternation fails. Before/after evidence and limitations are recorded on #265. No new procedure, hook, live-mail operation, or benchmark framework.

Requirements and who asked: Lukasz requested #265's minimal wording, versionctl transition, before/after evaluation and independent exact-head review.
Deleted: No new procedure, hook, live Gmail test, or benchmark framework; retain one reusable synthetic fixture.
Bottleneck: The negative-result gate's delegated-only trigger misses the planner's own unsupported impossibility claims.
Checked myself: Read the Issue and named files, root instructions/direction, existing negative-result tests and version policy; ran the fixture's oracle in Python and inspected the full diff. The change advances the Objective's evidence-first reviewed delivery contract.

Version: 17.4.6 -> 17.4.7, patch correction to agent wording, prepared by bin/versionctl with all five mirrors and changelog in the same commit.

Proof at 95e2123d24b917134db08ebc9763f02e99160783: full repository suite 512 pass; versionctl suite 7 pass; clean-commit release-check, exact-base CI doctor, fixture JSON/oracle check, diff check and GitHub validate pass. Independent exact-head review: PASS, no findings; review/independent success by lukemaj. Grok eval rubric scores: 2/3 before, 3/3 after, with route retention passing both. Both Codex evals were interrupted at the fixed five-minute bound and remain ungraded; one completed synthetic task establishes no general behavioral improvement.

Website/SEO impact: none, internal contract correction without URL, setup or capability changes. Behavioral Live Verification remains pending user evidence; this synthetic eval makes no live-work claim.

Workspace retained at /home/luke/.t3/worktrees/agentsmd/agentsmd-265-evidence for review and the merge decision. The canonical live install is unchanged. No merge, release, distribution, installation or deployment is authorized or performed.

@lukemaj

lukemaj commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor Author

What: Independent exact-head review passed with no actionable findings.
Why: The reviewer checked the candidate against Issue #265 and the repository contract.
So what: Final proof also passes; Lukasz retains the merge decision.

Reviewer verdict relayed unchanged:

PASS: no actionable findings on exact head 95e2123d24b917134db08ebc9763f02e99160783, reviewed by gpt-6.1-sol at high effort.

The two paragraph edits satisfy #265 without changing authority or research scope; the patch transition 17.4.6 → 17.4.7, five mirrors, existing test changes, fixture prompt/rubric/oracle, and inspected CI logs are consistent.

Proof limits: the full suite remains pending, cancelled Codex runs are ungraded, and Grok supports only the narrow evidence-timing improvement while route retention passed both arms; none establishes Live Verification.

Subsequent proof update from the implementation owner: the full clean-head suite completed successfully, 512 tests in 282.103s; versionctl tests passed, 7 tests. No code or version changed after the reviewed SHA.

Supplemental alternative review also completed; verdict relayed unchanged:

PASS on the text and diff at 95e2123d24b917134db08ebc9763f02e99160783 (base 49329745df32a430e1911233da7aa5d7ef2a41bb). No findings.

Reviewed as grok-4.7-build-fast (Grok harness). Effort is not exposed separately from that model id. The earlier review is still pending, with no findings and no verdict. That silence is not proof and is not a technical impossibility. This is the one completed exact-head verdict.

The Judgment paragraph now requires a same-message experiment or primary source before an impossibility claim or a replacement of a user-selected approach, says "unverified" and proposes the experiment when neither is in reach, and keeps that approach until the evidence exists. The negative-result paragraph now covers the agent's own conversational claim as well as a relayed result. Authority and the existing research rule are unchanged. The patch is 17.4.6 to 17.4.7 on VERSION, the changelog, and all five mirrors in .version-policy.json. The fixture oracle matches a local re run: the original pattern raises PatternError, and (?<!Mr)(?<!Mrs)\. returns (11, 12) and (17, 18). The negative-result test updates the two clauses the contract changed and still requires the attempt list, primary sources, and challenger. Snapshot SHA-256 values in the handoff match both files at base and head. The worktree is clean at that head.

Execution proof stays separate and is not a pass. The completed Grok pair is one synthetic case: the before arm fails rubric 3 because a progress message rejects the original pattern before any experiment in that message, and the after arm passes all three rubrics with the command and output in the final answer. Both arms keep standard-library re, so there is no observed route-retention gain. Both Codex arms were interrupted at the five-minute bound and are ungraded. The first full suite was invalidated by a mid-run version bump. The fresh full suite was still running. This review did not re-run the author's focused tests, versionctl tests, release-check, or CI doctor. Tests and this fixture are not Live Verification.

Current owner update: the first reviewer also completed PASS before this supplemental result arrived; both verdicts cover the same head. The final clean-head full suite subsequently passed all 512 tests; versionctl suite passed all 7. The optional second-round Codex candidate was interrupted once the required first verdict arrived. No candidate changed after review.

@lukemaj
lukemaj merged commit bdc91d1 into main Oct 26, 2026
2 checks passed
@lukemaj
lukemaj deleted the agentsmd-265-evidence branch October 26, 2026 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Require evidence before any agent says a chosen approach can't work

1 participant