Skip to content

ruff_python_spo: callee-name residual detail + the drill's proposer stage - #97

Merged
AdaWorldAPI merged 1 commit into
mainfrom
claude/plain-spo-config-drill
Aug 18, 2026
Merged

AdaWorldAPI merged 1 commit into
mainfrom
claude/plain-spo-config-drill

Conversation

@AdaWorldAPI

Copy link
Copy Markdown
Owner

Summary

Increment 2 of the self-adaptive drill loop (#96), scoped to what could actually be measured this round. Two pieces.

1. Callee-name detail on non_literal_assign

x = <call>() residuals used to carry the generic "call" tag. Now, when the callee resolves to a terminal name, detail = "call:<name>" (field(...) → "call:field"); an unresolvable callee (a call whose own callee is a call, a lambda, a subscript — never guessed) keeps the generic "call" tag.

Measured payoff on dismech: the 451-row call bucket from #96's histogram decomposes losslessly into three named LinkML schema factories:

355  non_literal_assign/call:PermissibleValue
 96  unresolved_annotation/binop
 91  non_literal_assign/call:EnumDefinition
  5  non_literal_assign/call:ConfigDict

355 + 91 + 5 = 451, exact — this is precisely the drill target #96's own histogram predicted without being able to name.

2. drill.rs — the proposer stage

propose(residuals, min_support) groups a ledger by (reason, detail) for the four reasons dense enough to carry one (curie_constant, non_literal_assign, unresolved_annotation, unresolved_base), thresholded on support. classify_across_corpora(labeled, min_support) runs propose over 2+ labeled corpora and splits each candidate into Generic (cleared the bar in every supplied corpus) vs CorpusScoped (some but not all) — an exact rule, not a similarity heuristic.

Scope boundary — stated in the module doc, and stated again here so it isn't overclaimed: this builds grouping + classification only. It does not build a promotion gate that re-runs extraction with a candidate row "active" and checks the coverage delta — that needs a config-consuming extractor (a trie the plain arm reads before deciding a site's classification) that doesn't exist yet. That's the next increment, not this one.

Measured — and it falsifies my own prediction

Ran plain_propose at min_support=2 over three real corpora (dismech / A2UI SDK / ruff/scripts):

18 candidate rows, 0 generic, 18 corpus_scoped

I'd predicted in the #96 handoff that some row would prove cross-corpus generic. It didn't — worth recording plainly rather than dropping quietly. The closest candidate, unresolved_annotation/binop, fires 96× in dismech and 12× in ruff/scripts but is silent on the A2UI corpus — so under the exact "fires in ALL supplied corpora" rule it stays CorpusScoped. That's the strict rule working as designed: three modest corpora is a small n for "generic," and a fuzzy magnitude-similarity threshold would have manufactured a Generic verdict this data doesn't support. curie_constant rows are, as expected, uniformly CorpusScoped to dismech.

Falsifier discipline

Every new grouping/threshold/classification site verified by a disable run — mutation asserted present, test observed red, restored, full suite green: the min_support filter, the Generic/CorpusScoped split (forced everything to Generic — both cross-corpus tests failed), and the callee-name fallback (both resolved-and-unresolved cases exercised in one fixture).

Test plan


Generated by Claude Code

Two pieces of increment 2, both scoped to what could actually be measured
this round.

1. non_literal_assign's detail now carries the callee name for a resolvable
   call (`field(...)` -> "call:field"), falling back to the generic "call"
   tag when the callee is itself unresolvable (a call, lambda, subscript -
   never guessed). Measured payoff on dismech: the 451-row "call" bucket
   from increment 1 decomposes losslessly into three named LinkML schema
   factories - call:PermissibleValue 355, call:EnumDefinition 91,
   call:ConfigDict 5 (355+91+5=451, exact) - which is exactly the kind of
   drill target increment 1's own histogram predicted without being able
   to name.

2. New drill.rs: the proposer stage. propose() groups a residual ledger by
   (reason, detail) for the four reasons dense enough to carry one, and
   thresholds on support. classify_across_corpora() runs propose() over
   2+ labeled corpora and splits candidates into Generic (cleared the bar
   in EVERY supplied corpus) vs CorpusScoped (some but not all) - an
   exact, falsifiable rule, not a similarity heuristic.

   Scope boundary, stated in the module doc and here: this builds
   grouping + classification ONLY. It does not build a promotion gate
   that re-runs extraction with a candidate row "active" and checks the
   coverage delta - that needs a config-consuming extractor (a trie the
   plain arm reads before deciding a site's classification) that does not
   exist yet. Claiming this stage ratifies rows would overstate what it
   measures.

Measured, min_support=2, three real corpora (dismech / A2UI SDK /
ruff/scripts): 18 candidate rows, ZERO generic. This falsifies my own
prediction from the increment-1 handoff that some row would prove
cross-corpus generic - worth recording plainly rather than quietly
dropping. unresolved_annotation/binop comes closest (96 in dismech, 12 in
ruff/scripts) but is silent on the A2UI corpus, so by the exact "fires in
ALL supplied corpora" rule it stays CorpusScoped. That is the strict rule
working as designed, not a bug: three modest corpora is a small n for
"generic," and the alternative (a fuzzy magnitude-similarity threshold)
would have manufactured a Generic verdict this data does not support.

plain_propose new example: runs classify_across_corpora over N labeled
corpora and prints candidates as TOML - data, not Rust, per the earlier
design note that the proposer must never write code.

Falsifier discipline: every new grouping/threshold/classification site
verified by a disable run (mutation asserted present, test observed red,
restored, full suite green) - the min_support filter, the Generic/
CorpusScoped split, and the callee-name fallback.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AGVLyRZNEKKBSfBDJfbY3V
@cursor

cursor Bot commented Aug 18, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_aba29efd-6479-4211-bcf0-90e524105a81)

@AdaWorldAPI
AdaWorldAPI marked this pull request as ready for review August 18, 2026 05:54
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@AdaWorldAPI
AdaWorldAPI merged commit a91c239 into main Aug 18, 2026
28 of 31 checks passed
AdaWorldAPI pushed a commit that referenced this pull request Oct 14, 2026
…nables

Increment 3 of the self-adaptive drill loop, closing the boundary drill.rs
named twice now ("this does NOT build a promotion gate"). One rule wired
end-to-end, not all four residual reasons at once - a real config surface
plus a real gate is worth more than a wide, partially-wired one.

PlainDrillConfig::unwrap_optional_annotation resolves `T | None` / `None
| T` to T's own field_type reading. Chosen because it was the only
cross-corpus candidate from #97's measurement (dismech 96, ruff/scripts
12, genuinely absent - not merely below threshold - on the A2UI SDK),
unlike the LinkML call:PermissibleValue/call:EnumDefinition factories,
which are corpus-scoped and would need a per-corpus row, not a generic
one. field_type_from_annotation recurses into the non-None operand;
a chained union (`str | int | None`) has a BinOp as its left operand, so
the shape check fails and it stays unresolved rather than guessing which
arm to keep. Off by default - PlainDrillConfig::default() is pinned
byte-identical to the config-free path.

drill::ratify_optional_unwrap is the gate: runs extraction with the rule
OFF and ON over one source, then independently verifies the claim two
ways that share no code with the resolver -
  1. a SEPARATELY-WRITTEN shape test (is_shaped_t_or_none), applied by
     re-parsing the source and re-walking the raw AST directly - not by
     trusting field_type_from_annotation's own success/failure - counting
     how many of the newly-resolved sites are genuinely T|None shaped.
     Must equal the resolver's own count exactly, in EITHER direction:
     the resolver claiming to fix something the independent check
     disputes is exactly as much a defect as missing one it should have
     caught.
  2. every resolved value is checked against a real-type-name shape
     (non-empty, alphanumeric) rather than trusted as "present, so fine".

Measured, real corpora, per-file (`plain_ratify` example):
  dismech (84 files):      baseline=96 resolved=91 verified=91 ratified=YES
  ruff/scripts (23 files): baseline=12 resolved=12 verified=12 ratified=YES
  a2ui sdk (127 files):    baseline=0  resolved=0  verified=0  ratified=YES

dismech's 96-91=5 gap is the honest remainder: non-optional binop shapes
(chained unions, `int | str`) the rule correctly declines rather than
guesses on - the ratification proves the claim is EXACT, not that the
rule resolves everything.

Falsifier discipline, including a finding worth recording rather than
discarding: the first disable run on the independent verifier
(is_shaped_t_or_none forced to always return true) reported the target
test green under mutation - genuinely uninformative, not a false pass,
because the independent-verification loop only visits sites the resolver
itself flipped, and a correct resolver structurally can never disagree
with a correct independent check without a SECOND bug existing first.
Traced to the right level: RatificationReport::ratified()'s own
disagreement-detection logic now has direct unit coverage
(ratified_is_false_when_the_counts_disagree_in_either_direction, four
hand-built cases), disable-run verified on THAT function directly
(forced to `true`, the new test failed; restored, green). Two more
disable runs on the resolver itself (optional_operand neutered; the
config gate bypassed) both verified red-then-green normally.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AGVLyRZNEKKBSfBDJfbY3V
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants