Repository navigation
ruff_r2il: slag_partition probe — two negatives, one against my own claim - #107
Conversation
…my own claim Built to test whether harvest_6502's 3742 residual rows split into 'opcode not yet classified' vs 'data lifted as code'. Uses a CONTROL GROUP rather than a heuristic: Elite ships known-pure-data binaries (SHIPS/WORDS/LODATA/IANTOK/SPRITE) beside known-pure-code (LOCODE/ HICODE), both from one build, both given identical treatment. Negative 1 — the premise was false, and the probe is what showed it. Elite keeps data in a SEPARATE source file (elite-data.asm -> SHIPS.bin et al); the harvest corpus gma4/5/6 = COMLOD/LOCODE/HICODE is code. The blueprints were never lifted as opcodes. Corroborated independently by the probe's own per-file numbers: both code files sit inside the CODE range on both metrics. Some tables ARE embedded (QQ23 market prices lives in elite-source.asm), so contamination is non-zero, but it is not the bulk of the slag. The reframing that claimed otherwise was wrong. Negative 2 — content statistics do not partition. Over 208 windows of 256 bytes (>=16 instructions each): decode-error rate reaches 72.1% balanced accuracy, control-flow density 77.0%. Neither is usable alone. Direction is counterintuitive and worth keeping: DATA shows HIGHER control-flow density (p50 26.6 vs 10.6) because random bytes decode into branch opcodes. Measured at WINDOW granularity after a correction. The first version reported whole-file furnace stats and produced two exact duplicate pairs across unrelated files of different sizes (SPRITE==gma5, SHIPS==gma6): it handed every sweep block to ONE FunctionBehavior, and CFG::from_blocks keeps only the component reachable from blocks[0], so those columns measured a tiny entry-reachable fragment rather than the file. Removed rather than reported. Remaining metrics come straight off the sweep and need no SSA. The untested hypothesis, named rather than implied: reachability. Code is what a call graph reaches; data is what is only ever loaded FROM. That cannot be measured on standalone control files (no callers) and needs the mixed-image setting.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_fa9ee2bc-85e0-4781-8849-74b438be9114) |
|
What failed: the runner could not download the That is a GitHub Actions DNS/infrastructure failure during action setup — it died before checkout finished and before any test body ran. No Rust code was compiled, so nothing in this diff could have influenced it. Why it cannot be this PR's, independently of the log: the only code here is a new example in Isolated: every other completed check on this head is green — 6 success, 12 still running, this one failure. No fix ported, and none exists to port. The failure is in a workflow file this PR does not touch, and the only "fix" would be re-pinning or vendoring a third-party action in ruff's CI — out of scope, and it would widen a probe-only PR into a CI change. I lack the means to re-run it: Keeping this PR watched until it is green, merged, or closed. Generated by Claude Code |
The prek check failed because a hook MODIFIED a file: the typos spell-checker read the loop variable `thr` as a misspelling of `the` and rewrote all four uses, which fails prek by definition (hooks that change files fail the run). `threshold` is the clearer name regardless, so this takes the spell-checker's side rather than adding an ignore entry. Mine, and avoidable: AGENTS.md says to run `uv run --only-group dev --locked prek` after changing files in this repo, and I did not. Verified now — typos and rustfmt both Passed on this file; the harvest artifacts under .claude/ are excluded from every hook. `--locked` cannot run in this container (pyproject.toml's `exclude-newer = "P7D"` fails to parse, so uv wants to re-lock), so the run was unlocked and the uv.lock it touched was restored. Probe output unchanged: 72.1% and 77.0% balanced accuracy.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_698bf65a-547d-4130-b221-a2e5fe176e20) |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e7bee5ac8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| //! `IANTOK.bin`), and the harvest corpus was `gma4/5/6` = COMLOD / LOCODE / HICODE, all code. | ||
| //! The per-file numbers below corroborate it independently: both code files sit inside the CODE | ||
| //! range on both metrics. Some tables ARE embedded in the code image (`QQ23`, the market prices, | ||
| //! is in `elite-source.asm`), so contamination is non-zero — but it is not the bulk of the slag, |
There was a problem hiding this comment.
The probe measures only gma5 and gma6, but this conclusion covers the full gma4/5/6 harvest and even acknowledges immediately afterward that an embedded market table was harvested as code. Without measuring or region-labeling gma4, the results cannot establish either that all three images are code or that embedded data is not a substantial source of the 3,742 residual rows; narrow the conclusion to the measured files or include the omitted image.
Useful? React with 👍 / 👎.
| let c = pool(Truth::Code, f); | ||
| let d = pool(Truth::Data, f); |
There was a problem hiding this comment.
When C64_CORPUS contains readable files from only one truth class—a supported scenario because every missing subject is merely skipped—one of these pools is empty. The percentile output then contains NaN, while the .max(1) denominators below still produce and label a seemingly meaningful balanced accuracy. Check that both pools contain scorable windows and skip the comparison otherwise.
Useful? React with 👍 / 👎.
| } else if best.1 >= 0.75 { | ||
| "weak — not alone" | ||
| } else { | ||
| "no better than guessing" |
There was a problem hiding this comment.
Every balanced accuracy below 75% is labeled "no better than guessing", even though chance is 50%; this mischaracterizes the probe's reported 72.1% result and any similar run. If 75% is intended as a usefulness gate, describe the result as weak or below that gate rather than asserting statistical equivalence to guessing.
Useful? React with 👍 / 👎.
| fn pad_window(bytes: &[u8], want: usize) -> Vec<u8> { | ||
| let mut w = bytes.to_vec(); | ||
| if w.len() < want { | ||
| w.resize(want, 0); |
There was a problem hiding this comment.
pad_window clones the entire supplied slice, and sweep supplies &code[offset..] on every decode attempt, so processing an image copies a quadratic number of bytes even though one lift needs only the minimum instruction window. This can make larger custom corpora disproportionately slow and memory-bandwidth intensive; borrow the remaining slice when it is long enough and allocate only the short padded tail.
Useful? React with 👍 / 👎.
| let mut best = (0.0f64, 0.0f64, false); | ||
| let mut cands: Vec<f64> = c.iter().chain(d.iter()).cloned().collect(); | ||
| cands.sort_by(|a, b| a.partial_cmp(b).unwrap()); |
There was a problem hiding this comment.
The percentile and threshold-selection code has no automated coverage, while the example exits successfully without exercising it whenever the external sibling spec or corpus is absent. Extracting the scorer and testing both threshold directions, ties, boundary values, and empty/one-class inputs would prevent incorrect experimental conclusions from silently passing normal checks.
AGENTS.md reference: AGENTS.md:L169-L171
Useful? React with 👍 / 👎.
…laim (#107) * ruff_r2il: slag_partition probe — two negatives, one of them against my own claim Built to test whether harvest_6502's 3742 residual rows split into 'opcode not yet classified' vs 'data lifted as code'. Uses a CONTROL GROUP rather than a heuristic: Elite ships known-pure-data binaries (SHIPS/WORDS/LODATA/IANTOK/SPRITE) beside known-pure-code (LOCODE/ HICODE), both from one build, both given identical treatment. Negative 1 — the premise was false, and the probe is what showed it. Elite keeps data in a SEPARATE source file (elite-data.asm -> SHIPS.bin et al); the harvest corpus gma4/5/6 = COMLOD/LOCODE/HICODE is code. The blueprints were never lifted as opcodes. Corroborated independently by the probe's own per-file numbers: both code files sit inside the CODE range on both metrics. Some tables ARE embedded (QQ23 market prices lives in elite-source.asm), so contamination is non-zero, but it is not the bulk of the slag. The reframing that claimed otherwise was wrong. Negative 2 — content statistics do not partition. Over 208 windows of 256 bytes (>=16 instructions each): decode-error rate reaches 72.1% balanced accuracy, control-flow density 77.0%. Neither is usable alone. Direction is counterintuitive and worth keeping: DATA shows HIGHER control-flow density (p50 26.6 vs 10.6) because random bytes decode into branch opcodes. Measured at WINDOW granularity after a correction. The first version reported whole-file furnace stats and produced two exact duplicate pairs across unrelated files of different sizes (SPRITE==gma5, SHIPS==gma6): it handed every sweep block to ONE FunctionBehavior, and CFG::from_blocks keeps only the component reachable from blocks[0], so those columns measured a tiny entry-reachable fragment rather than the file. Removed rather than reported. Remaining metrics come straight off the sweep and need no SSA. The untested hypothesis, named rather than implied: reachability. Code is what a call graph reaches; data is what is only ever loaded FROM. That cannot be measured on standalone control files (no callers) and needs the mixed-image setting. * fix(prek): rename thr -> threshold; the typos hook rewrote it to 'the' The prek check failed because a hook MODIFIED a file: the typos spell-checker read the loop variable `thr` as a misspelling of `the` and rewrote all four uses, which fails prek by definition (hooks that change files fail the run). `threshold` is the clearer name regardless, so this takes the spell-checker's side rather than adding an ignore entry. Mine, and avoidable: AGENTS.md says to run `uv run --only-group dev --locked prek` after changing files in this repo, and I did not. Verified now — typos and rustfmt both Passed on this file; the harvest artifacts under .claude/ are excluded from every hook. `--locked` cannot run in this container (pyproject.toml's `exclude-newer = "P7D"` fails to parse, so uv wants to re-lock), so the run was unlocked and the uv.lock it touched was restored. Probe output unchanged: 72.1% and 77.0% balanced accuracy. --------- Co-authored-by: Claude <noreply@anthropic.com>
A probe, not a feature. It asks whether
harvest_6502's 3,742 residual rows split into "opcode not yet classified" vs "data lifted as code" — and answers with a control group instead of a heuristic: Elite ships known-pure-data binaries (SHIPS/WORDS/LODATA/IANTOK/SPRITE) beside known-pure-code (LOCODE/HICODE), same build, identical treatment.Negative 1 — the premise was false, and the probe is what showed it
I claimed the harvest had lifted Elite's ship blueprints and market tables as opcodes. It had not. Elite keeps data in a separate source file (
elite-data.asm→SHIPS.binet al); the harvest corpusgma4/5/6= COMLOD/LOCODE/HICODE is code. Corroborated independently by the probe's own per-file numbers — both code files sit inside the CODE range on both metrics.Some tables are embedded (
QQ23, the market prices, lives inelite-source.asm), so contamination is non-zero. But it is not the bulk of the slag, and the reframing that said otherwise was wrong.Negative 2 — content statistics do not partition
208 windows of 256 bytes (≥16 instructions each):
Direction is counterintuitive and worth keeping: DATA shows higher control-flow density (p50 26.6 vs 10.6), because random bytes decode into branch opcodes.
A measurement correction inside the probe itself
The first version reported whole-file furnace stats and produced two exact duplicate pairs across unrelated files of different sizes (
SPRITE==gma5,SHIPS==gma6). Cause: it handed every sweep block to oneFunctionBehavior, andCFG::from_blockskeeps only the component reachable fromblocks[0]— so those columns measured a tiny entry-reachable fragment, not the file. Removed rather than reported; the remaining metrics come straight off the sweep and need no SSA. Measurement moved to window granularity, which is also what the real use needs (mark regions within one image).The untested hypothesis, named rather than implied
Reachability. Code is what a call graph reaches; data is what is only ever loaded from. That cannot be measured on standalone control files — they have no callers — and needs the mixed-image setting. It is the one hypothesis these results leave standing.
Gates:
clippy --example slag_partition -- -D warningsclean,cargo fmtclean, probe runs end-to-end. Corpus read from a local sibling checkout, never redistributed; a missing file is a printed skip.Generated by Claude Code