docs(claude): scoring hygiene and the probe/ALFA split - #69
Open
opariffazman wants to merge 1 commit into
Open
Conversation
The issue #57 noise floor is the ALFA scorer, not the training: ~1% of tasks return a different verdict for a byte-identical command because the task is not deterministic (find -mtime against build-time stamps, unsorted find, ps aux, dig, gzip/tar header mtimes). Records what to do about it without touching the scorer or the container: verdict cache on (index, command), one image build per comparison, majority-of-3 on the known-flaky set, and the embed model digest alongside the llama build. Also records that the probe and ALFA measure different tools, which is the trade runs 10-13 made without naming it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Writes the PR #68 findings into CLAUDE.md as rules, with the references behind them.
Corrects the noise-floor bullet. Issue #57's ≤0.013 floor is the scorer, not the training — ~1% of tasks give a different verdict for a byte-identical command. The causes are all task-side:
find -mtime/-mmin/-atimeagainst build-time file stamps, unsortedfind -print,ps aux,dig, and the mtime gzip and tar write into their own headers. InterCode-ALFA's README documents none of it.Adds a "Scoring hygiene" section. Scorer and container stay unmodified; everything listed controls the measurement around it — the same split
lm-evaluation-harnessdraws with--use_cache:(index, command)— 61% of our scoring calls repeat a command already scored for that taskmxbai-embed-largedigest next to the llama buildlibfaketime— changes the environment, ~10× slower, buys nothing one session doesn'tAdds "The probe and ALFA do not measure the same thing." Only 17% of ALFA-BM tasks the shipped model passes use a beginner tool;
findalone is a third. Runs 10–13 traded ALFA for probe one inside-CI step at a time. Names the trade so the next run makes it deliberately.References: lm-eval interface, arXiv 2405.14782, arXiv 2505.23419, arXiv 2310.15642, reproducible-builds.org, libfaketime.
Docs only — no code, no dataset, no model. The cache itself isn't implemented yet.
🤖 Generated with Claude Code