Repository navigation
research: withdraw the "11 features match 5,511" claim after nested selection - #30
Merged
hallelx2 merged 1 commit intoOct 25, 2026
Conversation
…election PR #29 shipped the parsimony ladder with a caveat: the ranking came from SHAP computed across folds of the whole dataset, so it had seen every row it was later tested on. Discharging that caveat changes the headline. Re-run with selection confined to the training fold -- fit on train with all features, rank by gain, take the top k of that ranking, refit on train, score the held-out fold -- nested k=11 falls below the full matrix in every regime: 0.466 against 0.529 stratified, 0.363 against 0.454 scaffold-grouped, 0.234 against 0.249 receptor-grouped. The optimism is worth 0.03-0.07 macro-F1 and sits exactly where the claim was strongest, at small k, where a ranking derived from the whole dataset has the most room to flatter itself. So "11 features match 5,511" and "two beat it under receptor-grouped" are withdrawn. What stands is narrower and still worth having: 80-320 nested features match or beat all 5,511 in every regime -- 0.540 against 0.529, 0.463 against 0.454, 0.258 against 0.249 -- a 17-70x reduction at no cost. One new thing fell out. Under receptor-grouped at k=40, random features (0.266) edge out gain-ranked ones (0.258). Within noise either way, and that is the finding: once receptors are held out every feature set lands near the 0.179 floor, and which few hundred you choose stops mattering. It is the same message the graph network sends from a different representation -- the ceiling is in the data, not in the feature set. Section 2 is marked superseded for quotable claims rather than deleted, so the size of the optimism stays visible.
hallelx2
deleted the
halleluyaholudele/hal-4224-correct-the-parsimony-claim-nested-selection-kills-11
branch
October 25, 2026 17:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR #29 shipped the parsimony ladder with an explicit caveat: the ranking came from SHAP computed across folds of the whole dataset, so it had seen every row it was later tested on. Discharging that caveat changes the headline, so this corrects it.
The honest ladder
Selection confined to the training fold — fit on train with all features → rank by gain → take top-k of that ranking → refit on train → score the held-out fold:
Withdrawn
"11 features match 5,511" and "2 beat it under receptor-grouped". Nested k=11 is below the full matrix in every regime. The optimism is worth 0.03–0.07 macro-F1, concentrated at small k — exactly where a ranking derived from the whole dataset has the most room to flatter itself, and exactly where the claim was strongest.
Stands
80–320 nested features match or beat all 5,511 in every regime: 0.540 vs 0.529 stratified, 0.463 vs 0.454 scaffold, 0.258 vs 0.249 receptor. A 17–70× reduction at no cost — real parsimony, an order of magnitude less dramatic than first reported.
New, and arguably the more interesting half
Under receptor-grouped at k=40, random features (0.266) edge out gain-ranked ones (0.258). Within noise either way — and that is the finding. Once receptors are held out, every feature set lands near the 0.179 floor and which few hundred you choose stops mattering. Same message the GIN sends from a different representation: the ceiling is in the data, not the feature set.
On presentation
§2 is marked superseded, not deleted. The size of the optimism is itself informative, and removing the original table would hide how the correction was arrived at.
Verification
ruff check scripts/experiments/clean. Artefact:nested_parsimony.json. 3 regimes × 3 seeds × 8 values of k, with matched random controls drawn inside the same fold.Still outstanding: the ERK 1,000-permutation null is running — the 100-draw p = 0.0495 is too fragile to quote, since one further extreme draw would make it 0.059.
Closes HAL-4224