Repository navigation
research: ERK null at 1,000 draws — no usable signal - #31
Merged
Merged
Conversation
…im it The 100-draw null gave p = 0.0495, close enough to the threshold that one further extreme permutation would have moved it to 0.059. Re-run at 1,000 draws the observed 0.5219 sits 0.0005 above the null's own 95th percentile (0.5214), for a final p = 0.0500 and an effect of +0.036 macro-F1 over always-predict-not-ERK, on 34 positives across 18 receptors. The trajectory is the real argument against claiming it. The running p went 0.0495, 0.0348, 0.0365, 0.0474, 0.0559, 0.0566, 0.0574, 0.0500 -- crossing the threshold in both directions. A conclusion drawn at 100 draws, or at 500, would have been an artefact of where the sampling stopped. More permutations will not settle this one; more data would. G-protein selectivity is left at 100 draws deliberately: at p = 0.42 with the observed value below the null mean, additional draws cannot change it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the last open checkbox on HAL-4224.
The 100-draw null gave p = 0.0495 — close enough to the threshold that one further extreme permutation would have put it at 0.059. Re-run at 1,000 draws:
The observed value sits 0.0005 above the null's own 95th percentile. That is as borderline as a result can be, on 34 positives spread across 18 receptors.
The trajectory is the argument
It crossed 0.05 in both directions. Any conclusion drawn at 100 draws, or at 500, would have been an artefact of where the sampling happened to stop — which is exactly the failure the re-run was meant to catch, and it caught it.
Read as: no usable ERK signal at this sample size. More permutations will not settle it; more data would.
G-protein selectivity is deliberately left at 100 draws — at p = 0.42 with the observed value below the null mean, additional draws cannot change the conclusion.
Scope
Documentation only;
scripts/experiments/erk_null_1000.pylanded in PR #30.Closes HAL-4224