Repository navigation
Add ekdist.mixtures: two populations in one set of estimates - #9
Merged
Merged
Conversation
Fit a mechanism to many simulated records and the estimates do not always form one cloud. Some records take the optimiser to a second maximum, typically one where most events are missed, and the results come back as two groups an order of magnitude apart. The mean of a set like that describes neither group. mixture2 fits two Gaussian components in log space, for groups that overlap or nearly do, and reports the proportion in each and their geometric means. gap_split finds the largest multiplicative step between neighbouring sorted values, for a few runaways cleanly separated from the body. The point of both is that the split comes from the data. Choosing a threshold by eye and counting either side makes the answer a property of the threshold; these let an exclusion be stated -- "16 values above a 55-fold gap" -- and checked by someone else. Log space because rate constants are positive with multiplicative errors, so a group of them is log-normal rather than normal; that also makes Mixture.means geometric means. Written for the reproduction of Colquhoun, Hatton & Hawkes (2003), whose Fig. 2 is this situation exactly, but nothing in the module is specific to that paper or to ion channels. numpy only. 23 tests, against data generated from a known mixture rather than against values the function produced itself, because a mixture fit cannot look wrong: it returns two plausible components whatever it is handed, and raises nothing. Proportions are recovered at 10, 27 and 50 per cent so the tolerances are not tuned to one case. Two tests keep the suite honest rather than checking arithmetic: one records that a unimodal set is split without complaint, since that is the mistake a user will make and the separation of the means is the only thing that tells them; the other shows the recovery tolerances would reject the most likely way for the fit to be broken while still looking reasonable. Suite goes from 75 to 98. The clean-install job needs no change -- it enumerates modules since #8, and picks this one up as the eighth. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
ekdist.mixtures— two functions for the case where a set of fitted estimates is not one population.Fit a mechanism to many simulated records and the results do not always form one cloud. Some records take the optimiser to a second maximum — typically one where most events are missed — and the estimates come back as two groups an order of magnitude apart. Any summary that ignores this is meaningless: the mean of such a set describes neither group.
mixture2gap_splitproportionWhy not just pick a threshold
Choosing a cut by eye and counting either side makes the answer a property of the cut. Both functions determine the split from the data, so an exclusion can be stated — "16 values above a 55-fold gap" — rather than assumed, and someone else can check it.
This is not hypothetical. In the work this came from I first reported a cluster mean from an ad-hoc
alpha2 < 2500threshold and got 2025; the mixture fit gave 2038.8, because the threshold had stranded five points on the wrong side.Why log space
Rate constants are positive and their errors are multiplicative, so a group of them is roughly log-normal, not normal. Fitting in logs keeps every fitted mean positive and makes each group closer to the Gaussian a mixture assumes. It also means
Mixture.meansare geometric means, which is said explicitly rather than left to be discovered.Tests
23, and the interesting thing about them is what they defend against.
A mixture fit cannot look wrong. It returns two components whatever it is handed, with plausible means and a plausible proportion, and raises nothing. So every recovery test runs on data generated from a known mixture, where the proportion and both means are known before the fit runs — never against values the function produced itself. Proportions are recovered at 10 %, 27 % and 50 % so the tolerances are not tuned to one case.
Two tests check nothing about the arithmetic and exist to keep the suite honest:
test_a_unimodal_set_is_split_without_a_gap— records that one population comes back as two, without complaint, because that is the mistake a user will make. Nothing here tests whether two components is the right number; the separation of the means is the only thing that tells the caller, and the test pins what "no real split" looks like (within a factor of two, against the eightfold separation of a genuine one).test_the_recovery_tests_could_fail— shows the tolerances would reject the most likely way for the fit to be broken while still looking reasonable (returning the overall mean for both components).Plus determinism, ordering (
means[0]is always the lower group, so it can be indexed blind), responsibilities summing to one, and thatgap_split's mask follows input order rather than sorted order.Suite goes from 75 to 98.
Notes
clean-installjob needs no change — it enumerates modules since Measure the correlation between an opening and its neighbouring shut times #8 and picks this up as the eighth. That fix paid for itself within a day.mixtures, deliberately notclusters: "cluster" already means a run of openings between long shuttings, and reusing it here would be actively confusing.🤖 Generated with Claude Code