Skip to content

Add ekdist.mixtures: two populations in one set of estimates - #9

Merged
remislp merged 1 commit into
masterfrom
ekdist-mixtures
Sep 8, 2026
Merged

remislp merged 1 commit into
masterfrom
ekdist-mixtures

Conversation

@remislp

@remislp remislp commented Sep 8, 2026

Copy link
Copy Markdown
Member

What this adds

ekdist.mixtures — two functions for the case where a set of fitted estimates is not one population.

Fit a mechanism to many simulated records and the results do not always form one cloud. Some records take the optimiser to a second maximum — typically one where most events are missed — and the estimates come back as two groups an order of magnitude apart. Any summary that ignores this is meaningless: the mean of such a set describes neither group.

for
mixture2 two groups that overlap or nearly do. Fits two Gaussian components in log space; reports the proportion in each and their geometric means
gap_split a handful of runaways cleanly separated from the body. Finds the largest multiplicative step between neighbouring sorted values
proportion a proportion and its binomial SE, both per cent

Why not just pick a threshold

Choosing a cut by eye and counting either side makes the answer a property of the cut. Both functions determine the split from the data, so an exclusion can be stated — "16 values above a 55-fold gap" — rather than assumed, and someone else can check it.

This is not hypothetical. In the work this came from I first reported a cluster mean from an ad-hoc alpha2 < 2500 threshold and got 2025; the mixture fit gave 2038.8, because the threshold had stranded five points on the wrong side.

Why log space

Rate constants are positive and their errors are multiplicative, so a group of them is roughly log-normal, not normal. Fitting in logs keeps every fitted mean positive and makes each group closer to the Gaussian a mixture assumes. It also means Mixture.means are geometric means, which is said explicitly rather than left to be discovered.

Tests

23, and the interesting thing about them is what they defend against.

A mixture fit cannot look wrong. It returns two components whatever it is handed, with plausible means and a plausible proportion, and raises nothing. So every recovery test runs on data generated from a known mixture, where the proportion and both means are known before the fit runs — never against values the function produced itself. Proportions are recovered at 10 %, 27 % and 50 % so the tolerances are not tuned to one case.

Two tests check nothing about the arithmetic and exist to keep the suite honest:

  • test_a_unimodal_set_is_split_without_a_gap — records that one population comes back as two, without complaint, because that is the mistake a user will make. Nothing here tests whether two components is the right number; the separation of the means is the only thing that tells the caller, and the test pins what "no real split" looks like (within a factor of two, against the eightfold separation of a genuine one).
  • test_the_recovery_tests_could_fail — shows the tolerances would reject the most likely way for the fit to be broken while still looking reasonable (returning the overall mean for both components).

Plus determinism, ordering (means[0] is always the lower group, so it can be indexed blind), responsibilities summing to one, and that gap_split's mask follows input order rather than sorted order.

Suite goes from 75 to 98.

Notes

  • numpy only. No new dependency, and the clean-install job needs no change — it enumerates modules since Measure the correlation between an opening and its neighbouring shut times #8 and picks this up as the eighth. That fix paid for itself within a day.
  • Named mixtures, deliberately not clusters: "cluster" already means a run of openings between long shuttings, and reusing it here would be actively confusing.
  • Written for the reproduction of Colquhoun, Hatton & Hawkes (2003), whose Fig. 2 is this situation exactly. Their examples appear in the docstrings because a concrete case reads better than an abstract one, but nothing in the module is specific to that paper or to ion channels.

🤖 Generated with Claude Code

Fit a mechanism to many simulated records and the estimates do not always form
one cloud. Some records take the optimiser to a second maximum, typically one
where most events are missed, and the results come back as two groups an order
of magnitude apart. The mean of a set like that describes neither group.

mixture2 fits two Gaussian components in log space, for groups that overlap or
nearly do, and reports the proportion in each and their geometric means.
gap_split finds the largest multiplicative step between neighbouring sorted
values, for a few runaways cleanly separated from the body.

The point of both is that the split comes from the data. Choosing a threshold
by eye and counting either side makes the answer a property of the threshold;
these let an exclusion be stated -- "16 values above a 55-fold gap" -- and
checked by someone else. Log space because rate constants are positive with
multiplicative errors, so a group of them is log-normal rather than normal;
that also makes Mixture.means geometric means.

Written for the reproduction of Colquhoun, Hatton & Hawkes (2003), whose Fig. 2
is this situation exactly, but nothing in the module is specific to that paper
or to ion channels. numpy only.

23 tests, against data generated from a known mixture rather than against
values the function produced itself, because a mixture fit cannot look wrong:
it returns two plausible components whatever it is handed, and raises nothing.
Proportions are recovered at 10, 27 and 50 per cent so the tolerances are not
tuned to one case. Two tests keep the suite honest rather than checking
arithmetic: one records that a unimodal set is split without complaint, since
that is the mistake a user will make and the separation of the means is the
only thing that tells them; the other shows the recovery tolerances would
reject the most likely way for the fit to be broken while still looking
reasonable.

Suite goes from 75 to 98. The clean-install job needs no change -- it
enumerates modules since #8, and picks this one up as the eighth.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@remislp
remislp merged commit f94d7ae into master Sep 8, 2026
6 checks passed
@remislp
remislp deleted the ekdist-mixtures branch September 8, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant