Repository navigation
Conversation
Half-point ratings were detected as continuous, so judge_alignment gave no ordinal agreement score for them. Any bounded quantization grid is now likert, using the detector compare() already uses, and kappa counts categories as grid positions rather than the ranks of the values observed. score_type= is now accepted by the column-based form too, and declaring likert on off-grid scores raises. Test-specific alignment is printed as rho^2 with its interval instead of rho, and the metric dicts gain rho2 fields. With every item labeled the printed report omits the multiplier and N_eff; the new n_total= parameter (an int, or a per-condition dict for the multi-condition form) gives the size of the full dataset so they can be projected from a validation set alone. Bump the version to 0.3.4. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ubuntu-latest now resolves to Ubuntu 26.04, for which setup-python has no Python 3.9 build, so the 3.9 job fails at "Set up Python" before any test runs. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
judge_alignment(evaldata, ...) on data with 2+ conditions now reports rho^2 and
N_eff for the tests compare() will run, in AlignmentResult.comparison and in
.summary(). The design is read from the item ids (shared items: within;
disjoint: between) unless design= is given, and test= is honoured on this form
instead of being silently ignored. compare() and judge_alignment read one table
of which test runs for which design, so they report the same numbers.
The {condition: (judge, human)} dict form is removed; its machinery stays as a
private function that compare() and the evaldata form both call, now with
options to skip intervals and the raw per-pair correlations. n_total= accepts a
per-condition dict as well as an int and now also reaches the test-specific
N_eff. The label-position check is skipped when n_total= says the rows passed
in are only a validation subset.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ianarawjo
changed the title
summary() now reports "Overall agreement" (pooled across conditions), "Agreement by condition" (kappa and r), and "Label efficiency", which lists rho^2 and N_eff for each condition's mean, the omnibus test and each pair, each named by what it corrects. The same word, rho^2, meant three different quantities in the old layout. Pairs are listed up to 10; beyond that the summary gives a range and verbose lists them. Per-condition rho^2 and N_eff are stored in per_condition_metrics. The score type is stated as inferred or declared instead of explained. compare().summary() no longer prints the note that the verdict follows the corrected CI while p is corrected independently; the correction methods are already listed above. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
judge_alignmentnow takes the same evaldata ascompare()and reports everything in one call. Bumps the version to 0.3.4.One call, one report. On data with 2+ conditions,
judge_alignment(evaldata, llm_metric=..., human_groundtruth=...)reports rho^2 and N_eff for the testscompare()will run (AlignmentResult.comparison, printed by.summary()), beside the agreement metrics. The design is read from the item ids (the same items in every condition is within-subjects; items that never repeat is between-subjects);design=overrides, and a design that can't be inferred is reported in the summary rather than raised unless the caller named a design or test.test=is now honoured on this form to report one named test; before, it was accepted and silently ignored.compare()andjudge_alignmentread one table of which test runs for which design, so they report the same numbers (a test checks this).Half-point scales get kappa. Scores on any bounded quantization grid are detected as
likertwith the detectorcompare()already uses, so half-point ratings get quadratic-weighted kappa instead of being treated as continuous.score_type=is accepted on the evaldata form, and declaringlikerton off-grid scores raises. Kappa now counts categories as positions on the grid rather than the ranks of the values observed, so an unused level still counts toward the distance between its neighbours; this can change kappa on whole-number data with an unused level.rho^2, not rho. Test-specific alignment is reported as rho^2 with its interval. The raw Pearson and Spearman lines stay as r. Metric dicts gain
rho2,rho2_ci_low,rho2_ci_high;estimateis unchanged.n_total=. With every item labeled there is no larger pool, so the report omits N_eff.n_total=gives the size of the full dataset (an int, or a{condition: size}dict) so N_eff can be projected from a validation set alone. The label-position check is skipped in that case, since the rows passed in are only a validation subset.Report layout
summary()is split into "Overall agreement" (pooled across conditions), "Agreement by condition" (kappa and r), and "Label efficiency". The last lists rho^2 and N_eff for each condition's mean, the omnibus test and each pair, each named by what it corrects: the same symbol was standing for three different quantities. Pairs are listed up to 10, then summarized as a range;summary(verbose=True)lists every pair. The header states whether the score type was inferred or declared. Per-condition rho^2 and N_eff are inper_condition_metrics.compare().summary()also drops the note that the verdict follows the corrected CI while p is corrected independently; the correction methods are already listed above it.Breaking
Removed
judge_alignment({condition: (judge, human)}, design=..., test=...). Pass the evaldata you givecompare(). The per-condition machinery stays as a private function thatcompare()and the evaldata form both call, now with options to skip intervals and the raw per-pair correlations.Also
Pins the release-check runner to
ubuntu-24.04:ubuntu-latestnow resolves to Ubuntu 26.04, which has no Python 3.9 build forsetup-python, so the 3.9 job failed before running any test.Tests
New tests cover grid detection, the comparison report (between, within, ambiguous design, overrides, single condition, agreement with
compare()), andn_total. Run and passing:test_alignment(120),test_quick_primitives(41),test_summary_api_agreement(22),test_unpaired(64),test_compare(84),test_ppi_corrections(361),test_compound_ppi_fwer(18). CI passes on 3.9 and 3.12.🤖 Generated with Claude Code