fix(cuad): stop collapsing an undefined AUPR to a score of 0 - #802
Open
promptsmith1990 wants to merge 1 commit into
Open
fix(cuad): stop collapsing an undefined AUPR to a score of 0#802promptsmith1990 wants to merge 1 commit into
promptsmith1990 wants to merge 1 commit into
Conversation
get_aupr() called np.trapz(processed_precisions, recalls) and, whenever that returned NaN because the integral was undefined, collapsed it to 0 - the worst attainable AUPR. An uncomputable score and a model that scored the worst possible result were indistinguishable in the returned metrics, and CUAD is a legal-contract benchmark where this aggregate is the number people quote. NaN happens whenever a sample's precision/recall is itself NaN, which compute_precision_recall produces for a 0/0 case - a question with no ground-truth answers and no predicted answer text (a legitimate "correctly abstained" sample). That NaN needs to survive two steps to reach the aggregate honestly, and only the second one was fixed by just touching get_aupr: 1. process_precisions() runs a cumulative max() from the high-recall end backward to build the precision envelope. Python's max() is order-dependent with NaN - max(0.5, nan) is 0.5, but max(nan, 0.5) is nan - so an undefined precision could get silently overwritten by a neighboring real value depending on which side of the pair it landed on. Fixed to propagate NaN regardless of argument order. 2. get_aupr() itself: removed the `if np.isnan(aupr): return 0` floor. Also fixed in the same function: np.trapz was removed outright in NumPy 2.0 (not just deprecated), and this package's own setup.py allows numpy>=1.17 with no upper bound in the base install (the numpy<2.0 pin only applies to the tensorflow extras), so get_aupr() raises AttributeError on any modern numpy - not something covered by the reported issue, but the same line, and worth fixing alongside it rather than leaving a second break in the same function. Picks np.trapezoid when available and falls back to np.trapz otherwise. Fixes huggingface#801. Tests (metrics/cuad/test_cuad.py, new): - process_precisions propagates NaN through the running max regardless of which side of the pair it starts on (both orderings) - get_aupr still returns a real 0.0 for a genuine single/zero-area curve, unaffected by the fix - get_aupr returns nan instead of flooring to 0 when given a NaN input - end-to-end compute_score() reports a nan aupr for a dataset with one normally-scored sample and one 0/0 (unanswerable, no prediction) sample, rather than silently scoring it as the worst possible AUPR All 5 pass locally (`pytest metrics/cuad/test_cuad.py`), and `black --check` / `isort --check-only` / `flake8` are clean on both touched files. Verified the existing docstring example in cuad.py still produces byte-identical output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #801.
Problem
get_aupr()callsnp.trapz(processed_precisions, recalls)and, whenever that returnsNaNbecause the integral is undefined, collapses it to0— the worst attainable AUPR:An uncomputable score and a model that scored the worst possible result are indistinguishable in the returned metrics. CUAD is a legal-contract benchmark, so this aggregate is the number people quote (as noted in the issue).
Root cause, and why one fix wasn't enough
NaNshows up whenever a sample's precision/recall is itselfNaN, whichcompute_precision_recallproduces for a 0/0 case — a question with no ground-truth answers and no predicted answer text (a legitimate "correctly abstained" sample). Tracing that NaN through to the aggregate, it has to survive two steps, not one:process_precisions()runs a cumulativemax()from the high-recall end backward to build the precision envelope. Python'smax()is order-dependent withNaN—max(0.5, nan)is0.5, butmax(nan, 0.5)isnan— so an undefined precision could get silently overwritten by a neighboring real value depending on which side of the pair it landed on. Fixed to propagateNaNregardless of argument order.get_aupr()itself — removed theif np.isnan(aupr): return 0floor.Only fixing (2) isn't sufficient on its own: in the realistic end-to-end case (a dataset with a mix of normal and degenerate samples), the
NaNgets swallowed by (1) before it ever reaches (2), so the bug survives unless both are fixed together.Also fixed in the same function
np.trapzwas removed outright in NumPy 2.0 (not just deprecated), and this package's ownsetup.pyallowsnumpy>=1.17with no upper bound in the base install — thenumpy<2.0pin only applies to thetensorflowextras. Soget_aupr()currently raisesAttributeErroron any modern numpy install. Not something the linked issue reported, but it's the same line I was already touching, so I fixed it alongside rather than leave a second break in the same function: picksnp.trapezoidwhen available and falls back tonp.trapzotherwise.Tests
metrics/cuad/test_cuad.py(new):process_precisionspropagatesNaNthrough the running max regardless of which side of the pair it starts on (both orderings)get_auprstill returns a real0.0for a genuine single-point/zero-area curve, unaffected by the fixget_auprreturnsnaninstead of flooring to0when given aNaNinputcompute_score()reports ananaupr for a dataset with one normally-scored sample and one 0/0 (unanswerable, no prediction) sample, rather than silently scoring it as the worst possible AUPRblack --check/isort --check-only/flake8are clean on both touched files (using the pinnedblack~=22.0). Verified the existing docstring example incuad.pystill produces byte-identical output after this change.Prepared with AI assistance under my review; the root cause, the two-step propagation chain, and the tests were verified locally before opening this PR.