Skip to content

fix(cuad): stop collapsing an undefined AUPR to a score of 0 - #802

Open
promptsmith1990 wants to merge 1 commit into
huggingface:mainfrom
promptsmith1990:fix/cuad-aupr-nan-coercion
Open

fix(cuad): stop collapsing an undefined AUPR to a score of 0#802
promptsmith1990 wants to merge 1 commit into
huggingface:mainfrom
promptsmith1990:fix/cuad-aupr-nan-coercion

Conversation

@promptsmith1990

Copy link
Copy Markdown

Fixes #801.

Problem

get_aupr() calls np.trapz(processed_precisions, recalls) and, whenever that returns NaN because the integral is undefined, collapses it to 0 — the worst attainable AUPR:

def get_aupr(precisions, recalls):
    processed_precisions = process_precisions(precisions)
    aupr = np.trapz(processed_precisions, recalls)
    if np.isnan(aupr):
        return 0
    return aupr

An uncomputable score and a model that scored the worst possible result are indistinguishable in the returned metrics. CUAD is a legal-contract benchmark, so this aggregate is the number people quote (as noted in the issue).

Root cause, and why one fix wasn't enough

NaN shows up whenever a sample's precision/recall is itself NaN, which compute_precision_recall produces for a 0/0 case — a question with no ground-truth answers and no predicted answer text (a legitimate "correctly abstained" sample). Tracing that NaN through to the aggregate, it has to survive two steps, not one:

  1. process_precisions() runs a cumulative max() from the high-recall end backward to build the precision envelope. Python's max() is order-dependent with NaNmax(0.5, nan) is 0.5, but max(nan, 0.5) is nan — so an undefined precision could get silently overwritten by a neighboring real value depending on which side of the pair it landed on. Fixed to propagate NaN regardless of argument order.
  2. get_aupr() itself — removed the if np.isnan(aupr): return 0 floor.

Only fixing (2) isn't sufficient on its own: in the realistic end-to-end case (a dataset with a mix of normal and degenerate samples), the NaN gets swallowed by (1) before it ever reaches (2), so the bug survives unless both are fixed together.

Also fixed in the same function

np.trapz was removed outright in NumPy 2.0 (not just deprecated), and this package's own setup.py allows numpy>=1.17 with no upper bound in the base install — the numpy<2.0 pin only applies to the tensorflow extras. So get_aupr() currently raises AttributeError on any modern numpy install. Not something the linked issue reported, but it's the same line I was already touching, so I fixed it alongside rather than leave a second break in the same function: picks np.trapezoid when available and falls back to np.trapz otherwise.

Tests

metrics/cuad/test_cuad.py (new):

  • process_precisions propagates NaN through the running max regardless of which side of the pair it starts on (both orderings)
  • get_aupr still returns a real 0.0 for a genuine single-point/zero-area curve, unaffected by the fix
  • get_aupr returns nan instead of flooring to 0 when given a NaN input
  • end-to-end compute_score() reports a nan aupr for a dataset with one normally-scored sample and one 0/0 (unanswerable, no prediction) sample, rather than silently scoring it as the worst possible AUPR
$ pytest metrics/cuad/test_cuad.py -v
...
5 passed in 0.12s

black --check / isort --check-only / flake8 are clean on both touched files (using the pinned black~=22.0). Verified the existing docstring example in cuad.py still produces byte-identical output after this change.


Prepared with AI assistance under my review; the root cause, the two-step propagation chain, and the tests were verified locally before opening this PR.

get_aupr() called np.trapz(processed_precisions, recalls) and, whenever
that returned NaN because the integral was undefined, collapsed it to
0 - the worst attainable AUPR. An uncomputable score and a model that
scored the worst possible result were indistinguishable in the
returned metrics, and CUAD is a legal-contract benchmark where this
aggregate is the number people quote.

NaN happens whenever a sample's precision/recall is itself NaN, which
compute_precision_recall produces for a 0/0 case - a question with no
ground-truth answers and no predicted answer text (a legitimate
"correctly abstained" sample). That NaN needs to survive two steps to
reach the aggregate honestly, and only the second one was fixed by
just touching get_aupr:

1. process_precisions() runs a cumulative max() from the high-recall
   end backward to build the precision envelope. Python's max() is
   order-dependent with NaN - max(0.5, nan) is 0.5, but max(nan, 0.5)
   is nan - so an undefined precision could get silently overwritten
   by a neighboring real value depending on which side of the pair it
   landed on. Fixed to propagate NaN regardless of argument order.
2. get_aupr() itself: removed the `if np.isnan(aupr): return 0` floor.

Also fixed in the same function: np.trapz was removed outright in
NumPy 2.0 (not just deprecated), and this package's own setup.py
allows numpy>=1.17 with no upper bound in the base install (the
numpy<2.0 pin only applies to the tensorflow extras), so get_aupr()
raises AttributeError on any modern numpy - not something covered by
the reported issue, but the same line, and worth fixing alongside it
rather than leaving a second break in the same function. Picks
np.trapezoid when available and falls back to np.trapz otherwise.

Fixes huggingface#801.

Tests (metrics/cuad/test_cuad.py, new):
- process_precisions propagates NaN through the running max regardless
  of which side of the pair it starts on (both orderings)
- get_aupr still returns a real 0.0 for a genuine single/zero-area
  curve, unaffected by the fix
- get_aupr returns nan instead of flooring to 0 when given a NaN input
- end-to-end compute_score() reports a nan aupr for a dataset with one
  normally-scored sample and one 0/0 (unanswerable, no prediction)
  sample, rather than silently scoring it as the worst possible AUPR

All 5 pass locally (`pytest metrics/cuad/test_cuad.py`), and
`black --check` / `isort --check-only` / `flake8` are clean on both
touched files. Verified the existing docstring example in cuad.py
still produces byte-identical output.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CUAD compute_score: undefined AUPR (NaN from np.trapz) is collapsed to 0, the worst attainable score

1 participant