Feature/release prep

[laurejt]

Associated Issue(s): resolves #15

Changes in this PR

• Update README
• Add spanerr-metrics project script to pyproject.toml (runs compute_metrics.py)
• Add --binary option to compute_metrics.py to for binarizing inputs
• Add --save-intermediate option to compute_metrics.py for saving intermediate (document-level) results

Notes

• Does it seem reasonable to table the lambda function documentation (Add documentation for explaining the lambda approach used for both alignment and scoring #16) until a future release?
• Does it seem reasonable to have different intermediate results for entity and document-level metrics? Basically, document-level gives more results since metrics are actually calculated at the document level.
• Would it make sense to change --binary to --binarize?

Notes for Testing

Case 1. Binarize labels
Note that these are the same strategy choices used in #14's test cases.

Command:

uv run spanerr-metrics tests/test_data/ref.jsonl tests/test_data/sys.jsonl entity select_first overlap_factor --binary

Final Output

Macro entity-level metrics for 9 documents:
- Precision = 0.5972
- Recall = 0.5701
- F1 = 0.5833

Command:

uv run spanerr-metrics tests/test_data/ref.jsonl tests/test_data/sys.jsonl document corppa jaccard --binary

Final Output

Macro document-level metrics for 9 documents:
- Precision = 0.6037
- Recall = 0.591
- F1 = 0.5961

Case 2. Save intermediate results

First, get the intermediate results when computing entity-level aggregated metrics
Command:

uv run spanerr-metrics tests/test_data/ref.jsonl tests/test_data/sys.jsonl entity corppa overlap_factor --save-intermediate intmd_ent.csv

Final Output

Macro entity-level metrics for 9 documents:
- Precision = 0.3682
- Recall = 0.4184
- F1 = 0.3917

Intermediate Results (intmd_ent.csv)

doc_id,n_ref,n_sys,relevance
no_matches,2,3,0
empty,0,0,0
exclusive_1,3,3,3
exclusive_2,3,3,0
exclusive_3,3,3,1.2857142857142856
exclusive_4,3,3,1.5333333333333332
multi_1,3,2,0.7857142857142857
multi_2,1,4,0.6
gamma_14,4,4,2

Then, get the intermediate results when computing document-level aggregated metrics

Command:

uv run spanerr-metrics tests/test_data/ref.jsonl tests/test_data/sys.jsonl document corppa overlap_factor --save-intermediate intmd_doc.csv

Final Output

Macro document-level metrics for 9 documents:
- Precision = 0.4425
- Recall = 0.478
- F1 = 0.4438

Intermediate Results (intmd_doc.csv)

doc_id,n_ref,n_sys,relevance,precision,recall,f-1
no_matches,2,3,0,0.0,0.0,0
empty,0,0,0,1,1,1.0
exclusive_1,3,3,3,1.0,1.0,1.0
exclusive_2,3,3,0,0.0,0.0,0
exclusive_3,3,3,1.2857142857142856,0.42857142857142855,0.42857142857142855,0.42857142857142855
exclusive_4,3,3,1.5333333333333332,0.5111111111111111,0.5111111111111111,0.5111111111111111
multi_1,3,2,0.7857142857142857,0.39285714285714285,0.2619047619047619,0.3142857142857143
multi_2,1,4,0.6,0.15,0.6,0.24
gamma_14,4,4,2,0.5,0.5,0.5

Reviewer Checklist

• The README is legible and sufficient
• Can successfully run spanerr-metrics project script locally (see notes above)
• Intermediate results output seem reasonable

[codecov]

Codecov Report

❌ Patch coverage is 97.12230% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.34%. Comparing base (2316e15) to head (b3e3643).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop      #18      +/-   ##
===========================================
+ Coverage    99.30%   99.34%   +0.03%     
===========================================
  Files           11       11              
  Lines         1159     1230      +71     
  Branches        77       80       +3     
===========================================
+ Hits          1151     1222      +71     
  Misses           4        4              
  Partials         4        4              

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rlskoeser]

Reviewed the readme and provided some suggestions/feedback.

[rlskoeser]

Are you planning to publish on pypi? I would recommend doing that (would be nice to claim the package name). The easiest way is to use github actions workflow and pypi trusted publishing.

[laurejt]

Yes, that's my plan. Perhaps we can talk tomorrow about making an official submission? I plan on adding a GitHub Action once we get the initial release on pypi (but I could be misremembering the best workflow here).