Conversation
- New crates/hotcoco/src/detection/streaming.rs: StreamingEval::new/add_image/ finalize runs COCOeval's per-image matching immediately as ground truth and detections become available, instead of waiting for a whole dataset before evaluate(). finalize() assembles an ordinary COCOeval for accumulate/ summarize/report. - Reuses matching::gather_pair/evaluate_cell unmodified; add_image builds a tiny single-image COCO per call so indexing cost scales with that image's annotation count, not the dataset's. No Open Images support (hierarchy expansion needs the whole GT dataset up front); the category list is frozen at construction so a category with zero annotations still gets a -1.0 slot; finalize()'s coco_gt/coco_dt carry categories only, not annotations, since accumulate/summarize/report never read annotations off those fields. - matching.rs: new pub(super) build_context() constructs EvalImgContext from parts, so streaming.rs can build its own per-image similarity map without widening tests/architecture.rs's similarity-cache allowlist (verified: all 10 architecture tests still pass unmodified). - New integration tests: streaming_matches_batch_on_tie_heavy_bbox reuses the shared tie_heavy_datasets() fixture to prove eval_imgs and the accumulated precision/recall/scores/ap_all_points arrays are bit-identical to the batch pipeline; streaming_matches_batch_on_lvis_federated_categories covers neg_category_ids/not_exhaustive_category_ids/plain-drop; streaming_rejects_ open_images pins the constructor error. Two fault injections proven before trusting the suite: sorting finalize()'s keys by (category, image) instead of (image, category), and dropping the last area-range slot in add_image (caught by an internal debug_assert_eq! on slot count). - Compact per-detection records (~20 B/det) are not implemented here: matching is order-sensitive per area range (matching::partition_gt reorders ground truths by area_rng), so a detection's matched status is [T*A], not the [T] the initial design sketch assumed. Building compaction against that unproven equivalence would risk a silent parity regression; left for a follow-up PR using this one's bit-identity tests as its oracle. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- New pyclass StreamingEval (crates/hotcoco-pyo3/src/lib.rs): new(categories, iou_type, lvis_style, params), add_image(image, gt_anns, dt_anns), and finalize() -> COCOeval, mirroring PyCOCOeval's constructor conventions. add_image()/finalize() on a spent (already finalized) evaluator raise RuntimeError instead of panicking or silently no-opping. - Re-exported as hotcoco.StreamingEval in python/hotcoco/__init__.py and added to __all__; stub declaration in python/hotcoco/__init__.pyi. Verified with scripts/test_stubs.py (name-drift check, 29/29 pass). - New crates/hotcoco-pyo3/tests/test_streaming_eval.py: get_results()/stats equality against a batch COCOeval, LVIS federated filtering (neg_category_ids/not_exhaustive_category_ids) verified by inspecting eval_imgs cells directly rather than AP arithmetic, and the spent-evaluator RuntimeError paths. One fault injection proven before trusting the suite: hardcoding EvalMode::Coco regardless of lvis_style in the constructor was caught by test_not_exhaustive_category_ignores_unmatched_dt, confirming the lvis_style -> EvalMode wiring itself is exercised, not just the Rust-side logic it delegates to. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CHANGELOG entry for the new hotcoco::StreamingEval / hotcoco.StreamingEval API: what it does, its scope (Coco/Lvis only, categories frozen at construction, finalize()'s coco_gt/coco_dt carry categories but no annotations), and what is explicitly left open (compact per-detection records, RF-DETR-side wiring). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new incremental evaluation feature,
StreamingEval, for COCO-style object detection evaluation, along with supporting Rust and Python bindings and comprehensive regression tests. The changes allow users to evaluate predictions image-by-image as data becomes available, rather than requiring the entire dataset at once. This improves efficiency, especially for large datasets or streaming scenarios, and ensures that evaluation results are bit-identical to the traditional batch pipeline. The implementation also ensures robust error handling and test coverage for both typical and edge cases.New Features:
StreamingEval(Rust:hotcoco::StreamingEval, Python:hotcoco.StreamingEval), enabling incremental (streaming) COCO evaluation by allowing images to be added one at a time and finalized into a standardCOCOevalobject for further processing. This moves expensive dataset loading and matching operations off the end-of-epoch critical path. [1] [2]StreamingEvalin the Python bindings and module exports, making it available for use in Python code.Testing and Robustness:
test_streaming_eval.py) to verify thatStreamingEvalreproduces the batch evaluation pipeline exactly, handles LVIS-style category filtering, and raises appropriate errors when misused after being finalized.Internal API and Codebase Improvements:
EvalImgContextin a controlled manner for per-image evaluation.