Repository navigation
Reduce full batch delivery time and preserve reliable recovery - #7
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch delivery spent substantial time constructing heatmap paths for automatic legend placement, expanding arrays that checkpoints immediately discarded, and recompressing new arrays during partial recovery. Resuming a complete delivery also regenerated artifacts unnecessarily; canceled recovery could overwrite prior evidence.
This change uses fixed legend placement, lossless low-search NPZ compression, compact checkpoint serialization, and one staging archive during partial recovery. Verified complete generations can retain their native outputs, reports and ZIP. Reuse binds inputs, configuration, analyzer/source/dependency identity, artifact hashes and write-sensitive file versions. Output-directory OS locks cover checkpointing through publication. Recovery retains original metadata and exact fit vectors, protects prior files on cancellation, and retries checkpoint-ahead frames whose native evidence was never published.
Validation on six real calibrated HDF5 frames, with identical parameters, one numerical thread and all native/PNG/SVG/PDF/ZIP outputs:
The original baseline needed the common missing-ellipse-estimate export correction before it could complete this workload. Existing scientific quality warnings remain visible; this is engineering/performance validation, not scientific acceptance. Real full2d performance, other input domains, larger production batches and cross-platform timing remain unmeasured. Reproducible entry points and evidence rules are documented in
docs/validation/batch_delivery_performance.md; private inputs and local validation artifacts are excluded from this PR.