Skip to content

Reduce full batch delivery time and preserve reliable recovery - #7

Merged
D-sudoasd merged 7 commits into
mainfrom
codex/batch-delivery-performance
Oct 13, 2026
Merged

D-sudoasd merged 7 commits into
mainfrom
codex/batch-delivery-performance

Conversation

@D-sudoasd

Copy link
Copy Markdown
Owner

Batch delivery spent substantial time constructing heatmap paths for automatic legend placement, expanding arrays that checkpoints immediately discarded, and recompressing new arrays during partial recovery. Resuming a complete delivery also regenerated artifacts unnecessarily; canceled recovery could overwrite prior evidence.

This change uses fixed legend placement, lossless low-search NPZ compression, compact checkpoint serialization, and one staging archive during partial recovery. Verified complete generations can retain their native outputs, reports and ZIP. Reuse binds inputs, configuration, analyzer/source/dependency identity, artifact hashes and write-sensitive file versions. Output-directory OS locks cover checkpointing through publication. Recovery retains original metadata and exact fit vectors, protects prior files on cancellation, and retries checkpoint-ahead frames whose native evidence was never published.

Validation on six real calibrated HDF5 frames, with identical parameters, one numerical thread and all native/PNG/SVG/PDF/ZIP outputs:

  • Alternating first-run measurements: 273.89 → 208.30 s mean (1.315×). Final source also completed a fresh run in 164.77 s. Shared HDD load was not isolated; a later baseline took 308.42 s with substantial waiting, so its larger single-run ratio is not treated as pure code acceleration.
  • Same-correctness partial recovery, two retained frames plus four fits: 193.80 → 155.05 s. Finalization CPU: 32.39 → 12.64 s; about 0.70 GB each of redundant reads/writes removed.
  • Same-source complete-reuse ablation: 100.50 → 47.81 s, zero fits on both sides; all 205 published files unchanged on normal reuse. Another normal reuse took 19.14 s.
  • Real abrupt exit after the third durable checkpoint preserved the prior native package and commit marker. Recovery completed with two retained frames and four fits. The slower recovery observation (305.15 s) is retained in the local evidence.
  • Five strict full-output audits passed: 205 files, 480 arrays / 342,191,758 values, 1,177,686 CSV rows, JSON, 49 PNGs, 96 SVG/PDF figures, and every one of 203 ZIP members. Arrays are bit-identical, including masks and NaNs. A separate repeated-baseline PDF had three last-digit path-coordinate differences; the final source matches the originally retained baseline exactly, and both variants render identically at 300/600 dpi. No comparison tolerance was added.
  • Focused checks: 244 passed / 2 skipped / 1 deselected, subsequent 114 recovery/checkpoint checks, 27 final recovery checks, and 8 final CLI/service seam checks. These sets overlap. Ruff across src/tests/scripts and diff whitespace checks passed.

The original baseline needed the common missing-ellipse-estimate export correction before it could complete this workload. Existing scientific quality warnings remain visible; this is engineering/performance validation, not scientific acceptance. Real full2d performance, other input domains, larger production batches and cross-platform timing remain unmeasured. Reproducible entry points and evidence rules are documented in docs/validation/batch_delivery_performance.md; private inputs and local validation artifacts are excluded from this PR.

@D-sudoasd
D-sudoasd merged commit 6a29f57 into main Oct 13, 2026
8 checks passed
@D-sudoasd
D-sudoasd deleted the codex/batch-delivery-performance branch October 13, 2026 01:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant