Repository navigation
Conversation
|
Superseded by #12, based on current main after merged #11 (a911c2e). Code comparison found that #11 covers the scalar-copy and timing improvements, while this PR's prepared-payload writer still has independent value. #12 preserves that part, reconciles exact source compatibility pins, handles nonfinite array-summary overflow at the construction boundary and retains #11's benchmark/tests. #12 passed 170 focused tests (3 Win32-only skips), strict fixed-clock cross-tree checkpoint equivalence and synthetic continuous/partial readback. Its published tree d65e925acedca6bbd250f3884e929936661a4d06 exactly matches the locally tested tree. Synthetic results are not real 121-frame production measurements; partial total speedup is not established. Closing this old-base conflicting PR to prevent accidental duplicate integration. Its branch is preserved; no force-push or changes to its working environment were made. Continue any checkpoint integration/review on #12. |
|
Cleanup completed after replacement #12 was verified and merged as 96c253e. At the user's explicit request, the unchanged obsolete branch at 1d17971 was removed with an exact-SHA lease. This PR remains closed and unmerged; its discussion and original commit record remain available. Its independent prepared-payload optimization is retained in #12, while the overlapping scalar-copy and timing implementations remain those from #11. |
Checkpoint snapshots traversed historical scientific records in _checkpoint_safe and then again in the writer's generic JSON conversion. Normalize metadata and scientific records at their existing boundaries, serialize the internally prepared snapshot directly, and reduce per-scalar recursive dispatch. Continue rebuilding history on every snapshot so mutations remain visible; public writer inputs still receive full conversion.
The version-1 fields, float precision, full history, commit frequency, synchronous flush/fsync/close/replace, and input/config/source/dependency/native-array checks remain intact. Add exact source pins for this reviewed storage change and the prior compact writer; preserve real producer identities and NPZ provenance.
Comparable serial 24-frame synthetic replay (20,001 profile values/frame, 25 synchronous commits): run_batch 4.127695 -> 2.706829 seconds (34.42% shorter), snapshot construction 0.964685 -> 0.558373 seconds, generic safety conversion 0.856871 -> 0.003305 seconds. The original 121-frame records were unavailable; these are synthetic results, not a production speedup claim. No scientific fitting, report, or package generation was performed.
Extend the existing benchmark with corrected non-nested safety timers, file write/flush/close timers, full/partial restoration replay, retained-record merge timing, and explicitly labeled synthetic fixtures. Recovery eligibility itself was small in these fixtures; keep those rules unchanged. Full historical reconstruction/serialization/write volume still scales quadratically.
Validation: 149 focused tests passed; 3 Win32-only metadata checks skipped on Linux. Strict field/type/float.hex comparisons pass for normal, full-resume and partial-resume replay; only runtime timestamps/durations and reviewed source identities differ. Tests cover all listed old source pairs, complete/partial native recovery, retained NPZ bytes/provenance, history mutations, signed zero, nonfinite values, commit failures, and real subprocess exits after fsync/replace. Focused Ruff fatal-error checks and git diff checks pass. Published Git tree matches the locally tested tree exactly.
Reproduction, stage tables, equivalence checks and remaining limits: docs/validation/checkpoint_snapshot_performance.md.