Skip to content

Reduce duplicate checkpoint snapshot conversion with stage-level replay evidence - #10

Closed
D-sudoasd wants to merge 1 commit into
mainfrom
codex/checkpoint-snapshot-conversion
Closed

D-sudoasd wants to merge 1 commit into
mainfrom
codex/checkpoint-snapshot-conversion

Conversation

@D-sudoasd

Copy link
Copy Markdown
Owner

Checkpoint snapshots traversed historical scientific records in _checkpoint_safe and then again in the writer's generic JSON conversion. Normalize metadata and scientific records at their existing boundaries, serialize the internally prepared snapshot directly, and reduce per-scalar recursive dispatch. Continue rebuilding history on every snapshot so mutations remain visible; public writer inputs still receive full conversion.

The version-1 fields, float precision, full history, commit frequency, synchronous flush/fsync/close/replace, and input/config/source/dependency/native-array checks remain intact. Add exact source pins for this reviewed storage change and the prior compact writer; preserve real producer identities and NPZ provenance.

Comparable serial 24-frame synthetic replay (20,001 profile values/frame, 25 synchronous commits): run_batch 4.127695 -> 2.706829 seconds (34.42% shorter), snapshot construction 0.964685 -> 0.558373 seconds, generic safety conversion 0.856871 -> 0.003305 seconds. The original 121-frame records were unavailable; these are synthetic results, not a production speedup claim. No scientific fitting, report, or package generation was performed.

Extend the existing benchmark with corrected non-nested safety timers, file write/flush/close timers, full/partial restoration replay, retained-record merge timing, and explicitly labeled synthetic fixtures. Recovery eligibility itself was small in these fixtures; keep those rules unchanged. Full historical reconstruction/serialization/write volume still scales quadratically.

Validation: 149 focused tests passed; 3 Win32-only metadata checks skipped on Linux. Strict field/type/float.hex comparisons pass for normal, full-resume and partial-resume replay; only runtime timestamps/durations and reviewed source identities differ. Tests cover all listed old source pairs, complete/partial native recovery, retained NPZ bytes/provenance, history mutations, signed zero, nonfinite values, commit failures, and real subprocess exits after fsync/replace. Focused Ruff fatal-error checks and git diff checks pass. Published Git tree matches the locally tested tree exactly.

Reproduction, stage tables, equivalence checks and remaining limits: docs/validation/checkpoint_snapshot_performance.md.

Copy link
Copy Markdown
Owner Author

Superseded by #12, based on current main after merged #11 (a911c2e). Code comparison found that #11 covers the scalar-copy and timing improvements, while this PR's prepared-payload writer still has independent value. #12 preserves that part, reconciles exact source compatibility pins, handles nonfinite array-summary overflow at the construction boundary and retains #11's benchmark/tests.

#12 passed 170 focused tests (3 Win32-only skips), strict fixed-clock cross-tree checkpoint equivalence and synthetic continuous/partial readback. Its published tree d65e925acedca6bbd250f3884e929936661a4d06 exactly matches the locally tested tree. Synthetic results are not real 121-frame production measurements; partial total speedup is not established.

Closing this old-base conflicting PR to prevent accidental duplicate integration. Its branch is preserved; no force-push or changes to its working environment were made. Continue any checkpoint integration/review on #12.

@D-sudoasd D-sudoasd closed this Oct 16, 2026
github-actions Bot added a commit that referenced this pull request Oct 16, 2026
Preserve #11 scalar conversion and retain the independent prepared-payload optimization from #10. All six Linux/Windows Python 3.11-3.13 CI jobs, lint and wheel smoke passed in run 38855207424. Scientific results, synchronous persistence and strict recovery identities remain unchanged.
@github-actions
github-actions Bot deleted the codex/checkpoint-snapshot-conversion branch October 16, 2026 05:43
@github-actions

Copy link
Copy Markdown
Contributor

Cleanup completed after replacement #12 was verified and merged as 96c253e. At the user's explicit request, the unchanged obsolete branch at 1d17971 was removed with an exact-SHA lease. This PR remains closed and unmerged; its discussion and original commit record remain available. Its independent prepared-payload optimization is retained in #12, while the overlapping scalar-copy and timing implementations remain those from #11.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant