Repository navigation
Harden the parity vehicles for the 1733d30 re-qualification - #396
Merged
Merged
Conversation
…fication Brings modal_openvla_export_parity_pinned.py, modal_openvla_export_cpu_pinned.py and modal_groot_export_parity_pinned.py to the pi0/pi0.5 pattern. - No SIGKILL watchdog. A self-killed container is reported as an internal failure, and the modal 1.4.0 client retries those regardless of the retry policy (_functions.py MAX_INTERNAL_FAILURE_COUNT = 8), so one hung job could bill up to 8 times. Now retries=0; export and harness each run in a subprocess with an explicit timeout; the Modal timeout is the sum of those timeouts plus a setup margin. - Restart guard: a marker keyed by --run-id under <family>/attempts/ on the volume; a second start of the same run returns aborted-restarted-attempt before doing work. The local entrypoint generates a run id if none is given. - New volume root /parity_out/requal-1733d30-20261030/<family>/ (same constant style as pi0's PARITY_OUT_PATH), so the 2026-09/10 evidence is never overwritten. - Provenance-checked reuse. The export's provenance (Tether commit, model revision, export library versions; GR00T also embodiment id) is written beside the export, outside --onnx-dir so it never enters the artifact hash, and only after a successful export. A rebuild first deletes the old record and clears the export dir, so a partial export can never match. - OpenVLA GPU job never exports and never reuses an unproven export: it refuses (verdict refused-export-provenance-mismatch) unless the CPU builder's provenance under the new root matches exactly, and its summary names the CPU builder command. The old export built at an older Tether commit sits in the old root with no provenance record, so it cannot match. The CPU builder defaults to reuse_export=False. - --tether-revision and --run-id on every vehicle, recorded in the returned summary and in <family>/runs/<run-id>.json on the volume. Default TETHER_REVISION is 1733d30. - Timeouts from measured times where recorded: OpenVLA GPU harness 1800 s (round-2 reuse runs 10.3 and 3.9 GPU-min), Modal 2400 s; GR00T export 900 s and harness 600 s against a measured 201-204 s for both together, Modal 2400 s. OpenVLA CPU phases are unmeasured and say so: export 3600 s, pre-check 2400 s, Modal 7200 s (the whole job previously fit inside a 5100 s watchdog). A100-40GB kept; no doc shows an OOM for either family. - GR00T --strict-fp32: runs the unmodified harness with NVIDIA_TF32_OVERRIDE=0 and writes groot-export-parity-receipt-strict-tf32.json (raw receipt plus an environment block with the raw sha256), the only GR00T receipt Studio's committed_parity accepts. Without the flag the default-env receipt is written as before; that path failed full_max_abs in round 2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its second pip layer (onnx, onnxruntime) upgraded numpy to 2.5.4, so the export's recorded provenance never matched the GPU parity image (numpy 1.26.4) and the parity vehicle refused it. Re-pin numpy in that layer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- pi0, pi0.5: default to 1733d30, the pin their output root names; keep export provenance beside the export (out of the hashed artifact); clear the export dir and old provenance before a rebuild; unlink a stale receipt before the harness; reuse only when model.onnx exists. - All five vehicles: `modal run` exits non-zero unless the verdict is passed (restart guard, provenance refusal, no receipt, failed); a non-zero harness exit with a passed receipt raises; a phase timeout prints the hung child's output. - pi0, pi0.5, GR00T: onnx and numpy join the export provenance. - OpenVLA GPU: refuse when its artifact hash differs from the CPU pre-check's. - GR00T harness records the NVIDIA_TF32_OVERRIDE it observed and the CUDA EP options; the vehicle checks the recorded value when present. - tests/test_modal_parity_vehicle_exit.py: 31 pass; 22 fail without this. The 2026-10-30 receipts stay valid: each records tether_commit 1733d30 and came from a single run into a new root. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These are the vehicles that re-qualified pi0, pi0.5, OpenVLA and GR00T at 1733d30 on 2026-10-30. All four pass (Studio PR records the receipts); the Modal cost was $3.33. Only
scripts/changes;src/is untouched.requal-1733d30-20261030, so the 2026-10-12 evidence isn't overwritten.retries=0, and no SIGKILL watchdog. Modal 1.4 retries internal failures up to 8 times regardless of retry policy, so a killed container could bill repeatedly.--run-id.--strict-fp32, which writes the strict-TF32 receipt.Checks:
py_compileand ruff are clean. Each vehicle was exercised for real in the runs above.Review fixes (3fdaca9)
An independent review found that pi0 and pi0.5 had moved to the new root without the hardening GR00T and OpenVLA got. The "hardened pi0 pattern" wording above was wrong. Fixed:
modal runexits non-zero unless the verdict ispassedNVIDIA_TF32_OVERRIDEit observed and the CUDA EP options.Test:
tests/test_modal_parity_vehicle_exit.py: 31 pass; 22 fail without the fix.The 2026-10-30 receipts are unaffected: each records
tether_commit1733d30 and came from a single run into a new root. pi0 and pi0.5artifact_sha256include the old in-dir provenance JSON. Studio displays that hash and doesn't match on it (committed_parity.py:358).Not done: a shared constraints file for both OpenVLA images (review item 8). The GPU provenance gate already fails loudly on that drift.
🤖 Generated with Claude Code