Skip to content

ci: make scheduled verification a cold build - #2217

Merged
Th0rgal merged 13 commits into
mainfrom
codex/phase1-periodic-cold-build
Jul 31, 2026
Merged

ci: make scheduled verification a cold build#2217
Th0rgal merged 13 commits into
mainfrom
codex/phase1-periodic-cold-build

Conversation

@Th0rgal

@Th0rgal Th0rgal commented Jul 30, 2026

Copy link
Copy Markdown
Member

Summary

  • make scheduled verification a clean exact-head build: no sticky Lake state, restored Lake cache, or compiler ccache
  • route proof, compiler, and Foundry work through the explicit build-heavy capability label and document/install that label consistently
  • rematerialize reused checkouts safely and make workflow edits trigger the full build/compiler graph
  • extend the generated verify-sync contract so clean-build, path-filter, step, and runner-label drift fails closed
  • declare explicit reentrancy trust on the four self-delegate multicall test fixtures required by current validation

Why

The daily workflow forced the full graph but still consumed runner-local build and compiler state, so it did not prove that pinned sources build cleanly. During validation, the existing generic build route also exposed a capped 20 GiB host and reused-workspace cleanup hazards. This slice turns the scheduled lane into a reproducible cold receipt and gives heavy jobs an explicit resource capability gate.

Validation

  • make check: 640 tests, exit 0, 1:40.34, 78,536 KiB peak RSS
  • python3 scripts/check_verify_sync.py: 9 invariant groups, exit 0
  • python3 -m unittest scripts.test_check_verify_sync scripts.test_ci_infra_maintenance: 46 tests, exit 0
  • lake build Compiler.Modules.CallsTest: exit 0, 2:26.70, 64 remote jobs/threads requested by the focused run, node old-agent
  • lake build PrintAxioms Compiler Contracts: 2,687 jobs, exit 0, 1:29:40, 17,152 KiB local orchestrator peak RSS, node old-agent
  • make test-evmyullean-fork: exit 0, 1:06:18, 17,152 KiB local orchestrator peak RSS; bridge lemmas/test, native harness, and EndToEnd receipts passed across ashur and babylon
  • exact-head GitHub run 30577938694 passed: build, compiler regressions/audits, patched Foundry, gas calibration, and all four Foundry shards
  • exact-head Codex review: no major issues; exact-head OCR: no findings; all review threads resolved

Lean 4.31.0, Lake 5.0.0, LEAN_NUM_THREADS=8 for final acceptance commands. EVMYulLean remains pinned to official revision f7e4ee0dc8f8d5265ce822a937ab5be771f182e9. No theorem, axiom, proof gate, allowlist, unsafe declaration, or bypass was added.

@vercel

vercel Bot commented Jul 30, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
verity Ready Ready Preview Jul 30, 2026 8:09pm

Request Review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenCodeReview first-pass review

🟠 1 finding(s) (1 high) — see inline comments.

✅ Posted 1 inline comment(s).

OCR pilot metrics & packet coverage

OCR pilot metrics

  • Routing: config-docs (router-v10)
  • Changed files: 3 supported / 3 total; Lean 0, trust docs 0, workflow/scripts 3, contracts 0, docs 0
  • Changed lines: 46 supported; thresholds large Lean >=3 files or >800 lines
  • OCR: status success; comments 1; files 3; tokens 1118795; tool calls 76; warnings 0; duration 283s
  • Largest changed files: .github/workflows/verify.yml (+11/-7), scripts/verify_sync_spec_source.py (+9/-7), scripts/verify_sync_spec.json (+6/-6)

Pilot mode: advisory only. Codex Review remains the merge gate.

Comment thread scripts/verify_sync_spec_source.py
@Th0rgal

Th0rgal commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 282d3e2c40

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenCodeReview first-pass review

🟢 No findings from the OCR first pass. Advisory only.

No comments generated. Looks good to me.

OCR pilot metrics & packet coverage

OCR pilot metrics

  • Routing: config-docs (router-v10)
  • Changed files: 3 supported / 3 total; Lean 0, trust docs 0, workflow/scripts 3, contracts 0, docs 0
  • Changed lines: 59 supported; thresholds large Lean >=3 files or >800 lines
  • OCR: status success; comments 0; files 3; tokens 403403; tool calls 42; warnings 0; duration 209s
  • Largest changed files: scripts/verify_sync_spec.json (+15/-6), scripts/verify_sync_spec_source.py (+13/-7), .github/workflows/verify.yml (+11/-7)

Pilot mode: advisory only. Codex Review remains the merge gate.

@Th0rgal

Th0rgal commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: e324c2164f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Th0rgal

Th0rgal commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

@codex review

@github-actions

Copy link
Copy Markdown
Contributor
\n### CI Failure Hints\n\nFailed jobs: `build`\n\nCopy-paste local triage:\n```bash\nmake check\nlake build\nFOUNDRY_PROFILE=difftest forge test -vv\n```

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: d3c4e21875

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Th0rgal

Th0rgal commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: b3956cfff6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Th0rgal

Th0rgal commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

@codex review

@Th0rgal

Th0rgal commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

@codex review

@Th0rgal

Th0rgal commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 58f031ef95

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/CI_RUNNERS.md Outdated
@Th0rgal

Th0rgal commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

@codex review

@cursor

cursor Bot commented Jul 30, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_74ed8a29-dfc2-49db-9249-c8b7866e0e53)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 55746168b0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenCodeReview first-pass review

🟠 2 finding(s) (1 high / 1 medium) — see inline comments.

✅ Posted 2 inline comment(s).

OCR pilot metrics & packet coverage

OCR pilot metrics

  • Routing: config-docs (router-v10)
  • Changed files: 7 supported / 7 total; Lean 0, trust docs 0, workflow/scripts 6, contracts 0, docs 1
  • Changed lines: 204 supported; thresholds large Lean >=3 files or >800 lines
  • OCR: status success; comments 2; files 7; tokens 1351812; tool calls 113; warnings 0; duration 598s
  • Largest changed files: scripts/verify_sync_spec.json (+55/-6), scripts/verify_sync_spec_source.py (+46/-7), .github/workflows/verify.yml (+34/-18), scripts/check_verify_sync.py (+22/-0), docs/CI_RUNNERS.md (+6/-3)

Pilot mode: advisory only. Codex Review remains the merge gate.

Comment thread docs/CI_RUNNERS.md Outdated
Comment thread docs/CI_RUNNERS.md Outdated
@Th0rgal

Th0rgal commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

@codex review

@cursor

cursor Bot commented Jul 30, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_411d57d9-6ba2-4096-b3da-01bfd3b36307)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Unknown error
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Th0rgal

Th0rgal commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

@codex review

@cursor

cursor Bot commented Jul 30, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_a9815d73-0133-4d7f-9d30-946a790e8b1e)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: d8a176e106

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

Copy link
Copy Markdown
Contributor

OpenCodeReview first-pass review

🟠 1 finding(s) (1 medium) — see inline comments.

✅ Posted 1 inline comment(s).

OCR pilot metrics & packet coverage

OCR pilot metrics

  • Routing: small-lean (router-v10)
  • Changed files: 8 supported / 8 total; Lean 1, trust docs 0, workflow/scripts 6, contracts 0, docs 1
  • Changed lines: 207 supported; thresholds large Lean >=3 files or >800 lines
  • OCR: status success; comments 1; files 8; tokens 1005488; tool calls 105; warnings 0; duration 504s
  • Largest changed files: scripts/verify_sync_spec.json (+55/-6), scripts/verify_sync_spec_source.py (+46/-7), .github/workflows/verify.yml (+34/-18), scripts/check_verify_sync.py (+22/-0), docs/CI_RUNNERS.md (+6/-2)

Pilot mode: advisory only. Codex Review remains the merge gate.

⚠️ Inline publication failed, so OCR findings are summarized here instead.

Inline findings that could not be posted

  • docs/CI_RUNNERS.md:16 — The Current Topology section still lists build-88-99-4-254-1 with labels: ..., verity, build, hetzner, hz2 and no build-heavy. This contradicts the rest of this change, which (a) adds build-heavy to every proof/compiler/Foundry job
Unprocessable Entity: "Line could not be resolved"

@Th0rgal

Th0rgal commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

@codex review

@cursor

cursor Bot commented Jul 30, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_1aa7ec12-232e-4eb9-9d9e-ff8ef78b24b8)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 695b1f633f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenCodeReview first-pass review

🟢 No findings from the OCR first pass. Advisory only.

No comments generated. Looks good to me.

OCR stderr highlights
[ocr]   ✘ file_read failed: invalid line range: start_line 470 is greater than end_line 100
OCR pilot metrics & packet coverage

OCR pilot metrics

  • Routing: small-lean (router-v10)
  • Changed files: 8 supported / 8 total; Lean 1, trust docs 0, workflow/scripts 6, contracts 0, docs 1
  • Changed lines: 209 supported; thresholds large Lean >=3 files or >800 lines
  • OCR: status success; comments 0; files 8; tokens 1906014; tool calls 130; warnings 0; duration 631s
  • Largest changed files: scripts/verify_sync_spec.json (+55/-6), scripts/verify_sync_spec_source.py (+46/-7), .github/workflows/verify.yml (+34/-18), scripts/check_verify_sync.py (+22/-0), docs/CI_RUNNERS.md (+7/-3)

Pilot mode: advisory only. Codex Review remains the merge gate.

@Th0rgal

Th0rgal commented Jul 31, 2026

Copy link
Copy Markdown
Member Author

Merging with corroborating evidence from last night: the un-provisioned cgroup-capped runner on agent-core picked up build/build-audits/build-compiler-binaries from the plain build label and failed them with exit 127 (scripts/ci_local_persistence.sh: No such file or directory — no sticky-disk/local-persistence provisioning on that host), which is precisely the mis-routing this PR's explicit build-heavy capability label prevents. I have already removed the build label from agent-core-verity-build-1 (it keeps fastlane, where it has been serving changes/checks/timeout-watchdog since yesterday). Full CI is green including the phased compiler-regressions.

@Th0rgal
Th0rgal merged commit 4895d4f into main Jul 31, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant