Skip to content

fix: stop worker jobs using stale ambient Codex - #4038

Merged
joshavant merged 1 commit into
mainfrom
security/patched-worker-inputs-20261023
Oct 25, 2026
Merged

joshavant merged 1 commit into
mainfrom
security/patched-worker-inputs-20261023

Conversation

@joshavant

@joshavant joshavant commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

What Problem This Solves

Fixes worker jobs continuing to execute an older ambient Codex when npm installs the reviewed version into another global prefix, and refreshes the available worker trust inputs.

User Impact

Worker jobs select and verify Codex 0.164.0 from their own job directory. The shared installer selects Node 24.22.1 for npm; stock Bun selectors move to 1.4.4, uv to 0.13.5, and the normal AIG hash lock to certifi 2026.10.23.

Why This Change Was Made

One local setup action owns native CLI installation, exact version verification and the GitHub PATH export for all eight callers. Renderer dependencies retain their separate install step. Existing credentials, branch guards, source pins and publication controls retain their owners.

Evidence

  • Actual real-npm reproduction: installation succeeds while the old ambient CLI remains selected. The repaired action selects the reviewed native executable without overwriting the ambient fixture; its SHA256 matches the separately qualified Codex 0.164.0 Linux ARM artifact. Two regression cases cover independent prefixes and rejection of a wrong installed version.
  • Final source: static checks, full unit suite (8,081 passed; 3 existing skips), root TypeScript check and seven focused workflow files (27 tests) passed. Workspace package/build gates and eight read-only HTTP/CLI checks passed with unchanged application/runtime inputs. Final P0–P3 independent review is scoped-clean.
  • Actual x86_64 Linux userspace under QEMU: source-pinned producer builds, full curated AIG dependency installation and certifi 2026.10.19→2026.10.23 hash-locked upgrade, dependency check and installed CLI help passed. Frozen SkillSpector installation passed 24 adapter tests.

Additional exact-head native AMD64 proof closes the complete installer gate:

  • Native Ubuntu26 full installers: producer/AIG/SkillSpector installers exit0, AIG259 passed, SkillSpector24 passed, dependency checks and installed CLI help passed.
  • Ubuntu24.04.5 userspace on the same native AMD64 AWS guest: four CPUs/8GiB, complete installers exit0, AIG259 passed, SkillSpector24 passed; exact setup-codex shell block with actual npm admits0.164.0 on verified Node24.22.1. Executed installer/lock inputs match candidate Git blobs. Installed Codex ELF SHA256 b93f6811d3f653f0fb83be4050d857fb9abbb111dfa2ce160563327daff7f306; source-built ClawScan ELF de4fe416e63b605d9a4442cef22e55bd30679256d6b278d4f658a8290152312f. Qualification image ID sha256:4a213773c1980db9de43e93a37da30e0eb062e0c50bdcc21499c30546a426453; evidence archive SHA256 180810940c7ac718f2cf24965c93159330752396691ddb6d58bef82c466a48b5.

These are native AMD64 installer results, with Ubuntu24 userspace using the guest's Ubuntu26 kernel. They do not establish exact hosted-image identity or deployed delivery. Historical emulation interrupted after71 failures/98passes; only its first timeout was paired, and the other70 remain unattributed. Assertions and deadlines are unchanged.

Policy-required freshly installed benign/malicious real scanner-and-judge acceptance remains incomplete. The bounded full run completed benign static/AIG/SkillSpector stages but failed both judge attempts; the exact consumer correctly rejected the artifact. Malicious input reached600s without a final artifact. Scanner requests observed expected controls and HTTP200 response headers, which do not prove native judge success or completion of timed-out work. TMPDIR diagnosis removed a helper warning but retained native authentication/transport failure markers. The subsequent same-command official Responses positive control returned HTTP401; the fail-closed harness skipped both planned native Codex0.160/0.164 comparisons. Root cause and PR attribution remain unresolved. A valid approved credential/endpoint path and successful exact-head policy acceptance are still needed. No security check, receipt, sandbox, credential boundary or workflow guard was relaxed.

Current-head openclaw/openclaw-secops owner approval is also outstanding. No source changes were needed or pushed during this additional validation.

Native AMD64 default-HTTP trust gap found; hold for an owning upstream/backend repair. The 63-case exact-ELF wire proof exercised all 14 incident endpoints. With no CA override, seven endpoints reached client Finished followed by large encrypted client data records; rejection of those compromised chains was not enforced. Companion current-endpoint chain captures match the original certificate DER identities and compromised SPKIs; system OpenSSL verification accepts the same seven chains. This disproves the default-backend certificate-rejection claim, without establishing remote authenticated HTTP receipt or successful inference. Native HTTP uses the system/default TLS backend; the selected-CA path forces Rustls. All 28 direct explicit historical-CA cases and all 14 selected-CA WebSocket-to-HTTP fallback cases rejected before Finished. Private trusted-CA HTTP and fallback POST controls passed, unknown/wrong-host private-CA controls rejected, and the public legitimate HTTP control returned an actual HTTP405. No real credential was used.

Exact candidate ELF remains b93f6811d3f653f0fb83be4050d857fb9abbb111dfa2ce160563327daff7f306; wire export SHA256 f8ec1a4c2fcbfa9b3c58d67938bdb4239293f5bf6f11b3c2a421a42c03d65706. Default/no-CA WebSocket-to-native-HTTP fallback is still unproven. The two released predecessor controls selected an explicit historical CA and therefore do not establish predecessor default-native behavior; this gap is not attributed as a new regression introduced by the upgrade. Closure requires an owning default-backend repair qualified through default HTTP, explicit CA and default/no-CA fallback, with legitimate controls. No local trust-policy workaround or security weakening was applied.

The later independent auth-only official Responses control also returned HTTP401 outside the product/guest path. This confirms an unavailable current authentication resource, without identifying credential/account/routing/provider cause. The user deferred credential-dependent work; real scanner/judge acceptance remains blocked until that approved resource is restored. SkillSpector's frozen upstream certifi 2026.06.17, compiled Xet/custom old-CA transports, separate proxy binaries and deployed-worker delivery remain distinct open gaps. Python root-data updates do not reject arbitrary explicit old CA stores.

Refreshed upstream default-native trust hold (October 24 UTC)

A fresh official GitHub/npm check still selects stable Codex 0.164.0, source 6aaac694803004a63c6f393d153c96d22521a6b9. The newer 0.165.0-alpha.1 and inspected main 970111f6cb62cc02fe36a82ea7d71b37a2889c97 retain the identical complete codex-rs/http-client tree (a3a9da460393dd54f4b1cef30aef4bd528af0287) and relevant TLS dependency lock entries. No released or maintained owning repair for this default-backend gap was identified. The owning source forces Rustls for a selected CA override and otherwise preserves the default builder.

The retained native AMD64 actual-CLI proof, ELF SHA256 b93f6811d3f653f0fb83be4050d857fb9abbb111dfa2ce160563327daff7f306, accepted seven of 14 matching incident chains through client Finished and subsequent encrypted client records on Ubuntu 26's native/default HTTP path. Hold affected default-native uses for an owning repair and actual-artifact default HTTP, default/no-CA WebSocket-to-HTTP fallback, explicit-CA and legitimate HTTPS/private-CA controls. This is an environment/backend-specific demonstrated gap, not an attributed new regression or invalidation of the separate ARM64/explicit-CA proofs above. Provider-authentication status does not establish certificate rejection; credentialed acceptance is tracked separately. No forced-CA workaround, source change or gate waiver is introduced by this status update.

@joshavant
joshavant requested a review from a team as a code owner October 23, 2026 19:15
@clawsweeper

clawsweeper Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@vercel

vercel Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
clawhub Ready Ready Preview Oct 23, 2026 7:17pm UTC

Request Review

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on maintainer Only explicit maintainer approvals or decisions remain; merge is still blocked. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Oct 23, 2026
@clawsweeper

clawsweeper Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Codex review: blocked before merge.

What this changes

This PR centralizes verified Codex installation and refreshes worker runtime and certificate pins.

Example: A worker already has Codex 0.160.0 on PATH while npm uses a separate global prefix.

  • Before: Installing another version can leave the worker executing Codex 0.160.0.
  • After: The shared action installs Codex 0.164.0 in the job directory, verifies it directly, and exports its directory through GITHUB_PATH.

Review scores

Measure Result What it means
Overall readiness 🦪 silver shellfish (2/6) The implementation is focused and head CI supports it, but installation qualification does not complete the explicitly required live acceptance.
Proof confidence 🦪 silver shellfish (2/6) Real behavior proof is not required for maintainer- or bot-authored pull requests.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: Workers can execute an older Codex despite an intended reviewed runtime installation.
Reason: Selecting the intended executable restores the existing pinned-runtime contract, and the maintainer-authored direction also includes the scoped trust-input refresh.

Merge readiness

⛔ Blocked before merge - 1 item remains

Keep this PR open: current main still permits stale ambient Codex, and no introduced correctness defect was found in the pinned diff.

Priority: P2
Reviewed head: ee77b14d90bddf1bb9af809190f4a2b9dc3f1ad6

Before merge

  • Await maintainer action - Obtain current-head approval from openclaw/openclaw-secops for the action and workflow changes required by CODEOWNERS.

Findings

None.

Tests

  • Missing end-to-end proof: Exact-head freshly installed benign and malicious scanner-and-judge acceptance must complete successfully under the existing policy; installer success and the unsuccessful live attempts do not establish that result.
Agent review details

How this fits together

GitHub Actions prepares the runtimes used by security scans, Skill Cards, evaluations, and design audits; those jobs consume queued work or fixtures and produce reports and artifacts.

flowchart TD
  A[Worker workflow] --> B[Select Node runtime]
  B --> C[Install job-local Codex]
  C --> D[Verify exact version]
  D --> E[Export executable path]
  E --> F[Run worker or diagnostic]
Loading

Technical review

Best possible solution:

Retain the shared installer and existing security boundaries, then satisfy the recorded acceptance and owner-review gates before landing.

Do we have a high-confidence way to reproduce the issue?

Current main demonstrably accepts any existing Codex in the Skill Card worker and does not bind global npm installation to subsequent executable selection; no reproduction was executed during this read-only review.

Is this the best way to solve the issue?

The shared setup action is the best fix location because it owns installation, exact-version verification, and executable selection for every replaced caller; updating version strings alone would preserve the prefix-selection defect.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against d93282970b63.

Provenance checked

  • Codex worker installation keeps the original intent (fix: update codex worker cli pin #2915: The earlier pin update retained required isolation and structured-output flags; the proposed action preserves those safeguards while preventing ambient executable selection.)
  • Scanner resolver and certificate pins keeps the original intent (6e6c57d: The previous update selected released trust fixes through the existing installer and hash lock; this PR follows the same ownership and installation boundaries.)
  • Stock Bun selectors changes intended behavior with a stated reason (911211f: The prior dependency qualification selected Bun 1.3.14 with a compiler repair; the new stock selection is accompanied by reported broad validation and passing head checks.)
  • Diagnostic attestation and security policy keeps the original intent (23bb691: Diagnostics attest the installed runtime rather than treating workflow success as recovery evidence; changing the expected Codex version keeps that attestation aligned with installation.)

Testing

Proof path: shipped entry point.

Security

None.

Evidence

What I checked:

  • Pinned introduction reviewed: The merge-base-to-head diff contains the shared action, eight caller replacements, runtime pin updates, regression coverage, and policy documentation; base-only diagnostic additions are excluded from PR ownership. (ee77b14d90bd)
  • Current-main defect: The Skill Card worker skips installation whenever any codex executable exists; other workers install globally without selecting an explicit installation prefix. (.github/workflows/skill-card-worker.yml:72, d93282970b63)
  • Executable selection repair: Installation and verification use the same explicit job-local prefix; PATH export happens only after the installed executable reports the required version. (.github/actions/setup-codex/action.yml:19, ee77b14d90bd)
  • Updated validation and retained failure: The complete live PR body reports passing native AMD64 installer checks, but benign judge attempts failed, malicious input produced no final artifact before timeout, and a later Responses control returned HTTP401 before either runtime comparison ran. The author's re-review comment explicitly retains these blockers: fix: stop worker jobs using stale ambient Codex #4038 (comment). (ee77b14d90bd)
  • Required real-provider acceptance: The existing policy requires freshly installed scanners and the real judge to complete benign and malicious text-only fixtures with bounded requests and truthful model and tier evidence. (specs/security-consumer-model-policy.md:144, ee77b14d90bd)
  • Security owner approval outstanding: CODEOWNERS requires openclaw/openclaw-secops review for actions and workflows; the live review endpoint contains no submitted reviews. (.github/CODEOWNERS:9, ee77b14d90bd)

Review metrics

Metric Value Why it matters
Patch size +174/-70 across 26 files Most production changes replace repeated installation steps or update existing pins; the larger test addition exercises prefix selection and wrong-version rejection.

Labels

Label changes:

  • add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
  • remove rating: 🧂 unranked krab: Current PR rating is rating: 🦪 silver shellfish, so this older rating label is no longer current.

Label justifications:

  • P2: This repairs worker runtime selection without evidence establishing an urgent production regression caused by the proposed changes.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
  • status: ⏳ waiting on maintainer: Only explicit maintainer approvals or decisions remain; merge is still blocked.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

History

Review history (1 earlier review cycle)
  • reviewed 2026-10-23T19:30:21.826Z sha ee77b14 :: blocked before merge. :: none

Reviewed October 23, 2026, 10:52 PM ET / October 24, 2026, 02:52 UTC (Revision 2).

@joshavant

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Exact head remains ee77b14d90bddf1bb9af809190f4a2b9dc3f1ad6; no source push. The updated body records complete native AMD64 installer gates passing, with run links, input/artifact identities and platform limits. Required live scanner-and-judge acceptance remains incomplete: benign judge failure, malicious600s timeout, and later official current-key Responses controlHTTP401. The latter correctly skipped both planned native runtime variants, so no base/candidate attribution is claimed. Secops owner approval remains outstanding. Please retain these explicit blockers when reviewing the new evidence.

@clawsweeper

clawsweeper Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
Exact review queued.

Re-review progress:

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Oct 24, 2026
@joshavant
joshavant merged commit 5235885 into main Oct 25, 2026
113 of 116 checks passed
@joshavant
joshavant deleted the security/patched-worker-inputs-20261023 branch October 25, 2026 02:19

This branch was successfully deployed

1 active deployment
Preview – clawhub — ee77b14d Deployed Oct 23, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal backlog priority with limited blast radius. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: ⏳ waiting on maintainer Only explicit maintainer approvals or decisions remain; merge is still blocked.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant