[joshavant]

What Problem This Solves

The repository's acquired local workerd verifier can accept the compromised emSign keys even while the invoking Node runtime is patched.

User Impact

Local Wrangler and applicable Workers tests use workerd 1.20261021.1, which rejects those keys. Cloudflare's managed production runtime remains supplier-owned; dependency installation alone does not establish deployed-fleet remediation.

Why This Change Was Made

Use the existing pnpm workspace override owner and regenerate its lockfile with the declared package manager. Keep Wrangler, Miniflare, source inputs, installation policy and service lifecycle unchanged. Replace every affected nested workerd graph; preserve legitimate private-CA trust.

Evidence

Candidate c38799f7231953634dd6f051281ca8a094119a6c, base 6eb82bab1edf008f86859879fdaea3f7e022c495; native artifact SHA256 e98c9ce9473a19405f80bfa8979d0b7aabae6807ec43093b8d3a660e319bbd83.

• Actual owning Wrangler baseline accepted 14/14 verified incident endpoints with matching chain/hostname controls. Candidate result: 0/14 accepted; 14/14 underlying CertDistrusted; legitimate public/private HTTPS controls: both pass on baseline and candidate.
• Owner checks: pnpm lint → 0; pnpm build:sdk → 0; pnpm --dir apps/admin build → 0; pnpm typecheck → 0; pnpm test → 0; pnpm --dir apps/api exec wrangler deploy --dry-run --outdir dist-proof → 0; pnpm --dir apps/demo build → 0; pnpm docs:build → 0; pnpm verify:sdk-packages → 0. Workers-pool compatibility and nested-engine identity: All installed workerd binaries match the official Linux ARM64 artifact; no old engine graph remains in the final lock. Existing real Worker suites are included above..
• Independent review: canonical independent P0–P3 review completed scoped-clean, no actionable findings; source/lock review is distinct from all-platform qualification. Current-head hosted checks: not yet observed for this new head; consult this PR Checks tab.
• Evidence/artifact manifest: patch SHA256 067db51e8d3e6c7625f8454664ebf0608ba89d0f3db591de90ff600253c6625e; chain/fixture identities and results below. Platform and trust-mode limits: Hosted native keyring/runtime matrix remains separate. The generated lock also consolidates better-auth peer graphs from jose6.2.12 to already-direct6.2.13 within declared ranges; full existing auth/Worker tests pass. No direct jose requirement changed..

Existing automatic effects: PR: CI matrix, CodeQL, ClawSweeper dispatch, native keyring matrix because lock changes. Main: same checks and Docs GitHub Pages deployment because pnpm-lock.yaml matches its path filter. Production Cloudflare deploy is manual-only. Application release v* and SDK npm release sdk-v* require tag/manual events and are not invoked.

No merge, deployment, release, image/package publication or cloud operation was performed during local qualification.

Reproduction and evidence limits

Both revisions use the actual repository-installed Wrangler and its frozen dependency graph, patched Node 24.22.1, identical isolated Pages fetch fixture, historical extra-CA plus system roots, and matching requests. This exercises the acquired local native verifier, not each authenticated application route. No production test seam or trust bypass was added. Each of the 14 endpoint chains passed independent OpenSSL chain/hostname verification and matched verified incident SPKI identities;14chains cover six TLS issuer keys, not the two additional code-signing keys. Supplier/source comparison separately matches the full eight-key denylist. Generic internal-error envelopes are corroborated by 14 exact underlying CertDistrusted log entries.

Fixture SHA256 8cf12c8291a4d9e76b2f1e63dd93ea1a533f9fe174dca7078e9992579bc18f16; additional-CA SHA256 7ddc10d975d45dcb1906a13d984edf8e5fcf13e6877f5a0dbbfd503c4e841e98. No host trust, production service, Docker daemon or Cloudflare setting was changed. Mac/Windows/native x64, outbound WSS, every provider, production managed-runtime identity and fleet uptake remain unverified by this lane.

Matched incident endpoint results

Each row has valid independent chain and exact-hostname verification. Baseline TLS completes; candidate fails with underlying CertDistrusted.

Endpoint	Incident issuer	Baseline / candidate
dv-g3b-emudhra.we-have-a-problem.com	EM DV TLS CA - G3B	accepted / CertDistrusted
dv-g3c-emudhra.we-have-a-problem.com	EM DV TLS CA - G3C	accepted / CertDistrusted
ov-g3b-emudhra.we-have-a-problem.com	EM OV TLS CA - G3B	accepted / CertDistrusted
ov-g3c-emudhra.we-have-a-problem.com	EM OV TLS CA - G3C	accepted / CertDistrusted
ev-g3b-emudhra.we-have-a-problem.com	EM EV TLS CA - G3B	accepted / CertDistrusted
ev-g3c-emudhra.we-have-a-problem.com	EM EV TLS CA - G3C	accepted / CertDistrusted
dv-g3b-aaa-emudhra.we-have-a-problem.com	EM DV TLS CA - G3B	accepted / CertDistrusted
dv-g3c-aaa-emudhra.we-have-a-problem.com	EM DV TLS CA - G3C	accepted / CertDistrusted
ov-g3b-aaa-emudhra.we-have-a-problem.com	EM OV TLS CA - G3B	accepted / CertDistrusted
ov-g3c-aaa-emudhra.we-have-a-problem.com	EM OV TLS CA - G3C	accepted / CertDistrusted
ev-g3b-aaa-emudhra.we-have-a-problem.com	EM EV TLS CA - G3B	accepted / CertDistrusted
ev-g3c-aaa-emudhra.we-have-a-problem.com	EM EV TLS CA - G3C	accepted / CertDistrusted
dv-g3b-noct-emudhra.we-have-a-problem.com	EM DV TLS CA - G3B	accepted / CertDistrusted
dv-g3b-aaa-noct-emudhra.we-have-a-problem.com	EM DV TLS CA - G3B	accepted / CertDistrusted

Chain-controls JSON SHA256 337b73f7d54256bbbc0af661e648ac2db90b55e066a273baab6d1a63609d94ab; full candidate server log SHA256 b80d5e03027cb48f70b451e4bd5b58a76bd08f6aa8de5abb762e09de66df66a1. Both legitimate controls complete HTTPS under the same additional trust. These are TLS acceptance results, not model/application success at the incident sites.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs maintainer review before merge.

What this changes

Pin local workerd tooling to 1.20261021.1, regenerate the dependency graph, and document the security fix.

Example: Local Wrangler fetches dv-g3b-emudhra.we-have-a-problem.com using the documented additional trust.

• Before: The supplied baseline accepts its incident certificate chain.
• After: The supplied candidate rejects it with CertDistrusted while legitimate public and private HTTPS controls continue to pass.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused security dependency repair supported by matched shipped-entry-point results, verified upstream behavior, and successful exact-head checks.
Proof confidence	🐚 platinum hermit (4/6)	Sufficient (live_output): The supplied exact-candidate results exercise the repository-installed Wrangler entry point on Linux ARM64 with Node 24.22.1: all 14 incident chains change from accepted to CertDistrusted, while legitimate public/private HTTPS controls remain successful. Supplier source corroborates rejection before TLS success and prevents the direct-trust fallback from overriding distrust. Evidence covers local verification, not managed production fleet uptake or every platform trust mode.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: Local Worker tooling can trust HTTPS certificate chains containing compromised eMudhra keys even when the invoking Node runtime is patched.
Reason: The supplied maintainer-direction context authorizes the narrow security repair, which uses an established dependency mechanism and preserves existing installation and application contracts.

Merge readiness

✅ Ready for maintainer review

Keep this PR open: it addresses a concrete local TLS verification vulnerability still present on main, and the reviewed diff introduces no actionable defect.

Priority: P2
Reviewed head: c38799f7231953634dd6f051281ca8a094119a6c

Before merge

None.

Findings

None.

Agent review details

How this fits together

Wrangler and the Cloudflare Vitest plugin acquire workerd through the workspace dependency graph; its native TLS verifier decides whether local Worker HTTPS connections proceed.

flowchart LR
  A[Workspace override] --> B[Frozen dependency graph]
  B --> C[Wrangler and Workers tests]
  C --> D[Native workerd]
  D --> E[TLS certificate verification]
  E --> F[HTTPS accepted or rejected]

Loading

Technical review

Best possible solution:

Land the verified workerd pin through the existing workspace dependency owner while preserving legitimate trust and keeping managed-fleet remediation explicitly outside its scope.

Do we have a high-confidence way to reproduce the issue?

Main selects the older native verifier, and the supplied matched Wrangler runs demonstrate certificate acceptance before the fix and rejection afterward; this review independently inspected source without executing the reproduction.

Is this the best way to solve the issue?

The existing workspace override is the smallest complete repository fix, and supplier source confirms that the selected verifier rejects the compromised keys without broadly removing private-CA trust.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 6eb82bab1edf.

Provenance checked

• Workspace overrides and native workerd selection changes intended behavior with a stated reason (build(deps): patch Node runtimes and refresh compatible dependencies #161: The prior dependency refresh selected compatible Cloudflare tooling while patching Node and retaining the two-day dependency cooldown; earlier merged work established security overrides for pinned transitive dependencies.)
• Better Auth JOSE dependency resolution keeps the original intent (build(deps): patch Node runtimes and refresh compatible dependencies #161: The previous refresh updated Better Auth and JOSE; the regenerated lock now shares the already-direct JOSE 6.2.13 resolution within the existing peer range, without changing the declared requirement.)

Testing

Proof path: shipped entry point.

Security

None.

Evidence

What I checked:

• Pinned change and current-main boundary: The introduced diff adds the existing-owner workerd override, replaces all five platform packages and Wrangler/Miniflare engine references, consolidates JOSE onto the already-present 6.2.13 resolution, and adds an Unreleased entry. Main still resolves workerd 1.20261019.1. Installation permissions, release cooldown, workflows, application source, and persisted schemas are unchanged. (pnpm-workspace.yaml:12, c38799f72319)
• Supplied production-entry-point evidence: The full PR body reports actual installed Wrangler on Linux ARM64 with patched Node 24.22.1: baseline accepts 14 incident endpoints, candidate rejects all 14 with underlying CertDistrusted, and public/private HTTPS controls pass on both revisions. Fixture, native artifact, chain-control, and server-log hashes identify the evidence. Captured sourceRevision is 6f0decd78824feabea879d3c39ff252fecc2aa4915a45ab76dec1e692c4a3131; the full body and both discussion comments were read. (c38799f72319)
• Supplier verifier dependency and fallback: The official v1.20261021.1 tag resolves to this supplier commit and locks rustls-webpki 0.103.16. Its TLS owner delegates certificate verification before connection success; direct-trust fallback accepts only specified chain-building errors, excluding CertDistrusted. This dependency contract applies because the PR directly replaces the native verifier consumed by Wrangler and Workers tests. (src/rust/cxx/kj-hyper/tls.rs:402, 1038810cb01a)
• Exact upstream security mechanism: The 0.103.16 change rejects distrusted SPKIs in leaves, intermediates, and trust anchors, including cross-signs, using eight explicit eMudhra key fixtures. Its tests preserve unrelated valid paths. This is the verifier implementation selected by the patched workerd dependency. (src/verify_cert.rs:59, bb7dcc8d5087)
• Existing dependency ownership and validation: Merged dependency work establishes the workspace override pattern and retained cooldown, including build(deps): refresh compatible dependencies and patch Undici #87 and build(deps): patch Node runtimes and refresh compatible dependencies #161. Exact-head checks show 11 successes and two skipped dispatch checks, with no failures. The checkout remains clean; no target code, tests, or autoreview helper was executed. (pnpm-workspace.yaml:11, c38799f72319)

Likely related people:

• Peter Steinberger: Raw commit f7f30f3 adds pnpm-workspace.yaml:15 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: f7f30f343461; files: pnpm-workspace.yaml)
• Vincent Koc: Raw commit 2dd2f43 adds pnpm-workspace.yaml:19 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: medium; commits: 2dd2f4385981; files: pnpm-workspace.yaml)

Labels

Label changes:

• add P2: This is a concrete security repair to local development and test tooling, with a bounded repository-owned surface and no demonstrated production outage.
• add proof: sufficient: Contributor real behavior proof is sufficient.
• add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Label justifications:

• P2: This is a concrete security repair to local development and test tooling, with a bounded repository-owned surface and no demonstrated production outage.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.
• proof: sufficient: Contributor real behavior proof is sufficient.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 26, 2026, 5:28 AM ET / 09:28 UTC.