fix(artifacts): rotate the signing key without a gap by harlan-zw · Pull Request #556 · skilld-dev/skilld.dev

harlan-zw · 2026-10-07T05:31:08Z

🔗 Linked issue

❓ Type of change

📚 Description

The Artifact signing key skilld-production-2026-08 closes at 2026-11-20T00:00:00Z. After that the signer refuses to sign and the site stops trusting stored attestations, so every skilld run fails. The signer could not change keys without a gap: it read one key ID var and one secret, and wrangler secret put deploys at once with the old key ID.

The signer now has a primary and a secondary key slot. A secret with no vars signs nothing, so the new key can be staged first. For each Resolution, the signer uses the newest key whose window holds both the statement createdAt and the current time.
The site verifier now also requires createdAt inside the key window. The skilld CLI already applied that rule, so the site could store and reuse an attestation that every CLI refuses.
scripts/trusted-root.ts generates a signing key, signs its trusted root entry with the offline root key, and checks a root by the CLI rules. The site never checks a root signature, so this check gates every change to ARTIFACT_TRUSTED_ROOT_JSON.
The daily check-in gains skilld.signing-key. It warns 30 days and fails 7 days before no signer key is both open and trusted.
The signing key rotation runbook gives the ordered steps, the latest safe date for each, checks, and rollback. The rollout guide's old rotation steps now link to it.

Every released CLI from 3.0.0 to 3.6.4 pins only the root key skilld-root-2026, so this rotation needs no CLI release.

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

The skilld CLI refuses a signature over a statement created outside the key window. The site checked only the current time, so it could store, reuse and serve an attestation every CLI refuses until the key expired. A signing key cutover made that case reachable.

One key ID var set and one secret could not change together: a secret put deployed at once with the old key ID, so every build failed until the vars deploy landed. A staged secret in the free slot now signs nothing until its vars deploy. The signer picks the newest key whose window holds both the statement creation time and now.

The site never checks a root signature, and every released CLI does. A hand-built key entry with a bad signature stops every skilld run with an upgrade message no upgrade fixes. The tool signs entries with the offline root key and checks a root by the CLI rules before it reaches the secret.

skilld-production-2026-08 closes on 2026-11-20 and nothing watched it. The check reads the signer windows and the live trusted root, and finds when no key is both open and trusted.

The rollout guide's rotation steps swapped the secret and key ID non-atomically and retired the old key with an unsigned status change.

harlan-zw · 2026-10-07T05:31:36Z

Agent comment (Harlan Agent Kit). A local dry run of the whole rotation, with throwaway keys only. Nothing touched production.

Setup. A throwaway root key with the production key ID skilld-root-2026. K1 stands in for skilld-production-2026-08. K2 is skilld-production-2026-11, window 2026-10-14 to 2027-05-20. scripts/trusted-root.ts built root v2 (K1 + K2) and root v3 (K2 only), and verify passed both. The real signer handler signed. The real verifyArtifactAttestation and parseTrustedRoot verified. CLIs ran skilld run skilld-dev/skills/demo against a local site. The released 3.3.0, 3.6.1 and 3.6.4 binaries had only their compiled root public key swapped for the throwaway one. An unpatched 3.6.1 is the control.

Released binaries 3.0.0 to 3.6.4 all embed the live root key ID and public key, and no signing key. verify_signature and verify_trusted_root are byte-identical in every tag from v3.0.0-beta.0 to v3.6.4. Releases before 3.3.0 have no SKILLD_API_URL, so they did not run locally.

Signer

Statement	Slots	Result
created 2026-10-10	K1	K1
created 2026-10-20	K1 + K2	K2
created 10 s before the K2 window, signed after it	K1 + K2	K1
created 2026-11-21	K1 + K2	K2
created 2026-11-21	K1 only, today's config	`SIGNING_KEY_NOT_ACTIVE`
created 2026-11-21	K2 only	K2

Site verifier

Attestation	Root	Site time	Verifies
K1	v1	2026-10-10	yes
K1	v2	2026-10-20	yes
K2	v2	2026-10-20	yes
K2	v1	2026-10-20	no
K1	v2	2026-11-20T00:00Z	no, so the next request builds again
K2	v2	2026-11-20T00:00Z	yes
K1	v3	2026-11-13	no
K2	v3	2026-11-13	yes
K2 over a statement created before the K2 window (the old single-key signer at a cutover)	v2	2026-10-14	no (it was yes before this change)

CLI (3.3.0, 3.6.1, 3.6.4 patched, and a build of skilld main pinned to the throwaway root gave identical results)

Attestation	Root	All CLIs	Unpatched 3.6.1
K1	v1	runs	`TRUSTED_ROOT_MISMATCH`
K1	v2	runs	`TRUSTED_ROOT_MISMATCH`
K2	v2	runs	`TRUSTED_ROOT_MISMATCH`
K1, created 10 s before the K2 window	v2	runs	`TRUSTED_ROOT_MISMATCH`
K2	v1	`SIGNING_KEY_UNKNOWN`	`TRUSTED_ROOT_MISMATCH`
K2	v3	runs	`TRUSTED_ROOT_MISMATCH`
K1	v3	`SIGNING_KEY_UNKNOWN`	`TRUSTED_ROOT_MISMATCH`
K2 over a statement created before the K2 window	v2	`SIGNING_KEY_INVALID`	`TRUSTED_ROOT_MISMATCH`
K2, entry signed by another root key	v2	`TRUSTED_ROOT_SIGNATURE_INVALID`	`TRUSTED_ROOT_MISMATCH`

skilld.signing-key against this branch and the live root returned Pass with 43 days left. With the clock set to 2026-10-25 it returned Warn with 26 days left.

harlan-github-agent · 2026-10-08T00:46:46Z

🤖 MERGED

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. Last updated: 2026-10-08 10:27 UTC.

GitHub merged this pull request.

Logged (20/100): The ceremony verify command defaults its pin to the live root URL, letting a no-pin invocation pass a root whose key the released CLIs would all refuse. View code
- Ask an agent to verify and fix this finding
Logged (12/100): The daily signing-key check parses the .jsonc signer config as strict JSON, so a routine comment added to that config breaks the rotation warning check. View code
- Ask an agent to verify and fix this finding
Logged (5/100): The untrusted-signing-key failure evidence omits signingEndsAt/daysLeft that the check-in prompt tells the operator to quote. View code
- Ask an agent to verify and fix this finding

c314599a-b55e-4263-95b0-3bb3b3d480cd

Selected findings run after merge and open separate pull requests.

harlan-github-agent

Harlan Agent Kit posted this automated review. AI open source policy.

Automated Review: BLOCKED.

Read the Review results

harlan-github-agent

Harlan Agent Kit posted this automated review. AI open source policy.

Automated Review: READY.

Read the Review results

harlan-zw added 5 commits October 7, 2026 16:29

feat(checkin): warn 30 days before Artifact signing stops

6843cd2

skilld-production-2026-08 closes on 2026-11-20 and nothing watched it. The check reads the signer windows and the live trusted root, and finds when no key is both open and trusted.

docs(runbooks): add the signing key rotation runbook

d9df472

The rollout guide's rotation steps swapped the secret and key ID non-atomically and retired the old key with an unsigned status change.

harlan-zw added the harlan-agent-review Pull request triage requires an adversarial Review. Adding this label manually always forces one. label Oct 7, 2026

harlan-github-agent Bot added harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. and removed harlan-agent-review Pull request triage requires an adversarial Review. Adding this label manually always forces one. labels Oct 7, 2026

harlan-zw mentioned this pull request Oct 7, 2026

fix(signer): typecheck the signer with the site #557

Merged

1 of 6 tasks

harlan-github-agent Bot added the harlan-agent-running An Agent holds a Task on this issue or pull request right now. label Oct 8, 2026

harlan-github-agent Bot reviewed Oct 8, 2026

View reviewed changes

harlan-github-agent Bot added harlan-agent-running An Agent holds a Task on this issue or pull request right now. and removed harlan-agent-blocked The automated Review found a material defect in this head commit. labels Oct 8, 2026

docs(runbooks): make the signing key vars snippet valid JSONC

bc9cc65

harlan-github-agent Bot reviewed Oct 8, 2026

View reviewed changes

harlan-zw merged commit 0705d88 into main Oct 8, 2026
8 checks passed

harlan-github-agent Bot removed the harlan-agent-ready The automated Review passed every gate on this head commit. label Oct 8, 2026

Navigation Menu

Repository navigation

fix(artifacts): rotate the signing key without a gap - #556

fix(artifacts): rotate the signing key without a gap#556
harlan-zw merged 6 commits into
mainfrom
fix/signing-key-rotation

harlan-zw commented Oct 7, 2026

Uh oh!

harlan-zw commented Oct 7, 2026

Uh oh!

harlan-github-agent Bot commented Oct 8, 2026 •

edited

Loading

Uh oh!

harlan-github-agent Bot left a comment

Uh oh!

harlan-github-agent Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

harlan-zw commented Oct 7, 2026

🔗 Linked issue

❓ Type of change

📚 Description

Uh oh!

harlan-zw commented Oct 7, 2026

Uh oh!

harlan-github-agent Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 MERGED

Uh oh!

harlan-github-agent Bot left a comment

Choose a reason for hiding this comment

Uh oh!

harlan-github-agent Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

harlan-github-agent Bot commented Oct 8, 2026 •

edited

Loading