feat(identityhub): provision and rotate the participant identity from the chart - #173
Draft
vramperez wants to merge 20 commits into
Draft
feat(identityhub): provision and rotate the participant identity from the chart#173vramperez wants to merge 20 commits into
vramperez wants to merge 20 commits into
Conversation
… the chart
A deployed identityhub was inert: the chart had no template, Job or values key that
talks to the Identity API, writes the signing key to vault or inserts a credential.
The procedure was documented as a manual curl sequence in doc/DSP_INTEGRATION.md and
only automated inside the integration tests, so every real deployment reimplemented it
as a hand-written Job.
This adds it as a chart feature, keeping upstream's payloads and helper logic:
* derive-key.sh derives the JWK pair from the participant's identity key, EC (P-256
and P-384) and RSA. The algorithm is explicit because the failure mode is silent:
extracting an RSA modulus from an EC key yields a valid-looking JWK full of garbage
that only breaks later, at the counterparty.
* register.sh writes the key to vault, repairs the super-user credential that
SuperUserSeedExtension leaves in an unusable shape, registers the participant
context idempotently and provisions the STS client secret. That secret is chosen by
us rather than read from the creation response, which is the only way to make it
survive a restart, and it is written under both spellings of the alias because EDC's
vault client double-encodes the colons in a DID.
* check-kid.sh verifies the issuer signs with <did>#<keyAlias> and aborts otherwise,
printing the one-off repair. It cannot fix it: Keycloak never re-imports an existing
realm and `kid` is not a declared property of its java-keystore provider, so an
Admin API PUT answers 204 and drops it.
* insert-credential.sh publishes the credential into the identityhub's own store,
replacing a stale copy, and refuses a credential signed under a different kid.
* rotate-key.sh replaces the signing key, keeping the alias and the DID fragment.
The rotation Job is opt-in and never a hook: there is no overlap window, so each run
invalidates everything signed with the previous key and re-issuance is part of the
operation. Its order - revoke, overwrite the vault alias, add - is the only safe one:
the keypairs `rotate` endpoint called with the same keyId appends a second
verificationMethod under the same id, and revoking either one then removes both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…figMaps Two separate production blockers around the bundled vault. The token was rendered into ConfigMaps in clear - identityhub-cm.yaml did it with a `required`, so the chart could not even render an identityhub without a plaintext token in values - and anyone able to read ConfigMaps in the namespace could read the credential that unlocks the participant's signing key. vault.hashicorp.existingSecret now injects it as EDC_VAULT_HASHICORP_TOKEN instead, and the ConfigMap omits the property entirely in that case rather than leaving an empty one behind for EDC to resolve as an empty token. The vault itself ran in dev mode, which is in-memory: every restart of that pod loses the participant's signing key, the identityhub super-user credential, the STS client secret and the transfer signing key fdsc-edc generates for itself, taking the DCP lane down until the bootstrap Job runs again. Persistent storage alone does not fix it, because a vault with storage comes back sealed and refuses every request. So vault.production adds the missing half: initialise an empty vault once, store the unseal keys in a Secret, and unseal whenever the vault is found sealed. It is a Deployment and not a Job because the restart that needs it happens long after any deploy. The trade-off is written into values.yaml rather than hidden: the unseal keys live in a Kubernetes Secret, which is what makes an unattended restart recover on its own, and autoUnseal: false leaves them out of the cluster at the cost of a manual unseal. The Secret is annotated to survive uninstall, since deleting it while the storage volume exists leaves a vault nobody can open. Production mode also refuses to render alongside dev mode instead of silently doing nothing. The unsealer runs under its own ServiceAccount with a Role scoped to the single Secret it owns, so it cannot read the participant's identity key or its credentials. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eval.sh renders every chart with default values only, and identityhub, vault, fdsc-edc and vcCredentials are all disabled by default - so none of those templates was ever reached by `helm template` or kubeconform. A broken manifest in that whole area passed CI silently. Each chart's ci/*-values.yaml is now rendered and validated as well. The first one turns on the EDC lane, including the opt-in rotation Job and the branch that keeps the vault token out of the ConfigMap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each of these cost time in a production deployment and none of them is visible from a
default render, which is why they survived: with identityhub, vault and fdsc-edc all
disabled by default, none of this code is exercised.
Dangling references, i.e. templates that named objects nothing created:
* identityhub-custom-cacerts: declaring identityhub.customCaCerts mounted an empty
emptyDir over the JVM truststore, replacing the bundled roots with nothing instead of
extending them. Now a ConfigMap holds the extra CAs and an init container imports
them into a writable copy of the truststore - which is what makes a vault or an issuer
behind a private CA reachable at all.
* identityhub-pvc-logs: enabling logging.persistence rendered a Deployment whose pod
could never be scheduled.
The trusted-issuers registration was effectively install-only. It is annotated
post-install, so a release that already exists never runs it again, and the generated
script used `curl -f`, which turns the 409 of an already-registered issuer into a failed
Job - so making it run on upgrade would have failed every time. Both fixed: the script
now treats 409 as success and reports the status code, and the hook includes
post-upgrade. This is what kept a change to the credentials configuration from ever
reaching a running cluster.
Defaults that cannot work or should not be defaults:
* fdsc-edc tmfExtension was enabled with seven empty addresses. TMFConfig converts them
with URI.create(...).toURL() while the runtime boots, so the default configuration
aborted startup with "URI is not absolute". Off by default now, and the k3s values
that relied on the old default enable it explicitly.
* oid4vp.trustAll defaulted to true - disabling TLS verification, and the only
DO-NOT-USE-IN-PRODUCTION marker in the chart.
* edc.log.level defaulted to DEBUG, which logs every DSP message including token
contents.
Credential attributes in the realm:
* vc.expiry_in_seconds and vc.refresh_interval_in_seconds are now kept in sync. The
admin console shows the first while the issued `exp` comes from the second, so
setting only one produced credentials that expired at a completely different time
than the console claimed - with Keycloak's default, a week. Setting neither still
leaves Keycloak's defaults alone.
* vc.signing_key_id defaults to the key id the DID document actually publishes when the
identityhub serves that document. Keycloak otherwise defaults it to the bare DID,
which is not resolvable there, and issuance fails with "No key for id '<did>' and
algorithm ... available".
Also: the accounts port is opened by the deployment but was missing from the service, so
the STS Accounts API was unreachable in-cluster; and Chart.yaml asked for keycloak
0.21.7 while Chart.lock and the vendored archive are at 0.21.19, so the next
`helm dependency update` would have silently downgraded Keycloak.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…renewal The chart pushed the credential into the identityhub's credential store once, from the bootstrap Job, and nothing kept that copy in step afterwards. The vc-operator renewed the Secret on its own schedule while the identityhub went on presenting the previous credential until it expired, which took the DCP lane down with nothing having been deployed and a Secret that looked healthy. vcCredentials.requests[].identityHub.enabled hands that job to the vc-operator, which already knows when a credential is renewed. Every address defaults to what identityhub.bootstrap declares, so a participant provisioned by this chart needs nothing beyond the toggle. The api key has to come from a Secret, because that is what the operator reads; rendering fails with an explanation rather than emitting a dangling reference when only a plaintext identityhub.superuser.key is configured. Requires the vc-operator release that understands spec.identityHub. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The chart's only unit tests covered OpenTelemetry tracing, so nothing asserted
anything about the templates this branch adds or repairs. 26 tests over two
suites, aimed at the things that were actually wrong in real deployments rather
than at rendering in general:
* the bootstrap steps run in a guaranteed order, and the credential insert is a
regular container so a missing credential fails the Job instead of looping in
an init retry;
* the hook includes post-upgrade, which is what a dev-mode vault requires;
* the key id is derived from the DID and the alias in one place;
* the rotation Job carries no Helm hook and revokes before adding, never
calling the keypairs rotate endpoint - which appends a duplicate
verificationMethod under the same id;
* the vault token is in the ConfigMap by default, and omitted rather than
emptied when it comes from a Secret;
* a production vault refuses to render next to dev mode;
* the unsealer's Role is scoped to the single Secret it owns;
* the objects the deployment referenced while nothing created them - the
truststore ConfigMap with its import step, and the logs claim - now exist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DSP_INTEGRATION.md described the participant setup as a curl sequence to run from a workstation, which is why every deployment reimplemented it. The automated path is now the recommended one and the manual procedure stays as the local-deployment and debugging path, with the two traps that are easy to miss when doing it by hand written down: why the STS client secret is chosen rather than read from the creation response, and why it is stored under two spellings of the alias. A new production guide covers what a default install does not: a Vault that survives a restart with its unseal trade-off stated rather than hidden, a non-root token kept out of the ConfigMaps, the bootstrap values, keeping the identityhub's credential copy in step, pinning cert-manager's rotationPolicy instead of relying on a default upstream intends to change, and the runbook for rotating the signing key - including the step people skip, restarting everything that read the key at startup and the apisix that caches the JWKS under an unchanged kid. Also noted where the manual insert is described: a credential put there by hand is a copy that nothing keeps current, which is the failure that takes the DCP lane down without any deployment having happened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The EDC values render the vc-operator's CredentialIssuer and VerifiableCredentialRequest, whose CRDs are in no public catalog, and kubeconform -strict treats an unknown kind as an error - so the validation pass those values were added for would itself have failed CI. Everything with a published schema is still validated strictly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Aligning Chart.yaml with the lock removes the downgrade hazard but leaves the lock's digest stale, so `helm dependency build` still refuses. Worth saying plainly rather than letting the previous note read as if the drift were resolved: it comes from the bump automation, which updates Chart.yaml only, and CI never noticed because eval.sh ignores that command's exit status. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… the chart
A deployed identityhub was inert: the chart had no template, Job or values key that
talks to the Identity API, writes the signing key to vault or inserts a credential.
The procedure was documented as a manual curl sequence in doc/DSP_INTEGRATION.md and
only automated inside the integration tests, so every real deployment reimplemented it
as a hand-written Job.
This adds it as a chart feature, keeping upstream's payloads and helper logic:
* derive-key.sh derives the JWK pair from the participant's identity key, EC (P-256
and P-384) and RSA. The algorithm is explicit because the failure mode is silent:
extracting an RSA modulus from an EC key yields a valid-looking JWK full of garbage
that only breaks later, at the counterparty.
* register.sh writes the key to vault, repairs the super-user credential that
SuperUserSeedExtension leaves in an unusable shape, registers the participant
context idempotently and provisions the STS client secret. That secret is chosen by
us rather than read from the creation response, which is the only way to make it
survive a restart, and it is written under both spellings of the alias because EDC's
vault client double-encodes the colons in a DID.
* check-kid.sh verifies the issuer signs with <did>#<keyAlias> and aborts otherwise,
printing the one-off repair. It cannot fix it: Keycloak never re-imports an existing
realm and `kid` is not a declared property of its java-keystore provider, so an
Admin API PUT answers 204 and drops it.
* insert-credential.sh publishes the credential into the identityhub's own store,
replacing a stale copy, and refuses a credential signed under a different kid.
* rotate-key.sh replaces the signing key, keeping the alias and the DID fragment.
The rotation Job is opt-in and never a hook: there is no overlap window, so each run
invalidates everything signed with the previous key and re-issuance is part of the
operation. Its order - revoke, overwrite the vault alias, add - is the only safe one:
the keypairs `rotate` endpoint called with the same keyId appends a second
verificationMethod under the same id, and revoking either one then removes both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…figMaps Two separate production blockers around the bundled vault. The token was rendered into ConfigMaps in clear - identityhub-cm.yaml did it with a `required`, so the chart could not even render an identityhub without a plaintext token in values - and anyone able to read ConfigMaps in the namespace could read the credential that unlocks the participant's signing key. vault.hashicorp.existingSecret now injects it as EDC_VAULT_HASHICORP_TOKEN instead, and the ConfigMap omits the property entirely in that case rather than leaving an empty one behind for EDC to resolve as an empty token. The vault itself ran in dev mode, which is in-memory: every restart of that pod loses the participant's signing key, the identityhub super-user credential, the STS client secret and the transfer signing key fdsc-edc generates for itself, taking the DCP lane down until the bootstrap Job runs again. Persistent storage alone does not fix it, because a vault with storage comes back sealed and refuses every request. So vault.production adds the missing half: initialise an empty vault once, store the unseal keys in a Secret, and unseal whenever the vault is found sealed. It is a Deployment and not a Job because the restart that needs it happens long after any deploy. The trade-off is written into values.yaml rather than hidden: the unseal keys live in a Kubernetes Secret, which is what makes an unattended restart recover on its own, and autoUnseal: false leaves them out of the cluster at the cost of a manual unseal. The Secret is annotated to survive uninstall, since deleting it while the storage volume exists leaves a vault nobody can open. Production mode also refuses to render alongside dev mode instead of silently doing nothing. The unsealer runs under its own ServiceAccount with a Role scoped to the single Secret it owns, so it cannot read the participant's identity key or its credentials. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eval.sh renders every chart with default values only, and identityhub, vault, fdsc-edc and vcCredentials are all disabled by default - so none of those templates was ever reached by `helm template` or kubeconform. A broken manifest in that whole area passed CI silently. Each chart's ci/*-values.yaml is now rendered and validated as well. The first one turns on the EDC lane, including the opt-in rotation Job and the branch that keeps the vault token out of the ConfigMap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each of these cost time in a production deployment and none of them is visible from a
default render, which is why they survived: with identityhub, vault and fdsc-edc all
disabled by default, none of this code is exercised.
Dangling references, i.e. templates that named objects nothing created:
* identityhub-custom-cacerts: declaring identityhub.customCaCerts mounted an empty
emptyDir over the JVM truststore, replacing the bundled roots with nothing instead of
extending them. Now a ConfigMap holds the extra CAs and an init container imports
them into a writable copy of the truststore - which is what makes a vault or an issuer
behind a private CA reachable at all.
* identityhub-pvc-logs: enabling logging.persistence rendered a Deployment whose pod
could never be scheduled.
The trusted-issuers registration was effectively install-only. It is annotated
post-install, so a release that already exists never runs it again, and the generated
script used `curl -f`, which turns the 409 of an already-registered issuer into a failed
Job - so making it run on upgrade would have failed every time. Both fixed: the script
now treats 409 as success and reports the status code, and the hook includes
post-upgrade. This is what kept a change to the credentials configuration from ever
reaching a running cluster.
Defaults that cannot work or should not be defaults:
* fdsc-edc tmfExtension was enabled with seven empty addresses. TMFConfig converts them
with URI.create(...).toURL() while the runtime boots, so the default configuration
aborted startup with "URI is not absolute". Off by default now, and the k3s values
that relied on the old default enable it explicitly.
* oid4vp.trustAll defaulted to true - disabling TLS verification, and the only
DO-NOT-USE-IN-PRODUCTION marker in the chart.
* edc.log.level defaulted to DEBUG, which logs every DSP message including token
contents.
Credential attributes in the realm:
* vc.expiry_in_seconds and vc.refresh_interval_in_seconds are now kept in sync. The
admin console shows the first while the issued `exp` comes from the second, so
setting only one produced credentials that expired at a completely different time
than the console claimed - with Keycloak's default, a week. Setting neither still
leaves Keycloak's defaults alone.
* vc.signing_key_id defaults to the key id the DID document actually publishes when the
identityhub serves that document. Keycloak otherwise defaults it to the bare DID,
which is not resolvable there, and issuance fails with "No key for id '<did>' and
algorithm ... available".
Also: the accounts port is opened by the deployment but was missing from the service, so
the STS Accounts API was unreachable in-cluster.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…renewal The chart pushed the credential into the identityhub's credential store once, from the bootstrap Job, and nothing kept that copy in step afterwards. The vc-operator renewed the Secret on its own schedule while the identityhub went on presenting the previous credential until it expired, which took the DCP lane down with nothing having been deployed and a Secret that looked healthy. vcCredentials.requests[].identityHub.enabled hands that job to the vc-operator, which already knows when a credential is renewed. Every address defaults to what identityhub.bootstrap declares, so a participant provisioned by this chart needs nothing beyond the toggle. The api key has to come from a Secret, because that is what the operator reads; rendering fails with an explanation rather than emitting a dangling reference when only a plaintext identityhub.superuser.key is configured. Requires the vc-operator release that understands spec.identityHub. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The chart's only unit tests covered OpenTelemetry tracing, so nothing asserted
anything about the templates this branch adds or repairs. 26 tests over two
suites, aimed at the things that were actually wrong in real deployments rather
than at rendering in general:
* the bootstrap steps run in a guaranteed order, and the credential insert is a
regular container so a missing credential fails the Job instead of looping in
an init retry;
* the hook includes post-upgrade, which is what a dev-mode vault requires;
* the key id is derived from the DID and the alias in one place;
* the rotation Job carries no Helm hook and revokes before adding, never
calling the keypairs rotate endpoint - which appends a duplicate
verificationMethod under the same id;
* the vault token is in the ConfigMap by default, and omitted rather than
emptied when it comes from a Secret;
* a production vault refuses to render next to dev mode;
* the unsealer's Role is scoped to the single Secret it owns;
* the objects the deployment referenced while nothing created them - the
truststore ConfigMap with its import step, and the logs claim - now exist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DSP_INTEGRATION.md described the participant setup as a curl sequence to run from a workstation, which is why every deployment reimplemented it. The automated path is now the recommended one and the manual procedure stays as the local-deployment and debugging path, with the two traps that are easy to miss when doing it by hand written down: why the STS client secret is chosen rather than read from the creation response, and why it is stored under two spellings of the alias. A new production guide covers what a default install does not: a Vault that survives a restart with its unseal trade-off stated rather than hidden, a non-root token kept out of the ConfigMaps, the bootstrap values, keeping the identityhub's credential copy in step, pinning cert-manager's rotationPolicy instead of relying on a default upstream intends to change, and the runbook for rotating the signing key - including the step people skip, restarting everything that read the key at startup and the apisix that caches the JWKS under an unchanged kid. Also noted where the manual insert is described: a credential put there by hand is a copy that nothing keeps current, which is the failure that takes the DCP lane down without any deployment having happened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The EDC values render the vc-operator's CredentialIssuer and VerifiableCredentialRequest, whose CRDs are in no public catalog, and kubeconform -strict treats an unknown kind as an error - so the validation pass those values were added for would itself have failed CI. Everything with a published schema is still validated strictly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
vramperez
marked this pull request as draft
August 17, 2026 11:54
…a-space-connector into feat/edc-production-readiness
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A deployed identityhub was inert: the chart had no template, Job or values key that
talks to the Identity API, writes the signing key to vault or inserts a credential.
The procedure was documented as a manual curl sequence in doc/DSP_INTEGRATION.md and
only automated inside the integration tests, so every real deployment reimplemented it
as a hand-written Job.
This adds it as a chart feature, keeping upstream's payloads and helper logic:
and P-384) and RSA. The algorithm is explicit because the failure mode is silent:
extracting an RSA modulus from an EC key yields a valid-looking JWK full of garbage
that only breaks later, at the counterparty.
SuperUserSeedExtension leaves in an unusable shape, registers the participant
context idempotently and provisions the STS client secret. That secret is chosen by
us rather than read from the creation response, which is the only way to make it
survive a restart, and it is written under both spellings of the alias because EDC's
vault client double-encodes the colons in a DID.
printing the one-off repair. It cannot fix it: Keycloak never re-imports an existing
realm and
kidis not a declared property of its java-keystore provider, so anAdmin API PUT answers 204 and drops it.
replacing a stale copy, and refuses a credential signed under a different kid.
The rotation Job is opt-in and never a hook: there is no overlap window, so each run
invalidates everything signed with the previous key and re-issuance is part of the
operation. Its order - revoke, overwrite the vault alias, add - is the only safe one:
the keypairs
rotateendpoint called with the same keyId appends a secondverificationMethod under the same id, and revoking either one then removes both.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com