Skip to content

fix: generate a per-deploy HyperDX session secret - #530

Merged
catinspace-au merged 2 commits into
mainfrom
fix/hyperdx-session-secret
Oct 6, 2026
Merged

catinspace-au merged 2 commits into
mainfrom
fix/hyperdx-session-secret

Conversation

@catinspace-au

@catinspace-au catinspace-au commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Two HyperDX keys the chart never set.

HyperDX falls back to a session secret published in upstream's source when EXPRESS_SESSION_SECRET is unset (packages/api/src/config.ts:16 in our fork, upstream 2.40.0), so every Kubernetes deployment signed HyperDX sessions with the same public key. And without TOKEN_ENCRYPTION_KEY it stores third-party tokens (Slack bot tokens, OAuth tokens) in plain text.

  • templates/generated-keys.yaml: one ESO Password generator plus ExternalSecret per key.
    • dfe-hyperdx-session / session-secret: the 48-char alphanumeric password itself.
    • dfe-hyperdx-token-encryption / token-encryption-key: {{ .password | sha256sum }}, 64 hex chars. The fork takes 64 hex chars or base64 of exactly 32 bytes and refuses to start on anything else (packages/api/src/utils/tokenEncryption.ts:150). ESO 2.10.0's template sha256sum returns hex.EncodeToString(sha256.Sum256(...)) (runtime/template/v2/sprig/crypto.go).
  • The pod reads both through secretKeyRef.
  • Stable across Argo syncs: refreshPolicy: CreatedOnce + refreshInterval: "0", so ESO writes each value once and a re-render or re-label never rotates it. Nothing is minted at render time, so helm template output is byte-identical every time. helm.sh/resource-policy: keep stops an uninstall taking either key with it. Rotation matters most for the token key: a new one makes every stored token unreadable.
  • sessionSecret.create: false / tokenEncryption.create: false hand the Secret to the deployment; the pod still reads it.
  • ESO is already a prerequisite of every mode that runs hyperdx (ferretdb's password and the engine JWT use the same generator).
  • dfe-stack's README and NOTES list both keys among the ESO-generated secrets.

Tests: test_render_stable.py covers both keys -- the default renders the generator and a CreatedOnce ExternalSecret with the exact value template, renders identically twice and wires the pod to it; create=false renders neither and the pod still reads the Secret.

The pinned HyperDX image (v0.2.7) reads EXPRESS_SESSION_SECRET today. It predates tokenEncryption.ts, so TOKEN_ENCRYPTION_KEY takes effect from the first fork release that carries it.

The compose half is hyperi-io/dfe-docker#204.

Done when CI is green and a fresh deploy's hyperdx pod starts with both keys from their minted Secrets.

HyperDX signs its session cookie with a key published in upstream's source whenever EXPRESS_SESSION_SECRET is unset, and the hyperdx chart never set it. So every Kubernetes deployment shared one key anyone can read.

The chart now generates dfe-hyperdx-session in-cluster with ESO's Password generator (48 alphanumeric chars) and the pod reads it through a secretKeyRef. Same mechanism as the engine JWT key: refreshPolicy CreatedOnce writes it once, nothing is minted at render time, so an Argo re-sync never rotates it and never ends a session. sessionSecret.create=false hands the Secret to the deployment.

test_render_stable.py covers both paths.
HyperDX stores third-party tokens (Slack bot tokens, OAuth tokens) in plain text unless TOKEN_ENCRYPTION_KEY is set, and the chart never set it. It now generates dfe-hyperdx-token-encryption in-cluster beside the session key, and the pod reads it through a secretKeyRef.

The fork takes 64 hex chars or base64 of exactly 32 bytes and refuses to start on anything else (packages/api/src/utils/tokenEncryption.ts:150), so the key is the sha256sum of an ESO-generated password: 64 hex chars. CreatedOnce writes it once; a rotated key would make every stored token unreadable.

The template is renamed generated-keys.yaml to hold both keys, and dfe-stack's README and NOTES list them among the ESO-generated secrets.
@catinspace-au
catinspace-au merged commit 47e0af8 into main Oct 6, 2026
8 checks passed
@catinspace-au
catinspace-au deleted the fix/hyperdx-session-secret branch October 6, 2026 00:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant