Repository navigation
fix: generate a per-deploy HyperDX session secret - #530
Merged
Merged
Conversation
HyperDX signs its session cookie with a key published in upstream's source whenever EXPRESS_SESSION_SECRET is unset, and the hyperdx chart never set it. So every Kubernetes deployment shared one key anyone can read. The chart now generates dfe-hyperdx-session in-cluster with ESO's Password generator (48 alphanumeric chars) and the pod reads it through a secretKeyRef. Same mechanism as the engine JWT key: refreshPolicy CreatedOnce writes it once, nothing is minted at render time, so an Argo re-sync never rotates it and never ends a session. sessionSecret.create=false hands the Secret to the deployment. test_render_stable.py covers both paths.
HyperDX stores third-party tokens (Slack bot tokens, OAuth tokens) in plain text unless TOKEN_ENCRYPTION_KEY is set, and the chart never set it. It now generates dfe-hyperdx-token-encryption in-cluster beside the session key, and the pod reads it through a secretKeyRef. The fork takes 64 hex chars or base64 of exactly 32 bytes and refuses to start on anything else (packages/api/src/utils/tokenEncryption.ts:150), so the key is the sha256sum of an ESO-generated password: 64 hex chars. CreatedOnce writes it once; a rotated key would make every stored token unreadable. The template is renamed generated-keys.yaml to hold both keys, and dfe-stack's README and NOTES list them among the ESO-generated secrets.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two HyperDX keys the chart never set.
HyperDX falls back to a session secret published in upstream's source when
EXPRESS_SESSION_SECRETis unset (packages/api/src/config.ts:16in our fork, upstream 2.40.0), so every Kubernetes deployment signed HyperDX sessions with the same public key. And withoutTOKEN_ENCRYPTION_KEYit stores third-party tokens (Slack bot tokens, OAuth tokens) in plain text.templates/generated-keys.yaml: one ESO Password generator plus ExternalSecret per key.dfe-hyperdx-session/session-secret: the 48-char alphanumeric password itself.dfe-hyperdx-token-encryption/token-encryption-key:{{ .password | sha256sum }}, 64 hex chars. The fork takes 64 hex chars or base64 of exactly 32 bytes and refuses to start on anything else (packages/api/src/utils/tokenEncryption.ts:150). ESO 2.10.0's templatesha256sumreturnshex.EncodeToString(sha256.Sum256(...))(runtime/template/v2/sprig/crypto.go).secretKeyRef.refreshPolicy: CreatedOnce+refreshInterval: "0", so ESO writes each value once and a re-render or re-label never rotates it. Nothing is minted at render time, sohelm templateoutput is byte-identical every time.helm.sh/resource-policy: keepstops an uninstall taking either key with it. Rotation matters most for the token key: a new one makes every stored token unreadable.sessionSecret.create: false/tokenEncryption.create: falsehand the Secret to the deployment; the pod still reads it.Tests:
test_render_stable.pycovers both keys -- the default renders the generator and a CreatedOnce ExternalSecret with the exact value template, renders identically twice and wires the pod to it;create=falserenders neither and the pod still reads the Secret.The pinned HyperDX image (
v0.2.7) readsEXPRESS_SESSION_SECRETtoday. It predatestokenEncryption.ts, soTOKEN_ENCRYPTION_KEYtakes effect from the first fork release that carries it.The compose half is hyperi-io/dfe-docker#204.
Done when CI is green and a fresh deploy's hyperdx pod starts with both keys from their minted Secrets.