fix: mint a per-deploy HyperDX session secret

.env.example

@@ -320,6 +320,13 @@
 ##   HYPERDX_POSTGRES_PASSWORD - HyperDX metadata store. Not published to the
 ##                              host, so smaller blast radius, but a password
 ##                              whose value is also its name is not one.
+##   HYPERDX_EXPRESS_SESSION_SECRET - signs HyperDX's session cookie. Unset,
+##                              HyperDX falls back to a key published in its
+##                              upstream source.
+##   HYPERDX_TOKEN_ENCRYPTION_KEY - encrypts the third-party tokens HyperDX
+##                              stores. 64 hex chars. Its fallback is empty
+##                              (plain text), not a sentinel, because HyperDX
+##                              refuses to start on a malformed key.
 ##   CLICKHOUSE_PASSWORD       - ClickHouse default user, which has full admin
 ##                              (DEFAULT_ACCESS_MANAGEMENT=1). Was blank. BREAKING
 ##                              on upgrade - a data volume created with the blank
@@ -691,3 +698,8 @@
 #
 ## HYPERDX_POSTGRES_PASSWORD is generated by `make init` - see "Generated
 ## secrets" above. Deliberately not given a placeholder here.
+#
+## Signs HyperDX's session cookie; `make init` generates it, see "Generated secrets".
+# HYPERDX_EXPRESS_SESSION_SECRET=
+## Encrypts the third-party tokens HyperDX stores; `make init` generates it.
+# HYPERDX_TOKEN_ENCRYPTION_KEY=

README.md

@@ -67,10 +67,11 @@ This creates `.env` from `.env.example` and `env/<service>.env` for every templa
 Re-running `make init` is safe - existing files are never overwritten. It does two
 things beyond the copy:
 
-- **Generates secrets.** A random value is minted for `DFE_UI_NEXTAUTH_SECRET` and
-  `HYPERDX_POSTGRES_PASSWORD`. An existing `.env` that predates a key is topped up,
-  so upgrading does not break your checkout. Compose carries a sentinel default for
-  both rather than hard-failing (a hard-fail would abort `make down` too); the
+- **Generates secrets.** A random value is minted for `DFE_UI_NEXTAUTH_SECRET`,
+  `HYPERDX_POSTGRES_PASSWORD`, `HYPERDX_EXPRESS_SESSION_SECRET` and
+  `HYPERDX_TOKEN_ENCRYPTION_KEY`. An existing `.env` that predates a key is topped up,
+  so upgrading does not break your checkout. Compose carries a sentinel (or empty)
+  default for each rather than hard-failing (a hard-fail would abort `make down` too); the
   power-on self test is what refuses to pass while a default is still in place.
 - **Reports drift.** The copy is one-shot, so an `.env` created months ago never
   learns that `.env.example` grew a setting. A re-run lists the keys you are

docker-compose.yml

@@ -1595,6 +1595,13 @@ services:
       NEXT_PUBLIC_DFE_UI_BASE_URL: ${DFE_HYPERDX_APP_URL:-${DFE_EXTERNAL_ORIGIN:-http://localhost}}:${DFE_UI_PORT:-3000}
       # HyperDX talks Mongo wire protocol; FerretDB serves it over Postgres.
       MONGO_URI: mongodb://${HYPERDX_POSTGRES_USER:-hyperdx}:${HYPERDX_POSTGRES_PASSWORD:-hyperdx}@hyperdx-ferretdb:27017/hyperdx?authSource=admin
+      # Unset, HyperDX signs sessions with a key published upstream, so `make init`
+      # generates one. Not a `:?` hard-fail -- see NEXTAUTH_SECRET.
+      EXPRESS_SESSION_SECRET: ${HYPERDX_EXPRESS_SESSION_SECRET:-RUN-make-init-TO-GENERATE-A-REAL-SECRET}
+      # Encrypts stored third-party tokens; `make init` generates 64 hex chars. The
+      # fallback is empty (plain text), not a sentinel: HyperDX refuses to start on
+      # a malformed key.
+      TOKEN_ENCRYPTION_KEY: ${HYPERDX_TOKEN_ENCRYPTION_KEY:-}
       NEXT_PUBLIC_THEME: ${HYPERDX_THEME:-dfe}
     volumes:
       - ./config/hyperdx/default-sources.json:/etc/hyperdx/default-sources.json:ro

docs/configuration.md

@@ -324,6 +324,8 @@ The same toggle points the engine at HyperDX: with it on, the engine receives `D
 | `HYPERDX_THEME`                                  | UI theme (NEXT_PUBLIC_THEME)                                                         | `dfe`                                                               |
 | `HYPERDX_POSTGRES_USER`                          | FerretDB/Postgres user                                                               | `hyperdx`                                                           |
 | `HYPERDX_POSTGRES_PASSWORD`                      | FerretDB/Postgres password                                                           | `hyperdx`                                                           |
+| `HYPERDX_EXPRESS_SESSION_SECRET`                 | Signs HyperDX's session cookie (`EXPRESS_SESSION_SECRET`)                            | generated by `make init`; sentinel otherwise, which `make post` fails |
+| `HYPERDX_TOKEN_ENCRYPTION_KEY`                   | Encrypts the third-party tokens HyperDX stores (`TOKEN_ENCRYPTION_KEY`), 64 hex chars | generated by `make init`; empty (plain text) otherwise, which `make post` fails |
 | `HYPERDX_FERRETDB_VERSION`                       | FerretDB image version                                                               | none -- `make stack` pins it from the DFE stack SSoT; unset is a hard-fail                |
 | `HYPERDX_POSTGRES_VERSION`                       | Postgres/DocumentDB image version                                                    | none -- `make stack` pins it from the DFE stack SSoT; unset is a hard-fail                |
 

docs/developing.md

@@ -44,8 +44,8 @@ make up                    # pull and start, self test, then print the login
 to `env/<service>.env`. Existing files are never overwritten, so it is safe to
 re-run. It does two things beyond the copy:
 
-- **Generates secrets.** `DFE_UI_NEXTAUTH_SECRET` and `HYPERDX_POSTGRES_PASSWORD`
-  get a random value. An existing `.env` that predates a key is topped up.
+- **Generates secrets.** `DFE_UI_NEXTAUTH_SECRET`, `HYPERDX_POSTGRES_PASSWORD`,
+  `HYPERDX_EXPRESS_SESSION_SECRET` and `HYPERDX_TOKEN_ENCRYPTION_KEY` get a random value. An existing `.env` that predates a key is topped up.
 - **Reports drift.** The copy is one-shot, so a `.env` made months ago never
   learns that `.env.example` grew a setting. A re-run lists the keys yours is
   missing and stops there -- editing your `.env` is yours to do.

docs/operating.md

@@ -346,10 +346,11 @@ because Compose interpolates every service before profiles filter anything. A
 Redpanda pin in `.env` is not a Redpanda deployment -- but if your licence
 position requires zero reference to the artefact, that is the remaining edge.
 
-## Secrets: three are generated
+## Secrets: generated by make init
 
 `make init` mints a random value for `DFE_UI_NEXTAUTH_SECRET`,
-`HYPERDX_POSTGRES_PASSWORD` and `CLICKHOUSE_PASSWORD`, including topping up an
+`HYPERDX_POSTGRES_PASSWORD`, `HYPERDX_EXPRESS_SESSION_SECRET`,
+`HYPERDX_TOKEN_ENCRYPTION_KEY` and `CLICKHOUSE_PASSWORD`, including topping up an
 existing `.env` that predates any of them (`scripts/init.py`). Compose carries a
 sentinel (or empty) default rather than a `${VAR:?}` hard-fail -- interpolation is
 not profile-gated, so a hard-fail would abort `make down` too, for a service the
@@ -462,8 +463,9 @@ nobody leaves enabled.
 
 The other outcomes are as informative as the PASS:
 
-- **FAIL on weak secrets** -- `DFE_UI_NEXTAUTH_SECRET` or
-  `HYPERDX_POSTGRES_PASSWORD` is still the committed default. Run `make init`.
+- **FAIL on weak secrets** -- `DFE_UI_NEXTAUTH_SECRET`, `HYPERDX_POSTGRES_PASSWORD`,
+  `HYPERDX_EXPRESS_SESSION_SECRET` or `HYPERDX_TOKEN_ENCRYPTION_KEY` is still the
+  committed default. Run `make init`.
 - **FAIL, schema not converged** -- dfe-engine did not report its `schema` readiness check true within 300 seconds. `GET /api/v1/system/schema` on the engine names the object that failed.
 - **SKIP, schema convergence not checked** -- the engine names no `schema` check on `/readyz` and answers 404 on `/api/v1/system/schema`, which is every engine before v1.21.0. The rest of the run still asserts the rows land.
 - **FAIL, not ready** -- the profile declares an ingest component that never

scripts/init.py

@@ -46,7 +46,8 @@
 )
 
 # Secrets that must not be left at their weak/empty default. scripts/post.py
-# enforces them: DFE_UI_NEXTAUTH_SECRET and HYPERDX_POSTGRES_PASSWORD via its
+# enforces them: DFE_UI_NEXTAUTH_SECRET, HYPERDX_POSTGRES_PASSWORD,
+# HYPERDX_EXPRESS_SESSION_SECRET and HYPERDX_TOKEN_ENCRYPTION_KEY via its
 # WEAK_SECRET_DEFAULTS service-map, CLICKHOUSE_PASSWORD via its own external-CH
 # check (its default is empty, not a sentinel string), and
 # DFE_AUTH_LOCAL_ADMIN_PASSWORD by logging in with it. Keep them in step.
@@ -76,18 +77,24 @@
 # or 32 bytes. The key tier's 48 is rejected outright, so this one is its own
 # length rather than a tier -- 32 for AES-256, the strongest of the three.
 _LEN_COOKIE = 32
+# HyperDX takes TOKEN_ENCRYPTION_KEY as 64 hex chars or base64 of exactly 32 bytes
+# and refuses to start on anything else; hex keeps it alphanumeric.
+_LEN_HEX_AES256 = 64
+_HEX_SECRETS = frozenset({"HYPERDX_TOKEN_ENCRYPTION_KEY"})
 
 # Name -> length tier. DB passwords are the 128-bit credential tier; the dfe-ui
-# NextAuth and engine JWT values are session-signing KEYS, so they take the
-# 256-bit tier.
+# NextAuth, engine JWT and HyperDX session values are session-signing KEYS, so
+# they take the 256-bit tier.
 GENERATED_SECRETS = {
     "CLICKHOUSE_PASSWORD": _LEN_CREDENTIAL,
     "DFE_AUTH_BREAKGLASS_PASSWORD": _LEN_CREDENTIAL,
     "DFE_AUTH_LOCAL_ADMIN_PASSWORD": _LEN_CREDENTIAL,
     "HYPERDX_POSTGRES_PASSWORD": _LEN_CREDENTIAL,
     "DFE_API_JWT_SECRET": _LEN_KEY,
     "DFE_UI_NEXTAUTH_SECRET": _LEN_KEY,
+    "HYPERDX_EXPRESS_SESSION_SECRET": _LEN_KEY,
     "DFE_OAUTH2_PROXY_COOKIE_SECRET": _LEN_COOKIE,
+    "HYPERDX_TOKEN_ENCRYPTION_KEY": _LEN_HEX_AES256,
 }
 
 # Matches a dotenv assignment: live (`KEY=value`) or a single-hash commented-out
@@ -120,6 +127,14 @@ def _generate_secret(length: int) -> str:
     return "".join(secrets.choice(_SECRET_ALPHABET) for _ in range(length))
 
 
+def _mint(key: str) -> str:
+    """Return a fresh value for one GENERATED_SECRETS key, in the format its consumer parses."""
+    length = GENERATED_SECRETS[key]
+    if key in _HEX_SECRETS:
+        return secrets.token_hex(length // 2)
+    return _generate_secret(length)
+
+
 def _setting_key(*, line: str) -> str | None:
     """Return the dotenv key a line assigns, commented or not - None if it assigns nothing."""
     match = _SETTING_RE.match(line)
@@ -157,7 +172,7 @@ def _render_secrets(*, text: str) -> str:
         key = _setting_key(line=line)
         if key in pending:
             pending.discard(key)
-            rendered.append(f"{key}={_generate_secret(GENERATED_SECRETS[key])}")
+            rendered.append(f"{key}={_mint(key)}")
             continue
         rendered.append(line)
 
@@ -166,10 +181,7 @@ def _render_secrets(*, text: str) -> str:
     if pending:
         rendered.append("")
         rendered.append("## Generated by `make init` - keep out of version control.")
-        rendered.extend(
-            f"{key}={_generate_secret(GENERATED_SECRETS[key])}"
-            for key in sorted(pending)
-        )
+        rendered.extend(f"{key}={_mint(key)}" for key in sorted(pending))
     return "\n".join(rendered) + "\n"
 
 
@@ -264,7 +276,7 @@ def _top_up_secrets(*, dotenv_path: Path) -> None:
         key = _setting_key(line=line)
         if key in pending and not (line.lstrip().startswith("#")):
             pending.discard(key)
-            rewritten.append(f"{key}={_generate_secret(GENERATED_SECRETS[key])}")
+            rewritten.append(f"{key}={_mint(key)}")
             continue
         rewritten.append(line)
 
@@ -273,10 +285,7 @@ def _top_up_secrets(*, dotenv_path: Path) -> None:
         rewritten.append(
             "## Generated by `make init` - the power-on self test fails without these."
         )
-        rewritten.extend(
-            f"{key}={_generate_secret(GENERATED_SECRETS[key])}"
-            for key in sorted(pending)
-        )
+        rewritten.extend(f"{key}={_mint(key)}" for key in sorted(pending))
 
     write_private(path=dotenv_path, text="\n".join(rewritten) + "\n", mode=DOTENV_MODE)
     _print(

scripts/post.py

@@ -307,6 +307,12 @@
 WEAK_SECRET_DEFAULTS = {
     "DFE_UI_NEXTAUTH_SECRET": ("RUN-make-init-TO-GENERATE-A-REAL-SECRET", "dfe-ui"),
     "HYPERDX_POSTGRES_PASSWORD": ("hyperdx", "hyperdx"),
+    "HYPERDX_EXPRESS_SESSION_SECRET": (
+        "RUN-make-init-TO-GENERATE-A-REAL-SECRET",
+        "hyperdx",
+    ),
+    # Empty, not a sentinel: HyperDX refuses to start on a malformed key.
+    "HYPERDX_TOKEN_ENCRYPTION_KEY": ("", "hyperdx"),
 }
 
 

scripts/tests/test_hyperdx.py

@@ -15,6 +15,8 @@
 console never gets its HyperDX view, and the engine answers 503 `hyperdx_absent`.
 HyperDX's own ClickHouse connection has to follow the engine's too, or an external
 ClickHouse leaves every HyperDX query aimed at a container that is not running.
+Its session signing key comes from `make init`, or the fork signs with a key
+published upstream; so does its token encryption key, in a format it parses.
 
 Read the way compose reads it: the `.profile.mk` values resolve_profile writes for
 a profile, expanded over the `${NAME:-default}` forms docker-compose.yml uses.
@@ -28,11 +30,17 @@
 
 import pytest
 
+import init
+import post
 import resolve_profile
 from _common import COMPOSE_FILE, REPO_ROOT
 
 _ENGINE_SERVICE = "dfe-engine"
 _HYPERDX_SERVICE = "hyperdx"
+_SESSION_KEY = "HYPERDX_EXPRESS_SESSION_SECRET"
+_TOKEN_KEY = "HYPERDX_TOKEN_ENCRYPTION_KEY"
+# The hex form of the two the fork's tokenEncryption.ts parseEncryptionKey accepts.
+_HEX_AES256 = re.compile(r"[0-9a-f]{64}")
 _IN_STACK_HYPERDX = "http://hyperdx:8000"
 _JWKS_URL = "http://dfe-engine:8000/.well-known/jwks.json"
 
@@ -266,6 +274,63 @@ def test_a_developer_with_no_engine_can_still_select_header_dev() -> None:
     )
 
 
+def test_hyperdx_signs_sessions_with_the_key_make_init_mints() -> None:
+    """Unset, the fork signs sessions with a key published in upstream's source."""
+    template = _service_environment(service=_HYPERDX_SERVICE)["EXPRESS_SESSION_SECRET"]
+    minted = init._mint(_SESSION_KEY)
+
+    assert init.GENERATED_SECRETS[_SESSION_KEY] >= 32
+    assert _expand(template=template, values={_SESSION_KEY: minted}) == minted
+    # The fallback compose runs on is exactly what `make post` refuses on HyperDX.
+    assert post.WEAK_SECRET_DEFAULTS[_SESSION_KEY] == (
+        _expand(template=template, values={}),
+        _HYPERDX_SERVICE,
+    )
+
+
+def test_the_minted_token_key_is_one_hyperdx_parses() -> None:
+    """HyperDX refuses to start unless the key is 64 hex chars or base64 of 32 bytes."""
+    minted = init._mint(_TOKEN_KEY)
+
+    assert _HEX_AES256.fullmatch(minted)
+    assert len(bytes.fromhex(minted)) == 32
+
+
+def test_hyperdx_encrypts_tokens_with_the_key_make_init_mints() -> None:
+    template = _service_environment(service=_HYPERDX_SERVICE)["TOKEN_ENCRYPTION_KEY"]
+    minted = init._mint(_TOKEN_KEY)
+
+    assert _expand(template=template, values={_TOKEN_KEY: minted}) == minted
+    # Empty reads as encryption off; a sentinel would read as a malformed key.
+    assert _expand(template=template, values={}) == ""
+    assert post.WEAK_SECRET_DEFAULTS[_TOKEN_KEY] == ("", _HYPERDX_SERVICE)
+
+
+@pytest.mark.parametrize("key", [_SESSION_KEY, _TOKEN_KEY])
+def test_an_env_that_predates_a_hyperdx_key_gets_one_and_keeps_it(
+    dotenv: Path, key: str
+) -> None:
+    older = "".join(
+        f"{other}=already-minted-value\n"
+        for other in init.GENERATED_SECRETS
+        if other != key
+    )
+    dotenv.write_text(older, encoding="utf-8", newline="\n")
+
+    init._top_up_secrets(dotenv_path=dotenv)
+    topped_up = dotenv.read_text(encoding="utf-8")
+    init._top_up_secrets(dotenv_path=dotenv)
+
+    assert topped_up.startswith(older)
+    minted = [
+        line.partition("=")[2]
+        for line in topped_up.splitlines()
+        if line.startswith(f"{key}=")
+    ]
+    assert [len(value) for value in minted] == [init.GENERATED_SECRETS[key]]
+    assert dotenv.read_text(encoding="utf-8") == topped_up
+
+
 def test_nothing_stamps_an_identity_hyperdx_no_longer_reads() -> None:
     texts = {
         name: (REPO_ROOT / name).read_text(encoding="utf-8") for name in _PROXY_CONFIGS