helm: make the Postgres role and bootstrap database configurable for validators

[SLoeuillet]

Fixes #6802. Scope reduced to validator and participant per the discussion below — the SV charts and splitwell move to a follow-up.

Why

splice-postgres is unsupported after 2026-11-12, so every operator on it has to move to a self-provisioned or managed Postgres. The charts assumed that migration target would offer a cnadmin superuser and a cantonnet database, which a managed offering may not permit creating at all — so the names blocked the very migration path upstream points at.

persistence.user and persistence.bootstrapDatabaseName are now read wherever those values were hardcoded, defaulting to cnadmin and cantonnet so rendered output is byte-identical unless you set them.

The part that was not just the init container

The participant needed more than the pg-init fix. canton-base's storage.conf reads user = ${?CANTON_PARTICIPANT_POSTGRES_USER}, and the chart never set it — so the node kept connecting as cnadmin even with persistence.user configured. That is a half-applied configuration: the init container uses the new role, the application does not, and nothing complains. Worse than not offering the value, since it looks configurable.

The image already supports the variable and the compose deployment already passes it (CANTON_PARTICIPANT_POSTGRES_USER=${SPLICE_DB_USER}); Helm was the only deployment path missing it. splice-validator was never affected — it writes user straight into its HOCON storage block.

Testing

Helm deployment on a kind cluster, against a Postgres initialised with POSTGRES_USER=appuser / POSTGRES_DB=bootstrapdb, so cnadmin and cantonnet do not exist:

• participant 1/1 Running — participant_test owner=appuser, CANTON_PARTICIPANT_POSTGRES_USER=appuser, 18 Flyway migrations, Canton started, and Postgres reports appuser -> participant_test (32 conn)
• validator app completes its DB work — validator_test owner=appuser, schema validator owner=appuser — then stops at the ledger-API wiring, which was deliberately not configured
• negative control: the unpatched charts with identical values loop on FATAL: password authentication failed for user "cnadmin" at Init:0/1, with no user env var rendered at all

Chart unit tests cover the defaults, an override, and that neither default survives in the rendered command; each was checked to fail without the change rather than assumed.

Not in this PR

splice-sv-node, splice-scan, splice-global-domain (sequencer + mediator), splice-domain and splice-splitwell-app carry the same hardcoding and the same fix, but SV deployments have more moving parts and the cluster test coverage for them is not there yet. That work is preserved and will follow once it is.

[isegall-da]

Thanks @SLoeuillet this seems sensible to me. Can you please add a release notes entry in release_notes_upcoming.rst please?

[isegall-da]

/cluster_test

[github-actions]

Deploy cluster test triggered for Commit 8922982c3fc846f1de44ca301cc5ed80fa47f146 in , please contact a Contributor to approve it in CircleCI: https://app.circleci.com/pipelines/github/DACH-NY/canton-network-internal/79923

[SLoeuillet]

Thanks @isegall-da — release notes entry added in f62c546, under a new Helm section in docs/src/release_notes_upcoming.rst.

I rebased onto main first: the branch was 44 commits behind, and release_notes_upcoming.rst had been reset in the meantime by #6888 (Clear release notes for 0.7.4), so an entry written against the old file would have landed on content that no longer exists. The rebase was clean and the functional diff is unchanged — still the same 18 files, +59/−12.

One consequence: the head commit moved from 8922982c to f62c546, so the /cluster_test you triggered is against the old commit and will need re-running when convenient.

[github-actions]

Deploy cluster test triggered for Commit f62c54648dae06d3835dc8abafa5ba8a79fade81 in , please contact a Contributor to approve it in CircleCI: https://app.circleci.com/pipelines/github/DACH-NY/canton-network-internal/79972

[martinflorian-da]

Thank you @SLoeuillet !

I think I would feel slightly better if we could add 1-2 helm tests (perhaps for validator + participant?) that cover the new value, to guard against regressions.

More on the nit side: we could also cover splitwell.yaml, for symmetry... (Really we should think about some larger refactoring that avoids all this code duplication, but doesn't have to be in this PR.)

Also please rebase to latest main again to fix your merge conflict and consume the fix for the reason your deployment test failed... sorry for the trouble.

[martinflorian-da]

Argh apologies for missing that earlier, we should also fix the sequencer:

> rg "\-\-dbname=cantonnet" cluster/helm/
cluster/helm/splice-splitwell-app/templates/splitwell.yaml
128:              until errmsg=$(psql -h {{ .Values.persistence.host }} -p {{ .Values.persistence.port }} --username=cnadmin --dbname=cantonnet -c 'create database {{ .Values.persistence.databaseName }}' 2>&1); do

cluster/helm/splice-global-domain/templates/sequencer.yaml
266:              until psql -h "$DB_HOST" -p "$DB_PORT" --username="$DB_USER" --dbname=cantonnet -tAc 'SELECT 1' >/dev/null 2>&1; do
270:              if [ "$(psql -h "$DB_HOST" -p "$DB_PORT" --username="$DB_USER" --dbname=cantonnet -tAc "SELECT 1 FROM pg_database WHERE datname = '${DB_NAME}'")" = "1" ]; then
274:              psql -v ON_ERROR_STOP=1 -h "$DB_HOST" -p "$DB_PORT" --username="$DB_USER" --dbname=cantonnet -c "CREATE DATABASE \"${DB_NAME}\"";

And even bigger miss: for participant/domain/mediator/sequencer persistence.user is not actually used for anything but the init container atm? You want to be overriding POSTGRES_USER somewhere: https://github.com/digital-asset/canton/blob/72c82f382942e3bda7d6418f72f95b1524f57ca8/docker/canton/images/canton-base/storage.conf#L9

Given that both @isegall-da and myself missed these things on the first pass makes me want to ask you for some e2e confirmation on a cluster that this is working as expected, with dbname overridden... Did you try these changes out?

[SLoeuillet]

Thanks — all three addressed, and good catch on the sequencer.

1. The missed call sites. Your rg found two, and both are now parameterized:

• splice-global-domain/templates/sequencer.yaml — the cantonbft-driver pg-init container. It already derived host, port and user from the driver's persistence block with a fallback to sequencer.persistence, so I gave the bootstrap database the same treatment rather than inlining a template expression into the shell script: a $dbBootstrap variable exposed as DB_BOOTSTRAP, used by all three psql invocations.
• splice-splitwell-app/templates/splitwell.yaml — had both the role and the bootstrap database hardcoded; now reads persistence.user and persistence.bootstrapDatabaseName like the other charts, and I documented the two values in its values-template.yaml.

rg -- '--dbname=cantonnet|--username=cnadmin' cluster/helm/ now returns nothing.

2. Tests. Added a postgres identity suite to both splice-validator and splice-participant, two tests each: one asserting the cnadmin/cantonnet defaults, one asserting an override propagates and that neither default survives in the rendered command. That second half is what actually guards against a regression — a new call site that forgets the values would still pass a defaults-only test.

One wrinkle worth mentioning: the obvious negative assertion (notMatchRegex: cnadmin|cantonnet) is wrong, because the target database is cantonnet_validator and legitimately contains the string. The assertion has to be anchored on the flags.

3. Rebased onto 6ab535819. The only conflict was release_notes_upcoming.rst, resolved by taking the current upstream file and re-appending the Helm entry.

Test suites for every touched chart, compared against upstream/main as a baseline: identical failure counts on both sides, with the 4 new tests passing (splice-validator 44→46, splice-participant 22→24). The handful of pre-existing failures reproduce unchanged on main.

On the wider refactoring you mentioned to remove the duplication across these init containers — happy to take that on in a follow-up PR if you think it's worth doing; it seemed too broad to fold in here.

[martinflorian-da]

@SLoeuillet I don't think you have addressed my concerns that the Canton nodes will not learn about the new database user to use.

You want to be overriding POSTGRES_USER somewhere: https://github.com/digital-asset/canton/blob/72c82f382942e3bda7d6418f72f95b1524f57ca8/docker/canton/images/canton-base/storage.conf#L9

Did you give some thought on how you could end-to-end test these changes? Deploy at least a validator with a custom DB config, ideally using both compose and Helm?

[SLoeuillet]

You're right, and I apologise — I answered only the first half of your review and claimed it was complete. The POSTGRES_USER point was the important one and I skipped it entirely.

Now fixed. For the participant, domain, mediator and sequencer, persistence.user reached the pg-init container but never the node: canton-base's storage.conf reads user = ${?CANTON_<NODE>_POSTGRES_USER} and the charts never set it, so the node kept connecting as the hardcoded cnadmin. Exactly the half-applied configuration you described — worse than not offering the value, because it looks configurable.

Two things that made the fix straightforward, and which I think strengthen the case for the PR:

• The images already support it. canton-base's app.conf reads CANTON_DOMAIN_POSTGRES_USER and CANTON_PARTICIPANT_POSTGRES_USER. Nothing needed changing outside the charts.
• Compose already passes it — CANTON_PARTICIPANT_POSTGRES_USER=${SPLICE_DB_USER} in both cluster/compose/validator/compose.yaml and cluster/compose/sv/compose.yaml. So a configurable DB user is already a supported deployment shape; Helm was simply the odd one out.

I also checked the Splice apps while I was there: splice-validator, splice-scan and splice-sv-node were never affected — they write user straight into their HOCON storage block, so they already honoured the value.

The env var is emitted only when persistence.user is set, so rendered output stays byte-identical for anyone not using it.

Tests: the value reaching the node's env, and nothing emitted when unset, on splice-participant and on the global-domain mediator — covering both variable names. I verified both fail when the env var is missing, rather than assuming.

On end-to-end: agreed it is warranted, and given two of you missed this on the first pass I would rather not self-certify. Before I burn a cluster deployment on it — would you rather see it validated through compose (which already has SPLICE_DB_USER wired, so it is the cheaper path and exercises the same conf), through Helm on a cluster, or both? Happy to do whichever you would actually trust.

Rebased onto current main; the only conflict was the release-notes entry, as usual.

[martinflorian-da]

Before I burn a cluster deployment on it — would you rather see it validated through compose (which already has SPLICE_DB_USER wired, so it is the cheaper path and exercises the same conf), through Helm on a cluster, or both?

@SLoeuillet Ah compose doesn't matter much here, does it? Apologies for even bringing that one up.

So I think we want to test with a Helm deployment here. Do you have a reasonable way to do that manually / locally? A confirmation for a validator (app + participant) would be sufficient for me. I guess you don't even have to connect to a synchronizer - if init passes until the "connect to synchronizer" step then we should be safe.

Alternatively I'm also open to you making an additional Pulumi commit that makes some validator deployment and perhaps the sv-runbook use the new fields. We use this to trigger a scratchnet cluster test on the PR, then we revert your commit (so it can be hacky). Either that or just a new PR stacked on this one.

Longer-term we want to cover your changes here in our regular cluster tests. That seems like we should better move it to a follow-up PR though.

[SLoeuillet]

Tested with a Helm deployment on a local kind cluster. Both the participant and the validator app come up against a Postgres whose role and bootstrap database are deliberately not cnadmin/cantonnet.

Setup — a plain postgres:14 initialised with POSTGRES_USER=appuser, POSTGRES_DB=bootstrapdb, so the old defaults do not exist at all:

SELECT count(*) FROM pg_roles    WHERE rolname='cnadmin';    -- 0
SELECT count(*) FROM pg_database WHERE datname='cantonnet';  -- 0

Both charts installed with persistence.user=appuser and persistence.bootstrapDatabaseName=bootstrapdb.

Participant — 1/1 Running:

• pg-init completed and created the database: participant_test owner=appuser
• the node's env carries CANTON_PARTICIPANT_POSTGRES_USER=appuser
• 18 Flyway migrations applied, then Successfully started all nodes / Canton started
• Postgres confirms who is actually connected: appuser -> participant_test (32 conn), and no other role has a single connection

Validator app — reaches and completes its DB work:

Database: jdbc:postgresql://postgres:5432/validator_test?currentSchema=validator (PostgreSQL 14.24)
Lock successfully acquired ... Finished setting up database schemas

with validator_test owner=appuser and schema validator owner=appuser. It then stops at Invalid host or port: 5002 — the ledger-API/synchronizer wiring I deliberately did not configure, which is the point past which you said we do not need to go.

Negative control, which I think is the part that actually proves it: installing the unpatched splice-participant from current main with the identical values leaves it stuck at Init:0/1, looping on

psql: error: FATAL: password authentication failed for user "cnadmin"

and its rendered env contains no CANTON_PARTICIPANT_POSTGRES_USER at all — persistence.user is simply ignored. So the values only take effect with this PR, in both the init container and the node itself.

Two incidental notes from doing this, neither related to the change:

• cluster/helm/splice-*/files/logback.xml is gitignored and supplied by your build, so a chart installed straight from a checkout crashes with Premature end of file on logback. Easy to work around, but it does mean helm install from a clone is not self-contained.
• splice-validator's schema rejects setting both the top-level scanAddress and scanClient.scanAddress, which took a moment to work out from 'not' failed. Not a problem, just unfriendly output.

Happy to also do the Pulumi/scratchnet commit you offered if you would still like the change exercised in your own cluster tests — say the word and I will stack it, or leave it for the follow-up PR you mentioned.

[martinflorian-da]

Thank you @SLoeuillet ! I think you have convinced me that the changes work for validators. The Helm test coverage there is also quite good.

After thinking about your PR some more though I still don't feel comfortable about merging the changes for SVs without having additional testing. Among other things, an SV deployment also consists of more moving parts - so harder to judge if we're not still missing something.

Adding relevant (or at least: easy to repurpose) cluster testing is actually planned soon as part of a security hardening story.

So how about this, assuming that you are mostly interested in the validator use case anyway:

• You cut down this PR to cover only validator and participant and we merge like that.
• You either open a follow-up PR with the support for SVs, or leave it to me to recover the "lost" parts from this PR here once the testing infrastructure is there.

If you prefer not to cut up your PR we could also just delay merging by a few weeks, but I'd actually prefer having your changes for validators in earlier.

[SLoeuillet]

That works for me — validators are exactly the use case I care about, so getting them in early is the better trade. Done: the PR is now validator + participant only, rebased onto current main, squashed to a single commit.

Kept: splice-participant, splice-validator, and splice-util-lib/_helpers.tpl — the last one is needed because both charts render the postgres-metrics sidecar through it. 10 files, +155/−6.

Dropped, and reverted to main: splice-sv-node, splice-scan, splice-global-domain (sequencer + mediator), splice-domain, and splice-splitwell-app. I put splitwell with the SV set rather than here — it is neither a validator nor an SV component, so it seemed to belong with the batch that waits for the new test coverage rather than with the one you are merging on the strength of the validator testing. Say so if you would rather have it in this PR; it is a two-line change.

The title, description and release-notes entry are all rescoped, and the entry now says explicitly that the SV charts still use the hardcoded values, so the note does not overclaim.

The SV work is preserved on a branch and nothing is lost. I am happy to open the follow-up PR whenever you prefer — either now so it is queued behind this one, or once the security-hardening testing lands and it can be reviewed against real coverage. Your call; if it is easier for you to recover the parts yourself at that point, that is fine too.

[martinflorian-da]

Perfect, thank you!

[martinflorian-da]

@SLoeuillet We had a bug on main, please rebase (hopefully the last time on this PR...)

[SLoeuillet]

Rebased onto ed7caed7f — no conflict this time, and the diff is unchanged at 10 files, +155/−6. Chart tests give the same results as main on both charts.

Note the force-push will have re-armed Approve CI for Forks again, sorry — that one needs your click whenever you next pass by.

[SLoeuillet]

Thanks for merging, and for the review — the POSTGRES_USER catch in particular was the one that mattered.

Opened the follow-up as #7069 with the SV charts plus splitwell, so the work is queued rather than lost. No urgency from my side: it is there for whenever the SV cluster coverage lands, and I have said as much in the description. Happy to rebase it whenever you need.