Add vault-utils and one-shot local bootstrap

[amitslog]

Summary

Self-hosted Vault CE cluster for the local platform: one-command setup, path-based tenant isolation, optional dynamic PostgreSQL credentials, and a vault-utils Go binary that owns all Vault logic. Shell is limited to Docker Compose orchestration.

Revision 3

• bootstrap local|aws|azure|gcp, tenants create|destroy, snapshot take|list|verify|restore, health subcommand structure; cloud platforms return not-implemented.
• Bootstrap only initializes a cluster (init, unseal, configure, issue tokens, revoke root). Key material flows through a KeyStore interface (FileKeyStore locally) so cloud platforms can plug in secret managers and share the bootstrap code.
• Tenants are no longer created at bootstrap; use tenants create <id>.
• BOOTSTRAP_DIR defaults to ./.bootstrap on the host, so vault-utils bootstrap local works without Compose; the container still uses /bootstrap.
• Policy templates render with Go text/template ({{.KVMount}}) instead of @@KEY@@.
• docker-compose.yml renamed to compose.yml; images are digest-pinned inline, image env vars removed.
• snapshot.sh and stop.sh folded into the binary (snapshot subcommands; stop is docker compose down). Remaining scripts (setup.sh, reset.sh, lib.sh, runtime-test.sh) only orchestrate Compose.

Test plan

• go test -short ./... and go test ./internal/vaultcluster (isolation, credentials, isolation-then-credentials) pass.
• All validate/test-local CI jobs replicated locally.
• E2E on the Compose stack: setup, tenant create, AppRole login, KV write, dynamic DB creds, cross-tenant 403, tenant destroy, snapshot take/verify, and a restore round trip.

[BSick7]

We're making good progress, but I'm having trouble tracking everything because it's hard to understand the organization scheme.

I think the only top-level directories should be:

• aws (nullstone tf module)
• azure (nullstone tf module)
• cmd (entrypoint for go apps)
• gcp (nullstone tf module)
• internal (libraries to support go apps)

That leaves me confused with scripts, tests, and config dir.
Help me understand the organization and/or improve the layout.

[amitslog]

@BSick7 The top-level config/, scripts/, and tests/ dirs are gone.

Current layout:

• cmd/ – Go entrypoints (vault-utils)
• internal/ – Go library, plus the policy templates and lint fixtures that library uses
• aws/, gcp/, azure/ – Nullstone Terraform modules (stubs for now)
• local/ – Docker Compose target (same idea as the cloud dirs, for a laptop)

Go tests live next to the library (go test ./...). Remaining bash is only under local/ because credentials conformance still uses it. Cloud modules will call vault-utils, not those scripts.

[BSick7]

Incrementally getting better.

• I see a local-bootstrap command in vault utils in go, but also a ./local/boostrap/ dir with scripts. Shouldn't those go away?
• I see a bunch of scripts in ./local/scripts/ that also seem to mirror commands that should be in the go commands
• go does a better job of building test suites, let's convert lint and conformance tests to go tests inside appropriate packages inside the internal/ package.
• There is a huge demo setup in ./local/postgres/init.sh. This seems like an example and not a setup for the postgres which backs the vault cluster. Let's drop it for now.

[amitslog]

Let me optimize it !

[amitslog]

Revision

• Vault bootstrap is only vault-utils. local/bootstrap/ is gone. Compose helpers are local/setup.sh, local/lib.sh, local/snapshot.sh.
• local/scripts/ is gone (it duplicated configure / tenant-create / tenant-offboard).
• Lint and conformance are Go: TestIsolationMatrix and TestCredentialsMatrix.
• local/postgres/init.sh only creates vault_admin and privilege roles. Demo app.customers / app.orders schema is gone.
• Isolation then ./setup.sh --with-credentials (no reset) mounts the database engine instead of 404ing on database/roles.

[BSick7]

• I don't think this latest is working. It's looking for a /bootstrap dir that doesn't exist anymore. I also don't see an init file.
• Let's remove tenant construction from local-bootstrap and keep it solely focused on initializing a new vault cluster. Theoretically, we might be able to share this between local, aws, azure, gcp if there's a pluggable way to save the key material. We should be reusing most of the bootstrap code between platforms. (there will be obvious differences)
• Let's restructure the vault-utils command to be commands/subcommands:
  • bootstrap local|aws|azure|gcp
  • tenants create
  • tenants destroy
  • health
• The @@KEY@@ interpolation is a bit janky. Can we replace with a go template or handlebar? (Since it's already *.tpl, go template might work best)
• Rename docker-compose.yml to compose.yml. Let's also not use env vars for the service images.
• There are still several bash scripts in ./local/. Can those be incorporated into the vault-utils binary?

[amitslog]

Revision:

• BOOTSTRAP_DIR defaults to ./.bootstrap on the host now, so vault-utils bootstrap local works outside Compose. The container still mounts it as /bootstrap. Init file is at local/.bootstrap/vault-init.json.
• Bootstrap only initializes the cluster. Key material goes through a KeyStore interface (FileKeyStore locally), so cloud platforms can plug in their secret managers and reuse the same bootstrap code. Tenants are created with tenants create <id>.
• CLI is bootstrap local|aws|azure|gcp, tenants create|destroy, snapshot take|list|verify|restore, health.
• Policy templates render with Go text/template ({{.KVMount}}).
• Renamed to compose.yml, images are digest-pinned inline, image env vars removed.
• snapshot.sh and stop.sh are folded into the binary. The remaining scripts (setup.sh, reset.sh, lib.sh, runtime-test.sh) only orchestrate docker compose.

Question: should runtime-test.sh (the Compose deployment smoke test) move into a Go test as well, or is thin shell fine for the Compose-only wrappers? @BSick7