Repository navigation
Install FASTDB with helm charts - #103
Open
elvinpoole wants to merge 53 commits into
Open
elvinpoole wants to merge 53 commits into
elvinpoole wants to merge 53 commits into
Conversation
Includes configurations for: - Local Kind cluster (values-local.yaml) - SLAC S3DF production (values-slac.yaml) - Development vCluster (values-ccosta-dev.yaml) Services: PostgreSQL, PostgreSQL standby, MongoDB, web app, query runner, shell pod, mailhog, WAL webserver Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- scripts/helm-deploy.sh: automates build, helm install, code copy to PVC via shell pod, and pod restart. Supports --registry-password, --skip-build, --skip-helm, and --release flags. - templates/secrets.yaml: generate dockerconfigjson secret from registryCredentials values so image pull secrets are Helm-managed. - values.yaml, values-ccosta-dev.yaml: add registryCredentials config. - HELM_HOWTO.md: rewrite deploy instructions around helm-deploy.sh, document registry credentials, deploy script options, and uninstall behavior. - ADR-HELM-DEPLOY.md: architecture decision record for the deploy script approach vs alternatives (bake into images, init containers, Helm hooks, S3). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The rkwebutil frontend uses @external_url@ placeholders in .js.in and .html.in templates, substituted by ./configure --with-external-url. For subdirectory deployments (e.g. /fastdb-ccosta-dev/), the frontend must be built with this path baked in. - helm-deploy.sh: add --external-url flag that overrides the makeinstall entrypoint to pass --with-external-url to configure - HELM_HOWTO.md: document subdirectory deployments, the relationship between --external-url (build-time JS/HTML) and webap.basePath (runtime Flask SCRIPT_NAME), and update all examples Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add -n <namespace> to all helm commands in Common Operations to prevent releases going to the default namespace. Reference the deploy script as the recommended deploy method and note that bare helm install leaves the code PVC empty. Clarify uninstall warning about PVC data loss and add values-ccosta-dev.yaml to file reference table. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Reframe the deploy script as an optional wrapper that saves repetitive typing during iterative development. Everything it does can be run manually via helm and kubectl. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add --context to pass through to all kubectl/helm commands so the
script targets the correct cluster instead of the current kubeconfig
context. Add --create-cluster to create a Kind cluster before
deploying, using sed to expand ${PWD} in the Kind config template.
Cluster name is derived from the namespace. Update defaults from
ccosta-dev to local. Update HELM_HOWTO.md to match.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The sessions, query-results, and logs PVCs were hardcoded to ReadWriteMany which Kind's local-path provisioner does not support. Template the access mode from values (default ReadWriteMany) and override to ReadWriteOnce in values-local.yaml. Also update local image registry/tag to match docker-compose output so images can be loaded into Kind without retagging. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
helm-deploy.sh: auto-detect podman/docker runtime, add --load-images flag for building and loading container images into Kind clusters (implied by --create-cluster), default EXTERNAL_URL to "/". values-ccosta-dev.yaml: add metallb.io/address-pool annotation for postgres LoadBalancer on S3DF. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Documents the steps to create a user in PostgreSQL, trigger a password-reset email via the web UI, and retrieve the reset link from Mailhog logs. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Gives helm-deploy.sh --create-cluster the missing kind-config template with hostPath mounts and port 8080 mapping.
Build the createdb target first so the makeinstall service can use the local fastdb-shell image instead of attempting to pull it from GHCR.
Split local cluster creation and FASTDB installation into independent, self-contained scripts. Deprecate the combined start-local workflow, pin and preload third-party images, and make database migrations wait for PostgreSQL readiness.
Restart webap, queryrunner, and brokerconsumer after rebuilding the host-mounted Python code so rerunning the installer cannot leave long-running processes using stale modules.
Add scripts and Helm values for building and installing FASTDB on a single-node K3s VM. Support configurable host paths, local container images, generated deployment secrets, and CLI-based user creation. Document the Arbutus workflow and mark external web access as future work.
Initialize Git submodules and create the host bind-mount directories before Docker Compose runs. Fail early with ownership instructions when the installation directories are not writable.
Remove Docker build cache before importing images and stream each image directly into K3s instead of creating a temporary archive.
Read the configured PostgreSQL password from the mounted Kubernetes Secret when running apply_migrations.py.
Configure the Arbutus deployment to consume LASS alerts and run the source importer every minute. Discover the K3s node address during installation so the same values file works across Arbutus VMs.
Check for existing Docker and K3s images before deployment and prompt the user to rebuild or reuse them. Build and import only missing or newly rebuilt images to make routine reinstalls faster.
Add a startup probe that gives first-time database and extension setup up to ten minutes before Kubernetes enables the PostgreSQL liveness probe.
Allow users to choose between prebuilt Harbor images and images built from the current checkout. Configure K3s to pull missing Harbor images using the existing Docker credentials, while retaining the local build and import workflow. Authenticate with sudo before local builds and keep the authorization active so long builds can complete unattended.
Drive the Arbutus installer from a supplied Helm values file and use its registry and tag for the makeinstall shell image. Add a CANFAR Harbor values file and move local image building and K3s loading into a dedicated script. Discover the installed shell image when creating users and document the separate Harbor and local-image workflows.
…will replace the job without deleting data
Explicitly clear the rollingUpdate field when MongoDB and PostgreSQL use the Recreate strategy, allowing existing deployments to upgrade cleanly.
Render only the Recreate strategy for MongoDB and PostgreSQL because Kubernetes rejects rollingUpdate fields when the strategy is Recreate.
Verify prebuilt FASTDB images using the registry configured in the Helm values file, use registry-neutral authentication guidance and pull-secret naming, and update the Arbutus documentation accordingly.
Rename the Helm guide to README.md and add a script-free installation quickstart. Rename the Arbutus installer to install-singlenode-fastdb.sh and update documentation, values files, and helper messages accordingly.
Remove the obsolete shared-PVC deployment script, its architecture record, deprecated wrapper, and dependent ccosta values file. Simplify the Helm documentation around the supported Kind and single-node workflows, and ignore environment-specific secrets values files.
Describe the MailHog-based local Kind workflow and direct single-node user creation workflow. Rename create-user-arbutus.sh to create-user-singlenode.sh and update its documentation references.
merge updates from main to feature branch
…u/elvinpoole/helm merge helm arbutus edits into helm branch
Wait for createdb to complete before setting up the processing version, then wait for processing-version setup before running source importer. Add narrowly scoped Job-status permissions and retain createdb so its completion status remains available.
…fs share This allows us to change where the data is stored. However, the k3s local-path provisioner by default will delete PersistentVolumeClaims when the node is deleted. So we may want to instead directly define a PersistentVolume and use that instead.
helm: change the default path for the k3s provisioner to use the cephfs share
changed lsdb and hats version from dev branch to latest release
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds helm charts for automated deployment of fastdb
See
helm/README.mdfor instructions on how to installIn theory we should be able to define a separate values.yaml file for each fastdb deployment
New files summary
helm/fastdb/templates/to deploy FASTDB services, databases, storage, secrets, ingestion, and supporting components.Chart.yaml, and example values files inhelm/fastdb/, including defaults and overrides for a local Kind deployment, and a single-node VMhelm/scripts/for FASTDB installation, plus cluster setup, image preparation, addings users, etcadmin/local/kind-config.yamlto configure a local Kind clusterhelm/README.mdandHOWTO_LOGIN.mdto document helm installation