feat(helm): add HStore deployment chart - #3218
Draft
bitflicker64 wants to merge 58 commits into
Draft
bitflicker64 wants to merge 58 commits into
bitflicker64 wants to merge 58 commits into
Conversation
…tore disabled The Server wrapper now writes auth.admin_pa from the auth Secret alongside usePD and pd.peers, so an auth-enabled release keeps its configured admin password with init_store.enabled=false instead of silently falling back to the public default. The Secret value is rejected when it contains properties-parser metacharacters that would inject config lines or store a different password than the Secret holds. The new hubble component deploys the Hubble UI as a single-replica Deployment with pd and direct wiring modes, optional Ingress and H2 persistence, schema validation, render-time guards, docs, and CI coverage. PD-meta installs (auth enabled, or Hubble in pd mode) also announce the Server client Service URL to PD via server.urls_to_pd and server.deploy_in_k8s so discovery clients receive a resolvable address instead of the 0.0.0.0 default, and the Hubble wrapper writes server.host so current images bind all interfaces. Because current Hubble images authenticate their login against the cluster, rendering Hubble without server.auth fails unless explicitly overridden. The CI invalid-value step now fails on every case rather than only its last line, and positive renders cover both Hubble modes. Validated against a composition of master 1716c77 plus the current heads of apache#3119 (edf07d0), apache#3126 (b40c42f), and apache#3130 (198de19): fresh auth-enabled installs reach Ready with zero restarts, the admin credential comes from the Secret while unauthenticated and default-password requests get 401, and Hubble logs in with the Secret credential and reads cluster metadata through PD discovery, with its H2 metadata persisted on the PVC.
…xtraEnv overrides A PD PodDisruptionBudget with minAvailable below floor(replicas/2)+1 permits voluntary evictions that leave PD without a Raft majority, so the render now requires the quorum floor in addition to the existing blocks-all-drains upper bound. With 2 replicas no valid budget exists; the README documents the even/odd contract. extraEnv entries render after the chart-owned variables and Kubernetes lets the last duplicate win, so a duplicate name could silently override a validated contract such as HG_SERVER_INIT_STORE_ENABLED=false. Each component's extraEnv now rejects its chart-managed variable names. Boundary and negative render cases for both rules are part of the CI invalid-value step.
The run failed at startup because azure/setup-helm@v4 is not on the ASF-approved actions allowlist; install the pinned helm release from the official tarball in a plain run step instead.
…cheduling defaults Fresh installs now seed PD with partition.default-shard-count derived from the Store count (3 when store.replicas is at least 3, otherwise 1, matching PD's odd-only constraint and its 2-to-1 clamp), so a default 3-Store deployment gets store-level HA instead of the image default of one shard replica per partition. The seed is delivered as -D system properties prepended into the PD JAVA_OPTS ahead of pd.javaOpts, preserving the start script's automatic heap sizing; it applies at first bootstrap only, after which PD metadata is authoritative, all documented together with the resulting initial partition count change. pd.antiAffinity and store.antiAffinity default to preferred so the chart schedules on clusters with fewer nodes than replicas; values-cluster.yaml keeps required for both, NOTES warns when PD quorum members may co-locate, and the README documents the upgrade implications. The Disaster Recovery documentation describes what current PD builds actually do: the scheduled patrol only marks silent stores Offline, shard reconciliation and tombstone processing run only via the manual /v1/task/patrolPartitions endpoint, and the pd.patrol-interval and store.max-down-time properties are bound but never read, which is why the chart does not expose them. Periodic leader balancing and recovery metrics are referenced as upstream feature requests. extraEnv now also rejects JAVA_OPTIONS for pd, store, and server, because the start scripts drop the chart-managed JAVA_OPTS entirely when it is set. Schema accepts numeric strings for the new keys, values-file integers at or above one million no longer fail as scientific notation, and CI asserts the rendered -D content, covers both shard-count validation messages, and kubeconforms the sharded render.
…#3137) Document that the creating Server is consistent at HTTP 200 after apache#3138, while cross-replica convergence and PD-owned creation remain open upstream.
Multi-replica Server pods must use one JWT signing key or Hubble login fails behind the Service; chart-manage or BYO via server.auth.tokenSecret.
When auth is enabled without existingSecret, create a stable release-admin Secret so operators are not forced to pre-create credentials.
Default installs get a chart-managed admin Secret; leave Hubble off so API-only clusters stay lean, and enable the UI with one flag when wanted.
Split server.auth into admin and token blocks so inline credentials (password/value) are distinct from Kubernetes Secret refs (existingSecret), matching the clearer values API for operators.
Let Server register a reachable URL with PD and optionally expose the PD client Service, so standalone Hubble can discover the cluster without in-cluster DNS. Fail closed when advertiseUrl is set without PD meta mode.
PD resolves its raft peer hostnames once at boot and freezes the result as the RPC allowlist. With podManagementPolicy: Parallel a PD can start before its peers A records are published, freeze a partial allowlist, and permanently block the elected leader, wedging a fresh install. Add a wait-for-pd-dns init container mirroring the store wait-for-pd gate: hold the pd container until every peer hostname resolves, bounded by pd.waitTimeoutSeconds (values: pd.waitEnabled escape hatch, waitImage, waitTimeoutSeconds, waitResources, schema-enforced). busybox nslookup ignores resolv.conf search domains, so the gate derives the cluster domain and retries the FQDN. The PD headless Service publishes not-ready addresses, so records appear at IP assignment and the gate cannot deadlock. First-boot mitigation only: a PD rescheduled with a new pod IP still requires the image-side allowlist refresh, noted under Limitations. Hardening and docs in the same pass: - explicit updateStrategy and persistentVolumeClaimRetentionPolicy (Retain/Retain; honored on K8s >=1.27) on the PD and Store StatefulSets, value-driven and schema-typed - checksum/auth pod annotation on the Server Deployment so rotating an existingSecret rolls Server pods on the next upgrade; hashes Secret names, keys, and resourceVersion, never Secret data. Best-effort: template-only pipelines render a constant, and the first upgrade after a fresh install rolls Server once - NOTES.txt no longer echoes the admin password - README: release-name assumption note, port-forward instead of exec curl in troubleshooting, Upgrading notes for the one-time PD and Server rollouts this version causes, and Limitations entries for the image-side PD allowlist and PD management REST issues (chart 0.1.1)
This PR introduces the chart, so there is no published 0.1.0 to bump from and the version should stay at the initial one until a release happens. Reword the Upgrading note for the same reason: it described rolls that happen when upgrading "from an earlier revision", which does not exist here. State the forward-looking behavior instead, that any Pod template change rolls that workload, and keep the PD allowlist caveat and the first-upgrade Server roll, both of which still apply.
The lint-and-render legacy guard caught a regression: pd.waitImage was added to the pd required list, so a release created before the field existed failed schema validation under --reuse-values. Make the field optional and default the image in the template instead, matching how waitTimeoutSeconds already degrades. Also flip the gate condition to "not explicitly disabled" so those same legacy releases pick the DNS gate up on upgrade rather than silently losing the first-boot protection because waitEnabled is absent from their values.
The guard rejected advertiseUrl unless auth was on or in-chart Hubble ran in pd mode, on the premise that only then does the chart register server.urls_to_pd. That premise stopped being true when the chart moved every Server to PD meta mode unconditionally: server-deployment.yaml sets $pdMeta := true, so urls_to_pd is always written. Rendering with advertiseUrl set, auth off and Hubble off now emits HG_SERVER_URLS_TO_PD with the configured URL, which the guard had been refusing to produce. The URL-format guard (absolute http:// or https:// only) is unchanged. Correct the README passage that documented the removed requirement.
PD resolves its raft peer allowlist once at boot, so under Kubernetes a peer whose Pod IP was unpublished at that moment, or changes later, is rejected: a first-boot race that could wedge an install, and a permanent rejection after any PD reschedule. The upstream raft.ip-whitelist.enabled switch fixes this at the source, leaving peer authentication to Kubernetes-level controls. Render -Draft.ip-whitelist.enabled=false into PD's derived JAVA_OPTS via the new pd.raftIpWhitelistEnabled value (default false, schema-typed; images predating the switch ignore the property), and remove the interim wait-for-pd-dns init container with its pd.wait* values, schema entries, and peer-hosts helper. The gate only ever mitigated the first-boot half of the problem. Verified on Kubernetes with a PD image carrying the switch: two deterministic installs, PD follower/leader crash, PD and Store majority loss, leader network partition, PVC reattach, rolling restart under write load, auth on every Server replica, a 0.1.2-to-this upgrade, Secret rotation, and a zero-override install followed through the documented first-run steps. Zero "Blocked connection" and zero "Could not resolve allowlist entry" lines across the entire campaign; the pod-IP recycle case that previously failed reproducibly now passes. Also update the chart CI's JAVA_OPTS assertions, which matched a quote-terminated shard-count prefix and therefore broke once the whitelist flag was appended, and bring the README in line: the Limitations and Upgrading sections described the removed init container, and the values table had no row for the new setting.
…ning Setting server.auth.token.value shorter than 32 bytes passed every chart check and then failed at runtime: all Server pods crash looped on the entrypoint assertion and helm install --wait timed out with no indication of the cause. Add a minLength to the schema so the value is rejected at render time with a precise path. The default values set no container resources, so each JVM sizes its heap against total node memory rather than a cgroup limit. On a multi-node cluster where several pods share a node the heaps oversubscribe it. Note the cluster preset in the README and warn from NOTES.txt. Images could only be referenced as repository:tag, so a release could not pin an immutable digest. Add an optional image.digest per component that takes precedence over the tag. Give the Helm test hook bounded default resources so it is admissible on quota-managed namespaces.
The chart shipped no unit tests. Cover the behaviour that fails silently: the PD raft whitelist flag and derived JAVA_OPTS, image digest pinning, the Server auth checksum annotation and Secret wiring, update strategy and PVC retention, quorum arithmetic at one three and five replicas, the test hook resource bounds, and every validateValues guard asserted against its real error message.
Record the values that had no documentation: the optional image.digest for each component, and the 32 byte minimum on server.auth.token.value that the schema now enforces. Add the credential lifecycle that operators hit first. The chart-managed auth Secret survives uninstall and is reused by a later install of the same release name, and helm template cannot read an existing Secret so the password it prints is only a render-time placeholder. Add a worked example for supplying an admin Secret instead of letting the chart generate one. Correct two stale statements: the Limitations note pointed to the Disaster Recovery section as being below when it is above, and the pinning advice said three component tags when there are four, since Hubble also tracks a mutable tag.
Picks up apache#3159 (321ba4d), the REST API adaptation for Hubble and the k8s-mode enhancements that the pending confirmation pass depends on, along with apache#3140, apache#3153, apache#3173, apache#3171, apache#3177, apache#3176, apache#3178 and apache#3149. No conflicts, and no change to helm/.
Every Server replica registered the shared client Service URL with PD, so the discovery registry collapsed three replicas into one logical entry and PD-discovered clients such as Hubble saw a single Server. The registration is a lease renewed by a per-instance heartbeat, so a replaced Pod's entry ages out on its own; announcing each Pod's own IP keeps the replica list truthful without leaving permanent stale entries. Empty server.advertiseUrl now announces the Pod IP through a POD_IP downward-API variable. Setting advertiseUrl keeps its meaning as the shared external endpoint for an outside Hubble. POD_IP joins the reserved environment names so extraEnv cannot shadow it. Ports c2d086d from helm-dev and adds a unit suite covering the fieldRef, the default URL shape, the advertiseUrl override and the reserved name.
…commands The Limitations entry said the PD management endpoints reject requests on current images, so the Disaster Recovery flow might be unavailable. Measuring it shows the opposite. PD compares the Basic-auth username against a fixed internal set and never reads the password, so any password, including an empty one, is accepted for those names. The endpoints are effectively unauthenticated rather than unusable, which is a caveat about exposure, not availability. That also makes the documented recovery commands wrong. Written without a credential they answer HTTP 200 with an Unauthorized body and the task never runs, so an operator mid-incident sees success and gets nothing. Add the credential the endpoints actually require and say why the password is empty. Also record that success, refusal and a missing credential all return HTTP 200, so the status code carries no signal for health checks.
…ment health vs ready PD's /v1/health answers 200 as soon as the REST listener is up and never consults raft, so every PD and Store probe and the Store init container's PD wait count listeners, not quorum members (apache#3183). The fix, apache#3185, adds /v1/ready from 1.8.0. - pd.readinessPath and store.waitPath, both defaulting to /v1/health, so the switch to /v1/ready is a values change made with the 1.8.0 pin; the schema rejects paths without a leading slash - README: Limitations entries for the liveness-only health endpoint and for the 45 second discovery lease (measured 30 to 35 seconds); the Store wait is described as a PD wait rather than a quorum wait; the Server now registers its Pod IP, not the Service URL - NOTES and the init container messages no longer claim a quorum - tests: pd_readiness_path_test.yaml, five cases
Points at apache#3185 and says the defaults flip with the 1.8.0 image pin, so the change is not lost once that PR merges.
PD images from 1.8.0 (apache#3189) check the Basic-auth password of every management call against auth.secret-key and refuse to start without one. The chart now keeps that value in a kept release-pd-auth Secret, or in pd.auth.existingSecret, and hands it to the three readers: PD as HG_PD_AUTH_SECRET_KEY, the Server storage wait as PD_AUTH_PASSWORD, and Hubble as operations.pd.password written into its properties file by the existing wrapper. A checksum/pd-auth annotation on the three Pod templates rolls them when the Secret changes; the Server annotations block is now rendered unconditionally for it. Priority and lookup semantics mirror server.auth.token. Older images ignore the password, so the wiring is harmless on the images the draft currently tracks. The values schema requires one of existingSecret, value or autoGenerate and refuses newlines, carriage returns and backslashes in an inline value, since it lands in a Java properties file; the template guard repeats the first rule for values that bypass the schema. The three chart-managed variables join the reserved extraEnv lists. README: Chart Details bullet, four parameter rows, Disaster Recovery calls carry the secret, and the Limitations bullet separates the 1.7.0 behaviour from 1.8.0. NOTES prints how to read the secret. New suite pd_auth_secret_test.yaml, 9 tests; 58 in total. Lint on three presets; renders 16 objects by default and 19 with Hubble. Measured on a kind cluster with images built from master plus apache#3185, apache#3187 and apache#3189: the Secret is created, PD starts with the variable, the Server storage wait passes with the credential, and Hubble lists all nine nodes.
Bring over the onboarding pieces the testing branch already had: a pre-install cluster check, --wait --timeout on the primary install command, the Kind/minikube local-build section (with the pullPolicy override called out as required, since this chart defaults Always), a Hubble row in the component table, and the resource naming pattern. Close three notes from the 2026-09-16 independent test run on apache#3132: document the liveness bound on a stalled component, document how to stage a rollout given the schema's one-replica floor, and enable the Server PodDisruptionBudget in values-cluster.yaml so a node drain cannot evict every Server at once (the default stays off, as recorded). Also move the server.ingress.allowPlainHttp rejection into the schema (a server-specific ingress definition without the key), so the refusal happens at validation with a clear message instead of mid-render; the template guard stays as a backstop.
A PD leader that vanishes without closing its sockets (a freeze or a network partition, unlike a crash) is waited on for the full raft.rpc-timeout per attempt, so the image default of 10000 ms leaves the cluster without a PD leader for about a minute. Measured twice on the chart: 67 s with a blackholed leader (2026-09-10 campaign) and 56 s with a SIGSTOP-frozen leader, against 8.5 s with the timeout at 3000 ms (independent k3s run on apache#3132, 2026-09-16). Data-plane writes were unaffected in both runs; everything that needs PD waited. Render -Draft.rpc-timeout from a new pd.raftRpcTimeoutMs value, defaulting to 3000, placed with the other derived -D properties ahead of pd.javaOpts so an explicit flag there still wins. Empty preserves the image default, and a values set stored before the key existed renders without the flag. Two unit tests cover the override and the opt-out; the CI verbatim JAVA_OPTS greps carry the new flag.
- render explicit zero grace periods for all workloads - preserve omitted fields for legacy values - cover zero overrides across four workload templates
- restrict startup and liveness success thresholds to one - preserve readiness thresholds above one - cover invalid probe overrides for all components
25 tasks
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3218 +/- ##
============================================
- Coverage 41.12% 36.59% -4.54%
+ Complexity 7209 4664 -2545
============================================
Files 802 545 -257
Lines 69393 51420 -17973
Branches 9237 7245 -1992
============================================
- Hits 28541 18815 -9726
+ Misses 37588 30288 -7300
+ Partials 3264 2317 -947 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The workflow lints, renders and validates but never invokes the helm-unittest suites under helm/hugegraph/tests/, so a change can break every one of them while CI stays green. Install the plugin pinned at v1.1.2 and run it after lint. Also check the helm and kubeconform tarballs against their published sha256 sums instead of piping an unverified download into the job.
Review fixes from #221: - Require store.pdb.minAvailable >= store.replicas - 1. Shards keep their copies on a subset of the Stores, so a budget that permits two concurrent evictions (the old default with four or five Stores) can remove a shard majority even while the PDB is satisfied. The default 3-Store topology is unchanged. - Reject podLabels that set app.kubernetes.io/name, instance or component on any of the four workloads: they render after the chart-managed labels and would break the selectors, Services and PDBs that match on them. - Accept an empty hubble.image.tag when a digest is set; the digest is a complete image identity.
Review fixes from #221: - The checksum annotations were lookup-only, so an upgrade that changes an inline pd.auth.value, admin password or JWT value hashed the OLD Secret's resourceVersion and did not roll the reading pods in that same upgrade. Fold a hash of the desired inline value into hugegraph.pd.authChecksum and hugegraph.server.authChecksum, only when that inline value is live (no existingSecret overriding it); the autoGenerate and existingSecret paths are unchanged. - Schema now rejects inline admin passwords the Server wrapper refuses at startup (leading whitespace, backslash, CR/LF), failing at install instead of CrashLooping after it. - pd.auth.value is restricted to printable ASCII with no leading whitespace and no backslashes: Hubble reads its properties file as ISO-8859-1, strips leading whitespace, and its Java properties parser unescapes backslashes, so such values silently stop matching what PD holds.
Review fixes from #221: Raft and shard membership are persisted, and deleting Pods does not reconfigure them: a 3-to-1 PD shrink permanently loses quorum, and removing a Store strands the shard copies it holds. validateValues now reads the live StatefulSet through lookup and fails an upgrade whose replica count is below it. Template-only renders have no live object and skip the guard, and an operator who finished the documented manual procedure has already scaled the live StatefulSet, so the upgrade passes once live and desired match. README: the manual scale-down procedure under Scaling, a note that kubectl scale does not survive the next helm upgrade, and an Upgrading bullet recommending store.updateStrategy=OnDelete for production image rolls, because Store readiness reports the listener while shard recovery is still catching up.
Review fixes from #221: - Quote the configurable Secret data keys in the three Secret templates, so a key like "on" or "01" stays the literal string the workloads reference instead of being reinterpreted by YAML. - A regression test renders a Secret with the key "on" and asserts the literal key survives; a CI render greps the quoted key verbatim, since the unit framework's YAML 1.2 parser cannot see the 1.1 coercion. - Escape dots in the NOTES.txt jsonpath commands, so a dotted key such as admin.password reads the flat Secret entry instead of querying a nested field and printing an empty credential. - Default the pd readiness path and the Store wait path to /v1/ready in the templates too, matching values.yaml, NOTES and the schema for a release whose stored values predate the keys. - Scope the BestEffort NOTES warning to the case where no component sets resources; a milder note already covers the partial case. - README: enable Hubble with --reuse-values so the upgrade keeps the release's existing overrides; add -n <namespace> to the troubleshooting commands; use four-space continuations in the new command blocks.
Review fixes from #221: - A non-ClusterIP pd.service now requires pd.service.allowInsecureExposure=true: the exposed gRPC port carries no authentication, raft membership RPCs included, so exposure must be an explicit decision taken after restricting reachability elsewhere. - A Server Ingress without tls is refused unless server.ingress.allowPlainHttp=true; auth is on by default, so a plain HTTP route carries Basic-auth credentials and JWTs. Replaces the old rule that rejected the key outright, and mirrors the Hubble opt-in. - hubble.securityContext.readOnlyRootFilesystem=true is refused: the Hubble wrapper writes its properties file inside the image at startup and the chart mounts no writable volume there. - HG_SERVER_STARTUP_TIMEOUT_S is now the startup probe budget minus the image's 300-second storage wait (150 s by default; the 450 s probe floor keeps the result at 150 or more, with the image's 120 s minimum kept as a guard), so the start command and kubelet give up together instead of the image outliving the probe. - The default PD announcement brackets an IPv6 POD_IP, which a URL requires; an explicit server.advertiseUrl stays untouched. - README documents the exposure acknowledgements, refreshes the validation-rules list for the new gates, adds an Upgrading note for releases that already expose PD or serve a TLS-less Ingress, and uses HTTPS in the outside-Hubble direct URL example, since login sends credentials over it.
Review fixes from #221: The recovery task endpoints execute locally on the PD that receives them and answer a follower with empty success, so the documented procedure could report three successes while recovering nothing. The runbook now identifies the leader through /v1/members and port-forwards that Pod before triggering tasks. A Store replaced with an empty PVC registers under a new Store ID while the old one stays Offline with its shard memberships; the patrol repairs only Tombstone members, so it never touches the Offline entry. The runbook adds the retirement step: mark the old ID Tombstone with POST /v1/store/<id> {"storeState":"Tombstone"}, which hands its shards to the patrol, then patrol and verify the shard groups. DELETE only erases the record and strands the memberships, so it is documented as cleanup after the patrol, never the retirement itself.
The v1.1.x plugin manifests carry a platformHooks field that Helm 3.16.2 rejects at plugin load, so the new unittest step failed in seconds on both PRs; the local gate missed it because this machine runs Helm 4. Verified against the CI's exact 3.16.2 binary with an isolated plugin home: v1.0.0 installs and runs the suites. Under v1.0.0 a schema abort is a suite error rather than a matchable failed render, which broke exactly one case, the pd.auth no-secret-source rejection. That input moves to the workflow's must-fail render guards, where the same class of schema rejection is already asserted; nothing is asserted less than before. 71 cases pass under 3.16.2 with v1.0.0 and under Helm 4 alike.
Follow-ups from CodeRabbit's 2026-09-18 pass on #221: - Install helm-unittest by the v1.0.0 tag's commit SHA, so a moved tag cannot change what executes on the runner. - The recovery runbook ran two port-forwards and three curls as one foreground sequence, which blocks at the first forward. Note the second terminal, stop the Service forward before the leader forward, and move the inline comments to their own lines within 100 columns. - Exclude a leading form feed from inline admin passwords: the config reader trims it when the admin is first created, so the Secret and the effective password would silently differ.
Review fixes from #221: The helm test Pod never received .Values.imagePullSecrets, so on an authenticated private mirror the four workloads pull with credentials while helm test enters ImagePullBackOff. The hook now propagates the setting like the workloads do, with a unit case. PD and Store storage sizes live in the StatefulSet volumeClaimTemplates, which Kubernetes forbids changing, so an upgrade with a new size was rejected in full with nothing telling the operator the supported path. The parameter tables mark both sizes install-time and Upgrading gains the resize procedure: patch each PVC, wait, recreate the StatefulSet with --cascade=orphan, upgrade with the matching value.
…very Two gaps a review of the replies surfaced before posting them: The pd.auth.existingSecret row now states the same value constraint as the inline path (printable ASCII, no leading whitespace, no backslashes); an external Secret bypasses the schema, so the rule has to live in the contract text. The recovery runbook now re-reads /v1/members after the task sequence: if leadership moved mid-sequence, the later tasks ran on a follower and did nothing, so they are rerun on the new leader. Also strips trailing whitespace the workflow picked up in e62cac3.
25 tasks
3 tasks
Add server.readinessPath (default /versions, so nothing changes on current images), mirroring pd.readinessPath: values.yaml, the schema (pattern ^/) and the Server readinessProbe in server-deployment.yaml, plus a README parameter row and a two-case unit suite (74 total). On a Server image that serves GET /readiness (apache#3212, proposed in apache#3221), setting the value to /readiness makes a Server that cannot serve graph traffic answer 503 and drop out of the Service instead of returning 500 to every graph request. Startup and liveness stay on /versions so a Server that merely lost its storage is not restarted. Implements the change proposed and measured by @SebastianGruza in #229 (six fault scenarios at 1 Hz sampling, zero readiness transitions across Store and PD rolling restarts).
The Store commits well beyond its heap. conf/application.yml includes the pd Spring profile, and conf/application-pd.yml sets rocksdb.total_memory_size to 32000000000; RaftRocksdbOptions splits that into a RocksDB write cache and block cache, so both are bounded by 32 GB rather than by the container. The jraft log storage registers a further 1 GiB LRU cache once per process. Against -Xmx1024m -XX:MaxDirectMemorySize=512m the old 4Gi limit sat below the steady state: a k3s run OOM-killed all three Stores after about 1 GB of data and they stayed in CrashLoopBackOff, while 8Gi held at 4.42 GiB anonymous RSS. Request 5Gi, limit 8Gi, with the accounting recorded in the preset and in Limitations. The limit bounds the damage, not RocksDB. The Store entrypoint rebuilds SPRING_APPLICATION_JSON from its own variables and the chart mounts no config file, so rocksdb.total_memory_size cannot be set from the chart today; that is noted as an image-side gap. Reported with measurements by @SebastianGruza on #221.
Both auth checksums appended the live Secret resourceVersion unconditionally, including when the credential was an active inline value. That rolled the Pods a second time on the no-change upgrade after a rotation: the rotation render still saw the old resourceVersion, and the next render saw the new one with the same desired value. The revision input is now chosen per credential. An active inline value contributes its digest alone, so a rotation rolls once. An external or chart-generated Secret keeps the live resourceVersion, which is the only signal it has. Server admin and token select independently. Tests pin the exact annotation for four source combinations, so the parts list cannot change unnoticed; the existingSecret cases set an inline value too and prove it stays out. A render carries no live Secret, so the lookup half stays for the lifecycle run. Answers the open thread on #221.
The OnDelete procedure told the operator to wait for the replaced Store to show Up in PD. That is not a recovery barrier. StoreNodeService.register() persists StoreState.Up and only then notifies the Store, where HgStoreEngine.stateChanged starts restoreLocalPartitionEngine(); a failure there is logged and the state stays Up. The Store is Up before it has restored anything, so an operator following the old text could delete the next replica while the first was still rejoining. Replace it with the strongest check these images support: per shard group, the full shard count, exactly one leader, and the replaced Store back in the groups it holds, read from the PD leader through /v1/shardGroups. State the residue plainly. That is PD's membership record, not proof that the Store finished loading its partitions and caught up, and no endpoint reports restoration-complete, so the text asks for a margin, keeps the PDB floor as the backstop, and names the missing image-side signal. Answers the P1 recheck on #221.
HugeConfig reads the Server properties file through Configurations.properties(), whose reader trims the value, so a password with a trailing space or tab was stored padded in the Secret and applied to the first-created admin account trimmed. Authenticating with the value the Secret holds then fails, with nothing in the install to suggest why. The admin-password pattern already refused leading whitespace but accepted a trailing space, tab or form feed; pd.auth.value accepted a trailing space. Both now require the last character to be non-whitespace, and a single character value stays valid. An inner space is still allowed, because only the ends are trimmed. Schema aborts cannot be matched by the unit framework at v1.0.0, so the CI job gains three must-fail cases and one positive control for the inner space. Answers the P2 recheck on #221.
Two halves of the same mistake: the chart guarded PD and Store shrinks but let a PD increase through, and the documented Store drain named two tasks that cannot retire a healthy Store. PD. The rendered peer list reaches raft only as NodeOptions.setInitialConf, which jraft applies when a node bootstraps without its own configuration. On an initialized group a 3-to-5 upgrade adds Pods and leaves the voting configuration at three. Membership moves through RaftEngine.changePeerList, which the PD client API reaches and no REST route exposes, so the chart refuses the upgrade and says where the operation actually lives instead of offering a sequence nobody has run here. Store. patrolPartitions reallocates groups whose shard count is wrong and hands off groups belonging to Stores already in Tombstone; balancePartitions spreads shards over the active Stores, the leaving one included. Neither retires it, so the old completion condition could never arrive. The procedure now transitions the leaving Store to Tombstone, the same path the recovery runbook uses, checks the remaining Stores still satisfy the persisted replication factor, maps ordinals to Store ids, and waits on /v1/shardGroups before the StatefulSet shrinks. The guard needs a live StatefulSet, so a fresh install at five PD replicas still renders; that case is pinned in the topology suite. Answers two P2 rechecks on #221.
PD's /v1/health returns 200 as soon as Jetty is up and never consults raft. With three PDs that is the right liveness signal, because restarting a follower for a normal election would turn one election into a rolling outage. With one PD there is no election to lose: a PD that steps down and cannot recover, as after a failed raft snapshot on a full disk (apache#3222), keeps answering /v1/health while serving no writes, and liveness never restarts it. Add pd.livenessPath, empty by default and derived: /v1/health above one replica, /v1/ready at one. The startup probe follows the same path, because Kubernetes suppresses liveness until startup succeeds, so leaving startup on /v1/health would give a single PD only the 60 s liveness budget to reach raft readiness after a restart and a slow log replay would crash-loop; following liveness charges that window to the 300 s startup budget instead. Multi-PD renders are unchanged. The value stops being needed if PD starts answering 503 from /v1/health in this state. Reported with measurements by @SebastianGruza on #221.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose of the PR
Add an official Helm installation path for the distributed HugeGraph HStore topology on Kubernetes. The chart packages the PD, Store and Server startup contract that otherwise has to be reconstructed by each operator, with authentication on by default and an optional Hubble UI.
This PR succeeds #3132 (closed, history and earlier review threads preserved there). Its head is the org branch
hugegraph:feat/hstore-helm-chart-3132, the same branch as hugegraph#221, so this PR and the org-side review stay on one branch and cannot drift. Review happens on hugegraph#221, where the GitHub review apps can run; ASF repository settings block their permissions here.Main Changes
helm/hugegraph/: chart 0.1.0 with PD and Store StatefulSets, a Server Deployment, an optional Hubble Deployment, three values presets (values.yaml,values-single.yaml,values-cluster.yaml), a fullvalues.schema.json, and 66 helm-unittest cases.init-storeand registers per-Pod with PD, the Store waits for a PD quorum on/v1/ready, PD readiness is quorum-aware while startup and liveness stay on/v1/health, andHG_SERVER_STARTUP_TIMEOUT_Sis derived from the startup probe budget.pd.auth) consumed by PD, the Server storage wait and Hubble; chart-managed admin and JWT Secrets with bring-your-ownexistingSecretpaths..github/workflows/helm-chart-ci.yml: lint, renders on all presets, fail-path guards, kubeconform, a legacy--reuse-valuesfixture render, and packaging.Verifying these changes
Current work queue, from the review on #221 (2026-09-17): fix the inline findings, run the helm-unittest suites in CI, and one current-image lifecycle run (managed PD/JWT rotation, Store replacement with a persisted record, the documented recovery procedure). Image tags and
appVersionstaylatestuntil a 1.8.x release exists; the pin is a follow-up commit.Does this PR potentially affect the following parts?
helm/and one CI workflow; no Java changes)Documentation Status
Doc - TODODoc - DoneDoc - No NeedDocumentation files in this PR or paired hugegraph-doc PR:
helm/hugegraph/README.md, plus the paired site page in apache/hugegraph-doc#494 (EN and CN, written against05f3d9e0b).