diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md new file mode 100644 index 000000000..1b5d66dcb --- /dev/null +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -0,0 +1,356 @@ +--- +title: "在 Kubernetes 上运维 HugeGraph" +linkTitle: "Kubernetes 运维 (Helm)" +weight: 5 +search_keywords: + - helm + - kubernetes + - operations + - disaster recovery + - networkpolicy +--- + +### 1 适用范围 + +本页面向已经用 Helm chart 部署好 HugeGraph 集群的运维场景。安装、升级、卸载见 +[部署页](/cn/docs/quickstart/hugegraph/hugegraph-helm/);完整参数参考仍在 +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration)。 + +下文命令假定 release 名为 `hugegraph`、namespace 为 `hugegraph`,请按需替换。两个凭据会反复用到,都从 +chart 管理的 Secret 读取: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin \ + -o jsonpath='{.data.password}' | base64 --decode)" +PD_SECRET="$(kubectl get secret -n hugegraph hugegraph-pd-auth \ + -o jsonpath='{.data.secret-key}' | base64 --decode)" +``` + +### 2 端口与健康 + +| 组件 | 端口 | 用途 | +|------|------|------| +| PD | `8686` | gRPC(Store 与 Server 客户端) | +| PD | `8620` | REST / 健康探测 | +| PD | `8610` | Raft | +| Store | `8500` | gRPC | +| Store | `8510` | Raft | +| Store | `8520` | REST / 健康探测 | +| Server | `8080` | Gremlin 与 REST API | + +所有端口都可以通过 values 配置;修改 `server.port` 会同时更新监听、容器端口和 Service。进程存活但停止响应的组件由 +存活探测终结:默认 20 秒周期、3 次失败阈值把一个卡死 Store 的影响时间限制在约一分钟内,重启后 raft 几秒内就会转移 +它持有的分区 leader。 + +通过 port-forward 访问 API: + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/graphs +``` + +### 3 调度 + +每个组件(`pd`、`store`、`server`、`hubble`)都提供 `nodeSelector`、`tolerations`、`affinity`、 +`topologySpreadConstraints` 和 `priorityClassName`。例如把 Store 固定到打了标签的节点: + +```yaml +store: + nodeSelector: + hugegraph/role: storage +``` + +`antiAffinity`(`required` | `preferred` | `disabled`)为 `pd`、`store`、`server` 渲染按主机名的 +pod 反亲和预设;Hubble 设计为单副本,没有这个配置项。设置原生 `affinity` 会整体替换预设。三者默认都是 +`preferred`,因此节点数少于副本数的集群也能调度。代价是:节点资源紧张时调度器可能把副本放到同一节点,单个节点 +故障就可能同时带走多个 PD 或 Store 副本。节点足够的生产集群应把 `pd.antiAffinity` 和 `store.antiAffinity` +固定为 `required`,`values-cluster.yaml` 即如此。 + +### 4 分区分片 + +全新安装会在 `store.replicas` 不低于 3 时把 PD 的分片数种子设为 3,否则设为 1;没有这个种子,PD 镜像会把 +`partition.default-shard-count` 固定为 1,chart 部署的集群就没有 Store 级高可用。chart 通过 PD 的 +`JAVA_OPTS` 传入 `-Dpartition.default-shard-count`,不影响镜像的 JVM 自动调优。 + +**种子只在首次引导时生效。** PD 第一次以空存储启动时把分片数持久化到自己的元数据里,此后存储的值才是权威:之后修改 +`pd.partition.defaultShardCount` 或跨过推导边界地伸缩 `store.replicas`,对已初始化的集群都没有效果。要修改运行中 +集群的分片数,请使用 PD 自己的配置 API(只接受不超过存活 Store 数的奇数),然后触发 +`GET /v1/task/patrolPartitions`,并预期分片组会发生再分配。 + +分片数还决定初始分区数:`store.replicas x storeMaxShardCount / shardCount`,只在引导时计算一次。按镜像默认的 +`store-max-shard-count`(12),默认 3 Store 安装得到 12 个分区;需要更多分区时调大 +`pd.partition.storeMaxShardCount`(同样只在首次引导时生效)。 + +显式设置的 `pd.partition.defaultShardCount` 必须是奇数且不超过 `store.replicas`;其他值会在渲染时被 chart +拒绝,因为 PD 会悄悄钳制或拒绝它们,渲染通过并不等于设置生效。 + +### 5 NetworkPolicy + +`networkPolicy.enabled` 为每个组件(PD、Store、Server,启用时还有 Hubble)渲染一个 NetworkPolicy。每个策略 +都对自己的 Pod 做双向隔离并列出所需流量,因此应用顺序无关紧要。`values.yaml` 默认关闭(基础配置无法预知你的 +客户端是谁),`values-cluster.yaml` 默认开启。 + +它只在集群网络插件强制执行 NetworkPolicy 时才生效(kind v0.25 及以上、k3s、Calico、Cilium);其他插件会接受 +对象但不执行。验证方法:在另一个 namespace 起一个不带 chart 标签的 Pod,`curl` PD 客户端 Service 的 REST +端口,必须超时。 + +策略开启后,release 只放行自己的流量: + +| 目标 | 来源与端口 | +|---|---| +| PD | PD:raft、gRPC。Store、Server、`pd` 模式的 Hubble:gRPC、REST | +| Store | Store:raft。Server:gRPC、REST。`pd` 模式的 Hubble:REST | +| Server | Hubble 和 `helm test` Pod:`server.port` | +| Hubble | 无(port-forward 走回环,不需要规则) | + +每个组件还可以走 53 端口解析 DNS。release 之外的任何来源都必须列入 +`networkPolicy..extraIngress` 才被放行,包括 Ingress 控制器和 NodePort / LoadBalancer Service +的客户端。在 `extraIngress` 为空时暴露 PD、Server 或 Hubble,或设置 `server.advertiseUrl`,渲染会直接失败 +而不是打开端口。该检查只看 chart 自己创建的暴露;你自行添加的 Service、Gateway 路由或代理需要自己的条目。 + +两个出向事实需要规划。PD、Store、Server 除 DNS 外不访问 release 以外的任何地址,因此需要外呼的功能在策略开启时 +不可用;默认唯一的外呼方是 Store 镜像,它每次启动都从 github.com 下载 `libjemalloc.so`。策略开启时该连接约两分 +钟后超时,Store 不带 jemalloc 继续启动(kind 上实测:就绪时间从 11 秒变为 151 秒);无外网的集群同理。另外 +`helm test` Pod 不被任何 chart 策略选中,因此在你自己的 namespace 级默认拒绝策略下,要给它放行到 `server.port` +和 DNS 的出向流量。 + +**放行其他工作负载。** 每条规则的 `from` 必须指明对端;要放行任意地址,请显式写 `0.0.0.0/0` 这样的 +`ipBlock`。同一个对端里的 `namespaceSelector` 和 `podSelector` 必须同时匹配;写成两个对端则满足其一即可。 +NodePort / LoadBalancer 客户端在 Pod 看来是什么地址,取决于网络插件、`externalTrafficPolicy` 和请求到达的 +节点。在双节点 kind 集群上用 NodePort Server 实测: + +- kindnet,以及开 kube-proxy replacement 的 Cilium:请求发到 Server 所在节点时带客户端地址;经另一节点转发时 + 带那个节点的地址。 +- Calico:经另一节点转发的请求来自该节点在 Pod CIDR 里的隧道地址。 +- 用 kube-proxy 的 Cilium:任何 `ipBlock` 规则都放行不了 NodePort 流量,因为 Cilium 用自己的节点身份而不是 + CIDR 识别节点地址。 + +请用你实际运行的插件测试,并按看到的来源地址写 CIDR。放行 Ingress 控制器、应用 namespace 和 Prometheus 的 +完整示例: + +```yaml +networkPolicy: + server: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: ingress-nginx + podSelector: + matchLabels: + app.kubernetes.io/name: ingress-nginx + ports: + - port: 8080 + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: apps + ports: + - port: 8080 + pd: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: monitoring + ports: + - port: 8620 +``` + +### 6 安全地滚动 Store 镜像 + +Store 的滚动更新以监听检查推进,而不是以分片恢复推进,控制器可能在上一个 Store 尚未重新加入分片组时就替换下一 +个。生产环境滚动镜像时,设置 `store.updateStrategy.type=OnDelete`,逐个删除 Store Pod,并在两次删除之间做检查。 + +PD 里的 `Up` 不是这个检查。PD 在注册时就把 Store 标为 `Up`,此时它还没恢复任何分区;已停止的 Store 也会在 +keep-alive 记录过期前(当前镜像为 300 秒)一直保持 `Up` 并留在所有分片组里:窗口内删除又回来的 Pod 根本不会离开 +`Up` 状态。检查要从 Pod 开始: + +```bash +kubectl -n hugegraph wait --for=condition=Ready \ + pod/hugegraph-store- --timeout=10m +``` + +然后从 PD leader 逐组检查分片成员与 leader(找 leader 的方法见下文"灾难恢复"): + +```bash +curl -s -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/shardGroups | jq ' + .shardGroups[] | {id: (.id // 0), + shards: [.shards[] | {storeId, role}], + leaders: [.shards[] | select(.role=="Leader")] | length}' +``` + +只有当被替换的 Pod 已 `Ready`、它的 Store id 在 `/v1/stores` 里显示新的 `lastHeartBeat`、且每个分片组都报告 +完整分片数和恰好一个 `Leader` 时,才删除下一个 Store。 + +要清楚这证明不了什么:分片列表是 PD 的成员记录,不代表该 Store 已追上 raft 日志。当前镜像没有任何端点报告"恢复 +完成"。想看得更近,port-forward 被替换的 Store,读它自己对某个分片组的视图:`GET :8520/v1/partition/` +返回该 Store 持有的 raft 角色、term 和已提交 index,Store 停机时会失败;term 和 index 要与对端 Store 上的同一 +组对比着看,不要单独读。复数形式的 `GET :8520/v1/partitions` 在 +[apache/hugegraph#3232](https://github.com/apache/hugegraph/pull/3232)(2026-09-24 合入)之前构建的镜像上, +对任何跟随分片组的 Store 返回 500;之后的镜像对每个 Store 都返回 200,被跟随分组的 `conf` 和 `peers` 为 +null。逐组路径在两类镜像上都可用。 + +成员检查通过后留出余量再删下一个 Pod,把 `store.pdb.minAvailable` 保持在 `replicas - 1`,意外的第二次驱逐会被 +拒绝;任何缺分片或没有 leader 的分组都应视为停止信号。真正的"分区恢复完成"信号是上游工作,在 +[apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229) 跟踪。 + +### 7 灾难恢复 + +当前版本 PD 自动做的事很少:60 秒一次的巡检只把停止心跳的 Store 标为 `Offline`。没有自动副本重建;丢失 Store +的副本重新放置、按分片数校正分片组、处理 Tombstone Store,都只在显式触发分区巡检时执行。 + +任务端点在收到请求的那台 PD 上本地执行,follower 会返回空的成功响应但什么也不做。port-forward 客户端 Service +选中的 PD 是任意的,所以先找 leader,再 port-forward 那个 Pod(forward 在前台运行,需要第二个终端): + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-pd-client 8620:8620 +# 读 .data.pdLeader.raftUrl,其主机名即 leader Pod。 +curl -su "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/members +# 停掉 Service 的 forward,改为 forward leader Pod。 +kubectl port-forward -n hugegraph pod/ 8620:8620 +# 校正分片组并处理 Tombstone Store。 +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/patrolPartitions +# 先摊平 Raft leader,再摊平分区数据。 +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balanceLeaders +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balancePartitions +``` + +任务跑完后再读一次 `/v1/members`:如果中途 leader 迁移,后面的任务其实跑在 follower 上、什么也没做。 +`balancePartitions` 之后至少等 180 秒再执行 `balanceLeaders`:`balancePartitions` 即使什么都没搬也会设置 +180 秒的 balance-shard 标志,窗口内的 `balanceLeaders` 会被拒绝。在 +[apache/hugegraph#3233](https://github.com/apache/hugegraph/pull/3233)(2026-09-24 合入)之前构建的镜像上, +拒绝表现为裸的 HTTP 500,原因只在 PD 日志里;之后的镜像把原因放进响应体: +`{"status":1001,"error":"balance shard is processing, please try later!"}`。 + +要分辨"真的执行了"和"空跑",得看 PD leader 的日志,响应本身分辨不了:`patrolPartitions` 无论有没有修复、在 +leader 还是 follower 上,都返回同样的空成功(看日志里的 `reallocShards`、`shardOffline`、`storeTurnoff`,或 +对比前后的 `/v1/shardGroups`);`balancePartitions` 在 leader 上返回 `{}`,在 follower 上返回空响应体。只有 +`balanceLeaders` 的响应体携带工作内容。可区分的任务响应是上游工作,在 +[apache/hugegraph#3231](https://github.com/apache/hugegraph/issues/3231) 跟踪。 + +替换一个回不来的 Store 之后执行 `patrolPartitions`,集群稳定后执行 `balancePartitions`,重启导致 leader 分布 +倾斜后执行 `balanceLeaders`。 + +**丢失 Store 卷。** 带着空 PVC 重建的 Store,在携带 +[apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234)(2026-09-24 合入)的镜像上可以原地 +恢复;更早的镜像,包括所有已发布版本的镜像,都不行。两种情况下替换者都会以**新的 Store ID** 注册,而 Pod 名、 +DNS 名、raft 地址不变,`/v1/stores` 会在同一地址下列出两个 ID。 + +在含 #3234 的镜像上,退役流程是可用的:在 `/v1/stores` 里找到旧 ID(被替换 Pod 地址下不是新注册的那一行),在 +PD leader 上 `POST /v1/store/` 并携带 `{"storeState":"Tombstone"}`,执行 +`GET /v1/task/patrolPartitions`,然后等待;确认每个分片组都恢复到完整分片数且恰好一个 leader、没有分组再引用 +旧 ID、被替换 Store 自己的 `:8520/v1/partition/` 对每个分组都返回 200;最后 +`DELETE /v1/store/` 清除退役记录。在 3+3+3 安装上用 `master` `dbb6663a` 构建的镜像实测:替换者 156 秒 +就绪,Tombstone 加巡检之后 1 秒内 12 个分组全部收敛到新 ID(空 Store 通过 raft 快照安装追平数据),持续写入 +没有丢失任何已确认的写。 + +在不含 #3234 的镜像上,同样的退役流程会执行但修不好分组:PD 发起配置变更,但分组 leader 看到该地址已在组内, +jraft 无可添加,分组继续记录旧 ID。实测:20 分钟、三次巡检之后,12 个分组仍全部引用退役 ID,替换 Store 不持有 +任何分区。健康面板完全看不出这个故障:`/v1/stores`、集群状态、Hubble、Pod 就绪全部显示健康,而每个分片组实际 +只运行在两个存活副本上。唯一能看出问题的检查是被替换 Store 自己的 `:8520/v1/partition/`。 + +因此默认做法不变:替换 Store Pod 时保留它的 PVC(Store id 存在数据目录里,Pod 会以同一 id 回来)。空 PVC 替换 +只作为含 #3234 镜像上的恢复手段;在已发布镜像上,卷真的丢了就当集群已降级处理,准备重建而不是原地恢复。 + +周期性 leader 均衡在 [apache/hugegraph#3135](https://github.com/apache/hugegraph/issues/3135) 跟踪; +灾难恢复指标在 [apache/hugegraph#3136](https://github.com/apache/hugegraph/issues/3136) 跟踪。 + +### 8 伸缩 + +PD 和 Store 在资源名里为最大 StatefulSet 序号预留了空间,伸缩不会改名 PersistentVolumeClaim、不会移动 Pod +身份;两者上限都是 99 副本。Server 通过 `server.replicas` 或 `server.hpa` 自由伸缩(HPA 开启时 Deployment +省略 `spec.replicas`,升级不会覆盖自动扩缩的副本数)。 + +分阶段上线没法写进 values 文件(schema 要求每个组件至少一个副本);用 `kubectl scale statefulset +hugegraph-store --replicas=0` 分阶段,准备好后再扩回来。Server 会保持未就绪等待 Store 注册,下一次 +`helm upgrade` 会按 values 恢复完整拓扑。 + +修改线上 release 的 PD 或 Store 副本数不是普通的 values 变更:raft 与分片成员关系是持久化的,Pod 本身不会 +重新配置它们。chart 通过读取线上 StatefulSet 拒绝 PD 的双向变更和 Store 的缩容,全新安装不受影响。 + +**PD,双向。** chart 渲染的对端列表只作为 raft 的引导配置生效,已初始化的组会忽略它:3 扩到 5 只是多起两个 +PD,投票配置仍是三个;3 缩到 1 直接失去多数派。成员变更走 PD 客户端 API(没有 REST 路由),chart 无法代劳。 +先通过 PD 改成员、在 `/v1/members` 确认新配置、`kubectl scale` 线上 StatefulSet、再用匹配的 values 执行 +`helm upgrade`;在你自己的构建上验证过这套流程之前,请按打算长期保留的 PD 数量安装。 + +**Store,缩容。** 迁出是状态转换,不是均衡:`patrolPartitions` 和 `balancePartitions` 都不会退役健康的 +Store。像灾难恢复退役被替换 Store 那样退役要下线的 Store: + +1. 确认剩余 Store 仍能承载持久化的分片数(`pd.partition.defaultShardCount`;留空时 `store.replicas` 不低于 + 3 推导为 3)。 +2. 通过 `/v1/stores` 按 Pod 地址把要删除的序号(最大的几个)映射到 Store id。 +3. 对每个要下线的 id `POST /v1/store/`,携带 `{"storeState":"Tombstone"}`。 +4. 等到 `/v1/shardGroups` 不再列出这些 id,且每个分组都报告完整分片数和一个 leader。 +5. `kubectl scale` 线上 StatefulSet,再用匹配的 values 执行 `helm upgrade`。 + +删除被移除序号的 PVC 是独立且不可逆的操作;只在第 4 步确认数据已迁走之后再做。 + +### 9 在集群外运行 Hubble + +集群内 Hubble(`hubble.enabled=true` 加 port-forward)是推荐路径,部署页已覆盖。Hubble 必须在集群外运行时有 +两条路径。 + +**直连 Server URL**(图、schema、数据、Gremlin;不需要 PD 发现):关闭 chart 内的 Hubble,暴露 Server +(`server.service.type` NodePort/LoadBalancer,或 Ingress),用如下配置运行独立的 Hubble 镜像: + +```properties +pd.enabled=false +server.direct_url=https://: +``` + +配置文件挂载到镜像内的 `/hubble/conf/hugegraph-hubble.properties`(工作目录是 `/hubble`)。 +`server.direct_url` 要用 HTTPS 或可信通道:登录会把 Server 凭据发到这个 URL。 + +**PD 发现**(集群外的 Hubble 向 PD 询问 Server 地址):`*.svc` 这类集群内名字在外部不可解析,chart 提供两个 +配置项。把 `server.advertiseUrl` 设为外部 Hubble 发现后要使用的绝对 `http(s)://` URL,chart 会用它替代集群内 +Service URL 注册到 PD。再暴露 PD 客户端 Service(`pd.service.type` NodePort/LoadBalancer),这需要 +`pd.service.allowInsecureExposure=true`,因为 PD gRPC 没有认证;先限制谁能访问它(chart 的 NetworkPolicy +开启时,调用方没有列入 `networkPolicy.pd.extraIngress` 前渲染会拒绝这种暴露)。然后用如下配置运行独立 +Hubble: + +```properties +pd.enabled=true +pd.peers=: +pd.server=: +``` + +取舍:设置 `server.advertiseUrl` 后,每个 Server 副本注册的都是同一个逻辑 URL,PD 会把它返回给所有发现客户 +端,包括集群内的 Hubble。留空则走默认的集群内路径,每个 Server Pod 注册自己的 IP。 + +本机快速验证(集群和 Hubble 在同一台机器):port-forward Server `8080` 和 PD 客户端 `8620`/`8686`,设置 +`server.advertiseUrl=http://127.0.0.1:8080`,用 `--network host` 和上面的 PD 配置运行独立 Hubble,再打开 +`8088` 端口。 + +### 10 Gremlin 报 "Could not rebind" 时 + +这个报错有两个成因,持续时间不同。 + +**建图后的收敛窗口。** 处理 `CreateGraph` 的 Server 会等自己的 Gremlin 绑定完成才返回 HTTP 200 +([#3138](https://github.com/apache/hugegraph/pull/3138)),所以在同一个 Server 上建图后立即查询是可靠的。 +其他副本独立收敛,在完成之前,经负载均衡 Service 路由到未收敛副本的 Gremlin 查询可能报 400,如 +`Could not rebind [g]`。带退避重试(窗口通常几秒内关闭),对"建图后立即验证"的流程用会话粘滞或 +port-forward,或先在每个副本上轮询 `/graphs` 再放开查询流量。集群级就绪在 +[#3137](https://github.com/apache/hugegraph/issues/3137) 跟踪。 + +**在 PD 滚动期间启动的 Server Pod。** Gremlin Server 只在启动时实例化一次图;那一刻 PD 客户端连不上的话, +这个 Pod 会终身通过就绪探测、正常提供 REST,而每个发到它的 Gremlin 请求都报 `Could not rebind [graph]`。 +它的 `hugegraph-server.log` 会写明: + +``` +Graph [DEFAULT-hugegraph] configured at [...] could not be instantiated and +will not be available in Gremlin Server +``` + +任何同时滚动了 PD 和 Server 的升级之后,逐个检查 Server Pod 的 Gremlin(镜像不带 curl,逐个 port-forward): + +```bash +kubectl port-forward -n hugegraph pod/ 8080:8080 +curl -s --compressed -u "admin:${PASSWORD}" -H 'Content-Type: application/json' \ + -X POST http://127.0.0.1:8080/gremlin \ + -d '{"gremlin":"graph.traversal().V().limit(1).count()","aliases":{"graph":"DEFAULT-hugegraph"}}' +``` + +健康的 Pod 返回 `result.data`;返回 `Could not rebind` 的 Pod 直接删除。只要 PD 稳定,替换者会正常完成绑定 +(实测:与 PD 滚动重叠的 12 次 Server 启动中 4 次命中,每次删除后都恢复)。 diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md new file mode 100644 index 000000000..15c001fec --- /dev/null +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md @@ -0,0 +1,251 @@ +--- +title: "使用 Helm 在 Kubernetes 上部署" +linkTitle: "Kubernetes 部署 (Helm)" +weight: 4 +search_keywords: + - helm + - kubernetes + - k8s +--- + +### 1 概述 + +Helm chart 在 Kubernetes 上部署一套分布式 HugeGraph 集群:PD、Store、Server,以及可选的 Hubble UI。chart 位于主仓库的 +[`helm/hugegraph`](https://github.com/apache/hugegraph/tree/master/helm/hugegraph) 目录。 + +| 组件 | 工作负载 | 默认副本数 | 作用 | +|------|----------|------------|------| +| PD | StatefulSet + PVC | 3 | 元数据管理:以 Raft 组跟踪 Store 与分区 | +| Store | StatefulSet + PVC | 3 | 图数据存储 (HStore) | +| Server | Deployment | 3 | Gremlin 与 REST 查询层 | +| Hubble | Deployment | 0(默认关闭) | Web UI,通过 `hubble.enabled=true` 启用 | + +分布式 HugeGraph 集群有一套启动约定(Server 不执行 `init-store`、每个 Server 都通过 PD 读写图元数据、Store 等待 PD +形成多数派、一个 PD REST 密钥由三个组件共用)。chart 把这套约定固化下来,运维人员无需手工处理;细节见 +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#chart-details)。日常运维(NetworkPolicy +细节、灾难恢复、伸缩、安全滚动 Store、在集群外运行 Hubble)见 +[运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/)。 + +```mermaid +flowchart LR + subgraph Kubernetes 集群 + PD[PD StatefulSet
3 节点 Raft 组,PVC] + Store[Store StatefulSet
3 副本,PVC] -->|注册、心跳| PD + Server[Server Deployment
3 副本] -->|元数据、服务发现| PD + Server -->|gRPC 读写| Store + Hubble[Hubble Deployment
可选 UI] -->|通过 PD 发现 Server| PD + end + Client[客户端 / hugegraph-client] -->|REST / Gremlin| Server +``` + +启动顺序由 chart 保证,而不是由运维人员保证:PD 先选出 leader,每个 Store Pod 的 init 容器等到多数 PD 汇报就绪后才启动, +Server 则反复等待存储层,直到 Store 完成注册。全新安装无需手工干预即可收敛。 + +### 2 前置条件 + +- Kubernetes 1.23 及以上(chart 渲染 `autoscaling/v2` 和 `policy/v1`) +- Helm 3;升级一节提到的 `--reset-then-reuse-values` 需要 Helm 3.14 及以上 +- 动态卷供给:有默认 StorageClass,或为 PD 和 Store 显式指定 `storageClassName` +- 默认拓扑要运行九个 JVM 进程,内存需留足;见安装一节的资源说明 + +chart 要求组件镜像包含 PD 就绪探测端点和 PD REST 认证(两者都已合入 1.7.0 之后的版本)。默认镜像 tag 指向的构建已包含 +这些改动;1.7.0 镜像不受支持。 + +### 3 安装 + +#### 3.1 获取 chart + +chart 尚未发布到 chart 仓库,从源码树安装: + +```bash +git clone https://github.com/apache/hugegraph.git +cd hugegraph +``` + +#### 3.2 使用默认值安装 + +先确认 `kubectl` 指向目标集群,且集群能供给存储卷。因缺少 StorageClass 而卡在 `Pending` 的 PVC 是最常见的首次安装 +故障: + +```bash +kubectl config current-context +kubectl get storageclass +``` + +然后安装: + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace --wait --timeout 15m +``` + +`--wait` 让 Helm 阻塞到所有工作负载就绪。对分布式集群来说,这个信号意味着 PD 已选出 leader、Store 已注册、Server 已 +启动;不加它,`helm install` 在对象创建完成后就返回。全新集群一般几分钟内收敛,15 分钟超时是给缓慢的镜像拉取留的余量。 + +继续之前需要了解两个默认值: + +- **默认不设置 resources。** 每个 Pod 都是 BestEffort,JVM 按节点总内存计算堆大小。单节点没有问题;多节点集群上各个堆 + 会超订节点内存导致进程中止。超出笔记本范围的部署请使用 `values-cluster.yaml`,或按组件设置 `resources`。 +- **镜像 tag 跟踪 `latest`**,且 `pullPolicy: Always`,直到下一个 HugeGraph 版本发布。生产环境请固定 tag 或 digest。 + +#### 3.3 拓扑预设 + +chart 附带三个 values 文件: + +| 文件 | 拓扑 | 适用场景 | +|------|------|----------| +| `values.yaml` | 3 PD + 3 Store + 3 Server | 默认;preferred 反亲和,认证开启,Hubble 关闭 | +| `values-single.yaml` | 1 + 1 + 1 | 单节点开发与 CI;PVC 更小 | +| `values-cluster.yaml` | 3 + 3 + 3 | 生产起点:JVM 堆与资源设置、PD/Store PodDisruptionBudget、`required` 反亲和、NetworkPolicy 开启 | + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace \ + -f helm/hugegraph/values-single.yaml --wait --timeout 15m +``` + +`values-cluster.yaml` 是起点而非容量保证:请按图规模和流量重新核算资源。其中一个数字值得说明:它给每个 Store 申请 +5Gi、限制 8Gi 内存,远高于 1Gi 的堆,因为 Store 的 RocksDB 缓存在 JVM 堆之外(4Gi 的限制在约 1 GB 数据后就把 +Store OOM 杀掉了)。两个数字要随数据量一起放大。完整参数参考(每个组件的探针、调度、Secret 配置项)见 +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration)。 + +#### 3.4 验证安装 + +```bash +helm test hugegraph --namespace hugegraph +``` + +读取自动生成的 admin 密码并调用 API: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin -o jsonpath='{.data.password}' | base64 --decode)" +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +``` + +以上命令假定 release 名为 `hugegraph`;用其他名字时,请替换成带 release 前缀的资源名(`kubectl get svc,secret -n +` 可以列出)。`helm install` 结束时打印的说明里包含填好名字的同样命令。 + +### 4 认证与 Secret + +认证默认开启,chart 管理三个 Secret。每个凭据按同一顺序取值:你预先创建的 `existingSecret` 优先,其次是内联值,最后 +是安装时随机生成。 + +| Secret | 键 | 用途 | 自带凭据的配置项 | +|--------|-----|------|------------------| +| `-admin` | `password` | Server admin 账号、Hubble 登录 | `server.auth.admin.existingSecret` | +| `-auth-token` | `token_secret` | 所有 Server 副本共用的 JWT 签名密钥 | `server.auth.token.existingSecret` | +| `-pd-auth` | `secret-key` | PD REST 认证,由 PD、Server、Hubble 读取 | `pd.auth.existingSecret` | + +要自行管理凭据,请在安装前创建 Secret 并把对应的 `existingSecret` 指向它;chart 不会改动任何不是它创建的 Secret。取值 +约束:admin 密码不能包含换行、回车或反斜杠;JWT 密钥至少 32 字节;PD 密钥必须是可打印 ASCII。非法值会在渲染时或启动 +包装脚本中被拒绝,不会被悄悄截断。 + +chart 管理的 Secret 在卸载时保留,同名 release 再次安装会复用它们。 + +
+轮换与注意事项 + +- admin 密码只在认证元数据首次创建时生效,之后修改 Secret 不会轮换已有集群的密码。请改用 Server 的 auth API 轮换。 +- 轮换 PD REST Secret 会在下一次 `helm upgrade` 时同时滚动 PD、Server 和 Hubble,保证三者持有的副本一致。纯模板流水线 + (`helm template`、GitOps 渲染器)看不到集群里的 Secret,因此在那里检测轮换的注解不起作用。 +- 三个 Secret 即使从不读取也都存在:安装结束打印的说明里有读取 admin 密码和 PD 密钥的 `kubectl get secret` 命令。 +
+ +### 5 健康检查与启动顺序 + +PD 暴露两个健康端点,chart 有意同时使用两者: + +- `/v1/health` 在 REST 监听建立后立刻返回 200。它不查询 Raft,因此感知不到多数派丢失。 +- `/v1/ready` 在 PD Raft 组选出 leader 之前返回 503,反映的是多数派状态而不只是进程存活。 + +chart 把 PD 的**就绪探测**和 Store init 容器的等待放在 `/v1/ready` 上:Store 只有在多数 PD 成为多数派成员后才启动, +失去 leader 的 PD 会退出 Service 端点,直到 leader 恢复。PD 的**启动和存活探测**按副本数推导(`pd.livenessPath` +可覆盖)。多副本时特意留在 `/v1/health` 上:只是失去 leader 的 PD 仍是健康的 Raft 成员,重启它只会让故障恶化。 +单副本 PD 是例外,推导为 `/v1/ready`:它没有选举可失去,而一个永久退位的 PD(例如磁盘写满导致 Raft 快照失败之后, +[apache/hugegraph#3222](https://github.com/apache/hugegraph/issues/3222))会一直用 `/v1/health` 返回 200 却不再 +服务写入;改用 `/v1/ready` 让 kubelet 把它重启。 + +Server 的启动获得相同的预算:镜像默认会在 120 秒后杀掉仍在启动的 Server,chart 因此从启动探针推导 +`HG_SERVER_STARTUP_TIMEOUT_S`(默认 450 秒),配置的探针预算低于该下限时会被抬高。如果存储层启动更慢,调大 +`server.probes.startup`,镜像的预算会跟着变。 + +### 6 启用 Hubble UI + +Hubble 默认关闭,纯 API 集群因此更精简。在运行中的 release 上启用: + +```bash +helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values --set hubble.enabled=true +``` + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-hubble 8088:8088 +``` + +打开 `http://127.0.0.1:8088`,用 3.4 节的 admin 密码以 `admin` 身份登录。Hubble 通过 PD 发现 Server,集群运维视图无需 +额外配置即可工作。Hubble 只提供明文 HTTP:请通过 port-forward 或做 HTTPS 终结的 Ingress 访问,绝不要直接暴露在不可信 +网络上。在集群外运行 Hubble 也可行,但配置更多;见 chart README 的 +[Reaching Hubble](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#reaching-hubble-pick-one-path)。 + +### 7 升级 + +```bash +helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values +``` + +`--reuse-values` 保留 release 的既有覆盖值;不加它,升级会以 chart 默认值重建 release。任何改变 Pod 模板的升级都会让 +对应工作负载滚动一次。需要提前规划的几点: + +- **全新安装后的第一次升级会让 PD、Server、Hubble 各滚动一次**,因为跟踪 Secret 的注解第一次观察到安装时创建的 + Secret。Store 不受影响。 +- **Store 的滚动更新以监听检查推进,而不是以分片恢复推进**,因此控制器可能在上一个 Store 尚未重新加入分片组时就替换 + 下一个。生产环境滚动镜像时,设置 `store.updateStrategy.type=OnDelete`,逐个替换 Store Pod。PD 里的 `Up` 不是两次 + 替换之间的检查:PD 在注册时就把 Store 标为 `Up`,此时分区尚未恢复,已停止的 Store 也要等 300 秒 keep-alive 过期 + 才离开 `Up`。应等被替换的 Pod 变为 `Ready`,再确认每个分片组都报告完整分片数和一个 leader;完整流程见 + [运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/)。PD 本来就逐个重启, + `pd.updateStrategy.type=OnDelete` 为维护窗口提供同样的手工控制。 +- **升级不能修改 PVC 大小**:Kubernetes 禁止修改 StatefulSet 的 `volumeClaimTemplates`,带新 `storage.size` 的升级会 + 被整体拒绝。chart README 记录了支持卷扩容的 StorageClass 上的扩容步骤。 + +Server 的扩缩容是普通的 values 变更(`server.replicas` 或 `server.hpa`)。**缩容 PD 或 Store 不是**:Raft 与分片成员 +关系是持久化的,删除 Pod 不会重新配置它们,PD 从 3 缩到 1 会永久失去多数派。chart 会拒绝副本数低于线上 StatefulSet +的升级;先迁移再缩容的手工步骤见 chart README 的 +[Scaling](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#scaling)。 + +### 8 卸载 + +```bash +helm uninstall hugegraph --namespace hugegraph +``` + +有两类状态是有意保留的。StatefulSet 创建的 PersistentVolumeClaim 会保留(Kubernetes 行为),确认数据不再需要后请显式 +删除。chart 管理的 Secret 也会保留,同名 release 再次安装时凭据不变。 + +### 9 限制 + +- `networkPolicy.enabled` 为每个组件渲染只放行本 release 流量的 NetworkPolicy;默认关闭,`values-cluster.yaml` + 中开启。它之所以重要,是因为 chart 在集群内关闭了 PD 的 Raft IP 白名单(Pod IP 会变化;白名单只在启动时解析一次 + 对端,之后就会拦截它们),raft 与 gRPC 端口的访问限制由这些策略承担。策略需要强制执行 NetworkPolicy 的网络插件 + (Calico、Cilium、kind v0.25 及以上、k3s),所有外部调用方(包括 Ingress 控制器)都必须列入 + `networkPolicy..extraIngress`,否则渲染失败。细节见 + [运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/)。 +- 镜像 tag 跟踪 `latest`,直到下一个 HugeGraph 版本发布带版本号的镜像;长期运行的环境请固定 tag 或 digest。 +- 创建图之后,其他 Server 副本在短暂窗口内可能尚未收敛,路由到这类副本的查询可能报 `Could not rebind [g]` 之类的 + 错误。请带退避重试,或对"创建后立即查询"的流程使用会话粘滞路由;集群级图就绪在 + [#3137](https://github.com/apache/hugegraph/issues/3137) 跟踪。 +- 当前版本的 Store 恢复由运维人员触发:Store 丢失后的副本重建、leader 均衡、分区再均衡都只在调用 PD 的 REST API 时 + 执行。丢失了卷的 Store 只有在携带 + [apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234)(2026-09-24 合入,尚未进入任何发布版本) + 的镜像上才能原地恢复;更早的镜像上,替换 Store Pod 时请保留它的 PVC。操作手册见 + [运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/)的灾难恢复一节。 +- 集群内无 TLS 终结,无备份工具,无 Operator,无内置监控栈。 + +### 10 排障 + +| 现象 | 先查什么 | +|------|----------| +| Store Pod 卡在 `Init:0/1` | PD 未就绪:`kubectl logs -c wait-for-pd`,再看 PD Pod | +| PVC 停在 `Pending` | 没有默认 StorageClass,或供给器故障:`kubectl get sc` | +| 多节点上 Pod 被 OOM 杀掉或反复重启 | 未设置 resources,JVM 按节点内存取堆:用 `values-cluster.yaml` | +| 建图后立刻查询失败 | 副本收敛窗口:见上文"限制" | + +每种情况的完整排查步骤见 +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#troubleshooting)。 diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md new file mode 100644 index 000000000..85d79fa56 --- /dev/null +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -0,0 +1,483 @@ +--- +title: "Operate HugeGraph on Kubernetes" +linkTitle: "Operations on Kubernetes (Helm)" +weight: 5 +search_keywords: + - helm + - kubernetes + - operations + - disaster recovery + - networkpolicy +--- + +### 1 Scope + +This page is for operating a HugeGraph cluster that the Helm chart already +deployed. Installing, upgrading, and uninstalling are on the +[deployment page](/docs/quickstart/hugegraph/hugegraph-helm/); the full +parameter reference stays in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration). + +Commands below assume the release is named `hugegraph` in namespace +`hugegraph`; substitute your own names. Two credentials come up repeatedly, +both read from chart-managed Secrets: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin \ + -o jsonpath='{.data.password}' | base64 --decode)" +PD_SECRET="$(kubectl get secret -n hugegraph hugegraph-pd-auth \ + -o jsonpath='{.data.secret-key}' | base64 --decode)" +``` + +### 2 Ports and health + +| Component | Port | Purpose | +|------|-------------|---------| +| PD | `8686` | gRPC (Store and Server clients) | +| PD | `8620` | REST / health probes | +| PD | `8610` | Raft | +| Store | `8500` | gRPC | +| Store | `8510` | Raft | +| Store | `8520` | REST / health probes | +| Server | `8080` | Gremlin and REST API | + +All ports are configurable through values; changing `server.port` updates the +listener, container port, and Service together. A stalled component (process +alive but frozen) is ended by its liveness probe: the default 20 s period and +3-failure threshold bound the blast radius of a stalled Store at roughly one +minute, and raft moves its partition leaders within seconds of the restart. + +Reach the API through a port-forward: + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/graphs +``` + +### 3 Scheduling + +Every component (`pd`, `store`, `server`, `hubble`) exposes `nodeSelector`, +`tolerations`, `affinity`, `topologySpreadConstraints`, and +`priorityClassName`. Pinning Store to labeled nodes is just: + +```yaml +store: + nodeSelector: + hugegraph/role: storage +``` + +`antiAffinity` (`required` | `preferred` | `disabled`) renders a hostname +pod-anti-affinity preset for `pd`, `store`, and `server`; Hubble is +single-replica by design and has no such key. Setting a raw `affinity` +replaces the preset entirely. All three default to `preferred`, so the chart +schedules on clusters with fewer nodes than replicas. The trade: under node +pressure the scheduler may co-locate replicas, and a single node failure can +then take more than one PD or Store replica with it. Production clusters with +enough nodes should pin `pd.antiAffinity` and `store.antiAffinity` to +`required`, as `values-cluster.yaml` does. + +### 4 Partition sharding + +A fresh install seeds PD with a partition shard count of 3 when +`store.replicas` is at least 3, and 1 otherwise; without the seed, the PD +image pins `partition.default-shard-count` to 1 and chart-deployed clusters +would run without store-level HA. The chart passes the value as +`-Dpartition.default-shard-count` in PD's `JAVA_OPTS`, so the image's JVM +auto-sizing is unaffected. + +**The seed applies at first bootstrap only.** PD persists the shard count the +first time it starts with empty storage; from then on the stored value is +authoritative, and changing `pd.partition.defaultShardCount` or scaling +`store.replicas` later has no effect on an initialized cluster. To change +the shard count of a running cluster, use PD's own config API (odd values +only, at most the live store count), then trigger +`GET /v1/task/patrolPartitions` and expect shard-group reallocation. + +The shard count also fixes the initial partition count: +`store.replicas x storeMaxShardCount / shardCount`, computed once at +bootstrap. With the image default `store-max-shard-count` of 12, a default +3-store install gets 12 partitions; raise +`pd.partition.storeMaxShardCount` (likewise seeded once) when more +partitions are wanted. + +An explicit `pd.partition.defaultShardCount` must be odd and at most +`store.replicas`; the chart rejects other values at render time because PD +would otherwise clamp or refuse them silently. + +### 5 NetworkPolicy + +`networkPolicy.enabled` renders one NetworkPolicy per component (PD, Store, +Server, and Hubble when enabled). Each policy isolates its own Pods in both +directions and lists the traffic they need, so apply order does not matter. +It is off in `values.yaml`, because the base values cannot know who your +clients are, and on in `values-cluster.yaml`. + +It only works when the cluster's network plugin enforces NetworkPolicy (kind +v0.25 or later, k3s, Calico, Cilium); other plugins accept the objects and +enforce nothing. To check, run a Pod without chart labels in another +namespace and `curl` the PD client Service on the REST port: it must time +out. + +With the policies on, the release admits only its own traffic: + +| To | From, ports | +|---|---| +| PD | PD: raft, gRPC. Store, Server, and Hubble in `pd` mode: gRPC, REST | +| Store | Store: raft. Server: gRPC, REST. Hubble in `pd` mode: REST | +| Server | Hubble and the `helm test` Pod: `server.port` | +| Hubble | nothing (port-forward uses loopback and needs no rule) | + +Every component may also resolve DNS on port 53. Nothing outside the release +is admitted unless it is listed in `networkPolicy..extraIngress`, +including the Ingress controller and the clients of a NodePort or +LoadBalancer Service. Exposing PD, Server or Hubble that way, or setting +`server.advertiseUrl`, with an empty `extraIngress` fails the render instead +of opening the port. The check sees only exposure the chart creates; a +Service, Gateway route or proxy you add yourself needs its own entry. + +Two egress facts to plan around. PD, Store and Server reach nothing outside +the release except DNS, so features that call out do not work with the +policies on; the one default caller is the Store image, which downloads +`libjemalloc.so` from github.com on every start. With the policies on that +connection times out after about two minutes and the Store starts without +jemalloc (measured on kind: Ready after 151 s instead of 11 s); the same +happens on any cluster without internet access. And the `helm test` Pod is +selected by no chart policy, so under a namespace-wide default-deny policy +of your own you must allow it egress to `server.port` and DNS. + +**Letting other workloads in.** Every rule must name its peers in `from`; +to admit any address, write an `ipBlock` such as `0.0.0.0/0` explicitly. A +`namespaceSelector` and a `podSelector` in the same peer must both match; +as two separate peers, either one is enough. What a NodePort or +LoadBalancer client looks like from the Pod depends on the network plugin, +`externalTrafficPolicy`, and the node the request arrives on. Measured with +a NodePort Server on two-node kind clusters: + +- kindnet, and Cilium with kube-proxy replacement: a call to the Server's + own node arrived with the client address; through the other node it + arrived with that node's address. +- Calico: through the other node the call arrived from that node's tunnel + address inside the Pod CIDR. +- Cilium with kube-proxy: no `ipBlock` rule admitted NodePort traffic, + because Cilium identifies node addresses by its own node identities + rather than by CIDR. + +Test with the plugin you run and name the CIDR you see arriving. A worked +example admitting an Ingress controller, an application namespace, and +Prometheus: + +```yaml +networkPolicy: + server: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: ingress-nginx + podSelector: + matchLabels: + app.kubernetes.io/name: ingress-nginx + ports: + - port: 8080 + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: apps + ports: + - port: 8080 + pd: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: monitoring + ports: + - port: 8620 +``` + +### 6 Rolling Store images safely + +Store rolling updates advance on a listener check, not on shard recovery, +so the controller can replace the next Store while the previous one is +still rejoining its shard groups. For a production image roll, set +`store.updateStrategy.type=OnDelete` and delete Store Pods one at a time, +checking between deletions. + +`Up` in PD is not that check. PD marks a Store `Up` at registration, before +the Store has restored any partition, and a stopped Store stays `Up` in +every shard group until its keep-alive entry expires (300 s on current +images): a Pod deleted and back inside that window never leaves `Up` at +all. Start with the Pod instead: + +```bash +kubectl -n hugegraph wait --for=condition=Ready \ + pod/hugegraph-store- --timeout=10m +``` + +Then check shard membership and leadership per group, read from the PD +leader (see Disaster Recovery below for finding the leader): + +```bash +curl -s -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/shardGroups | jq ' + .shardGroups[] | {id: (.id // 0), + shards: [.shards[] | {storeId, role}], + leaders: [.shards[] | select(.role=="Leader")] | length}' +``` + +Delete the next Store only when the replaced Pod is `Ready`, its Store id +shows a fresh `lastHeartBeat` in `/v1/stores`, and every group reports its +full shard count with exactly one `Leader`. + +Know what this does not prove: the shard list is PD's membership record, +not a statement that the Store caught up on the raft log. No endpoint on +current images reports restoration-complete. For a closer look, +port-forward the replaced Store and read its own view of a group: +`GET :8520/v1/partition/` returns the raft role, term, and +committed index that Store holds, and fails while the Store is down; +compare term and index with a peer Store rather than reading them alone. +The plural `GET :8520/v1/partitions` answers 500 on any Store that follows +a group on images built before +[apache/hugegraph#3232](https://github.com/apache/hugegraph/pull/3232) +(merged 2026-09-24); after it, it answers 200 on every Store, with `conf` +and `peers` null for followed groups. The per-group path works on both. + +Leave a margin after the membership check, keep `store.pdb.minAvailable` +at `replicas - 1` so an accidental second eviction is refused, and treat a +group that is short a shard or has no leader as a stop. A real +restoration-complete signal is upstream work, tracked in +[apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229). + +### 7 Disaster recovery + +What PD automates on current builds is narrow: a 60-second patrol only +marks Stores that stopped sending heartbeats as `Offline`. There is no +automatic re-replication; re-placing the replicas of a lost Store, +reconciling shard groups, and processing tombstoned Stores run only when a +partition patrol is triggered explicitly. + +The task endpoints execute locally on the PD that receives them, and a +follower answers with an empty success while doing nothing. +Port-forwarding the client Service selects an arbitrary PD, so identify +the leader first, then port-forward that Pod (two terminals: the forward +runs in the foreground): + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-pd-client 8620:8620 +# Read .data.pdLeader.raftUrl; its host names the leader Pod. +curl -su "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/members +# Stop the Service forward, then forward the leader Pod instead. +kubectl port-forward -n hugegraph pod/ 8620:8620 +# Reconcile shard groups and process tombstoned Stores. +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/patrolPartitions +# Spread Raft leaders, then partition data. +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balanceLeaders +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balancePartitions +``` + +Read `/v1/members` again after the tasks: if leadership moved mid-sequence, +the later tasks ran on a follower and did nothing. Wait at least 180 s +before rerunning `balanceLeaders` after a `balancePartitions` call: +`balancePartitions` sets a balance-shard flag for 180 s even when it moves +nothing, and `balanceLeaders` inside that window is refused. On images +built before +[apache/hugegraph#3233](https://github.com/apache/hugegraph/pull/3233) +(merged 2026-09-24) the refusal is a bare HTTP 500 whose reason appears +only in the PD log; after it, the reason comes back in the body as +`{"status":1001,"error":"balance shard is processing, please try later!"}`. + +Telling a real run from a no-op takes the PD leader's log, because the +responses do not: `patrolPartitions` answers the same empty success on the +leader and on a follower, whether or not it repaired anything (look for +`reallocShards`, `shardOffline` or `storeTurnoff` lines, or diff +`/v1/shardGroups` before and after), and `balancePartitions` answers `{}` +on the leader and an empty body on a follower. `balanceLeaders` is the one +call whose body carries the work. Distinguishable task responses are +upstream work, tracked in +[apache/hugegraph#3231](https://github.com/apache/hugegraph/issues/3231). + +Run `patrolPartitions` after replacing a Store that is not coming back, +`balancePartitions` once the cluster is stable again, and `balanceLeaders` +after restarts that skewed leader placement. + +**Losing a Store volume.** A Store rebuilt with an empty PVC recovers in +place on images carrying +[apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234) +(merged 2026-09-24); on every earlier image, including all published +release images, it does not. In both cases the replacement registers under +a new Store ID while its Pod name, DNS name, and raft address are +unchanged, so `/v1/stores` lists two IDs at one address. + +On post-#3234 images the retirement works: find the old ID in `/v1/stores` +(the row at the replaced Pod's address that is not the newly registered +one), `POST /v1/store/` with `{"storeState":"Tombstone"}` on the PD +leader, run `GET /v1/task/patrolPartitions`, and wait; verify every shard +group is back to full shard count with one leader, no group names the old +ID, and the replaced Store's own `:8520/v1/partition/` answers +200 for every group; then `DELETE /v1/store/` to erase the retired +record. Measured on a 3+3+3 install with images built from `master` at +`dbb6663a`: the replacement was Ready in 156 s, all 12 groups converged +onto the new ID 1 s after the Tombstone and patrol (the empty Store caught +up by raft snapshot install), and a continuous writer lost 0 acknowledged +writes. + +On images without #3234 the same retirement runs and does not repair the +groups: PD fires the configuration change, but the group leader sees the +address already in the group, so jraft has nothing to add and the group +keeps the old ID. Measured: 20 minutes and three patrols later, all 12 +groups still listed the retired ID and the replacement Store held no +partitions. Nothing in the health surface shows the failure; `/v1/stores`, +cluster state, Hubble, and Pod readiness all read healthy while every +shard group runs on two live replicas. The one check that shows it is the +replaced Store's own `:8520/v1/partition/`. + +So the default remains: replace a Store Pod, keep its PVC (the Store id +lives in the data path, and the Pod comes back under the same id). Treat +the empty-PVC procedure as recovery for post-#3234 images only; on a +released image a genuinely lost volume leaves the cluster degraded, and +expect to rebuild rather than to recover in place. + +Periodic leader balancing is tracked in +[apache/hugegraph#3135](https://github.com/apache/hugegraph/issues/3135); +disaster-recovery metrics in +[apache/hugegraph#3136](https://github.com/apache/hugegraph/issues/3136). + +### 8 Scaling + +PD and Store reserve the maximum StatefulSet ordinal in their resource +names, so scaling never renames a PersistentVolumeClaim or shifts a Pod +identity; both are capped at 99 replicas. Server scales freely through +`server.replicas` or `server.hpa` (with HPA on, the Deployment omits +`spec.replicas` so upgrades do not overwrite the autoscaler). + +A staged rollout cannot be written in a values file (the schema requires +one replica per component); stage it with `kubectl scale statefulset +hugegraph-store --replicas=0` and scale back up when ready. The Servers +wait, not ready, until Stores register, and the next `helm upgrade` +restores the full topology from values. + +Changing PD or Store replicas on a live release is not a values change: +raft and shard membership are persisted, and Pods alone do not reconfigure +them. The chart rejects both directions for PD and a shrink for Store by +reading the live StatefulSet, so fresh installs at any count are +unaffected. + +**PD, either direction.** The rendered peer list reaches raft only as its +bootstrap configuration, which an initialized group ignores; a 3-to-5 +upgrade leaves the voting configuration at three, and a 3-to-1 shrink +loses quorum outright. Membership changes go through the PD client API +(no REST route exposes them), which the chart cannot wrap. Change the +membership through PD, confirm it in `/v1/members`, scale the live +StatefulSet, then `helm upgrade` with the matching value; until that +sequence is verified on your own build, install the PD count you intend +to keep. + +**Store, shrinking.** Draining is a state transition, not a balance: +neither `patrolPartitions` nor `balancePartitions` retires a healthy +Store. Retire the leaving Stores the way Disaster Recovery retires a +replaced one: + +1. Check the remaining Stores still cover the persisted shard count + (`pd.partition.defaultShardCount`; empty derives 3 when + `store.replicas` is at least 3). +2. Map the ordinals the shrink will delete (the highest ones) to Store + ids through `/v1/stores`, matching on the Pod address. +3. `POST /v1/store/` with `{"storeState":"Tombstone"}` for each + leaving id. +4. Wait until `/v1/shardGroups` no longer lists those ids and every group + reports its full shard count with one leader. +5. Scale the live StatefulSet, then `helm upgrade` with the matching + value. + +Deleting the PVCs of the removed ordinals is separate and permanent; do it +only after step 4 reports the data moved. + +### 9 Running Hubble outside the cluster + +In-cluster Hubble (`hubble.enabled=true` plus a port-forward) is the +recommended path and is covered on the deployment page. Two paths exist +for a Hubble that must run outside the cluster. + +**Direct Server URL** (graph, schema, data, and Gremlin; no PD +discovery): leave in-chart Hubble off, expose Server +(`server.service.type` NodePort/LoadBalancer, or Ingress), and run a +standalone Hubble image with: + +```properties +pd.enabled=false +server.direct_url=https://: +``` + +Mount the file at `/hubble/conf/hugegraph-hubble.properties` (the image +workdir is `/hubble`). Use HTTPS or a trusted channel for +`server.direct_url`: login sends the Server credentials over that URL. + +**PD discovery** (the outside Hubble asks PD for the Server address): +in-cluster names such as `*.svc` are not reachable from outside, so the +chart provides two knobs. Set `server.advertiseUrl` to the absolute +`http(s)://` URL the outside Hubble will use after discovery; the chart +registers it with PD in place of the in-cluster Service URL. And expose +the PD client Service (`pd.service.type` NodePort/LoadBalancer), which +requires `pd.service.allowInsecureExposure=true` because PD gRPC has no +authentication; restrict who can reach it first (with the chart's +NetworkPolicy on, the render refuses the exposure until its callers are +listed in `networkPolicy.pd.extraIngress`). Then run standalone Hubble +with: + +```properties +pd.enabled=true +pd.peers=: +pd.server=: +``` + +Trade-off: when `server.advertiseUrl` is set, every Server replica +registers that same logical URL and PD returns it to every discovery +client, including an in-cluster Hubble. Leave it empty for the default +in-cluster path, where each Server Pod registers its own IP. + +Local quick test (cluster and Hubble on one machine): port-forward Server +`8080` and PD client `8620`/`8686`, set +`server.advertiseUrl=http://127.0.0.1:8080`, run standalone Hubble with +`--network host` and the properties above, then open Hubble on `8088`. + +### 10 When Gremlin fails with "Could not rebind" + +The error has two causes with different lifetimes. + +**A convergence window after creating a graph.** The Server that handles +`CreateGraph` waits for its own Gremlin binding before returning HTTP 200 +([#3138](https://github.com/apache/hugegraph/pull/3138)), so +create-then-query on the same Server is reliable. Other replicas converge +independently, and until they finish, a Gremlin query routed through the +load-balanced Service to a not-yet-converged replica can fail with a 400 +such as `Could not rebind [g]`. Retry with backoff (the window normally +closes in seconds), use sticky routing or a port-forward for +create-then-verify flows, or poll `/graphs` on each replica before +opening query traffic. Cluster-wide readiness is tracked in +[#3137](https://github.com/apache/hugegraph/issues/3137). + +**A Server Pod that started while PD was rolling.** Gremlin Server +instantiates the graph once at startup; if the PD client cannot connect +at that moment, the Pod passes readiness and serves REST while every +Gremlin request on it fails with `Could not rebind [graph]`, for the life +of the Pod. Its `hugegraph-server.log` names it: + +``` +Graph [DEFAULT-hugegraph] configured at [...] could not be instantiated and +will not be available in Gremlin Server +``` + +After any upgrade that rolled PD and Server together, check Gremlin on +each Server Pod (the image ships no curl, so port-forward each Pod): + +```bash +kubectl port-forward -n hugegraph pod/ 8080:8080 +curl -s --compressed -u "admin:${PASSWORD}" -H 'Content-Type: application/json' \ + -X POST http://127.0.0.1:8080/gremlin \ + -d '{"gremlin":"graph.traversal().V().limit(1).count()","aliases":{"graph":"DEFAULT-hugegraph"}}' +``` + +A healthy Pod answers with `result.data`; delete a Pod that answers +`Could not rebind`. Its replacement binds normally once PD is stable +(measured: 4 of 12 Server starts that overlapped a PD roll hit this, and +every deletion recovered). diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md new file mode 100644 index 000000000..b2d4feb22 --- /dev/null +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md @@ -0,0 +1,282 @@ +--- +title: "Deploy on Kubernetes with Helm" +linkTitle: "Deploy on Kubernetes (Helm)" +weight: 4 +search_keywords: + - helm + - kubernetes + - k8s +--- + +### 1 Overview + +The Helm chart deploys a distributed HugeGraph cluster on Kubernetes: PD, Store, and Server, plus the optional +Hubble UI. It lives in the main repository under +[`helm/hugegraph`](https://github.com/apache/hugegraph/tree/master/helm/hugegraph). + +| Component | Workload | Default replicas | Purpose | +|-----------|----------|------------------|---------| +| PD | StatefulSet + PVC | 3 | Placement driver: a Raft group tracking Stores and partitions | +| Store | StatefulSet + PVC | 3 | Graph data storage (HStore) | +| Server | Deployment | 3 | Gremlin and REST query layer | +| Hubble | Deployment | 0 (off) | Web UI, enabled with `hubble.enabled=true` | + +A distributed HugeGraph cluster has a startup contract (no `init-store` on Server, every Server using PD for graph +metadata, Store waiting for a PD quorum, one PD REST secret shared by three readers). The chart encodes that +contract so operators do not have to; the details are in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#chart-details). +Day-2 work (NetworkPolicy details, disaster recovery, scaling, safe Store +rolls, running Hubble outside the cluster) is on the +[operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/). + +```mermaid +flowchart LR + subgraph Kubernetes cluster + PD[PD StatefulSet
Raft group of 3, PVC] + Store[Store StatefulSet
3 replicas, PVC] -->|register, heartbeat| PD + Server[Server Deployment
3 replicas] -->|metadata, discovery| PD + Server -->|gRPC read/write| Store + Hubble[Hubble Deployment
optional UI] -->|discovers Servers via PD| PD + end + Client[Client / hugegraph-client] -->|REST / Gremlin| Server +``` + +Startup order is enforced by the chart, not by the operator: PD pods elect a leader first, each Store pod waits in +an init container until a majority of PD peers report ready, and Servers keep restarting their storage wait until +Stores have registered. A fresh install converges without manual steps. + +### 2 Prerequisites + +- Kubernetes 1.23 or later (the chart renders `autoscaling/v2` and `policy/v1`) +- Helm 3; the `--reset-then-reuse-values` flag mentioned under Upgrade needs Helm 3.14 or later +- Dynamic volume provisioning: a default StorageClass, or an explicit `storageClassName` for PD and Store +- Memory for nine JVMs in the default topology; see the resource note under Install + +The chart requires component images that carry the PD readiness endpoint and PD REST authentication (both merged +for the release after 1.7.0). The default image tags already point at builds that include them; 1.7.0 images are +not supported. + +### 3 Install + +#### 3.1 Get the chart + +The chart is not published to a chart repository yet, so install it from the source tree: + +```bash +git clone https://github.com/apache/hugegraph.git +cd hugegraph +``` + +#### 3.2 Install with default values + +First confirm `kubectl` points at the intended cluster and that it can provision volumes. PVCs stuck in `Pending` +for want of a StorageClass are the most common first-run failure: + +```bash +kubectl config current-context +kubectl get storageclass +``` + +Then install: + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace --wait --timeout 15m +``` + +`--wait` makes Helm block until every workload is ready, which for a distributed cluster is the signal that PD +elected a leader, Stores registered, and Servers came up; without it `helm install` returns as soon as the objects +are created. A fresh cluster normally converges in a few minutes; the 15 minute timeout leaves room for slow image +pulls. + +Two defaults to know before going further: + +- **No resources are set.** Every pod is BestEffort and each JVM sizes its heap against total node memory. That is + fine on a single node; on a multi-node cluster the heaps oversubscribe the nodes and pods abort. Use + `values-cluster.yaml` or set `resources` per component for anything beyond a laptop. +- **Image tags track `latest`** until the next HugeGraph release is published, with `pullPolicy: Always`. Pin tags + or digests for production. + +#### 3.3 Topology presets + +The chart ships three values files: + +| File | Topology | Intended use | +|------|----------|--------------| +| `values.yaml` | 3 PD + 3 Store + 3 Server | Default; preferred anti-affinity, auth on, Hubble off | +| `values-single.yaml` | 1 + 1 + 1 | Single-node development and CI; smaller PVCs | +| `values-cluster.yaml` | 3 + 3 + 3 | Production starting point: JVM heap and resource settings, PD/Store PodDisruptionBudgets, `required` anti-affinity, NetworkPolicy on | + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace \ + -f helm/hugegraph/values-single.yaml --wait --timeout 15m +``` + +`values-cluster.yaml` is a starting point, not a capacity guarantee: recalculate resources for your graph size and +traffic. One of its numbers deserves a note: it requests 5Gi and limits at 8Gi of memory per Store, well above the +1Gi heap, because the Store's RocksDB caches live outside the JVM heap (a 4Gi limit OOM-killed Stores after about +1 GB of data). Scale both numbers with data size. The full parameter reference (every component, probe, +scheduling, and Secret knob) is kept in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration). + +#### 3.4 Verify the install + +```bash +helm test hugegraph --namespace hugegraph +``` + +Read the generated admin password and call the API: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin -o jsonpath='{.data.password}' | base64 --decode)" +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +``` + +The commands above assume the release is named `hugegraph`; with another name, substitute the release-prefixed +resource names (`kubectl get svc,secret -n ` lists them). The post-install notes printed by +`helm install` repeat these commands with the right names filled in. + +### 4 Authentication and Secrets + +Authentication is on by default and the chart manages three Secrets. Each credential resolves in the same order: +an `existingSecret` you created wins, then an inline value, then a random value generated at install time. + +| Secret | Key | Used for | Bring your own with | +|--------|-----|----------|---------------------| +| `-admin` | `password` | Server admin account, Hubble login | `server.auth.admin.existingSecret` | +| `-auth-token` | `token_secret` | JWT signing key shared by all Server replicas | `server.auth.token.existingSecret` | +| `-pd-auth` | `secret-key` | PD REST authentication, read by PD, Server, and Hubble | `pd.auth.existingSecret` | + +To manage a credential yourself, create the Secret before installing and point the matching `existingSecret` value +at it; the chart never modifies a Secret it did not create. Value constraints: the admin password must not contain +newlines, carriage returns, or backslashes; the JWT key must be at least 32 bytes; the PD secret must be printable +ASCII. Invalid values are rejected at render time or by the startup wrapper, not silently truncated. + +Chart-managed Secrets are kept on uninstall and reused by a later install under the same release name. + +
+Rotation and caveats + +- The admin password is applied only when the auth metadata is first created, so changing the Secret later does not + rotate an existing cluster's password. Rotate it through the Server's auth API instead. +- Rotating the PD REST Secret rolls PD, Server, and Hubble together on the next `helm upgrade`, which keeps their + copies in step. Template-only pipelines (`helm template`, GitOps renderers) cannot see live Secrets, so there the + rotation-detecting annotation is inert. +- All three Secrets exist even if you only ever read one: the post-install notes print the exact `kubectl get + secret` commands for the admin password and the PD secret. +
+ +### 5 Health checks and startup order + +PD exposes two health endpoints, and the chart deliberately uses both: + +- `/v1/health` answers 200 as soon as the REST listener is up. It never consults Raft, so it cannot see a lost + quorum. +- `/v1/ready` answers 503 until the PD Raft group has a leader, so it reports quorum, not just a live process. + +The chart puts PD **readiness** and the Store init-container wait on `/v1/ready`: a Store only starts once a +majority of PD peers are quorum members, and a PD that lost its leader drops out of Service endpoints until a +leader is back. PD **startup and liveness** derive from the replica count (`pd.livenessPath` overrides the +choice). With more than one PD they stay on `/v1/health` on purpose: a PD that merely lost its leader is still a +healthy Raft member, and restarting it would make the outage worse. A single PD is the exception and derives to +`/v1/ready`: it has no election to lose, and one that steps down for good, as after a failed Raft snapshot on a +full disk ([apache/hugegraph#3222](https://github.com/apache/hugegraph/issues/3222)), would answer `/v1/health` +forever while serving no writes; the kubelet restarts it instead. + +Server startup gets a matching budget: the image would normally kill a Server still starting after 120 seconds, so +the chart derives `HG_SERVER_STARTUP_TIMEOUT_S` from the startup probe (450 seconds by default) and raises a lower +configured probe budget to that floor. Raise `server.probes.startup` if your storage takes longer to come up, and +the image budget follows. + +### 6 Enable the Hubble UI + +Hubble is off by default so API-only clusters stay lean. Enable it on a running release: + +```bash +helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values --set hubble.enabled=true +``` + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-hubble 8088:8088 +``` + +Open `http://127.0.0.1:8088` and log in as `admin` with the admin password from Section 3.4. Hubble discovers the +Servers through PD, so the cluster operations view works without extra wiring. Hubble serves plain HTTP: reach it +through a port-forward or an HTTPS-terminating Ingress, never directly from an untrusted network. Running Hubble +outside the cluster is possible but takes more wiring; see +[Reaching Hubble](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#reaching-hubble-pick-one-path) in +the chart README. + +### 7 Upgrade + +```bash +helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values +``` + +`--reuse-values` keeps the release's existing overrides; without it the upgrade rebuilds the release from chart +defaults. Any upgrade that changes a Pod template rolls that workload once. Points worth planning around: + +- **The first upgrade after a fresh install rolls PD, Server, and Hubble once**, when the Secret-tracking + annotations first observe the install-created Secrets. Store is untouched. +- **Store rolling updates advance on a listener check, not on shard recovery**, so the controller can replace the + next Store while the previous one is still rejoining its shard groups. For a production image roll, set + `store.updateStrategy.type=OnDelete` and replace Store pods one at a time. `Up` in PD is not the between-pods + check: PD marks a Store `Up` at registration, before it restores partitions, and a stopped Store stays `Up` + until a 300 s keep-alive expires. Wait for the replaced Pod to be `Ready`, then confirm every shard group + reports its full shard count with one leader; the full procedure is on the + [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/). PD restarts are one pod at a time + either way, and `pd.updateStrategy.type=OnDelete` gives the same manual control for maintenance windows. +- **PVC sizes cannot be changed by upgrade**: Kubernetes forbids changing StatefulSet `volumeClaimTemplates`, so an + upgrade with a new `storage.size` is rejected in full. The chart README documents the resize procedure for + StorageClasses that support volume expansion. + +Scaling Server up and down is a values change (`server.replicas`, or `server.hpa`). Scaling **PD or Store down is +not**: Raft and shard membership are persisted, deleting pods does not reconfigure them, and a 3-to-1 PD shrink +permanently loses quorum. The chart rejects an upgrade whose replica count is below the live StatefulSet; the +manual drain-then-scale procedure is in the chart README under +[Scaling](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#scaling). + +### 8 Uninstall + +```bash +helm uninstall hugegraph --namespace hugegraph +``` + +Two kinds of state survive on purpose. PersistentVolumeClaims created by the StatefulSets are kept (Kubernetes +behavior); delete them explicitly once the data is no longer needed. The chart-managed Secrets are also kept, so a +later install under the same release name comes back with the same credentials. + +### 9 Limitations + +- `networkPolicy.enabled` renders one NetworkPolicy per component that admits only the release's own traffic; it + is off by default and on in `values-cluster.yaml`. It matters because the chart disables the PD Raft IP + allowlist in-cluster (pod IPs change; the allowlist resolves peers once at boot and then blocks them), so the + policies are what restricts the raft and gRPC ports. They need a network plugin that enforces NetworkPolicy + (Calico, Cilium, kind v0.25 or later, k3s), and every outside client, the Ingress controller included, must be + listed in `networkPolicy..extraIngress` or the render fails. Details are on the + [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/). +- Image tags track `latest` until the next HugeGraph release publishes versioned images; pin tags or digests for + anything long-lived. +- After creating a graph, other Server replicas can lag for a short window before they serve queries for it, so a + query routed to a not-yet-converged replica can fail with an error such as `Could not rebind [g]`. Retry with + backoff, or use sticky routing for create-then-query flows; cluster-wide graph readiness is tracked in + [#3137](https://github.com/apache/hugegraph/issues/3137). +- Store recovery is operator-triggered on current builds: re-replication after Store loss, leader balancing, and + partition rebalancing run only when called through PD's REST API. A Store whose volume is lost can be recovered + in place only on images carrying + [apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234) (merged 2026-09-24, in no release yet); + on earlier images, keep a Store's PVC when replacing its Pod. The Disaster Recovery section of the + [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/) is the runbook. +- No TLS termination inside the cluster, no backup tooling, no Operator, and no bundled monitoring stack. + +### 10 Troubleshooting + +| Symptom | First check | +|---------|-------------| +| Store pods stuck in `Init:0/1` | PD is not ready: `kubectl logs -c wait-for-pd`, then the PD pods | +| PVCs stay `Pending` | No default StorageClass, or the provisioner is down: `kubectl get sc` | +| Pods OOM killed or restarting on multi-node | No resources set, JVM heaps sized to node memory: use `values-cluster.yaml` | +| Query fails right after creating a graph | Replica convergence window: see Limitations above | + +Longer walkthroughs for each case are in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#troubleshooting). diff --git a/data/version_routes.json b/data/version_routes.json index bd7e4351e..6207b4b3c 100644 --- a/data/version_routes.json +++ b/data/version_routes.json @@ -715,6 +715,20 @@ "1.3": null, "1.0": null }, + "cn:quickstart/hugegraph/hugegraph-helm": { + "latest": "cn/docs/quickstart/hugegraph/hugegraph-helm/", + "1.7": null, + "1.5": null, + "1.3": null, + "1.0": null + }, + "cn:quickstart/hugegraph/hugegraph-helm-operations": { + "latest": "cn/docs/quickstart/hugegraph/hugegraph-helm-operations/", + "1.7": null, + "1.5": null, + "1.3": null, + "1.0": null + }, "cn:quickstart/hugegraph/hugegraph-server": { "latest": "cn/docs/quickstart/hugegraph/hugegraph-server/", "1.7": "cn/docs/quickstart/hugegraph/hugegraph-server/", @@ -1422,6 +1436,20 @@ "1.3": null, "1.0": null }, + "en:quickstart/hugegraph/hugegraph-helm": { + "latest": "docs/quickstart/hugegraph/hugegraph-helm/", + "1.7": null, + "1.5": null, + "1.3": null, + "1.0": null + }, + "en:quickstart/hugegraph/hugegraph-helm-operations": { + "latest": "docs/quickstart/hugegraph/hugegraph-helm-operations/", + "1.7": null, + "1.5": null, + "1.3": null, + "1.0": null + }, "en:quickstart/hugegraph/hugegraph-server": { "latest": "docs/quickstart/hugegraph/hugegraph-server/", "1.7": "docs/quickstart/hugegraph/hugegraph-server/",