[Bug] A single-node PD never recovers raft leadership after a failed periodic snapshot (disk full), even once the disk is freed; /v1/health stays 200 and /v1/ready hides the error state

[SebastianGruza]

Bug Type (问题类型)

others (availability)

Before submit

• 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

Environment (环境信息)

• PD and Store :latest of 2026-09-18 (master after fix(server): wait for the stores at startup instead of exiting on a cold start #3210), jraft-core 1.3.13; chart of feat(helm): add HStore deployment chart #3218 / feat(helm): add HStore deployment chart hugegraph/hugegraph#221, values-single.yaml (1 PD + 1 Store + 1 Server) on a 3-node k3s
• PD data on its own 512 MB volume (loop-backed ext4 as a hostPath PV), so ENOSPC hits PD only
• The same outcome was seen on 2026-09-15 on VMs with PD 1.7.0 and a root disk at 98 %

Expected & Actual behavior (期望与实际表现)

Expected: with a full disk PD stops accepting writes and tells the probes so; once the disk is freed it resumes on its own, or at least /v1/health fails so an orchestrator restarts it.

Actual (two passes, results/pd-disk-full-single in the validation repo):

t	event	/v1/ready	/v1/health	clients
0	volume 49 % used (raft log preallocation)	200, STATE_LEADER	200	schema writes 202, reads 200
+60 s	dd fills the volume to 100 %	still 200 for ~1 min	200	unchanged
+116 s	the periodic raft snapshot fails on ENOSPC: RaftException: ERROR_TYPE_SNAPSHOT from SnapshotExecutorImpl.doSnapshot → RaftStateMachine.onError → Raft lost leader	503, {"ready":false,"state":"STATE_FOLLOWER","isLeader":false}	200	every gRPC/REST call: IllegalStateException: Leader is not ready (63 in 5 min); schema writes through the Server time out; reads through the Server still 200
+241 s	filler removed, volume back to 49 %	503 for the next 180 s, not a single election attempt in the log	200	writes still fail
+422 s	PD pod restarted	200, STATE_LEADER after ~30 s	200	writes 202; oracle over the whole run: 0 mismatches, 0 acknowledged writes lost

Analysis

• The trigger is a failed periodic snapshot (NodeImpl.handleSnapshotTimeout), not a log write, so PD looks healthy for about a minute after the disk fills and drops at a random moment relative to the fault.
• After onError, jraft 1.3.13 steps the node down and never starts an election again (the node stays in its error state until the process restarts); in a one-member group nobody else can become leader, so PD stays leaderless for good, even after the cause is gone. With 3 PD the other two elect a leader and the symptom is one PD permanently out of the quorum.
• RaftStateMachine.onError sets ProbeView(STATE_ERROR, false), but onLeaderStop, which jraft calls a moment later, unconditionally overwrites it with ProbeView(STATE_FOLLOWER, false); /v1/ready therefore shows "a follower without a leader", not an error. /v1/health is return "", 200 as long as Jetty is up, so no liveness probe ever restarts such a PD.

Proposal

1. onLeaderStop / onStopFollowing must not overwrite STATE_ERROR (or getRaftStatus() asks node.getNodeState()), so that /v1/ready tells the truth.
2. /v1/health answers 503 when the raft node is in STATE_ERROR: that state is terminal until restart, so it is the right liveness signal; on a plain loss of leader (3 PD) health stays 200 as today.
3. Longer term: after a snapshot error PD could retry the snapshot and, once the cause is gone, re-initialise its raft node without a process restart; at minimum an upgrade note for single-node installations.

Repro: cluster/k3s_pd_disk_full.py plus pd_fill_capture.sh in https://github.com/SebastianGruza/hugegraph-validation; the PD log of pass 2 in results/pd-disk-full-single/pass2/.

Vertex/Edge example (问题点 / 边数据举例)

No response

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

No response