[Feature] Server needs a storage-aware readiness signal (readiness stays green with zero Stores)

[bitflicker64]

Feature Description (功能描述)

The Server has no endpoint a Kubernetes readiness probe can use that reflects whether it can actually serve graph traffic. /versions (what the #3132 chart uses today) answers 200 as long as the REST layer is up, even when the cluster has no Store at all.

Measured independently by @SebastianGruza on 2026-09-16 (#3132, comment of 19:31 CEST; scripts in SebastianGruza/hugegraph-validation, docs/helm-chart-faults.md scenario 7): with all Stores scaled to 0 on a running cluster, the Servers stayed Running and ready=true with 0 restarts for 150+ seconds, /versions answered 200 throughout, while every GET /graph/vertices/<id> ended in a 500 after the 30 s request bound (#3204). A Kubernetes Service therefore keeps routing traffic to Servers whose every real request fails.

This is the Server-side twin of #3183, which was fixed for PD by #3185's unauthenticated /v1/ready.

Proposal: an unauthenticated, cheap readiness endpoint on the Server (for example /apis/readiness or a flag on /versions) that returns 503 unless:

1. the Server's PD client currently has a usable PD connection, and
2. at least one Store is registered and Up from that Server's view (the graph list from PD plus one cheap Store call is the shape suggested in the feat(helm): add HStore deployment chart #3132 thread).

It must not require Basic auth: a Kubernetes httpGet probe cannot carry a credential without leaking it into the pod spec, which is exactly why #3185 made /v1/ready unauthenticated and why the auth interceptor already excludes /actuator/**, /v1/health and /v1/ready on PD.

Chart side once this exists: server.readinessPath flips from /versions to the new path the same way pd.readinessPath moved to /v1/ready, and a Server with no working storage drops out of the Service instead of serving 500s.

Visual summary

[SebastianGruza]

I'm taking this one, as agreed with the chart author. Design below so it can be rejected before the PR; the code exists locally, the E2E on k3s is running, and the PR will come with the measurements.

Endpoint

GET /readiness, unauthenticated (added to FIXED_WHITE_API_SET next to versions), 200 or 503, JSON without addresses:

200 {"ready":true,"storage":"hstore","reason":"ok","active_stores":3,"answered_store":1844095761336398711,
     "pd_millis":4,"store_millis":6,"cached":false}
503 {"ready":false,"storage":"hstore","reason":"no active store registered in pd","active_stores":0,
     "answered_store":null,"pd_millis":3,"store_millis":0,"cached":false}

A separate path rather than a flag on /versions, because /versions is the chart's startup and liveness probe today and should stay "the REST layer is up"; readiness has a different meaning and a different blast radius. /apis/readiness from the proposal is out, since /apis belongs to ProfileAPI.

Definition of "ready"

1. This server's PD client answers: PDClient.getActiveStores() within the time budget.
2. At least one of the stores PD reports as active answers a direct, cheap call from this server: a node session through HgStoreNodeManager and existsTable("g+v"), i.e. one Table/EXISTS RPC that touches no data. Answering counts, not the answer. Stores are tried in turn; the first one that answers ends the probe.

Why "at least one" and not "all": readiness answers whether this server can serve traffic, not whether the cluster is healthy. During a rolling restart of the stores one is always on its way; requiring all of them would pull every server out of the Service on every roll, which is exactly the flapping the chart author asked about. The zero-store case from #3132 is caught either way. PD's view can be stale (measured: PD showed a store as Up 200 s after kill -9), hence step 2 is mandatory.

Cost and bounding

• One shared budget for the whole probe, readiness.timeout (ms, default 1000): PD and each store get the remaining budget, calls go through an executor with Future.get(remaining), so a hung PD or store yields a 503 with "did not answer within N ms", never a hung probe.
• The result is cached for readiness.cache_ttl (ms, default 2000): kubelet, a load balancer and a human with curl share one PD call and one store RPC per window. 0 disables the cache.
• The storage is shared by every hstore graph in the process, so one graph is probed; a server with no hstore graph (rocksdb, tests) answers 200 with "storage":"embedded".

Where it lives

• hugegraph-hstore: HstoreStorageProbe (pure logic over two functional interfaces, the active-store list and the ping, plus the wiring through HstoreSessionsImpl.getDefaultPdClient() and HgStoreNodeManager); HstoreStore registers a storage_readiness meta handler.
• hugegraph-api: ReadinessAPI (@Path("readiness"), @PermitAll) and StorageReadiness (graph selection, cache, mapping to 200/503), called through the existing HugeGraph.metadata(null, "storage_readiness", timeout) hook. The api module does not depend on hstore or on the store client, and stays that way.
• ServerOptions: readiness.timeout, readiness.cache_ttl.

Tests

• Unit: HstoreStorageProbeTest (9: PD fails, PD hangs within the budget, zero stores, every store fails, first answering store after two failures, hung stores within the budget, no addresses in the result), StorageReadinessTest (5: cache, TTL 0, probe exception as 503 with a reason, budget passed through, copy isolation).
• E2E on a three-node k3s with the chart from feat(helm): add HStore deployment chart #3218 / feat(helm): add HStore deployment chart hugegraph/hugegraph#221, with the same harness as the 2026-09-16 battery (oracle before the fault, load through the Service, 1 s sampling of /readiness, /versions, the Ready condition and the Service endpoints on every server pod): baseline, stores → 0 and back, one store deleted, rolling restart of the stores under load, PD → 0 and back, rolling restart of PD, SIGSTOP on the store leading the most partitions. Criteria: with stores → 0 every server leaves the Service; on every roll not a single Ready → NotReady transition. Results go into the PR and to feat(helm): add HStore deployment chart hugegraph/hugegraph#221.

Chart side

server.readinessPath defaulting to /versions, switched to /readiness on images that serve it, exactly like pd.readinessPath; startup and liveness stay on /versions so a server that merely lost its storage is not restarted. The diff for #221 is ready and will go with the measurements.

What I'm not doing: probing every graph and every store, checking partition leaders, an authenticated variant of the endpoint. If any decision above is wrong, now is the cheapest moment to say so.

[SebastianGruza]

PR is up: #3221.

The design held, with two corrections the k3s runs forced (both kept as evidence in the validation repo):

1. The first probe pinged each Store with the store client's existsTable (a raft-routed call over the shared session pool). It flapped on a rolling restart of the Stores and of PD (200→503→200 several times per Server) because a healthy node's ping waited behind a leaving one. Replaced by a direct, local, read-only HgStoreState.getScanState per node over the probe's own channels, fired in parallel, first answer wins.
2. Probing PD on every request made readiness track PD, not storage: PD→0 turned Servers NotReady while the data plane served fine. PD is now refreshed in the background and the last known Store list is used immediately; PD down/slow/restarting only flips pd_reachable.

Final E2E on a 3-node k3s with the #3218 chart, 1 Hz sampling of /readiness, /versions, the Ready condition and the Service endpoints per Server pod, load through the Service: 7 PASS. Stores→0 pulls every Server out of the Service in ~30 s and back ~20 s after they return; rolling restart of Stores and of PD show zero readiness transitions; PD→0 stays 200 with pd_reachable=false; SIGSTOP on the busiest Store shows zero transitions. Scripts, per-scenario JSON and the two superseded designs: https://github.com/SebastianGruza/hugegraph-validation/tree/master/results/issue-3212

Chart side (server.readinessPath, default /versions, mirroring pd.readinessPath) is ready as a diff for hugegraph#221.