[Bug] PD reports "Started" and keeps serving after failing to open its RocksDB (lock held by the previous instance): every request fails, /v1/health stays 200, no retry, no exit

[SebastianGruza]

Bug Type (问题类型)

others (availability)

Before submit

• 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

Environment (环境信息)

PD master 83ef9f3f (the code path is unchanged since 1.7.0: HgKVStoreImpl.init), single PD, Java 17. Second occurrence: a 3-node PD cluster restarted by systemd during a load; the new JVMs started before the old ones released the RocksDB lock, all three came up half-initialised, stores lost the PD leader, servers timed out, and the cluster stood still until an operator deleted the LOCK files by hand.

Expected & Actual behavior (期望与实际表现)

Expected: a PD that cannot open its KV store either retries the open (the lock is transient during a restart race) or exits non-zero so the supervisor restarts it.

Actual (lab, results/pd-halfalive-lock):

15:19:10 ERROR HgKVStoreImpl - Failed to open RocksDB from ./pd_data/rocksdb/
         org.rocksdb.RocksDBException: While lock file: ./pd_data/rocksdb//LOCK: Resource temporarily unavailable
15:19:10 ERROR HgKVStoreImpl - Failed to open data file,{}
15:19:13 INFO  HugePDServer - Started HugePDServer in 5.856 seconds
lock released at 15:19:50; three minutes later:
  GET /v1/health   -> 200
  GET /v1/ready    -> 503 {"ready":false,"state":"STATE_UNINITIALIZED","isLeader":false}
  GET /v1/members  -> 401 {"status":-1,"error":"Unauthorized"}   (auth cannot read anything either)
  gRPC and raft ports listening; stores: Failed to get the PD leader
  "Failed to open RocksDB" logged exactly once: no retry, no exit

Analysis

• HgKVStoreImpl.init() catches the PDException from openRocksDB() and only logs it (HgKVStoreImpl.java:66-80); this.db stays null. Spring Boot finishes startup, Tomcat, gRPC and the raft node come up, and every KV access later fails, surfacing as "Unauthorized" on REST and "leader not ready" for the stores.
• The lock error is the normal outcome of a restart race: RocksDB's LOCK is an fcntl lock held until the previous JVM exits, and a supervisor that restarts on the shell wrapper (or a manual start right after stop) lands in that window. jraft's own log storage in pd_raft/log has the same lock; when that one is held as well the process eventually exits, so the half-alive state needs the raft side to open while the KV side fails.
• With /v1/health hard-coded to 200 (see [Bug] A single-node PD never recovers raft leadership after a failed periodic snapshot (disk full), even once the disk is freed; /v1/health stays 200 and /v1/ready hides the error state #3222) nothing external restarts such a PD; a 3-node cluster where all three hit the window stays down until an operator intervenes, and deleting LOCK files is the wrong fix when the old process is still alive.

How to reproduce (lab, no systemd needed, repro_pd_lock.sh)

1. Copy a PD data directory; from another process hold an fcntl lock on pd_data/rocksdb/LOCK (Python fcntl.lockf) for 40 s.
2. Start PD on that directory while the lock is held.
3. Release the lock. PD never recovers: /v1/ready 503 STATE_UNINITIALIZED, REST 401, health 200, process alive.

Proposal

1. HgKVStoreImpl.init() retries the open for a bounded time when the failure is a lock conflict (the previous instance is shutting down), then rethrows; the caller lets Spring startup fail, so the process exits non-zero and the supervisor restarts it cleanly.
2. Until the store is open, /v1/health answers 503 (the same direction as [Bug] A single-node PD never recovers raft leadership after a failed periodic snapshot (disk full), even once the disk is freed; /v1/health stays 200 and /v1/ready hides the error state #3222: health should reflect a terminal error state, ready reflects leadership, feat(pd): add quorum-aware /v1/ready endpoint and raft gauges #3185).
3. A note in the operations guide that the LOCK files must never be deleted while a previous JVM is alive.

Related: #3222 (single-node PD stays leaderless after a snapshot error, health still 200), #3185, #3189.