[SPARK-58021][CONNECT] Add local server pool lifecycle management

[ericm-db]

What changes were proposed in this pull request?

This is layer 3 of the local Connect pool stack: #57684 -> #57685 -> this PR. Until the lower layers merge, GitHub shows their cumulative diff; the new review unit here is commit 76def5665b6.

This layer adds process lifecycle management on top of the storage model:

• janitor rules for dead or timed-out launches;
• retirement of dead, unreachable, idle, and orphaned claimed servers;
• graceful termination followed by SIGKILL escalation;
• cleanup of stale member directories and half-started daemons;
• idempotent release of a claimed member; and
• a forceful pool purge escape hatch.

It does not add background acquisition or SparkSession dispatch.

Why are the changes needed?

Pool members outlive individual Python call frames and can be orphaned by crashes or SIGKILL. Their cleanup rules need to be explicit, idempotent, and testable before attendants and acquisition loops begin invoking them concurrently.

Does this PR introduce any user-facing change?

No. The pool is not wired into SparkSession in this layer.

How was this patch tested?

Expanded LocalConnectServerPoolUnitTests with fake-process coverage for dead and live pending launches, unreachable and idle members, orphaned claims, shutdown escalation, release, and purge.

python -m unittest -v pyspark.sql.tests.connect.test_connect_local_server_pool.LocalConnectServerPoolUnitTests

All 14 tests passed. Ruff check, Ruff format check, and git diff --check also passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Fable 5) and OpenAI Codex (GPT-5)