[SPARK-58021][CONNECT] Integrate local server pools with SparkSession

[ericm-db]

What changes were proposed in this pull request?

This is layer 5 of the local Connect pool stack: #57684 -> #57685 -> #57686 -> #57687 -> this PR. JIT warmup is isolated in follow-up #57688. Until the lower layers merge, GitHub shows their cumulative diff; the new review unit here is commit 320bffceae6.

This layer wires the pool into the public SparkSession construction path:

• SparkSession.builder.remote("local[*]") selects the pool when spark.local.connect.pool=true or SPARK_LOCAL_CONNECT_POOL=1 is set;
• pool mode takes precedence over persistent-server reuse;
• the claimed member is released if session construction fails;
• the owning Connect session registers an idempotent stop callback;
• cloned and new sessions initialize their own lifecycle state but do not inherit ownership of the pooled server;
• user documentation describes the experimental single-use pool and purge command; and
• real-server tests cover sequential isolation, teardown, and concurrent cold clients.

Each opted-in application run claims a booted server that has never served another run, starts a replacement in the background, and retires its server when the owning session stops. No server serves two application runs.

Why are the changes needed?

Local Connect startup pays for a JVM and SparkContext in every Python process. Persistent reuse removes that cost but can carry SparkContext-backed state across runs. The single-use pool preserves fresh-server-per-run isolation while moving server startup off the critical path after the initial pool bootstrap.

Keeping this integration separate from storage, lifecycle, acquisition, and warmup makes the user-facing dispatch and session ownership rules independently reviewable.

Does this PR introduce any user-facing change?

Yes, only when the experimental opt-in is enabled. With SPARK_LOCAL_CONNECT_POOL=1 or spark.local.connect.pool=true, a local remote getOrCreate() claims a never-used local Connect server from the pool instead of starting an in-process server. The claimed server is retired when the owning session stops. With the opt-in unset, behavior is unchanged. The mode is POSIX-only, like persistent local server reuse.

How was this patch tested?

The pool suite includes unit coverage for storage, lifecycle, acquisition, release, purge, and platform handling plus end-to-end tests using real local Connect servers.

python -m unittest -v pyspark.sql.tests.connect.test_connect_local_server_pool
python -m unittest -v pyspark.sql.tests.connect.test_connect_session
python -m unittest -v pyspark.sql.tests.connect.test_connect_clone_session

Results on this branch:

• pool suite: 19 tests passed, including both real-server end-to-end cases;
• Connect session suite: 26 tests passed;
• clone-session suite: 8 tests passed; and
• Ruff check, Ruff format check, and git diff --check passed.

The initial clone-session attempt used the host default Python 3.6 worker and failed because that interpreter lacks importlib.resources; rerunning with the test Python 3.11 venv explicitly selected for workers passed all 8 tests.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Fable 5) and OpenAI Codex (GPT-5)

[ericm-db]

This PR is now stacked on #57684, which contains the behavior-preserving local-server launch refactor. Please review that foundation first; after it merges, this PR will reduce to the pool implementation, SparkSession integration, documentation, and pool tests.