docs(runtime): one instance identity for every shared-state feature

website/docs/features/cdc/postgres-replication.md

@@ -304,6 +304,20 @@ Every Spice replica must have its own replication slot. Spice hashes the replica
 |---------------------------------|----------------------------------------------------------------|
 | `SPICE_INSTANCE_ID` env         | Preferred — set it explicitly per replica.                     |
 | `HOSTNAME` / `COMPUTERNAME` env | Fallback — works on Kubernetes where each pod has a distinct hostname. |
+| The host name                   | Last fallback, when none of the variables above is set.        |
+
+This is the [instance identity](../../reference/spicepod/runtime#instance-identity) that every Spice feature coordinating through shared state uses. An empty variable counts as unset. When an instance's identity changes, for example when `SPICE_INSTANCE_ID` is set on an instance that had none, its default slot name changes too: Spice creates a new slot, and the old slot keeps retaining WAL on the source until it is dropped. Find slots that no replica consumes with `SELECT slot_name, active FROM pg_replication_slots WHERE slot_name LIKE 'spice_%'`, then drop a confirmed-abandoned slot with `SELECT pg_drop_replication_slot('<slot name>')`.
+
+:::warning[Upgrading from v2.4.0-rc.2 and earlier]
+
+Earlier versions used the first of `SPICE_INSTANCE_ID`, `HOSTNAME`, and `COMPUTERNAME` that was set, even when its value was empty, and `unknown` when none was set. Later versions skip empty variables and fall back to the host name. Two kinds of instances therefore get a new default slot name, and create a new slot, on upgrade:
+
+- An instance with none of the variables set, such as a `systemd` service. Its old slot's name ends in `_3f484bfe`.
+- An instance whose earlier version selected an empty variable, for example an empty `SPICE_INSTANCE_ID`, or an unset `SPICE_INSTANCE_ID` with an empty `HOSTNAME`. It now uses the next non-empty variable, or the host name. Its old slot's name ends in `_38d394c2`.
+
+The old slot keeps retaining WAL until it is dropped. An instance whose earlier version selected a non-empty variable keeps its slot name.
+
+:::
 
 ### Example: Kubernetes StatefulSet
 

website/docs/features/data-acceleration/snapshots.md

@@ -149,7 +149,11 @@ Dataset 'orders' is not creating snapshots while instance 'spice-1' holds its sn
 
 A lease lasts twice the dataset's snapshot interval, kept between 30 seconds and 24 hours. The interval is `snapshots_trigger_threshold` with `snapshots_trigger: time_interval` and `refresh_check_interval` with `refresh_complete`, and otherwise 10 minutes. Another instance takes over a lease that has gone unrenewed that long, as measured on its own clock, so clock skew between instances does not expire a live lease. A holder whose snapshot fails releases the lease at once. A holder that loses the lease during an upload does not publish that snapshot over a newer one from the instance that took over.
 
-Each instance is identified by the `SPICE_INSTANCE_ID` environment variable, or by its host name when the variable is unset, so a restarted instance resumes the lease it held. Give each replica a distinct `SPICE_INSTANCE_ID`: two processes with the same identity both upload, and each logs a warning naming the identity.
+Each instance is identified by its [instance identity](../../reference/spicepod/runtime#instance-identity): the `SPICE_INSTANCE_ID` environment variable, else `HOSTNAME`, else `COMPUTERNAME`, else the host name. The identity is stable across restarts, so a restarted instance resumes the lease it held. Give each replica a distinct identity: two processes with the same identity both upload, and a process that finds its lease renewed by the other logs an error naming the identity and the snapshot location:
+
+```text
+Another running instance uses SPICE_INSTANCE_ID 'spice-1' at state location 's3://my-bucket/snapshots/'. Give each instance its own SPICE_INSTANCE_ID (for example the pod name).
+```
 
 ### Snapshot location and the Cayenne data tier
 

website/docs/features/distributed-query/index.md

@@ -126,6 +126,14 @@ spiced --role executor \
 
 Specifying `--scheduler-address` implies `--role executor`.
 
+### Instance Identity
+
+Each scheduler and executor registers under its [instance identity](../../reference/spicepod/runtime.md#instance-identity): the `SPICE_INSTANCE_ID` environment variable, else `HOSTNAME`, else `COMPUTERNAME`, else the host name. Nodes on separate hosts or Kubernetes pods already have distinct identities. Processes on one host share its host name, so two schedulers, or two executors, on one host each need their own `SPICE_INSTANCE_ID`. This is in addition to what any two `spiced` processes on one host already need, such as their own instance directory, set with `SPICE_CONFIG_DIR`, because each process holds an exclusive lock on its instance directory.
+
+The identity is stable across restarts, so the scheduler recognizes an executor that restarts under the same identity as the same executor. It logs `Executor '<id>' restarted, so the tasks and shuffle output of its earlier run are rescheduled` and reschedules that work instead of waiting for the earlier run to time out.
+
+In v2.4.0-rc.2 and earlier, a scheduler registered under its `host:port` address and an executor under its advertise address, so two nodes on one host did not need distinct identities.
+
 ### Version Compatibility
 
 Run the same Spice version on every scheduler and executor in a cluster, and upgrade all of them together. A scheduler checks the cluster protocol version that each executor reports when it registers and polls for work, and rejects an executor built for a different Spice version. A rejected executor receives no work. The scheduler logs `Rejecting executor <id>: protocol version mismatch`, and the executor logs a warning like the following on each retry:

website/docs/reference/spicepod/runtime.md

@@ -1158,6 +1158,26 @@ Spice stops at startup and exits with an error in these cases:
 
 v2.4.0-rc.1 did not check the location. After an upgrade from rc.1, an instance whose `runtime.state` credentials are read-only does not become ready until they can write, and an instance whose store does not enforce conditional writes stops at startup.
 
+### Instance identity
+
+Every feature that coordinates with other instances through shared state names the instance with one identity. Spice reads the identity once per process from the first of these that is set and not empty: the `SPICE_INSTANCE_ID` environment variable, the `HOSTNAME` environment variable, the `COMPUTERNAME` environment variable, and otherwise the operating system's host name. The identity stays the same across restarts, so a restarted instance resumes the state it held.
+
+| Feature | Uses the identity for |
+| ------- | --------------------- |
+| [Cluster scheduler](../../features/distributed-query#instance-identity) | Its registration in the shared cluster state. |
+| [Cluster executor](../../features/distributed-query#instance-identity) | Its executor ID when it registers with the schedulers. |
+| [Acceleration snapshots](../../features/data-acceleration/snapshots#several-instances-creating-one-datasets-snapshots) | The holder of a dataset's snapshot writer lease. |
+| [HTTP rate control](#limits-across-instances) | The name of each process's share of the per-window budget, paired with a per-process ID so that a restarted instance does not reuse its earlier run's share. |
+| [PostgreSQL replication](../../features/cdc/postgres-replication#multi-replica-deployments) | The default replication slot name. |
+
+Give each instance that shares state its own identity. On Kubernetes, `HOSTNAME` defaults to the pod name, which is already distinct. Instances on one host share its host name, so two schedulers or two executors started on the same machine each need their own `SPICE_INSTANCE_ID`. When Spice finds another running process with the same identity, it logs an error such as:
+
+```text
+Another running instance uses SPICE_INSTANCE_ID 'spice-0' at state location 's3://my-bucket/spice-state/'. Give each instance its own SPICE_INSTANCE_ID (for example the pod name).
+```
+
+A scheduler logs this error, and its registration is refused, when the scheduler already registered under its identity keeps sending heartbeats. A scheduler restarted after a crash is not reported as a duplicate, because the heartbeat its predecessor left stops advancing. A scheduler also logs the error when a second executor process connects under an ID that a running executor already uses, and a snapshot writer logs it when another process renews its lease. HTTP rate control does not detect a shared identity.
+
 ### State location parameters
 
 These rules apply to `runtime.state.params`, and to the `params` of a feature that sets its own state location, such as [`runtime.scheduler`](#runtimescheduler). A [snapshot location](../../features/data-acceleration/snapshots#configure-snapshot-storage) set with `snapshots.location` takes its parameters from `snapshots.params`. Snapshots stored under `runtime.state` use `runtime.state.params`, unless `snapshots.params` is set, which replaces them.