test(durability): assert shared-nothing part replication and disk-loss recovery

[tdakkota]

The suite deploys three nodes on per-node file PVCs — the shared-nothing case
storage.cluster.private_backend exists for — and nothing set it. That is oteldb/oteldb#1262
reproduced by the suite that is supposed to catch it. The oteldb side landed in oteldb/oteldb#1264;
this wires it into the cluster under test and, more importantly, re-runs the disk-loss experiment
to find out what the flag actually changes.

The empirical answer

The README claimed, as a durability note, that a node returning under the same identity with an
empty disk "is not auto-restored from replicas within the query window", and the disk-loss spec
asserted only self-healing plus availability.

That note is wrong once private_backend is on, and the mechanism it implied was never the
reason. Measured on kind (3 nodes, backend: file, replicationFactor: 2, flushInterval: 10s),
wiping a node's PVC and letting the StatefulSet return it:

{"msg":"Joining storage cluster", ... ,"private_backend":true}
{"msg":"bootstrap: gained shard with no local engine","shard":"default/_s0"}
{"msg":"bootstrap: shard signal loaded","shard":"default/_s0","signal":"log"}
part_sync: {"passes":8,"mirrored":1,"copied":9,"copied_bytes":557}

The node had all 9 of its part objects back roughly 20s — two maintenance ticks — after becoming
Ready, and every record ingested before the wipe stayed queryable throughout. The recovery is
bootstrapGainedTenants finding a shard the node owns but has no engine for, cluster/partsync
mirroring that shard's backend objects from a peer, and the engine loading them. There is no
query-window wait involved: the pass runs on the engine's maintenance tick, whose cadence is
spec.engine.flushInterval.

Without the flag, in the same cluster, it never recovers — not "not within the query window",
but structurally never:

{"msg":"Joining storage cluster", ... ,"private_backend":false}
{"msg":"bootstrap: gained shard with no local engine","shard":"default/_s3"}
{"msg":"bootstrap: gained shard with no local engine","shard":"default/_s3"}
... once per maintenance tick, indefinitely; the data dir stays empty

bootstrapShard calls syncParts, which no-ops for a backend the node believes its peers can
already read, so no bucket index appears locally, so no engine is created, so hasAnyEngine stays
false and the next tick rediscovers the same shard. Observed for 3 minutes; the loop has no exit.

The more alarming finding is upstream of the wipe. With the flag off, flushed parts are on one
node each despite replicationFactor: 2 — the ring replicates the head, and only partsync
replicates parts:

oteldb-0  /default/_s1/{metrics,traces}, /default/_s3/metrics   35 files
oteldb-1  (nothing)                                              0 files
oteldb-2  /default/_s0/logs                                     13 files

With the flag on, each flushed part lands on replicationFactor nodes, as intended.

None of this is visible from any API the cluster exposes: writes still reach every replica through
the primary-authoritative path and every query is answered from whichever owner has the data. So
the suite now asserts the mode directly.

Changes

• manifests/cluster.yaml sets spec.cluster.privateBackend: true.
• internal/nodeadmin — client for a node's admin API (/api/v1/storage, port 8090). The operator
  does not publish that listener on the Service, which suits these specs: "did this node flush a
  part, did this node mirror a peer's" are per-node questions a Service-level view hides behind
  the read fan-out.
• durability_test.go asserts (a) every node reports part-sync stats, which only a node in
  shared-nothing mode does, and (b) a flushed part reaches two nodes. (a) is the one that fails
  loudly if the operator drops the field — see the blocker below — rather than silently testing a
  shared-store cluster that is not one.
• The disk-loss spec now asserts recovery: it waits for a flushed part that already has a peer copy,
  wipes a node holding one, then asserts that node mirrors parts back, holds them again, and that
  the pre-disruption telemetry is still queryable.
• README's durability note replaced with the measured mechanism, both ways round.

Blocker

oteldb/operator has no spec.cluster.privateBackend, and storage.cluster is a reserved
extraConfig path, so there is currently no way to express this through the CR at all — filed
as oteldb/operator#16 with a two-hunk patch sketch. Against oteldb/operator@main the run stops at
kubectl apply with strict decoding error: unknown field "spec.cluster.privateBackend", which is
what CI on this PR shows. Draft until that lands. durability_test.go is the backstop for the
quieter failure mode — an operator that accepts the field and drops it.

Testing

make test-keep against a 4-node kind cluster, with an operator carrying the proposed
privateBackend field: 7/7 specs pass. Run twice back to back — once against a cold cluster (fresh
etcd and PVCs) and once warm — since an earlier iteration of the flushed-part helper waited for a
part count to grow, which a merge folding parts back into one makes unreachable on a warm cluster.
The helper now waits on a state, not a change.

Refs oteldb/oteldb#1262, oteldb/oteldb#1264, oteldb/operator#16.