Repository navigation
Change: skein-soak-hardening #123
Description
Proposal
Why
Charter Phase 8 requires soak testing and hardening: the platform must survive long
running operation, crashes, and distributed load without losing work or double-executing.
The earlier phases build the durable primitives (durable loops, DurableAgentExecution,
distributed claims, reconciliation); this change proves them under sustained, adversarial
conditions and fixes what the soak reveals.
What Changes
- A soak harness that runs the platform (control plane + project servers + agents) for a
sustained period under realistic load, injecting failures (server kills, network
partitions, lease expiries, claim races, GPU slot loss). - Invariant checks during the soak: no lost work, no double-execution of non-idempotent
operations, no orphaned claims, no leaked leases, no unbounded memory growth. - Crash-recovery tests: kill -9 at arbitrary points; restart; verify recovery from durable
records + actual repo state (not transcripts). - Distributed soak: multiple hosts, multiple repos, cross-repo dependencies.
- A hardening pass that fixes the defects the soak reveals, each with a regression test
that fails without the fix.
Impact
- Affected specs:
skein-soak-hardening(new),skein-control-plane(distributed soak),
project-server(crash recovery),peer-conversation-reliability(delivery under
failure). - A soak harness + invariant checker; no new runtime features (only fixes the soak
reveals).
Tasks
Phase 1: Soak harness
- 1.1 Build a soak harness that runs control plane + project servers + agents for a
sustained period under realistic load. - 1.2 An invariant checker that asserts, continuously: no lost work, no double-
execution of non-idempotent ops, no orphaned claims, no leaked leases, no unbounded
memory growth.
Phase 2: Failure injection
- 2.1 Server kills (kill -9) at arbitrary points; restart; verify recovery from
durable records + actual repo state, not transcripts. - 2.2 Network partitions between control plane and project servers; verify claims are
released and re-claimed, not lost. - 2.3 Lease expiries and claim races; verify exactly-one-winner and no double-booking.
- 2.4 GPU/inference slot loss mid-execution; verify the work is re-queued, not
silently dropped.
Phase 3: Distributed soak
- 3.1 Multiple hosts, multiple repos, cross-repo dependencies; verify correct ordering
and no double-claim. - 3.2 Sustained soak (target duration per charter §21); collect the §17 metrics.
Phase 4: Hardening pass
- 4.1 Fix each defect the soak reveals, with a regression test that fails without the
fix. - 4.2 Document the invariants and the soak results in the change.
Activity