test: verify fenced retries and complete job conservation

docs/features/postgres-multibroker.md

@@ -1403,6 +1403,26 @@ completion-retention suite repeats the authoritative lookup with a zero-sized
 local result cache. These files are part of the
 normal `test:postgres` command and therefore run in every PostgreSQL 16–18 CI
 matrix entry; they are not an opt-in soak.
+
+The public Queue/Worker lifecycle checker retains every processor invocation's
+token, attempt, stage and monotonic start/end timestamps. With no applied failure,
+each ID must execute once. A repeated invocation is accepted only after the
+specifically covered pre-log Progress error: a matching token-bearing Worker
+failure, durable failed/retried events, the next attempt and its failure timeline
+must agree. Repeated tokens, overlapping invocations, recovery/release events,
+missing predecessors and other processor errors fail the checker. A repeatable-read
+snapshot additionally requires one terminal row, one completion tombstone and
+one durable completed event per original ID, with the exact result.
+
+`test/repro-postgres-public-progress-retry.test.ts` holds a real PostgreSQL row
+lock after public Worker claim, observes the blocked Progress request, and waits
+for its configured SQL lock timeout. It proves rollback at attempt zero, an
+applied failure, a new token at attempt one, then one result and log. This
+regression distinguishes legal retry from an unexplained duplicate; it does not
+identify the cause of historical CI43559077874, whose logs lacked that evidence.
+`test/postgres-public-delivery-oracle.test.ts` checks that malformed delivery and
+completion histories are rejected. Runtime retry, lease and heartbeat settings
+are unchanged.
 Concurrent brokers claim each job once under both ordinary and 16-consumer
 high-contention campaigns, stale owners are fenced, broker/client shutdown is safe, remotely
 renewed leases survive disconnect, shared limits and workers work, missed cron

docs/src/content/docs/changelog.md

@@ -214,6 +214,9 @@ head:
 
 ### Testing
 
+- Record the completed retry-delivery and job-conservation test corrections
+  with their isolated gate results, case-preservation checks, skeptic approvals
+  and green source CI evidence.
 - Test assertions now observe real lease ownership and renewal, rate-limit
   admission/refill, concurrency-policy removal, exact DLQ pagination, stored job
   logs, waiting-event IDs and forced closure with a running processor. Fixture
@@ -236,15 +239,6 @@ head:
 - Record the completed test-quality work item with its isolated gate totals,
   controlled-defect and case-preservation proofs, skeptic approvals and green
   source PR CI evidence. The final record changes documentation and state only.
-- Track an intermittent PostgreSQL 17 public-worker processing-count failure
-  observed during PR validation. Its single failed-job rerun passed; a separate
-  reproduce-first work item retains the assertion and records the evidence
-  without claiming a cause or a fix.
-- Track a separate intermittent legacy chaos-accounting test failure in unit
-  CI. Its partial count omits delayed work, but the missing state/operation
-  trace leaves the observed cause unproved. The single unit-job rerun passed;
-  a reproduce-first follow-up preserves the oracle and requires full job
-  conservation rather than a relaxed threshold.
 
 - The CPU-intensive workers guide test "a non-yielding handler starves
   heartbeats and gets recovered" now observes the recovery it claims
@@ -530,6 +524,22 @@ head:
 
 ### Bug fixes
 
+- Fix the legacy chaos test's incomplete job accounting. Delayed retries were
+  omitted and an empty pull ended the drain before retries became due, so a
+  valid 49-completed/51-delayed population failed a half-count threshold.
+  Seeded concurrent operations now check every original ID, exclusive state
+  membership and all counters exactly; regressions cover the delayed partition,
+  a missing retained job and a retry deadline crossing between state reads.
+  Split the race suites into smaller files with individually owned temporary
+  SQLite directories.
+- Fix the PostgreSQL public Worker test's absolute invocation-count assertion:
+  a rolled-back Progress command can legitimately trigger the default job retry.
+  A real SQL row-lock regression now proves that path. Tests require distinct
+  tokens, sequential attempts, matching applied failures and durable history,
+  nonoverlapping processing and exactly one committed completion with the right
+  result. Unexplained repeats still fail with per-attempt diagnostics. This
+  corrects the oracle; the original intermittent CI cause remains unclassified.
+
 - **Harness command cancellation covers child startup.** A command can start
   before `spawn()` returns. SIGTERM in that window previously terminated the
   supervisor before its signal handlers were registered, leaving the detached

docs/testing.md

@@ -212,6 +212,41 @@ Run benchmarks only on native macOS, never in an OrbStack Machine or container.
   Docker Compose is only for functional tests requiring external services and
   must use a unique project name and disposable volumes.
 
+### Concurrent job conservation
+
+The race-condition suites use a separate temporary SQLite directory for each
+test. `test/race-conditions.test.ts` records 200 operations from a test-owned
+seeded generator without changing runtime retry jitter. Conservation compares
+every original ID against mutually exclusive waiting, prioritized, delayed,
+active, completed, failed/DLQ and waiting-children listings and their counters.
+An empty pull does not mean delayed retries have disappeared, and a pull's wait
+argument is not a processing timeout. The suite checks the retained population
+after all operations settle instead of sleeping briefly and requiring only half
+the jobs to appear in a partial count.
+
+A controlled 51-delayed/49-completed history protects that boundary. A separate
+case deliberately removes one original job through the persistence path and
+proves that the checker rejects the missing ID even with 99 completions. Set
+`BUNQUEUE_RACE_TRACE=1` for the seeded operation/outcome trace, attempts, retry
+deadlines and complete state snapshot. The seed reproduces operation choices;
+timestamps and completion order remain part of the captured evidence.
+The snapshot guard reads retry deadlines independently of the state-filtered
+listings: a job becoming due between waiting and delayed reads can be absent
+from both. A controlled-clock regression covers that boundary; only a crossed
+deadline repeats the snapshot, while assertion failures propagate immediately.
+
+The PostgreSQL public Worker lifecycle test correlates tokens, attempts, applied
+Worker failures and durable transition events. Its absolute processor-once
+assertion was invalid when a Progress command rolled back and the default retry
+policy applied. The replacement requires one invocation when no failure applied,
+and permits the covered pre-log Progress retry only with a complete predecessor
+history. Distinct tokens, nonoverlap, contiguous attempts, exact results and one
+durable completion per ID are mandatory. A real row-lock/SQL-timeout regression
+proves the retry; checker mutation cases reject unexplained repeats and duplicate
+completions. See [PostgreSQL validation](features/postgres-multibroker.md#validation).
+The specific historical CI failure remains unclassified because its original
+logs did not retain the necessary per-attempt evidence.
+
 ### Behavioral test oracles and controlled observation
 
 Removing a case needs a coverage map: the surviving test must use the same