Cut backend CI runtime 66%: 16m → 5m with isolated semantic test lanes

.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md

@@ -1,12 +1,13 @@
 # CHUNK MAP: WS-CI-001
 
 | Chunk | Title | Risk | Status |
-|---|---|---:|---|
+| --- | --- | ---: | --- |
 | `WS-CI-001-01` | Parallel Full-Suite Coverage | L1 | Completed and merged in PR #163 |
-| `WS-CI-001-01R1` | Timeout Cleanup Repair | L1 | Implemented and internally reviewed; awaiting hosted repair PR proof/human review |
-| `WS-CI-001-02` | Safe Routing, Cache, and Timing Refinement | L1 | Future; not started and requires separate approval |
+| `WS-CI-001-01R1` | Timeout Cleanup Repair | L1 | Completed and merged in PR #164 |
+| `WS-CI-001-02` | Migrate-Once Backend Test Runtime | L1 | Source-level fixture consolidation in local verification; canonical start required before PR |
 
-Each chunk maps to one PR. Chunk 01 preserves the full suite and every coverage
-gate. Chunk 01R1 is the bounded post-merge response to CodeRabbit's timeout
-cleanup finding. Chunk 02 may be planned only from measured 01 evidence and
-cannot be activated without separate human approval.
+Each chunk maps to one PR. Chunk 01 introduced parallel coverage and chunk 01R1
+repaired timeout cleanup. Measured evidence later proved that repeated per-test
+migrations—not module assignment—were the primary bottleneck. The user directed
+chunk 02 to fix that source problem locally and remove redundant sharding. It
+must not be pushed or opened as a PR before canonical start and internal review.

.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md

@@ -2,149 +2,102 @@
 
 ## Objective
 
-Reduce Backend workflow wall-clock time while preserving complete test execution,
-real service integration, exact-head provenance, and every current coverage and
-branch-protection gate.
-
-## Proposed approach
-
-### 1. Deterministic inventory and partitioning
-
-Add a repository-owned planner with one shared validation implementation.
-Canonical module completeness comes from symlink-safe filesystem discovery of
-`backend/tests/test_*.py`, with an explicit declaration excluding only
-`test_isolated_database_runner.py`. A successful pytest collection supplies the
-canonical per-test-base cardinality signatures and per-module weights; every node must map to one
-discovered canonical module. Collection errors, zero-test modules, foreign paths,
-malformed node IDs, and parameterized-node collisions fail.
-
-Assign every non-excluded module exactly once using deterministic
-largest-weight-first bin packing with lexical tie-breaking. Emit canonical,
-schema-versioned JSON binding schema, actual checked-out tree SHA from
-`git rev-parse HEAD`, ordered normalized module paths, explicit exclusion,
-ordered stable node signatures, collected counts, shard count, weights, and assignments. Its
-SHA-256 digest identifies the exact executable inventory and plan.
-
-The initial weight is collected test count per module. Runtime telemetry may
-replace or augment weights only in the later optional chunk after its provenance
-and stability are reviewed.
-
-### 2. Fast prerequisite job
-
-Retain evidence, install, lint, docstring, isolated-runner self-test, collection,
-and shard-plan validation ahead of the expensive matrix. Upload the immutable
-manifest for the current commit. A collection or planning failure prevents all
-shards.
-
-### 3. Isolated parallel shards
-
-Run four matrix jobs initially. Each validates its actual checked-out tree SHA
-against the manifest, provisions its own PostgreSQL service and independently
-owned migrated database through `run_isolated_tests.py`, starts MinIO, and runs
-only manifest modules through Python argv/subprocess construction rather than
-shell-expanded module strings. It executes the manifest's validated whole-module
-paths and uses repository-owned pytest hooks to record final collection and
-runtime completion in that same pytest process. It requires those exact runtime
-sets to match and their stable test-base cardinalities to match preflight, writes
-a unique coverage file, and uploads a fixed-name bundle containing shard ID,
-tree SHA, manifest digest, collected/completed node IDs and counts,
-allowlisted non-secret metadata, coverage, and SHA-256 of the coverage bytes.
-
-No shard shares a database, role, filesystem artifact, or coverage filename.
-No test is selected by an untrusted PR-provided shell fragment.
-
-### 4. Concurrent API contract proof
-
-Run the existing real API contract E2E command as a separate dependent job after
-fast prerequisites and concurrently with shards. Preserve its isolated database
-and timeout behavior.
-
-### 5. Fail-closed final fan-in
-
-Keep workflow name `Backend` and final job ID `test`, preserving the existing
-`Backend / test` required-check context. Other jobs use different IDs. Run the
-final job after every shard and API proof with `if: always()`. It must reject any
-failed, cancelled, or skipped prerequisite; missing/extra shard; tree-SHA or
-manifest-digest mismatch; malformed metadata; duplicate/omitted module or node;
-collection-count mismatch; missing coverage; coverage-digest mismatch; symlink,
-path traversal, non-regular file, or unexpected path. It accepts exactly four
-fixed artifact names containing tree SHA and shard ID, downloads without wildcard
-surplus, and recomputes canonical bindings rather than trusting repeated labels.
-
-Only after provenance validation does it combine coverage data and enforce the
-unchanged repository-wide and twelve protected coverage reports. The final job
-uploads concise timing and shard-balance evidence.
-
-### 6. Evidence and rollout
-
-Test the planner and fan-in validator locally, including a real repository
-collection-to-plan-to-four-shard-selection-to-fan-in dry run. Statically test the
-workflow in the agent-gate suite and run the implementation PR on GitHub. Compare hosted
-wall time and shard distribution with the PR #161 baseline. Do not claim a
-latency target passed until exact hosted evidence exists.
-
-Matrix job state supplies visible progress; final shard duration/balance metadata
-supplies timing evidence. Live pytest output is not required because the
-unchanged isolated runner intentionally buffers and redacts child output.
-
-## Alternatives rejected
-
-- Test or coverage sampling.
-- Path-based Backend skipping in the first chunk.
-- Shared-database xdist processes.
-- Unpinned marketplace sharding actions.
-- Mutable external timing data without commit/inventory provenance.
-- Allowing the aggregator to pass when an upstream job is skipped or cancelled.
-
-## Architecture and security boundaries
-
-- Planning and execution logic remain CI-only under backend scripts and GitHub
-  workflows; product modules are not imported as orchestration services.
-- Planner input is treated as data and emitted module paths are validated before
-  command construction.
-- Artifacts are accepted only for the actual checked-out tree and expected
-  canonical manifest digest, and coverage bytes are independently hashed.
-- Top-level and job permissions explicitly set `contents: read` plus only minimum
-  run-scoped artifact access; no repository write exists and
-  `pull_request_target` is prohibited.
-- No database URL/password, MinIO credential, environment dump, or runner
-  database metadata outside an allowlisted non-secret schema enters artifacts.
-
-## Verification strategy
-
-- Planner tests: filesystem/node completeness, parameterized nodes, zero-test and
-  collection failure, determinism, exact-once assignment, stable ties,
-  empty/malformed rejection, traversal/symlink rejection, shard bounds, and
-  canonical manifest-digest binding.
-- Fan-in tests: missing, duplicate, extra, foreign-SHA, foreign-digest, malformed,
-  failed, cancelled, skipped, missing/duplicate/foreign node, count mismatch,
-  coverage-byte mismatch, symlink, traversal, unexpected-file, and coverage-file
-  mismatch rejection.
-- Workflow tests: immutable action pins, fixed matrix size, isolated database per
-  job, no shared coverage path, explicit failure propagation, stable final check,
-  unchanged coverage floors, and no path-based skip.
-- Existing runner, agent-gate, backend, coverage, and API E2E tests.
-- Hosted exact-checked-out-tree timing report against PR #161.
-
-## Operator documentation
-
-Update canonical `docs/operations_backend_testing.md` with preflight, shard, API,
-and fan-in diagnosis; fixed artifact contents and retention; whole-workflow versus
-failed-job reruns and stale rejection; parallel runner cost; stable check identity;
-progress semantics; and one-PR rollback.
+Reduce Backend workflow wall-clock time without test sampling, arbitrary
+weighted shards, weaker coverage, or weaker PostgreSQL contract evidence.
+
+## Selected design
+
+### 1. Dependency-based process lanes
+
+`backend/scripts/run_test_lanes.py` assigns every test module exactly once to the
+no-PostgreSQL, schema-contract, control-plane, or execution-plane lane. All four
+processes run concurrently in one job and write private coverage files. The
+three database lanes share one PostgreSQL server but invoke
+`run_isolated_tests.py` with independent temporary databases and roles. Both
+runners emit secret-free heartbeats and bound every lane at 20 minutes.
+
+### 2. Cheap isolation for ordinary tests
+
+`backend/tests/conftest.py` owns the normal database reset. Before each
+database-backed test it disposes pooled connections and truncates all mutable
+public tables in one transaction while preserving:
+
+- `alembic_version`;
+- immutable `actor_profile_migration_state` evidence; and
+- the schema, constraints, and triggers installed by migrations.
+
+The fixture temporarily disables only the five known truncate guards, restores
+the baseline `authority_control` row, and re-enables every guard in the same
+transaction. Route and service modules do not run Alembic.
+
+### 3. Explicit PostgreSQL schema-contract lane
+
+Tests marked `postgres_schema_contract` own whole-schema Alembic transitions.
+They receive a rebuilt head schema rather than the truncate reset and restore
+`head` during teardown. This intentionally slower path is limited to migration
+contracts. Lock, trigger, deferred-constraint, `asyncpg`, and concurrent
+transaction proofs remain on real PostgreSQL.
+
+PGlite is not adopted for the Python suite. The schema migrated in the spike,
+but the socket multiplexer did not close `asyncpg` connections reliably and does
+not guarantee production PostgreSQL concurrency behavior.
+
+### 4. One required GitHub job
+
+The required context remains `Backend / test`. One job provides one PostgreSQL
+service and one MinIO service, then runs in this order:
+
+1. internal evidence, lint, docstrings, and isolated-runner tests;
+2. four semantic processes with exact module inventory validation;
+3. local coverage combine and the destructive real API contract drill;
+4. the unchanged 78 percent repository floor and every protected 90 percent
+   floor.
+
+There is no GitHub matrix, arbitrary planner, timing artifact, or cross-job
+fan-in. Product tests execute exactly once across the lanes; runner lifecycle
+tests execute once in their dedicated prerequisite step.
+
+## Correctness boundaries
+
+- No skipped, sampled, or reclassified product tests.
+- No mutable database is shared between pytest processes.
+- No production database is accepted by the isolated runner.
+- No database URL, password, MinIO credential, or environment dump enters logs
+  or metadata.
+- Destructive migration tests remain serialized by the database-scoped DDL lock.
+- Coverage cannot compensate for a failed test or API contract drill.
+
+## Local verification
+
+- Cross-module smoke: 9 passed in 8.19 seconds.
+- Projects: 235 passed in 179.53 seconds, down from 706–729 seconds.
+- Tasks: 142 passed in 145.89 seconds, down from 535–798 seconds.
+- Converted database modules: 534 passed in 124.21 seconds.
+- Schema/reset contracts: 34 passed in 178.63 seconds.
+- Semantic lanes: 1,826 passed exactly once in 314.40 seconds; 330.70 seconds
+  including process startup and local coverage combine.
+- Previous single process: 1,815 passed in 919.48 seconds; 975.62 seconds wall.
+- Lane inventory and failure-custody tests: 11 passed.
+- Isolated-runner lifecycle: 17 passed in 13.52 seconds.
+- Agent workflow gates: 91 passed.
+- Real API contract drill: passed.
+- Coverage: 87.58 percent global; protected reports range from 90.90 to
+  100.00 percent.
+- Ruff and `git diff --check`: passed.
+
+Hosted timing and exact checked-out-commit proof remain required before claiming
+the GitHub workflow result.
 
 ## Rollback
 
-If hosted sharding is unstable or slower, revert the implementation PR as one
-workflow/script unit to restore the existing sequential job. Do not lower
-thresholds, disable shards, or silently fall back inside the workflow.
+If semantic lanes violate exact execution or resource isolation, revert the lane
+runner and workflow invocation while retaining the central reset. Restore
+correctness through one isolated runner; do not lower coverage, skip tests, or
+add arbitrary weighted shards to hide fixture cost.
 
-## Delivery order
+## Delivery state
 
-1. `WS-CI-001-01`: deterministic parallel full-suite execution and centralized
-   coverage fan-in.
-2. `WS-CI-001-02`: optional measured routing/cache/telemetry improvements after
-   01 has stable hosted evidence.
-
-Implementation does not begin until the reviewed 01 contract receives explicit
-human approval.
+`WS-CI-001-02` supersedes the earlier shard/fan-in implementation. Local
+implementation and proof are complete under the user's direction. Push or PR is
+still prohibited until canonical loop state records an explicit signed start for
+this chunk.

.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/RISKS.md

@@ -1,18 +1,17 @@
 # RISKS: WS-CI-001 - Backend CI Acceleration
 
 | ID | Risk | Severity | Mitigation |
-|---|---|---:|---|
-| R1 | A test is omitted or duplicated | Critical | Filesystem modules plus stable preflight cardinality signatures and exact same-process collection/completion validation |
-| R2 | Coverage is combined from incomplete, foreign, or altered evidence | Critical | Bind fixed artifacts and coverage SHA-256 to checked-out tree, shard ID, schema, and manifest digest |
-| R3 | Upstream failure is hidden by dependency skipping | Critical | Always-run final check explicitly validates every dependency result |
-| R4 | Shards interfere through shared database state | Critical | One isolated migrated database and role per shard process |
-| R5 | Coverage thresholds are weakened | Critical | Preserve exact 78/90 commands and add workflow regression assertions |
-| R6 | MinIO tests run without a real provider | High | Start pinned MinIO in all shards initially or prove a safe module map before narrowing |
-| R7 | One large module controls wall time | Medium | Measure hosted shard duration; consider reviewed node-level split only later |
-| R8 | Parallel jobs cost more runner minutes | Medium | Begin with four shards and compare aggregate minutes with PR #161 |
-| R9 | Artifact names collide | High | Per-commit/per-shard names and strict unique-set fan-in |
-| R10 | Untrusted paths reach shell execution | High | Canonical path validation and argument-array execution in repository script |
-| R11 | Required check identity changes | Critical | Preserve final Backend `test` job and verify in workflow tests/GitHub PR |
-| R12 | Plan scope expands into path-based skipping | High | Defer routing to separately approved 02 contract |
-| R13 | Mutable PostgreSQL tag changes CI behavior | High | Replace `postgres:16` with a reviewed digest pin in 01 |
-| R14 | Parameter display values change across pytest processes | Critical | Execute whole modules; compare exact collection/completion within one process and bind only stable test-base cardinalities across processes |
+| --- | --- | ---: | --- |
+| R1 | A test is omitted or duplicated | Critical | One ordinary pytest process discovers and executes the complete suite exactly once |
+| R2 | Reset leaks data between tests | Critical | Truncate every mutable public table in one transaction before each database-backed test |
+| R3 | Reset damages schema or migration evidence | Critical | Preserve `alembic_version` and actor migration evidence; run destructive migration proofs separately |
+| R4 | Immutable triggers block reset or remain disabled | Critical | Disable only five reviewed truncate guards and re-enable them in the same transaction |
+| R5 | Coverage thresholds are weakened | Critical | Preserve the exact 78/90 commands and workflow regression assertions |
+| R6 | MinIO tests run without a real provider | High | Start one pinned MinIO service before the complete suite |
+| R7 | The child hangs silently | High | Emit a secret-free heartbeat every 60 seconds and enforce a 20-minute child bound |
+| R8 | PGlite diverges from production PostgreSQL | High | Keep asyncpg, locks, triggers, migrations, and concurrency proofs on real PostgreSQL |
+| R9 | Parallel infrastructure returns without evidence | High | Regression-test that the workflow has one job and no matrix, shard tool, or fan-in artifacts |
+| R10 | API drill state contaminates pytest | High | Run the API drill in a sequential isolated-runner invocation |
+| R11 | Required check identity changes | Critical | Preserve the single `Backend / test` job and verify it in workflow tests |
+| R12 | Fixture consolidation changes application behavior | High | Limit implementation to tests, runner supervision, workflow, and documentation |
+| R13 | Mutable PostgreSQL tag changes CI behavior | High | Retain the reviewed digest-pinned PostgreSQL service image |

.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md

@@ -1,9 +1,16 @@
 # STATUS: WS-CI-001 - Backend CI Acceleration
 
-- Phase: post-merge timeout cleanup repair
-- Active implementation chunk: `WS-CI-001-01R1`
-- Proposed first chunk: `WS-CI-001-01`
-- Human direction: prioritize full-suite CI acceleration before
-  `WS-ENG-001-04B`
-- Current gate: exact rebased-head review, repair PR CI, and human review
-- `WS-ENG-001-04B`: remains inactive
+- Phase: local implementation and verification complete; hosted proof pending
+- Canonical active implementation chunk: none for this initiative
+- Local candidate chunk: `WS-CI-001-02`
+- Human direction: keep production code unchanged; optimize only tests,
+  fixtures, runners, CI, and test documentation
+- Verified runtime wins: projects 706–729s to 179.53s; tasks 535–798s to
+  145.89s; converted database modules 534 tests in 124.21s
+- Complete local proof: four dependency lanes executed all 1,826 tests once in
+  314.40s and 330.70s wall, down from 975.62s wall; real API contract passed;
+  global coverage 87.58%; protected reports 90.90–100%; no temporary database or
+  role remained; agent workflow gates and Ruff passed
+- Remaining environment proof: exact checked-out commit on hosted `Backend / test`
+- Current repository gate: `WS-CI-001-02` requires an explicit signed start
+  before push or PR