test(bench): add enrichment completion harness and provider simulator (M0)

docs/benchmarks/enrichment-baseline.md

@@ -0,0 +1,92 @@
+# Enrichment completion baseline (M0)
+
+**Developer-laptop baseline.** These numbers come from one development machine
+with simulated providers. Use them as a regression reference only. They are
+not capacity claims and do not meet the M0 exit gate in
+[the backend plan](../plans/go-python-backend-rewrite.md), which needs a
+controlled environment, PostgreSQL, steady-state duration, and API
+measurements. Raw reports:
+[enrichment-baseline-2026-10-14.json](enrichment-baseline-2026-10-14.json).
+For how to run the harness and what each knob does, see
+[operations/worker-concurrency.md](../operations/worker-concurrency.md).
+
+## Identity
+
+| Field | Value |
+| --- | --- |
+| Commit | `9188fc4d040c186f3495902fe77e0d041deba3bd` (application code; harness files added on top, uncommitted at run time) |
+| Hardware | Apple M2, 8 cores (8 logical), 16 GB RAM |
+| OS | macOS 26.4.1 (arm64) |
+| Python | CPython 3.14.5 (uv-managed venv) |
+| Database | SQLite 3.50.4 (temporary file, WAL); PostgreSQL 18.4 (Homebrew, local socket, superuser role, so RLS policies were bypassed) |
+| Redis | Reachable on localhost (cell broadcasts enabled) |
+| Worker processes | 1 harness worker process (`QueueService`), 1 job child per slot |
+| Defaults used unless noted | `WORKER_CONCURRENCY=1`, `WORKER_MAX_ACTIVE_PER_WORKSPACE=2`, `row_concurrency=12`, `provider_workers=8`, `provider_timeout=10`, `max_providers=8`, `retry_passes=1`, heartbeat 30 s, `run_workbook` timeout 1800 s |
+| Workload | N rows x 1 `enrichment` column on a declarative HTTP provider (manifest -> pebble provider worker -> httpx -> simulator), `verify: false`, `cost_per_lookup: 0`, one provider call per cell |
+| Simulator | Seed 42; latency, distribution, and error rates per row below |
+| Date | 2026-10-14 |
+
+## Results
+
+Throughput is committed completions/sec from enqueue to the last result
+commit; the number in parentheses excludes the job-start overhead. "E2E"
+means enqueue to the cell's result commit (one batch enqueue, so it includes
+queueing). "Cell" is one `enrich_cell` execution: provider call plus commit.
+
+| Run | DB | Rows | Simulator | Config change | Done | Completions/s | E2E p50 / p95 / p99 (ms) | Cell p50 / p95 / p99 (ms) | Wall (s) |
+| --- | --- | ---: | --- | --- | ---: | ---: | --- | --- | ---: |
+| A | SQLite | 500 | fixed 50 ms | defaults | 500 | 70.1 (77.2) | 4227 / 6852 / 7121 | 87 / 132 / 702 | 7.5 |
+| B | SQLite | 2000 | fixed 50 ms | defaults | 2000 | 80.6 (82.9) | 13248 / 23697 / 24587 | 85 / 130 / 139 | 25.2 |
+| C | SQLite | 2000 | fixed 0 ms | defaults | 2000 | 214.6 (230.5) | 5347 / 8950 / 9243 | 40 / 47 / 94 | 9.7 |
+| D | SQLite | 2000 | lognormal 50 ms, sigma 0.5, 1% 5xx, 2% 429 | defaults | 1999 | 61.7 (63.0) | 16487 / 30490 / 32078 | 93 / 170 / 234 | 32.9 |
+| E | SQLite | 2000 | fixed 50 ms | `provider_workers=12` | 2000 | 112.9 (117.3) | 9826 / 16948 / 17532 | 84 / 94 / 133 | 18.0 |
+| F | SQLite | 2000 | fixed 50 ms | 4 workbooks / 2 workspaces, `WORKER_CONCURRENCY=4` | 2000 | 189.2 (203.7) | 6882 / 10063 / 10389 | 122 / 195 / 2482 | 11.0 |
+| G | PostgreSQL | 500 | fixed 50 ms | defaults | 500 | 45.1 (48.1) | 6204 / 10672 / 11099 | 183 / 219 / 662 | 11.4 |
+| H | PostgreSQL | 2000 | fixed 50 ms | defaults | 2000 | 37.8 (38.3) | 20794 / 49865 / 52167 | 214 / 411 / 469 | 53.3 |
+
+Provider attempts equal rows in every run except D, which made 2061 requests:
+1999 returned 200, 38 returned 429, and 24 returned 5xx. The retry pass
+recovered all but one cell. Peak simulator in-flight requests matched
+`provider_workers` per job (8, 12, or 32 = 4 jobs x 8). Job startup (claim,
+child process, provider registry, pool start) took 0.65-0.75 s to the first
+cell. Peak RSS per process was about 160-170 MB.
+
+## Observations to carry into M0/M3
+
+1. **Provider pool and batching set throughput.** With the defaults, 12 rows
+   per batch share 8 provider processes, and each batch waits for its slowest
+   row. Raising `provider_workers` to match `row_concurrency` gave +40%
+   (B -> E). The zero-latency ceiling (C) is about 215/s per job on SQLite.
+2. **Row concurrency above the DB pool deadlocks a run.** With
+   `row_concurrency=32` (allowed by Settings, up to 64), the job child blocked
+   its event loop on a SQLAlchemy `QueuePool` checkout ("limit of size 5
+   overflow 10 reached"). A 300-row run finished 0 cells in 240 s, and an
+   earlier 2000-row attempt finished 14 cells in about 10 minutes. This
+   should be a bug fix (bound or decouple sessions from in-flight provider
+   calls), not a tuning note.
+3. **PostgreSQL is slower than SQLite here and degrades with N.** Cell p95 went
+   from 219 ms (G, 500 rows) to 411 ms (H, 2000 rows) at the same provider
+   latency. Per-cell database work appears to grow with workbook size. Profile
+   this before M3 ("bounded workbook refresh queries").
+4. **Failure detail is lost.** The cell that exhausted retries in D recorded
+   `no_data`. The provider's `http_429`/`http_500` error does not reach the
+   cell, so 429s are not visible per cell and do not count as rate-limit
+   signals.
+5. **Slots scale across workbooks, not within one.** Four slots and four
+   workbooks reached 189/s (F). A single large workbook uses one slot no
+   matter what `WORKER_CONCURRENCY` is set to.
+
+## Reproduce
+
+```bash
+uv run python scripts/bench/enrichment_harness.py --rows 500  --latency-ms 50 --output a.json
+uv run python scripts/bench/enrichment_harness.py --rows 2000 --latency-ms 50 --output b.json
+uv run python scripts/bench/enrichment_harness.py --rows 2000 --latency-ms 0  --output c.json
+uv run python scripts/bench/enrichment_harness.py --rows 2000 --latency-ms 50 --latency-dist lognormal \
+  --sigma 0.5 --error-rate 0.01 --rate-429 0.02 --output d.json
+uv run python scripts/bench/enrichment_harness.py --rows 2000 --latency-ms 50 --provider-workers 12 --output e.json
+uv run python scripts/bench/enrichment_harness.py --rows 2000 --latency-ms 50 --workbooks 4 --workspaces 2 \
+  --worker-concurrency 4 --output f.json
+BENCH_DATABASE_URL='postgresql+psycopg://USER@/opengtm_bench?host=/tmp' \
+  uv run python scripts/bench/enrichment_harness.py --rows 2000 --latency-ms 50 --output h.json
+```

docs/operations/worker-concurrency.md

@@ -0,0 +1,147 @@
+# Worker slots and enrichment concurrency
+
+Workbook enrichment concurrency is bounded at four nested levels. Each level
+caps a different resource, and the effective parallelism is the *minimum*
+along the path, not the product. This page lists every knob, its default, what
+it bounds, and how to tune it. Measure changes with the
+[enrichment completion harness](#measuring-with-the-enrichment-harness).
+
+```
+replicas x WORKER_CONCURRENCY slots            (durable jobs at once)
+  └─ capped per tenant by WORKER_MAX_ACTIVE_PER_WORKSPACE
+     └─ one killable job child per slot         (run_workbook)
+        └─ row_concurrency rows per batch       (asyncio, one DB session per cell)
+           └─ provider_workers pebble processes (killable provider calls)
+```
+
+## Durable queue (per worker process)
+
+| Setting | Default | Range | Bounds |
+| --- | --- | --- | --- |
+| `WORKER_CONCURRENCY` | `1` | 1-64 | Claim slots per worker process (the in-API worker or each `python -m apps.api.worker` replica). Each slot claims one job and runs it in its own killable child process (`apps.api.job_process`). Total job parallelism is `replicas x WORKER_CONCURRENCY`. |
+| `WORKER_MAX_ACTIVE_PER_WORKSPACE` | `2` | 0-64 (0 disables) | Jobs one workspace may have in `processing` at once, across all replicas. PostgreSQL enforces it with a per-workspace advisory lock during the claim; global jobs (no workspace) are uncapped. This is the tenant-fairness bound. |
+| `WORKER_SHUTDOWN_GRACE_SECONDS` | `30` | 0-300 | How long shutdown drains active slots before cancelling (and killing) their job children. |
+| `RUN_INLINE_WORKER` | `1` | `0`/`1` | Whether the API process also runs a worker. Set `0` when standalone worker replicas own processing. |
+| `run_workbook` timeout | 1800 s | code (`JOB_TIMEOUTS`) | Wall-clock limit on one enrichment job; the child is killed on expiry. |
+| Heartbeat interval | 30 s | code | Lease renewal; dead-worker reaping uses it. |
+
+A single `run_workbook` job occupies one slot for its whole run, so raising
+`WORKER_CONCURRENCY` speeds up *several* workbooks (or tenants) running at once,
+not one large workbook. With the default of one slot, a second workbook waits
+until the first finishes.
+
+## Per-run enrichment settings (Settings -> Enrichment performance)
+
+These are stored in the settings database as `ENRICH_<KEY>` (set them in the UI
+or with `PUT /api/settings/enrichment-perf`, admin only). The API snapshots them
+into each `run_workbook` job payload at enqueue time, so a change affects the
+next run, not one already queued.
+
+| Setting | Default | Range | Bounds |
+| --- | --- | --- | --- |
+| `row_concurrency` | `12` | 1-64 | Rows enriched concurrently inside one job. Rows run in batches of this size and each batch waits for its slowest row before the next starts. Each in-flight cell holds one database session. |
+| `provider_workers` | `8` | 1-64 | Size of the pebble process pool for provider calls inside one job child. Caps concurrent provider calls per job; a hung call costs one killed process, not the run. |
+| `provider_timeout` | `10` s | 3-120 | Per provider call; the provider process is killed when it expires. |
+| `max_providers` | `8` | 0-20 (0 = full chain) | Waterfall depth per cell. |
+| `retry_passes` | `1` | 0-5 | Extra passes over cells still in `error` after the main pass. |
+
+Environment fallbacks used when a payload or the settings database lacks a
+value: `WORKBOOK_RUN_CONCURRENCY` (12), `WORKBOOK_PROVIDER_WORKERS` (8),
+`WORKBOOK_PROVIDER_TIMEOUT` (10), `WORKBOOK_MAX_PROVIDERS_PER_CELL` (0).
+
+## Database connection pools (per process)
+
+Every job child creates its own SQLAlchemy engine:
+
+| Backend | Pool |
+| --- | --- |
+| SQLite | SQLAlchemy default `QueuePool`: 5 connections + 10 overflow, 30 s checkout timeout |
+| PostgreSQL | `pool_size=10`, `max_overflow=20` (30 connections), 30 s checkout timeout |
+
+PostgreSQL `max_connections` must cover
+`replicas x (1 + WORKER_CONCURRENCY x per-child pool)` plus the API.
+
+## Tuning
+
+1. **Keep `row_concurrency` below the job child's pool capacity**: at most 12
+   on SQLite (15 connections) and about 25 on PostgreSQL (30 connections).
+   Cells keep their session open across the provider call, and a checkout
+   that has to wait blocks the event loop of the job child. Every in-flight
+   cell then stalls until the 30 s pool timeout fires. The M0 harness
+   reproduced this: `row_concurrency=32` on SQLite finished 14 cells in about
+   ten minutes. The Settings UI currently allows up to 64. Track this as a
+   bug; do not tune around it.
+2. **Match `provider_workers` to `row_concurrency`** for single-provider
+   columns. A provider call is the only work that uses the pool, so with
+   `provider_workers < row_concurrency` part of each batch queues behind the
+   rest. Each provider worker is a full Python process (about 100-160 MB RSS),
+   so per host memory is roughly
+   `replicas x WORKER_CONCURRENCY x provider_workers` processes.
+3. **Raise `WORKER_CONCURRENCY`** when several workbooks or tenants run at
+   once. Keep `WORKER_MAX_ACTIVE_PER_WORKSPACE` below it so one tenant cannot
+   take every slot.
+4. **Use PostgreSQL** for concurrent writers. SQLite serializes writes and is a
+   development backend.
+5. Re-measure after every change with the same harness parameters and record
+   the identity block next to the result.
+
+## Measuring with the enrichment harness
+
+`scripts/bench/enrichment_harness.py` runs the real path: durable queue claim,
+`run_workbook` job child, declarative HTTP provider in the pebble pool, and
+result commit. Calls go to a deterministic local provider simulator
+(`scripts/bench/provider_simulator.py`).
+
+```bash
+# Shipped defaults, 500 rows, fixed 50 ms provider latency, temp SQLite
+uv run python scripts/bench/enrichment_harness.py --rows 500 --latency-ms 50 \
+  --output artifacts/bench/enrichment-500.json
+
+# Fault mix: lognormal latency, 1% 5xx, 2% 429, fixed seed
+uv run python scripts/bench/enrichment_harness.py --rows 2000 --latency-ms 50 \
+  --latency-dist lognormal --sigma 0.5 --error-rate 0.01 --rate-429 0.02 --seed 42
+
+# Override any knob (defaults are the shipped values above)
+uv run python scripts/bench/enrichment_harness.py --rows 2000 --workbooks 4 \
+  --workspaces 2 --worker-concurrency 4 --row-concurrency 12 --provider-workers 12
+
+# PostgreSQL: a disposable, migrated database only
+BENCH_DATABASE_URL=postgresql+psycopg://.../opengtm_bench \
+  uv run python scripts/bench/enrichment_harness.py --rows 2000
+```
+
+The report (`--output` writes JSON, `--json` prints it) includes:
+
+- **Results:** committed completions/sec (from enqueue, and excluding the
+  job-start overhead); end-to-end latency per cell (enqueue to result commit)
+  and cell execution latency at p50/p95/p99; startup time to the first cell;
+  cell status and error counts; job statuses; simulator request and status
+  counts (provider attempts, 429s, 5xx); approximate CPU and peak RSS.
+- **Identity block:** git commit and dirty flag; CPU model, cores, RAM, and OS;
+  Python version; database dialect and version; Redis reachability; every
+  worker and enrichment setting above; workload parameters (rows, workbooks,
+  workspaces, simulator latency, error rates, and seed).
+
+How it works and what it changes:
+
+- Simulator outcomes are a pure function of `(seed, row domain, attempt)`, so
+  identical parameters produce the same provider behavior regardless of
+  scheduling.
+- Benchmark processes load `scripts/bench/_shim/sitecustomize.py` through
+  `PYTHONPATH`. It makes three changes: it exempts exactly the simulator's
+  loopback origin from the SSRF guard (any other URL is still checked), it
+  registers the generated manifest through the normal compile/register path,
+  and it records per-cell timings. It refuses any non-loopback origin. None of
+  this ships in the application.
+- With the default temporary SQLite database, nothing persists. With a
+  supplied database, the harness refuses to start if other jobs are
+  pending/processing. It claims only its own fire-key-tagged jobs and deletes
+  only the rows it created.
+- The email verify cascade is disabled on the benchmark column (it would
+  measure live DNS/SMTP). Pass `--cost-per-lookup` > 0 to route calls through
+  the spend-reservation path.
+- `kill -USR1 <job child pid>` dumps all thread stacks of a stalled benchmark
+  process to stderr.
+
+Developer-laptop numbers are a regression reference only. They are not capacity
+claims; see [benchmarks/enrichment-baseline.md](../benchmarks/enrichment-baseline.md).

docs/plans/go-python-backend-rewrite.md

@@ -164,7 +164,7 @@ process termination and parent-side failure reconciliation.
 
 | Milestone | Deliverable | Exit gate |
 | --- | --- | --- |
-| M0: baseline | Fixed fixtures, provider simulator, representative workloads, resource profiles, and current queue/API measurements | Reproducible baseline with hardware, commit, configuration, and workload identity |
+| M0: baseline | Fixed fixtures, provider simulator, representative workloads, resource profiles, and current queue/API measurements ([harness and settings](../operations/worker-concurrency.md), [developer-laptop baseline](../benchmarks/enrichment-baseline.md)) | Reproducible baseline with hardware, commit, configuration, and workload identity |
 | M1: foundations | Go module, sqlc queries, versioned contracts, telemetry, additive executor routing/fencing, and Compose development setup | Python-only operation still passes; mixed-version claims, RLS, and lease recovery tests pass |
 | M2: one worker | One declarative HTTP enrichment job type in Go, including result/evidence persistence, billing coordination, and progress | Behavioral parity, crash/cancellation tests, benchmark comparison, feature flag, and exercised rollback |
 | M3: execution improvements | Bounded concurrency, transport pooling, shared limits, batching, and bounded workbook refresh queries | Measured improvement without degraded fairness, correctness, or resource bounds |
@@ -241,6 +241,10 @@ PostgreSQL correctness, or reproducible performance testing. Follow
 [CONTRIBUTING.md](../../CONTRIBUTING.md), including DCO sign-off.
 
 - [ ] Record M0 baseline and select the first enrichment job type.
+  - Harness, simulator, and a first developer-laptop baseline are in
+    [benchmarks/enrichment-baseline.md](../benchmarks/enrichment-baseline.md).
+    Still needed: a controlled-environment PostgreSQL baseline, API (k6)
+    measurements, and the first job type selection.
 - [ ] Define domain ownership, task contracts, executor routing, and lease fencing.
 - [ ] Add the Go development skeleton, pgx/sqlc access, and tenant transaction helper.
 - [ ] Preserve Python-only operation and exercise mixed-executor claims.