test(bench): add enrichment completion harness and provider simulator (M0)

[debpalash]

Part of #33. This covers the first M0 step: the harness, the simulator, the settings docs, and a first local baseline. It does not tick the M0 checkbox. That still needs a controlled-environment PostgreSQL baseline, API (k6) numbers, and the first job-type selection.

What's added

• scripts/bench/provider_simulator.py: a deterministic local provider. Latency can be fixed, uniform, or lognormal, with a seed, plus configurable 5xx and 429 rates. Each outcome is a pure function of (seed, domain, attempt), so request scheduling does not change it. /stats exposes request and status counts.
• scripts/bench/enrichment_harness.py: creates N workbook rows with a declarative HTTP provider column pointing at the simulator, then enqueues run_workbook with the same payload shape as POST /run. It runs a real QueueService: claim → killable job child → pebble provider worker → httpx → result commit. It reports JSON and a short summary: completions/sec, end-to-end and per-cell p50/p95/p99, errors, provider attempts/429s, CPU/RSS, and the identity block (commit, hardware, Python, DB, worker config, workload). SQLite is the default. Set BENCH_DATABASE_URL to use a disposable Postgres; the harness claims and deletes only its own tagged rows.
• SSRF handling: there is no supported dev override for check_url, so the guard is not changed. Benchmark processes load scripts/bench/_shim/sitecustomize.py via PYTHONPATH. It exempts exactly the simulator's loopback origin (scheme+host+port) and refuses any non-loopback origin. It also registers the generated manifest through the normal compile/register path and records per-cell timings. Nothing ships in the app package.
• docs/operations/worker-concurrency.md: every worker slot and concurrency knob, with defaults, ranges, what it bounds, how to tune it, and how to run the harness.
• docs/benchmarks/enrichment-baseline.md plus raw JSON, linked from the plan's M0 row and checklist.
• tests/test_enrichment_bench_harness.py: simulator determinism, the shim's loopback-only rule, and a 5-row end-to-end smoke run (about 3 s).

Baseline (developer laptop, regression reference only, not capacity)

Identity: commit 9188fc4 (app code) · Apple M2, 8 cores, 16 GB, macOS 26.4.1 arm64 · CPython 3.14.5 · SQLite 3.50.4 temp file / PostgreSQL 18.4 local (superuser, so RLS was bypassed) · Redis reachable · 1 worker process · defaults WORKER_CONCURRENCY=1, WORKER_MAX_ACTIVE_PER_WORKSPACE=2, row_concurrency=12, provider_workers=8, provider_timeout=10, max_providers=8, retry_passes=1 · 1 declarative HTTP column, verify off, cost 0 · simulator seed 42.

Run	DB	Rows	Simulator	Config	Done	Completions/s	E2E p50/p95/p99 ms	Cell p50/p95/p99 ms
A	SQLite	500	50 ms fixed	defaults	500	70.1	4227/6852/7121	87/132/702
B	SQLite	2000	50 ms fixed	defaults	2000	80.6	13248/23697/24587	85/130/139
C	SQLite	2000	0 ms	defaults	2000	214.6	5347/8950/9243	40/47/94
D	SQLite	2000	lognormal 50 ms σ0.5, 1% 5xx, 2% 429	defaults	1999	61.7	16487/30490/32078	93/170/234
E	SQLite	2000	50 ms fixed	provider_workers=12	2000	112.9	9826/16948/17532	84/94/133
F	SQLite	2000	50 ms fixed	4 workbooks/2 ws, slots=4	2000	189.2	6882/10063/10389	122/195/2482
G	Postgres	500	50 ms fixed	defaults	500	45.1	6204/10672/11099	183/219/662
H	Postgres	2000	50 ms fixed	defaults	2000	37.8	20794/49865/52167	214/411/469

Findings worth follow-up issues

1. row_concurrency above the DB pool size deadlocks a run. With row_concurrency=32 on SQLite (the UI allows up to 64), the job child's event loop blocks on a SQLAlchemy QueuePool checkout (size 5 + overflow 10). A 300-row run finished 0 cells in 240 s. Postgres has the same failure mode above 30 connections.
2. Postgres per-cell cost grows with workbook size. Cell p95 went from 219 ms to 411 ms (500 → 2000 rows) at the same provider latency. This is relevant to M3's bounded queries.
3. Declarative provider HTTP errors (429/5xx) reach the cell as no_data. Error detail and rate-limit signal are lost.
4. Throughput is gated by provider_workers < row_concurrency and by batch-synchronous row execution. Matching them gave +40%.

Tests

• uv run pytest -q tests/test_enrichment_bench_harness.py tests/test_declarative_providers.py tests/test_declarative_array_projection.py tests/test_queue_claim.py tests/test_queue_concurrency.py tests/test_queue_load.py tests/test_job_process_runner.py tests/test_http_column.py → 52 passed in 4.3s
• uv run pytest -q → 2218 passed, 139 skipped in 74s

🤖 Generated with Claude Code