perf(workbook): select stale refresh cells with bounded SQL

[debpalash]

Part of #33 (cheap fix: move refresh stale-row selection into bounded SQL).

What changed

_stale_row_columns in apps/api/services/workbook/refresh.py used to load all workbook rows and all overlays into Python. It now:

• keyset-paginates workbook_rows.id (WHERE workbook_id = :wb [AND id > :after] ORDER BY id LIMIT :n), page size from the new setting WORKBOOK_REFRESH_SCAN_BATCH_SIZE (default 1000);
• per page, cross-joins the page's rows with a literal UNION ALL set of enrichment column ids (each with its own cutoff and config ordinal) and LEFT JOINs workbook_enrichments on (workbook_id, COALESCE(row.lead_id, row.id), column_id);
• returns only stale (row, column) pairs, so only stale candidates reach Python.

Semantics preserved

A cell is stale exactly as before: overlay missing, status not complete (NULL included), updated_at NULL, or updated_at < now - staleness_ttl_days[target_field or id] (default 30). The execution key is still the linked lead id, falling back to the row id. The output is still ordered by row id, with columns in config order. Every page is scanned, so the result doesn't depend on page size. The setting bounds the work done per query and does not cap how much a refresh covers.

Timestamps: updated_at is naive UTC DateTime on both dialects, so cutoffs are bound as naive UTC. That avoids session-timezone coercion on PG and mismatched string formats on SQLite.

Tenancy: same as before. There's an explicit workbook_id filter, and RLS applies on both tables. The PG test checks that another tenant gets {} for the same workbook id.

The one intentional difference is that enrichment columns with no id (malformed config) are now skipped. Before, they were reported as a None column on every row. Duplicate ids are de-duplicated, which the caller already did.

The ADR docs/adr/20261012-refresh-dependent-and-new-row-cells.md has an addendum covering this.

Tests

• New file tests/test_workbook_refresh_stale_sql.py compares the new function against the old implementation, which is kept verbatim as an oracle in tests/refresh_stale_support.py. It uses randomized workbooks covering fresh, stale, missing, error, pending, NULL status, NULL updated_at and ±1h TTL-edge cells, linked and unlinked rows, per-column TTLs, TTL looked up via target_field, the default TTL, a TTL of 0, and a sibling workbook sharing execution keys. It runs page sizes 1/3/7/40/1000 and also checks the round-trip count under the setting.
• tests/test_workbook_refresh_cell_scope.py: an end-to-end refresh with page size 1 gives the same scope.
• New file tests/test_workbook_refresh_stale_sql_pg.py: the same comparison on PostgreSQL under FORCE RLS as the app role, plus the cross-tenant isolation check. It is added to the CI Postgres job.

Run locally:

• uv run pytest -q -m 'not live and not postgres' → 2235 passed, 134 skipped, 8 deselected
• TEST_DATABASE_URL=postgresql+psycopg://…/<disposable db> uv run pytest -q tests/test_create_merge_pg.py tests/test_entity_identity_pg.py tests/test_pg_tenancy_rls.py tests/test_lead_paging_pg.py tests/test_workbook_refresh_stale_sql_pg.py → 23 passed (local PostgreSQL 18.4)

🤖 Generated with Claude Code