Migration: facts, derivations, projections

[kensac]

Tracking issue for the move described in docs/agents/architecture-migration.md. That document is the plan and the rules; this issue is the working state.

Phases

• 0 — the layering is enforced. feat: phase 0, the layering is enforced and the migration plan is written down #507. import-linter holds core does not depend on api; core/devseed.py moved to api/. Verified the contract can fail.
• 1 — check types are a registry. Inventory below.
• 2 — one definition of board membership. Needs a person before merge.
• 3 — catalog observations are facts. jobs.relisted_at (feat: a posting its feed puts back is read again, so a closure stops being permanent #504) retires into an append-only log.
• 4 — derivations are content addressed. Verdict identity becomes (input_hash, recipe_id); model becomes provenance. Migrates the verdict cache; needs a person.
• 5 — files move to the shape. Last, not first. Includes untangling the api / api.tasks cycle.

Phase 1 inventory (measured 2026-09-10)

148 occurrences of a check-type literal: 87 in src, 61 in tests. Six values exist: closed, clearance, custom, content, plus extraction and comp, which are read or asserted but never written by src. The only place all six are enumerated is a test.

category	src	tests
READ-FILTER (SQL predicates)	58	25
WRITE (recording a verdict)	13	34
DISPATCH (branching on type)	9	0
DECLARATION (lists, enums, indexes)	7	2

The registry belongs in core/checks.py, which is already half of one: its docstring says the prompt and the response model are one pair per check, and it is the import target of both dispatch ladders.

Per-type behaviour varies on six axes, currently spread across seven files: instructions, result schema, verdict extraction lambda, model routing, staleness rule, and cache topology. A registry entry has to carry all six or the ladders come back.

Findings worth acting on regardless of the phase:

• routers/jobs.py:518-540 and routers/admin.py:1142-1174 are the same ~30-line dispatch ladder, duplicated and drifted: admin has a hash: addressing mode and a with_reason schema switch that jobs does not.
• clearance has no staleness rule at all. It is only ever backfilled when missing, so a clearance verdict is never revisited. That is the same class of defect as the closed verdict being permanent, which feat: a posting its feed puts back is read again, so a closure stops being permanent #504 fixed.
• custom keys its own cache by prompt hash while every other type shares _latest_cache, and its staleness is identity rather than age — which is the entanglement phase 4 removes.
• orm.py:722,731 bake check-type literals into two partial indexes, repeated in alembic/versions/f3a4b5c6d7e8. They must move in lockstep with the registry.

Invariant

openapi.json is canon and tests/test_openapi_current.py fails the build when routes and schema disagree. No phase changes an operation's shape. A frontend repo and a browser extension read that contract.

[kensac]

Phase 0 merged as 17f8cb6 (#507). import-linter holds core does not depend on api in CI; core/devseed.py moved to api/.

Phase 1a open as #510 — the registry exists and both dispatch ladders use it. Takes the 9 DISPATCH sites.

Two things learned while doing it, recorded here because they change later slices:

1. The import contract immediately constrained the design. JobClosedVerdict lived in api/tasks/models.py, and a registry entry in core could not name it. It moved to core/checks.py beside the check it answers for. Expect more of this: anything a registry entry needs has to be reachable from core, and phase 0 now makes that a build failure rather than a discovery.

2. Phase 2 is visible from here. Writing the test for the job route, a bare user_jobs row was not enough to make a job visible — visibility.FAST admits an untouched row only through board_visible. The test had to set a status. That is the same three-definitions problem phase 2 exists to collapse, showing up as friction in an unrelated test.

Next slice, phase 1b: the 58 READ-FILTER sites, check-type literals inside SQL predicates in store.py, verify.py, board.py, board_eligibility.py, visibility.py, filter_insights.py, health.py and others. The two partial indexes at orm.py:722,731, repeated in alembic/versions/f3a4b5c6d7e8, have to move in lockstep.

[kensac]

Phase 1a merged as 0dbc146 (#510). Phase 1b open as #511, with a scope correction: the 58 read-filter sites are three different things, not one job. Only 5 name a set; the rest either name one check deliberately, or (board_eligibility.STRUCTURAL) give the two checks different policies, which is phase 2 and must not be widened mechanically. Details in the PR.

[kensac]

Finding: the clearance check is permanent, and it hides eleven times more than the closed check did

The phase 1 inventory noted that clearance has no staleness rule: it is only backfilled when missing, never revisited. Measured on production:

           verdicts on active jobs    >7d     >30d   oldest      rejected
closed                     58,288  11,560      962  2026-06-09      1,152
clearance                  58,162  11,948    1,502  2026-06-09     12,444

urls whose clearance check has EVER run twice:  1,444
urls whose closed check has ever run twice:     3,737

12,444 active postings are held rejected by a clearance verdict, against 1,152 for closed. A rejected clearance verdict means "requires a security clearance, or will not sponsor" and, for anyone who has not set bypass_sponsorship_filter, it removes the posting from the board through board_eligibility.STRUCTURAL.

That verdict is never re-read. 1,444 urls have ever had the clearance check run twice, out of 58,162 — and most of those will be from a model change rather than a deliberate re-check.

This is the same defect as #504 (a closed verdict was permanent, fixed with a re-listing trigger), with two differences that make it worse:

1. Scale. 12,444 affected postings versus the 1,125 the closed version affected.
2. No trigger exists. feat: a posting its feed puts back is read again, so a closure stops being permanent #504 keyed the re-check on a feed re-listing, which is an observable event. A posting does not emit an event when its sponsorship language changes, so there is nothing equivalent to hook. It needs either an age rule or a re-check when the page content hash changes — and the content hash is exactly what phase 4 makes the derivation key.

Not acted on yet. Recorded here because it is a live product effect, not a refactor concern: a posting wrongly judged to require clearance is invisible to that person forever.

[kensac]

Handoff: where the migration stands, 2026-09-10

Read docs/agents/architecture-migration.md first; it is the rules. This is the state.

Phases

#	phase	state
0	layering enforced	merged 17f8cb6
1a	check registry	merged 0dbc146
1b	one definition of the check set	merged 1ee11d8
2a	board row vs working set, named and pinned	merged 76a180b
2b	move the sweeps' scope off user_jobs	deferred, needs a cutover comparison
3	catalog observations are facts	merged df47474
4	content-addressed derivations	dropped, measured
5	tasks beside api; apply a package	merged 9ff0524, 0c37fbf
6	long files split	in progress: #519 merged, #521 open
7	typed rows	not started

Product defects found on the way, and fixed

These were not refactor concerns. They were live.

• feat: a posting its feed puts back is read again, so a closure stops being permanent #504 A closed verdict was permanent. demote_closed deleted the board row, the sweep took candidates from board rows, and the full run asked for verdicts that PASSED, so nothing re-read the page. 1,125 postings whose feed still listed them were held closed. Fixed by keying a re-check on a re-listing.
• fix: the re-check answers both axes, so a clearance verdict stops being permanent #513 A clearance verdict was permanent, for the same reason one layer up: the sweep asked CLOSED_INSTRUCTIONS and recorded closed alone, while only the first pass ever asked both axes. 12,444 active postings were held rejected on an answer nothing would re-read. The sweep now asks the question the first pass asks; the page is already fetched, so the second axis costs output tokens.
• The internship bar. Not a migration item but the same session: the enabled filter never mentioned internships, so high-paying ones cleared the pay bar legitimately. 159 were on the board.

Where the plan was wrong

Three revisions, all evidence-driven, all recorded rather than quietly edited.

• Phase 2 was the wrong shape. demote_closed, materialize_passing and visibility.FULL were not three definitions of one thing. FULL admits an untouched row only through a branch that never references user_jobs, so an untouched board row makes nothing visible. Those rows carry scope, not membership. Measured before assuming it was expensive: 214 untouched jobs eligible only through a board row, which is not a cost problem.
• Phase 4 was dropped. Its stated goal contradicted a decision already recorded in tasks/filters.py ("a verdict from a different model is not this model's verdict"), and reframed as staleness it measured at 0-3% of verdicts.
• Phase 5 had no real reason until phase 6 gave it one. The layering cannot be enforced while the handlers are a child of the services: import-linter's forbidden skips a target inside its source.

Primitives added

• import-linter in CI, two contracts, both verified to fail on a planted violation
• make migration — bare alembic revision --autogenerate reads DATABASE_URL, which is production
• core/checks.py registry, core/answers.py shapes, api/queue.py primitive, job_listing_events

The one number to know

The tangle between services and handlers is nine ignored imports in three causes, and the largest is four routers calling four helpers (resume_text, writing_style, instructions, locations.store) that live in a handler because that is where they were first needed. That is the next real piece of work, and it is small.

Picking this up cold

The plan is docs/agents/architecture-migration.md. The state is this issue. Every phase ships on its own and leaves the system working. make check is the gate; CI runs the same thing plus the contracts.

[kensac]

State, 2026-09-11

Merged since the last handoff: #524 phase 7 begins and the measurement reframing it, #525 coverage measured, #526 the declared refusal shape and the apply lists.

Open: #529 the jobs write path (recreated from #527, which GitHub closed when its base was deleted on merge). #528 the CI auto-fix fix, deferred by Kanishk pending the AUTOFIX_TOKEN secret.

What phase 7 turned into

Not "typed rows". Every operation declares what it returns, because openapi.json cannot be dropped into a frontend today: 173 of 190 operations return an undeclared object. Typed rows are the mechanism that makes declaring cheap.

Failures are now declared too. api/problem.py holds the one refusal shape and REFUSALS attaches at each include_router, so the schema went from declaring no failure at all to declaring one on 188 operations, from 21 one-line changes.

Four rules the remaining ~170 endpoints need

All in docs/agents/architecture-migration.md, all learned by something breaking:

1. The wire must not move. A dict omits a key it has no value for; a model emits it as null. Use response_model_exclude_none=True where the old dict was conditional. Four tests caught this on PATCH /user/jobs/{id}.
2. A response model must be defined above the route that returns it, or PydanticUserError at import with a message that does not say why.
3. db.query_as(Model, sql, params) for rows. A column the model does not declare raises immediately, which is the point.
4. A SELECT * cannot be typed until it names its columns. 24 of them.

Order of work

By what the frontend calls, not alphabetically. lib/job-tracker/client.ts is the list. The admin surface is most of the remaining count and almost none of the remaining value.

Coverage, first measured

87%, 11,800 statements, 1,528 missed. The shape matters more than the total: core/store.py 59%, tasks/verify.py 70%, tasks/ingest.py 81%, tasks/filters.py 82%, everything else above 85. Both live defects found this session (#504, #513) lived in tasks/verify.py. No threshold gate, deliberately: one now would bless exactly what is thin.

[kensac]

Architecture program resumed from the current implementation state.

Merged #563: phase 7 guards are active and the full API/failure contract is enforced.

Merged #564: removed the experiments router-to-handler dependency. Experiment request construction and scoring now live in a shared service; task orchestration is separate. The import contract now carries one genuine behavior exception, location storage, plus the two intentional worker-runner edges.

In progress as separate reviewable changes:

• location domain service extraction
• first mechanical split of resolve contracts and choice policy

The user_jobs working-set split has been inventoried. It will use an additive table, dual writes, parallel set comparison, and a separate cutover. No data-shape change will merge before its ambiguous populations and paid-selection differences are measured.

[kensac]

Second architecture wave merged:

• refactor: separate location storage from task orchestration #565 separates location normalization/storage from batch orchestration. The service-to-handler import contract now carries only the two intentional worker-runner edges.
• refactor: separate resolution contracts and choice policy #566 extracts resolution contracts and shared message-choice policy.
• refactor: separate resolution read models #567 extracts resolution history and review-rate read models.
• refactor: give report taxonomy a shared home #568 gives the report taxonomy one shared domain home.
• refactor: separate resolution commands #569 extracts resolution commands with explicit owner/actor identities.
• refactor: separate person-state writes from the jobs router #570 gives user_jobs person-state writes one service used by jobs and apply.
• refactor: separate resolution queue construction #571 extracts resolution ranking, item producers, and queue composition. The resolve router is now a 160-line HTTP adapter.

All merged changes passed static/type checks, six test shards, corpus tests, generated checks, and secret scans. Their OpenAPI documents were byte-identical before merge.

A generic projection framework was investigated and rejected: only board_visible has the full rebuild, lock, freshness, and deduplicated-refresh lifecycle. The other projection-like stores have materially different semantics, so a base abstraction would currently add ceremony.

Next blocking phase: split user_jobs into person-authored state and a rebuildable per-user working-set projection. The additive/dual-write/parallel-compare migration is designed, but digest population, analytics vocabulary, and irrecoverably ambiguous all-default legacy rows require explicit product semantics before schema cutover.

[kensac]

Architecture migration progress:

• refactor: separate user task status route #572 extracted user task-status HTTP handling from the jobs router.
• Extract job upload route boundary #573 extracted direct job uploads into a focused router while preserving OpenAPI and behavior.
• Share derivation input limits #574 centralized compensation and verification input caps across production tasks and experiment builders, with a parity test that detects drift.

All three changes passed the full CI matrix and are merged. The next active slices are job-report router extraction and a typed immutable experiment-step contract. The working-set and catalog-fact schema migrations remain intentionally gated on the documented product-semantics decisions rather than encoding ambiguous behavior into storage.

[kensac]

Architecture migration progress:

• Extract job report route boundary #575 extracted person-filed job reports into a focused router.
• Type experiment step contracts #576 replaced untyped experiment step dictionaries with an immutable typed contract and step-owned deployed-result loaders.
• Extract job explanation route boundary #577 extracted on-demand explanation handling into a focused router.
• Define verification request recipe #578 introduced a verification-owned request recipe shared by production verification paths and experiments.
• Extract job detail read model #579 extracted the single-job detail read model.
• Centralize structured batch specifications #580 centralized strict structured-response BatchSpec construction across all nine response sites while leaving embeddings and domain policy separate.
• Extract person job command routes #581 extracted person-state patch/delete HTTP adapters around the existing service boundary.
• Extract job board read routes #582 extracted board options/list reads and reduced jobs.py to the ordered router aggregator.

All changes passed the full CI matrix and are merged. The safe modularization and derivation-contract tracks are complete. The next schema-level phases remain gated on the previously documented product-semantics choices for working-set membership and source-observation truth.

[kensac]

Documentation is reconciled in #583 and merged. The phase table now reflects completed registry, layout, and named long-module work; the architecture guide names the current person-state, experiment, and structured-batch owners; and the unresolved phase 2b/phase 3 product semantics are recorded explicitly so no migration can silently invent them.

[kensac]

Phase 2b resumed with the approved rule that automated discovery is working-set state and explicit interaction is person state. #587 adds the working-set relation and a nullable explicit-interaction marker without changing reads or classifying legacy rows. Explicit patch, upload, and accepted mail decisions stamp the marker; automated board materialization and mail inference do not. Full CI passed and the PR is merged.

[kensac]

Phase 2b progress: PR #591 is merged after full green CI. It adds an admin-admitted, resumable legacy split backfill with a fixed database cutoff, bounded atomic checkpoints, task-claim fencing, explicit legacy_unknown accounting, continuation generations after failure/cancellation, post-commit task events, and admission gating on a fresh same-release worker eligible for the task kind. No production backfill was run. The next slice is the read-only shadow comparator required before any read cutover.

[kensac]

Phase 2b progress: PR #592 is merged after full green CI. The admin-only read report compares legacy and proposed pair membership, AI eligibility, the full stale re-verification population, and scheduled users from one snapshot; it separates known mismatches from legacy_unknown rows and returns digest comparison as cannot_tell because the new working set has no admission timestamp. No production SELECT or write was performed. Read cutover remains gated on running the backfill, observing at least one complete filter cycle, and collecting the report's production evidence.