Skip to content

Pre-#211 delivery: accept the routing backfill, make On the main reading entry, and consolidate main for the storage move #219

Description

@holden

Outcome

One tested main that holds the accepted routing foundation and a working On → subject reader, with every remaining branch, worktree and database dependency accounted for, so that #211 may begin. This issue owns everything between the 27 September audit and #211's entry checkpoint. It relocates nothing.

Finish line = the entry checkpoint #211 inherits from #193: accepted main plus at most one designated integration branch; all local changes and evidence accounted for; migration-relevant restore defects addressed; preserved unaccepted code explicitly identified. Plus the owner's requirement, new here: On is the main reading entry, replacing Define, working locally on a safe dataset, with subjects linking to their distinct namespaces.

What this issue delivers and what it does not. Corrected, independently accepted code; a working On → subject reader proven on an isolated dataset; and a consolidated main handed to #211. It does not enable namespace addresses on the owner's working installation: that needs the reviewed population and reviewers, which #211 defers past relocation. That enablement is CP4, a linked follow-up issue opened at CP3, and #219 never describes itself as completing full Stage 2 acceptance.

The implementer, issue breakdown and PR sequence are open. Keep the foundation already built; do not start another architecture exercise. The prelaunch scope review (local: data/audits/2026-09-27-stage2-independent-review/PRELAUNCH-SCOPE-REVIEW.md) applies: no legacy-URL compatibility, no new lifecycle administration, no publication work.

Starting state, verified 27 September 2026 21:45 UTC

Item State
main 2a851e8 (#215)
#208 recovery repair 7860b9f, mergeable, A−. CodeRabbit Major on target_config/1 unfixed; two stale lines in docs/routing/stage-1-foundation.md
#216 backfill, stacked on #208 0415e5b, B−. The two audit defects (P1, P2) are still present: 0415e5b changed only the export reader and the task's argument parsing. A session is live in its worktree (dictionary-issue-194-stage-1-218d99)
#202 curated opening 7711e31, accepted A− on 26 September, unmerged, owner's merge decision
#210 curation runtime cb2e236, not accepted, no independent audit. Its two migrations are already applied to devils_dictionary_v2
#218 exemplar items (#212) 4839c6f, unrelated to routing; decided on its own merits, not by this issue
Consolidation branch codex/consolidation-2026-09-27 697ac8a Stale: predates #215, the last three #208 commits and #216
devils_dictionary_v2 main's schema plus #210's 20260927114948 and 20260927131023; routing and curation tables present and empty; nothing backfilled
Processes dev server 4007 SIGSTOPped since 25 September with ngrok --subdomain=wordhoard still pointed at it; #202 preview on 4017; scratch cluster on 5433 (/Volumes/LLM Models/dictionary-stage2a, 22 rehearsal copies)
Internal disk 5 GiB free; the breakdown and reclaim candidates are in the #211 comment
Independent audit local data/audits/2026-09-27-stage2-independent-review/: REPORT.md, PRELAUNCH-SCOPE-REVIEW.md, IMPLEMENTATION-HANDOFF.md, the probes in independent_backfill_test.exs, logs and artifact comparisons
The owner's example in the corpus Exactly one entity labelled Mars: object 1831413, stored kind concept, Q111, the planet. Seven lexemes with slug mars. No Mars deity exists; it is a clearly identified fixture. Mercury likewise: one entity, the planet

Re-verify all of this before writing anything. Conversational claims are not merge state.

A. Bring the backfill to acceptance

Work on #216's branch only in coordination with the live session there (or after it has committed and pushed and gone idle). Do not create a second version of the same fix.

A1 — P1, stale class evidence escapes the guard. Backfill.stale_pins/1 checks only the pins the evaluator emitted, and Policy.evidence_for/2 pins only matched paths; the run classifies from the exported graph. Required: every graph record the walk visited, matched or not, and every record that was missing at export time and now exists, is validated for currency before a record is classified. Either pin them all in the evaluator result or re-read the batch's graph subset from source_records and compare it with the export. A change defers the record as evidence_changed and requires a fresh decision and a fresh review fingerprint. Regressions: changed unmatched ancestry (the audit's Polish probe), newly available ancestry, contradictory evidence.

A2 — P2, a refused allocation leaves a permanent override. In Backfill.confirm/3 the override is written before Pages.ensure/3 and Ledger.allocate/3; available/2 never reads page lifecycle; classification_decisions refuses DELETE, so the stray override is permanent history. Required: confirmation is atomic per record: after any refusal, decision, page, path and ledger state equal what they were before the record was processed, the record's item still says refused with its reason, and the batch continues. Checking lifecycle in available/2 and wrapping override → page → allocate in an explicit SAVEPOINT / ROLLBACK TO SAVEPOINT via Repo.query! is one implementation; a nested Repo.transaction rollback is not, because it aborts the whole batch (Stage 1 decision 7: refusals are per record). Concurrency behaviour under two callers is part of the acceptance, not only the happy path. Regressions: retired page (the audit's probe), :target_has_other_role, :target_kind_mismatch, a path taken between available/2 and allocate/3, a canonical appearing between them. Each asserts decision, page, path and ledger state unchanged after the refusal.

A3 — Documentation. Correct docs/routing/stage-2/backfill.md lines 22, 35 and 75, the README's "against the evidence they reviewed", and the ADR §8 atomicity statement's coverage. Keep the audit's probes as tests, not as prose.

A4 — #208. Fix target_config/1: drop every inherited endpoint key (socket_dir, socket, endpoints, hostname, port) before merging a URL target, then resolve. Test: a URL without a port under DD_DATABASE_PORT=5433 reaches 5432. Update docs/routing/stage-1-foundation.md lines 140 and 278. This tool is what #211 will use to prove the move across clusters.

A5 — Evidence. Focused routing tests and mix precommit on a private MIX_TEST_PARTITION. On a fresh copy on 5433 restored from the frozen devils_dictionary_stage2r_c1, re-run docs/routing/stage-2/rehearsal/backfill.exs: repeat-run identity, SIGKILL resume, exact restore. The P1–P3 projection evidence and the catch-up measurements may be reused only if the final diff touches nothing the projection depends on (Absorb.*, materializers, resolver, fingerprints); state that check against the actual merged diff, not against the plan. No full-corpus re-projection is needed for this issue; say so and why.

A6 — Reassessment. An independent reading of the corrected heads against REPORT.md. Record the grade on #194. Accepted means: both probes pass as regressions, the rehearsal gates hold on the corrected code, and the documentation matches the code.

B. On is the main reading entry

B1 — Reconcile the ADR. Amend docs/adr/0004-public-routing.md §4 and §6:

  • /on/:slug is the everyday lexical entry, resolved exactly as /define/:slug is today (Lexicon.lookup/1, Lexicon.WordPage), for any lexeme. It needs no pages row: there are 1.5 million lexemes and an On page cannot depend on someone authoring an overview. That reconciles the owner's requirement with §4's "lexical availability cannot depend on someone writing an overview".
  • An authored overview stays optional and layers onto the same URL: a pages row with role overview whose path is /on/<routing slug> (the on editorial prefix in priv/routing/namespaces.json), allocated through the ledger only when authored. No second content or approval system; the page–composition binding stays the deferred Stage 4 migration.
  • /define/:slug is removed and every internal link moves in the same PR. No redirect layer: nothing has been public. /words/:id/:slug stays as the exact lexeme.
  • URL ownership. /on/:slug is the aggregate reader for everything the lexeme slug reaches, as /define/:slug is today: several lexemes share a slug by design (C++, C+ and c; Mars, mars, MARS). An exact selection of a word, from search or from a card, navigates to /words/:id/:slug, the existing identity-carrying route, and survives reload. The slug alone never identifies a word, so On never claims to.
  • Authored overview precedence. When an overview page exists, its current revision renders above the lexical aggregate on the same URL; with no lexeme behind it, the overview renders alone (200).
  • Overview ↔ lexical association is identity, never a shared label or slug (28 Sep re-audit, correction 2). An overview is loaded only by its allocated address: /on/:slug resolves the requested path through the ledger, and nothing re-slugifies a displayed lemma to find one. It is presented as the treatment of the lexical aggregate only when its current revision holds a supplies_lexical_material membership naming one of the aggregate's lexemes (or one of their senses). An overview whose address collides with an unrelated lexical slug is shown as a separate, labelled choice on that page, never as one treatment. From a lexical page (/on/:slug or /words/:id/:slug), the overviews linked are exactly those whose membership names one of the page's lexemes, at their allocated path in the current mode; the overview links back to each exact word it names. Exact selections keep their identity through navigation and reload (/words/:id/:slug, drawers included). The overview's path is Address.build("on", Policy.slug(lemma)); where that differs from the lexeme slug (c-plus-plus vs c), the lexical page links to the overview's allocated path and the overview links back to the exact word. No silent redirect between the two spellings, and never a different word or an unrelated overview chosen by a shared slug.
  • Subject pages live at /people/:slug, /organizations/:slug, /places/:slug, /events/:slug, /works/:slug, /concepts/:slug, /nature/:slug, /subjects/:slug: eight explicit routes, never a catch-all prefix, so account, evidence and operational paths are untouched. They resolve only through Routing.Resolver, which reads the ledger and nothing else. /entities/:id/:slug stays as the exact-identity route for a subject with no allocated address.
  • Reading modes. On current main, Resolver.decide/4 answers :unavailable for any page that is not published and Resolver.link/1 returns :error for it; the backfill creates only drafts and publication is Stage 5. So the reader needs one explicit internal mode, shared by link generation and direct namespace requests: Resolver.resolve(raw, mode:) and Resolver.link(page_id, mode:) with :public (today's behaviour, the default) and :internal (a draft page with a canonical path resolves and links, marked as draft in the UI). Internal mode is on in dev and test configuration and for an authenticated internal contributor; it never changes publication_state, approvals or the ledger. Tests prove public mode still withholds every draft.
  • Reading-mode eligibility, applied everywhere (28 Sep re-audit, correction 1). The mode comes only from trusted configuration (:internal_reading, set in config/dev.exs and config/test.exs and refused by a test in prod.exs, runtime.exs and config.exs) or from an authenticated internal contributor or reviewer; no request parameter can grant it. Public mode shows only published pages; internal mode shows published and draft (marked as draft). withdrawn content is withheld in both modes, and lifecycle rules (merged, split, retired, tombstones) are identical in both. The same eligibility applies to authored overviews, to their members, and to aliases and equivalent spellings, which answer with the destination's outcome in the same mode: a public request never redirects to a draft. Existing public tombstone semantics are unchanged. Configuration enabling internal mode for every local request is a development convenience: a dev server exposed through a tunnel exposes drafts, so the tunnel is closed first (D4).
  • Standards rationale: schema.org and Wikidata inform which family a subject belongs to (evidence, via classification-rules.json); they do not prescribe the URL taxonomy, which is ours. Say this once in the ADR and stop.

B2 — Routes and behaviour.

Request Public mode Internal mode
GET /on/:slug, lexeme(s) found 200: the word page (headword, senses, sources, quotations, culture, examples, the thing) plus the Subjects section; title "On "; an authored overview, if any, above same
GET /on/:slug, no lexeme, no overview 404 with did-you-mean (existing behaviour); never a subject substituted by label same
GET /on/:slug, overview only 200 if the overview is published, else 404 200 for a published or draft overview (draft marked); withdrawn 404
GET /on/:slug, lexeme(s) and a draft or withdrawn overview 200: lexical content only, the overview withheld 200 with the draft overview (marked); a withdrawn one withheld
GET /on/:slug, an overview whose lexical membership names none of the slug's lexemes 200: the lexical aggregate, with the overview as a separate labelled choice (if visible in the mode) same
GET /<family>/:slug, canonical, published 200 subject page 200
GET /<family>/:slug, canonical, draft 404 (:unavailable) 200 with a draft mark
equivalent spelling (case, Unicode form, trailing slash) 301 to the canonical if the destination is published, else the destination's outcome (404) 301 if the destination is published or draft
alias of a moved or merged page 301, one hop, same destination rule 301, same destination rule
missing 404 404
retired / tombstone 410 410
invalid request (bad encoding, dot segment, NUL) 400 400
corrupt ledger state 500 with diagnostics, logged 500
choice / split a plain list of successors; no design work, no such data exists same
GET /entities/:id/:slug, GET /words/:id/:slug unchanged unchanged
Enter in search (go) Lexicon.lookup/1 hit → /on/<lexeme slug>; miss → results same
a selected word result /words/:id/:slug (exact identity) same
a selected subject result Resolver.link(page_id, mode) when it answers a path, else /entities/:id/:slug same

Every internal link to a subject goes through one helper that asks the resolver in the current mode and falls back to the /entities route. No caller derives a namespace from a stored kind or label, and no caller rebuilds a slug.

B3 — Subjects section contract. Two sources, kept distinct on the page:

  1. Curated membership is authoritative. When an overview page has a current revision, its page_memberships in stored order supply the curated content: discusses_subject members are subjects, editorial_association members are shown as associations (Putin/poutine is an association, never identity), supplies_lexical_material members are lexical. A member whose name differs from the page title is still a member. Withdrawn or ineligible content is withheld, not replaced. Actual approvals are read, never inferred.
  2. Discovered candidates are labelled as such. The union of %WordPage{}.thing (concept and disagreement), Encyclopedia.candidates_for/2's may_refer_to set, and registry entities whose object_names.name or entities.preferred_label equals the lemma after NFC and case folding. Deduplicated by object identity, never by label; a candidate already curated is not shown twice.

Each subject is one card. Pages of role subject and edition count (the backfill creates edition pages in /works). State comes from the current classification_decisions row and the page, read in one bounded query per On page (limit, stable order: addressed first, then family, then label; locale en; active identities only), measured on corpus-scale data before merge because NFC/case-folded matching is not indexed today.

Card state Link Shown
address allocated, page resolves in the current mode canonical path family badge, label, description; "draft" mark in internal mode
address allocated, draft, public mode /entities/:id/:slug "not yet public"
mapped, no address yet /entities/:id/:slug "no public address yet"
ambiguous or unmapped /entities/:id/:slug "awaiting classification review" with the candidate families
no decision /entities/:id/:slug "unclassified"

Nothing draft is presented as public; the section reads state, it never writes it. The page–composition binding (ADR §4) stays deferred: this issue renders authored overview revisions and existing memberships, and does not claim Stage 4. The curated opening (#202) keeps its own development-only fixture reader; wiring Curation.Published into the web layer is named here as unfinished, not delivered.

B4 — Slugs. The On route accepts the lexeme slug; an authored overview's path uses Routing.Policy.slug/1. The two can differ (C++ → lexeme slug vs c-plus-plus). Fixtures for C++, C+, c, Polish/polish, capitalization collisions and a combining-mark label each have a passing outcome under B1's precedence rule: the aggregate On page lists every word the slug reaches, the exact word is reachable at /words/:id/:slug, an authored overview is reachable at its allocated path and linked from the lexical page, and nothing is silently redirected or substituted.

B5 — Wireframes.

/on/mars                                              ┌ search ─────────┐
On Mars                                               └─────────────────┘
────────────────────────────────────────────────────────────────────────
[headword · pronunciation · forms]           ┌ the thing ─────────────┐
                                             │ Mars  Q111 ↗           │
Senses (by source, as today)                 │ fourth planet …        │
  1. …                                       └────────────────────────┘
  2. …
Quotations · Culture · Examples (as today)

Subjects                                          3 subjects, 2 addressed
┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
│ NATURE               │ │ SUBJECTS  (fixture)  │ │ WORKS                │
│ Mars                 │ │ Mars                 │ │ Mars                 │
│ fourth planet from … │ │ Roman god of war     │ │ 2012 album           │
│ → /nature/mars       │ │ → /subjects/mars     │ │ no public address yet│
└──────────────────────┘ └──────────────────────┘ │ → /entities/…        │
                                                  └──────────────────────┘
/nature/mars
NATURE · Mars                                  On Mars ↗   Q111 ↗
────────────────────────────────────────────────────────────────────────
[the existing entity rendering: description, details, cited as,
 named under, examples, culture — EntityLive's content, unchanged]

[provenance ▸]  (a drawer: address, allocated <date> by <reviewer>, evidence <fingerprint>)

Only the planet is a real Mars entity; the deity and the album are marked fixtures. The subject page shows content and identity; review actors, fingerprints and allocation timestamps live in an inspection drawer, not a permanent block. The card states use the existing card, badge and eyebrow components; run /ui before writing markup.

B6 — Tests. ConnTest for every row of the B2 table, direct HTTP status and location headers included; LiveView navigation for search → On → subject and back; the Mars pair (planet allocated to /nature/mars, the deity fixture to /subjects/mars); a same-namespace collision with distinct qualified addresses (two of the corpus's Butterfly works); B4's slug fixtures; a lexeme with three entities of which one is addressed; the /words/:id exact lookup returning 404 for a missing id; public mode withholding a draft that internal mode shows; an editorial member whose name differs from the page title; an association that is not identity; withdrawn content withheld; an edition page in /works; an authored overview whose address collides with an unrelated lexical slug (shown as a separate choice, not one treatment), checked through navigation and reload; public-mode overview/alias requests never reaching a draft; and a URL parameter unable to switch the mode. Fixtures are marked and never minted into the corpus.

B7 — Demonstration. Run the dev server against a disposable demo database on the 5433 scratch cluster, restored with the corrected tooling from the frozen devils_dictionary_stage2r_c1 and backfilled with the rehearsal review file (105 rehearsal allocations, clearly labelled rehearsal). Do not write into devils_dictionary_stage2r_bfa2, the verified restored copy: it is retained audit evidence. Allocate the Mars fixtures on the demo database with a marked rehearsal review, in internal mode. Record screenshots of search → On Mars → /nature/mars and /subjects/mars, and of a subject card in each of the five defined states (B3's table). The "real working population" on the development corpus is demonstrated only after the C decisions and the move; do not conflate the two.

C. Corpus decisions: prepare, do not act

Nothing here writes to devils_dictionary_v2.

  • Schema reconciliation — decided by the owner on 27 September: (i) merge Curation runtime stage A: one private local model behind one slot (#195) #210 after an independent audit. The audit must check one thing the PR description admits: what ran against v2 at 13:11Z was an early draft of 20260927131023. Compare the four runtime tables' live schema on v2 (columns, constraints, indexes, trigger functions) with the schema the PR's final migrations produce on a fresh database. Identical → v2 equals main on merge and nothing else is needed. Different → record the difference and the repair (a follow-up migration, never a manual edit of v2) before calling the schema reconciled. Options (ii) and (iii), the rehearsed rollbacks, remain the fallback if the audit fails.
  • Deferred to after relocation, per Repeatable dictionary migration: prove external setup against the retained working baseline #211: the corpus catch-up (option A), the six artwork images, the 170-record population, the named reviewers. Record on Routing before launch: On overviews, standards-informed namespaces, and stable subject URLs #194 exactly what remains and why the checkpoint is nevertheless satisfied. After A1–A4, regenerate the export, the population and every review fingerprint before any persistent run; rehearsal confirmations are not approvals; the other 100,614 entities stay visible with their dispositions.

D. Consolidate and hand off

D1 — Merge order. #208 (after A4) → #216 (after A1–A3, rebased onto the new main, base retargeted to main) → #210 (after the independent audit in C) → #202 (accepted; the owner's merge decision recorded) → the On reader PR(s) from B, with their docs. Merge commits, not squash. Validate main after each merge: compile with warnings as errors, the combined migration path from zero and from v2's current version, mix precommit.

D2 — Worktrees — approved by the owner on 27 September. Preservation comes first (28 Sep re-audit, correction 3): before a worktree is retired, any unique ignored configuration it holds is preserved privately (or shown to be byte-identical to a retained copy) and any evidence it alone holds is copied into the retained evidence; credentials never go into git or an issue comment. The owning tool's archive is used only for a worktree that tool actually manages (a session registered with it), not because of a directory name. Before removing any worktree: no process has it as its working directory (lsof -d cwd), git status --ignored shows nothing beyond build caches, .env and the data symlink, and the branch is merged with no commits ahead. App-managed worktrees (.claude/worktrees/*) whose sessions are idle are reclaimed through the managed archive (the session archive tool), which stops the session and removes its worktree; the rest with git worktree remove after the data symlink is confirmed a symlink and removed. claude/dictionary-word-page-grouping-2c67e4's two dirty files (a local port override) are preserved as a patch under data/worktree-reclaim-2026-09-27/ before removal. Leave the locked claude/158-build1, the two ~/.codex worktrees, and any worktree with a live process to their owners. Remove the #216 worktree only after its merge and with its session gone. Record what was removed and the space recovered.

D3 — Databases. A name is not a disposition (#211). Record one inventory row per database on both clusters: server identity (system_identifier), name, owner, purpose, what depends on it, and a proposed disposition. This issue inventories and proposes; it drops no database. A drop needs separately established authorization naming that exact database and server; an ended session, a test-partition name or this inventory is not that authorization. devils_dictionary_bing135, devils_dictionary_74_verify and devils_dictionary_dev (12 GB; config/dev.exs still names _dev as the Gate 0 baseline) and the 22 scratch copies on 5433 get rows and a recommendation; keep at least stage2r_c1, stage2r_caughtup, stage2r_bfa2 and the dumps.

D4 — Processes. The owner's terminal: fg the 4007 server and stop it cleanly, close the ngrok tunnel, stop the 4017 preview. Record that quiescence in the handoff.

D5 — Checkpoint. Re-cut the consolidation checkpoint against the new main; verify local main equals origin/main; name the integration branch, if any, and what it preserves. Then repeat the B6 journey and the B7 demonstration from that exact final main revision, with #202's curated opening present, because both changes touch the reader. Evidence from an earlier branch does not satisfy this.

D6 — Documentation. README routing paragraph, ADR §4/§6/§8, docs/routing/stage-2/README.md, backfill.md, stage-1-foundation.md, and a status comment on #194 with the honest foundation grade.

D7 — Handoff to #211. One comment on #211: accepted code SHA and schema baseline (schema_migrations head), migrations main will expect after the move (#216's 20260927200717, and #210's if unmerged), preserved outstanding work, cleared blockers, and the reclaim already done.

Schema, read against devils_dictionary_v2

Section B is expected to add no tables and no columns; if the B3 contract turns out to need the smallest binding change, the PR justifies it and this section is updated first. It reads:

pages                     public_paths              classification_decisions
  id                        id                        id
  role                      path                      object_id
  locale                    kind                      origin · status · family
  target_object_id          original_page_id          candidate_families[]
  publication_state         destination_page_id       rule_ids[] · reasons[] · warnings[]
  lifecycle_state           last_route_change_id      policy_version
  merged_into_page_id       inserted_at · updated_at  evidence_fingerprint
  current_revision_id                                 source_pins[]
  canonical_path_id                                   reviewer_actor_id · reason
  last_route_change_id                                supersedes_id · is_current
  inserted_at · updated_at                            inserted_at

plus entities, object_names (object_id, name, language_tag, name_kind, source_record_revision_id) and the existing lexicon. #216 adds routing_backfill_runs and routing_backfill_items (migration 20260927200717); per #211 it is applied to the source only after preservation passes.

Decisions

Decided on 27 September 2026:

  1. Schema reconciliation: (i), merge Curation runtime stage A: one private local model behind one slot (#195) #210 after an independent audit (C above).
  2. Worktree reclaim: approved under D2's checks. Databases are not covered by this approval; D3 records dispositions only.

Decided on 28 September:
3. Merge #202: yes. Merged as e1d60b0, in D1 order.

Still open:
4. Stop 4007, the tunnel and 4017 (owner's terminal).
5. After the move, not now: catch-up A, the six images, the population, the reviewers, and CP4's enablement on the working installation.

Checkpoints

Non-goals

Relocating PostgreSQL, source or agent storage; legacy redirects; publication, indexing, sitemaps or Stage 5; new move/merge/split/retire administration; any write to devils_dictionary_v2 (backfill, catch-up, migration) before the move; minting corpus records for demonstrations.

Acceptance

Independent review of the code and the evidence, not the implementer's grade. The final comment states what works in the app with the demonstration, which defects were fixed, checks run and their limits, what was merged, what remains outside main, the foundation grade, whether #211 can begin, and which dataset the demonstration ran on (isolated demo now; the working installation only at CP4).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions