Repeatable dictionary migration: prove external setup against the retained working baseline

[holden]

Outcome

Move the working dictionary installation to the owner's external 2 TB drive by building and using the repeatable installation, transfer and verification process itself. Deliver both a usable external installation and a documented procedure for a future machine/server. This is one combined implementation, not a manual move followed by a separate bootstrap project.

Combines the practical execution of #209 (storage/worktree/database migration) and #130 (fresh-clone bundle, bootstrap and readiness checks). Those issues remain requirement sources; this issue owns the shared implementation and evidence. Do not close either automatically.

The current installation is the baseline. Preserve it and a verified recovery copy until the destination is accepted. Compare old and new at the same code revision, schema and captured data state. A page that merely loads, matching row counts, or a successful command exit is insufficient.

Entry checkpoint and boundaries

• First complete the consolidation/preservation checkpoint in Delivery: curated opening section with five local persona editors and visit-driven refresh #193 (comment): accepted main plus at most one designated integration branch, all local changes/evidence accounted for, and migration-relevant restore defects addressed. Preserved unaccepted code must remain explicitly identified.
• Full Routing Stage 2 completion is not a prerequisite. Leave unfinished replay/backfill/classification/review work visible in Routing before launch: On overviews, standards-informed namespaces, and stable subject URLs #194 and resume it after migration. This supersedes older finish-all-Stage-2-first scheduling in Move dictionary storage, PostgreSQL and agent workspaces to the external 2 TB drive — execution in #211 #209.
• Read current PR/review state; do not assume the handoff's commits are still current. The relevant work at planning time is Curated opening section, Phase 1: manual fixture, Bierce-first lead, sourced highlights (#156) #202, Curation persistence slice 1: default configuration, manual compositions and publication authority (#196, #201) #206, Stage 2 recovery: rehearse on the development corpus, then repair and re-run (#194) #208 and Curation runtime stage A: one private local model behind one slot (#195) #210.
• The owner's explicit dictionary relocation instruction supersedes the older internal-source/PostgreSQL preference. Protect unrelated projects and shared services.
• This issue specifies implementation, not evidence that any move has already occurred. No personalities, new curation behavior, routing publication, corpus rebuilding or semantic cleanup belongs in the move.

1. Inventory and record an executable plan

• Inventory source/Git history, worktree registrations, unpushed/detached commits, dirty/untracked and necessary ignored files, source archives, media, models, builds/caches, databases, audit evidence, project-specific agent state and saved paths. Record size, owner, destination, preservation method and verification.
• Account for runtime benchmark databases and routing recovery artifacts; no database is disposable solely because of its name. Models/runtime assets are already external: reuse verified artifacts rather than download duplicates. Reconcile the runtime's database/cluster authority binding deliberately after restore.
• Verify external volume identity, filesystem suitability, permissions, available space and measured I/O. A directory at the expected mount path is not proof of the intended mounted drive. Measure current internal headroom and all temporary-space requirements; write bulk transfer artifacts directly externally where safe.
• Detect shared PostgreSQL resources. Prefer a dedicated external dictionary cluster when moving a shared cluster would affect other projects. Record required PostgreSQL major/tool versions, extensions, encoding, locale/collation, ownership and grants; pin the application toolchain.
• Produce a path/configuration map covering source, database, models, archives, worktrees, caches, logs and temporary files. Paths must be configurable rather than embedded throughout scripts.
• Coordinate active agents/processes before changing their paths. Use supported Git and agent-app mechanisms; do not edit live shared agent databases. Record explicit measured exceptions for shared metadata that cannot be relocated independently.

2. Implement the reusable transfer/setup interface

Implement or adapt the bundle, bootstrap and doctor capabilities proposed in #130. Names may follow existing conventions; document the actual supported commands rather than presenting proposed commands as available.

Bundle/export: versioned manifest with code revision, schema/migration state, tool compatibility, file sizes and cryptographic digests; consistent database dump(s); required source/replay/media inputs; and inventory of optional pinned model artifacts. Carry exact pinned inputs that cannot be re-downloaded from rolling URLs. A manifest must bind verification metadata to the actual dump and captured data state.

Bootstrap/restore: explicit destination and mode; verify prerequisites, mount and bundle before mutation; restore into a verified separate empty database; install/build application dependencies on the destination; configure paths and run readiness checks. Reject source aliases and unknown/mismatched/ambiguous target identities. Never silently overwrite an existing installation. Resume interrupted transfers and safely repeat setup without duplicate writes or destructive re-restoration. Existing matching state is verified; conflicting state is reported clearly.

Doctor: read-only machine-readable and human-readable checks for tool versions, destination identity, database/extension/schema compatibility, required files/checksums, configuration, build, endpoints, background-job mode and optional model readiness. Distinguish usable core operation from missing optional provider credentials/model service. No model generation or background imports as a side effect.

Separate credentials from transferable artifacts. Treat a full working database backup as private, regardless of whether .env is excluded. A future public starter bundle needs an explicit data/rights/privacy scope; do not publish the working backup. Operator credentials and grants must be transferred/recreated deliberately through private configuration.

Keep three explicit modes: restore this exact installation; initialize a new development installation from an approved bundle; rebuild from pinned source inputs. Only the first is the migration path. A source rebuild cannot replace restoration of durable identities/editorial state.

3. Establish the comparison baseline

• Record the consolidated code revision, actual source schema/migration state and relevant feature flags/configuration. Do not apply new migrations to the source merely to simplify relocation.
• Define the exact comparison contract before moving: table contents keyed by durable identity, references, sequence state, routing decisions/history/paths, curation records/approvals, source manifests and required file hashes. Compare intended role/grant mappings and extensions as well. Exclusions must be narrow, documented and justified; do not hide semantic differences as bookkeeping.
• Capture representative application behavior on the source: ordinary word page, sourced quotation/artwork page, entity page, search, enabled curated opening, routing lookup where applicable, and relevant authenticated behavior with a controlled test account. Record expected identities, links, attribution, feature availability and errors; screenshots alone are not acceptance.
• Quiesce writers through their owning services and capture a consistent, immutable baseline. Keep restored jobs, cron, discovery and inference disabled during comparisons. Read-only access to the old installation is allowed only with writers reliably disabled.
• If an early rehearsal occurs while source writes continue, record its snapshot boundary; before cutover, quiesce and capture/restore a final consistent snapshot. Do not compare a stale snapshot against a changing source and call it exact.

4. Create and verify the external installation

• Use the documented reusable commands to create the destination; every manual workaround discovered must become an explicit step, configuration input or automated check.
• Preserve source/worktree history and local work through supported relocation; prove a fresh checkout can also use the setup process without undocumented dependencies on the old checkout. Build artifacts must be produced for the destination platform, not assumed portable to a future Linux server.
• Use supported PostgreSQL logical backup/restore and isolated endpoints. Never copy individual files out of a running database cluster. Handle cluster-level roles separately without moving unrelated databases.
• First prove exact restoration at the captured schema. Any migrations needed by the consolidated code are a separate, recorded step after preservation passes; test them on the destination with their expected effects specified.
• Run exact baseline comparisons, doctor, representative UI/API reads and file/source access. Exercise writes and jobs on an isolated validation database or controlled destination state; never contaminate the baseline comparison or trigger duplicate external work.
• Run focused integration tests and mix precommit from the external checkout using an external isolated test database. Report capacity-blocked/unrun checks honestly.

5. Prove repeatability and failure behavior

• Repeat setup from the same code and bundle into a second clean external target, or a documented disposable target, using only the quickstart. No hidden access to the old installation. Verify equivalent preserved state and functionality.
• Re-run setup on an initialized target: no duplicate data, lost state or silent restore. Demonstrate an interrupted transfer/setup can resume safely.
• Reject corrupted/missing bundle parts, incompatible tools, incorrect mount, insufficient capacity, source/destination aliasing and an already-populated conflicting target before destructive work.
• Verify service restart and missing/wrong/unwritable-drive behavior without unplugging a running database. No fallback directories, database, models or bulk caches may be silently created internally.
• Verify a resumed agent session and a newly created task/worktree use intended paths. Keep unrelated projects operational.
• Measure representative query/build/test performance and storage growth; document any material regression and acceptance decision. Choose numeric thresholds from the measured baseline before judging results.

6. Cut over, retain rollback, then reclaim space

• Permit only one intended writable dictionary installation. Update all clients, worktrees, scripts, launchers and jobs; verify their actual endpoints before enabling writes.
• Preserve the old installation unchanged through destination acceptance. Record a rollback procedure and acceptance window. Once destination writes occur, switching to the old snapshot alone loses those writes: preserve/reconcile them before rollback.
• After successful verification/cutover, reclaim old internal bulk copies through owning tools and recorded dispositions. Never delete in-use worktrees or database files manually. Retain verified baseline/recovery artifacts externally for later comparisons, with an independent backup destination for drive-failure recovery; keeping all old internal files forever would not meet the disk-relief goal.
• Report measured reclaimed space, remaining internal exceptions and an operating-headroom target. Verify new routine work continues to write externally.

Deliverables and acceptance

• One committed quickstart and operations runbook describing prerequisites, exact setup/restore/verify commands, credentials handling, paths, failure recovery, backup/retention and rollback.
• Versioned bundle/manifest, safe reusable bootstrap/restore and read-only doctor capabilities implemented and used for the actual move.
• Private evidence maps source revision/schema/snapshot to destination; exact state comparisons pass, with every intended difference explained.
• The external application is usable: representative reads/writes, jobs, restart and source access pass; code checks and mix precommit results are recorded.
• A second clean-target setup and repeat/interruption tests demonstrate the documented procedure is reproducible.
• No hidden dependency on the old installation; future dictionary tasks/worktrees/tests use external storage; unrelated projects survive.
• Cutover and rollback are verified, old state retained through acceptance, and internal space actually reclaimed afterward.
• Independent review accepts the final code and operational evidence. Neither a successful copy nor an implementer's self-assigned grade is completion.
• Kickstart on a new machine: one command from a fresh clone to the full corpus (bundle, doctor, README quickstart) #130 and Move dictionary storage, PostgreSQL and agent workspaces to the external 2 TB drive — execution in #211 #209 receive shared evidence and explicit remaining scope; Routing before launch: On overviews, standards-informed namespaces, and stable subject URLs #194 retains unfinished Stage 2 work. Reconcile stale scheduling text and project storage instructions.

Future server deployment reuses the data manifest, restore, configuration and verification contracts. Server-specific release builds, supervision, TLS/networking and inference hosting remain explicit future work; the external-drive rehearsal is not proof that a Linux production deployment has been tested.

[holden]

Readiness audit, 27 September 2026: three live blockers, and the disk picture

Companion to the routing audit on #194: #194 (comment). Read-only, measured on the machine today; nothing was relocated, deleted, stopped or rolled back.

Entry checkpoint is stale

The consolidation checkpoint handoff (697ac8a) holds main c6bd882 + #202 + #210 + #208 at 2d495e8. Since then main moved to 2a851e8 (#215), #208 moved three commits to 7860b9f, and #216 (stacked on #208) did not exist. Once #208 and #216 merge (order and preconditions on #194), the integration branch only needs #202 and #210, and the checkpoint should be re-cut against that main.

Blockers to the move that the plan does not list

1. The 4007 development server is SIGSTOPped, not stopped. beam.smp 13180 has been in state T for two and a half days. It holds idle connections to devils_dictionary_v2 (22 idle connections on the database in total, the oldest from that server), and ngrok http --subdomain=wordhoard 4007 is still running against it. It needs fg in its own terminal and a clean stop, and the tunnel closed, before quiescence means anything. The Curated opening section, Phase 1: manual fixture, Bierce-first lead, sourced highlights (#156) #202 preview on 4017 (beam.smp 32882) also holds connections.
2. devils_dictionary_v2 still carries Curation runtime stage A: one private local model behind one slot (#195) #210's two unmerged migrations (20260927114948, 20260927131023, applied 13:11Z in the incident). The three main migrations above 20260924222346 are now legitimately main's schema (Routing Stage 1: durable pages, path ledger and resolver (#194) #205, Curation persistence slice 1: default configuration, manual compositions and publication authority (#196, #201) #206 merged). Recommendation: roll back only the two, from the consolidation branch, with mix ecto.rollback --to 20260927094256, so the source schema equals main's exactly and no unmerged branch is a dependency of the baseline. The full five-migration rollback the incident comment rehearsed remains the alternative. Owner's decision.
3. A session is live in the Stage 2: resumable routing backfill, rehearsed on copies of the corpus (#194) #216 worktree (dictionary-issue-194-stage-1-218d99) with uncommitted edits answering CodeRabbit. It must commit and push before the worktree inventory is final.

The disk is the emergency

Where	Measured
Internal disk free	5 GiB of 460
Postgres cluster on 5432 (Postgres.app var-18, internal, shared)	111 GB
of which dictionary databases	about 26 GB: devils_dictionary_v2 14 GB, devils_dictionary_bing135 4.2 GB, devils_dictionary_74_verify 4.2 GB, devils_dictionary_dev 3.3 GB, about twenty devils_dictionary_test* partitions 0.6 GB
of which other projects	about 83 GB: cinegraph_dev 45, volfefe_machine_dev 17, argus_dev 16, eventasaurus_dev 5
data/ (wiktextract 2.8 GB, not re-downloadable)	2.9 GB
30 dictionary worktrees	about 10 GB
of which on branches already merged to main, zero commits ahead	6.8 GB across 21 worktrees; 20 have no dirty files, claude/dictionary-word-page-grouping-2c67e4 has two, one (claude/158-build1) is locked
External /Volumes/LLM Models free	1.5 TiB
of which dictionary-stage2a scratch cluster on 5433	233 GB, 22 rehearsal databases of 9–13 GB each, plus 2.3 GB of dumps

Two conclusions for the plan:

• Moving the dictionary frees roughly 40 GB, not the drive. Three quarters of the internal cluster is other projects, which this issue rightly protects. The internal disk stays tight after the move unless those get their own decision; worth saying so in the acceptance criteria rather than discovering it at the reclaim step.
• Immediate, reversible headroom exists before any move: git worktree remove on the 20 clean merged worktrees (branches stay), and dropping the finished devils_dictionary_test* partitions. Both need the owner's word under Stage 1's "a merged PR does not by itself make a checkout disposable". The three retained databases (bing135, 74_verify, dev, 12 GB together) are the larger internal win and need an explicit disposition; config/dev.exs still names devils_dictionary_dev as the Gate 0 baseline.
• The 22 scratch copies on 5433 are external and harmless to the internal disk, but the inventory must give them a disposition too; they are the only copies of the Stage 2 rehearsal states.

Recommended order

1. Merge Stage 2 recovery: rehearse on the development corpus, then repair and re-run (#194) #208 and Stage 2: resumable routing backfill, rehearsed on copies of the corpus (#194) #216 as set out on Routing before launch: On overviews, standards-informed namespaces, and stable subject URLs #194 (two fixes first).
2. Reclaim the clean merged worktrees and test partitions with the owner's approval.
3. Unfreeze and stop 4007, close the tunnel, stop 4017; roll back Curation runtime stage A: one private local model behind one slot (#195) #210's two migrations; re-cut the consolidation checkpoint.
4. Execute this issue. The corpus catch-up, the population, the reviewers and the persistent backfill all wait for the external installation.

[holden]

The pre-migration work is now specified in #219. Its CP3 is this issue's entry checkpoint: accepted main, at most one named integration branch, the three live blockers cleared or named, and a handoff comment here with the code and schema baseline.

[holden]

Handoff from #219 (28 September)

The code: main at 16e35ca. The code is identical to 04f97bc; only docs merged after it. Merged under #219 with merge commits, branches kept, validated after each merge:

#208	484d21e
#216	903f82b
#220	64f12cd
#202	e1d60b0, on the owner's decision of 28 September
#225, the On reader	04f97bc
#226, docs	16e35ca

#210 had merged before, as d31d212. The final precommit gave 19 doctests, 2,540 tests, 0 failures. The procedure and the full inventories are in docs/operations/consolidation-checkpoint-2026-09-28.md.

Schema baseline. devils_dictionary_v2 is at 20260927131023 (24 migrations; read-only check on 28 September). main expects exactly two more, applied only after preservation passes:

1. 20260927200717 (Stage 2: resumable routing backfill, rehearsed on copies of the corpus (#194) #216): routing_backfill_runs and routing_backfill_items.
2. 20260927220937 (Every database carries the merged local_model_configs_shape, whichever draft it ran (#195, #219) #220): re-states local_model_configs_shape, which the 13:11Z incident left in its early-draft form. Check SELECT count(*) FROM local_model_configs first. It was 0 on 28 September, and the table is append-only.

A schema-only copy of v2 on the scratch cluster was migrated through these two. The result equals a database migrated from zero (26 migrations), apart from whitespace in function bodies.

Take a fresh capture. The rehearsal copies come from c1.dump (27 September, 13:29Z). v2 has been written since then, and the 4007 server is still connected to it. The move preserves v2 as it is once quiesced, not c1.

Blocker before #211 starts: quiescence (D4), the owner's terminals. Still running at 06:10Z on 28 September:

• the 4007 dev server (PID 13180, in the main checkout, 22 connections to v2);
• its ngrok tunnel (PID 14391);
• Curated opening section, Phase 1: manual fixture, Bierce-first lead, sourced highlights (#156) #202's 4017 preview (PID 32882).

Then:

1. Fast-forward the main checkout's local main, which is 66 commits behind: git -C ~/Code/projects-2026/dictionary pull --ff-only. It was left alone because a dev server is running on it against v2.
2. Do not start that dev server against v2 again before preservation. On the new main, Phoenix's development page for pending migrations would offer to apply the two migrations to v2.
3. This session stops its own two demo servers (4219, 4220). They use the scratch cluster only.

The procedure. It is the 27 September one, with these changes:

• The recovery tooling is repaired (Stage 2 recovery: rehearse on the development corpus, then repair and re-run (#194) #208), so mix dd.routing.verify may be run as a second check after pg_dump/pg_restore and the independent verification, not instead of them.
• The incident is decided: option (i). No rollback; Every database carries the merged local_model_configs_shape, whichever draft it ran (#195, #219) #220 repairs the drifted constraint after the move.

Preserved outstanding work:

• Examples in the curated opening: an exemplar item kind, one provenance projection, and the manual proof (integration under #193) #212's open PRs, Exemplar composition items and the manual proof (#212, Build 1) #218 and Examples.Provenance and 'Why this example is here' (#212, Build 2) #223;
• Runtime stage A follow-ups from the #210 audit: stale dispatch refusal, orphaned ollama runners on stop, two vacuous benchmark cells (#195) #221, Curation runtime stage A: one private local model behind one slot (#195) #210's three should-fix items;
• the unmerged branches claude/143-spotify-music-6bb8a8 (1 commit) and claude/144-kit-phase3 (5 commits);
• codex/consolidation-2026-09-27, superseded; its one unique commit is now docs/operations/consolidation-checkpoint-2026-09-27.md;
• the evidence, local and ignored, in data/audits/2026-09-28-219/;
• the dumps under /Volumes/LLM Models/dictionary-stage2a/stage2r/dumps/.

Blockers cleared:

• the recovery target's inherited endpoint (A4);
• the backfill's stale-evidence (P1) and permanent-override (P2) defects;
• Curation runtime stage A: one private local model behind one slot (#195) #210's schema drift, repaired by Every database carries the merged local_model_configs_shape, whichever draft it ran (#195, #219) #220;
• the Curated opening section, Phase 1: manual fixture, Bierce-first lead, sourced highlights (#156) #202 decision.

Reclaim. On 27 September nineteen idle sessions were archived; their directories are the owner's to remove. On 28 September 16 more worktrees (3.6 GB) were confirmed removable: no process, nothing untracked, all content on main. The #219 closing comment lists them with the commands. This session removed none, because its permission mode blocks git worktree remove. No database was dropped. Both clusters are inventoried, with proposed dispositions that need separate authorization per database and server.

After the move: CP4, #224. The owner's decisions (catch-up, population, reviewers), the backfill with real reviews on the working database, and namespace addresses for the everyday reader.

[holden]

The handoff above was independently checked on 28 September (audit on #219): precommit on 16e35ca reproduced (2,540 tests, 0 failures), the B2 reader table reproduced over HTTP from main in both modes on the retained demo copy, and the migrated-v2 schema equals the fresh reference apart from whitespace. Entry checkpoint: accepted main 16e35ca; the one integration branch is #223 (stacked on #218, leave both unmerged until the move is accepted, since #218 adds a migration). Still before this issue starts, all the owner's: stop the SIGSTOPped 4007 server (PID 13180), the ngrok tunnel (14391) and the 4017 preview (32882); fast-forward the main checkout; remove the 32 clean worktrees. Then take the fresh capture.

[holden]

D4 done, 28 September 09:12 (+02). The owner stopped the ngrok tunnel (14391) and the 4017 preview (32882); the SIGSTOPped 4007 server (13180) could not take a TERM while stopped by job control, so it was killed with SIGKILL after checking its one open transaction was an empty BEGIN from the moment it resumed. Verified: no process on 4007, zero connections to devils_dictionary_v2, no beam.smp or ngrok left. The source is quiet; a fresh capture may be taken once D5 (fast-forward the main checkout) and D2 (worktree removal) are done.