Self-hosting phase 2: the canonical-default flip — sdp import, the checkout migration, the decision fold, and the 58-spec corpus

[darko-mijic]

What this is

Phase 1 taught the Protocol to describe itself: fifteen of its own specs in the ruled .sdp.md carrier, extracted into the same graph and checked by the same validators as any consumer project. But it left the repo running a deliberately interim, three-state carrier world: new spec IDs born Markdown-canonical, pre-existing IDs still TypeScript-canonical, and the worked example (checkout-v1) TS end to end. This phase ends that interim state — the transition clause of the carrier ruling (MD-18) is discharged, and one carrier rule now governs everything: Specs default to Markdown; Packs remain TS until a Pack syntax ruling; the TS DSL survives as import source and a lawful per-ID option.

Getting there was not a rename. It forced a durable product verb (sdp import), a machine-checked fidelity proof for the migration, a parser-hardening pass that replaced phase 1's pinned non-claim with a bounded parity claim, a fold of the prose decision diary into typed decision specs, and four corpus waves that grew the Protocol's delivery model of itself from 15 to 58 interlinked specs. The work ran as owner-gated sessions (G1–G8, ledger in plans/18-self-hosting-phase-2.md), closed by an archived adversarial code review (reviews/07) whose two blockers were fixed and re-probed before close.

What landed

sdp import — a durable import surface, not a migration script.

• A pure, fs-free emit/import core (src/import/): emitMarkdownSpec renders a reified spec into the ruled Markdown grammar; importTypeScriptSpec orchestrates the TS reifier → emitter path; both exported from the package barrel. Findings flow through the existing extract/* currency plus a verb-local import/* namespace — never a second validation path (one validation path, MD-14).
• The emitter is fidelity-gated: it re-reifies its own output and requires exact authored-data equality (normalizing only canonical relation order and the physical empty-relations spelling). Anything it cannot reproduce faithfully becomes import/unsupported-construct with no emitted target — a refusal, never a silently changed document.
• The CLI verb is batch-atomic: all sources validate before anything publishes; targets are written via temp files and published together; a failed batch leaves no partial migration behind. Write-beside only, --dry-run, refuses overwrites (import/target-exists) and Pack carriers (import/pack-unsupported), never deletes sources.
• CONTEXT.md's glossary entry was amended on the record: one import verb, many source adapters — the TS adapter landed; the gen-1 .feature adapter stays designed-in and deferred.

The checkout-v1 migration, machine-proven. All eleven example specs moved to .sdp.md in one atomic commit (every emit paired with its TS delete — no commit ever holds both surfaces of an ID), via the importer itself rather than hand edits. Fidelity was proven at two layers under a pre-declared delta catalog: authored-level (reify → emit → re-reify → deep-equal, zero prose normalization) and graph-level (pre/post graphs equal up to the carrier file suffix). The walkthrough README was rewritten for the Markdown surface.

Parser hardening and the parity claim. The review-06 grammar-hardening cluster landed (YAML 1.2 scalar classification under the failsafe schema, seeded line counters so finding lines are carrier-relative, the exact ... closer, per-limit summary findings under cap flood, stricter heading/GWT law, the dead branches removed, and a total catch-all boundary on the public Markdown reifier mirroring the TS carrier). The bounded TS↔Markdown refusal-parity matrix was then executed: each of the TS reifier's ten authored-fidelity refusal classes maps to a same-class Markdown finding or an explicitly named, recorded non-claim. Phase 1's blanket "no parity claimed" caveat is resolved with pinned evidence.

The canonical-default flip, one commit. Every operative record moved together: AGENTS.md, CONTEXT.md, the decision registry, the root README (born this phase), the concept docs, and the JTBD stories now state the post-flip rule; the interim-rule audit script was renamed (check-carrier-interim.mjs → check-carrier-rule.mjs) and re-pinned to the new sentence, so drift in the carrier rule's wording fails the audit loudly. The per-ID canonical-surface config idea was repaired to explicitly aspirational — the current realization is file-existence-only.

The decision fold. Nineteen durable rulings left the prose diary and became decision-kind specs under specs/decisions/, each refines-linked to the most specific subsystem spec it rules, with decidedBy back-edges from the specs it shapes. docs/concept/DECISIONS.md shrank from a ~580-line diary to a lean registry: name, one-line gloss, spec pointer. Diary bodies are deleted — git history is the archive. Measured-evidence figures moved onto the specs that own them.

Four corpus waves: 15 → 58 specs. The model subsystem (the primitive, descriptors, sections, relations, stable IDs, the Pack, anchors), the validation family (one rule spec per independently statable law, with pre-declared bundles), the consumers (reader, agent surface, projections, Design Review, edit model), and extraction/graph (excludes, claim taxonomy, regenerability, schema versioning, executable contracts). Anchors bind only named real entrypoints (1–3 per anchored spec, no decorative anchors); readiness is stated only where the floor honestly clears — the closing distribution is 7 ready / 51 defined, zero idea/scoped, no orphans. A per-spec coverage ledger maps each concept-doc section to its carrying spec.

Kept honest along the way. The Design Review projection got a ruled determinism fix (lexicographic dynamic-key ordering; one escaping policy across every rendered field). The adversarial review (reviews/07) found two real blockers in the import surface — silent content loss on multi-entry TS specs, and partial publication on failed batches — both fixed, re-probed, and archived with dispositions.

The pre-merge docket, closed on the branch. A second, independent five-dimension review of the full branch diff (reviews/08) recorded 8 major and 23 smaller findings against the durable import/render surfaces. Rather than carry them into main, a follow-up session (plans/19) fixed every accepted one — and three of them closed by fixing a contract rather than a call site:

• Builder trust is physical module identity, never a raw relative spelling. Recognizing anchors by generic specifiers like "./ids.js" meant a consumer file importing a symbol named codeAnchor from their own ./code-anchor.js would mint impl: anchors → satisfies edges → derived implemented facts from non-protocol code. Trust now resolves the importer-relative path (including the TS .js→.ts convention) against this runtime package's canonical builder modules; consumer lookalikes mint nothing, and deeply nested Protocol source stays eligible.
• Import is a transaction. Scan → plan → prepare → publish: bounded canonical discovery sharing the extractor's exclusions and following no directory symlink, every source reported before anything is written, publication by atomic hard link so an existing target cannot be clobbered, and rollback through the never-throw removal helpers that build/view already used. A failed batch publishes nothing and names every survivor instead of crashing.
• Markdown escaping follows syntactic context. The one-policy ruling had been applied to markdown-literal positions where escapes are never decoded, so authored & and < displayed mangled inside ```json fences. Prose, headings, links, and tables escape; fenced JSON and inline code carry literal data under their own encoders.

The rest landed as the verb's diagnostic cluster (the emitter now names the first divergent authored path; refusals no longer swallow the Pack finding; explicit non-carrier operands are errors), a reserved-vocabulary parity repair, and a records sweep. Three findings are recorded observations with no invented policy: control-character latitude, corpus-test granularity (owned by the tests-to-executable-specs program), and the separate example id namespace.

Gate machinery kept honest too. The anchor-trust ruling initially cost preflight its clean room — a copied source tree driven by this checkout's binary lawfully loses every relative-import anchor, so the check was reduced to regenerating in place, which downstream of generate:* and both --check-clean legs proves little. The clean room is restored correctly: the scratch root carries dist/ and package.json and runs its own sdp, so trust holds inside the copy and the committed tree is compared against a regeneration performed entirely outside this checkout. The tracked slice of the ignored .omo/ agent workspace was untracked, and the temporal guard's exemption for it deleted — enumeration already honours .gitignore.

Verification

• npm run check green on the full twelve-leg chain — locally, on a fresh git clone --no-local clone pinned to the phase SHA, and against the packed, installed tarball (which also proves the sdp import bin wiring and the barrel exports).
• Migration fidelity is machine-checked at two layers (authored deep-equality with zero prose normalization; graph equality under the pre-declared delta catalog), not eyeballed.
• The self-hosting graph at close: 58 Specs, 1 Pack, 36 anchors → 95 nodes, 180 edges, zero validation errors or warnings.
• All eight owner gates accepted (ledger in plans/18-self-hosting-phase-2.md); the adversarial review's remediations re-probed by focused suites across the import/CLI/package surface. The suite stands at 478 repository tests plus the isolated 54-test CLI suite.
• A second, independent five-dimension pre-merge review of the full branch diff is archived at reviews/08-self-hosting-phase-2-pre-merge-review.md. Its verdict: sound to merge — the fidelity gate refused every lossy shape probed at it, batch atomicity held, the flip is consistent across every operative record, and the corpus floor is honest. Its 8 major and 23 smaller findings were then closed on this branch rather than carried into main (plans/19, with the per-finding disposition table and the review's own remediation addendum); the fixes were re-probed by focused suites and a repeated full gate.
• preflight's clean room proves regeneration is relocation-independent: the committed generated/ tree matches a regeneration run outside this checkout, by a relocated copy of the built CLI, byte-for-byte.
• The executed orchestration plan is archived byte-for-byte at plans/18a-self-hosting-phase-2-execution.md (same convention as the phase-1 archive), so the coverage, fold, and docket ledgers survive workspace cleanup.

What remains for full self-hosting

The ratified term is self-hosting, and the native carrier already is Markdown — so what remains is not a format question but the last leg of the thesis applied to itself: moving the remaining untyped prose exposition and the untyped test suite into the typed, graph-carried model. Concretely, in rough order:

1. The tests-to-executable-specs rewrite (the big one). The duplicate-IDs tracer from phase 1 is the template: author the parent's gwt-vocabulary, bind an example child to a point, generate the contracts, and rewrite the corresponding test/ assertions as bindExample handlers carrying a test anchor — so has-verifier is earned in the graph rather than merely claimed by a green runner. Phase 2 deliberately did not start this systematically; it is the named remaining work in the status header. This tranche is also what will finally force the unfired syntax watch items — table sugar for truth-table material, the single-literal vocabulary form, the multi-entry constraint form — which stay unruled until real material forces them (all five watch items closed this phase unfired, with recorded reasons).

2. Readiness maturation of the corpus. 51 of the 58 specs sit honestly at defined. Raising them to ready is not paperwork — each needs acceptance-grade content and, where the verify loop is genuinely cheap, a bound verifier. This proceeds spec by spec as the rewrite above supplies verifiers.

3. Per-doc concept deletion, gated by the dissolution decision. spec:decisions.concept-docs-dissolve is the recorded criterion: a concept doc may be deleted only once its semantic contract is fully carried by specs and lean registries, judged at a Design Review — never a bulk purge. The coverage ledger now maps each doc section to its carrying spec, so the per-doc audits have a concrete basis. CONTEXT.md (the glossary), the lean DECISIONS.md registry, and plans/ deliberately never dissolve.

4. The deferred tail, on the record. The Markdown Pack syntax ruling (Packs stay TS until a real caller forces it); the gen-1 .feature import adapter (designed-in, deferred); the no-reparse read seam and the temporal-guard token assembly (both deferred with named reasons in the docket ledger).

[greptile-apps]

Too many files changed for review. (230 files found, 100 file limit)

Bypass the limit by tagging @greptile-apps to review.