Self-hosting phase 1: the Protocol's own specs in the ruled Markdown carrier

[darko-mijic]

What this is

Until this branch, the Protocol validated an example (the checkout-v1 tracer bullet) but described itself only in prose — eight concept documents, a decision diary, a glossary. This phase turns the product's own thesis back on the product: the Protocol's delivery model is now authored as typed Spec documents in the ruled .sdp.md carrier, extracted into the same one graph, checked by the same conformance and honesty validators, and rendered by the same Design Review projection that consumers get. The repo's own npm run check now fails if the Protocol's specs about the Protocol stop being honest.

Getting there forced the carrier ruling (MD-18) out of the exhibit stage and into product code: a real Markdown parser, a public carrier seam, prose carried in the graph, and an executable spec whose generated contract binds a real test against the real extractor. Every syntax question the corpus forced was ruled and recorded; every one it didn't force stays an explicitly named watch item rather than speculative surface.

The work ran as four owner-gated sessions (schema freeze → corpus and readiness → executable loop → decisions, rendering, and the repository gate), each closed by a Design Review over the generated view, followed by an archived adversarial code review with a verified remediation pass. The full plan, its docket ledger, and the gate record live in plans/17-self-hosting-v1.md.

What landed

The Markdown carrier, product-grade and deliberately bounded.

• .sdp.md discovery beside .sdp.ts; a bounded YAML frontmatter envelope (exact yaml@2.9.0 pin, strict acceptance subset, resource limits, reserved derived vocabulary refused — Markdown is never an honesty bypass) and an owned line-based body grammar (ruled heading → section map, gwt/gwt-vocabulary fences reusing the one notation parser).
• The parser never throws on authored content: refusals land as findings under four pinned IDs (extract/invalid-frontmatter, extract/invalid-markdown-structure, extract/unrecognized-heading, extract/unowned-prose), excluding only the offending carrier while healthy siblings continue. The grammar is corpus-scoped by design — full refusal parity with the TS carrier is explicitly not claimed and is pinned as a phase-2 obligation.
• A public two-seam carrier API: CarrierReification with reifyTypeScriptCarrier / reifyMarkdownCarrier, and deriveGraph promoted to the package root — verified on source, built output, and an installed tarball.

Prose in the graph (the prose-ownership law, MD-19). Spec-level narrative and per-section descriptions are first-class graph content with deterministic ownership rules; unowned prose refuses loudly. Graph schema bumps to 0.4.0 with fixed key positions; the only checkout-v1 golden diff is the recorded version-driven lines. The Design Review renders prose strictly from graph data.

The self-hosting corpus. Fifteen Markdown-canonical specs under specs/ (carrier, extraction, validation, model, protocol, decisions domains) plus pack:self-hosting-v1 — every readiness rung stated only where its floor honestly clears, with precise inline anchors on the real entrypoints so implemented and has-verifier are derived, never authored.

The executable tracer. spec:validation.duplicate-ids declares a typed example space; its example child binds one point; sdp build generates the step contracts under the repo's own untracked generated/; a bindExample test drives the real extractor over dual-carrier fixtures and carries a test anchor — so has-verifier is earned in the graph, not merely claimed by a green run. Spec-side edits redden the test: the drift alarm now works for the Protocol's own specs.

Extraction-root exclusions (the strict consumer-exclusion contract, MD-20). Repeatable --exclude on every extracting CLI leg with root-relative path-prefix semantics, unioned with the fixed tooling excludes — required so the repo's root build carries its own specs without sweeping examples/ and explorations/.

The repository gate. npm run check now runs the full clean-clone-safe twelve-leg sequence (temporal guard → lint → format → build → generate self-hosting → generate example → typecheck ×2 → test → self-hosting --check-clean → example check → preflight), proved on a clean snapshot, a git clone --no-local clone, and an installed package. Root generated state is isolated from parallel tests, and the temporal guard covers tracked plus non-ignored untracked durable files.

Records kept honest. The carrier ruling's transition clause was amended on the record before the first Markdown-canonical ID (DECISIONS MD-18, CONTEXT.md, and AGENTS.md in step): new spec IDs may be born Markdown-canonical; pre-existing IDs and checkout-v1 stay TS-canonical until the ruled flip. Two new durable decisions entered the diary (MD-20, MD-21), two phase-1 decisions were born directly as decision-kind specs, and an anti-misleading pass repaired every active "TS is the sole canonical surface" claim across the concept docs and JTBD stories. An archived code review (reviews/06) drove a remediation wave — security-boundary refusals for the frozen grammar, permutation/byte-identity coverage, and diagnostic fixes — with each finding's disposition recorded.

Verification

• npm run check green on the full sequence above, locally, on a clean snapshot, and on a true clean clone; determinism is regenerate-and-compare (--check-clean), never committed goldens.
• The tracer test is green via the generated contract, with faithful RED evidence captured before each behavioral step; parser refusals and the dual-carrier duplicate are pinned by tests (~1,050 lines of Markdown-reifier tests alone).
• All four owner Design Review gates accepted (ledger in plans/17-self-hosting-v1.md §6); the post-review remediation was re-verified end to end.
• npm audit: zero production vulnerabilities; five dev-chain advisories (Vitest/Vite), dispositioned in the plan.

What remains for full self-hosting

Two framing notes: the ratified term is self-hosting (the repo deliberately rejects "dogfooding"), and the native format already is Markdown — .sdp.md with the typed envelope, free prose, and the owned notation. So the road ahead is not "away from Markdown" but from untyped prose exposition (docs/concept/00–07) to typed, graph-carried, largely executable Specs. The north star is itself recorded as a decision spec this phase authored — spec:decisions.concept-docs-dissolve: concept documents may dissolve only after their semantic contract is carried by executable Specs and lean registries; deletion is later work, never part of phase 1.

1. Phase 2 — end the interim carrier rule (scheduled by the owner at Gate 4 as one effort). The sdp import emitter (spec:carrier.sdp-import, honestly authored at idea), the checkout-v1 migration, and the canonical-default flip — discharging MD-18's transition clause. The like-for-like parser-hardening baseline (the pinned "no full refusal parity" non-claim) rides in the same effort, and the table-sugar ruling is expected to fire under it — useful for multi-case rule and test material anyway. This is the precondition for the rest: the corpus can't absorb the concept docs while the repo runs a three-state carrier world (Markdown-canonical new IDs, TS-canonical old IDs, TS checkout).

2. The decision-spec fold. Every durable row in the ratified-name registry becomes a decision-kind spec, with DECISIONS.md shrinking to a lean registry of pointers. Phase 1 seeded the pattern — two decision specs exist and the registry already points at them — but also left a recorded namespace divergence to resolve at the fold: the registry reserves spec:protocol.decisions.* while the authored specs were born spec:decisions.*. This is the cheapest dissolution tranche, since diary entries map one-to-one onto the decision section shape.

3. Corpus expansion — one subsystem at a time, anchored to the real engine. Grow the pack beyond the current 15 so each concept doc's semantic contract is carried: the core model (02) → model/behavior/rule specs for the primitive, descriptors, sections, and relations; extraction (03) → derivation and determinism specs (started this phase); validation (05) → one rule-kind spec per validator with the floor clauses as evidence; consumers (06) → reader and projection specs. Precise anchors onto existing entrypoints keep implemented honest; readiness is stated only where the floor clears.

4. The tests-to-executable-specs rewrite. The duplicate-ID tracer is the template: for each behavior worth verifying, author the parent's gwt-vocabulary, an example child binding a point, generate the contracts, and rewrite the corresponding test/ assertions as bindExample handlers carrying a test anchor — earning has-verifier in the graph rather than just a green runner. This tranche is what forces the remaining syntax rulings (table sugar for truth-table material, the single-literal vocabulary form, possibly the multi-entry constraint form and the array-section prose sub-owner) — ruled under fire, as designed.

5. Per-doc deletion, gated by the dissolution decision. Each concept doc is deleted individually, only once its content is fully carried — the criterion is the recorded decision spec, checked at a Design Review, never a bulk purge. What deliberately never dissolves: CONTEXT.md stays the ratified glossary, DECISIONS.md survives as the lean registry, and plans/ remains the process record. The end state is the thesis applied to itself completely: the spec corpus is the documentation, the graph is the read surface, and prose survives as narrative and description slots on typed owners.

Greptile Summary

This PR adds the self-hosting Markdown carrier path for the Protocol's own specs. The main changes are:

• .sdp.md discovery and Markdown carrier reification.
• Bounded YAML frontmatter parsing and owned body prose grammar.
• Graph-carried narrative and section descriptions with schema 0.4.0.
• Self-hosting specs, pack generation, and duplicate-ID executable coverage.
• CLI --exclude support and deterministic self-hosting check legs.
• Design Review rendering from Reader/graph data.

Confidence Score: 5/5

This PR appears safe to merge based on the reviewed changed paths.

The carrier parsing, discovery, CLI gating, serialization, Reader, and projection changes are internally consistent in the reviewed code. No blocking findings remain.

No files require special attention.

T-Rex Logs

What T-Rex did

• Verified that the setup probe completed with exit code 0 and that dependencies and generated outputs are present.
• Verified that the npm check exited with code 1 after 10 seconds due only to prettier warnings in environment files.
• Verified that the focused fallback command exited with code 0 after 17 seconds and exercised the changed self-hosting Markdown carrier/generation/tests/clean-check path.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
src/extract/markdown.ts	Adds Markdown carrier reification by parsing bounded frontmatter and owned body content into standard reified spec data; no blocking issues found.
src/extract/discover.ts	Extends discovery to .sdp.md carriers and root-relative exclude prefixes with deterministic sorting; behavior appears consistent with the CLI contract.
src/cli/sdp.ts	Adds repeatable --exclude, self-hosting generation support, artifact cleanup, and determinism checks for graph/contracts/view output; no blocking issues found.
src/extract/markdown-body.ts	Implements the ruled Markdown body grammar and prose ownership mapping; the bounded grammar is covered by refusal findings and no definite issue was identified.
src/extract/markdown-yaml-policy.ts	Adds YAML frontmatter policy checks for tags, aliases, depth/node/byte limits, scalar handling, and relation target validation; no blocking issues found.
src/extract/index.ts	Routes discovered .sdp.md files through the Markdown carrier and keeps duplicate-ID exclusion and graph derivation shared across carriers.
src/extract/serialize.ts	Updates canonical graph serialization for narrative and section descriptions while preserving deterministic key ordering.
src/projections/design-review.ts	Renders graph-carried narrative and section descriptions through the Reader without reparsing source; no blocking issues found.
src/reader/reader.ts	Exposes narrative in summaries/search and continues deriving context from the graph and validators; no blocking issues found.
package.json	Updates the check pipeline to generate and verify self-hosting artifacts with exclusions and pins yaml for the Markdown carrier.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant CLI as sdp CLI
participant Discovery as discoverFiles
participant TS as reifyTypeScriptCarrier
participant MD as reifyMarkdownCarrier
participant Graph as deriveGraph
participant Contracts as generateContracts
participant Reader as createReader
participant View as renderDesignReview

CLI->>Discovery: root + --exclude prefixes
Discovery-->>CLI: .sdp.ts/.sdp.md carriers + TS anchor candidates
CLI->>TS: TypeScript carriers
TS-->>CLI: reified specs/packs/findings
CLI->>MD: Markdown carriers
MD-->>CLI: reified specs/findings with owned prose
CLI->>Graph: specs + packs + anchors
Graph-->>CLI: graph with delivery facts
CLI->>Contracts: graph
Contracts-->>CLI: generated contract files + findings
CLI->>Reader: graph
Reader->>View: decoded graph contexts
View-->>CLI: generated Design Review pages

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant CLI as sdp CLI
participant Discovery as discoverFiles
participant TS as reifyTypeScriptCarrier
participant MD as reifyMarkdownCarrier
participant Graph as deriveGraph
participant Contracts as generateContracts
participant Reader as createReader
participant View as renderDesignReview

CLI->>Discovery: root + --exclude prefixes
Discovery-->>CLI: .sdp.ts/.sdp.md carriers + TS anchor candidates
CLI->>TS: TypeScript carriers
TS-->>CLI: reified specs/packs/findings
CLI->>MD: Markdown carriers
MD-->>CLI: reified specs/findings with owned prose
CLI->>Graph: specs + packs + anchors
Graph-->>CLI: graph with delivery facts
CLI->>Contracts: graph
Contracts-->>CLI: generated contract files + findings
CLI->>Reader: graph
Reader->>View: decoded graph contexts
View-->>CLI: generated Design Review pages

Loading

_{Reviews (2): Last reviewed commit: "docs(plans): file the bootstrap executio..." | Re-trigger Greptile}