fix(frontmatter): warn on comment loss, and lift rationale into the body

README.md

@@ -111,6 +111,7 @@ flowchart TB
   - For tracks where you don't want to bother curating at all, set `next_up_auto: true` in the track's frontmatter — `brief` will then derive the list live each invocation, ignoring whatever's stored.
   - **Ranking presets** — when `next_up_auto: true` is on, the default ranking is `flow` (milestone → dependency → priority → recency). Override per-track with `set-next-up <track> --preset=<name>`, or set `next_up_default: <name>` in your config for a global fallback. Named presets: `flow` (the default), `priority-driven` (priority first, no milestone bias — good for backlogs with no milestones), `backlog` (oldest issues first — surfaces stalled work). Custom criterion order: `set-next-up <track> --order=aging,priority,dependency`. Clear a track's override with `--clear`. Toggle auto-derivation itself with `--auto=on|off` (no hand-editing frontmatter required).
 - **Weekly** → `hygiene` runs `refresh-md --all` + `reconcile --all` + `dedupe-tiers` (report-only) + `duplicates` in sequence to keep status icons, GitHub labels, tier dedup, and issue-dedup state honest.
+  - ⚠️ **Keep rationale in the body, not in frontmatter comments.** Frontmatter is rewritten by these commands and YAML comments cannot survive the write (#491). `lift-rationale` migrates existing ones.
 
 > **When should I run `refresh-md`?** Any time you close or merge issues and want the track body to reflect the new state. `handoff` rewrites the status table for one track on every run, but `brief` reads GitHub live without writing anything back — so a track you haven't `handoff`'d recently stays stale on disk. `refresh-md <track>` (or **Sync Issue States from GitHub** in VS Code) fixes that on-demand; `hygiene` sweeps all tracks weekly.
 
@@ -533,6 +534,7 @@ See `docs/usage-examples.md` for end-to-end scenarios (morning brief, mid-work h
 | `hygiene [--repo=<key>]` | Weekly all-in-one: `refresh-md` + `reconcile` + `dedupe-tiers` (report-only) + `duplicates`. With `--repo=<key>`, steps 1–3 scope to that repo and the global `duplicates` step is skipped. |
 | `doctor [--json] [--fix]` | Detect config drift: a renamed local folder or GitHub repo that `config.yml` no longer matches, a non-git local path, duplicate entries, an invalid/missing `notes_root`, an orphaned notes folder, or a stale per-track `github.repo`. Run this right after any rename/move. `--fix` corrects only the two mechanically-safe cases (a GitHub-confirmed rename, a stale track slug) and always re-scans afterward. `--json` for machine output. |
 | `dedupe-tiers [--repo=<key>] [--apply]` | Remove private track copies that a shared twin in a repo's `.work-plan/` supersedes (#359). When a track is promoted to the shared tier, its private original under `notes_root` is sometimes left behind (bulk/manual promotion, or a failed unlink during `push-track`) — `discover_tracks` then warns `exists in both shared and private` on every run with no cleanup path. This removes the safe orphans and **refuses** any whose private copy references issue numbers the shared one lacks (no silent data loss; the invariant is `issue_refs(private) ⊆ issue_refs(shared)`). Covers active and archived tiers. Default is a **dry-run report**; `--apply` deletes (auto-committed to `notes_root`, so undoable via `notes-vcs undo`). `--repo=<key>` scopes to one repo. |
+| `lift-rationale [--repo=<key>] [--track=<name>] [--apply]` | Move a track's rationale out of YAML frontmatter comments into a `## Ranking rationale` body section, where writes preserve it (#491). Frontmatter is round-tripped through JSON, which has no comment concept, so **every** writer (`refresh-md`, `reconcile`, `slot`, `hygiene`) erases **every** frontmatter comment — a routine `hygiene` run silently deleted 213 lines of ranking rationale from a real track while the `next_up` *order* survived intact, which is precisely what made the loss invisible. Comments attached to a `next_up` entry become bullets naming that issue; section headers become paragraphs. The body is also the only place rationale is **visible** — frontmatter comments never render as markdown, never reach the VS Code viewer, and never appear in `export --json`. Dry-run by default; `--apply` writes. |
 | `list [--all] [--sort=recent\|priority]` | List active tracks (or all including parked/archived). `--sort=recent` orders by `last_touched` (most recent first); `--sort=priority` orders by `launch_priority` (P0→P3) with recency as tiebreaker. Default keeps discovery order. |
 | `init <path> [--priority=P0..P3] [--milestone=<m>]` | Add frontmatter to a brand-new track .md file (the file must already exist). Pass `--priority=`/`--milestone=` to skip the prompts. |
 | `init-repo <key> --github=<slug> [--local=<path>] [--update [--clear-local]]` | Bootstrap a new repo: create `<notes_root>/<key>/archive/{shipped,abandoned}/` and add the repo block to your config. `--github` is required for an add; `--local` is optional. `--update` on an existing key changes its local/github; `--update --clear-local` forgets the saved local path (keeps github + other fields). `--clear-local` and `--local` are mutually exclusive. |

skills/work-plan/commands/lift_rationale.py

@@ -0,0 +1,229 @@
+"""lift-rationale — move frontmatter comments into the body, where they survive (#491).
+
+## Why this exists
+
+`lib/frontmatter.write_file` round-trips frontmatter through JSON. JSON has no
+comment concept, so EVERY write erases EVERY YAML comment — `refresh-md`,
+`reconcile`, `slot`, `hygiene`, all of them. A routine hygiene run deleted 213
+lines of ranking rationale from a real track. The `next_up` order survived
+perfectly, which is what made the loss invisible: the data was fine, the
+reasoning was gone.
+
+The body does NOT have this problem — `write_file` passes it through verbatim.
+So the durable place for rationale is a body section, not frontmatter comments.
+
+There is a second reason to move, independent of durability: frontmatter
+comments are invisible everywhere except the raw file. They do not render in
+markdown, do not reach the VS Code viewer, and do not appear in `export --json`.
+Rationale kept there was already hidden from every surface anyone reads.
+
+## What it does
+
+Extracts comment lines from a track's frontmatter and appends them, as markdown,
+under a `## Ranking rationale` heading in the body. Comments attached to a
+`next_up` entry are rendered as a bullet naming that issue, so the association
+survives the move; free-standing comment blocks become paragraphs.
+
+The frontmatter keys themselves are untouched — only comments move.
+
+Dry-run by default. `--apply` writes.
+
+Usage:
+    work_plan.py lift-rationale [--repo=<key>] [--track=<name>] [--apply]
+"""
+import re
+import sys
+from pathlib import Path
+
+from lib.config import load_config, ConfigError
+from lib.frontmatter import FRONTMATTER_RE, parse_file, write_file
+from lib.prompts import parse_flags
+from lib.tracks import discover_tracks
+
+KNOWN = {"--repo", "--track", "--apply"}
+
+HEADING = "## Ranking rationale"
+
+_ENTRY_RE = re.compile(r"^(\s*)-\s*(\d+)\s*(?:#\s?(.*))?$")
+_COMMENT_RE = re.compile(r"^(\s*)#\s?(.*)$")
+
+
+def _raw_frontmatter(path: Path) -> str:
+    try:
+        text = Path(path).read_text(encoding="utf-8")
+    except OSError:
+        return ""
+    m = FRONTMATTER_RE.match(text)
+    return m.group(1) if m else ""
+
+
+def extract_rationale(frontmatter_text: str) -> list:
+    """Group frontmatter comments into blocks, associating trailing comment runs
+    with the `next_up` entry they follow.
+
+    Returns a list of (issue_number_or_None, [lines]) in document order. A block
+    keyed to an issue number is rationale written under that entry; a block keyed
+    None is a free-standing comment (a section header, a preamble).
+
+    Association rule is INDENTATION, not adjacency. A comment indented deeper
+    than the most recent `- <number>` entry is that entry's rationale; a comment
+    at or above the entry indent is free-standing. This is how the format is
+    actually written:
+
+        # TIER 1 — the golden path is broken     <- indent 2, header
+        - 6374   # first line, inline            <- entry, indent 2
+                 # continues here                <- indent 9, belongs to 6374
+        - 6368   # ...
+
+    Adjacency alone cannot distinguish those two cases: both a trailing comment
+    and a section header sit between two entries.
+
+    An inline comment on the entry line itself starts that entry's block — the
+    real format puts the first (and often most important) line of rationale
+    there, so dropping it would lose the summary of every entry.
+    """
+    blocks: list = []
+    owner = None          # owner of the block being accumulated
+    current: list = []
+    last_issue = None
+    entry_indent = 0
+
+    def flush():
+        if current:
+            blocks.append((owner, list(current)))
+            current.clear()
+
+    for line in frontmatter_text.split("\n"):
+        em = _ENTRY_RE.match(line)
+        if em:
+            flush()
+            entry_indent = len(em.group(1))
+            last_issue = int(em.group(2))
+            inline = (em.group(3) or "").rstrip()
+            owner = last_issue
+            if inline:
+                current.append(inline)
+            continue
+
+        cm = _COMMENT_RE.match(line)
+        if cm:
+            indent = len(cm.group(1))
+            text = cm.group(2).rstrip()
+            this_owner = (last_issue if (last_issue is not None
+                                         and indent > entry_indent) else None)
+            if current and this_owner != owner:
+                flush()
+            owner = this_owner
+            current.append(text)
+            continue
+
+        # Any other YAML line ends the current run. A new top-level key also
+        # ends the entry context, so a comment under an unrelated key later in
+        # the document is not misattributed to the last issue seen.
+        flush()
+        if line.strip() and not line.startswith((" ", "\t")):
+            last_issue = None
+            owner = None
+
+    flush()
+    return blocks
+
+
+def render_rationale(blocks: list) -> str:
+    """Render extracted blocks as a markdown section body (no heading)."""
+    out: list = []
+    for issue, lines in blocks:
+        text = " ".join(l for l in lines if l).strip()
+        if not text:
+            continue
+        if issue is None:
+            out.append(text)
+        else:
+            out.append(f"- **#{issue}** — {text}")
+    return "\n\n".join(out)
+
+
+def strip_frontmatter_comments(frontmatter_text: str) -> str:
+    """Frontmatter with comment-only lines removed (values keep their order)."""
+    kept = [l for l in frontmatter_text.split("\n")
+            if not l.strip().startswith("#")]
+    return "\n".join(kept)
+
+
+def _append_section(body: str, rendered: str) -> str:
+    """Append (or replace) the rationale section at the end of the body."""
+    marker = f"\n\n{HEADING}\n\n"
+    idx = body.find(f"\n{HEADING}\n")
+    if idx != -1:
+        body = body[:idx].rstrip()
+    return body.rstrip() + marker + rendered + "\n"
+
+
+def run(args: list) -> int:
+    flags, _ = parse_flags(args, KNOWN)
+    for f in ("--repo", "--track"):
+        if flags.get(f) is True:
+            print("usage: work_plan.py lift-rationale [--repo=<key>] "
+                  "[--track=<name>] [--apply]", file=sys.stderr)
+            return 2
+    repo_key = flags.get("--repo")
+    track_name = flags.get("--track")
+    apply = bool(flags.get("--apply"))
+
+    try:
+        cfg = load_config()
+    except ConfigError as e:
+        print(f"ERROR: {e}", file=sys.stderr)
+        return 1
+
+    tracks = discover_tracks(cfg)
+    if repo_key:
+        k = repo_key.lower()
+        tracks = [t for t in tracks
+                  if (t.folder or "").lower() == k or (t.repo or "").lower() == k]
+    if track_name:
+        tracks = [t for t in tracks if t.name == track_name]
+
+    if not tracks:
+        print("No matching tracks. Nothing to lift.")
+        return 0
+
+    touched = 0
+    for t in tracks:
+        raw = _raw_frontmatter(t.path)
+        blocks = extract_rationale(raw)
+        if not blocks:
+            continue
+        rendered = render_rationale(blocks)
+        if not rendered:
+            continue
+        n_lines = sum(len(lines) for _, lines in blocks)
+        touched += 1
+        print(f"\n{t.name}  ({n_lines} comment line(s) in {len(blocks)} block(s))")
+        if not apply:
+            preview = rendered.split("\n\n")[:3]
+            for p in preview:
+                print(f"    {p[:100]}")
+            if len(rendered.split('\n\n')) > 3:
+                print("    ...")
+            continue
+
+        meta, body = parse_file(t.path)
+        new_body = _append_section(body, rendered)
+        # write_file re-dumps frontmatter from `meta`, which never carried the
+        # comments — so they are already gone from what gets written. The body
+        # now holds them, which is the whole point.
+        write_file(t.path, meta, new_body)
+        print(f"    -> lifted into '{HEADING}'")
+
+    print()
+    if touched == 0:
+        print("No frontmatter comments found. Nothing to lift.")
+    elif apply:
+        print(f"✓ Lifted rationale in {touched} track(s) into the body, where "
+              "writes preserve it.")
+    else:
+        print(f"{touched} track(s) carry frontmatter comments that ANY write will "
+              "destroy (#491).")
+        print("Re-run with --apply to move them into the body.")
+    return 0

skills/work-plan/lib/frontmatter.py

@@ -20,20 +20,54 @@ def parse_file(path: Path) -> Tuple[dict, str]:
     return (meta, match.group(2))
 
 
+def count_frontmatter_comments(path: Path) -> int:
+    """Number of `#` comment lines in a file's EXISTING frontmatter (#491).
+
+    Frontmatter is written by round-tripping through JSON (`_yaml_to_dict` ->
+    `_dict_to_yaml`), and JSON has no comment concept — so every write erases
+    every comment, structurally. This counts what a pending write would destroy
+    so the loss can be announced instead of silent.
+
+    Never raises: an unreadable or frontmatter-less file simply has nothing to
+    lose, and a warning path must not be able to break a write.
+    """
+    try:
+        text = Path(path).read_text(encoding="utf-8")
+    except OSError:
+        return 0
+    match = FRONTMATTER_RE.match(text)
+    if not match:
+        return 0
+    return sum(1 for line in match.group(1).split("\n")
+               if line.strip().startswith("#"))
+
+
 def write_file(path: Path, meta: dict, body: str) -> None:
     """Write markdown with frontmatter. Empty meta = body only.
 
     Refuses to write through a symlink (#195): a track file that is a symlink to
     a target outside the notes tree would otherwise let a write land on an
     arbitrary file. Track files are never legitimately symlinks, so this rejects
     nothing valid; raises ValueError if one is encountered.
+
+    WARNS on frontmatter comment loss (#491). The JSON round-trip below cannot
+    preserve comments, so a routine `hygiene` run silently deleted 213 lines of
+    ranking rationale from a real track — the `next_up` ORDER survived intact,
+    which is exactly what made it invisible. This does not prevent the loss (the
+    durable fix is to keep rationale in the BODY, which passes through this
+    function untouched); it makes the loss announce itself.
     """
     p = Path(path)
     if p.is_symlink():
         raise ValueError(f"refusing to write through symlink: {p}")
     if not meta:
         p.write_text(body, encoding="utf-8")
         return
+    lost = count_frontmatter_comments(p)
+    if lost:
+        print(f"WARNING: {p.name}: dropping {lost} frontmatter comment line(s) — "
+              "YAML comments cannot survive a write (#491). Move rationale into "
+              "the body (see `/work-plan lift-rationale`), where it is preserved.")
     yaml_text = _dict_to_yaml(meta)
     p.write_text(f"---\n{yaml_text}---\n{body}", encoding="utf-8")