feat(memory)!: bring the wiki up to date now with wiki update

docs/admin-guide.md

@@ -562,7 +562,7 @@ The persistent layer of the household brain: a curated knowledge vault, rendered
 
 Setup seeds the vault with three things: the classification ontology (the topics and document types the archivist files against), household facts, and a hand-curated correspondents layer with aliases, so "Springfield Insurance" and "Springfield Ins. Co." resolve to the same page instead of becoming duplicates.
 
-The wiki maintains itself. A curator sidecar (`stack-memory-curator`) watches the vault: when new filings settle it regenerates the pages of the family members involved plus the home page (a couple of LLM calls, a few minutes after the burst), and once a night it rebuilds everything — topic pages, cross-references, the lot — while the GPU has nothing better to do. `./stack memory wiki` stays available as the manual trigger, and `[memory]` in `stack.toml` holds the knobs (see [Configuration](#configuration)). Edits happen in Forgejo (every wiki page links to its source), so the commit log doubles as the household's learning history.
+The wiki maintains itself. A curator sidecar (`stack-memory-curator`) watches the vault: when new filings settle it regenerates the pages of the family members involved plus the home page (a couple of LLM calls, a few minutes after the burst), and once a night it rebuilds everything — topic pages, cross-references, the lot — while the GPU has nothing better to do. To bring the wiki up to date now instead of waiting, `./stack memory wiki update` regenerates what the latest filings touched and `./stack memory wiki update --all` runs the whole nightly rebuild; both return when the pages are written. `--member`, `--topic` and `--home` regenerate just those pages, and `[memory]` in `stack.toml` holds the knobs (see [Configuration](#configuration)). Edits happen in Forgejo (every wiki page links to its source), so the commit log doubles as the household's learning history.
 
 **The family diary.** Everything posted in `#Memories` becomes a diary entry. Each night the curator transcribes the new recordings, has the AI read each entry (a title, a summary, the people, topics from the ontology, facts), writes one card per entry to the vault under `family/diary/entries/`, and rebuilds the diary pages in the wiki from those cards. At 07:30 local time the archivist posts each new card as a quiet notice in the thread under its memory, and one short notice with the count (`diary_job_at` in the archivist's `bot.toml` sets the time; it catches up after a night the Mac slept through). A family member corrects a card by replying in its thread, typed or spoken; the correction is committed to the vault under their name, and a corrected or hand-edited card is never overwritten by a later run. The archivist says nothing else in `#Memories`. The user guide has the family's side: [The Memories Room](user-guide.md#the-memories-room).
 
@@ -575,7 +575,9 @@ Useful commands:
 
 ```bash
 ./stack memory search <term>      # full-text query over the vault
-./stack memory wiki               # regenerate home, member, and topic pages with the LLM
+./stack memory wiki update             # bring the latest filings into the wiki, and wait
+./stack memory wiki update --all       # the nightly rebuild now: diary, sources, every page
+./stack memory wiki update --member homer   # regenerate one page
 ./stack memory ontology           # sync the vault ontology with the shipped seed
 ./stack memory lookup <text>      # which topic or doctype does a term resolve to?
 ./stack memory correspondents     # inspect the correspondent layer

docs/agent/dev.md

@@ -132,7 +132,7 @@ Shell hooks (`.sh`) receive env vars: all rendered vars plus `FAMSTACK_DATA_DIR`
 
 ## CLI plugin contract
 
-Files under `stacklets/<id>/cli/<cmd>.py` (excluding `_*.py` and `post_setup.py`) become `./stack <id> <cmd>`.
+Files under `stacklets/<id>/cli/<cmd>.py` (excluding `_*.py` and `post_setup.py`) become `./stack <id> <cmd>`. Naming, verbs and flags follow the command rules in [../stack-reference.md § CLI Commands](../stack-reference.md#command-rules); check a new or changed command against them.
 
 ```python
 HELP = "One-line description for --help"

docs/agent/ops.md

@@ -114,7 +114,7 @@ Port collisions: do not silently rebind. Surface them. The user's fix is "stop t
 | Voice messages fail on a remote AI setup | `./stack ai connect <url>` output, "Voice messages" line | The AI server lists no speech-to-text model. Pass `--whisper <url>` for a speech server. |
 | Disk full | `./stack host` | Likely the photo library. Move `data_dir` to external SSD. |
 | Element warns "browser not supported", then never loads | Which address is open? | Port mode over the LAN IP: plain HTTP is not a secure context. On the server Mac open `http://localhost:42030`; other computers use the Element desktop app with `http://<ip>:42031`. |
-| Wiki pages stale after filings | `docker logs stack-memory-curator` | Curator debounces (~3 min quiet) before rebuilding; topic pages wait for the nightly sweep. Manual override: `./stack memory wiki`. |
+| Wiki pages stale after filings | `docker logs stack-memory-curator` | Curator debounces (~3 min quiet) before rebuilding; topic pages wait for the nightly sweep. Manual override: `./stack memory wiki update` (what the last filings touched) or `./stack memory wiki update --all` (everything, as at night). |
 | Wiki shows old content entirely | `docker logs stack-memory-curator` | The curator owns the vault `git pull` (the wiki container is a read-only view). Curator down/stuck → vault and wiki go stale together. |
 | Curator logs "waiting for vault" | `./stack status` → memory, code | Vault not cloned yet — the memory install hooks own the initial clone. |
 

docs/stack-reference.md

@@ -810,6 +810,32 @@ The convention:
 Files starting with `_` are private helpers (e.g., `_matrix.py`), not
 exposed as commands.
 
+### Command rules
+
+Every stacklet adds commands, so these keep them reading as one CLI.
+Check a new command, or a changed one, against them.
+
+1. **Name the thing or the action, as an admin sees it.** Never the
+   component that does the work (`curator`, `archivist`) and never when
+   it usually runs (`nightly`): those change, the admin's intent does not.
+2. **A thing with more than one action is a noun with verbs.**
+   `stack memory wiki update`, `stack memory wiki clean`,
+   `stack messages room list`. The bare noun lists its verbs, and the
+   noun leaves room for a later verb such as `show`.
+3. **Flags narrow or modify, and one word means one thing everywhere.**
+   `--all` everything, `--dry-run` preview and write nothing, `--yes`
+   skip the confirmation, `--as <member>` act as that family member.
+   `--json` belongs to the framework. Flags that contradict each other
+   are refused in one line that names both.
+4. **A command that changes something applies by default.** `--dry-run`
+   is the opt-in preview, never the other way round.
+5. **A command that asks a background worker waits until the work is
+   done**, prints each step and each item while it waits, and on timeout
+   says how far it got. The request itself survives the timeout.
+6. **A command returns a dict**, `{"error": ...}` on failure, which exits
+   non-zero. With `--json` that dict is all of stdout; what the command
+   prints for people goes to stderr.
+
 A command that records a choice in `stack.toml` calls
 `config["set_cfg"](section, key, value)`, the plugin-side twin of a
 hook's `ctx.cfg(key, value)`. It changes that one line in that section

lib/stack/cli.py

@@ -11,6 +11,11 @@
 
 All framework logic lives in the Stack class. Docker operations use
 the docker module. This file is the glue.
+
+Stacklet commands (`stacklets/<id>/cli/<cmd>.py`) follow the command
+rules in docs/stack-reference.md § CLI Commands: named for what the
+admin wants, nouns with verbs, one meaning per flag, apply by default,
+wait with progress, return a dict.
 """
 from __future__ import annotations
 

stacklets/memory/bot/cli/wiki.py

@@ -1,17 +1,22 @@
-"""`stack memory wiki` — regenerate the family wiki's entry pages.
+"""`stack memory wiki update` — regenerate the family wiki's entry pages.
 
 Walks the memory vault, pulls every document's `> [!summary]` callout,
 and asks the LLM to compose the browsable pages a family lands on:
 
-    stack memory wiki                   home + members + topics (apply)
-    stack memory wiki --home            just the household home page
-    stack memory wiki --member homer    just Homer's page
-    stack memory wiki --topic camping   just one topic's page
-    stack memory wiki --topics          every topic page, no home/members
-    stack memory wiki --dry-run         preview to stdout, no writes
-    stack memory wiki clean             delete every generated page (asks first)
-    stack memory wiki clean --dry-run   list the pages clean would delete
-    stack memory wiki clean --yes       skip the confirmation (scripted rebuild)
+    (no flags)             home + members + topics: the curator's nightly pass
+    --home                 just the household home page
+    --member homer         just Homer's page
+    --topic camping        just one topic's page
+    --topics               every topic page, no home/members
+    --dry-run              preview to stdout, no writes
+    clean                  delete every generated page (asks first)
+    clean --dry-run        list the pages clean would delete
+    clean --yes            skip the confirmation (scripted rebuild)
+
+These are the arguments this module takes inside the bot-runner. Admins
+reach the page flags as `stack memory wiki update --member homer` and
+`clean` as `stack memory wiki clean`; a bare `stack memory wiki update`
+asks the curator instead (host side, `cli/wiki.py`).
 
 `--member` and `--topic` repeat and combine with `--home`: any
 selection flag switches from the full sweep to "generate exactly this

stacklets/memory/bot/curator.py

@@ -28,7 +28,7 @@
 
 Failure shape: a failed rebuild keeps the SHA where it was and
 retries after another quiet window — the trigger is never lost, and
-`stack memory wiki` always works as the manual override. If the
+`stack memory wiki update` always works as the manual override. If the
 curator is down, the vault (and therefore the wiki) goes stale
 together — one component, one failure story. Disabling
 `wiki_auto_rebuild` stops the rebuilds, NOT the pull: the wiki must
@@ -43,6 +43,7 @@
 from __future__ import annotations
 
 import asyncio
+import json
 import os
 import subprocess
 import sys
@@ -61,11 +62,15 @@
 from memory.lib import (  # noqa: E402
     MIRROR_SHA_NAME,
     MIRROR_TRIGGER_NAME,
+    NIGHTLY_RUN_NAME,
+    NIGHTLY_TRIGGER_NAME,
+    REBUILT_SHA_NAME,
     PRESERVE_LOCAL,
     RESET_LOCAL,
     SyncResult,
     _parse_frontmatter,
     authenticated_remote,
+    report_progress,
     brain_remote_url,
     is_auth_failure,
     reconcile_with_remote,
@@ -403,14 +408,14 @@ def _fm(reader, path: str) -> dict:
     return ops
 
 
-def consume_trigger(state_dir: Path) -> bool:
-    """Consume a pending mirror-now trigger. True when one was there.
+def consume_trigger(state_dir: Path, name: str = TRIGGER_NAME) -> bool:
+    """Consume a pending trigger (mirror-now by default). True when one was there.
 
     Deletion is the consumption: a concurrent second consumer loses the
     unlink race and correctly reports no trigger.
     """
     try:
-        (state_dir / TRIGGER_NAME).unlink()
+        (state_dir / name).unlink()
         return True
     except OSError:
         return False
@@ -764,7 +769,7 @@ def _reconcile(self) -> SyncResult:
 
 # ── Rebuild ──────────────────────────────────────────────────────────────
 
-async def compile_diary() -> bool:
+async def compile_diary(state_dir: Path | None = None) -> bool:
     """One diary pass over the memories room, on the nightly sweep.
 
     The compiler re-reads the whole room every time, because a reply or
@@ -774,19 +779,26 @@ async def compile_diary() -> bool:
     the nightly pays for recordings and readings that are genuinely new
     and reuses the rest.
     """
-    return await _run_command("diary", [], "compiling diary")
+    return await _run_command("diary", [], "compiling the diary", state_dir)
 
 
-async def rebuild(selection: list[str]) -> bool:
+async def rebuild(selection: list[str], state_dir: Path | None = None) -> bool:
     """One wiki generation pass via the CLI entrypoint — the same code
-    path `stack memory wiki` execs, in a subprocess so a wedged LLM
-    call dies with the child instead of inside this loop."""
-    label = " ".join(selection) if selection else "(full sweep)"
-    return await _run_command("wiki", selection, f"rebuilding wiki: {label}")
+    path `stack memory wiki update` execs, in a subprocess so a wedged
+    LLM call dies with the child instead of inside this loop."""
+    label = " ".join(selection) if selection else "every page"
+    return await _run_command("wiki", selection, f"regenerating wiki pages: {label}", state_dir)
 
 
-async def _run_command(command: str, selection: list[str], what: str) -> bool:
-    logger.info("[curator] {}", what)
+async def _run_command(command: str, selection: list[str], what: str,
+                       state_dir: Path | None = None) -> bool:
+    """Run one entrypoint command in a child process; True when it succeeded.
+
+    The child's lines (one per page: published, unchanged, skipped) go to
+    the log and, with a `state_dir`, to the progress file a waiting
+    `stack memory wiki update` prints from, as they come.
+    """
+    _progress(state_dir, what)
     # Starting the child and waiting on it are separate failure modes, and
     # only the second one has a child to kill. Keeping them in one block
     # left `proc.kill()` reachable on a path where `proc` was never bound.
@@ -800,10 +812,19 @@ async def _run_command(command: str, selection: list[str], what: str) -> bool:
         logger.warning("[curator] {} failed to start: {}", command, e)
         return False
 
+    lines: list[str] = []
+
+    async def follow() -> None:
+        assert proc.stdout is not None
+        async for raw in proc.stdout:
+            line = raw.decode(errors="replace").rstrip()
+            lines.append(line)
+            if line.strip() and not line.startswith("{"):
+                _progress(state_dir, line.strip()[:200])
+        await proc.wait()
+
     try:
-        out_bytes, _ = await asyncio.wait_for(
-            proc.communicate(), timeout=REBUILD_TIMEOUT_SECS,
-        )
+        await asyncio.wait_for(follow(), timeout=REBUILD_TIMEOUT_SECS)
     except TimeoutError:
         proc.kill()
         logger.warning("[curator] {} timed out after {}s",
@@ -813,16 +834,26 @@ async def _run_command(command: str, selection: list[str], what: str) -> bool:
         logger.warning("[curator] {} failed: {}", command, e)
         return False
 
-    output = out_bytes.decode(errors="replace").strip()
     if proc.returncode != 0:
-        tail = "\n".join(output.splitlines()[-5:])
+        tail = "\n".join(lines[-5:])
         logger.warning("[curator] {} rc={}: {}", command, proc.returncode, tail)
         return False
-    published = sum(1 for ln in output.splitlines() if ln.startswith("published "))
+    published = sum(1 for ln in lines if ln.startswith("published "))
     logger.info("[curator] {} done — {} page(s) published", command, published)
     return True
 
 
+def _progress(state_dir: Path | None, what: str, *, fresh: bool = False) -> None:
+    """One step, to the log and, when someone may be waiting, the progress file."""
+    logger.info("[curator] {}", what)
+    if state_dir is None:
+        return
+    try:
+        report_progress(state_dir, what, fresh=fresh)
+    except OSError as e:
+        logger.warning("[curator] could not record progress: {}", e)
+
+
 # ── Main loop ────────────────────────────────────────────────────────────
 
 async def main() -> None:
@@ -832,7 +863,7 @@ async def main() -> None:
     brain_dir = Path(os.environ.get("BRAIN_REPO_DIR", "/data/memory/brain"))
     shared_bucket = os.environ.get("SHARED_BUCKET", "family")
     state_dir = Path(os.environ.get("CURATOR_STATE_DIR", "/data/memory/curator"))
-    sha_file = state_dir / "last-rebuilt-sha"
+    sha_file = state_dir / REBUILT_SHA_NAME
     mirror_file = state_dir / MIRROR_SHA_NAME
     nightly_file = state_dir / "last-nightly-date"
 
@@ -872,7 +903,7 @@ def _read(path: Path) -> str:
         return path.read_text(encoding="utf-8").strip() if path.exists() else ""
 
     # First boot starts from the current HEAD: existing content is the
-    # operator's to backfill (`stack memory wiki`), not something to
+    # operator's to backfill (`stack memory wiki update --all`), not something to
     # surprise a fresh install with. From then on the SHA survives
     # restarts, so a rebuild interrupted mid-flight is simply redone.
     last = _read(sha_file)
@@ -976,20 +1007,29 @@ async def mirror_reconcile() -> bool:
         # date is recorded even on failure: one attempt per night, the
         # incremental path and the manual CLI cover the gap. A full
         # source reconcile precedes the regen so brain self-heals any
-        # drift the incremental mirror missed.
-        if nightly_due(NIGHTLY, _read(nightly_file), time.localtime()):
-            _write(nightly_file, time.strftime("%Y-%m-%d", time.localtime()))
+        # drift the incremental mirror missed. `stack memory wiki update --all`
+        # asks for the same sweep now; that does not count as tonight's.
+        asked = consume_trigger(state_dir, NIGHTLY_TRIGGER_NAME)
+        if asked or nightly_due(NIGHTLY, _read(nightly_file), time.localtime()):
+            if not asked:
+                _write(nightly_file, time.strftime("%Y-%m-%d", time.localtime()))
+            _progress(state_dir, "rebuilding the whole wiki", fresh=True)
+            _progress(state_dir, "reconciling the source vault with the brain")
             if await mirror_reconcile():
                 mirror_sha = _write(mirror_file, head)
             # Diary first: it writes pages into the same working tree,
             # and the commit below should carry both nights' work.
-            await compile_diary()
-            if await rebuild([]):
+            await compile_diary(state_dir)
+            rebuilt = await rebuild([], state_dir)
+            if rebuilt:
                 # Generation wrote pages into brain's working tree; commit
                 # and push them (one commit alongside the reconcile).
                 await brain.commit_push(f"{BRAIN_COMMIT_PREFIX} nightly rebuild")
                 last = _write(sha_file, head)
                 debounce.reset()
+            # The record a waiting `stack memory wiki update --all` reads.
+            _write(state_dir / NIGHTLY_RUN_NAME,
+                   json.dumps({"at": time.time(), "ok": rebuilt, "asked": asked}))
             continue
 
         # ── Incremental: persons + home, debounced ────────────────────
@@ -1015,7 +1055,8 @@ async def mirror_reconcile() -> bool:
             debounce.reset()
             continue
 
-        if await rebuild(selection):
+        _progress(state_dir, "regenerating the pages new filings touched", fresh=True)
+        if await rebuild(selection, state_dir):
             # Commit + push the regenerated pages the wiki CLI wrote into
             # brain's working tree.
             await brain.commit_push(f"{BRAIN_COMMIT_PREFIX} incremental rebuild")