diff --git a/Agent.md b/Agent.md index 117e7b01..4569459c 100644 --- a/Agent.md +++ b/Agent.md @@ -22,7 +22,7 @@ EMRG is a self-evolving AI agent architecture experiment: a **micro-kernel** — - A change is not done until it is pushed and its guard runs in CI — local green is necessary, never sufficient - Server logs are discarded (`stderr=DEVNULL`); client logs go to `./.emrg/emrg-client.log` -- The server outlives any client, and a client auto-detects and starts it +- A client auto-detects the server and starts it if absent - `README.md` and `Agent.md` are English; `README.cn.md` is the Chinese README - Commits: `emrg: `, English, one intent per commit - A derived number is never written where a guard can measure it; a rule that can be mechanised is mechanised, because prose decays @@ -59,7 +59,7 @@ Renderer: `cd emrg/gui/renderer && npm run typecheck && npm test` (714: 5 snapsh - CI: pytest on ubuntu **and windows-2025**, the GUI suites, and an actionlint gate over `.github/workflows/`; re-trigger a dropped push with `scripts/re-trigger-ci.sh `, never an empty commit - Merge gates, run before merging — each answers a question no other gate asks: `uv run --no-sync python3 scripts/check-vote-count.py` (approvals still valid) · `scripts/check-pr-base.py` (base reaches master) · `scripts/check-merge-freshness.py` (that green CI is about the landing tree) · `scripts/check-merge-order.py` (dirt and what it lands) · `scripts/check-merge-landing-diff.py` (what the landing changes) · `scripts/check-merge-pairs.py` / `scripts/check-merge-sequence.py` (two PRs together; every step of an order) · `scripts/check-merge-tree-health.py` (the merged tree passes one guard) · `scripts/check-merge-plan-suite.py` (the plan's final tree passes the suite) — all under `uv run --no-sync python3` - Post-merge audit: `uv run --no-sync python3 scripts/check-merge-landed.py ` (did the merge land a tree its reviews named) -- Other tools: `uv run --no-sync python3 scripts/check-node-test-count.py --write` (syncs the Node totals above), `scripts/check-memory-index.py` (a MEMORY.md against the index rule's numbers and its rows' links), `scripts/check-patch-files.py` (a rebuilt patch carries every file the previous one did), `scripts/cast-vote.py` (a vote the counter counts), `bump-version.py`, `classify-conflict.py`, `sync-master-from-api.py`, `push-branch-from-api.py` +- Other tools: `uv run --no-sync python3 scripts/check-node-test-count.py --write` (syncs the Node totals), `scripts/check-memory-index.py` (a MEMORY.md against the index rule), `scripts/check-patch-files.py` (a rebuilt patch carries its predecessor's files), `scripts/check-memory-rotation.py` (which projects' memory moved), `scripts/cast-vote.py` (a vote the counter counts), `bump-version.py`, `classify-conflict.py`, `sync-master-from-api.py`, `push-branch-from-api.py` Every guard that reads a working tree names it before its verdict; a question it cannot answer is reported unmeasurable, never as a pass. diff --git a/scripts/check-memory-rotation.py b/scripts/check-memory-rotation.py new file mode 100644 index 00000000..839ef005 --- /dev/null +++ b/scripts/check-memory-rotation.py @@ -0,0 +1,520 @@ +#!/usr/bin/env python3 +"""Which registered projects' memory moved since their bookmark - step 0 of the rotation. + +Usage +----- + uv run --no-sync python3 scripts/check-memory-rotation.py --bookmark + uv run --no-sync python3 scripts/check-memory-rotation.py --bookmark --all + uv run --no-sync python3 scripts/check-memory-rotation.py --bookmark --json + + # the defaults both point at the paths this host uses + uv run --no-sync python3 scripts/check-memory-rotation.py + +Why this exists +--------------- +The evolution prompt's §1.5 makes every cycle answer one question before it reads +anything: **which registered projects' memory is newer than that project's +bookmark line, and what is the newest file**. Step 0 is the step that is never +skipped, and nothing in this repository performed it - `check-memory-index.py` +measures an index against the index rule's two numbers and never looks at +`projects.yml`. Every cycle therefore hand-wrote the probe in a `bash` heredoc, +and it answered wrongly in two ways that both read as a clean result. + +**Keying.** A probe that keys rows by `projects.yml`'s `name` field cannot +resolve every row the bookmark carries. The bookmark writes a row's key as +whatever disambiguates that project, and this host's bookmark has three forms: +`emrg` (a bare name), `aitokenpool (github)` (a name plus the parenthetical that +separates two projects sharing it) and `gitlab/shaoaq/mem` (a path suffix, for a +project whose name alone would be ambiguous). Measured in the cycle that filed +issue #2408: a name-keyed probe reported **12 of 31** projects as changed or +unvisited, of which **10** were artefacts of the keying - the rows existed, keyed +another way, so ten projects were invisible to the rotation while the reading +said they had nothing to show. + +**The comparison.** `find -newermt ''` parses a stamp written +`%Y-%m-%d %H:%M:%S` and answers **0 files** for the ISO `T` form - the shell +form's silent failure, which makes every project read as unchanged for ever and +is why the bookmark states that measurement as a rule. This tool does not hand +the stamp to any external parser: it parses the stamp itself and compares it +against an `os.stat` mtime, so the format is a reading this file can be wrong +about out loud (a stamp that does not parse is exit **2**, never "unchanged"). + +What it reads, and what it prints first +-------------------------------------- +Two subjects, both named before any verdict: the `projects.yml` (the register of +projects this installation tracks) and the **bookmark** (one Markdown-table row +per project: the newest memory modification time already read, and the cycle +that read it). Per project it looks under `.emrg/memory/**/*.md` and under +`.emrg/sessions/*/memory/**/*.md` - the two places the rotation's rule names - and +keeps the newest file strictly newer than that project's stamp. + +Each project lands in exactly one state: + + changed a memory file is newer than its stamp; the newest is printed + unchanged nothing newer under either memory root + no memory root the project is on disk and holds no memory file at all (a memory + directory with nothing in it is nothing to read) + path missing the project's registered path is not on disk + +The last two are states a row may **carry as a note** instead of a stamp, and a note is a reading +taken on a day rather than a property: both are re-measured on every run, and a note the disk now +contradicts (`row says path missing`, `row says no memory root`) is a fault with the row named. +Repeating a note would hide a project that came back, or one that started writing memory, for +ever - the same stale-past-state defect as comparing a whole second against a sub-second mtime. + no bookmark row the project has a line in `projects.yml` and none in the + bookmark: it has never been visited, and §1.5 reads it + +Exit codes +---------- + 0 a reading was taken, every project has a row, and every row belongs to a project + 1 the coverage the bookmark exists to guarantee is broken: a project with no row, + a row naming no project, a row that matches more than one, or a row whose state + field is none of the three forms above + 2 nothing was measured: no `projects.yml`, no bookmark, or a stamp that cannot be + ordered against an mtime - which is never a pass + +What it does not do +------------------- +It **measures and never repairs**: it does not rewrite the bookmark, add a row, or +touch any project's memory. Advancing a row needs a reader's judgement (which +project was actually read, and to what depth), and §1.5 already says who makes it. + +`--all` lists every project with its state; the default lists the projects that +moved plus every fault, because the ordinary cycle has two or three of the first +and none of the second, and 31 unchanged lines are 31 lines of noise in a prompt. +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import sys +from datetime import datetime +from pathlib import Path + +#: The one stamp format a bookmark row may carry, and the one the shell form parses. +#: Spelled as a constant because the comparison below is the only reader of it: a +#: value that does not fit is unmeasurable, and this is the string that decides. +STAMP_FORMAT = "%Y-%m-%d %H:%M:%S" + +#: The two states a row may carry instead of a stamp. Both are the reader's own note +#: about the project rather than a time, so neither can be ordered against an mtime. +NO_MEMORY_ROOT = "no memory root" +PATH_MISSING_PREFIX = "path missing" + +#: A body row's **key cell**: a code span, with an optional disambiguator that may sit +#: inside the span or just after it. Both forms are in this host's bookmark - +#: `aitokenpool (github)` and `` `1` (Downloads/1) `` - and a parser that insisted on one +#: of them dropped the other row in silence: measured in the cycle that filed issue #2408, +#: a span-only pattern read 30 of 31 rows and reported the missing project as never +#: visited. +KEY_CELL = re.compile(r"^`(?P[^`]+)`\s*(?:\((?P[^)]*)\))?$") + +#: What a row's **value cell** carries: a stamp, or one of the two notes. A cell may append +#: a parenthetical - this host writes `path missing (gone from disk)`, another writes the +#: stamp followed by the memory file it was read from - so the note is read from the cell +#: with any trailing parenthetical removed. +CELL_PARENTHETICAL = re.compile(r"\s*\([^)]*\)\s*$") + +#: A markdown table's separator line, one cell at a time: `---`, `:---`, `---:`, `:---:`. +SEPARATOR_CELL = re.compile(r"^:?-+:?$") + +#: What a stamp cell looks like **before** it is parsed: a leading date. The cell is located +#: by this shape and then parsed strictly by `project_state`, so a stamp written in a form +#: this tool cannot order - the ISO `T` form, whose silent failure is one of the reasons the +#: tool exists - is located here and refused there, with the value named in the message. +#: Parsing here instead would answer "this row is unreadable", which is true but says less. +DATE_SHAPED = re.compile(r"^\d{4}-\d{1,2}-\d{1,2}[ T]") + +#: A row is **not** a fixed number of cells. §1.5 fixes what a row *states* - the project, +#: the stamp, and the cycle that read it - not how many columns a table lays those out in: +#: this installation writes three (`| `emrg` | stamp | files |`) and another writes four +#: (`| project | path | stamp | read by |`). Measured on the three-cell-anchored pattern an +#: earlier revision carried, the four-column file matched **0 rows**, and the reading that +#: followed was a false statement about live data ("every project is at or before its +#: stamp") rather than an unmeasurable one. So the value cell is found rather than counted: +#: it is the first cell after the key that carries a stamp or a note, whatever its index. + +#: A project's memory, in the two places §1.5's rule names. +MEMORY_SUBDIR = (".emrg", "memory") +SESSIONS_SUBDIR = (".emrg", "sessions") + +#: Both defaults are this host's paths. They are defaults rather than required +#: arguments because step 0 runs every cycle on this installation; any other +#: layout is reached by naming the two paths, and the report says which it read. +DEFAULT_PROJECTS = "~/.emrg/projects.yml" +DEFAULT_BOOKMARK = "~/.emrg/evolution/emrg/.emrg/memory/project-memory-rotation-bookmark.md" + + +class Unmeasurable(Exception): + """A reading could not be taken. Reported as exit 2, never as a pass.""" + + +def load_projects(path: Path) -> list[dict]: + """The register: one record per tracked project, each with a `name` and a `path`.""" + try: + import yaml + except ImportError as exc: # pragma: no cover - the repository depends on PyYAML + raise Unmeasurable(f"PyYAML is not importable, so no register can be read: {exc}") from exc + if not path.is_file(): + raise Unmeasurable(f"no project register at {path}") + try: + data = yaml.safe_load(path.read_text(encoding="utf-8")) + except (OSError, ValueError) as exc: + raise Unmeasurable(f"the project register at {path} could not be read: {exc}") from exc + if not isinstance(data, list): + raise Unmeasurable(f"the project register at {path} is not a list of records") + records = [] + for record in data: + if isinstance(record, dict) and record.get("name") is not None: + # A register is machine-written YAML, so a project whose name looks numeric + # (`1`, this host has one) arrives as an `int`. Coerced here because the + # bookmark writes that key as text: left alone, the row is a `str` and the + # project an `int`, so `==` answers no and the project reads as never visited. + records.append(dict(record, name=str(record["name"]))) + return records + + +def _cells(line: str) -> list[str]: + """A table line's cells, stripped, with the two halves outside its outer pipes gone.""" + return [cell.strip() for cell in line.strip()[1:-1].split("|")] + + +def _is_table_line(line: str) -> bool: + """Whether this line is a row of a markdown table: it opens and closes on a pipe.""" + stripped = line.strip() + return len(stripped) > 2 and stripped.startswith("|") and stripped.endswith("|") + + +def _value_of(cells: list[str]) -> str | None: + """The stamp or note a row carries, from wherever its table puts it. + + Found rather than counted, because the column's index belongs to the installation + writing the bookmark and not to §1.5's rule (see the block above `SEPARATOR_CELL`). A + cell qualifies when it is one of the two notes or opens on a date, so a path, a count + or a cycle id in the same row is skipped instead of mistaken for the value. Whether a + date-shaped cell is a stamp this tool can order is `project_state`'s question. + """ + for cell in cells: + for candidate in (CELL_PARENTHETICAL.sub("", cell).strip(), cell): + if candidate == NO_MEMORY_ROOT or candidate.startswith(PATH_MISSING_PREFIX): + return candidate + if DATE_SHAPED.match(candidate): + return candidate + return None + + +def parse_bookmark(path: Path) -> list[dict]: + """Every body row of the bookmark's table, in file order, as key/value pairs. + + A table this reader cannot read is **not** an empty table. A row whose key cell is a + code span - the shape every bookmark row has - but which carries no stamp and no note + in any of its cells is reported as unmeasurable, because reading it as "no row" would + report its project as never visited while its row sits in the file, and that is the + defect this tool was written to end (issue #2408). + """ + if not path.is_file(): + raise Unmeasurable(f"no bookmark at {path}") + try: + text = path.read_text(encoding="utf-8") + except (OSError, ValueError) as exc: + raise Unmeasurable(f"the bookmark at {path} could not be read: {exc}") from exc + lines = text.splitlines() + rows = [] + unreadable = [] + for index, line in enumerate(lines): + if not _is_table_line(line): + continue + cells = _cells(line) + if not cells or all(SEPARATOR_CELL.match(cell) for cell in cells if cell): + continue # the header's separator, not a body row + key_cell = KEY_CELL.match(cells[0]) if cells else None + if key_cell is None: + continue # prose, or a header written without a code span + following = next((l for l in lines[index + 1:] if l.strip()), "") + if _is_table_line(following) and all( + SEPARATOR_CELL.match(cell) for cell in _cells(following) if cell): + continue # a header written as `| `project` | stamp | … |` + value = _value_of(cells[1:]) + if value is None: + unreadable.append(f"line {index + 1}: {line.strip()}") + continue + key = key_cell.group("key") + hint = key_cell.group("hint") + rows.append({"key": f"{key} ({hint})" if hint else key, "value": value}) + if unreadable: + raise Unmeasurable( + f"the bookmark at {path} carries {len(unreadable)} row(s) this reader cannot " + f"read - a row names its project in a code span and then no stamp and no note " + f"in any cell - so the reading below would call their projects never visited: " + + "; ".join(unreadable) + ) + return rows + + +def _segments(path: str) -> list[str]: + """A path's segments, host and platform separators alike, empty ones dropped.""" + return [s for s in re.split(r"[\\/]+", path) if s] + + +def _key_matches_path(key: str, project_path: str) -> bool: + """Whether a path-suffix key names this project's path. + + The bookmark's third form is a suffix (`gitlab/shaoaq/mem`), written because the + project's own name would not identify it. A segment matches when it equals the + path's own segment or is a prefix of it, so `gitlab` names `gitlab.xinluex.com` + without the row having to repeat the host's whole domain. + """ + key_segments = _segments(key) + path_segments = _segments(project_path) + if not key_segments or len(key_segments) > len(path_segments): + return False + tail = path_segments[-len(key_segments):] + return all(k == p or p.startswith(k) for k, p in zip(key_segments, tail)) + + +def resolve_rows(rows: list[dict], projects: list[dict]) -> tuple[dict, list[dict], list[dict]]: + """Match each row to the project it names. + + Three forms, tried in this order and for this reason: a bare `name` first (the + common case, and it cannot be ambiguous while only one project carries that name); + then a name with a parenthetical disambiguator, resolved by matching the hint + against the project's path (`aitokenpool (github)`); then a path suffix + (`gitlab/shaoaq/mem`). A row that matches no project, or more than one, is + returned rather than guessed at. + """ + by_key: dict[str, dict] = {} + orphans: list[dict] = [] + ambiguous: list[dict] = [] + for row in rows: + key = row["key"] + candidates: list[dict] = [] + name, _, hint = key.partition(" (") + hint = hint.rstrip(")") + named = [p for p in projects if p["name"] == name] + if named and not hint: + candidates = named + elif named and hint: + candidates = [p for p in named if hint in (p.get("path") or "")] + if not candidates: + candidates = [p for p in named if _key_matches_path(hint, p.get("path") or "")] + if not candidates: + candidates = [p for p in projects + if _key_matches_path(key, p.get("path") or "") + and p["name"] != key] + if len(candidates) > 1: + ambiguous.append({"key": key, "projects": [p["name"] for p in candidates], + "paths": [p.get("path") for p in candidates]}) + continue + if not candidates: + orphans.append(row) + continue + by_key[candidates[0].get("path") or candidates[0]["name"]] = row + return by_key, orphans, ambiguous + + +def memory_roots(root: Path) -> list[Path]: + """The project's memory directories: its own, and each of its sessions'.""" + found = [] + own = root.joinpath(*MEMORY_SUBDIR) + if own.is_dir(): + found.append(own) + sessions = root.joinpath(*SESSIONS_SUBDIR) + if sessions.is_dir(): + try: + names = sorted(sessions.iterdir()) + except OSError: + names = [] + for entry in names: + candidate = entry / "memory" + if candidate.is_dir(): + found.append(candidate) + return found + + +def memory_files(root: Path) -> list[Path]: + """Every `.md` under the project's memory roots - the files the rotation's rule binds. + + A directory is not a reading. Measured in the cycle that filed issue #2408: three + projects on this host carry an **empty** `.emrg/sessions/*/memory/` directory, and a + reading that counted the directory as "this project keeps memory" reported all three as + contradicting their `no memory root` note. Nothing was there to read. + """ + found = [] + for memory_root in memory_roots(root): + for dirpath, _dirnames, filenames in os.walk(memory_root): + for name in filenames: + if name.endswith(".md"): + found.append(Path(dirpath) / name) + return found + + +def newest_memory(root: Path, since: int) -> tuple[Path, float] | None: + """The newest `.md` under the project's memory roots that is strictly newer than `since`. + + `since` is a **whole second**, and an mtime is truncated to one before it is compared. + The bookmark is written from `stat`'s `%Sm`, which carries no fraction, so a + sub-second comparison reports the very file a row was written from as newer for ever: + measured in the cycle that filed issue #2408, **21 of 23** projects read as changed + against stamps that already named their newest file - `.real` at `10:19:27.829` against + a `10:19:27` row. Truncating both sides is what makes the reading idempotent, and it + loses nothing: a file that really moved lands in a later second. + """ + best: tuple[Path, float] | None = None + for path in memory_files(root): + try: + mtime = path.stat().st_mtime + except OSError: + continue + if int(mtime) <= since: + continue + if best is None or mtime > best[1]: + best = (path, mtime) + return best + + +def project_state(project: dict, row: dict | None) -> dict: + """One project's line: its state, and the reading behind it.""" + name = project["name"] + raw_path = project.get("path") or "" + state = {"name": name, "path": raw_path, "key": row["key"] if row else None} + if row is None: + state.update(state="no bookmark row", bookmark=None, newest=None) + return state + value = row["value"] + root = Path(os.path.expanduser(raw_path)) + if value == NO_MEMORY_ROOT: + # A note is a reading taken on a day, so it is re-measured rather than repeated: a + # project that has since written a memory root is one the note would hide for ever, + # which is the same class of stale-past-state defect as the truncated stamp above. + state.update(state=NO_MEMORY_ROOT if not memory_files(root) else "row says no memory root", + bookmark=None, newest=None) + return state + if value.startswith(PATH_MISSING_PREFIX): + state.update(state=PATH_MISSING_PREFIX if not root.is_dir() else "row says path missing", + bookmark=None, newest=None) + return state + try: + stamp = datetime.strptime(value, STAMP_FORMAT) + except ValueError as exc: + raise Unmeasurable( + f"the row for {name} carries {value!r}, which is not a {STAMP_FORMAT} stamp " + f"and cannot be ordered against a file's mtime ({exc})" + ) from exc + state["bookmark"] = stamp.strftime(STAMP_FORMAT) + if not root.is_dir(): + state.update(state="path missing", newest=None) + return state + if not memory_files(root): + state.update(state=NO_MEMORY_ROOT, newest=None) + return state + newest = newest_memory(root, int(stamp.timestamp())) + if newest is None: + state.update(state="unchanged", newest=None) + return state + state.update(state="changed", newest={"path": str(newest[0]), + "mtime": datetime.fromtimestamp(newest[1]).strftime(STAMP_FORMAT)}) + return state + + +def read(projects_path: Path, bookmark_path: Path) -> dict: + """The whole reading: both subjects, every project's state, and the coverage faults.""" + projects = load_projects(projects_path) + rows = parse_bookmark(bookmark_path) + by_path, orphans, ambiguous = resolve_rows(rows, projects) + states = [project_state(p, by_path.get(p.get("path") or p["name"])) for p in projects] + faults = [] + for state in states: + if state["state"] == "no bookmark row": + faults.append(f"{state['name']} ({state['path']}) has a line in the register and none in " + f"the bookmark: it has never been visited, and §1.5 reads it") + for state in states: + if state["state"] == "row says path missing": + faults.append(f"the bookmark's row for {state['name']} says its path is missing, but " + f"{state['path']} is on disk: the row is a note from an earlier day and " + f"needs a real stamp") + elif state["state"] == "row says no memory root": + faults.append(f"the bookmark's row for {state['name']} says it has no memory root, but " + f"{state['path']} has one now: the row is a note from an earlier day and " + f"needs a real stamp") + for row in orphans: + faults.append(f"the bookmark's row {row['key']!r} names no registered project") + for item in ambiguous: + faults.append(f"the bookmark's row {item['key']!r} names {len(item['projects'])} projects " + f"({', '.join(item['projects'])}): the key has to disambiguate them") + return { + "projects_file": str(projects_path), + "bookmark_file": str(bookmark_path), + "project_count": len(projects), + "row_count": len(rows), + "states": states, + "faults": faults, + } + + +def render(reading: dict, show_all: bool) -> str: + """The report, with both subjects named before any verdict.""" + out = [ + f"projects: {reading['projects_file']} ({reading['project_count']} project(s))", + f"bookmark: {reading['bookmark_file']} ({reading['row_count']} row(s))", + ] + changed = [s for s in reading["states"] if s["state"] == "changed"] + counts: dict[str, int] = {} + for state in reading["states"]: + counts[state["state"]] = counts.get(state["state"], 0) + 1 + if changed: + out.append(f"changed since their bookmark: {len(changed)}") + for state in changed: + out.append(f" {state['name']} {state['bookmark']} -> {state['newest']['mtime']} " + f"{state['newest']['path']}") + else: + out.append("changed since their bookmark: 0 - every project is at or before its stamp") + if show_all: + others = [s for s in reading["states"] if s["state"] != "changed"] + if others: + out.append("every other project:") + for state in others: + out.append(f" {state['state']:<16} {state['name']} {state['path']}") + summary = ", ".join(f"{name} {count}" for name, count in sorted(counts.items())) + out.append(f"states: {summary}" if summary else "states: none recorded") + if reading["faults"]: + out.append(f"coverage faults: {len(reading['faults'])}") + out.extend(f" {fault}" for fault in reading["faults"]) + out.append("a project with no row is read this cycle, and the bookmark's next write carries " + "a line for it; a row naming no project is a stale or mistyped key") + else: + out.append(f"coverage: every project has a row and every row names a project " + f"({reading['project_count']} of {reading['project_count']})") + return "\n".join(out) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="Which registered projects' memory moved since their bookmark line.") + parser.add_argument("--projects", default=DEFAULT_PROJECTS, + help=f"the project register (default {DEFAULT_PROJECTS})") + parser.add_argument("--bookmark", default=DEFAULT_BOOKMARK, + help=f"the rotation bookmark (default {DEFAULT_BOOKMARK})") + parser.add_argument("--all", action="store_true", + help="list every project with its state, not only the ones that moved") + parser.add_argument("--json", action="store_true", help="the same reading, as one document") + args = parser.parse_args(argv) + + projects_path = Path(os.path.expanduser(args.projects)) + bookmark_path = Path(os.path.expanduser(args.bookmark)) + try: + reading = read(projects_path, bookmark_path) + except Unmeasurable as exc: + print(f"not measurable: {exc}", file=sys.stderr) + return 2 + if args.json: + print(json.dumps(reading, indent=2, ensure_ascii=False)) + else: + print(render(reading, args.all)) + return 1 if reading["faults"] else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/test_a_tree_reading_guard_names_its_tree.py b/tests/test_a_tree_reading_guard_names_its_tree.py index 4dd76d18..35222ee9 100644 --- a/tests/test_a_tree_reading_guard_names_its_tree.py +++ b/tests/test_a_tree_reading_guard_names_its_tree.py @@ -312,6 +312,19 @@ def _refusal_reason(out: str) -> str: "answer anything (the repo it read is printed first instead)" ), "check-patch-files.py": "reads the patch files it is given as arguments", + "check-memory-rotation.py": ( + "reads two host-state subjects it names before any verdict - the project register " + "(`~/.emrg/projects.yml`) and the rotation bookmark (a Markdown table of the newest " + "memory modification time already read per project) - then walks each project's " + "`.emrg/memory/` and `.emrg/sessions/*/memory/`, which are gitignored host data no " + "checkout carries. A working tree is not its subject and no tree's name would answer " + "anything: the questions are which projects moved and whether every registered " + "project has a row, and both are answered about the host's data. Its unmeasurable " + "answer (rc 2: no register, no bookmark, a stamp that cannot be ordered) and its " + "coverage faults (rc 1) are pinned by tests/test_check_memory_rotation.py - added by " + "issue #2408, where step 0 was a hand-written probe that read 30 of 31 rows and " + "reported 21 projects as changed against rows that already named their newest file" + ), "check-batching-share.py": ( "reads a corpus of session history files - `history_*.jsonl` under the roots given " "as positional subjects, defaulting to the host's `~/.emrg` - and counts the tool " diff --git a/tests/test_check_memory_rotation.py b/tests/test_check_memory_rotation.py new file mode 100644 index 00000000..584c0e65 --- /dev/null +++ b/tests/test_check_memory_rotation.py @@ -0,0 +1,467 @@ +"""The rotation reading: `scripts/check-memory-rotation.py`. + +What this file holds +-------------------- +The tool answers §1.5's step 0 - which registered projects' memory is newer than that +project's bookmark row - and it exists because two hand-written probes answered it +wrongly while reading as clean (issue #2408). So the assertions here are the three +halves of "does it answer that question, and can a wrong answer look right": + +* **each boundary is read in both states, and they are built from what the tool reads, + never from a literal** - a file *at* its stamp is `unchanged` and one *after* it is + `changed`; the file a row was written from, carrying a sub-second fraction, is + `unchanged` (a comparison that kept the fraction reported **21 of 23** projects as + changed on this host while their rows already named that very file); a project with a + memory root and one without; a path on disk and a path that is gone. +* **all three key forms the bookmark writes are resolved** - a bare name, a name with a + parenthetical disambiguator (inside the code span and just after it, because this + host's bookmark carries both), and a path suffix. A form the parser drops is a project + that leaves the rotation in silence, which is the failure the tool was written for. +* **a reading that could not be taken is never a pass** - no register, no bookmark, a + stamp that does not parse, and a row that matches two projects all answer exit **2** + or exit **1** with the reason named, and `0` is reserved for a reading that covered + every project and every row; +* **a note from an earlier day is re-measured, not repeated** - `no memory root` and + `path missing` are readings, so a row carrying one the disk now contradicts is a fault + rather than a verdict that hides a project for ever. + +It **measures and never repairs**: one assertion reads the bookmark's bytes before and +after a run, because a tool that quietly rewrote rows would hide exactly the drift its +report exists to show. + +Loading +------- +By path (`spec_from_file_location`), the shape `test_check_memory_index.py` uses: the +filename is not importable by name (the dash), and the module stands alone - it imports +nothing from the package, so there is no tree to put on `sys.path`. +""" + +from __future__ import annotations + +import importlib.util +import json +import os +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parent.parent +SCRIPT = REPO_ROOT / "scripts" / "check-memory-rotation.py" + +#: One stamp, in the tool's own format. Every fixture time is an offset from it, so the +#: reading and the test cannot drift apart by a hand-written date. +BASE = "2026-10-20 16:06:24" + + +def _load(): + spec = importlib.util.spec_from_file_location("check_memory_rotation", SCRIPT) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +@pytest.fixture(scope="module") +def mod(): + return _load() + + +def _stamp(mod, seconds: int = 0) -> float: + """`BASE` plus `seconds`, as an mtime the tool will compare against.""" + from datetime import datetime, timedelta + + return (datetime.strptime(BASE, mod.STAMP_FORMAT) + timedelta(seconds=seconds)).timestamp() + + +def _write_memory(root: Path, relative: str, mtime: float) -> Path: + """A memory file at `relative` under `root`, stamped `mtime`.""" + path = root / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("# memory\n", encoding="utf-8") + os.utime(path, (mtime, mtime)) + return path + + +def _projects(path: Path, records: list[dict]) -> Path: + """The register, written the way the daemon writes it: every scalar quoted.""" + text = "".join(f"- name: '{r['name']}'\n path: '{r['path']}'\n" for r in records) + path.write_text(text, encoding="utf-8") + return path + + +def _bookmark(path: Path, rows: list[tuple[str, str]]) -> Path: + """A bookmark with a title, the table's two header lines, and one row per pair.""" + lines = ["# Bookmark", "", "| project | newest read | files |", "| --- | --- | --- |"] + lines += [f"| {key} | {value} | 0 |" for key, value in rows] + path.write_text("\n".join(lines) + "\n", encoding="utf-8") + return path + + +def _reading(mod, tmp_path: Path, projects: list[dict], rows: list[tuple[str, str]]): + register = _projects(tmp_path / "projects.yml", projects) + book = _bookmark(tmp_path / "bookmark.md", rows) + return mod.read(register, book) + + +def _root(tmp_path: Path, name: str) -> str: + root = tmp_path / "projects" / name + root.mkdir(parents=True, exist_ok=True) + return str(root) + + +# --------------------------------------------------------------------------- the states + + +def test_a_file_at_its_stamp_is_unchanged_and_one_second_later_is_not(mod, tmp_path): + """The boundary, in both states, from the stamp the tool itself parses.""" + at, after = _root(tmp_path, "at"), _root(tmp_path, "after") + _write_memory(Path(at), ".emrg/memory/MEMORY.md", _stamp(mod)) + _write_memory(Path(after), ".emrg/memory/MEMORY.md", _stamp(mod, 1)) + reading = _reading(mod, tmp_path, + [{"name": "at", "path": at}, {"name": "after", "path": after}], + [("`at`", BASE), ("`after`", BASE)]) + states = {s["name"]: s for s in reading["states"]} + assert states["at"]["state"] == "unchanged" + assert states["after"]["state"] == "changed" + assert states["after"]["newest"]["mtime"] == "2026-10-20 16:06:25" + assert reading["faults"] == [] + + +def test_a_sub_second_fraction_does_not_re_flag_the_file_its_row_was_written_from(mod, tmp_path): + """The row is written from `stat`'s second-precision `%Sm`; the file keeps a fraction. + + This is the defect the tool was written against: a comparison that kept the fraction + reported 21 of 23 projects as changed on this host while their rows already named the + very files it was reporting. + """ + root = _root(tmp_path, "fraction") + _write_memory(Path(root), ".emrg/memory/MEMORY.md", _stamp(mod) + 0.825689) + reading = _reading(mod, tmp_path, [{"name": "fraction", "path": root}], [("`fraction`", BASE)]) + assert reading["states"][0]["state"] == "unchanged" + + +def test_a_sessions_memory_counts_and_a_non_markdown_file_does_not(mod, tmp_path): + """Both roots §1.5 names are read, and only the files its rule names.""" + root = _root(tmp_path, "sessioned") + _write_memory(Path(root), ".emrg/memory/MEMORY.md", _stamp(mod)) + _write_memory(Path(root), ".emrg/sessions/s_1/memory/note.md", _stamp(mod, 5)) + _write_memory(Path(root), ".emrg/memory/notes.txt", _stamp(mod, 9)) + reading = _reading(mod, tmp_path, [{"name": "sessioned", "path": root}], + [("`sessioned`", BASE)]) + state = reading["states"][0] + assert state["state"] == "changed" + assert Path(state["newest"]["path"]).parts[-3:] == ("s_1", "memory", "note.md"), state["newest"] + assert not state["newest"]["path"].endswith(".txt") + + +def test_the_newest_of_several_is_the_one_named(mod, tmp_path): + """A reading that kept the first file it walked would name the wrong one.""" + root = _root(tmp_path, "several") + _write_memory(Path(root), ".emrg/memory/a.md", _stamp(mod, 3)) + newest = _write_memory(Path(root), ".emrg/memory/b.md", _stamp(mod, 7)) + reading = _reading(mod, tmp_path, [{"name": "several", "path": root}], [("`several`", BASE)]) + assert reading["states"][0]["newest"]["path"] == str(newest) + + +def test_a_project_with_no_memory_root_and_one_whose_path_is_gone(mod, tmp_path): + """Two states that are not a time: neither can be ordered against an mtime.""" + bare = _root(tmp_path, "bare") + gone = str(tmp_path / "projects" / "gone") + reading = _reading(mod, tmp_path, + [{"name": "bare", "path": bare}, {"name": "gone", "path": gone}], + [("`bare`", "no memory root"), ("`gone`", "path missing (gone from disk)")]) + states = {s["name"]: s for s in reading["states"]} + assert states["bare"]["state"] == "no memory root" + assert states["gone"]["state"] == "path missing" + assert reading["faults"] == [] + + +def test_a_note_the_disk_contradicts_is_a_fault_in_both_directions(mod, tmp_path): + """A row may carry `no memory root` / `path missing` instead of a stamp - and those are + readings from a day, not properties. Repeating a note would hide a project that started + writing memory or came back on disk, for ever: the class of stale-past-state defect the + sub-second comparison belongs to as well. Each direction is asserted, and each is + asserted where the note is still *true* too, so the fault cannot fire unconditionally. + """ + now_memory = _root(tmp_path, "now-memory") + _write_memory(Path(now_memory), ".emrg/memory/MEMORY.md", _stamp(mod)) + still_bare = _root(tmp_path, "still-bare") + back = _root(tmp_path, "back") + still_gone = str(tmp_path / "projects" / "still-gone") + + reading = _reading(mod, tmp_path, + [{"name": "now-memory", "path": now_memory}, + {"name": "still-bare", "path": still_bare}, + {"name": "back", "path": back}, + {"name": "still-gone", "path": still_gone}], + [("`now-memory`", "no memory root"), ("`still-bare`", "no memory root"), + ("`back`", "path missing (gone from disk)"), + ("`still-gone`", "path missing (gone from disk)")]) + states = {s["name"]: s["state"] for s in reading["states"]} + assert states["now-memory"] == "row says no memory root" + assert states["still-bare"] == "no memory root" + assert states["back"] == "row says path missing" + assert states["still-gone"] == "path missing" + assert len(reading["faults"]) == 2, reading["faults"] + assert any("has one now" in f for f in reading["faults"]) + assert any("is on disk" in f for f in reading["faults"]) + + +def test_an_empty_memory_directory_is_nothing_to_read(mod, tmp_path): + """A directory is not a reading: three projects on this host carry an empty + `.emrg/sessions/*/memory/`, and counting the directory reported all three as + contradicting their `no memory root` note while nothing was there to read.""" + empty = _root(tmp_path, "empty-root") + (Path(empty) / ".emrg" / "sessions" / "s_1" / "memory").mkdir(parents=True) + reading = _reading(mod, tmp_path, [{"name": "empty-root", "path": empty}], + [("`empty-root`", "no memory root")]) + assert reading["states"][0]["state"] == "no memory root" + assert reading["faults"] == [], reading["faults"] + + +# ----------------------------------------------------------------------- the three keys + + +def test_all_three_key_forms_the_bookmark_writes_resolve(mod, tmp_path): + """A bare name, a parenthetical that separates two same-named projects, a path suffix. + + The middle form is asserted in **both** spellings, because this host's bookmark + carries both: the disambiguator inside the code span (`aitokenpool (github)`) and just + after it (`1` (Downloads/1)). A parser that insisted on one dropped the other row and + reported its project as never visited. + """ + git = _root(tmp_path, "github/aitokenpool") + lab = _root(tmp_path, "gitlab.xinluex.com/shaoaq/aitokenpool") + mem = _root(tmp_path, "gitlab.xinluex.com/shaoaq/mem") + download = _root(tmp_path, "Downloads/1") + for root in (git, lab, mem, download): + _write_memory(Path(root), ".emrg/memory/MEMORY.md", _stamp(mod)) + projects = [ + {"name": "aitokenpool", "path": git}, + {"name": "aitokenpool", "path": lab}, + {"name": "mem", "path": mem}, + {"name": "1", "path": download}, + ] + reading = _reading(mod, tmp_path, projects, [ + ("`aitokenpool` (github)", BASE), # a name, hint inside the span + ("`gitlab/shaoaq/aitokenpool`", BASE), # a path suffix, same name as the first + ("`gitlab/shaoaq/mem`", BASE), # a path suffix for an unambiguous name + ("`1` (Downloads/1)", BASE), # a name, hint just after the span + ]) + by_key = {s["key"]: s for s in reading["states"]} + assert by_key["aitokenpool (github)"]["path"] == git + assert by_key["gitlab/shaoaq/aitokenpool"]["path"] == lab + assert by_key["gitlab/shaoaq/mem"]["path"] == mem + assert by_key["1 (Downloads/1)"]["path"] == download + assert reading["faults"] == [], reading["faults"] + + +def test_a_project_whose_name_looks_numeric_still_matches_its_row(mod, tmp_path): + """Machine-written YAML hands `1` over as an `int`, and the row writes it as text. + + Left as the loader found it, the comparison is `int == str` and answers no: the project + reads as never visited while its row sits in the file. This host registers a project by + exactly that name. + """ + root = _root(tmp_path, "numeric") + _write_memory(Path(root), ".emrg/memory/MEMORY.md", _stamp(mod)) + register = tmp_path / "projects.yml" + register.write_text(f"- name: 1\n path: '{root}'\n", encoding="utf-8") + book = _bookmark(tmp_path / "bookmark.md", [("`1`", BASE)]) + reading = mod.read(register, book) + assert reading["states"][0]["name"] == "1" + assert reading["states"][0]["state"] == "unchanged" + assert reading["faults"] == [] + + +def test_a_bare_name_shared_by_two_projects_is_reported_rather_than_guessed(mod, tmp_path): + """Two projects, one name, one bare row: which was read is a judgement, not a match.""" + one, two = _root(tmp_path, "one"), _root(tmp_path, "two") + reading = _reading(mod, tmp_path, + [{"name": "twin", "path": one}, {"name": "twin", "path": two}], + [("`twin`", BASE)]) + assert len(reading["faults"]) == 3, reading["faults"] # ambiguous row + two unrowed + assert any("names 2 projects" in f for f in reading["faults"]) + assert sum(1 for f in reading["faults"] if "never been visited" in f) == 2 + +# -------------------------------------------------------- measuring, and never a pass + + +def test_a_row_is_found_in_a_table_of_any_width(mod, tmp_path): + """§1.5 fixes what a row states, not how many columns the table lays it out in. + + This installation writes three cells (`| `name` | stamp | files |`); another writes + four (`| project | path | stamp | read by |`), and three cells was the whole of the + pattern an earlier revision matched on. Measured against a four-column bookmark, that + pattern read **0 rows**, so every project came back `no bookmark row` and the report + said "every project is at or before its stamp" - a false statement about live data, + where the shape is exactly what the repository has twice refused to answer (#2275, + #2277: *could not read* narrowed into *read as empty*). Both files must read alike. + """ + wide = _root(tmp_path, "wide") + narrow = _root(tmp_path, "narrow") + for root in (wide, narrow): + _write_memory(Path(root), ".emrg/memory/MEMORY.md", _stamp(mod, 60)) + register = _projects(tmp_path / "projects.yml", [ + {"name": "wide", "path": wide}, {"name": "narrow", "path": narrow}]) + + # The same two rows, in the two tables the two installations write. The wide one puts + # the project's path and the reading cycle on either side of the stamp, so a reader + # that counted cells would take the path for the value. + narrow_book = _bookmark(tmp_path / "narrow.md", [("`wide`", BASE), ("`narrow`", BASE)]) + wide_book = tmp_path / "wide.md" + wide_book.write_text("\n".join([ + "# Bookmark", "", + "| project | path | newest | read by |", + "| --- | --- | --- | --- |", + f"| `wide` | {wide} | {BASE} (`MEMORY.md`) | cyc20261020-141450 |", + f"| `narrow` | {narrow} | {BASE} | cyc20261020-141450 |", + ]) + "\n", encoding="utf-8") + + narrow_reading = mod.read(register, narrow_book) + wide_reading = mod.read(register, wide_book) + assert wide_reading["row_count"] == 2 + assert wide_reading["faults"] == [], wide_reading["faults"] + assert [(s["key"], s["state"], s["bookmark"]) for s in wide_reading["states"]] == \ + [(s["key"], s["state"], s["bookmark"]) for s in narrow_reading["states"]] + # And it is a reading, not an empty one that happens to agree: both files say `changed`. + assert [s["state"] for s in wide_reading["states"]] == ["changed", "changed"] + + +def test_a_row_that_carries_no_stamp_is_unmeasurable_rather_than_dropped(mod, tmp_path): + """The other half: a table this reader cannot read must not answer as an empty one. + + With the rows located by shape, the remaining failure is a row that names its project + in a code span and then carries neither a stamp nor a note in any cell. Dropping it + would report that project as never visited while its row sits in the file - the + original defect - so it is exit 2, and the message names the line it could not read. + """ + root = _root(tmp_path, "unreadable") + register = _projects(tmp_path / "projects.yml", [{"name": "unreadable", "path": root}]) + book = _bookmark(tmp_path / "bookmark.md", [("`unreadable`", "read on Tuesday")]) + with pytest.raises(mod.Unmeasurable) as excinfo: + mod.read(register, book) + message = str(excinfo.value) + assert "cannot" in message and "never visited" in message + assert "line 5" in message and "`unreadable`" in message + + +def test_a_header_written_with_a_code_span_is_not_a_body_row(mod, tmp_path): + """The rows are located by shape, so the header is the one shape that must not qualify. + + A table may label its columns with code spans (`| `project` | `newest` |`), and read as a + body row that header would ask for a stamp the header does not carry - turning a + readable bookmark into an unmeasurable one. Markdown decides a header by the separator + under it, which is what this pins, and the body rows beneath it still read. + """ + root = _root(tmp_path, "spanned") + _write_memory(Path(root), ".emrg/memory/MEMORY.md", _stamp(mod, 60)) + register = _projects(tmp_path / "projects.yml", [{"name": "spanned", "path": root}]) + book = tmp_path / "bookmark.md" + book.write_text("\n".join([ + "# Bookmark", "", + "| `project` | `newest` | `files` |", + "| --- | --- | --- |", + f"| `spanned` | {BASE} | 0 |", + ]) + "\n", encoding="utf-8") + reading = mod.read(register, book) + assert reading["row_count"] == 1 + assert reading["states"][0]["state"] == "changed" + assert reading["faults"] == [], reading["faults"] + + +def test_a_project_with_no_row_is_a_fault_and_is_reported_as_never_visited(mod, tmp_path): + root = _root(tmp_path, "unrowded") + reading = _reading(mod, tmp_path, [{"name": "unrowded", "path": root}], []) + assert reading["states"][0]["state"] == "no bookmark row" + assert len(reading["faults"]) == 1 + assert "never been visited" in reading["faults"][0] + + +def test_a_row_that_names_no_project_is_a_fault(mod, tmp_path): + root = _root(tmp_path, "kept") + reading = _reading(mod, tmp_path, [{"name": "kept", "path": root}], + [("`kept`", BASE), ("`ghost`", BASE)]) + assert len(reading["faults"]) == 1 + assert "names no registered project" in reading["faults"][0] + + +def test_a_stamp_that_does_not_parse_is_unmeasurable_never_unchanged(mod, tmp_path): + """The shell form's silent failure, answered out loud: the `T` form cannot be ordered.""" + root = _root(tmp_path, "iso") + _write_memory(Path(root), ".emrg/memory/MEMORY.md", _stamp(mod, 1)) + with pytest.raises(mod.Unmeasurable) as excinfo: + _reading(mod, tmp_path, [{"name": "iso", "path": root}], + [("`iso`", "2026-10-20T16:06:24")]) + assert "cannot be ordered" in str(excinfo.value) + assert "2026-10-20T16:06:24" in str(excinfo.value) + + +def test_no_register_and_no_bookmark_are_unmeasurable(mod, tmp_path): + for subject in ("register", "bookmark"): + register = _projects(tmp_path / "projects.yml", [{"name": "x", "path": str(tmp_path)}]) + book = _bookmark(tmp_path / "bookmark.md", [("`x`", BASE)]) + if subject == "register": + register.unlink() + else: + book.unlink() + with pytest.raises(mod.Unmeasurable) as excinfo: + mod.read(register, book) + assert "no project register" in str(excinfo.value) or "no bookmark" in str(excinfo.value) + + +def test_the_bookmark_is_not_rewritten(mod, tmp_path): + """It measures and never repairs: the bytes are the same after a run.""" + root = _root(tmp_path, "stable") + _write_memory(Path(root), ".emrg/memory/MEMORY.md", _stamp(mod, 4)) + register = _projects(tmp_path / "projects.yml", [{"name": "stable", "path": root}]) + book = _bookmark(tmp_path / "bookmark.md", [("`stable`", BASE)]) + before = book.read_bytes() + mod.read(register, book) + assert book.read_bytes() == before + + +# --------------------------------------------------------------------------- the report + + +def test_both_subjects_are_named_before_any_verdict(mod, tmp_path): + root = _root(tmp_path, "named") + _write_memory(Path(root), ".emrg/memory/MEMORY.md", _stamp(mod, 2)) + reading = _reading(mod, tmp_path, [{"name": "named", "path": root}], [("`named`", BASE)]) + lines = mod.render(reading, False).splitlines() + assert lines[0].startswith("projects: ") and str(tmp_path) in lines[0] + assert lines[1].startswith("bookmark: ") and "1 row(s)" in lines[1] + assert "changed since their bookmark: 1" in lines[2] + assert "coverage: every project has a row" in "\n".join(lines) + + +def test_an_empty_reading_says_zero_rather_than_printing_nothing(mod, tmp_path): + """A report that omits the changed section reads as a tool that did not look.""" + root = _root(tmp_path, "quiet") + _write_memory(Path(root), ".emrg/memory/MEMORY.md", _stamp(mod)) + reading = _reading(mod, tmp_path, [{"name": "quiet", "path": root}], [("`quiet`", BASE)]) + assert "changed since their bookmark: 0" in mod.render(reading, False) + + +def test_the_exit_codes_are_0_1_and_2_and_2_is_never_a_pass(mod, tmp_path): + """The contract a caller reads: clean, a coverage fault, unmeasurable.""" + root = _root(tmp_path, "codes") + _write_memory(Path(root), ".emrg/memory/MEMORY.md", _stamp(mod)) + register = _projects(tmp_path / "projects.yml", [{"name": "codes", "path": root}]) + clean = _bookmark(tmp_path / "clean.md", [("`codes`", BASE)]) + faulty = _bookmark(tmp_path / "faulty.md", [("`codes`", BASE), ("`ghost`", BASE)]) + assert mod.main(["--projects", str(register), "--bookmark", str(clean)]) == 0 + assert mod.main(["--projects", str(register), "--bookmark", str(faulty)]) == 1 + assert mod.main(["--projects", str(tmp_path / "absent.yml"), + "--bookmark", str(clean)]) == 2 + + +def test_json_is_one_document_that_names_its_subjects(mod, tmp_path, capsys): + root = _root(tmp_path, "documented") + _write_memory(Path(root), ".emrg/memory/MEMORY.md", _stamp(mod, 1)) + register = _projects(tmp_path / "projects.yml", [{"name": "documented", "path": root}]) + book = _bookmark(tmp_path / "bookmark.md", [("`documented`", BASE)]) + assert mod.main(["--projects", str(register), "--bookmark", str(book), "--json"]) == 0 + document = json.loads(capsys.readouterr().out) + assert document["project_count"] == 1 + assert document["states"][0]["state"] == "changed" + assert document["projects_file"] == str(register)