From 35a9fef922d91f9933528cbb5fc9b3040d0d8767 Mon Sep 17 00:00:00 2001 From: Arthur Date: Fri, 9 Oct 2026 09:36:20 +0200 Subject: [PATCH 1/6] docs(memory): design the person registry Who the family is and who it knows was guessed in four places. The design now names one model for it: - A person is one entity with two sources. The family declares each person in a file of their own, `family/people/.md`: the full name, the other names they go by, their chat account if they have one. The wiki's person page is the compiled projection of it. - Membership is derived, not declared. In the first iteration a person with an account is a member; a later iteration suggests members from the records and asks the family first. - Records gain `mentions`, the people outside the household they name, written by the same call that classifies them. - A person registry is compiled from those mentions: names, aliases, the kind of relation, how present a person is, and the records it cites. The family's corrections are edits to the same declared files. The wiki's people tiers are a projection of the members plus the registry. - A nightly plausibility pass (the dream cycle) is named as later work. - ADR-011 gets an update: the agent may answer from compiled pages whose statements cite their records, reading the records filed since the last compile. --- docs/adr/adr-011-vault-brain-projection.md | 25 +++++++ docs/design/brain/domain-model.md | 76 +++++++++++++++++++--- docs/design/brain/family-ontology.md | 11 +++- docs/design/brain/vault-format.md | 71 +++++++++++++++++++- 4 files changed, 170 insertions(+), 13 deletions(-) diff --git a/docs/adr/adr-011-vault-brain-projection.md b/docs/adr/adr-011-vault-brain-projection.md index d2552dac..36eec3d5 100644 --- a/docs/adr/adr-011-vault-brain-projection.md +++ b/docs/adr/adr-011-vault-brain-projection.md @@ -118,3 +118,28 @@ room) or edited by hand is theirs: the compile never overwrites it, and the vault commit that changed it names who did. The diary is therefore the one chat filing not rebuilt by replaying its thread (see `docs/agent/dev.md`, "Reprocessing replays the source"). + +## Update 2026-10-09: compiled projections are the agent's fast path + +The rule above keeps the agent off projections so that a stale page +never becomes a wrong answer. It assumed the agent could afford to +read the records for every question. With a local model it cannot: +reading dozens of records per question is slow, and the compiled pages +exist precisely to have read them once. + +The vault stays the truth. The agent may answer from a compiled +projection when two things hold: + +- every statement it uses cites the records it came from, as the home + page's numbered references and the person pages' `sources` do, so + the answer can name its source and a person can check it; +- it also reads the records filed since that projection was last + compiled, so nothing newer is missed. + +What this does not remove is a projection that is wrong rather than +stale. The citations make that checkable, family corrections fix it, +and a nightly plausibility pass (the "dream cycle" in +`docs/design/brain/domain-model.md`) is meant to find it. + +The agent already reads the brain (`MEMORY_VAULT_DIR` points at it), +so this update describes what it does and puts conditions on it. diff --git a/docs/design/brain/domain-model.md b/docs/design/brain/domain-model.md index 3e65e7d2..d30baa71 100644 --- a/docs/design/brain/domain-model.md +++ b/docs/design/brain/domain-model.md @@ -35,17 +35,40 @@ placement stays the deletion test: does deleting it lose information? ## Aggregates in the knowledge core -**Household** — the instance itself. Owns Members, the shared bucket -slug, language, and the scope rules. `shared_bucket` being +**Household** — the instance itself. Owns its People, the shared +bucket slug, language, and the scope rules. `shared_bucket` being configurable is the tell that the real concept is a circle of people sharing an archive; a family is the primary instantiation, a non-family deployment is the same aggregate with different vocabulary. -**Member** — identity is the triple equality -`matrix localpart == bucket slug == git author`. That equality is -load-bearing: it is what makes attribution (`--by homer`), scoping, -and the vault chronicle line up. A Member owns a personal bucket with -the same shape as the shared one. +**Person** — someone the family knows, declared in one file each: +`family/people/.md` (vault-format §7). Identity is the file's id +(`maggie`). The file holds the full name, the other names the family +uses for them, and the person's chat account if they have one. It is a +state document: the family writes and corrects it, by hand or through +the CLI, and `stack up memory` adds one for every account that has +none. The wiki's person page (`maggie/about.md`) is the compiled +projection of it: the declared names win, the compiled detail fills in +the rest. One entity, two sources, so the family's corrections to +what the stack learned ("Ed is Ned", "Grampa is Homer's father") are +edits to the same file, never a second format. + +**Member** — a person who belongs to the household. Not a separate +entity and not a declared field: membership is derived, and the stack +makes an opinionated guess. In the first iteration a person is a +member when they have an account. For a member with an account the +triple equality `matrix localpart == bucket slug == git author` holds, +and it is what makes attribution (`--by homer`), scoping, and the +vault chronicle line up; such a member owns a personal bucket with the +same shape as the shared one. A person without an account (a baby, a +grandparent) is still declared, so the classifier, the transcription +vocabulary and the diary know them by every name, and the wiki gives +them a page; they are not listed among the members. The next iteration +suggests membership from the records when they make it likely (Maggie +lives in the house, the records say so), and asks the family before +it counts. Deriving members from folders is what put `media/` on the +wiki as a family member, which is why the guess starts from accounts +and nothing weaker. **Record** — the generalization that ate "document". A capture, a document mirror, an email thread, a voice memo. Identity is @@ -89,7 +112,11 @@ from frontmatter; it wants to live in a registry (below). Scope (the privacy rule as a type: shared bucket or one member's bucket, derived from room membership). SourceContent (text + title + origin URI, the ingestion contract). Classification (correspondent, -doctype, persons, topics, summary, facts, action items). Provenance. +doctype, persons, mentions, topics, summary, facts, action items). +`persons` are the members a record belongs to; `mentions` are the +people outside the household it names, extracted in the same model +call that classifies it, so a record can be replayed to recompute +them like every other part of its classification. Provenance. Attribution. CaptureHash / PaperlessId / ThreadRoot. VaultPath (the deterministic path builders are its factories). SummaryCallout with Facts and ActionItems. PermaLink (`/go/docs|topic|person`, the @@ -110,6 +137,22 @@ connects Conversation to Knowledge (this room feeds that topic). identity authority. This is also the foundation for typed entities inside standing topics: a Policy, an Account, a Vehicle, each linked to a Correspondent and to source Records. + *Being built, people first.* A person entry is compiled from the + `mentions` of every record: canonical name, aliases, the kind of + relationship to the household (relative, friend, neighbor, teacher, + doctor, ...) and the stated relation, how present the person is + (records, months, first and last seen), and the records it cites. + It is a projection; the family's corrections ("Ed is Ned", "Grampa + is Homer's father") are the person's declared file (Person, above), + which wins over it and survives every rebuild. Correspondents become a second kind in the + same registry, and `pet`, `vehicle`, `place` after them, with the + same aliases, citations and corrections rather than a mechanism + each. The wiki's people tiers (family, broader family, close, + other) are a projection of the members plus the registry: tier + from the relationship kind, order within a tier from presence. + Evidence: `tools/family-memories/spec.people.en.yaml` and the + people-tiers prototype (2026-10-08). Relationship kind put 21 of 22 + people in the right tier; mention counts alone, 14. - **FactStore** — `facts.toml` holds facts (rule | fact | habit) with no aggregate managing them. Typed facts with as-of validity are what turn a standing topic from a folder into knowledge: policy number, @@ -117,6 +160,17 @@ connects Conversation to Knowledge (this room feeds that topic). - **Reminder** — time-bound intent attached to a Member or Topic. Standing-topic facts with dates (renewals, expiries) feed it almost for free. +- **Dream cycle** — a nightly pass after the compile that checks what + was compiled for plausibility: one person with two fathers, a + relation that contradicts another source, near-duplicate entries, + dates that cannot both be true. Cheap deterministic checks select + the suspects; the model rereads only their cited records. It never + rewrites the family's own documents: a contradiction the sources + cannot settle becomes a question to the family in chat, and the + answer becomes a correction. Majority voting is not enough: in the + prototype Grampa was "Marge's father" in 3 records and "Homer's + father" in 2, because Marge writes "Grampa" from the children's + side. Earlier sketch: `ontology-design.md`, "Maintenance". ## Rules of thumb @@ -127,4 +181,8 @@ connects Conversation to Knowledge (this room feeds that topic). 3. New containers are topics until proven otherwise; prefer a provenance flag over a new mechanism. 4. Facts cite Records. A fact without a source link is a rumor. -5. The agent is a client of the database, never of the projections. +5. The vault is the truth; compiled projections are the agent's fast + path. The agent may answer from a projection whose statements cite + their records, and reads the records filed since that projection + was last compiled. A slow local model cannot search every record + for every question. (ADR-011, update 2026-10-09.) diff --git a/docs/design/brain/family-ontology.md b/docs/design/brain/family-ontology.md index 699e4eba..ae9c950f 100644 --- a/docs/design/brain/family-ontology.md +++ b/docs/design/brain/family-ontology.md @@ -1,6 +1,15 @@ # Family ontology: declared entities priming classification -Status: design draft, 2026-06-12. Implementation slice of +Status: design draft, 2026-06-12. Partly superseded 2026-10-09: the +people pages below are built (`vault-format.md` §5, `person` +(declared)), with `type: person` and `title` instead of a folder-derived +kind and `canonical`; membership is derived (an account, in the first +iteration), not a `member:` field; people outside the household are +compiled from the `mentions` on records into the person registry +(`domain-model.md`, EntityRegistry); relations are a `relation_kind` +plus words, not a `role`. Onboarding, the write-path contract and +experiment round 6 +still stand. Implementation slice of `ontology-design.md` (2026-04-14), which designs the full living ontology (persons, organizations, relations, growth loop, dream-cycle maintenance, search expansion). Companion to `wiki-page-anatomy.md` and diff --git a/docs/design/brain/vault-format.md b/docs/design/brain/vault-format.md index 37bbc5e0..23053100 100644 --- a/docs/design/brain/vault-format.md +++ b/docs/design/brain/vault-format.md @@ -82,9 +82,18 @@ cheap). - `diary` — one entry from the memories room (a recording, photo or note, with its joined fragments and replies). +**State documents** (the family's current truth; mutable through the +CLI or by hand; never carry `generated`; see §7): +- `todos` — a topic's todo list. +- `person` (declared) — someone the family knows, one file each under + `/people/`. + **Entities & structure** (generated projections; MUST carry `generated: true`): -- `person` — a household member or known person page. +- `person` — the wiki page for a person: the declared file compiled + with what the records say, and later anyone the records mention (the + person registry). `person` is the one type that is both: declared in + `people/`, generated everywhere else, told apart by `generated`. - `correspondent` — an external party page. - `topic` — a topic overview page. - `index` — a folder navigation page. @@ -97,6 +106,14 @@ Legend: **R** required, **O** optional (present-when-nonempty — absence is a signal, so never emit an empty list/string). List fields are one-deep string lists per §2. +Two person fields recur on records and mean different things: +`persons` lists the household members the record belongs to, by +display name; `mentions` lists the people outside the household it +names, each by the most specific name the record gives, or a +description through a member when it gives none ("Marge's mother"). +Both come from the call that classifies the record. The person +registry is compiled from `mentions`; the record never links to it. + ### `document` | Field | | Notes | |---|---|---| @@ -111,6 +128,7 @@ one-deep string lists per §2. | `document_type` | O | Paperless subtype (invoice, contract) — a different axis from `type`. | | `category` | O | | | `persons` | O | list. | +| `mentions` | O | list. | | `tags` | O | list. | | `paperless_url` | O | base URL. | | `processing` | O | `ai_formatted` \| `ocr` \| `original`. | @@ -124,6 +142,7 @@ one-deep string lists per §2. | `title` | R | | | `timestamp` | R | | | `persons` | O | list. | +| `mentions` | O | list. | | `filed_by` | O | Matrix localpart of the filer (mirrors the git author). | | `tags` | O | list. | | `resource` | O | source URL (OKF `resource`; bookmarks always have one). | @@ -160,6 +179,7 @@ commit (`correct: `, authored by whoever wrote it), not on the card; its event id joins `event_ids`. | `description` | O | one sentence. | | `persons` | O | list. | +| `mentions` | O | list. | | `tags` | O | list. | | `filed_by` | O | Matrix localpart of the sender. | | `addressee` | O | who the message is spoken to, as it names them; a correction can change it. | @@ -177,16 +197,51 @@ and last the family's own words under `## Transcript`, `## Text` or `## Caption`. The words come last so nothing in them can be mistaken for a section. +### `person` (declared) +The family's file for one person, `<shared_bucket>/people/<id>.md`. The +file name is the id. + +| Field | | Notes | +|---|---|---| +| `type` | R | `person` | +| `title` | R | the full name (OKF `title`). | +| `aliases` | O | list; every other name the family uses (Obsidian reads the same key). | +| `account` | O | the person's chat user name (Matrix localpart), when they have one. | + +``` +--- +type: person +title: Maggie Simpson +aliases: + - Margaret +--- +``` + +A member of the household is a person with an `account` (first +iteration; `domain-model.md`, Member). Membership is derived, never a +field the family sets. A file that breaks the §2 subset or this table +is reported by path and left out, and the wiki retires no member page +while any file is reported. + ### `person` (generated) | Field | | Notes | |---|---|---| | `type` | R | `person` | | `generated` | R | `true` | | `title` | R | canonical display name (OKF `title`). | -| `slug` | R | bucket slug (== Matrix localpart for members). | +| `slug` | R | the person's id: the declared file's name (== Matrix localpart and bucket slug for a member); otherwise derived from the canonical name. | | `canonical` | R | canonical name (our H1/canonicalization logic). | | `aliases` | O | list of known surface forms (the EntityRegistry coupling field). | -| `role`, `member`, `birthday`, `employer`, `owners`, `relation` | O | family-semantic custom fields. | +| `relation_kind` | O | `relative` \| `friend` \| `neighbor` \| `teacher` \| `doctor` \| `employer` \| `colleague` \| `service` \| `other`; absent for members. | +| `relation` | O | the relation in words, through a member ("Marge's sister"). | +| `first_seen`, `last_seen` | O | dates of the first and last record that mentions the person. | +| `sources` | O | list of the vault paths of the records it cites. | +| `birthday`, `employer`, `owners` | O | family-semantic custom fields. | + +Whether someone is a member is never a field: it is derived from the +declared file's `account`. The tier the wiki shows (broader family, close, other) is +not stored either; the wiki derives it from `relation_kind` and +presence. ### `correspondent` (generated) | Field | | Notes | @@ -225,6 +280,16 @@ carries `generated`. It is a markdown checklist; frontmatter is optional and, if present, uses `type: todos`. State documents get read-your-writes through the CLI (ADR-011); they are not regenerated. +`<shared_bucket>/people/<id>.md` is a declared person (§5): the +family's own facts about someone, starting with their names. On every +start `stack up memory` adds one for each account that has none, as +a commit by the memory bot, and never rewrites an existing file. +A family member adds a person without an account (a baby) or an alias +by writing the file in Obsidian, Forgejo or with `stack memory write`; +a hand edit is a commit like every other. The family's corrections to +what the registry compiles ("Ed is Ned", "Grampa is Homer's father") +go into the same file when the registry is built. + `ontology.toml` / `facts.toml` are TOML config at the vault root, outside this markdown-frontmatter spec. From 31ea06532b784f102da655fd42841090b8a4d7cd Mon Sep 17 00:00:00 2001 From: Arthur <hello@arthware.dev> Date: Fri, 9 Oct 2026 12:59:30 +0200 Subject: [PATCH 2/6] feat(memory): give every person the family knows a file of their own The wiki guessed who the family is from vault folders and the names in documents, so a `media/` folder became a member page carrying the baby's data, and a person without a chat account had no page at all. Each person now has a file, `family/people/<id>.md`, in the vault's frontmatter format: `type: person`, the full name as `title`, the other names the family uses as `aliases`, and `account` for a person with a chat account. A person with an account is a member of the household. Every `stack up memory` adds a file for each account that has none and never rewrites an existing one, so an existing install switches over with a restart of the memory stacklet. Until a file exists, the wiki keeps its old roster. With the files, the wiki gives every declared person a page under the names their file gives, also before anything names them, and its home page lists the members. Names in documents resolve through the aliases, generated person pages nobody declares are removed, and a change to any file rebuilds every person page (`--members`). A file that breaks the schema is never guessed at: the readers name it and leave it out, no page is removed while one is broken, and `stack memory write` refuses it with the reason. The schema sits with the other types in the frontmatter validator, which now knows a declared `person` and fields that must hold a single value. An update reaches it with `./stack restart memory`; add anyone without an account (a baby) as a file of their own in `family/people/`. --- docs/admin-guide.md | 16 +- lib/stack/frontmatter.py | 37 ++- lib/stack/people.py | 238 ++++++++++++++ stacklets/memory/bot/cli/wiki.py | 259 ++++++++++++++-- stacklets/memory/bot/curator.py | 12 +- stacklets/memory/cli/wiki.py | 5 +- stacklets/memory/cli/write.py | 20 ++ stacklets/memory/hooks/on_start_ready.py | 28 ++ stacklets/memory/lib.py | 43 +++ stacklets/memory/quartz/quartz.config.ts | 4 +- tests/unit/framework/test_people.py | 175 +++++++++++ tests/unit/stacklets/test_frontmatter.py | 29 +- tests/unit/stacklets/test_memory_people.py | 291 ++++++++++++++++++ .../stacklets/test_memory_token_repair.py | 6 +- tests/unit/stacklets/test_memory_write_cli.py | 23 ++ 15 files changed, 1133 insertions(+), 53 deletions(-) create mode 100644 lib/stack/people.py create mode 100644 tests/unit/framework/test_people.py create mode 100644 tests/unit/stacklets/test_memory_people.py diff --git a/docs/admin-guide.md b/docs/admin-guide.md index 22e0394e..99ba4627 100644 --- a/docs/admin-guide.md +++ b/docs/admin-guide.md @@ -567,6 +567,20 @@ Setup seeds the vault with three things: the classification ontology (the topics The wiki maintains itself. A curator sidecar (`stack-memory-curator`) watches the vault: when new filings settle it regenerates the pages of the family members involved plus the home page (a couple of LLM calls, a few minutes after the burst), and once a night it rebuilds everything — topic pages, cross-references, the lot — while the GPU has nothing better to do. To bring the wiki up to date now instead of waiting, `./stack memory wiki update` regenerates what the latest filings touched and `./stack memory wiki update --all` runs the whole nightly rebuild; both return when the pages are written. `--member`, `--topic` and `--home` regenerate just those pages, and `[memory]` in `stack.toml` holds the knobs (see [Configuration](#configuration)). Edits happen in Forgejo (every wiki page links to its source), so the commit log doubles as the household's learning history. +**Who the family is.** Every person the family knows has a file of their own in the vault, `family/people/<id>.md`: the full name, the other names the family uses, and the chat account if they have one. + +```markdown +--- +type: person +title: Bart Simpson +aliases: + - Bartholomew +account: bart +--- +``` + +Each `./stack up memory` adds a file for every account in `users.toml` that has none, so an update reaches an existing install with a restart of the memory stacklet. An existing file is never rewritten: it is the family's. Add someone without an account, such as a baby, by adding a file with `type`, `title` and any `aliases`, and add the names a person goes by to their file. A person with an account is a member of the household. The wiki gives every person a page under the names their file gives, and its home page lists the members. A file the wiki cannot read is named in the curator's log (`docker logs stack-memory-curator`) and left out, and no page is removed until it reads again. A change to any file rebuilds the person pages within a few minutes, or right away with `./stack memory wiki update --home --members`. Edit the files in Forgejo, in Obsidian, or with `./stack memory write family/people/<id>.md --by <you> --from <file>`. + **The family diary.** Everything posted in `#Memories` becomes a diary entry. Each night the curator transcribes the new recordings, has the AI read each entry (a title, a summary, the people, topics from the ontology, facts), writes one card per entry to the vault under `family/diary/entries/`, and rebuilds the diary pages in the wiki from those cards. At 07:30 local time the archivist posts each new card as a quiet notice in the thread under its memory; the first time, one notice in the room says how to correct a card, and after two weeks without a new memory one notice invites the next, once per quiet stretch (`diary_job_at` in the archivist's `bot.toml` sets the time; it catches up after a night the Mac slept through). A family member corrects a card by replying in its thread, typed or spoken; the correction is committed to the vault under their name, and a corrected or hand-edited card is never overwritten by a later run. The archivist says nothing else in `#Memories`. The user guide has the family's side: [The Memories Room](user-guide.md#the-memories-room). | | | @@ -630,7 +644,7 @@ Key things to know: ### users.toml -Your family members. Generated by the installer. User accounts are seeded on the first `./stack up` of each stacklet. +The accounts: who can sign in, to what. Generated by the installer. User accounts are seeded on the first `./stack up` of each stacklet. The people the family knows are a separate list, the memory stacklet's `family/people/` (see [Memory](#memory-memory-optional)): a baby has no account but is family. ```toml [[users]] diff --git a/lib/stack/frontmatter.py b/lib/stack/frontmatter.py index 7f00cf61..7f466fbc 100644 --- a/lib/stack/frontmatter.py +++ b/lib/stack/frontmatter.py @@ -13,7 +13,7 @@ from __future__ import annotations -from dataclasses import dataclass +from dataclasses import dataclass, field from typing import Optional @@ -40,9 +40,10 @@ class FrontmatterError(ValueError): @dataclass class TypeSchema: - """Schema for a single type: required fields and optional list fields.""" + """Schema for a single type: required fields, list fields, single-value fields.""" required: set[str] list_fields: set[str] + scalar_fields: set[str] = field(default_factory=set) # Build the type-to-schema mapping from §5 of the spec. @@ -86,6 +87,18 @@ class TypeSchema: } +# Types the family may also declare by hand, as a state document without +# `generated`: the family's own file for an entity, which the generated +# page of the same type is compiled from (§5, `person` (declared)). +DECLARED_SCHEMAS: dict[str, TypeSchema] = { + "person": TypeSchema( + required={"type", "title"}, + list_fields={"aliases"}, + scalar_fields={"title", "account"}, + ), +} + + # ── Parser ──────────────────────────────────────────────────────────── def parse(text: str) -> dict: @@ -323,7 +336,9 @@ def validate(fm: dict) -> list[str]: - `type` is present and in the closed vocabulary - Required fields for the type are present - List fields are actually lists (or can be coerced) - - `generated` presence matches record vs. projection class + - Single-value fields are not lists + - `generated` presence matches record vs. projection class; a + declarable type without it is checked as the family's own file Args: fm: Parsed frontmatter dict @@ -349,7 +364,11 @@ def validate(fm: dict) -> list[str]: ) return errors - schema = SCHEMAS[entry_type] + # A declarable type without the marker is the family's own file. + is_projection = entry_type in PROJECTION_TYPES + has_generated = fm.get("generated") is True + declared = entry_type in DECLARED_SCHEMAS and not has_generated + schema = DECLARED_SCHEMAS[entry_type] if declared else SCHEMAS[entry_type] # Check required fields for req_field in schema.required: @@ -365,11 +384,13 @@ def validate(fm: dict) -> list[str]: f"field `{list_field}` must be a list, got {type(val).__name__}" ) - # Check `generated` invariant (§6) - is_projection = entry_type in PROJECTION_TYPES - has_generated = fm.get("generated") is True + # Check single-value fields are not lists + for scalar_field in schema.scalar_fields: + if isinstance(fm.get(scalar_field), list): + errors.append(f"field `{scalar_field}` must be a single value, not a list") - if is_projection and not has_generated: + # Check `generated` invariant (§6) + if is_projection and not has_generated and not declared: errors.append( f"projection type `{entry_type}` must have `generated: true`" ) diff --git a/lib/stack/people.py b/lib/stack/people.py new file mode 100644 index 00000000..7cb67dae --- /dev/null +++ b/lib/stack/people.py @@ -0,0 +1,238 @@ +"""The family's people: one declared file each. + +Every person the family knows has a file in the shared bucket, +`people/<id>.md`, in the vault's frontmatter format: + + --- + type: person + title: Maggie Simpson + aliases: + - Margaret + --- + +The file name is the id. `title` is the full name, `aliases` the other +names the family uses, and `account` the chat user name of a person who +has one. The schema lives with the others in `stack.frontmatter` +(`DECLARED_SCHEMAS`), and a file that breaks it is reported by path +and left out; nothing here guesses at what a broken file meant. + +A member of the household is a person with an account. That is the +first, opinionated guess at membership (docs/design/brain/domain-model.md, +Member); it is derived, never a field the family sets. A person +without an account (a baby) is still known by every name: the +classifier, the transcription vocabulary and the diary read all +people, the wiki's member list reads the members. + +The readers return `None` while no person file exists, so a caller can +keep its previous behaviour until `stack up memory` has written them. + +Pure functions and file reads. No Forgejo, no git: the memory stacklet +writes the files (`ensure_people` in its library). Lives in the +framework because the readers span stacklets: the wiki, the diary, the +archivist, transcription. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, field +from pathlib import Path +from typing import Callable, Iterable + +from .frontmatter import FrontmatterError, dump, parse, validate +from .vault import DEFAULT_SHARED_BUCKET, slug + +PEOPLE_DIR = "people" + +# An id as the vault uses it for buckets and Matrix for localparts. A +# file name that is not one is slugged ("Grampa Simpson.md" is +# `grampa-simpson`), so a file made in Obsidian still names a person. +_ID = re.compile(r"^[a-z0-9][a-z0-9._-]*$") + +Report = Callable[[str], None] + + +@dataclass(frozen=True) +class Person: + """One person the family declared. + + >>> p = Person("maggie", "Maggie Simpson", ("Margaret",)) + >>> p.all_names() + ['Maggie Simpson', 'Maggie', 'Margaret'] + >>> p.member + False + """ + + id: str + name: str + aliases: tuple[str, ...] = field(default_factory=tuple) + account: str = "" + + @property + def member(self) -> bool: + """Whether the person belongs to the household: they have an account.""" + return bool(self.account) + + @property + def first_name(self) -> str: + return self.name.split()[0] if self.name.split() else self.id + + def all_names(self) -> list[str]: + """Full name, first name, then the aliases; each once, in that order.""" + out: list[str] = [] + for name in (self.name, self.first_name, *self.aliases): + if name and name.lower() not in {n.lower() for n in out}: + out.append(name) + return out + + +class PersonError(ValueError): + """A person file that does not follow the format; the message says how.""" + + +# ── Reading ────────────────────────────────────────────────────────────── + +def people_dir(root: Path, shared_bucket: str = DEFAULT_SHARED_BUCKET) -> Path: + """Where the person files live under a vault or brain working copy.""" + return Path(root) / shared_bucket / PEOPLE_DIR + + +def person_path(person_id: str, shared_bucket: str = DEFAULT_SHARED_BUCKET) -> str: + """The repo-relative path of one person's file.""" + return f"{shared_bucket}/{PEOPLE_DIR}/{person_id}.md" + + +def person_id(file_stem: str) -> str: + """The id a person file's name gives.""" + return file_stem if _ID.match(file_stem) else slug(file_stem) + + +def read_person(file_stem: str, text: str) -> Person: + """The person one file declares, or `PersonError` saying what is wrong.""" + try: + fm = parse(text) + except FrontmatterError as e: + raise PersonError(f"frontmatter does not parse: {e}") from e + if fm.get("type") != "person": + raise PersonError("not a person file: it needs `type: person`") + if fm.get("generated") is True: + raise PersonError("a generated page does not belong in people/") + if errors := validate(fm): + raise PersonError("; ".join(errors)) + return Person( + id=person_id(file_stem), + name=str(fm["title"]).strip(), + aliases=tuple(str(a).strip() for a in fm.get("aliases") or [] if str(a).strip()), + account=str(fm.get("account") or "").strip(), + ) + + +def load_people( + root: Path | None, + shared_bucket: str = DEFAULT_SHARED_BUCKET, + *, + report: Report | None = None, +) -> list[Person] | None: + """Every person declared under `root`, members first, or `None` if none is. + + A file that cannot be read as a person is passed to `report` with + its path and the reason, and left out. `None` covers no folder, an + empty folder and a folder of broken files alike: the caller keeps + its previous behaviour rather than act on nobody. + """ + if root is None: + return None + folder = people_dir(root, shared_bucket) + people: dict[str, Person] = {} + for path in sorted(folder.glob("*.md")): + rel = f"{shared_bucket}/{PEOPLE_DIR}/{path.name}" + try: + person = read_person(path.stem, path.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError) as e: + _report(report, f"{rel}: cannot be read ({e})") + continue + except PersonError as e: + _report(report, f"{rel}: {e}") + continue + if person.id in people: + _report(report, f"{rel}: {person.id!r} is declared twice; the first file counts") + continue + people[person.id] = person + if not people: + return None + return sorted(people.values(), key=lambda p: (not p.member, p.name.lower())) + + +def find_people( + roots: Iterable, + shared_bucket: str = DEFAULT_SHARED_BUCKET, + *, + report: Report | None = None, +) -> list[Person] | None: + """The people from the first of `roots` that declares any. + + A process often sees two copies of the vault: the working copy the + family's edits land in, and the brain the curator mirrors it to. + Callers list the fresher one first. Empty and None roots are skipped. + """ + for root in roots: + if root and (people := load_people(Path(root), shared_bucket, report=report)) is not None: + return people + return None + + +def members(people: list[Person]) -> list[Person]: + """The people who belong to the household.""" + return [p for p in people if p.member] + + +def resolve(name: str, people: list[Person]) -> Person | None: + """The person a name means, by any name they go by, or None. + + >>> resolve("Margaret", [Person("maggie", "Maggie Simpson", ("Margaret",))]).id + 'maggie' + """ + wanted = name.strip().lower() + for person in people: + if wanted == person.id or wanted in (n.lower() for n in person.all_names()): + return person + return None + + +def _report(report: Report | None, message: str) -> None: + if report is not None: + report(message) + + +# ── Writing ────────────────────────────────────────────────────────────── + +def people_from_accounts(users: list[dict]) -> list[Person]: + """A person for everyone with an account, as `users.toml` lists them. + + The id follows `users.user_id` (an explicit `id`, else the first name + lowercased), so it is the person's Matrix localpart and their bucket. + Entries without a name are skipped. Bots never appear in `users.toml`. + """ + people: list[Person] = [] + seen: set[str] = set() + for user in users: + name = str(user.get("name") or "").strip() + if not name: + continue + account = str(user.get("id") or name.split()[0].lower()) + if account in seen: + continue + seen.add(account) + people.append(Person(account, name, account=account)) + return people + + +def render_person(person: Person) -> str: + """A person's file as famstack first writes it.""" + fm = dump({ + "type": "person", + "title": person.name, + "aliases": list(person.aliases), + "account": person.account, + }) + return f"---\n{fm}\n---\n\n# {person.name}\n" diff --git a/stacklets/memory/bot/cli/wiki.py b/stacklets/memory/bot/cli/wiki.py index ae4765c3..b6b1e178 100644 --- a/stacklets/memory/bot/cli/wiki.py +++ b/stacklets/memory/bot/cli/wiki.py @@ -6,6 +6,7 @@ (no flags) home + members + topics: the curator's nightly pass --home just the household home page --member homer just Homer's page + --members every person's page --topic camping just one topic's page --topics every topic page, no home/members --dry-run preview to stdout, no writes @@ -72,6 +73,7 @@ extract_summary_callout, ) from stack.vault import correspondents_dir, slug, slugify_person # noqa: E402 +from stack.people import Person, find_people, members, resolve # noqa: E402 from stack.ai.client import LLM, LLMUnavailableError # noqa: E402 @@ -114,11 +116,11 @@ # Within-bucket reserved subdirectories. A child folder of a known # bucket with one of these names is part of the bucket's own shape -# (capture-type folder, correspondent index, rescue folder), not a -# topic. The topic-discovery walker skips these names. +# (capture-type folder, correspondent index, person files, rescue +# folder), not a topic. The topic-discovery walker skips these names. _RESERVED_BUCKET_SUBDIRS = { "notes", "bookmarks", "documents", - "correspondents", "_unfiled", + "correspondents", "people", "_unfiled", } # Subdir names a topic folder must contain at least one of to be @@ -243,6 +245,7 @@ async def run(llm: LLM, argv: list[str]) -> int: home_sel = "--home" in argv topics_only = "--topics" in argv members_sel = _arg_values(argv, "--member") + members_all = "--members" in argv topics_sel = _arg_values(argv, "--topic") correspondents_sel = _arg_values(argv, "--correspondent") @@ -272,10 +275,20 @@ async def run(llm: LLM, argv: list[str]) -> int: _err("no documents with summary callouts found in vault") return 1 - # The roster the home page links into and the default --members loop - # walks. Computed once: bucket dirs on disk plus everyone named in - # document frontmatter. - roster = _member_slugs(vault, index, shared_bucket) + # The people the wiki carries a page for, and the members among them + # the home page lists: the person files once the family has them, the + # old guess (bucket dirs plus names in frontmatter) until + # `stack up memory` has written them. A person file that does not + # read is reported, and while any is, no page is retired: a typo in + # a file must not delete a page. + problems: list[str] = [] + people = _people(shared_bucket, problems) + if people is not None: + _resolve_persons(index, people) + roster = _roster(vault, index, shared_bucket, people) + member_ids = [p.id for p in members(people or [])] or roster + persons = {p.id: p for p in people or []} + retire = people is not None and not problems # Topic locations are discovered against the roster: a topic under # a bucket whose owner the wiki doesn't generate a page for would @@ -295,20 +308,21 @@ async def run(llm: LLM, argv: list[str]) -> int: # and slugs both land on the bucket. Unknown selections warn and skip — # an auto-run fed a name with no match must not sink the batch — and the # run fails only when NOTHING in the selection matched. - if home_sel or topics_only or members_sel or topics_sel or correspondents_sel: + if (home_sel or topics_only or members_sel or members_all + or topics_sel or correspondents_sel): generated = 0 if home_sel: rc = await _generate_home( - llm, index, roster=roster, + llm, index, roster=member_ids, people=persons, shared_bucket=shared_bucket, lang=lang, write=not dry_run, ) if rc == 0: generated += 1 - member_slugs: list[str] = [] + member_slugs: list[str] = list(roster) if members_all else [] for raw in members_sel: - slug = slugify_person(raw) + slug = _member_id(raw, people) if slug and slug not in member_slugs: member_slugs.append(slug) for slug in member_slugs: @@ -316,11 +330,13 @@ async def run(llm: LLM, argv: list[str]) -> int: _err(f"no member bucket for '{slug}' — skipping") continue rc = await _generate_member( - llm, slug, index, vault, + llm, slug, index, vault, person=persons.get(slug), shared_bucket=shared_bucket, lang=lang, write=not dry_run, ) if rc == 0: generated += 1 + if members_all and retire: + _retire_undeclared(roster, shared_bucket, write=not dry_run) # Correspondents match by canonical name or slug against the roster # discovered from document frontmatter. @@ -366,11 +382,13 @@ async def run(llm: LLM, argv: list[str]) -> int: # ── Default loop ────────────────────────────────────────────────── # Bare invocation: home, every member, every correspondent, every topic. rc = await _generate_home( - llm, index, roster=roster, + llm, index, roster=member_ids, people=persons, shared_bucket=shared_bucket, lang=lang, write=not dry_run, ) if rc != 0: return rc + if retire: + _retire_undeclared(roster, shared_bucket, write=not dry_run) # A member with no content is skipped (not an error) so one # empty bucket doesn't sink the whole run. `_generate_member` @@ -378,7 +396,7 @@ async def run(llm: LLM, argv: list[str]) -> int: # here -- the overall run still succeeds. for member_slug in roster: await _generate_member( - llm, member_slug, index, vault, + llm, member_slug, index, vault, person=persons.get(member_slug), shared_bucket=shared_bucket, lang=lang, write=not dry_run, ) @@ -400,16 +418,19 @@ async def run(llm: LLM, argv: list[str]) -> int: async def _generate_home( llm: LLM, index: list[dict], *, - roster: list[str], shared_bucket: str, lang: str, write: bool, + roster: list[str], people: dict[str, Person], + shared_bucket: str, lang: str, write: bool, ) -> int: """Compose the household home page from every filed summary. The roster (every member that has a page) is woven into the Members - section so each name links to that person's page. Quartz turns those + section so each name links to that person's page; `people` gives + the declared full names. Quartz turns those links into backlinks and graph edges for free, so the home page and the member pages become navigable in both directions. """ - prompt = _build_home_prompt(index, roster=roster, lang=lang) + names = {slug: p.name for slug, p in people.items()} + prompt = _build_home_prompt(index, roster=roster, names=names, lang=lang) try: page = (await llm.complete("overview", prompt, temperature=_TEMPERATURE)).strip() except LLMUnavailableError as e: @@ -454,15 +475,26 @@ async def _generate_home( async def _generate_member( llm: LLM, slug: str, index: list[dict], vault: Path, *, + person: Person | None = None, shared_bucket: str, lang: str, write: bool, ) -> int: - """Compose one member's page from their slice of the vault.""" + """Compose one person's page from their slice of the vault. + + `person` is their declared file; without person files the name comes + from the slug, as before. + """ entries = _member_entries(index, slug) + if not entries and person is not None: + # A declared person always has a page: the home page and the + # People menu link to it. Nothing filed yet means nothing for the + # model to write, so the page says so without asking it. + return _publish_person( + _empty_person_body(person), slug, person, [], write=write) if not entries: _err(f"no content involving '{slug}' — skipping") return 1 - display = slug.capitalize() + display = person.name if person else slug.capitalize() facts = _load_facts(vault, slug) # Anchored regen: renumber the existing page's citations to the @@ -512,17 +544,17 @@ async def _generate_member( shared_bucket=shared_bucket, write=write, ) return 0 - rc = _publish( - page, target_path=f"{slug}/about.md", - # First-creation frontmatter seeds the person entity registry on - # the page itself: `canonical` is the longest synonym (usually - # the formal first name when the family also uses a nickname), - # `synonyms` are the other variants seen in document - # frontmatter. The splice keeps everything outside the markers - # on re-runs, so a hand edit or a future deriver pass takes - # ownership of the registry from here. - default_preamble=_member_preamble(slug, display, _member_synonyms(index, slug)), - ) + learned = _member_synonyms(index, slug) + if person is None: + # No person files: first-creation frontmatter seeds the names + # on the page itself (`canonical` the longest synonym seen in + # documents) and the splice keeps them from then on. + rc = _publish( + page, target_path=f"{slug}/about.md", + default_preamble=_member_preamble(slug, display, learned), + ) + else: + rc = _publish_person(page, slug, person, learned, write=True) _publish_capture_indexes( index, page_dir=slug, display=display, shared_bucket=shared_bucket, write=write, @@ -690,6 +722,163 @@ def _norm_str_list(value) -> list[str]: return [v.strip() for v in value if isinstance(v, str) and v.strip()] +def _publish_person(page: str, slug: str, person: Person, learned: list[str], *, + write: bool) -> int: + """Publish a declared person's page under the names their file gives. + + The file owns the names: the full name is canonical, the family's + aliases come first, the variants documents used follow. Refreshed on + every rebuild, so an alias added to the file reaches the page the + next time. + """ + names = _distinct([*person.aliases, person.first_name, *learned], exclude=person.name) + if not write: + print(f"\n<!-- {slug}/about.md -->\n{page}") + return 0 + return _publish( + page, target_path=f"{slug}/about.md", + transform=lambda prior: _with_member_names( + _splice_generated(prior, page, default_preamble=_member_preamble( + slug, person.name, [person.name, *names])), + canonical=person.name, synonyms=names, + ), + ) + + +def _empty_person_body(person: Person) -> str: + """The page of a person nothing has been filed about yet.""" + return (f"# {person.name}\n\n" + f"Nothing filed about {person.first_name} yet. Documents, notes and " + f"memories that name {person.first_name} will show up here.") + + +def _people(shared_bucket: str, problems: list[str]) -> list[Person] | None: + """The declared people, or None while there are no person files. + + Read from the vault working copy (`MEMORY_VAULT_DIR`) when this + process has one: an edit to a file is on disk there before the + curator has mirrored it into the brain. The brain is the fallback. + Every file that does not read is logged and added to `problems`. + """ + def report(problem: str) -> None: + problems.append(problem) + _err(f"person file skipped: {problem}") + + return find_people( + (os.environ.get("MEMORY_VAULT_DIR"), os.environ.get("BRAIN_REPO_DIR")), + shared_bucket, report=report, + ) + + +def _roster(vault: Path, index: list[dict], shared_bucket: str, + people: list[Person] | None) -> list[str]: + """The ids the wiki carries person pages for, in the order it shows them.""" + if people is not None: + return [p.id for p in people] + return _member_slugs(vault, index, shared_bucket) + + +def _member_id(raw: str, people: list[Person] | None) -> str: + """The person a `--member` value or a `persons:` name means. + + Through every name the person files give ("Margaret" is Maggie), + by first word otherwise, the way buckets are named. + """ + if people is not None and (person := resolve(raw, people)): + return person.id + return slugify_person(raw) + + +def _resolve_persons(index: list[dict], people: list[Person]) -> None: + """Point every `persons:` name in the index at its declared person. + + The index slugs names by first word, which files "Margaret" under a + bucket of her own. With person files the names resolve through the + family's aliases instead. Names no person answers to keep their + first-word slug, as before. + """ + for entry in index: + entry["persons"] = [ + _member_id(name, people) for name in entry.get("person_names", []) + ] + + +def _retire_undeclared(roster: list[str], shared_bucket: str, *, + write: bool) -> list[str]: + """Delete generated person pages for anyone no person file declares. + + Before the person files, a vault folder or a surname could become a + member page (`media/about.md`). Only pages the wiki generated as a + person are removed; anything a person wrote is left alone, and so is + the folder's other content. + """ + try: + brain = _brain_dir() + except RuntimeError: + return [] + skip = _NON_MEMBER_DIRS | {shared_bucket} + retired: list[str] = [] + for about in sorted(brain.glob("*/about.md")): + slug = about.parent.name + if slug in roster or slug in skip or slug.startswith("."): + continue + try: + text = about.read_text(encoding="utf-8") + except OSError: + continue + if _parse_frontmatter(text).get("type") != "person" or not _is_generated_page(text): + continue + rel = f"{slug}/about.md" + if write: + about.unlink() + _err(f"retired {rel}: no person file declares {slug}") + else: + _err(f"would retire {rel}: no person file declares {slug}") + retired.append(rel) + return retired + + +def _distinct(names: list[str], *, exclude: str = "") -> list[str]: + """Names in order, each once (case-insensitive), without `exclude`.""" + seen = {exclude.lower()} if exclude else set() + out: list[str] = [] + for name in names: + if name and name.lower() not in seen: + seen.add(name.lower()) + out.append(name) + return out + + +def _with_member_names(content: str, *, canonical: str, synonyms: list[str]) -> str: + """Set a member page's `title`, `canonical` and `synonyms` frontmatter. + + Every other frontmatter line is kept as it is. A page without + frontmatter is returned unchanged; the splice always gives a member + page one. + """ + if not content.startswith("---\n"): + return content + end = content.find("\n---\n", 4) + if end < 0: + return content + kept: list[str] = [] + skipping_list = False + for line in content[4:end].splitlines(): + if skipping_list and line.startswith((" ", "\t")): + continue + skipping_list = False + key = line.partition(":")[0].strip() + if not line.startswith((" ", "\t")) and key in ("title", "canonical", "synonyms"): + skipping_list = key == "synonyms" + continue + kept.append(line) + names = [f"title: {_yaml_str(canonical)}", f"canonical: {_yaml_str(canonical)}"] + if synonyms: + names.append("synonyms:") + names.extend(f" - {_yaml_str(s)}" for s in synonyms) + return "---\n" + "\n".join(names + kept) + content[end:] + + def _member_slugs(vault: Path, index: list[dict], shared_bucket: str) -> list[str]: """Every family member the wiki should carry a page for. @@ -1429,7 +1618,8 @@ def _arg_values(argv: list[str], flag: str) -> list[str]: # ── Prompts ────────────────────────────────────────────────────────────── -def _build_home_prompt(entries: list[dict], *, roster: list[str], lang: str) -> str: +def _build_home_prompt(entries: list[dict], *, roster: list[str], lang: str, + names: dict[str, str] | None = None) -> str: """Single-shot prompt: feed all summaries, ask for one home page. The section layout is fixed -- we want the same headings every time @@ -1439,8 +1629,10 @@ def _build_home_prompt(entries: list[dict], *, roster: list[str], lang: str) -> closed set of page paths so the Members section can link each name to a member page that actually exists -- the model resolves identity (the baby "Margaret" maps to the `maggie/` page), but it can only - pick from the paths we give it. + pick from the paths we give it. With person files, `names` carries + each member's full name too, and the model fills in only the detail. """ + names = names or {} evidence = _format_evidence(entries) if roster: # Closed enumeration: the model copies this list instead of @@ -1448,7 +1640,8 @@ def _build_home_prompt(entries: list[dict], *, roster: list[str], lang: str) -> # small models split one person into two ("Maggie" and # "Margaret") or drop the page links. skeleton = "\n".join( - f"- **[<full name of {slug.capitalize()}>]({slug}/about)** — <detail>. [N]" + f"- **[{names[slug]}]({slug}/about)** — <detail>. [N]" if slug in names + else f"- **[<full name of {slug.capitalize()}>]({slug}/about)** — <detail>. [N]" for slug in roster ) member_links = ( diff --git a/stacklets/memory/bot/curator.py b/stacklets/memory/bot/curator.py index c76e5ba8..2aa97224 100644 --- a/stacklets/memory/bot/curator.py +++ b/stacklets/memory/bot/curator.py @@ -80,6 +80,7 @@ ) from wiki import COMMIT_PREFIX # noqa: E402 +from stack.people import PEOPLE_DIR # noqa: E402 POLL_SECS = int(os.environ.get("CURATOR_POLL_SECS", "30")) QUIET_SECS = int(os.environ.get("WIKI_REBUILD_QUIET_SECS", "180")) @@ -211,11 +212,15 @@ def member_selection( folders touched by captures. Returns e.g. `["--home", "--member", "Homer Simpson", "--topic", "camping"]`, or `[]` when nothing relevant changed (only generated pages or skipped dirs). Home is - included whenever anything relevant changed at all. + included whenever anything relevant changed at all. A change to a + person file adds `--members`: who the family knows changed, so every + person page is rebuilt and pages for people no file declares are + retired. """ members: list[str] = [] topics: list[str] = [] relevant = False + people_changed = False for path in paths: parts = [p for p in path.split("/") if p] @@ -225,6 +230,9 @@ def member_selection( if _has_generated_marker(fm): continue relevant = True + if path.startswith(f"{shared_bucket}/{PEOPLE_DIR}/"): + people_changed = True + continue if parts[0] != shared_bucket and len(parts) > 1 and parts[0] not in members: members.append(parts[0]) topic = _topic_slug_from_capture_path(parts, shared_bucket=shared_bucket) @@ -241,6 +249,8 @@ def member_selection( if not relevant: return [] argv = ["--home"] + if people_changed: + argv.append("--members") for member in members: argv += ["--member", member] for topic in topics: diff --git a/stacklets/memory/cli/wiki.py b/stacklets/memory/cli/wiki.py index 972d1dbb..aace73c6 100644 --- a/stacklets/memory/cli/wiki.py +++ b/stacklets/memory/cli/wiki.py @@ -5,6 +5,7 @@ stack memory wiki update --all the whole wiki, as at night: diary, sources, every page stack memory wiki update --home just the household home page stack memory wiki update --member homer just one member's page + stack memory wiki update --members every person's page, as the person files declare them stack memory wiki update --topic camping just one topic's page stack memory wiki update --topics every topic page, no home or members stack memory wiki update --dry-run preview home, member and topic pages, write nothing @@ -37,12 +38,12 @@ VERBS = { "update": "bring the latest filings into the wiki; --all rebuilds all of it, " - "--home/--member/--topic/--topics regenerate those pages", + "--home/--member/--members/--topic/--topics regenerate those pages", "clean": "delete every generated page (asks first)", } # Flags that name pages to regenerate directly, rather than asking the curator. -PAGE_FLAGS = ("--home", "--member", "--topic", "--topics", "--dry-run", "--dry") +PAGE_FLAGS = ("--home", "--member", "--members", "--topic", "--topics", "--dry-run", "--dry") def run(args, stacklet, config): diff --git a/stacklets/memory/cli/write.py b/stacklets/memory/cli/write.py index f8da7bf8..fcbde641 100644 --- a/stacklets/memory/cli/write.py +++ b/stacklets/memory/cli/write.py @@ -62,6 +62,7 @@ from stack.frontmatter import FrontmatterError, parse as parse_frontmatter # noqa: E402 from stack.links import go_page, public # noqa: E402 +from stack.people import PEOPLE_DIR, PersonError, read_person # noqa: E402 from stack.list_doc import diff # noqa: E402 from stack.page_patch import apply_edits # noqa: E402 @@ -94,6 +95,24 @@ def _check_frontmatter(before: str, after: str, page: str) -> None: "frontmatter block exactly as you read it.") from e +def _check_person(after: str, page: str) -> None: + """Reject a person file the readers would skip. + + Every reader leaves out a file that breaks the person schema, so + the write is where the family hears about it, not a log. + """ + parts = page.split("/") + if len(parts) != 3 or parts[1] != PEOPLE_DIR: + return + try: + read_person(Path(page).stem, after) + except PersonError as e: + raise ValueError( + f"{page} is a person file and {e}. It needs `type: person`, " + "a `title` with the full name and, optionally, `aliases` as a " + "list.") from e + + def run(args, stacklet, config): argv = list(args or []) actor, argv = _opt(argv, "--by") @@ -149,6 +168,7 @@ def _replace(prior: str) -> str: # seam sees every mutation, so the check lives here. ValueError # becomes the error message the caller relays to the model. _check_frontmatter(seen["before"], seen["after"], repo_path) + _check_person(seen["after"], repo_path) # A whole-page rewrite of a list is a restructure. A restructure # never legitimately reopens or removes an item (measured # 2026-09-15: rewrites drop [x] marks, and prompt rules do not diff --git a/stacklets/memory/hooks/on_start_ready.py b/stacklets/memory/hooks/on_start_ready.py index 5f432364..b9d5b7f3 100644 --- a/stacklets/memory/hooks/on_start_ready.py +++ b/stacklets/memory/hooks/on_start_ready.py @@ -25,6 +25,7 @@ authenticated_remote, brain_path_for, ensure_brain_projection_admin, + ensure_people, ensure_vault_cloned, host_code_url, point_remote_at, @@ -35,6 +36,30 @@ vault_path_for, vault_remote_url, ) +from stack.forgejo import ForgejoClient, ForgejoError # noqa: E402 +from stack.vault import DEFAULT_SHARED_BUCKET # noqa: E402 + + +def _ensure_people(ctx, code_url: str, token: str) -> None: + """Add a person file for every account that has none. + + Runs before the clone or pull, so the files are on disk when this + hook returns. A failure is reported and never blocks startup: every + reader keeps its previous behaviour until the files exist, and the + next `stack up memory` tries again. + """ + if not (code_url and token): + return + bucket = ctx.env.get("SHARED_BUCKET") or DEFAULT_SHARED_BUCKET + try: + added = ensure_people( + ForgejoClient(url=code_url, token=token), ctx.users, shared_bucket=bucket) + except (ForgejoError, OSError) as e: + # OSError: urllib's URLError when Forgejo is not reachable. + ctx.warn(f"Memory: could not write the person files: {e}") + return + if added: + ctx.step(f"Memory: added {', '.join(added)} to {bucket}/people/") def run(ctx): @@ -83,11 +108,14 @@ def remote_for(tok: str) -> str: if reason: if fresh := reissue_write_token(code_url, admin_user, admin_password): ctx.secret("MEMORY_BOT_TOKEN", fresh) + token = fresh remote = remote_for(fresh) ctx.step(f"{reason}; issued a new one") else: ctx.step(f"{reason} and it could not be replaced") + _ensure_people(ctx, code_url, token) + # If the vault never got cloned (install hook ran before code # stacklet was reachable, for example), try once more here. This # is the recovery path — same idempotent shape as install. diff --git a/stacklets/memory/lib.py b/stacklets/memory/lib.py index 02a31159..244d59c0 100644 --- a/stacklets/memory/lib.py +++ b/stacklets/memory/lib.py @@ -1568,6 +1568,49 @@ def install_seeds( return {"created": created, "skipped": skipped} +def ensure_people( + client: ForgejoClient, + users: list[dict], + *, + shared_bucket: str = DEFAULT_SHARED_BUCKET, +) -> list[str]: + """Add a person file for every account that has none; return their ids. + + The family's people are one file each under `<shared_bucket>/people/` + (`stack.people`). Every account holder gets one, marked with their + account, which makes them a member. An existing file is never read + or rewritten here, so a family edit (an alias, a corrected name) + survives every restart; a new account gets its file on the next + `stack up memory`. + + That makes this the whole upgrade path for an existing install: a + restart of the memory stacklet writes the files, and every reader + switches from its old guess to them as soon as they appear. + """ + from stack.forgejo import FileChange + from stack.people import PEOPLE_DIR, people_from_accounts, person_id, person_path, render_person + + accounts = people_from_accounts(users) + if not accounts: + return [] + listing = client.list_dir(REPO_OWNER, REPO_NAME, f"{shared_bucket}/{PEOPLE_DIR}") + existing = { + person_id(Path(entry["path"]).stem) + for entry in listing + if entry.get("type") == "file" and str(entry.get("path", "")).endswith(".md") + } + new = [p for p in accounts if p.id not in existing] + if not new: + return [] + client.change_files( + REPO_OWNER, REPO_NAME, + [FileChange("create", person_path(p.id, shared_bucket), render_person(p)) for p in new], + message="people: add a file for each chat account", + author_name=BOT_USERNAME, author_email=BOT_EMAIL, + ) + return [p.id for p in new] + + # Seed paths beginning with `_shared/` get retargeted to the # configured bucket slug. Everything else lands at the vault root # verbatim — keeps `ontology.toml`, `facts.toml`, `README.md` at the diff --git a/stacklets/memory/quartz/quartz.config.ts b/stacklets/memory/quartz/quartz.config.ts index 98db3973..9e3f6b2a 100644 --- a/stacklets/memory/quartz/quartz.config.ts +++ b/stacklets/memory/quartz/quartz.config.ts @@ -60,9 +60,11 @@ const config: QuartzConfig = { // `**/diary/entries/**` keeps the diary's records off the site. They // are the raw source the diary pages are compiled from, and stay in // the brain for search and the agent; the family reads the pages. + // `*/people/*.md` keeps the declared person files off the site for + // the same reason: each person's page is compiled from theirs. ignorePatterns: [".git", ".obsidian", "private", "templates", "README.md", "**/README.md", "media/**/*.json", "media/**/.tmp-*", - "**/diary/entries/**"], + "**/diary/entries/**", "*/people/*.md"], defaultDateType: "modified", theme: { // Self-hosted. The comment above about analytics applies with more diff --git a/tests/unit/framework/test_people.py b/tests/unit/framework/test_people.py new file mode 100644 index 00000000..7b0a351c --- /dev/null +++ b/tests/unit/framework/test_people.py @@ -0,0 +1,175 @@ +"""The family's people: one declared file each, read strictly. + +`<shared_bucket>/people/<id>.md` is the only place the family says who +it knows (docs/design/brain/vault-format.md §5, `person` (declared)). +These tests pin what the readers depend on: the format, that a member +is a person with an account, that a broken file is reported by path +and never guessed at, and that "no people" is a different answer from +"these people". +""" + +from __future__ import annotations + +from stack.people import ( + Person, + find_people, + load_people, + members, + people_dir, + people_from_accounts, + read_person, + render_person, + resolve, +) + + +def _write(root, name: str, text: str, bucket: str = "family") -> None: + folder = people_dir(root, bucket) + folder.mkdir(parents=True, exist_ok=True) + (folder / name).write_text(text, encoding="utf-8") + + +HOMER = "---\ntype: person\ntitle: Homer Simpson\naccount: homer\n---\n\n# Homer Simpson\n" +MAGGIE = ("---\ntype: person\ntitle: Maggie Simpson\naliases:\n - Margaret\n" + " - Mags\n---\n\nSays her first word in season four.\n") + + +class TestReadingAPersonFile: + def test_the_file_name_is_the_id_and_the_frontmatter_the_names(self): + assert read_person("maggie", MAGGIE) == Person( + "maggie", "Maggie Simpson", ("Margaret", "Mags")) + + def test_a_person_with_an_account_is_a_member(self): + """The first iteration's guess at membership: an account.""" + assert read_person("homer", HOMER).member + + def test_a_person_without_an_account_is_known_but_not_a_member(self): + assert not read_person("maggie", MAGGIE).member + + def test_the_body_is_the_familys_and_never_read(self): + assert read_person("maggie", MAGGIE).aliases == ("Margaret", "Mags") + + def test_every_name_a_person_answers_to(self): + """Full name, first name, then aliases: what the classifier and + transcription listen for.""" + assert read_person("maggie", MAGGIE).all_names() == [ + "Maggie Simpson", "Maggie", "Margaret", "Mags"] + + def test_a_file_named_in_obsidian_still_names_a_person(self): + text = "---\ntype: person\ntitle: Abe Simpson\n---\n" + assert read_person("Grampa Simpson", text).id == "grampa-simpson" + + +class TestABrokenFileIsReportedNotGuessed: + """A family edits these by hand. A mistake must say where it is.""" + + def _problems(self, tmp_path, text: str) -> list[str]: + _write(tmp_path, "homer.md", HOMER) + _write(tmp_path, "maggie.md", text) + problems: list[str] = [] + people = load_people(tmp_path, report=problems.append) + assert [p.id for p in people or []] == ["homer"] + return problems + + def test_frontmatter_outside_the_format(self, tmp_path): + problems = self._problems( + tmp_path, "---\ntype: person\ntitle: Maggie\naliases: |\n Margaret\n---\n") + assert problems and problems[0].startswith("family/people/maggie.md: ") + + def test_one_alias_where_a_list_belongs(self, tmp_path): + problems = self._problems( + tmp_path, "---\ntype: person\ntitle: Maggie Simpson\naliases: Margaret\n---\n") + assert "`aliases` must be a list" in problems[0] + + def test_a_file_without_a_name(self, tmp_path): + problems = self._problems(tmp_path, "---\ntype: person\naliases:\n - Margaret\n---\n") + assert "`title`" in problems[0] + + def test_a_note_that_is_not_a_person(self, tmp_path): + problems = self._problems(tmp_path, "Ideas for Maggie's birthday\n") + assert "type: person" in problems[0] + + def test_bytes_that_are_not_text(self, tmp_path): + _write(tmp_path, "homer.md", HOMER) + (people_dir(tmp_path) / "maggie.md").write_bytes(b"---\ntype: \xff\xfe\n---\n") + problems: list[str] = [] + assert [p.id for p in load_people(tmp_path, report=problems.append)] == ["homer"] + assert "cannot be read" in problems[0] + + def test_an_id_declared_twice_counts_once(self, tmp_path): + _write(tmp_path, "homer.md", HOMER) + _write(tmp_path, "Homer!.md", HOMER) + problems: list[str] = [] + assert len(load_people(tmp_path, report=problems.append)) == 1 + assert "declared twice" in problems[0] + + +class TestNoPeopleIsNotAnEmptyHousehold: + def test_no_folder_reads_as_none(self, tmp_path): + """Readers keep their previous behaviour until the files exist.""" + assert load_people(tmp_path) is None + + def test_a_folder_of_broken_files_reads_as_none(self, tmp_path): + """Nobody readable is never acted on as "nobody".""" + _write(tmp_path, "maggie.md", "not a person\n") + assert load_people(tmp_path) is None + + def test_the_files_live_in_the_shared_bucket(self, tmp_path): + _write(tmp_path, "homer.md", HOMER, bucket="familie") + assert [p.id for p in load_people(tmp_path, "familie")] == ["homer"] + assert load_people(tmp_path) is None + + def test_the_first_root_with_people_wins(self, tmp_path): + """The working copy is fresher than the brain mirror of it.""" + vault, brain = tmp_path / "vault", tmp_path / "brain" + _write(brain, "homer.md", HOMER) + assert [p.id for p in find_people((None, vault, brain))] == ["homer"] + _write(vault, "maggie.md", MAGGIE) + assert [p.id for p in find_people((vault, brain))] == ["maggie"] + + +class TestWhoIsWho: + def test_members_come_first_then_everyone_else_by_name(self, tmp_path): + _write(tmp_path, "maggie.md", MAGGIE) + _write(tmp_path, "marge.md", "---\ntype: person\ntitle: Marge Simpson\naccount: marge\n---\n") + _write(tmp_path, "homer.md", HOMER) + people = load_people(tmp_path) + assert [p.id for p in people] == ["homer", "marge", "maggie"] + assert [p.id for p in members(people)] == ["homer", "marge"] + + def test_a_name_resolves_by_any_name_the_person_goes_by(self): + maggie = read_person("maggie", MAGGIE) + assert resolve("margaret", [maggie]) is maggie + assert resolve("Maggie Simpson", [maggie]) is maggie + assert resolve("Simpson", [maggie]) is None + + +class TestTheFilesFamstackWrites: + """`stack up memory` adds one for every account that has none.""" + + def test_every_account_holder_becomes_a_member(self): + users = [ + {"name": "Homer Simpson", "role": "admin"}, + {"name": "Marge Simpson"}, + {"name": "Bart", "id": "bart"}, + ] + assert people_from_accounts(users) == [ + Person("homer", "Homer Simpson", account="homer"), + Person("marge", "Marge Simpson", account="marge"), + Person("bart", "Bart", account="bart"), + ] + + def test_the_id_is_the_chat_user_name(self): + """An explicit `id` in users.toml is the Matrix localpart; so is the person id.""" + assert people_from_accounts([{"name": "Homer Simpson", "id": "hjs"}])[0].id == "hjs" + + def test_an_entry_without_a_name_is_skipped(self): + assert people_from_accounts([{"id": "ghost"}]) == [] + + def test_a_written_file_reads_back_as_the_same_person(self): + homer = Person("homer", "Homer Simpson", ("Homie",), account="homer") + assert read_person("homer", render_person(homer)) == homer + + def test_a_written_file_follows_the_vault_format(self): + page = render_person(Person("homer", "Homer Simpson", account="homer")) + assert page.startswith("---\ntype: person\ntitle: Homer Simpson\naccount: homer\n---\n") diff --git a/tests/unit/stacklets/test_frontmatter.py b/tests/unit/stacklets/test_frontmatter.py index 93523fbd..c948eb8e 100644 --- a/tests/unit/stacklets/test_frontmatter.py +++ b/tests/unit/stacklets/test_frontmatter.py @@ -322,16 +322,35 @@ def test_record_carrying_generated_error(self): assert any("must not carry `generated`" in e for e in errors) def test_projection_missing_generated_error(self): - """Validator rejects projection type (person) missing generated marker.""" + """Validator rejects projection type (correspondent) missing generated marker.""" fm = { - "type": "person", - "title": "Homer", - "slug": "homer", - "canonical": "Homer", + "type": "correspondent", + "title": "Duff Insurance", + "canonical": "Duff Insurance", } errors = validate(fm) assert any("must have `generated: true`" in e for e in errors) + def test_a_person_without_the_marker_is_the_familys_own_file(self): + """vault-format §5, `person` (declared): type and title, nothing generated.""" + fm = {"type": "person", "title": "Maggie Simpson", + "aliases": ["Margaret"], "account": "maggie"} + assert validate(fm) == [] + + def test_a_declared_person_needs_a_title(self): + errors = validate({"type": "person", "aliases": ["Margaret"]}) + assert errors == ["missing required field for `person`: `title`"] + + def test_a_declared_persons_aliases_are_a_list(self): + """`aliases: Margaret` is one name where the format asks for a list.""" + errors = validate({"type": "person", "title": "Maggie Simpson", + "aliases": "Margaret"}) + assert any("`aliases` must be a list" in e for e in errors) + + def test_a_declared_persons_name_is_one_value(self): + errors = validate({"type": "person", "title": ["Maggie", "Margaret"]}) + assert any("`title` must be a single value" in e for e in errors) + def test_list_field_not_list(self): """Validator rejects list field that is a scalar.""" fm = { diff --git a/tests/unit/stacklets/test_memory_people.py b/tests/unit/stacklets/test_memory_people.py new file mode 100644 index 00000000..fd22553e --- /dev/null +++ b/tests/unit/stacklets/test_memory_people.py @@ -0,0 +1,291 @@ +"""The memory stacklet and the family's person files. + +Every person the family knows has a file, `family/people/<id>.md` +(`stack.people`). This file pins how the memory stacklet treats them: + +- `stack up memory` adds a file for every account that has none, on + every start, and never rewrites one: the restart of an existing + install is the upgrade, and the family's edits survive every restart. +- The wiki gives every declared person a page under the names their + file gives, and its home page lists the members (people with an + account). Before any file exists the wiki keeps its old roster, so + nothing changes until the restart. +- Generated person pages nobody declares (`media/about.md`, from the + old roster) are retired, but never while a person file is broken. +- A change to a person file rebuilds every person page. + +Forgejo is stubbed with `pytest-httpserver`; the wiki works on real +files in a temporary brain and vault. +""" + +from __future__ import annotations + +import asyncio +import base64 +import json +import sys +from pathlib import Path + +import pytest +import yaml + +_REPO_ROOT = Path(__file__).resolve().parent.parent.parent.parent +sys.path.insert(0, str(_REPO_ROOT / "stacklets" / "memory")) +sys.path.insert(0, str(_REPO_ROOT / "stacklets")) +sys.path.insert(0, str(_REPO_ROOT / "stacklets" / "memory" / "bot")) +sys.path.insert(0, str(_REPO_ROOT / "stacklets" / "memory" / "bot" / "cli")) + +from lib import REPO_NAME, REPO_OWNER, ensure_people # noqa: E402 +from stack.forgejo import ForgejoClient # noqa: E402 +from stack.people import Person, read_person # noqa: E402 + +import wiki # noqa: E402 +from curator import member_selection # noqa: E402 + +CONTENTS = f"/api/v1/repos/{REPO_OWNER}/{REPO_NAME}/contents" +PEOPLE = f"{CONTENTS}/family/people" + +SIMPSONS = [ + {"name": "Homer Simpson", "role": "admin"}, + {"name": "Marge Simpson"}, + {"name": "Bart Simpson"}, + {"name": "Lisa Simpson"}, +] + +HOMER = "---\ntype: person\ntitle: Homer Simpson\naccount: homer\n---\n" +MARGE = "---\ntype: person\ntitle: Marge Simpson\naccount: marge\n---\n" +MAGGIE = "---\ntype: person\ntitle: Maggie Simpson\naliases:\n - Margaret\n---\n" + + +def _client(httpserver) -> ForgejoClient: + return ForgejoClient(url=httpserver.url_for(""), token="bottoken") + + +def _listing(*names: str) -> list[dict]: + return [{"type": "file", "path": f"family/people/{n}", "sha": "abc"} for n in names] + + +def _committed(httpserver) -> dict[str, str]: + """The files the one commit created, path to content.""" + post = next(r for r, _ in httpserver.log if r.method == "POST") + return {f["path"]: base64.b64decode(f["content"]).decode() + for f in json.loads(post.data)["files"]} + + +# ── Writing the files on start ─────────────────────────────────────────── + +class TestTheStartAddsAFilePerAccount: + def test_an_install_without_files_gets_one_per_account_in_one_commit(self, httpserver): + httpserver.expect_request(PEOPLE, method="GET").respond_with_data("", status=404) + httpserver.expect_request(CONTENTS, method="POST").respond_with_json({}, status=201) + + added = ensure_people(_client(httpserver), SIMPSONS) + + assert added == ["homer", "marge", "bart", "lisa"] + files = _committed(httpserver) + assert sorted(files) == [f"family/people/{i}.md" for i in ("bart", "homer", "lisa", "marge")] + homer = read_person("homer", files["family/people/homer.md"]) + assert homer == Person("homer", "Homer Simpson", account="homer") + assert homer.member + + def test_an_existing_file_is_never_rewritten(self, httpserver): + """The family gave Homer an alias; a restart adds only who is missing.""" + httpserver.expect_request(PEOPLE, method="GET").respond_with_json( + _listing("homer.md", "marge.md", "bart.md", "maggie.md")) + httpserver.expect_request(CONTENTS, method="POST").respond_with_json({}, status=201) + + assert ensure_people(_client(httpserver), SIMPSONS) == ["lisa"] + assert list(_committed(httpserver)) == ["family/people/lisa.md"] + + def test_nothing_missing_commits_nothing(self, httpserver): + httpserver.expect_request(PEOPLE, method="GET").respond_with_json( + _listing("homer.md", "marge.md", "bart.md", "lisa.md")) + + assert ensure_people(_client(httpserver), SIMPSONS) == [] + assert not [r for r, _ in httpserver.log if r.method == "POST"] + + +# ── The wiki's people ──────────────────────────────────────────────────── + +@pytest.fixture +def instance(tmp_path, monkeypatch): + """A vault and brain holding what the old roster got wrong. + + A `media/` folder (diary recordings), a bot's bucket, documents that + call the baby "Margaret" and name a surname, and a generated person + page for `media/` left over from the old roster. + """ + vault, brain = tmp_path / "vault", tmp_path / "brain" + for root in (vault, brain): + for folder in ("homer", "marge", "media", "mail-bot", "family/documents"): + (root / folder).mkdir(parents=True, exist_ok=True) + (brain / "media" / "about.md").write_text( + "---\ntitle: Media\nslug: media\ntype: person\ngenerated: true\n---\n\n" + "<!-- begin: generated -->\n\nMaggie's page, under the wrong folder.\n\n" + "<!-- end: generated -->\n") + monkeypatch.setenv("MEMORY_VAULT_DIR", str(vault)) + monkeypatch.setenv("BRAIN_REPO_DIR", str(brain)) + index = [ + {"rel": "family/documents/a.md", "persons": ["margaret"], + "person_names": ["Margaret"], "title": "Birth certificate"}, + {"rel": "family/documents/b.md", "persons": ["simpson", "homer"], + "person_names": ["Simpson", "Homer"], "title": "Car insurance"}, + ] + return vault, brain, index + + +def _declare(root: Path, **files: str) -> None: + folder = root / "family" / "people" + folder.mkdir(parents=True, exist_ok=True) + for person_id, text in files.items(): + (folder / f"{person_id}.md").write_text(text) + + +def _people(problems: list[str] | None = None): + return wiki._people("family", problems if problems is not None else []) + + +class TestTheWikisPeople: + def test_before_any_file_exists_the_old_roster_stands(self, instance): + """Nothing changes until `stack up memory` has written the files.""" + vault, brain, index = instance + people = _people() + assert people is None + assert wiki._roster(brain, index, "family", people) == wiki._member_slugs( + brain, index, "family") + + def test_every_declared_person_has_a_page_members_first(self, instance): + vault, brain, index = instance + _declare(vault, homer=HOMER, marge=MARGE, maggie=MAGGIE) + assert wiki._roster(brain, index, "family", _people()) == ["homer", "marge", "maggie"] + + def test_names_resolve_through_the_familys_aliases(self, instance): + """The birth certificate says "Margaret"; it belongs on Maggie's page.""" + vault, brain, index = instance + _declare(vault, homer=HOMER, maggie=MAGGIE) + wiki._resolve_persons(index, _people()) + assert index[0]["persons"] == ["maggie"] + assert wiki._member_entries(index, "maggie") == [index[0]] + + def test_a_name_nobody_answers_to_gets_no_page(self, instance): + vault, brain, index = instance + people = [Person("homer", "Homer Simpson", account="homer")] + wiki._resolve_persons(index, people) + assert "simpson" not in wiki._roster(brain, index, "family", people) + + def test_the_files_are_read_from_the_vault_before_the_brain(self, instance): + """An edit is on disk in the vault before the curator mirrors it.""" + vault, brain, index = instance + _declare(brain, homer=HOMER) + _declare(vault, homer=HOMER, maggie=MAGGIE) + assert [p.id for p in _people()] == ["homer", "maggie"] + + def test_a_broken_file_is_reported_and_left_out(self, instance): + vault, brain, index = instance + _declare(vault, homer=HOMER, maggie="---\ntype: person\naliases: Margaret\n---\n") + problems: list[str] = [] + assert [p.id for p in _people(problems)] == ["homer"] + assert problems[0].startswith("family/people/maggie.md: ") + + def test_member_option_takes_any_name_the_person_answers_to(self): + people = [Person("maggie", "Maggie Simpson", ("Margaret",))] + assert wiki._member_id("Margaret", people) == "maggie" + assert wiki._member_id("Maggie Simpson", people) == "maggie" + + +class TestEveryDeclaredPersonHasAPage: + def test_a_person_nothing_names_yet_gets_a_page_without_the_model(self, instance): + """The home page and the People menu link every person's page.""" + vault, brain, index = instance + maggie = Person("maggie", "Maggie Simpson", ("Margaret",)) + llm = None # never called: there is nothing for it to write about + + rc = asyncio.run(wiki._generate_member( + llm, "maggie", [], brain, person=maggie, + shared_bucket="family", lang="en", write=True)) + + page = (brain / "maggie" / "about.md").read_text() + assert rc == 0 + assert yaml.safe_load(page.split("---\n")[1])["canonical"] == "Maggie Simpson" + assert "Nothing filed about Maggie yet" in page + + def test_without_person_files_an_empty_bucket_is_still_skipped(self, instance): + vault, brain, index = instance + rc = asyncio.run(wiki._generate_member( + None, "media", [], brain, shared_bucket="family", lang="en", write=True)) + assert rc == 1 + + +class TestUndeclaredPersonPages: + def test_a_generated_page_nobody_declares_is_retired(self, instance): + vault, brain, index = instance + retired = wiki._retire_undeclared(["homer", "maggie"], "family", write=True) + assert retired == ["media/about.md"] + assert not (brain / "media" / "about.md").exists() + + def test_a_page_a_person_wrote_is_left_alone(self, instance): + vault, brain, index = instance + (brain / "media" / "about.md").write_text("---\ntitle: Media\n---\n\nOur photos.\n") + assert wiki._retire_undeclared(["homer"], "family", write=True) == [] + assert (brain / "media" / "about.md").exists() + + def test_a_dry_run_names_the_page_and_keeps_it(self, instance): + vault, brain, index = instance + assert wiki._retire_undeclared(["homer"], "family", write=False) == ["media/about.md"] + assert (brain / "media" / "about.md").exists() + + def test_no_page_is_retired_while_a_person_file_is_broken(self, instance, monkeypatch): + """Maggie's file has a typo: her page must survive until it is fixed.""" + vault, brain, index = instance + _declare(vault, homer=HOMER, maggie="---\ntype: person\naliases: Margaret\n---\n") + (brain / "maggie").mkdir() + (brain / "maggie" / "about.md").write_text( + "---\ntitle: Maggie Simpson\nslug: maggie\ntype: person\ngenerated: true\n---\n") + + async def _no_model(*args, **kwargs): + return 0 + monkeypatch.setattr(wiki, "_generate_home", _no_model) + monkeypatch.setattr(wiki, "_generate_member", _no_model) + monkeypatch.setattr(wiki, "_index_vault", lambda vault: index) + monkeypatch.setattr(wiki, "_brain_dir", lambda: brain) + + asyncio.run(wiki.run(None, ["--members"])) + + assert (brain / "maggie" / "about.md").exists() + assert (brain / "media" / "about.md").exists() + + +class TestPersonPageNames: + def test_the_declared_names_replace_what_the_page_had(self): + page = ("---\ntitle: Margaret\nslug: maggie\ntype: person\ncanonical: Margaret\n" + "synonyms:\n - Maggie\nbirthday: \"2025-01-12\"\n---\n\nbody\n") + updated = wiki._with_member_names( + page, canonical="Maggie Simpson", synonyms=["Margaret", "Maggie"]) + frontmatter = yaml.safe_load(updated.split("---\n")[1]) + assert frontmatter == { + "title": "Maggie Simpson", "canonical": "Maggie Simpson", + "synonyms": ["Margaret", "Maggie"], + "slug": "maggie", "type": "person", "birthday": "2025-01-12", + } + assert updated.endswith("---\n\nbody\n") + + def test_the_home_page_is_told_the_full_names(self): + prompt = wiki._build_home_prompt( + [], roster=["homer", "marge"], lang="en", + names={"homer": "Homer Simpson", "marge": "Marge Simpson"}) + assert "- **[Marge Simpson](marge/about)**" in prompt + assert "<full name of" not in prompt + + +# ── Rebuilding on a change ─────────────────────────────────────────────── + +class TestTheCuratorRebuildsOnAChange: + def test_a_changed_person_file_rebuilds_every_person_page(self): + argv = member_selection(["family/people/maggie.md"], lambda path: {}, + shared_bucket="family") + assert argv == ["--home", "--members"] + + def test_other_changes_select_as_before(self): + argv = member_selection(["homer/notes/2026/10/x.md"], lambda path: {}, + shared_bucket="family") + assert argv == ["--home", "--member", "homer"] diff --git a/tests/unit/stacklets/test_memory_token_repair.py b/tests/unit/stacklets/test_memory_token_repair.py index 684fcfdb..49a26854 100644 --- a/tests/unit/stacklets/test_memory_token_repair.py +++ b/tests/unit/stacklets/test_memory_token_repair.py @@ -106,8 +106,9 @@ class _RecordingCtx: """The slice of hook context `on_start_ready` actually reads. A real `StackContext` wants an instance on disk, a parsed config and - a Docker runtime. The hook wants four things: the data dir, the - rendered env, the secret store, and somewhere to log progress and warnings. + a Docker runtime. The hook wants five things: the data dir, the + rendered env, the secret store, the accounts, and somewhere to log + progress and warnings. Standing up only those four is what lets the hook be driven end to end here, which matters because the bug this file guards lived in `run()` itself and not in any helper it calls. @@ -117,6 +118,7 @@ def __init__(self, data_dir: Path, env: dict, secrets: dict): self.stack = SimpleNamespace(data=data_dir) self.env = env self.secrets = secrets + self.users: list[dict] = [] self.steps: list[str] = [] def secret(self, name, value=None): diff --git a/tests/unit/stacklets/test_memory_write_cli.py b/tests/unit/stacklets/test_memory_write_cli.py index 4959b816..5637dfdb 100644 --- a/tests/unit/stacklets/test_memory_write_cli.py +++ b/tests/unit/stacklets/test_memory_write_cli.py @@ -321,3 +321,26 @@ def test_a_write_the_readers_have_not_caught_up_with_is_still_done(store, tmp_pa assert result["mirrored"] is False out = capsys.readouterr().out assert "Wrote" in out and "catch up" in out + + +# ── person files ───────────────────────────────────────────────────────── + +def test_a_person_file_the_readers_would_skip_is_refused(store, tmp_path): + """Every reader leaves out a broken person file; the write says so first. + + `aliases: Margaret` is one name where the format asks for a list + (vault-format §5, `person` (declared)). + """ + result = _run(store, "---\ntype: person\ntitle: Maggie Simpson\naliases: Margaret\n---\n", + path="family/people/maggie.md", tmp=tmp_path) + + assert "error" in result + assert "`aliases` must be a list" in result["error"] + assert store.commits == [] + + +def test_a_person_file_in_the_format_is_written(store, tmp_path): + result = _run(store, "---\ntype: person\ntitle: Maggie Simpson\naliases:\n" + " - Margaret\n---\n", path="family/people/maggie.md", tmp=tmp_path) + + assert result["committed"] is True From 8c8d735c4abe8bfa174433f83b117f473897c796 Mon Sep 17 00:00:00 2001 From: Arthur <hello@arthware.dev> Date: Fri, 9 Oct 2026 13:01:36 +0200 Subject: [PATCH 3/6] feat(archivist): file documents under every person by any name The archivist knew the family from the Paperless `Person:` tags, which are seeded from the accounts' first names. A person without an account had no tag and was never filed, and a document naming "Margaret" could not be matched to Maggie. With person files, the classifier and the capture prompt list every declared person with every name the family gave them, accounts or not. A returned name resolves through those names to the person's first name, and a person without a `Person:` tag gets one the first time a document names them, so no restart of the docs stacklet is needed. `stack memory prompt` shows the block the classifier receives and names any person file that was skipped, with the reason. Without person files, nothing changes. Refs: #154 --- stacklets/docs/bot/archivist.py | 1 + stacklets/docs/bot/capture_pipeline.py | 21 ++++++-- stacklets/docs/bot/document_pipeline.py | 2 + stacklets/docs/bot/matching.py | 32 ++++++++++++ stacklets/docs/bot/pipeline.py | 18 +++++++ stacklets/docs/bot/vault_context.py | 27 ++++++++-- stacklets/memory/cli/prompt.py | 29 ++++++++--- stacklets/memory/lib.py | 19 +++++++ tests/unit/stacklets/test_memory_people.py | 57 +++++++++++++++++++++ tests/unit/stacklets/test_pipeline.py | 59 ++++++++++++++++++++++ 10 files changed, 249 insertions(+), 16 deletions(-) diff --git a/stacklets/docs/bot/archivist.py b/stacklets/docs/bot/archivist.py index 999ed406..e4ba8ea3 100644 --- a/stacklets/docs/bot/archivist.py +++ b/stacklets/docs/bot/archivist.py @@ -557,6 +557,7 @@ def _build_services(self) -> ArchivistServices: capture_tag_prompt_size=self.capture_tag_prompt_size, vision_max_pdf_pages=self.vision_max_pdf_pages, llm=classifier.llm if classifier is not None else None, + vault=vault, ) return ArchivistServices( paperless=paperless, diff --git a/stacklets/docs/bot/capture_pipeline.py b/stacklets/docs/bot/capture_pipeline.py index f34ecd08..839afb84 100644 --- a/stacklets/docs/bot/capture_pipeline.py +++ b/stacklets/docs/bot/capture_pipeline.py @@ -165,8 +165,12 @@ def __init__( capture_tag_prompt_size: int, vision_max_pdf_pages: int = DEFAULT_VISION_MAX_PDF_PAGES, llm=None, + vault=None, ): self._url_extractor = url_extractor + # The memory vault, for the person files. None in callers + # that run without one; the Person tags stand in for them then. + self._vault = vault self._text_extractor = text_extractor self._classifier = classifier self._mirror = mirror @@ -772,6 +776,17 @@ async def _publish( scope=entity_slug, ) + async def _family_names(self) -> list[str]: + """The family's first names: the person files, else the Person tags.""" + people = self._vault.people() if self._vault is not None else None + if people is not None: + return [p.first_name for p in people] + person_tags = await self._paperless.get_tags() + return [ + t.replace("Person: ", "") for t in person_tags + if t.startswith("Person: ") + ] + async def _classify( self, source, sender_name: str, *, images: list[ImageAttachment] | None = None, @@ -795,11 +810,7 @@ async def _classify( # Bot was brought up without AI configured; fall through to the # minimal classification below so the capture still files. return {} - person_tags = await self._paperless.get_tags() - person_names = [ - t.replace("Person: ", "") for t in person_tags - if t.startswith("Person: ") - ] + person_names = await self._family_names() existing_tags = ( self._capture_tags.top(self.capture_tag_prompt_size) if self._capture_tags else [] diff --git a/stacklets/docs/bot/document_pipeline.py b/stacklets/docs/bot/document_pipeline.py index c03044b2..8d2c966d 100644 --- a/stacklets/docs/bot/document_pipeline.py +++ b/stacklets/docs/bot/document_pipeline.py @@ -378,6 +378,7 @@ async def _enrich( ontology_section=ontology.classifier_prompt_section(self.language), correspondents_section=self._vault.correspondents_section(), persons_section=self._vault.persons_section(), + people=self._vault.people(), ontology=ontology, lang=self.language, date_filed=date_filed, @@ -465,6 +466,7 @@ async def reprocess( ontology_section=ontology.classifier_prompt_section(self.language), correspondents_section=self._vault.correspondents_section(), persons_section=self._vault.persons_section(), + people=self._vault.people(), ontology=ontology, lang=self.language, is_reprocess=True, diff --git a/stacklets/docs/bot/matching.py b/stacklets/docs/bot/matching.py index 47ab6d11..804558b1 100644 --- a/stacklets/docs/bot/matching.py +++ b/stacklets/docs/bot/matching.py @@ -188,6 +188,38 @@ def match_persons(names: str | list | None, tags: dict[str, Any]) -> list[str]: return matched_tags +def declared_person_names(names: str | list | None, people) -> list[str]: + """LLM-returned person names, each as its declared person's first name. + + The first name is what a person's `Person:` tag carries. A name the + family declared under any spelling ("Margaret", "Maggie Simpson", + "Person: Maggie") becomes that person's first name; any other name + is passed on unchanged for `match_persons` to try, so a document + still matches by the tags alone, as it did before the person files. + + >>> from stack.people import Person + >>> people = [Person("maggie", "Maggie Simpson", ("Margaret",))] + >>> declared_person_names(["Margaret", "Homer"], people) + ['Maggie', 'Homer'] + """ + from stack.people import resolve + + if isinstance(names, str): + names = [names] + if not isinstance(names, list): + return [] + out: list[str] = [] + for name in names: + if not isinstance(name, str) or _is_empty(name): + continue + clean = name.replace("Person: ", "").strip() + person = resolve(clean, people) + resolved = person.first_name if person else clean + if resolved and resolved not in out: + out.append(resolved) + return out + + def submitter_person_tag( mxid: str | None, tags: dict[str, Any], ) -> str | None: diff --git a/stacklets/docs/bot/pipeline.py b/stacklets/docs/bot/pipeline.py index 14b7f4c9..f807cf0b 100644 --- a/stacklets/docs/bot/pipeline.py +++ b/stacklets/docs/bot/pipeline.py @@ -42,6 +42,7 @@ MAX_TITLE_LENGTH, _is_empty, fuzzy_match_entity, + declared_person_names, match_persons, match_topics, submitter_person_tag, @@ -78,6 +79,7 @@ ImageAttachment = LLMImage if TYPE_CHECKING: + from stack.people import Person from stack.ontology import Ontology @@ -1637,6 +1639,8 @@ def extract_bot_summary(doc: dict) -> str: _ISO_DATE = re.compile(r"^\d{4}-\d{2}-\d{2}$") _NEW_TOPIC_COLOR = "#4caf50" +# Same colour `seed.py` gives the person tags it creates for accounts. +_PERSON_TAG_COLOR = "#2196f3" async def enrich_document( @@ -1657,6 +1661,7 @@ async def enrich_document( initial_classification: dict | None = None, submitter_mxid: str | None = None, write_in: str | None = None, + people: "list[Person] | None" = None, ) -> EnrichResult: """Classify a doc, reconcile entities, PATCH Paperless. Pure data out. @@ -1758,6 +1763,19 @@ async def enrich_document( # tag name; we strip the prefix for the resolved-name list callers # want to render. persons_raw = classification.get("persons") or classification.get("person") + if people is not None: + # The person files name who the family knows, accounts or not: + # each name resolves through the person's aliases, and a person + # who has no `Person:` tag yet (no account, so none was seeded) + # gets one the first time a document names them. + persons_raw = declared_person_names(persons_raw, people) + firsts = {p.first_name for p in people} + for name in persons_raw: + tag = f"Person: {name}" + if name in firsts and tag not in tags: + if new_id := await paperless.create_tag(tag, _PERSON_TAG_COLOR): + tags[tag] = new_id + result.created_new.append(f'tag "{tag}"') for pt in match_persons(persons_raw, tags): tag_ids.append(tags[pt]) result.resolved_persons.append(pt.replace("Person: ", "")) diff --git a/stacklets/docs/bot/vault_context.py b/stacklets/docs/bot/vault_context.py index 61111600..da4c821e 100644 --- a/stacklets/docs/bot/vault_context.py +++ b/stacklets/docs/bot/vault_context.py @@ -15,13 +15,17 @@ import os from pathlib import Path +from loguru import logger + from memory.lib import ( correspondents_prompt_section, get_ontology, load_correspondents_from_vault, load_persons_from_vault, + people_prompt_section, persons_prompt_section, ) +from stack.people import Person, load_people class VaultContext: @@ -65,15 +69,30 @@ def correspondents_section(self) -> str: correspondents = load_correspondents_from_vault(path, self.shared_bucket) return correspondents_prompt_section(correspondents) + def people(self) -> list[Person] | None: + """The family's declared people; None until a person file exists. + + A file that does not read is logged with its path and left out. + """ + return load_people( + self._vault_path(), self.shared_bucket, + report=lambda problem: logger.warning("[vault] person file skipped: {}", problem), + ) + def persons_section(self) -> str: - """Build the persons block from `<slug>/about.md` across the vault. + """The persons block: the person files, else `<slug>/about.md` pages. - Empty when the vault isn't seeded yet — the prompt then falls back - to the flat Paperless Person roster (canonical first names only, - no synonym signal). + With person files the block lists every declared person with every + name they go by, accounts or not. Without them it is built from + person pages in the vault, and empty when there are none -- the prompt + then falls back to the flat Paperless Person roster (canonical + first names only, no synonym signal). """ path = self._vault_path() if path is None: return "" + people = self.people() + if people is not None: + return people_prompt_section(people) persons = load_persons_from_vault(path, self.shared_bucket) return persons_prompt_section(persons) diff --git a/stacklets/memory/cli/prompt.py b/stacklets/memory/cli/prompt.py index d41bb221..33b1483b 100644 --- a/stacklets/memory/cli/prompt.py +++ b/stacklets/memory/cli/prompt.py @@ -29,9 +29,11 @@ get_ontology, load_correspondents_from_vault, load_persons_from_vault, + people_prompt_section, persons_prompt_section, vault_path_for, ) +from stack.people import load_people # noqa: E402 HELP = "Show the vocabulary the models are fed: topics, family, correspondents" @@ -65,8 +67,22 @@ def run(args, stacklet, config): # Person pages are generated projections and live in the brain. data_dir = (config or {}).get("data_dir") brain = brain_path_for(Path(data_dir)) if data_dir else None - persons = persons_prompt_section( - load_persons_from_vault(brain, bucket) if brain and brain.exists() else []) + # The person files are what the classifier and the diary are given + # once they exist; the person pages are the fallback before. A file + # the readers skip is listed under the block, since this is where an + # admin looks when a name is not recognised. + problems: list[str] = [] + people = load_people(vault, bucket, report=problems.append) + if people is not None: + persons = people_prompt_section(people) + persons_used_by = "document classifier, captures, diary cards and transcription" + else: + persons = persons_prompt_section( + load_persons_from_vault(brain, bucket) if brain and brain.exists() else []) + persons_used_by = ("diary cards (the document classifier looks in the vault " + "and finds none)") + if problems: + persons += "\n\nSkipped, fix these files:\n" + "\n".join(f" - {p}" for p in problems) # So are correspondent pages. correspondents = correspondents_prompt_section( load_correspondents_from_vault(brain, bucket) if brain and brain.exists() else []) @@ -75,12 +91,11 @@ def run(args, stacklet, config): _block("Topics and document types", "document classifier, search query rewrite, diary cards", ontology, "(the ontology is empty)"), - # The document classifier reads both of these from the vault, + # The document classifier reads correspondents from the vault, # where generated pages no longer are, so today it receives - # neither. The labels say so until it reads the brain. - _block("Family members", - "diary cards (the document classifier looks in the vault and finds none)", - persons, "(no person pages yet)"), + # none. The label says so until it reads the brain. + _block("Family members", persons_used_by, + persons, "(no person files and no person pages yet)"), _block("Correspondents", "nothing yet (the document classifier looks in the vault and finds none)", correspondents, "(no correspondent pages yet)"), diff --git a/stacklets/memory/lib.py b/stacklets/memory/lib.py index 244d59c0..6fa38155 100644 --- a/stacklets/memory/lib.py +++ b/stacklets/memory/lib.py @@ -1290,6 +1290,25 @@ def persons_prompt_section(persons: List[Person]) -> str: return "\n".join(lines) +def people_prompt_section(people) -> str: + """The persons block from the person files, in the shape above. + + The canonical name is the person's first name, the name their + `Person:` tag carries; the full name and every alias the family + listed follow in parens, so "Margaret" on a birth certificate is + recognised as Maggie. A person without a chat account is listed like + any other. + """ + if not people: + return "" + lines = ["Family members (canonical first name; synonyms in parens):"] + for p in people: + others = [n for n in p.all_names() if n != p.first_name] + lines.append(f" - {p.first_name} ({', '.join(others)})" if others + else f" - {p.first_name}") + return "\n".join(lines) + + def get_ontology(vault_path: Optional[Path] = None) -> Ontology: """Return the live ontology from the vault, else the shipped seed. diff --git a/tests/unit/stacklets/test_memory_people.py b/tests/unit/stacklets/test_memory_people.py index fd22553e..c7bbb44e 100644 --- a/tests/unit/stacklets/test_memory_people.py +++ b/tests/unit/stacklets/test_memory_people.py @@ -289,3 +289,60 @@ def test_other_changes_select_as_before(self): argv = member_selection(["homer/notes/2026/10/x.md"], lambda path: {}, shared_bucket="family") assert argv == ["--home", "--member", "homer"] + + +# ── What the archivist is told ─────────────────────────────────────────── + +class TestTheClassifierIsToldThePeople: + """The document classifier's persons block comes from the files once they exist.""" + + @staticmethod + def _vault_context(tmp_path, monkeypatch): + sys.path.insert(0, str(_REPO_ROOT / "stacklets" / "docs" / "bot")) + from vault_context import VaultContext + monkeypatch.setenv("MEMORY_VAULT_DIR", str(tmp_path)) + (tmp_path / "family").mkdir(exist_ok=True) + return VaultContext(language="en", shared_bucket="family") + + def test_every_person_with_every_name(self, tmp_path, monkeypatch): + """Maggie has no account and is listed like anyone else.""" + vault = self._vault_context(tmp_path, monkeypatch) + _declare(tmp_path, homer=HOMER, maggie=MAGGIE) + assert vault.persons_section() == ( + "Family members (canonical first name; synonyms in parens):\n" + " - Homer (Homer Simpson)\n" + " - Maggie (Maggie Simpson, Margaret)") + + def test_without_the_files_the_old_block(self, tmp_path, monkeypatch): + """No files, no person pages in the vault: empty, and the prompt + falls back to the Person tags as before.""" + vault = self._vault_context(tmp_path, monkeypatch) + assert vault.people() is None + assert vault.persons_section() == "" + + +class TestThePromptCommandShowsThePeople: + """`stack memory prompt` is where an admin looks when a name is not recognised.""" + + @staticmethod + def _prompt(tmp_path, capsys) -> str: + import importlib.util + spec = importlib.util.spec_from_file_location( + "memory_cli_prompt", _REPO_ROOT / "stacklets" / "memory" / "cli" / "prompt.py") + prompt = importlib.util.module_from_spec(spec) + spec.loader.exec_module(prompt) + prompt.run([], None, {"data_dir": str(tmp_path)}) + return capsys.readouterr().out + + def test_the_block_the_classifier_receives(self, tmp_path, capsys): + _declare(tmp_path / "memory" / "vault", homer=HOMER, maggie=MAGGIE) + out = self._prompt(tmp_path, capsys) + assert " - Maggie (Maggie Simpson, Margaret)" in out + assert "Used by: document classifier" in out + + def test_a_skipped_file_is_named_with_the_reason(self, tmp_path, capsys): + _declare(tmp_path / "memory" / "vault", homer=HOMER, + maggie="---\ntype: person\ntitle: Maggie Simpson\naliases: Margaret\n---\n") + out = self._prompt(tmp_path, capsys) + assert "Skipped, fix these files:" in out + assert "family/people/maggie.md: field `aliases` must be a list" in out diff --git a/tests/unit/stacklets/test_pipeline.py b/tests/unit/stacklets/test_pipeline.py index a150a72b..4cb52405 100644 --- a/tests/unit/stacklets/test_pipeline.py +++ b/tests/unit/stacklets/test_pipeline.py @@ -18,6 +18,7 @@ sys.path.insert(0, str(_REPO_ROOT / "lib")) sys.path.insert(0, str(_REPO_ROOT / "stacklets" / "docs" / "bot")) +from stack.people import Person # noqa: E402 from pipeline import ( # noqa: E402 LLMModelNotFoundError, LLMTimeoutError, @@ -635,6 +636,64 @@ async def test_unknown_person_not_created(self, seeded_paperless): # ── Fresh-reprocess semantics ───────────────────────────────────────────── +class TestEnrichWithThePersonFiles: + """With person files, people are filed by who they are, not by spelling. + + The files declare everyone the family knows, accounts or not, with + the names the family uses. A name resolves through those; a person + who has no `Person:` tag yet (no account, so none was seeded) gets + one the first time a document names them. Without the files nothing + changes. + """ + + PEOPLE = [ + Person("homer", "Homer Simpson", account="homer"), + Person("marge", "Marge Simpson", account="marge"), + Person("bart", "Bart Simpson", ("Bartholomew",), account="bart"), + Person("maggie", "Maggie Simpson", ("Margaret",)), + ] + + @pytest.mark.asyncio + async def test_an_alias_files_under_the_persons_tag(self, seeded_paperless): + classifier = StubClassifier(payload={"title": "t", "persons": ["Bartholomew"]}) + result = await enrich_document( + paperless=seeded_paperless, classifier=classifier, doc=_doc(), + people=self.PEOPLE, + ) + assert result.resolved_persons == ["Bart"] + + @pytest.mark.asyncio + async def test_a_person_without_an_account_gets_a_tag_when_first_named( + self, seeded_paperless): + """Maggie has no account and so no seeded tag; she is still family.""" + classifier = StubClassifier(payload={"title": "t", "persons": ["Margaret"]}) + result = await enrich_document( + paperless=seeded_paperless, classifier=classifier, doc=_doc(), + people=self.PEOPLE, + ) + assert result.resolved_persons == ["Maggie"] + assert ("Person: Maggie", "#2196f3") in seeded_paperless.created_tags + assert 'tag "Person: Maggie"' in result.created_new + + @pytest.mark.asyncio + async def test_a_name_nobody_answers_to_gets_no_person_tag(self, seeded_paperless): + classifier = StubClassifier(payload={"title": "t", "persons": ["Simpson"]}) + result = await enrich_document( + paperless=seeded_paperless, classifier=classifier, doc=_doc(), + people=self.PEOPLE, + ) + assert result.resolved_persons == [] + assert not seeded_paperless.created_tags + + @pytest.mark.asyncio + async def test_without_person_files_no_tag_is_created(self, seeded_paperless): + classifier = StubClassifier(payload={"title": "t", "persons": ["Margaret"]}) + await enrich_document( + paperless=seeded_paperless, classifier=classifier, doc=_doc(), + ) + assert not seeded_paperless.created_tags + + class TestEnrichFreshReprocess: """enrich_document treats each run as a full fresh classification: prior tags are dropped, prior document_type is overwritten.""" From d22e0a3d898697dfbd3ba8d34fda95fb236fe3d7 Mon Sep 17 00:00:00 2001 From: Arthur <hello@arthware.dev> Date: Fri, 9 Oct 2026 13:02:33 +0200 Subject: [PATCH 4/6] feat(diary): listen for every person the family declared Transcription and the diary cards took the family from `FAMILY_NAMES`, the accounts' names, plus the wiki's person pages. A baby without an account was missing from the transcription vocabulary, and a person page the old wiki roster made for a folder counted as family. With person files, the vocabulary, the transcript correction pass and the card people are the declared people and every name they go by; person pages count only for declared people and add the spellings documents used. Cards list people by first name, as their `Person:` tags do. Until the files exist, `FAMILY_NAMES` stands in as before. Closes: #154 --- docs/admin-guide.md | 2 +- stacklets/memory/bot/cli/diary.py | 53 ++++++++++++- .../stacklets/test_memory_diary_people.py | 79 ++++++++++++++++++- 3 files changed, 127 insertions(+), 7 deletions(-) diff --git a/docs/admin-guide.md b/docs/admin-guide.md index 99ba4627..228ccfac 100644 --- a/docs/admin-guide.md +++ b/docs/admin-guide.md @@ -579,7 +579,7 @@ account: bart --- ``` -Each `./stack up memory` adds a file for every account in `users.toml` that has none, so an update reaches an existing install with a restart of the memory stacklet. An existing file is never rewritten: it is the family's. Add someone without an account, such as a baby, by adding a file with `type`, `title` and any `aliases`, and add the names a person goes by to their file. A person with an account is a member of the household. The wiki gives every person a page under the names their file gives, and its home page lists the members. A file the wiki cannot read is named in the curator's log (`docker logs stack-memory-curator`) and left out, and no page is removed until it reads again. A change to any file rebuilds the person pages within a few minutes, or right away with `./stack memory wiki update --home --members`. Edit the files in Forgejo, in Obsidian, or with `./stack memory write family/people/<id>.md --by <you> --from <file>`. +Each `./stack up memory` adds a file for every account in `users.toml` that has none, so an update reaches an existing install with a restart of the memory stacklet. An existing file is never rewritten: it is the family's. Add someone without an account, such as a baby, by adding a file with `type`, `title` and any `aliases`, and add the names a person goes by to their file. A person with an account is a member of the household. The wiki gives every person a page under the names their file gives, and its home page lists the members. The archivist files a document under a person by any of their names ("Margaret" on a birth certificate is Maggie) and creates a `Person:` tag for a person without an account the first time a document names them. Transcription and the diary listen for every name in the files. A file the wiki cannot read is named in the curator's log (`docker logs stack-memory-curator`) and left out, and no page is removed until it reads again. A change to any file rebuilds the person pages within a few minutes, or right away with `./stack memory wiki update --home --members`. Edit the files in Forgejo, in Obsidian, or with `./stack memory write family/people/<id>.md --by <you> --from <file>`. **The family diary.** Everything posted in `#Memories` becomes a diary entry. Each night the curator transcribes the new recordings, has the AI read each entry (a title, a summary, the people, topics from the ontology, facts), writes one card per entry to the vault under `family/diary/entries/`, and rebuilds the diary pages in the wiki from those cards. At 07:30 local time the archivist posts each new card as a quiet notice in the thread under its memory; the first time, one notice in the room says how to correct a card, and after two weeks without a new memory one notice invites the next, once per quiet stretch (`diary_job_at` in the archivist's `bot.toml` sets the time; it catches up after a night the Mac slept through). A family member corrects a card by replying in its thread, typed or spoken; the correction is committed to the vault under their name, and a corrected or hand-edited card is never overwritten by a later run. The archivist says nothing else in `#Memories`. The user guide has the family's side: [The Memories Room](user-guide.md#the-memories-room). diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 70d37b44..03a50bed 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -81,12 +81,14 @@ from stack.forgejo import FileChange, ForgejoClient, ForgejoError # noqa: E402 from stack.ai import transcripts # noqa: E402 from stack import media # noqa: E402 +from stack.people import Person as DeclaredPerson, find_people # noqa: E402 from memory.lib import ( # noqa: E402 BOT_EMAIL, BOT_USERNAME, REPO_NAME, REPO_OWNER, + Person, get_ontology, load_persons_from_vault, persons_prompt_section, @@ -290,15 +292,55 @@ def _length(ms: int | None) -> str: return f"{total // 60}:{total % 60:02d}" +def _declared() -> list[DeclaredPerson] | None: + """The family's person files; None until `stack up memory` wrote them. + + A file that does not read is logged with its path and left out. + """ + return find_people( + (os.environ.get("MEMORY_VAULT_DIR"), os.environ.get("BRAIN_REPO_DIR")), + os.environ.get("SHARED_BUCKET") or "family", + report=lambda problem: _err(f"person file skipped: {problem}"), + ) + + def _household_people() -> list[str]: - """The household's names: everyone in users.toml, then the wiki's person pages. + """The household's names: the person files, then the wiki's person pages. - The pages carry the family's own spelling of each name and its - variants. Best-effort: no brain means the users.toml names alone. + The person files declare everyone the family knows, a baby without + an account too, with every name the family uses for them. Until they + exist the accounts stand in (FAMILY_NAMES, from users.toml). The + pages add the spellings documents used. Best-effort: no brain means + the declared names alone. """ brain = Path(os.environ.get("BRAIN_REPO_DIR", "")) pages = [_frontmatter(about) for about in sorted(brain.glob("*/about.md"))] if brain.is_dir() else [] - return diary.household(os.environ.get("FAMILY_NAMES", ""), pages) + declared = _declared() + if declared is None: + return diary.household(os.environ.get("FAMILY_NAMES", ""), pages) + ids = {p.id for p in declared} + names = ",".join(n for p in declared for n in p.all_names()) + return diary.household(names, [p for p in pages if p.get("slug") in ids]) + + +def _declared_persons(declared: list[DeclaredPerson], learned: list[Person]) -> list[Person]: + """The person files as the classifier's persons: first name canonical. + + The first name is the name cards carry and the `Person:` tag the + archivist files under, so a card about "Margaret" lists Maggie the + same way a document about her is tagged. The full name, the + family's aliases and the spellings documents used are the synonyms. + """ + by_slug = {p.slug: p for p in learned} + persons = [] + for d in declared: + names = [n for n in d.all_names() if n != d.first_name] + seen = by_slug.get(d.id) + for n in seen.all_known_names() if seen else []: + if n.lower() not in {x.lower() for x in [d.first_name, *names]}: + names.append(n) + persons.append(Person(canonical=d.first_name, slug=d.id, synonyms=names)) + return persons def _household_vocabulary() -> str: @@ -1314,6 +1356,9 @@ def _vocabulary(bucket: str): # Person pages are generated, so they live in the brain, not the vault. brain_dir = Path(os.environ.get("BRAIN_REPO_DIR", "")) persons = load_persons_from_vault(brain_dir, bucket) if brain_dir.is_dir() else [] + declared = _declared() + if declared is not None: + persons = _declared_persons(declared, persons) people = diary.card_people(persons, _household_people()) # The pages describe the people they know; the rest of the household # joins the list by name, so the model can still say a note is about them. diff --git a/tests/unit/stacklets/test_memory_diary_people.py b/tests/unit/stacklets/test_memory_diary_people.py index 39f83c03..d943477c 100644 --- a/tests/unit/stacklets/test_memory_diary_people.py +++ b/tests/unit/stacklets/test_memory_diary_people.py @@ -3,8 +3,9 @@ Whisper decodes a voice note against the family's names, and the diary cards list the people a note is about from the same list. A name missing here is misheard ("Maggie" as "Eggie") and left off the card. So the list -starts from everyone in users.toml, rendered into FAMILY_NAMES, and the -wiki's person pages add the family's own spellings and nicknames. +starts from the person files (`family/people/`), or from everyone in +users.toml (FAMILY_NAMES) until they exist, and the wiki's person pages +add the spellings documents used. """ from __future__ import annotations @@ -56,3 +57,77 @@ def test_a_member_without_a_page_is_still_someone_a_card_can_name(self): def test_before_any_page_the_household_alone(self): assert diary.card_people([], ["Homer", "Bart"]) == {"homer": "Homer", "bart": "Bart"} + + +# ── With person files ────────────────────────────────────────────── + +def _diary_command(): + """Load `bot/cli/diary.py` as part of its package, under a name of its own. + + Several stacklets ship a `cli` package; importing this one as `cli` + would shadow theirs for every later test. + """ + import importlib + import importlib.util + # In the bot-runner, `voice` and the framework sit at /app. + sys.path.insert(0, str(_REPO_ROOT / "stacklets" / "core" / "bot-runner")) + sys.path.insert(0, str(_REPO_ROOT / "stacklets")) + name = "memory_bot_cli" + if name not in sys.modules: + pkg_dir = _REPO_ROOT / "stacklets" / "memory" / "bot" / "cli" + spec = importlib.util.spec_from_file_location( + name, pkg_dir / "__init__.py", submodule_search_locations=[str(pkg_dir)]) + package = importlib.util.module_from_spec(spec) + sys.modules[name] = package + spec.loader.exec_module(package) + return importlib.import_module(f"{name}.diary") + + +HOMER = "---\ntype: person\ntitle: Homer Simpson\naccount: homer\n---\n" +MAGGIE = "---\ntype: person\ntitle: Maggie Simpson\naliases:\n - Margaret\n---\n" + + +class TestThePersonFilesDecideWhoTheDiaryKnows: + """Once person files exist, the diary listens for everyone they declare. + + The accounts (FAMILY_NAMES) only stand in until then. Maggie has no + account; her file makes her known, and her aliases are names whisper + should expect and a card can use. + """ + + @staticmethod + def _instance(tmp_path, monkeypatch, people: dict[str, str] | None): + vault, brain = tmp_path / "vault", tmp_path / "brain" + (vault / "family").mkdir(parents=True) + (brain / "media").mkdir(parents=True) + (brain / "media" / "about.md").write_text( + "---\ntitle: Margaret\nslug: media\ntype: person\ncanonical: Margaret\n---\n") + for person_id, text in (people or {}).items(): + (vault / "family" / "people").mkdir(parents=True, exist_ok=True) + (vault / "family" / "people" / f"{person_id}.md").write_text(text) + monkeypatch.setenv("MEMORY_VAULT_DIR", str(vault)) + monkeypatch.setenv("BRAIN_REPO_DIR", str(brain)) + monkeypatch.setenv("SHARED_BUCKET", "family") + monkeypatch.setenv("FAMILY_NAMES", "Homer Simpson,Marge Simpson") + return _diary_command() + + def test_before_the_files_the_accounts_and_pages_as_before(self, tmp_path, monkeypatch): + cli = self._instance(tmp_path, monkeypatch, None) + assert cli._household_people() == ["Homer Simpson", "Marge Simpson", "Margaret"] + + def test_with_files_every_person_and_every_name_they_go_by(self, tmp_path, monkeypatch): + cli = self._instance(tmp_path, monkeypatch, {"homer": HOMER, "maggie": MAGGIE}) + assert cli._household_people() == [ + "Homer Simpson", "Homer", "Maggie Simpson", "Maggie", "Margaret"] + + def test_a_page_left_over_for_a_folder_is_not_family(self, tmp_path, monkeypatch): + """The old roster made `media/` a person page; it no longer counts.""" + cli = self._instance(tmp_path, monkeypatch, {"homer": HOMER}) + assert cli._household_people() == ["Homer Simpson", "Homer"] + + def test_a_card_about_margaret_is_about_maggie(self, tmp_path, monkeypatch): + cli = self._instance(tmp_path, monkeypatch, {"homer": HOMER, "maggie": MAGGIE}) + _ontology, people, section, _lang = cli._vocabulary("family") + assert people["margaret"] == "Maggie" + assert people["maggie simpson"] == "Maggie" + assert " - Maggie (Maggie Simpson, Margaret)" in section From 0b466fcf6ad76fac58f5d08dbabda6d46bc5d473 Mon Sep 17 00:00:00 2001 From: Arthur <hello@arthware.dev> Date: Fri, 9 Oct 2026 11:11:22 +0200 Subject: [PATCH 5/6] feat(wiki): show the other names a person goes by A member's names from the household page reached their page's frontmatter, but nothing showed them. The page header now adds "Also known as" with the other names next to the date: Bart's page says "Also known as Bartholomew". A name made only of words in the title ("Maggie" under "Maggie Simpson") is left out. Sender pages show their aliases the same way. --- .../memory/quartz/components/familyModel.ts | 26 +++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/stacklets/memory/quartz/components/familyModel.ts b/stacklets/memory/quartz/components/familyModel.ts index fecf83dd..f3ad2906 100644 --- a/stacklets/memory/quartz/components/familyModel.ts +++ b/stacklets/memory/quartz/components/familyModel.ts @@ -63,6 +63,7 @@ const LABELS = { topic: "Topic", sender: "Sender", filedBy: "filed by", + alsoKnownAs: "Also known as", updated: "updated", familyWiki: "family wiki", welcome: "Everything the family has kept: documents, notes, the diary, and the people they belong to.", @@ -106,6 +107,7 @@ const LABELS = { topic: "Thema", sender: "Absender", filedBy: "abgelegt von", + alsoKnownAs: "Auch bekannt als", updated: "aktualisiert", familyWiki: "Familienwiki", welcome: "Alles, was die Familie aufbewahrt: Dokumente, Notizen, das Tagebuch und die Menschen, zu denen sie gehören.", @@ -295,6 +297,26 @@ interface Topic { const fm = (f: QuartzPluginData) => (f.frontmatter ?? {}) as Record<string, any> const asList = (v: unknown): string[] => (Array.isArray(v) ? v.map(String) : v ? [String(v)] : []) +// The other names a person or sender goes by: `synonyms` on a person page +// (the household page's aliases, then what documents called them), +// `aliases` on a correspondent. A name made only of words already in the +// title says nothing new ("Maggie" under "Maggie Simpson") and is left out. +const otherNames = (m: Record<string, any>): string[] => { + const title = String(m.title ?? m.canonical ?? "").toLowerCase() + const titleWords = new Set(title.split(/\s+/).filter(Boolean)) + const seen = new Set<string>() + return [...asList(m.synonyms), ...asList(m.aliases)].filter((name) => { + const key = name.trim().toLowerCase() + if (!key || seen.has(key)) return false + seen.add(key) + return !key.split(/\s+/).every((w) => titleWords.has(w)) + }) +} +const alsoKnownAs = (m: Record<string, any>): string[] => { + const names = otherNames(m) + return names.length ? [`${L.alsoKnownAs} ${names.join(", ")}`] : [] +} + function parseDate(v: unknown): Date | undefined { if (!v) return undefined const d = new Date(String(v)) @@ -726,8 +748,8 @@ function build(ctx: BuildCtx, allFiles: QuartzPluginData[]): Model { } } const updated = f?.dates?.modified - if (m.type === "person") return { kicker: [L.person], meta: [], date: updated, chips: [] } - if (m.type === "correspondent") return { kicker: [L.sender], meta: [], date: updated, chips: [] } + if (m.type === "person") return { kicker: [L.person], meta: alsoKnownAs(m), date: updated, chips: [] } + if (m.type === "correspondent") return { kicker: [L.sender], meta: alsoKnownAs(m), date: updated, chips: [] } if (m.type === "topic") { const t = [...topics.values()].find((x) => x.slug === slug) const area = onto.areas.find((a) => a.id === t?.area) From 580c5196ea25d647f1aef166f7f5286bc91c71ee Mon Sep 17 00:00:00 2001 From: Arthur <hello@arthware.dev> Date: Thu, 8 Oct 2026 23:09:51 +0200 Subject: [PATCH 6/6] test(diary): add a memories corpus that names people over 30 months The base memories corpus covers four weeks and tests dating and threading. `spec.people.en.yaml` spreads 47 memories from March 2024 to September 2026 and names the people around the Simpsons the way a real room does: grandparents rarely but for years, a best friend often, a teacher for one school year, a doctor for one month, a bully often without being close. Its `people` block is an answer key of who each person is to the family, written independently of how often the corpus names them, and every item lists who it mentions. `generate.py --corpus` and `ingest.py --corpus` render and replay a named spec next to the locale corpora. --- tools/family-memories/generate.py | 8 +- tools/family-memories/ingest.py | 5 +- tools/family-memories/spec.people.en.yaml | 637 ++++++++++++++++++++++ 3 files changed, 648 insertions(+), 2 deletions(-) create mode 100644 tools/family-memories/spec.people.en.yaml diff --git a/tools/family-memories/generate.py b/tools/family-memories/generate.py index 8e530cba..fd14e9c0 100644 --- a/tools/family-memories/generate.py +++ b/tools/family-memories/generate.py @@ -177,8 +177,14 @@ def main() -> None: ap.add_argument("--only", default=None, help="substring filter on item id") ap.add_argument("--locale", choices=["de", "en"], default=None, help="render one locale (default: all)") + ap.add_argument("--corpus", default=None, + help="render spec.<corpus>.yaml into out/<corpus>/ " + "instead of the locale specs, e.g. people.en") args = ap.parse_args() - locales = [args.locale] if args.locale else ["de", "en"] + if args.corpus: + locales = [args.corpus] + else: + locales = [args.locale] if args.locale else ["de", "en"] for locale in locales: render_locale(locale, args) diff --git a/tools/family-memories/ingest.py b/tools/family-memories/ingest.py index ba549597..4a28b642 100644 --- a/tools/family-memories/ingest.py +++ b/tools/family-memories/ingest.py @@ -157,12 +157,15 @@ def main(): ap.add_argument("--login", action="append", required=True, metavar="USER:PASSWORD") ap.add_argument("--locale", choices=["de", "en"], default="de") + ap.add_argument("--corpus", default=None, + help="replay out/<corpus>/ (from generate.py --corpus) " + "instead of out/<locale>/") ap.add_argument("--delay", type=float, default=2.0, help="seconds between non-burst items") ap.add_argument("--force-i-know", action="store_true") args = ap.parse_args() global OUT - OUT = OUT / args.locale + OUT = OUT / (args.corpus or args.locale) household = configured_household(REPO_ROOT) if not args.force_i_know and \ diff --git a/tools/family-memories/spec.people.en.yaml b/tools/family-memories/spec.people.en.yaml new file mode 100644 index 00000000..30ef8924 --- /dev/null +++ b/tools/family-memories/spec.people.en.yaml @@ -0,0 +1,637 @@ +# spec.people.en.yaml: two and a half years of a memories room, for +# measuring how the people around a family show up in it. +# +# The base corpus (spec.en.yaml) covers four weeks and exercises dating +# and threading. This one spreads ordinary memories from March 2024 to +# September 2026 and lets the people outside the household come and go +# the way they do in a real room: grandparents rarely but for years, a +# best friend all the time, a teacher for one school year, a doctor for +# one month, a bully often without being close. +# +# Render and replay: +# FAMSTACK_TTS_URL=https://<speech-host>/v1/audio/speech \ +# python tools/family-memories/generate.py --corpus people.en +# python tools/family-memories/ingest.py --corpus people.en ... +# +# Every item states its date in its text, with the year, because a +# replay lands on the day it runs (see README). Compile with the default +# burst window: items go out --delay apart and none of them is a burst. +# +# Ground truth +# ------------ +# `people` is the answer key, written before any pipeline ran. `tier` +# follows who these people are to the Simpsons in the show, not how +# often this corpus names them, so a counting rule can be scored against +# it rather than confirmed by it: +# family the household +# broader_family relatives outside the household +# close people the family has a lasting personal bond with +# other everyone else, however often they come up +# `names` lists the variants the corpus uses for each person. +# +# `mentions` on each item lists the people it refers to outside the +# household, by key, including references without a name ("my sisters"). + +voices: + marge: alloy + homer: alloy + +people: + homer: {tier: family, names: [Homer, Homie, Dad]} + marge: {tier: family, names: [Marge, Mom]} + bart: {tier: family, names: [Bart]} + lisa: {tier: family, names: [Lisa]} + maggie: {tier: family, names: [Maggie]} + abe: {tier: broader_family, names: [Grampa, Abe, Grandpa Simpson, my dad]} + jacqueline: {tier: broader_family, names: [Grandma Bouvier, Mom, my mother]} + patty: {tier: broader_family, names: [Patty, Aunt Patty]} + selma: {tier: broader_family, names: [Selma, Aunt Selma]} + herb: {tier: broader_family, names: [Herb]} + ling: {tier: broader_family, names: [Ling]} + milhouse: {tier: close, names: [Milhouse]} + ned: {tier: close, names: [Ned, Flanders, Ned Flanders]} + lenny: {tier: close, names: [Lenny]} + carl: {tier: close, names: [Carl]} + moe: {tier: close, names: [Moe]} + janey: {tier: close, names: [Janey]} + krabappel: {tier: other, names: [Mrs. Krabappel, Edna Krabappel]} + hoover: {tier: other, names: [Ms. Hoover]} + skinner: {tier: other, names: [Skinner, Principal Skinner]} + hibbert: {tier: other, names: [Dr. Hibbert, Hibbert]} + nelson: {tier: other, names: [Nelson]} + largo: {tier: other, names: [Mr. Largo]} + burns: {tier: other, names: [Mr. Burns]} + krusty: {tier: other, names: [Krusty]} + apu: {tier: other, names: [Apu]} + lovejoy: {tier: other, names: [Reverend Lovejoy]} + +items: + - id: people-2024-03-grampa-dinner + true_date: 2024-03-09 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [abe] + text: > + Saturday, March 9th 2024. Grampa came for dinner and told the + story about his war years again, the one with the hat. Bart + pretended to fall asleep in his mashed potatoes, but Lisa asked + him three questions and he lit up like a Christmas tree. + + - id: people-2024-03-ramp + true_date: 2024-03-23 + date_source: spoken + pattern: people-mention + kind: voice + sender: homer + mentions: [milhouse] + text: > + Hi Bart, today is March twenty-third, 2024. You and Milhouse built + a ramp out of the old garage door. Milhouse went first, because + you said it was an honor. His glasses are fine. The garage door is + not. Your mother doesn't know yet. + + - id: people-2024-04-aunts + true_date: 2024-04-14 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [patty, selma] + text: > + April 14th 2024. Patty and Selma came over for Sunday lunch and + brought their iguana. Homer spent the whole afternoon in the + garage. Lisa played her new piece for her aunts, and Aunt Selma + said it was the first thing all year that didn't bore her. + + - id: people-2024-05-sleepover + true_date: 2024-05-05 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [janey] + text: > + May 5th 2024. Lisa's first sleepover here, with Janey. They + giggled until two in the morning and made a newspaper about our + street. Headline: Local Dad Snores. + + - id: people-2024-05-hedge + true_date: 2024-05-19 + date_source: spoken + pattern: people-mention + kind: voice + sender: homer + mentions: [ned] + text: > + Hey Lisa, today is May nineteenth, 2024. Ned Flanders lent me his + hedge trimmer again. Fourth time this year. You made me bring it + back before dinner, and he gave me a lemonade for it. That man is + impossible to stay mad at. + + - id: people-2024-06-grandma + true_date: 2024-06-08 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [jacqueline] + text: > + June 8th 2024. Mom's birthday. Grandma Bouvier taught Lisa how to + play bridge and beat her four times in a row. Maggie fell asleep + on her lap and she didn't move for an hour. + + - id: people-2024-06-bowling + true_date: 2024-06-22 + date_source: written + pattern: people-mention + kind: text + sender: homer + mentions: [lenny, carl] + text: > + June 22nd 2024. Took Bart bowling with Lenny and Carl for the + first time. He bowled a 63 and Carl let him keep the score sheet. + Lenny says he has my form. I'm choosing to take that as a + compliment. + + - id: people-2024-07-fireworks + true_date: 2024-07-04 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [ned] + text: > + July 4th 2024. Fireworks at the Flanders' barbecue next door. Ned + let Bart light one sparkler and Bart behaved for a whole evening. + Homer ate eleven hot dogs. + + - id: people-2024-07-retirement-castle + true_date: 2024-07-20 + date_source: spoken + pattern: people-mention + kind: voice + sender: marge + mentions: [abe] + text: > + Hi Maggie, today is July twentieth, 2024. We visited Grampa at the + retirement home, and he held you the whole afternoon. You fell + asleep on him and then he fell asleep too, and the nurses tiptoed + around the two of you. + + - id: people-2024-08-game-night + true_date: 2024-08-10 + date_source: written + pattern: people-mention + kind: text + sender: homer + mentions: [moe] + text: > + August 10th 2024. Moe closed the bar early and came over to watch + the game on our couch. He brought pretzels and stayed until Marge + started vacuuming around him. + + - id: people-2024-08-squishee + true_date: 2024-08-24 + date_source: written + pattern: people-mention + kind: text + sender: homer + mentions: [apu] + text: > + August 24th 2024. Hottest day of the year. Apu gave Maggie a free + Squishee at the Kwik-E-Mart. Her whole face was blue until + bedtime. + + - id: people-2024-09-fourth-grade + true_date: 2024-09-03 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [krabappel] + text: > + September 3rd 2024. First day of fourth grade for Bart. His new + teacher is Mrs. Krabappel. She called at noon. On the first day. + + - id: people-2024-09-lunch-money + true_date: 2024-09-14 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [nelson] + text: > + September 14th 2024. Nelson took Bart's lunch money again. Bart + says it doesn't matter and that he has a plan. I am worried about + both. + + - id: people-2024-10-robot + true_date: 2024-10-05 + date_source: spoken + pattern: people-mention + kind: voice + sender: homer + mentions: [milhouse] + text: > + Hi Bart, today is October fifth, 2024. Milhouse slept over and you + two tried to build a robot out of the vacuum cleaner. It works + better now than before. I don't know how, and I don't want to + know. + + - id: people-2024-10-conference + true_date: 2024-10-26 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [krabappel] + text: > + October 26th 2024. Parent-teacher conference with Edna Krabappel. + She says Bart is clever but bored, and that he corrected her + spelling on the board. She didn't sound angry. She sounded tired. + + - id: people-2024-11-saxophone + true_date: 2024-11-02 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [largo] + text: > + November 2nd 2024. Mr. Largo moved Lisa up to first saxophone. She + practiced on the porch until the neighbors applauded. + + - id: people-2024-11-thanksgiving + true_date: 2024-11-28 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [abe, patty, selma, jacqueline] + text: > + Thanksgiving, November 28th 2024. Everybody at our table: Grampa, + Patty and Selma, and Mom. Homer carved the turkey without injuring + anyone. Grampa fell asleep before dessert and Patty put a napkin + over his face. + + - id: people-2024-12-bonus + true_date: 2024-12-15 + date_source: written + pattern: people-mention + kind: text + sender: homer + mentions: [burns, lenny] + text: > + December 15th 2024. Mr. Burns cancelled the Christmas bonus again. + Lenny says next year for sure. We still got the tree, the kids + decorated it, and Bart put a picture of Burns at the top. + + - id: people-2024-12-christmas-eve + true_date: 2024-12-24 + date_source: spoken + pattern: people-mention + kind: voice + sender: marge + mentions: [ned, jacqueline] + text: > + Hi kids, today is Christmas Eve, 2024. Ned came over with a plate + of cookies shaped like the three wise men, and Grandma Bouvier + read you the Christmas story. Bart, you didn't interrupt once. I + want you to know I noticed. + + - id: people-2025-01-snow-fort + true_date: 2025-01-11 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [milhouse, nelson] + text: > + January 11th 2025. Snow day. Bart and Milhouse built a snow fort in + the front yard. Nelson knocked it over, and then, to everyone's + surprise, helped them build it again. + + - id: people-2025-02-arm + true_date: 2025-02-03 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [hibbert] + text: > + February 3rd 2025. Bart broke his arm falling off the garage roof. + Dr. Hibbert set it at the hospital and laughed that laugh of his + the whole time. Bart was brave. Homer fainted. + + - id: people-2025-02-xray + true_date: 2025-02-10 + date_source: written + pattern: people-mention + kind: text + sender: homer + mentions: [hibbert, krabappel] + text: > + February 10th 2025. Back at Dr. Hibbert's for the X-ray. The bone + is healing. Bart got the whole school to sign the cast, even Mrs. + Krabappel, who wrote "finally sitting still". + + - id: people-2025-02-cast-check + true_date: 2025-02-24 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [hibbert] + text: > + February 24th 2025. Cast check with Hibbert. Two more weeks. Bart + has been using it as a drum. + + - id: people-2025-03-cast-off + true_date: 2025-03-03 + date_source: spoken + pattern: people-mention + kind: voice + sender: homer + mentions: [hibbert] + text: > + Hey Bart, today is March third, 2025. Dr. Hibbert took the cast + off this morning. Your arm looks like a noodle and smells worse, + but you can climb things again. Please don't climb things again. + + - id: people-2025-03-science-fair + true_date: 2025-03-22 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [janey] + text: > + March 22nd 2025. School science fair. Lisa and Janey built a model + of the water cycle that actually rained. Second place, behind a + volcano. Lisa says the judges were biased toward explosions. + + - id: people-2025-04-birthday-party + true_date: 2025-04-01 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [krusty, milhouse, nelson] + text: > + April 1st 2025, Bart's birthday. Homer booked Krusty for the party. + He came an hour late and left after twenty minutes, but Bart will + talk about it for years. Milhouse cried when the balloon animal + popped. Nelson came too, and brought a present. + + - id: people-2025-04-dads-birthday + true_date: 2025-04-12 + date_source: written + pattern: people-mention + kind: text + sender: homer + mentions: [abe] + text: > + April 12th 2025. My dad's birthday. We took Abe out of the + retirement castle for lunch at the diner. He told the waitress + three times that he was 29. She gave him a free pie. + + - id: people-2025-05-patty + true_date: 2025-05-18 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [patty, selma] + text: > + May 18th 2025. Patty came alone this time, Selma has a cold. She + and Lisa watched MacGyver all afternoon and quoted it at dinner. + + - id: people-2025-06-last-day + true_date: 2025-06-13 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [krabappel] + text: > + June 13th 2025. Last day of school. Bart gave Mrs. Krabappel a + card he made himself. She cried a little, I think. He says she + had something in her eye. + + - id: people-2025-06-summer-concert + true_date: 2025-06-28 + date_source: spoken + pattern: people-mention + kind: voice + sender: marge + mentions: [largo] + text: > + Hi Lisa, today is June twenty-eighth, 2025. Your summer concert + with Mr. Largo's band. You played your solo a little faster than + he wanted, and it was the best part of the evening. + + - id: people-2025-07-camping + true_date: 2025-07-19 + date_source: written + pattern: people-mention + kind: text + sender: homer + mentions: [ned] + text: > + July 19th 2025. Camping at Lake Springfield with Ned and his boys. + Ned brought a tent for each of us, labeled. Ours collapsed anyway. + We slept in his. + + - id: people-2025-08-roof + true_date: 2025-08-16 + date_source: written + pattern: people-mention + kind: text + sender: homer + mentions: [lenny, carl, moe] + text: > + August 16th 2025. Lenny and Carl helped me fix the roof, the part + Bart fell off. Moe sent a case of beer and a note saying he + doesn't do ladders. + + - id: people-2025-09-school + true_date: 2025-09-02 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [hoover, milhouse] + text: > + September 2nd 2025. School again. Bart has Ms. Hoover this year. + Milhouse is in his class again, so I give it a week. + + - id: people-2025-09-herb + true_date: 2025-09-20 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [herb] + text: > + September 20th 2025. Herb, Homer's half-brother, called for the + first time in years. They talked for an hour. Homer was quiet + afterwards, in a good way. + + - id: people-2025-10-halloween + true_date: 2025-10-31 + date_source: spoken + pattern: people-mention + kind: voice + sender: homer + mentions: [ned] + text: > + Hi Maggie, today is Halloween, 2025. You went as a pacifier. Ned + gave out apples again and Bart egged nobody, which I count as + growth. + + - id: people-2025-11-thanksgiving + true_date: 2025-11-27 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [abe, patty, selma, jacqueline] + text: > + Thanksgiving, November 27th 2025. Grampa, my sisters and Mom all + came. Grampa and Mom argued about the gravy and both lost. Maggie + said "turkey" for the first time, or something close. + + - id: people-2025-12-carols + true_date: 2025-12-24 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [jacqueline, janey, lovejoy] + text: > + Christmas Eve 2025. Mom stayed over. Lisa and Janey sang carols at + church, and Reverend Lovejoy let them do a second verse. Homer + stayed awake for all of it. + + - id: people-2026-01-detention + true_date: 2026-01-17 + date_source: written + pattern: people-mention + kind: text + sender: homer + mentions: [milhouse, skinner] + text: > + January 17th 2026. Bart and Milhouse got detention together for + the thing with the school bell. Skinner called both families. I + told Bart I was disappointed and then asked him how they did it. + + - id: people-2026-02-valentine + true_date: 2026-02-14 + date_source: written + pattern: people-mention + kind: text + sender: homer + mentions: [moe] + text: > + Valentine's Day 2026. Moe gave me advice for Marge. I ignored all + of it and made her pancakes shaped like hearts. Mostly like + hearts. + + - id: people-2026-03-war-stories + true_date: 2026-03-01 + date_source: spoken + pattern: people-mention + kind: voice + sender: marge + mentions: [abe] + text: > + Hi Bart, today is March first, 2026. Grampa told you about the war + again, and this time you listened all the way to the end. He told + me afterwards that you are the only one who does. + + - id: people-2026-04-janeys + true_date: 2026-04-25 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [janey] + text: > + April 25th 2026. Lisa slept over at Janey's for the first time. + She called at nine to say goodnight and at nine thirty to say she + was fine. + + - id: people-2026-05-ling + true_date: 2026-05-09 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [selma, ling] + text: > + May 9th 2026. Selma visited with her daughter Ling. Ling and + Maggie played with the blocks all afternoon without a single + fight, which is more than I can say for their mothers. + + - id: people-2026-06-room + true_date: 2026-06-20 + date_source: written + pattern: people-mention + kind: text + sender: homer + mentions: [abe] + text: > + June 20th 2026. Grandpa Simpson moved to a room with a window on + the garden side. We brought his old armchair. He says it smells + like home, and that home smelled terrible. + + - id: people-2026-07-nelson-dinner + true_date: 2026-07-11 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [nelson] + text: > + July 11th 2026. Nelson came over for dinner. He said please twice + and thank you once. Bart says they are not friends, they just + have an arrangement. + + - id: people-2026-08-barbecue + true_date: 2026-08-08 + date_source: spoken + pattern: people-mention + kind: voice + sender: homer + mentions: [lenny, carl] + text: > + Hey Lisa, today is August eighth, 2026. Lenny and Carl came over + for a barbecue and you explained to them why you don't eat meat. + Carl had a veggie burger. Lenny had three regular ones, out of + respect. + + - id: people-2026-08-milhouse-moves + true_date: 2026-08-29 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [milhouse] + text: > + August 29th 2026. Milhouse's family is moving to Capital City. + Bart went to his room and didn't come down for dinner. Later he + asked if we could visit them in the fall. + + - id: people-2026-09-mom-fall + true_date: 2026-09-05 + date_source: written + pattern: people-mention + kind: text + sender: marge + mentions: [jacqueline] + text: > + September 5th 2026. My mother fell in her kitchen. She's fine, a + bruise and a lot of opinions. Lisa made her a get-well card with + a saxophone on it.