diff --git a/docs/agent/dev.md b/docs/agent/dev.md index 542532e8..330447b1 100644 --- a/docs/agent/dev.md +++ b/docs/agent/dev.md @@ -28,7 +28,7 @@ Pre-1.0 stance: we keep backwards compatibility only for critical parts. Everywh ``` famstack/ ├── stack 4-line bash wrapper → lib/stack -├── lib/stack/ CLI core (Python, stdlib only) +├── lib/stack/ CLI core (Python; top level stdlib only) ├── stacklets/ one directory per stacklet (id == dirname) │ ├── core/ always-on (Caddy, Watchtower, bot-runner) │ ├── messages/ Matrix + Element @@ -52,7 +52,7 @@ famstack/ ## Five framework invariants 1. **`.env` is a derived artifact.** Generated on every `stack up` from `stack.toml` + `[env.defaults]` + secrets. Never edit, never commit, never read as source of truth. (See [adr-006](../adr/adr-006-env-as-derived-artifact.md).) -2. **The CLI is stdlib-only.** `lib/stack/` imports nothing from pip. Container code carries its own deps; host-side tooling (tests, hooks) gets deps via `pyproject.toml [project.optional-dependencies] test`. +2. **The host CLI is stdlib-only.** `lib/stack/*.py` imports nothing from pip, because `./stack` runs on the host's system Python with no venv. A subpackage that only ever loads inside a container may use pip deps declared by the images that mount it: `lib/stack/ai/` imports `openai` and `loguru`. Container code carries its own deps; host-side tooling (tests, hooks) gets deps via `pyproject.toml [project.optional-dependencies] test`. 3. **State is derived.** No "enabled stacklets" registry. `docker ps -a` + `~/famstack-data//` decide state. 4. **Convention over configuration.** If a file exists with the documented name (`hooks/on_install.py`, `cli/foo.py`, `caddy.snippet`, `bot/bot.toml`), it is picked up. No registration step. 5. **The `stack` CLI is the sanctioned agent interface.** Bots, hooks, and external automations call `./stack ` - they do not import from `lib/`. Commands have stable exit codes, JSON output, and idempotent semantics. diff --git a/docs/design/brain/memory-pipeline-architecture.md b/docs/design/brain/memory-pipeline-architecture.md index 387563cc..0b2b6896 100644 --- a/docs/design/brain/memory-pipeline-architecture.md +++ b/docs/design/brain/memory-pipeline-architecture.md @@ -174,6 +174,120 @@ model had ignored. | undatable memories | Matrix has no compose-time field; offline recordings carry sync time | spoken-date extraction; honest "unrecoverable" state; recording habit: say the date aloud | | stale caches serving old wording | caches were keyed by content only; prompt changes did not change the keys | every cached artifact stores a fingerprint (hash) of the prompt or pass parameters that produced it; a prompt edit invalidates exactly the affected artifacts on the next compile | +## Media archive + +Built September 2026. Classification and filing already worked; what +was missing was that the original bytes were never retained. A page +linked an `mxc://` URL and the only copy lived in Synapse's media +store. One archive serves three consumers: note and bookmark captures, +diary entries, and anything else that files an upload. + +### Location + +`{data_dir}/memory/media///.`, its own store, +named by `MEDIA_ARCHIVE_DIR`. + +It is deliberately not a folder inside brain. Brain is a projection: +its README says it is generated and rebuilt from source, and it can be +deleted and re-cloned at any time. These bytes cannot be rebuilt from +anything, and they are excluded from git, so git is not their second +copy either. Irreplaceable data does not belong inside a tree that is +documented as disposable. + +The wiki container mounts the store into the rendered tree at +`/vault/media`, so pages and the originals they embed are served from +one site. That mount is what makes the link form work: Quartz emits +only what is under its content root, and a separate origin would need +an absolute URL that is wrong in one of the two deployment modes and is +then baked into pages read for decades. Files keep root-relative links +(`/media/2026/03/.jpg`), which Quartz and Obsidian resolve +identically at any page depth. + +Two consequences worth stating, because neither is obvious from the +compose file: + +- **Docker cannot create the mountpoint.** Its parent is a read-only + bind mount, `mkdir` fails there, and the container exits before + Quartz runs. `hooks/on_start.py` makes `brain/media/` first, on every + start rather than once at install, because a re-cloned brain carries + no empty directories: git does not track them. +- **The mountpoint is excluded from brain's git**, in + `.git/info/exclude` rather than `.gitignore`. Nothing should write + there, but a stale call site that did must not have its bytes + committed into the projection. The local file is the right one + because Quartz globs its content directory with `gitignore: true` + (`quartz/util/glob.ts`), and the `Assets` emitter uses that same + glob, so a tracked rule removes the archive from the build and every + link into it answers 404. Measured in the wiki container against the + pinned globby: + + | Rule in | Assets emitter sees | + |---|---| + | `.gitignore` | `[]` | + | `.git/info/exclude` | the files | + +Git honours both; globby reads only the first. + +The store is declared as a `[[backup.archive]]` source by the memory +stacklet. Every other irreplaceable store in the stack does the same; +this one needs it more, because everything else under `{data_dir}/memory` +is a git repo with a copy in Forgejo and the archive is not. + +### Derivatives + +ffmpeg, installed in both the bot-runner image and the curator image. +Both are needed because `curator.py` runs the nightly diary as a +subprocess inside its own container, while a manual `stack memory +diary` runs in the bot-runner. + +- Audio gets an `.m4a` beside the original. This exists for exactly one + reason: Safari does not decode Opus in Ogg, and Matrix voice messages + are Ogg/Opus. +- Images get a JPEG bounded to 1600px. JPEG rather than WebP because + mjpeg is built into ffmpeg itself while WebP needs libwebp linked in + at build time, and a build without it produces no derivative at all, + so the page falls back to embedding the full-size original. That is + the one outcome the derivative exists to prevent. WebP would be about + a quarter smaller, which does not buy that risk. +- ffmpeg missing or failing is never fatal. The page shows the original, + or the room link. + +### The record beside the file + +Every stored artifact gets `.json` next to it: event id, room, +sender, original filename, mime, size, sha256, capture and store times, +kind, which pipeline wrote it, and the derivatives it owns. JSON rather +than TOML to match the transcript store, which already writes one JSON +per Matrix event. + +The point is that the archive explains itself. Dated ordinary files +beside Markdown pages, readable with no famstack installed. `derived` +exists so a later re-encode replaces its own output instead of orphaning +it. + +Sidecars are excluded from the published site through Quartz's +`ignorePatterns` (`media/**/*.json`). They name the room and the +sender's full mxid, which is worth having on disk and not worth +serving. + +### Rendering + +`![[path]]` in the page, which Quartz v4.5.2 turns into real HTML by +extension: `