Repository navigation
feat(memory): keep the original file behind a wiki page - #92
Merged
Merged
Conversation
Adds stack.media, the archive that holds the bytes a page describes: one file per event id under <site>/media/<yyyy>/<mm>/, written once, addressed by a link that works from any page. Brain's seeded .gitignore now excludes it, so deleting a message can really delete the file. No caller yet; the capture and diary paths follow.
The archive now writes <id>.json beside every file it keeps: who sent it, from which room, under what name, when, and a checksum. Someone finding the tree years from now can place a file back in its context without famstack. Adds the second copies a page can show - AAC for recordings Safari would otherwise not play, WebP bounded to 1600px so a month of photographs is not a hundred-megabyte page. The original is kept either way, and the sidecar lists what was derived from it.
The diary runs in the curator container (curator.py subprocesses the compiler) and on demand in the bot-runner, and both now need to convert what they archive: an AAC copy of a voice note, a width-bounded copy of a photograph. Neither image had ffmpeg. Without it the pages fall back to the room link, which is what they do today.
The compile now keeps every file the memories room carried and puts it on the page: a voice note plays, a photo and a video show, a document is offered as a download. The link back into the room stays. Files already kept are not fetched again, so only the first compile pays for the room's history. Anything that fails - the homeserver, the disk, a missing ffmpeg - costs that entry its file and nothing else; the page renders as it did before.
An upload into a capture room is archived alongside its entry instead of living only in the homeserver's media store. The entry names the file under the name it was posted as, and shows it when showing it is useful: a picture goes on the page, a document stays a download. The bytes are kept in the archivist, which is the layer that has the room, the sender and the event the upload arrived on. Documents filed to Paperless are not archived - Paperless holds those durably and the entry already names them. Nothing here can stop a filing: without a brain to write into, the entry is the one famstack published before.
Writing the file and converting it can take seconds, and every bot in the stack shares one loop, so the archivist hands that work to a thread. The diary compiler now names each file as it fetches it: the first run over a room with years in it is minutes of downloading and was silent throughout.
A reply-to-correct re-renders the entry from the prior one and never sees the original bytes again, so the archived file would have disappeared from the page on the first correction. The reference is read back off the entry being replaced.
WebP needs libwebp linked into ffmpeg at build time. A build without it produces no derivative at all, so the page embeds the full-size original and the bound the derivative exists to enforce is silently gone. The ffmpeg on this machine is such a build, which is why the end-to-end thumbnail test was gated and skipped rather than run. mjpeg is built into ffmpeg itself, so any build produces the derivative. JPEG at -q:v 3 is roughly a quarter larger than the WebP would have been, which is a smaller cost than a hundred-megabyte page that nothing reports. The test drops its encoder gate and now runs, covering the scale arithmetic end to end.
Quartz globs its content directory through globby with `gitignore: true` (quartz/util/glob.ts), and the Assets emitter uses that same glob. A `media/` line in brain's committed `.gitignore` therefore removed the whole archive from the build: every embed and every link into it would have answered 404, with nothing anywhere reporting it. Verified in the wiki container against the pinned globby: with the rule in `.gitignore` the emitter returns an empty list; with the rule in `.git/info/exclude` it returns the files. Git honours both, globby reads only the first. So the rule moves to `.git/info/exclude`, which also removes the migration problem: nothing is committed, so an upgraded instance needs no seed change. The `media/` line comes back out of BRAIN_SEED_GITIGNORE. The tests now assert through `git check-ignore` rather than by reading a file, and one pins the invariant that broke: the rule must not appear in the tracked ignore file.
Replaces the diary-only media note with the design that was built: one archive for captures and diary entries, why it sits inside brain, why the exclusion is local rather than tracked, the ffmpeg and JPEG decisions, and the sidecar record. dev.md claimed `lib/stack/` imports nothing from pip. It is not true and has not been for a while: `lib/stack/ai/` imports openai and loguru. The invariant that actually holds is about the host CLI path, where ./stack runs on system Python with no venv. The wrong version of this sent an implementation down a path that avoided a dependency nothing forbids. Also records two open items the build surfaced: nothing prunes the archive, and a new recording is fetched twice on its first compile.
`run()` bound the archived-media map to `kept`, then the month-summary
loop rebound the same name to a cached summary string. `pages_for` was
handed that string and rendering died in `_entry_block` on
`'str' object has no attribute 'get'`.
It only fails once a summary is cached. On a first run the loop leaves
`kept` as "", and `"" or {}` yields an empty dict, so the compile
succeeds and every entry silently renders without its media. That is
why the unit suite is green: nothing drives `run()` end to end with a
warm summary cache. Found on the demo rig.
The loop variable is now `cached`.
Found on the demo rig. A 2400px screenshot archived as a 16 KB PNG produced a 1600px JPEG of 64 KB: four times the bytes, and blurred text as well. Flat colour and glyph edges are what PNG compresses best and what JPEG handles worst, and screenshots pasted into notes are a main case for this archive, not an edge one. The derivative exists only to make a page lighter, so one that is not lighter is deleted and the caller shows the original. Audio is exempt: it is transcoded for a codec Safari can play, so a larger file is still the only playable one. The image fixtures split accordingly, because one pattern cannot show both behaviours: a noisy source stands in for a photograph and keeps its derivative, a flat test pattern stands in for a screenshot and loses it.
The archive lived inside brain's working copy. Brain is a projection:
its README says it is generated and rebuilt from source, and it is
re-cloned whenever that happens. The originals cannot be rebuilt from
anything, and they are excluded from git, so git is not their second
copy either. Re-cloning brain for any reason destroyed every original
and left the pages pointing at nothing.
The store moves to {data_dir}/memory/media, named by MEDIA_ARCHIVE_DIR,
and the wiki container mounts it into the rendered tree at
/vault/media. Pages are unchanged: links were already site-rooted and
independent of where the bytes sit, and the mount puts the store where
they already point. Verified on the rig that Quartz serves through a
nested mount and that its watcher sees files added after start.
Docker cannot create that mountpoint, because its parent is a
read-only bind and mkdir fails there, taking the container down before
Quartz runs. A new on_start hook makes the directory on every start,
not once at install, because a re-cloned brain has no empty
directories. The same hook owns brain's exclude rule, which is where it
belonged: the memory stacklet owns brain, the framework just stores
bytes.
Both callers now resolve the store one way, through open_archive().
They previously disagreed, with different behaviour when it was
missing, and the archivist read an env var the memory stacklet owns.
Also declares the archive as a backup source. It had none, and unlike
everything else under {data_dir}/memory it is not a git repo with a
copy in Forgejo, so it had no second copy anywhere.
arthware-dev
added a commit
that referenced
this pull request
Sep 23, 2026
An image copy that is not smaller than its original was renamed into the media archive and deleted again a moment later. The wiki watches the archive, and a file that appears and disappears between two rebuilds can reach Quartz as a deletion of a file it never copied, which fails the rebuild the same way an archive temporary file did. The size is now compared while the copy still has its temporary name, so a rejected copy never appears under its final name. What is kept is unchanged. Fixes: 03e1708 ("feat(memory): keep the original file behind a wiki page (#92)")
arthware-dev
added a commit
that referenced
this pull request
Sep 23, 2026
A screenshot filed from a capture room showed on its wiki page as a broken image, and its File link answered 404, until the wiki container was next recreated. The media archive writes through a temporary `.tmp-*` file beside the target and renames it into place; ffmpeg derivatives are written the same way and take long enough for the wiki's file watcher to report them. Quartz then asks its Assets emitter to copy or unlink a path that is already gone, and the emitter throws. Quartz 4.5.2 keeps every change since startup and replays them on each rebuild, so the failure repeats: the process exits, or later assets never reach the site. Exclude `media/**/.tmp-*` from Quartz's content, which also removes the files from what the watcher reports. Fixes: 03e1708 ("feat(memory): keep the original file behind a wiki page (#92)")
3 tasks
arthware-dev
added a commit
that referenced
this pull request
Sep 25, 2026
On a first `./stack up memory` the brain projection was never cloned. `on_start` creates the wiki's media mountpoint inside `~/famstack-data/memory/brain` before the containers start, and the `family/brain` repo only exists once they are up, so `git clone` met a non-empty directory and refused. The step still printed a success mark, and the curator waited for the projection forever: no wiki page was regenerated and no diary was compiled. The brain clone now turns an existing directory into a working copy in place: tracked files take the repository's version, untracked ones such as the mountpoint stay. Brain is a projection that can be rebuilt, so that is safe there; the vault keeps the strict clone. A failed vault or brain clone is now reported as a warning. Fixes: 03e1708 ("feat(memory): keep the original file behind a wiki page (#92)") Closes: #148
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Classification and filing already worked. The original bytes were never
kept: a page linked an
mxc://URL and the only copy lived in Synapse'smedia store.
Uploads are now archived under
{data_dir}/memory/media/<yyyy>/<mm>/with a JSON record beside each file, and the page describing one embeds
or links it. Images inline, PDFs linked, recordings get a player. The
room link stays as the timeline anchor; it just stops being the only
copy.
The store is its own directory rather than a folder inside brain, and
the wiki container mounts it into the rendered tree at
/vault/media.Brain is a projection that gets re-cloned whenever it is rebuilt; these
bytes cannot be rebuilt from anything. It is also declared as a backup
source, which it needed more than the rest of
{data_dir}/memory,because everything else there is a git repo with a copy in Forgejo.
Two things worth not rediscovering during review:
/vault/mediamountpoint: its parent is aread-only bind,
mkdirfails, and the container exits before Quartzruns.
hooks/on_start.pymakes the directory on every start, becausea re-cloned brain has no empty directories.
.git/info/exclude, not.gitignore.Quartz globs its content directory with
gitignore: true, so atracked rule removes the archive from the build and every link into
it 404s.
ffmpeg goes into both the bot-runner and the curator images, because
the nightly diary runs as a subprocess inside the curator.
Existing diary pages gain their media on the next compile, and months
with no media are left unchanged. Captures filed before this keep only
their
mxc://link; there is no backfill.