diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cd26af43..28d4588d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,5 +1,6 @@ # Tier 1 of the release gate (docs/agent/dev.md, "Releases"): the checks -# that need no Docker and no macOS. Runs on every push/PR; on version +# that need no Docker and no macOS, including that every commit subject +# parses as a changelog entry. Runs on every push/PR; on version # tags it additionally enforces that the tag, the CLI VERSION, and # pyproject.toml all agree. Tiers 2 (integration suite) and 3 (fresh # install, ai stacklet) stay manual — see docs/cleanup-backlog.md. @@ -61,6 +62,21 @@ jobs: print("Versions agree:", ", ".join(f"{s}={v}" for s, v in versions.items())) EOF + - name: Commit convention + # The changelog is generated from these subjects, so one that + # does not parse is an entry missing from the release notes. + # Squash merging makes the PR title the subject that lands, so + # it is checked too. + if: github.event_name == 'pull_request' + env: + PR_TITLE: ${{ github.event.pull_request.title }} + BASE: ${{ github.event.pull_request.base.sha }} + HEAD: ${{ github.event.pull_request.head.sha }} + run: | + git fetch --no-tags --depth=50 origin "$BASE" "$HEAD" 2>/dev/null || true + python3 tools/commit-lint --title "$PR_TITLE" + python3 tools/commit-lint --range "$BASE..$HEAD" + - name: Framework tests run: uv run --extra test pytest tests/framework/ --ignore=tests/framework/test_config_to_container.py -q diff --git a/.gitignore b/.gitignore index 2c637ded..cdc6a04e 100644 --- a/.gitignore +++ b/.gitignore @@ -56,6 +56,11 @@ tests/integration/eval/runs/ .claude/skills/* !.claude/skills/famstack-lsp/ +# Extension stacklets live in ~/-extensions, not here. This is +# the obvious place to try anyway, so keep anything dropped here out of +# git rather than in someone's next commit. +/extensions/ + # Internal development notes impl/ diff --git a/AGENTS.md b/AGENTS.md index 7d840af6..04803e72 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -83,7 +83,7 @@ Apply to every role, every session. 3. **Never `git push` without explicit human approval.** Every push, every branch, every time. 4. **Never commit to `main`.** Feature branches only. 5. **No `Co-Authored-By:` trailers** in commit messages. -6. **Semantic commit prefixes:** `feat:`, `fix:`, `docs:`, `refactor:`, `chore:`, `test:`, `ci:`, `style:`. +6. **Commit subjects are the changelog.** `(): `, then a body that says what and why, in plain technical English. Types, scopes and the `Upgrade:` footer are in [docs/agent/dev.md](docs/agent/dev.md#commits-the-subject-is-the-changelog). 7. **Never read or copy a user's production family vault.** Reference paths only; fabricate test data. 8. **Destructive ops need confirmation.** `stack destroy`, `stack uninstall`, `rm -rf`, `git reset --hard`, force-push. 9. **Announce actions before running them.** No silent long running or integration test runs, scripts, or background commands. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 535394d7..62948270 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -10,9 +10,13 @@ One entry point, [Homebrew](https://brew.sh), which brings the rest: ```bash brew install uv -uv sync --extra test +tools/init-repo ``` +`tools/init-repo` is the one-time setup for a clone, and is safe to re-run. It checks the two things that otherwise fail confusingly hours later (an old Python, a missing `uv`), creates the test virtualenv, points git at the repo's hooks, and installs the language server binary. Everything it does is local to your clone. + +The hooks are the part git cannot do for you: a fresh clone must not be able to run code, so no repo can enable its own hooks on clone. `make test-unit` and `make typecheck` also enable them, so in practice they are on before your first commit either way. + `uv sync` creates `.venv` in the repo root. Nothing needs activating: `uv run` and `uvx` find it, and the type checker is pointed at it by `pyproject.toml`. The `test` extra declares every dependency the test suite needs, including the bot runtime libraries that tests import directly. The CLI itself depends only on the standard library. Stacklet containers carry their own runtime dependencies, which is why `.venv` will not contain all of them: `nanobot` and `fastapi`, for example, exist only inside their images. @@ -31,7 +35,7 @@ Neither is a merge gate, and the type checker is not clean today. Run it on what ## Language server -Agents working in this repo get a language server, so they can ask for a definition or every reference to a symbol instead of grepping for a name and hoping it is unique. Install the binary once: +Agents working in this repo get a language server, so they can ask for a definition or every reference to a symbol instead of grepping for a name and hoping it is unique. `tools/init-repo` installs the binary; by hand it is: ```bash uv tool install basedpyright @@ -69,4 +73,12 @@ A bare `make typecheck` reports on `lib`, `stacklets`, `tools` and `hooks`. `tes ## Tests, style, commits -All three live in [docs/agent/dev.md](docs/agent/dev.md), which is the canonical reference and stays shorter than a duplicate here would. The short version: `make test-unit` before every commit, module tests over unit tests, semantic commit prefixes, feature branches only, never push without asking. +All three live in [docs/agent/dev.md](docs/agent/dev.md), which is the canonical reference and stays shorter than a duplicate here would. The short version: the hooks lint every commit for you, `make test-unit` before a push or when a piece of work is done, module tests over unit tests, feature branches only, never push without asking. Commit subjects are the changelog famstack.dev renders, so they follow `(): ` and every one has to parse. + +The hooks check ruff on staged files and the commit subject before a commit exists, rather than in CI. `tools/init-repo` turns them on, as do `make test-unit` and `make typecheck`. To do only that: + +```bash +make hooks # or: git config core.hooksPath hooks +``` + +An explicit `core.hooksPath` is never overwritten, so pointing it elsewhere, or nowhere, is respected. diff --git a/Makefile b/Makefile index 7b765e20..b17d7eba 100644 --- a/Makefile +++ b/Makefile @@ -1,34 +1,81 @@ -# famstack development tasks +# famstack development tasks. +# +# One name per lane. The aliases that used to exist (unit, fast, test, +# demo, lifecycle, e2e, smoke, test-all, test-integration) are gone: they +# were eighteen names for six jobs, and `test-all` ran only the lifecycle +# lane, which is worse than not existing. +# +# `make` on its own prints the table. PYTEST = uv run --extra test pytest +.DEFAULT_GOAL := help + +help: + @printf '%s\n' \ + "" \ + " Linting is automatic: the hooks run ruff on staged files and check" \ + " the commit subject. Nothing to remember per commit." \ + "" \ + " Run the cheapest lane that proves what you changed." \ + "" \ + " LANE TIME NEEDS EXCLUSIVE RUN IT WHEN" \ + " make lint <1s nothing no the whole tree, when you want it" \ + " make test-unit ~75s nothing no before a push, or when a piece of work is done" \ + " make test-lifecycle ~6m Docker yes lifecycle, env, compose, ports, health" \ + " make test-demo ~4m demo instance yes it must work against the running bots" \ + " make test-smoke ? test-owned rig yes quick cross-service check" \ + " make test-e2e 20-30m uninstalled yes end of a branch, before review" \ + "" \ + " make typecheck types across the shipped code, not a gate" \ + " make hooks point git at the repo's hooks" \ + "" \ + " Exclusive means it owns fixed container names and ports, so exactly" \ + " one run at a time on this Mac. ? means nobody has timed it yet;" \ + " if you run one, write the number down. Details: tests/README.md" \ + "" + +# The whole tree, the way CI runs it. Per commit the hook already covers +# the staged files, which is faster and is why nothing here is per commit. +lint: hooks + uvx ruff check . + # Type check the shipped code. Not a gate: read what it says about the files # you touched. `uvx basedpyright ` narrows it further. -typecheck types: +typecheck: hooks -uvx basedpyright -# Fast unit tests: no Docker. Run before every commit. -test-unit unit fast test: - $(PYTEST) tests/framework tests/stacklets -v --ignore=tests/framework/test_config_to_container.py +# Point git at the repo's own hooks. Git will not do this on clone, by +# design: a fresh clone must not be able to run code. The nearest honest +# thing is to do it the first time someone uses the repo's tooling, which +# is what the lanes below depend on. +# +# Only when unset, so an explicit choice (an absolute path, a different +# directory, deliberately no hooks) is never overwritten. +hooks: + @tools/init-repo --hooks-only -# Live demo-rig tests: uses the already-running Simpsons demo instance. -test-demo demo-rig demo: - tests/integration/stacktests demo-rig +# Offline. No Docker, no live services, nothing shared. Fast enough to run +# when a piece of work is finished, too slow to run on every commit. +test-unit: hooks + $(PYTEST) tests/framework tests/stacklets -v --ignore=tests/framework/test_config_to_container.py -# Docker lifecycle tests: validates stack up/down/destroy and container env. -test-lifecycle container-lifecycle lifecycle: +# Docker lifecycle: up, down, destroy, env rendering, container environment. +# Owns its own throwaway instance, but on fixed names and ports. +test-lifecycle: $(PYTEST) tests/framework/test_config_to_container.py -v -# Managed container e2e tests: starts the test rig and runs every e2e test. -test-e2e container-e2e e2e: - tests/integration/stacktests e2e +# Against the demo instance already running in this checkout. Tests create +# unique data and clean up after themselves; they never reset the instance. +test-demo: + tests/integration/stacktests demo-rig -# Quick managed-rig e2e subset. -test-smoke smoke: +# A small subset of the managed-rig e2e lane, for a quick answer. +test-smoke: tests/integration/stacktests smoke -# Backwards-compatible names. -test-all: test-lifecycle -test-integration: test-e2e +# The full managed rig: seeds a test-owned instance and runs every e2e test. +test-e2e: + tests/integration/stacktests e2e -.PHONY: typecheck types test-unit unit fast test test-demo demo-rig demo test-lifecycle container-lifecycle lifecycle test-e2e container-e2e e2e test-smoke smoke test-all test-integration +.PHONY: help hooks lint typecheck test-unit test-lifecycle test-demo test-smoke test-e2e diff --git a/docs/README.md b/docs/README.md index 8a0e7d1c..cd0f91e3 100644 --- a/docs/README.md +++ b/docs/README.md @@ -21,4 +21,9 @@ - [ADR-007](adr/adr-007-port-convention.md) 42xxx port convention - [ADR-008](adr/adr-008-convention-based-bot-runner.md) Convention-based bot runner - [ADR-009](adr/adr-009-managed-ai-provider.md) Managed AI with oMLX as default -- [RFC-001](adr/rfc-001-dashboard-and-companion-apps.md) Dashboard and companion apps \ No newline at end of file +- [ADR-010](adr/adr-010-event-pipeline.md) Event pipeline +- [ADR-011](adr/adr-011-vault-brain-projection.md) Vault to brain projection +- [ADR-012](adr/adr-012-nanobot-fork.md) nanobot fork +- [ADR-013](adr/adr-013-stacklet-locations-and-stages.md) Stacklet locations and stages +- [RFC-001](adr/rfc-001-dashboard-and-companion-apps.md) Dashboard and companion apps +- [RFC-002](adr/rfc-002-installation-instances-and-updates.md) Installation, instances and updates \ No newline at end of file diff --git a/docs/admin-guide.md b/docs/admin-guide.md index e61f69e7..ca96f056 100644 --- a/docs/admin-guide.md +++ b/docs/admin-guide.md @@ -828,16 +828,156 @@ Stash the tarball on a USB stick or in 1Password. Container images update themselves nightly at 3am via Watchtower. You do nothing for that. -For the famstack code itself: +The famstack code is a git checkout, and a release is a tag. Updating means moving your checkout to a newer tag. ```bash -git pull -./stack restart # for any stacklet whose code changed +./stack update ``` -If `git pull` shows changes to `lib/stack/` or to a stacklet you are running, restart that stacklet. If in doubt, restart everything. +That is the whole thing from v0.3.0-beta.4 onward. It fetches the releases, shows what the jump changes, moves the checkout, and sets your local edits aside and puts them back. -Pre-1.0 caveat: occasionally a release changes a config schema or env var. Release notes will say so. Read them before pulling. +It does not restart anything. New code on disk is not new code running: a stacklet picks up a release when its containers are recreated, and when your family loses a service for thirty seconds is your call, not a side effect of updating. So the command works out which restarts the release actually earns and prints them: + +``` + ✓ Updated to v0.3.0-beta.4 + + Restart to pick it up + ./stack restart docs + ./stack doctor +``` + +```bash +./stack update --dry-run # show the plan, change nothing +./stack update v0.3.0-beta.4 # a specific release instead of the newest +./stack update --yes # skip the confirmation +``` + +Every form fetches tags first, including `--dry-run`: without it the answer +would only be as fresh as your last fetch. It opens with the version and, +once it knows, the move it is proposing: + +``` + famstack 0.3.0-beta.3 + Fetching releases... + + Update v0.3.0-beta.3 → v0.3.0-beta.4 +``` + +A stacklet is named only if the release changed files inside it and it is running, so a documentation release asks for nothing. If the release changed anything under `lib/`, every running stacklet is named, because that is the code they all share. + +### Updating by hand + +`stack update` ships in v0.3.0-beta.4. Moving *to* that release, or moving between older ones, is the same steps typed out: + +```bash +cd ~/famstack +git fetch --tags +git tag | tail -5 # what is available +git checkout v0.3.0-beta.4 # the one you want +./stack doctor +``` + +Then restart what the release touched. This prints the list: + +```bash +git diff --name-only v0.3.0-beta.3 v0.3.0-beta.4 -- stacklets/ | cut -d/ -f2 | sort -u +``` + +Restart any of those you run. If the same command against `lib/` prints anything, the framework itself changed and everything wants a restart: + +```bash +./stack down all && ./stack up all +``` + +Nothing in the update touches your instance. `stack.toml`, `users.toml`, `.stack/`, `~/famstack-data/` and `~/famstack-extensions/` are all outside git's reach, so your config, secrets, data and your own stacklets come through a tag switch untouched. + +### If you have edited files in the checkout + +`stack update` handles this for you. It lists the files it is about to set aside, warns in advance when one of them is a file the release also changes, and puts them back afterwards. + +If they collide, it winds the whole update back: you end up on the release you started from, with your edits exactly where they were and nothing left in the stash. Then it tells you how to go ahead deliberately: + +``` + ⚠ Your edits collide with v0.3.0-beta.4. Nothing changed. + Back on v0.3.0-beta.3 with your edits where they were. + The collision is in: + • stacklets/docs/docker-compose.yml + To take the release anyway, deal with that file first: + git checkout -- drop your version for the release's + git stash keep it, then pop and merge by hand + then run ./stack update again. +``` + +A half-applied update is worse than no update: a compose file with conflict markers in it does not parse, so the services you restart next fail for a reason that has nothing to do with the release. What follows is the same work done by hand, which is what an older release needs. + +git refuses to move while one of your edits is in the way: + +``` +error: Your local changes to the following files would be overwritten by checkout: + stacklets/docs/docker-compose.yml +Please commit your changes or stash them before you switch branches. +Aborting +``` + +Set them aside, switch, put them back: + +```bash +git stash +git checkout v0.3.0-beta.3 +git stash pop +``` + +If the release did not touch the same file, `git stash pop` re-applies your edit and you are done. + +If the release changed the same lines, it reports a conflict and **keeps your stash**, so nothing is lost: + +``` +CONFLICT (content): Merge conflict in stacklets/docs/docker-compose.yml +The stash entry is kept in case you need it again. +``` + +From there, look at what your edit was, then keep it or drop it: + +```bash +git stash show -p # your change + +# keep it: edit the file to merge both sides, then +git add stacklets/docs/docker-compose.yml && git stash drop + +# or take the release's version and give up your edit +git checkout HEAD -- stacklets/docs/docker-compose.yml && git stash drop +``` + +Editing a shipped stacklet means doing this on every update. There is no override mechanism: a stacklet in `~/famstack-extensions/` can add to what famstack ships, not replace it. If your change is one other families would want, an issue or a pull request costs you less in the long run. + +### Knowing where you are + +```bash +git describe --tags # v0.3.0-beta.3, or a tag plus commits past it +``` + +Checking out a tag leaves you in "detached HEAD". That is normal and nothing to repair, but it does mean `git pull` no longer works: + +``` +$ git pull +You are not currently on a branch. +Please specify which branch you want to merge with. +``` + +Do not take git's suggestion to run `git pull origin main`. It succeeds, and it quietly moves you off releases onto unreleased development code while `./stack version` carries on reporting the last release number. Use `git fetch --tags` and `git checkout `. + +If you cloned and never checked out a tag, you are on `main`, which is development. `git pull` there gives you unreleased work rather than the next release. `./stack update` moves you onto the newest release and says so before it does: your branch is left where it is, and `git switch main` goes back to it. If `main` already contains the newest release, which it will if you pulled recently, it tells you that instead of moving you backwards. + +**On a fork**, `origin` is your copy and has none of the project's releases in it. `./stack update` fetches tags from every remote, so add the project as one: + +```bash +git remote add upstream https://github.com/famstack-dev/famstack.git +./stack update +``` + +Without it there is nothing to update to, and the command says which remotes it looked at. + +Pre-1.0 caveat: occasionally a release changes a config schema or env var. Release notes say so. Read them before updating. ### Paperless-ngx 3.x (docs stacklet) @@ -848,7 +988,7 @@ This release moves the `docs` stacklet from Paperless-ngx 2.20.15 to 3.0.4. ```bash ./stack down docs tar -czf ~/docs-pre-3x-backup.tar.gz -C ~/famstack-data docs -git pull +git fetch --tags && git checkout v0.3.0-beta.3 ./stack up docs ``` diff --git a/docs/adr/adr-013-stacklet-locations-and-stages.md b/docs/adr/adr-013-stacklet-locations-and-stages.md new file mode 100644 index 00000000..aff53e47 --- /dev/null +++ b/docs/adr/adr-013-stacklet-locations-and-stages.md @@ -0,0 +1,114 @@ +# ADR-013: Stacklet Locations and Stages + +## Status +Accepted + +## Context +A stacklet was "a directory with a `stacklet.toml` under `stacklets/`". One +location, one git repo, one release. Everything that did not fit that shape +needed a workaround: a `.git/info/exclude` entry for a stacklet that was not +meant to be public, a long-lived branch for one that was not finished. Both +grow stale, and neither survives a rebase of `main`. + +Three kinds of stacklet do not belong in `stacklets/`: + +- **Private.** Written for one household or one office, not for the project. +- **Third-party.** Someone else's stacklet, with its own repo and release cycle. +- **Unfinished.** Ours, but not ready to be presented as part of a release. + +The first two are a question of location. The third is orthogonal: a stacklet +in either location can be half-built, and an operator who starts one deserves +to be told before the containers come up, not after their family is using it. + +Alternatives considered for location: + +- **Keep one directory, gate with git.** What we were doing. A + `.git/info/exclude` entry is local and invisible to everyone else, and a + branch that carries a stacklet has to be rebased forever. +- **A directory under `data_dir`.** It would be covered by the one backup the + admin guide mandates, and `stack uninstall` would clear it out with + everything else. Rejected: `~/famstack-data//` is what running services + write, and a stacklet is source code. Putting code there also makes + `extensions` a reserved stacklet id. +- **`extensions/` inside the repo, beside `stacklets/`.** The most legible + option, and the one that reads best in the tree. Rejected: it only works + because today's install method is "clone a git repo". The repo is what an + upgrade replaces, so a location inside it is a promise we can only keep + while that stays true, and defaults are sticky long after. + +The three populations this serves do not want the same thing. An incubating +stacklet of ours is disposable and wants to be near the repo; a household's +own stacklet must outlive every reinstall; a third-party one wants an +installer and a trust boundary. Only the first exists today, but a default is +picked once, so it is picked for the one with the strictest requirement. +- **Install by URL** (`stack extensions add `). Useful later, and it + only makes sense once a second location exists at all. + +## Decision +Discovery searches the repo's `stacklets/` first, then every directory in +`[core] extension_dirs`, in the order given. That key is a list and defaults +to `["~/-extensions"]`, so `~/famstack-extensions` on a famstack +instance; a bare string is read as a one-entry list. A directory with a +`stacklet.toml` is a stacklet in any of them. + +Nothing creates that directory. An admin who has no extensions has no such +directory, and discovery skips a configured path that is not there. It is +outside both the repo and the data dir on purpose: the repo is replaced on +upgrade, and the data dir is what running services write. + +The first tree to claim an `id` keeps it. The repo therefore wins a clash with +an extension, and an earlier extension dir wins a clash with a later one, the +way a `PATH` resolves. An extension can add a stacklet, never replace one the +release ships. + +Everything downstream resolves a stacklet by `id` rather than by path, so +hooks, CLI plugins, secrets, `{photos_url}`-style template variables and +`stack up ` work identically wherever it lives. Origin is reported rather +than inferred: every stacklet carries `source` (`repo` or `extension`) and +`path` through `discover`, `list`, `status` and `up`, `list`/`status` also +report `extension_dirs`, and both printers group extensions under their own +heading instead of mixing them into one column. + +The bot runner gets a second read-only bind mount, +`${EXTENSIONS_DIR}:/extensions:ro`, and searches both trees in the runtime's +order, so an extension stacklet can ship a bot. A compose file cannot iterate +a list, so exactly one extension dir is mounted: the first. A stacklet that +ships a bot from any other one would fail silently, so `stack up` says so. + +A bind mount needs its source to exist, which collides with creating nothing. +While the mounted dir is absent the mount source resolves to an empty +stand-in, `.stack/no-extensions`, and core's `on_start` creates whichever of +the two applies. Any `stack up` re-renders core's env, so the real directory +takes over on its own once there is one. + +A manifest declares how finished it is with `stage`. The default is `"stable"` +and is silent. Any other value is repeated back: `stack list` marks it, and +every `stack up` warns that the stacklet can change or disappear and is not +for production. `"beta"` and `"incubating"` have their own wording; an +unrecognised value gets a generic warning rather than a validation error, +because the manifest is making a claim and the runtime's job is to pass it on. + +## Consequences +- A private or incubating stacklet needs no ignore rule and no branch. It is + not in the repo at all, so `git status` never sees it and it can be its own + git repo. +- A second directory is a search path, not a full second home: its stacklets + install, run and get CLI plugins, but a bot there is invisible to the runner + until the directory is first in the list. Mounting all of them needs a + generated compose override, which is a framework change and its own decision. +- An extension now survives everything famstack does: upgrade, `stack + uninstall`, a fresh clone elsewhere. Nothing backs it up either. The backup + section of the operator docs lists it, and an extension that matters should + have a git remote. +- An admin who would rather keep it elsewhere, including inside the data dir, + points `extension_dirs` there. The uninstall prompt names any extension dir + it is about to delete. +- The default is derived from `[core] name`, so an instance that renames the + stack reads `~/-extensions`. The framework stays domain-agnostic; + nothing in `lib/stack/` spells "famstack". +- The install wizard still scans only the repo. Extensions are installed after + setup, with `stack up `. +- Symlinking a stacklet into the extensions dir works for discovery and hooks, + but not for a bot: the bind mount exposes the link, not its target. +- `stack extensions add ` becomes possible later without another + decision. It would clone into this directory and pin a commit. diff --git a/docs/adr/rfc-002-installation-instances-and-updates.md b/docs/adr/rfc-002-installation-instances-and-updates.md new file mode 100644 index 00000000..5e1822ea --- /dev/null +++ b/docs/adr/rfc-002-installation-instances-and-updates.md @@ -0,0 +1,301 @@ +# RFC-002: Installation, Instances and Updates + +## Status +Draft. No decision taken. Written after the extension-stacklets work surfaced +how much of the instance lives inside the code tree. + +## Context + +Today one directory is four things at once: + +| Role | What lives there | +|---|---| +| The product | `lib/stack/`, `stacklets/`, `docs/` | +| The installation | the git checkout you run `./stack` from | +| The instance | `stack.toml`, `users.toml`, `.stack/secrets.toml`, setup markers | +| The git repo | `main`, tags, your local edits | + +Data is already out (`~/famstack-data/`), and extension stacklets are now out +(`~/famstack-extensions/`). Everything else is entangled. + +Updating, as documented in the admin guide: + +```bash +git pull +./stack restart # for any stacklet whose code changed +``` + +Container images update on a separate track: Watchtower pulls nightly at 3am +for every container whose stacklet declares `[upstream] channel = "patch"`, +which is the default (`lib/stack/stack.py:542`). + +## The problems, precisely + +**P1. You cannot install a release.** The README says `git clone`, which puts +you on `main`. Reaching `v0.3.0-beta.3` means `git checkout v0.3.0-beta.3`, +which detaches HEAD, after which `git pull` no longer does what the admin +guide says it does. There is no `stack update`. Five tags exist and nothing in +the product knows about them. + +**P2. The version string was a constant in the working tree.** *(fixed)* +`VERSION = "0.3.0-beta.3"` was what `stack version` and `stack status` +reported, so on a checkout between tags it named a hundred different trees. +Every surface now derives it from `git describe --tags --dirty`, which +answers by comparing the tree against the commit a tag points at. + +**P3. A release does not describe what runs.** Across the stacklets there are +19 image references. Exactly one is pinned to an exact upstream version +(`paperless-ngx:3.0.4`). Five are `:latest` (element-web, synapse, tika, +adguard, watchtower), one is `:main` (open-webui), one carries no tag at all +(openedai-speech-min), five float on a major or minor (`forgejo:14`, +`postgres:16-alpine`, `valkey:8-alpine`, `gotenberg:8`, `caddy:2-alpine`), and +Immich takes `${IMMICH_VERSION:-release}`. Two people installing the same tag +a month apart run different software, and neither can say which. + +**P4. Two update channels that do not know about each other.** Watchtower +moves images nightly; git moves code when the admin remembers. The Paperless +3.x incident is written up in the admin guide: Watchtower rolled instances +from 2.20 to 3.0 across a major version while the code that configures it +stayed behind, and the documented recovery is a backup most people do not +have. That is the failure mode of two uncoordinated channels, not a +Paperless-specific accident. + +**P5. Nothing migrates.** There is no migration code anywhere in `lib/stack/`. +A release that changes a config key, a secret name or a data layout relies on +the admin reading release notes. A live example found while writing this: the +installer's `write_stack_toml()` never wrote `[core] name`, so every instance +it has ever created calls itself "stack" rather than "famstack". That is +cosmetic until something derives a path from it, which the extensions work +just did. New installs are fixed; every existing instance still has the old +file and nothing tells it. + +**P6. Nothing knows whether the running system matches the checkout.** The +admin guide's instruction is "if in doubt, restart everything". `stack doctor` +already compares each container's environment against freshly rendered env and +reports drift, which is the right shape; it just has no equivalent for image +digests or code. + +**P7. Documented commands that do not exist.** `docs/agent/ops.md` listed +`./stack updates`. There is no such command. The row is gone, and both the +operator docs and the admin guide now describe the real tag-to-tag procedure, +including the stash dance a local edit forces. + +## What "proper" would mean + +Four pillars, in dependency order: + +1. **A versioned artifact.** Installing names a version, and that version says + what runs, including image digests. +2. **A stable on-disk layout.** An instance lives somewhere an upgrade cannot + touch, so the artifact can be replaced wholesale. +3. **An update command.** One verb: fetch, show what changes, migrate, apply, + verify, and say what it restarted. +4. **Instance identity.** Every runtime name derives from an instance, so more + than one can coexist. + +Pillars 2 and 4 are the same refactor seen from two angles. That is the +central claim of this document: **moving instance state out of the code tree +unlocks packaging, safe updates and multiple instances at once.** Everything +else is downstream of it. + +## Option space: what we ship + +### A. Git checkout plus `stack update` (evolve what we have) + +`stack update` fetches tags, shows the release notes for the delta, checks out +the tag, runs migrations, restarts what changed, and runs `doctor`. + +- **For:** days of work, not weeks. Keeps clone-and-hack, which is the + contributor funnel. No packaging infrastructure, no signing, no formula. + Rollback is `git checkout ` for the code half. +- **Against:** the admin still needs git and a toolchain. A dirty working tree + blocks the update, which we have to handle in prose. It cannot be + unattended, because a checkout can conflict. + +### B. Homebrew tap + +`brew install famstack-dev/tap/famstack`, then `brew upgrade famstack`. + +- **For:** the native macOS answer for a macOS-only product. Homebrew is + already a hard prerequisite, so it adds no new dependency. Versioning, + rollback and uninstall come for free. `brew upgrade` is a verb people + already run. +- **Against:** we maintain a formula per release. The prefix is replaced on + upgrade, so pillar 2 is a hard prerequisite rather than a nice-to-have. A + hacked stacklet inside the prefix is destroyed by the next upgrade, so the + dev path has to be explicitly supported (`STACK_DIR` plus a `--root`). + +### C. PyPI wheel plus pipx + +- **For:** `pip install famstack` is familiar, and the CLI is stdlib-only so a + wheel is trivial. Needs `[build-system]` and `[project.scripts]`, neither of + which exists today. +- **Against:** buys nothing over Homebrew on a macOS-only product, and adds a + Python-version matrix we do not want. The stacklets tree is data (compose + files, hooks executed from disk, taxonomy), which fits a wheel badly. + +### D. Signed tarball and a `curl | sh` installer + +- **For:** total control over layout and upgrade, no third-party index. +- **Against:** we own the updater, the signing and eventually notarization. + This is the most work for the least differentiation, and `curl | sh` is a + hard sell to the privacy-minded audience this product is for. + +**Recommendation:** A now, B as the mainstream path once pillar 2 lands. Not +C. D only if we ever ship something that must be notarized. + +## Multi-instance: what is actually in the way + +Config isolation already exists: `STACK_DIR` points the CLI at a different +`stack.toml`, `users.toml` and `.stack/` while sharing one stacklets tree +(`lib/stack/cli.py find_instance_dir`). The integration rig uses it. + +Runtime isolation does not exist. Every name below is machine-global: + +| Resource | Where it is fixed | +|---|---| +| Compose project and `container_name` (`stack-`) | every `docker-compose.yml` | +| Docker network `stack` | `lib/stack/docker.py:117` | +| Service ports 42xxx | each `stacklet.toml` | +| Host API port 42001 | `stacklets/core/famstack-api.py:34` (env-overridable, but the launchd wrapper does not set it) | +| launchd label `dev.famstack.api` | `stacklets/core/hooks/on_start.py:11` | +| launchd label `dev.famstack.whisper` | `stacklets/ai/hooks/on_install.py:39` | +| The user's crontab entry | backup stacklet hooks | +| `omlx` via Homebrew and `~/.omlx/models` | machine-wide by nature | + +One subtler than the rest: `project_states()` (`lib/stack/docker.py:254`) maps +any compose project named `stack-*` to a stacklet id. A second instance would +report the first instance's containers as its own, so `stack list` lies before +anything collides. + +This is why `tests/integration/stacktests` takes the dev instance down rather +than running beside it, and why the operator docs say two instances cannot run +on one Mac. + +**Honest read on demand:** the only real consumers today are the test rig and +a developer wanting dev beside prod. A household needs one instance. An office +deployment might want more, and that is the question to answer before building +the UX. +The namespacing prep is worth doing anyway, because `project_states()` +cross-talk is a correctness bug waiting for the first person who tries. + +## Recommended path + +Each phase stands alone and ships on its own. Gates are what proves it. + +### Phase 0: make a release installable (days) + +- Install instructions clone and check out the latest tag. *(open)* +- ~~`stack update`: fetch tags, print the delta, stash a dirty tree, check + out, restart stacklets whose files changed, run doctor.~~ Shipped. It + refuses to restart anything when the stash conflicts, and prints the + recovery commands instead. +- ~~`stack version` reports the checkout's tag, the SHA, and whether the tree + is dirty.~~ Shipped. `git describe --tags --dirty` compares the tree against + the commit a tag points at, and every surface prints the same derived + string. The constant stays in `cli.py` for the release gate to check + against the tag; it is no longer what anyone is shown. +- ~~Delete `./stack updates` from ops.md or implement it.~~ Done: the row is + removed and the manual procedure is documented, which is what `stack update` + has to automate. + +*Gate:* on the rig, from `v0.3.0-beta.2`, `stack update` lands on beta.3, +restarts exactly the stacklets that changed, and `stack doctor` is clean. + +### Phase 1: instance state leaves the code tree (the enabling refactor) + +- `stack.toml`, `users.toml` and `.stack/` move to `~/.famstack//`, + or `~/Library/Application Support/famstack//`. +- `STACK_DIR` stays the override; the default stops being the checkout. +- `stack migrate` moves an existing instance and prints what it did. +- A checkout that still holds a `stack.toml` fails loudly and names the + command, rather than silently running a half-migrated instance. Pre-1.0 + says no compatibility shims; a one-shot migration with a hard cut honours + that better than a fallback path. + +*Gate:* a fresh clone at a new path, pointed at a migrated instance, brings +the whole stack up with no state in the checkout. `git clean -xdf` in the +checkout destroys nothing but build artifacts. + +### Phase 2: a release describes what runs (days, independent of 1) + +- Pin every image to an exact upstream version, as `docs` already is. +- Watchtower keeps running, but against exact tags it can only deliver digest + rebuilds, never a major jump. +- A release manifest (`versions.toml` or generated from the compose files) + lists the image set for that tag, and `stack doctor` compares what is + running against it. + +*Gate:* two fresh installs of the same tag, a month apart, produce the same +image digests. The Paperless failure mode cannot recur without a code change. + +### Phase 3: migrations (weeks) + +- `schema_version` in `stack.toml`, a `migrations/` directory, and `stack + update` applying them in order before restarting anything. +- Forward-only. Say so plainly: Paperless proved that a data migration is not + reversible, so the safety net is backups, not rollback. +- Which makes backup coverage (Postgres dumps, not just files) a prerequisite + for any unattended updater. + +*Gate:* an instance created by today's installer, with no `[core] name`, is +migrated by `stack update` and ends up identical to a fresh install. + +### Phase 4: packaged artifact (weeks, needs 1 and 3) + +Homebrew tap. `brew upgrade famstack` replaces the prefix; the instance is +untouched because of Phase 1; migrations run on the next `stack` invocation +because of Phase 3. + +*Gate:* `brew install`, `brew upgrade`, `brew uninstall` leave a working +instance, an upgraded instance, and an instance whose data survived. + +### Phase 5: instance namespacing (weeks, only on demand) + +Derive every runtime name from an instance id: compose project prefix, +network, port offsets, launchd labels, API port. Scope `project_states()` to +the instance's own prefix. Add `stack --instance ` and a registry. + +*Gate:* two instances up at once on one Mac, each `stack list` reporting only +its own containers, both reachable. + +## Pros and cons versus today's model + +| | Today (git checkout) | Proposed (Phases 0 to 4) | +|---|---|---| +| Install | `git clone`, one command, no packaging work | `brew install`, or clone for contributors | +| What you get | whatever `main` is at that minute | a named version with a known image set | +| Update | `git pull` plus manual restarts by feel | one command that migrates, restarts and verifies | +| Rollback | `git checkout` for code; data is one-way regardless | same, but stated honestly and backed by backups | +| Hackability | edit anything in the tree, it takes effect | preserved via a dev checkout; the packaged prefix is read-only | +| Instance state | inside the checkout, gitignored | outside it, survives any reinstall | +| Multi-instance | config only, runtime collides | possible, once namespaced | +| Cost to us | zero | a formula per release, a migration per schema change, image pins to maintain | +| Risk | drift nobody can see | a wrong migration hits every instance at once | + +The honest summary: today's model is cheap for us and fine for someone who +reads release notes. It fails the person who installed once in March, let +Watchtower run, and now cannot say what they are running or how to move to +the current tag. That person is the target audience. + +## Non-goals and risks + +- **Do not break clone-and-hack.** It is how contributors arrive and how + stacklet authors work. Any packaged path keeps a documented dev checkout. +- **Do not promise rollback.** Forward-only with backups is the truthful + contract while data migrations are one-way. +- **Do not build the multi-instance UX before there is demand.** Do the + namespacing prep, which is a bug fix either way. +- **Watchtower's default deserves a product decision.** Nightly auto-update is + a real feature for families and it is also what broke Paperless. Exact pins + make it safe; keeping floating tags and auto-update is the combination that + bites. + +## Open questions + +1. Is clone-and-hack a supported install path at 1.0, or the contributor path + only? +2. Does an office deployment need several instances per machine, or one per + box? +3. Are we willing to own a Homebrew tap, formula and bottle, per release? +4. Should Watchtower stay on by default once releases pin exact versions? diff --git a/docs/agent/dev.md b/docs/agent/dev.md index 330447b1..8544eced 100644 --- a/docs/agent/dev.md +++ b/docs/agent/dev.md @@ -89,6 +89,8 @@ Optional, all convention-named: ### Stacklet rules +- **Location.** `stacklets//` is for what the release ships. A private, incubating or third-party stacklet goes in `~/-extensions/` (`[core] extension_dirs` adds more search paths) and is discovered the same way, bot included. Searched in order, first claim on an id wins, and only the first dir is mounted into the bot runner. +- **`stage`.** Anything not finished declares `stage = "beta"` or `"incubating"`. `stack list` marks it, `stack up` warns. - **`id` == directory name.** Lowercase, no hyphens, no spaces. Used in container names, env namespacing, secret keys. - **Container name:** `stack-` (single-service) or `stack--` (multi). Set both `services.` and `container_name`. - **Network:** every container joins the `stack` network, declared `external: true`. Cross-stacklet refs go by container name (`http://stack-docs-paperless:8000`). @@ -207,14 +209,21 @@ Keep that file outside the repo. A committed baseline would quietly turn the typ Test runner: **`uv run --extra test pytest`**. The `test` extra in `pyproject.toml` declares every dep. Do NOT re-spell with `uvx --with`. ```bash -make test-unit # fast offline framework + stacklet tests -make test-demo # live Simpsons demo instance tests -make test-lifecycle # Docker lifecycle/container environment tests -make test-smoke # quick managed-rig e2e subset -make test-e2e # full managed-rig e2e suite +make # the lane table: cost, what each needs, when to run it +make lint # <1s, the whole tree. The hook already covers staged files. +make test-unit # ~75s, offline, no Docker. Before a push, or when a + # piece of work is done. Not per commit. +make test-lifecycle # ~6m, Docker. Lifecycle, env rendering, compose, health. +make test-demo # against the running demo instance. +make test-smoke # quick managed-rig e2e subset. +make test-e2e # full managed-rig e2e suite. ``` -Profile details live in [../../tests/README.md](../../tests/README.md). +Run the cheapest lane that proves what you changed. Every lane but +`test-unit` owns fixed container names and ports, so exactly one runs at a +time on a Mac: check nothing else is mid-run first. One name per lane, no +aliases. Details, and the timings nobody has measured yet, live in +[../../tests/README.md](../../tests/README.md). Testing rules: @@ -247,21 +256,161 @@ Testing rules: - **Re-read the full error line before calling a failure a duplicate.** Check sender, target, specific IDs. Two errors that look similar at a glance often differ in the load-bearing field. - **No backwards-compatibility shims** in pre-1.0 code. Change the code, update callers, ship. -## Commit & branch rules +## Commits: the subject is the changelog + +The release log is the changelog. `stack update` prints it, the GitHub release +quotes it, famstack.dev renders it. A subject that does not parse is missing +from the release notes. + +Messages follow Conventional Commits and the Git and Linux kernel +guidelines, in plain technical English. + +``` +()!: + + + +