Skip to content

stack-reference.md

Latest commit

 

History

History
1337 lines (1070 loc) · 54.1 KB

stack-reference.md

File metadata and controls

1337 lines (1070 loc) · 54.1 KB
Raw

Stacklet Reference

A stacklet is the unit of deployment. The stack is a composition of stacklets. The stack CLI is the runtime. It discovers stacklets by walking the filesystem, reads their manifests, and manages their lifecycle. No central registry — if it's a directory under stacklets/ with a stacklet.toml, it exists.

It is convention over configuration to "sew" services together. Think Spring Boot for self-hosted services on a Mac.

Stacklet vs core

The distinction is infrastructure vs feature. Core hosts cross-cutting infrastructure that every running stack needs: Watchtower (auto-updates), the bot runtime, the API socket, the LLM tools server. Vertical stacklets host features a user might rationally decline: photos, documents, backup, the AI stack. The test: could a user legitimately want this off? If yes, it belongs in its own stacklet, even when multiple other stacklets opt into it via manifest contracts (e.g. [[backup.archive]] declared by photos and docs, consumed by the backup stacklet).

Directory Structure

A stacklet is a directory containing at minimum a stacklet.toml manifest. Everything else is optional — include only what you need.

Two directories are searched, in this order:

Location What belongs there
stacklets/ in the repo The stacklets a release ships and supports.
an extension dir ([core] extension_dirs, default ["~/<product>-extensions"]) Everything else: a private stacklet, a community one, one still being designed. Outside the repo, so an upgrade cannot take it, and each can be its own git repo.

extension_dirs is a list, searched in the order given. The first tree to claim an id keeps it, so the repo wins a clash with an extension and an earlier extension dir wins a clash with a later one. Nothing creates these directories; one that is not there is skipped.

The runtime resolves a stacklet by id, never by path, so stack up <id>, hooks, CLI plugins, secrets and template variables like {photos_url} behave identically wherever it lives. stack list and stack status group extensions in their own section, and every stacklet in the JSON output carries source (repo or extension) and path.

One asymmetry: a compose file cannot iterate a list, so the bot runner bind-mounts the first extension dir only. A stacklet that ships a bot belongs there, and stack up says so when it is somewhere else. While that directory does not exist, the mount gets an empty stand-in under .stack/, because a bind mount needs a source and an instance without extensions should not grow a directory in the admin's home. Any stack up re-renders core's env, so the real one takes over by itself.

stacklets/photos/
  stacklet.toml          ← manifest (required)
  docker-compose.yml     ← container definitions
  caddy.snippet          ← reverse proxy route (domain mode)
  hooks/
    on_configure.py      ← first run: interactive prompts
    on_install.py        ← first run: create dirs, install deps
    on_install_success.py← first run: obtain tokens, seed data
    on_start.py          ← every up: validate config, start native services (before containers)
    on_start_ready.py    ← every up: seed data, sync accounts (after health checks)
    on_stop.py           ← every down: stop native services
    on_destroy.py        ← teardown: remove native services
  cli/
    seed.py              ← CLI subcommand: stack photos seed

Every file is a convention. The runtime looks for it by name. If it's there, it's used. If it's not, it's skipped. Hooks can be .py (preferred) or .sh — the runtime checks for Python first.

File When it runs Purpose
stacklet.toml Always read Identity, config, dependencies
docker-compose.yml stack up / stack down Container definitions
caddy.snippet Caddyfile assembly (domain mode) Reverse proxy route
hooks/on_configure Once on first stack up Interactive prompts (API keys, server names)
hooks/on_install Once on first stack up Create directories, install native deps
hooks/on_install_success Once after first healthy start Obtain tokens, seed data
hooks/on_start Every stack up, before containers Validate config, start native services
hooks/on_start_ready Every stack up, after health checks Seed data, sync accounts (service is healthy)
hooks/on_stop Every stack down Stop native services
hooks/on_destroy On stack destroy Remove native services
cli/*.py On demand via stack <id> <command> Stacklet-specific CLI commands

Manifest: stacklet.toml

The manifest declares what the stacklet is and what it needs. The runtime reads it — the stacklet never reads it itself.

Required Fields

id          = "photos"
name        = "Photos"
description = "Family photo library and mobile backup (Immich)"
version     = "0.1.0"
category    = "media"
Field Type Description
id string Unique identifier. Lowercase, no spaces. Used in CLI commands, file paths, container names, and secret namespacing.
name string Human-readable display name. Shown in stack list and status output.
description string One-line description.
version string Stacklet version (semver).
category string One of: infrastructure, media, ai, communication, productivity, development, automation. Used for grouping in stack list.

Optional Fields

port        = 42010
always_on   = true
type        = "host"
requires    = ["core", "messages"]
stage       = "incubating"
Field Type Default Description
port int none LAN port for port mode. All ports live in the 42xxx range.
ports table none Additional named ports: [ports] → element = 42030, synapse = 42031.
always_on bool false If true, stack destroy refuses to remove it (only core uses this). stack down still works.
type string "docker" "docker" (default) or "host". Host stacklets install native macOS software (brew, compiled binaries) alongside optional Docker containers.
requires list [] Stacklet IDs that must be enabled before this one. The runtime enforces ordering on stack up and prevents destroying dependencies.
build bool false If true, the stacklet has a local Dockerfile. stack up rebuilds the image on every run instead of pulling from a registry. Use for stacklets with custom code (bots, agents).
stage string "stable" How finished the stacklet is. "stable" is silent. Any other value ("beta", "incubating", or your own) is shown in stack list and warned about on every stack up.
required_secrets list [] Secret names (unprefixed, as ctx.secret() reads them) the stacklet cannot work without. stack doctor reports any that are absent and points at stack setup <id>. Declare a secret here when it is minted by on_install_success, since that hook never runs again on an instance that is already installed.

Upstream

Declares the primary Docker image for auto-update tracking.

[upstream]
image   = "ghcr.io/immich-app/immich-server"
channel = "patch"
Field Values Description
image string Docker image reference. Watchtower monitors this for updates.
channel "patch" / "none" patch: Watchtower auto-updates. none: manual only.

Environment

The environment system renders configuration from templates. No .env files to maintain — the runtime generates them on every stack up from these declarations.

[env]
generate = ["DB_PASSWORD", "SECRET_KEY"]

[env.defaults]
UPLOAD_LOCATION  = "{data_dir}/photos/library"
DB_DATA_LOCATION = "{data_dir}/photos/postgres"
TZ               = "{timezone}"
DB_USERNAME      = "postgres"
ADMIN_USER       = "{admin_email}"
ADMIN_PASSWORD   = "{admin_password}"

generate — list of env var names. Values are auto-generated as cryptographically random strings and stored in .famstack/secrets.toml, namespaced by stacklet ID (photos__DB_PASSWORD). Idempotent — existing secrets are never overwritten. Destroying a stacklet does not remove its secrets, so re-enabling reuses the same credentials.

defaults — key-value pairs with {template} variables. Rendered against stack.toml values on every stack up. The rendered output is written to the stacklet's .env file.

Available template variables:

Variable Source
{data_dir} stack.toml → [core].data_dir
{domain} stack.toml → [core].domain
{ip} [core].host if set, else auto-detected LAN IP (falls back to localhost) — used in port-mode URLs
{language} stack.toml → [core].language (falls back to [ai].language, then en)
{timezone} stack.toml → [core].timezone
{shared_bucket} stack.toml → [core].shared_bucket (defaults to "family") — slug for the shared/institutional bucket inside the memory vault
{stacklet_id} The stacklet's own id field
{admin_username} Tech admin username (stackadmin)
{admin_email} Tech admin email (stackadmin@home.local)
{admin_password} Tech admin password, generated and stored in secrets.toml
{family_names} Comma-separated first names of everyone in users.toml
{ai_openai_url} Derived from stack.toml → [ai].openai_url
{ai_openai_url_docker} Same, rewritten for container access via host.docker.internal
{ai_openai_key} stack.toml → [ai].openai_key
{ai_whisper_url_docker} Derived from stack.toml → [ai].whisper_url, or the AI server when unset
{ai_whisper_key} stack.toml → [ai].whisper_key with a dedicated speech server, otherwise the AI key
{ai_whisper_model} stack.toml → [ai].whisper_model, default whisper-1
{ai_default_model} stack.toml → [ai].default
{ai_tts_voice} Derived from [ai].language
{messages_server_name} stack.toml → [messages].server_name
{url}, {<id>_url}, {home_url} Public URLs: http://<ip>:<port> in port mode; https://<id>.<domain> in domain mode once [core].dns_provider is set, http:// before that
{browser_url} The address to open in a browser on this Mac: {url} in domain mode, http://localhost:<port> in port mode, because a web app that needs WebCrypto (Element) only runs over HTTPS or on localhost
{url_host} The host of {url}: <id>.<domain> in domain mode, {ip} in port mode. For a protocol the stacklet serves under the same name besides HTTP, such as Forgejo's SSH
{<id>__<NAME>} A secret from secrets.toml, e.g. {docs__API_TOKEN}, {infra__DNS_API_TOKEN}
{oidc_issuer}, {oidc_client_id}, {oidc_client_secret} Single sign-on for a stacklet with an [oidc] table, see Single Sign-On. All three empty until the provider has registered the stacklet

Hints

Post-setup messages shown to the user after stack up. Template variables from the environment plus {url} and {ip} are available.

hints = [
    "Open {url} and create your admin account",
    "Install the Immich app on your phone — enter {url} as the server",
]

Health

Defines how the runtime confirms the service is actually responding, not just that the container is running. Checked after stack up and by stack status.

[health]
url    = "http://localhost:42010/api/server/ping"
path   = "$.res"
expect = "pong"
Field Description
url HTTP endpoint to poll.
expect Expected HTTP status code (as string) or response body value.
path JSONPath into the response body. If set, expect is compared against the extracted value instead of the status code.

For several endpoints, use [[health.checks]], which additionally takes a name, a hint shown when the check fails, and [health.checks.headers].

[[health.checks]]
name          = "TTS"
url           = "http://localhost:42063/"
hint          = "TTS not running — check 'docker logs stack-ai-speech'"
skip_when_env = "STACK_AI_NO_VOICE"

skip_when_env names an environment variable that, when set to "1", drops the check entirely. Use it for a part of a stacklet the user can choose not to start: stack up ai --no-voice leaves the speech containers out, and without this the start sequence waits a full timeout for each of them and then reports containers it was told to skip as broken. Any other value, including "0" or empty, leaves the check in place, so an accidentally exported variable cannot quietly disable monitoring.

A stacklet that can be served by another machine declares a top-level remote = "{template}" in its manifest. When it renders to an address, stack list shows the row as remote with that host, or localhost when the address is this machine, meaning something on it that the stack does not manage serves it. Only the checks marked remote = true probe that server; the others probe this Mac, so they run only while the stacklet's own containers run here as well.

Native Services (host stacklets)

Host stacklets (type = "host") declare native macOS services that run outside Docker. The runtime checks each service on stack up and optionally starts it.

[services.omlx]
name        = "oMLX"
description = "MLX inference with SSD caching (Metal GPU)"
check_url   = "{ai_openai_url}/models"
start       = "brew services start omlx"
stop        = "brew services stop omlx"

[services.whisper]
name        = "Whisper"
description = "Speech-to-text (whisper.cpp, Metal GPU)"
check_url   = "http://localhost:42062/"
start       = "launchctl load ~/Library/LaunchAgents/dev.famstack.whisper.plist"
stop        = "launchctl unload ~/Library/LaunchAgents/dev.famstack.whisper.plist"
Field Description
name Display name for status output.
check_url URL polled to determine if the service is running. Supports {template} variables from stack.toml (e.g. {ai_openai_url}).
start Shell command to start the service if not responding. Optional.
stop Shell command to stop the service. Used by down.sh. Optional.

Bot Convention

A stacklet ships a bot by adding a bot/ directory with a bot.toml manifest and a Python file. The bot runner (in core) discovers bots across all enabled stacklets and runs them in one async process.

Bot IDs always end with -bot (e.g. archivist-bot, scribe-bot).

stacklets/docs/
  bot/
    bot.toml          # declaration
    archivist.py      # MicroBot subclass
    messages/         # i18n templates (optional)

# stacklets/docs/bot/bot.toml
id          = "archivist-bot"
name        = "Archivist"
description = "Auto-files documents with AI classification"
room        = "documents"
room_topic  = "Drop files here — they get filed automatically."

[settings]
classify = true
reformat = true
Field Description
id Bot identifier, ends with -bot. Becomes the Matrix username (@archivist-bot:home).
name Display name in Matrix.
room Room alias to create/join. Optional — omit for bots that only respond to DMs/invites.
room_topic Topic set on auto-created room.
join_rooms Aliases of rooms another stacklet owns that the bot takes part in. Joined on every start when they exist; never created, and no one else is joined to them.
settings Arbitrary key-value pairs passed as kwargs to the bot constructor.

Module convention: strip -bot from the ID → archivist.py → class ArchivistBot.

Bot passwords are declared in the stacklet's [env].generate (e.g. "ARCHIVIST_BOT_PASSWORD" in docs/stacklet.toml). The bot runner reads passwords from .stack/secrets.toml on startup.

Bot Events

Matrix is famstack's canonical event ledger — every stacklet emits typed envelopes into the room timeline, and any state in the system must be derivable from replaying those events. Two delivery shapes:

  1. Visible event — m.room.message carrying a dev.famstack.event content field. The body renders for humans in Element; bots and the deriver read the structured envelope alongside it. One event, full picture.
  2. Silent event — dev.famstack.event-typed message. Element ignores unknown event types, so the event acts as a pure machine signal with no chat noise.

Both shapes carry the same envelope schema:

{
  "source": "<stacklet id>",     // "docs", "messages", "photos", …
  "type":   "<verb.noun>",       // "document.filed", "voice.transcribed"
  "summary": "<one-liner>",      // human-readable activity description
  "data":   { … },               // structured payload
  "actor":  "@user:home" | "bot",
  "ts":     "<ISO-8601 UTC>"
}

Examples shipping today:

Type Delivery Emitter data keys
document.filed visible (on m.room.message) archivist-bot paperless_id, title, date, topics[], persons[], correspondent, document_type, summary, facts[], action_items[], url
document.reclassified visible archivist-bot same as document.filed + user_hint

Finding famstack events in a room: enumerate the timeline and treat any event whose content has a dev.famstack.event key OR whose Matrix type is dev.famstack.event as a ledger entry. Both deliver the same envelope.

To send a silent event from a bot, call MicroBot.emit_event:

await self.emit_event(
    room_id,
    "dev.famstack.event",
    {"source": "docs", "type": "service.started", "summary": "…",
     "data": {…}, "actor": "archivist-bot", "ts": "…"},
)

To attach an envelope to a visible message, pass metadata on a text-sending helper (see archivist.py's _send):

await self._send(
    room_id, body_text, reply_to,
    metadata={"dev.famstack.event": envelope},
)

Both paths are best-effort; transport failures are logged but never raised. A downstream bot being offline must not take the emitter's main path with it.

Subscribing to custom events on the bot side isn't wired yet — that comes with the first consumer bot. The emit contract is stable and tested end-to-end (see tests/e2e/test_archivist_e2e.py).

Backup

A stacklet declares the data it wants backed up. The backup stacklet discovers these declarations across all enabled stacklets and routes them through configured targets (see stack.toml → [backup.targets.*]). The stacklet itself never reads the manifest — discovery is the runtime's job.

# stacklets/photos/stacklet.toml
[[backup.archive]]
name = "library"
path = "{data_dir}/photos/library/library"
Field Description
name Short slug for this source. Combined with the stacklet id, this becomes the global source id (photos/library), which is also the directory it occupies on a vault (data/photos/library/). Used in stack backup status output and (future) --source= selection.
path Filesystem path to sync. Template variables from the rendered environment are available ({data_dir}, etc.).

There is no threshold to declare. The engine judges each source against the number of files it held on the previous run, recorded in its own history, and refuses to sync one that is suddenly empty or has lost more than half its files. This replaced a min_files constant, which could not work: a number written at authoring time cannot know the scale of the household it guards, so a library of 50,000 photos reduced to 11 passed min_files = 10 without complaint. A source that has never held anything is skipped rather than failing the run.

[[backup.archive]] declares an append-only store: files are added, never modified, never deleted. The engine commits to kernel-enforced immutability where the filesystem supports it. If a stacklet's data is genuinely append-only (photo originals, archived PDFs), this is the right section. (Storage-industry vocabulary calls this WORM — Write Once Read Many.)

[[backup.snapshot]] declares mutable state that cannot be rsynced. A database changes under you continuously and a copy taken mid-write will not restore, so it is dumped instead: one pg_dump per run, packed with whatever small files must travel beside it, into a dated .tar.gz. Each run adds a tarball and never touches an older one, which gives mutable state the same append-only shape an archive has.

# stacklets/messages/stacklet.toml
[[backup.snapshot]]
name     = "synapse"
postgres = { container = "stack-messages-db", database = "synapse", user = "synapse" }
include  = ["{data_dir}/messages/synapse/homeserver.yaml",
            "{data_dir}/messages/synapse/*.signing.key"]
Field Description
name Short slug, as for an archive. Becomes the source id (messages/synapse).
postgres How to capture the state, namespaced by what captures it. container is where the database runs (the dump goes through docker exec), database and user are what to dump and as whom.
include Files that must travel with the dump for a restore to be possible. Globs allowed; a path this install never created is skipped rather than failing the snapshot.

The capture key is namespaced so a stacklet keeping state somewhere other than Postgres can declare a snapshot later without the contract having to pretend every database looks like this one. Driving a containerised Postgres lives in stack.postgres, shared with whatever eventually restores one, rather than inside the backup coordinator where only it could reach it.

Snapshots run before the sync, so a dump is never newer than the files it references. pg_dump takes its own consistent view, so nothing stops and nobody is logged out while it runs. The tarball records the image versions and digests that produced it; nothing reads that yet, but it is the one part of a snapshot that cannot be added afterwards, and a restore has to be able to refuse an incompatible target.

A stacklet may declare zero, one, or several entries of each kind. Sources flow to every configured target whose engine supports the declared section type.

Single Sign-On (OIDC)

A stacklet can let people log in through an OpenID Connect provider that another stacklet runs. The two sides do not name each other: the client does not know which provider it gets, and the provider does not know the client's env.

A client declares [oidc] and reads three template variables:

[env.defaults]
OIDC_ISSUER        = "{oidc_issuer}"
OIDC_CLIENT_ID     = "{oidc_client_id}"
OIDC_CLIENT_SECRET = "{oidc_client_secret}"

[oidc]
name      = "Documents"     # shown on the provider's consent page; default: the stacklet name
callbacks = ["{url}/accounts/oidc/sso/login/callback/"]

callbacks are rendered against the client's own template variables, so {url} is the client's public URL. The three {oidc_*} variables are set together or not at all: they stay empty while no provider exists, and until the provider has registered the client. Test one of them in the compose file to switch the login on, and keep the service's own login as the fallback.

A provider declares [oidc_provider] (an empty table). The issuer is its public URL, {url}. Its hooks do the registration through the Stack instance in ctx.stack:

Method What it does
oidc_clients() Every stacklet with [oidc]: stacklet, name, rendered callbacks, and the stored client_id and client_secret (None before the first registration)
store_oidc_client(stacklet_id, client_id, client_secret) Stores the credentials the provider issued

Credentials are kept in the provider's secret namespace (<provider>__CLIENT_<ID>_ID and _SECRET). stack destroy <provider> deletes them, and each client falls back to its own login on its next stack up. A client reads new credentials only when its containers are recreated, so a provider should not rotate a stored secret on every run, and should tell the admin which client to stack up after storing new credentials. If more than one stacklet declares [oidc_provider], the first one discovered is used.

Brought up in one command, such as stack up all, the provider starts before its clients, so they render their env after it has registered them. A client brought up alone does not pull the provider in.

stack doctor warns about a running client that a running provider has not registered, and about a second provider being ignored. A provider that is present but was never brought up raises nothing.

A provider may need domain mode with HTTPS, and should refuse to start without it in on_install and on_start. A passkey login is the usual reason: browsers allow WebAuthn only on an https:// origin. The framework does not check this, because the requirement belongs to the provider, not to OIDC.

Lifecycle

States

A stacklet is in exactly one of three states. There is no separate "enabled" registry — state is derived from what actually exists.

                stack up
  AVAILABLE ──────────────► RUNNING
       ▲                    │     ▲
       │                    │     │
       │ stack destroy      │     │ stack up
       │                    │     │
       │                stack down│
       │                    │     │
       │                    ▼     │
       └──────────────── STOPPED
            stack destroy
State How to detect Meaning
Available No containers (docker ps -a), no data dir Defined in the repo, never started or fully destroyed
Running Containers exist and are running Active, serving requests
Stopped Containers exist but not running Paused, data intact, stack up resumes

stack up <id>

Brings a stacklet to the running state. Idempotent — safe to run repeatedly. Every run refreshes config so changes in stack.toml take effect.

 1. Check requires — fail if dependencies not running/stopped
 2. Render .env from templates + generate missing secrets
 3. First run only:
    a. hooks/on_configure.py — interactive prompts
    b. hooks/on_install.sh — create dirs, install deps, build
 4. Write .env to stacklet directory
 5. Bot runner discovers bots (if stacklet has bot/bot.toml)
 6. hooks/on_start.sh — start native services (host stacklets)
 7. Write the Caddyfile, counting this stacklet as up (domain mode)
 8. Build or pull Docker images
 9. Start containers (docker compose up -d)
10. Reload Caddy (domain mode)
11. Wait for health check
12. First run only:
    a. hooks/on_install_success.py — obtain tokens, seed data
13. hooks/on_start_ready.py — service is healthy, seed data, sync accounts
14. Show welcome screen with URL, login, hints

First run detection: .famstack/{id}.setup-done marker. Absent means first run. Created after on_install completes. Deleted by destroy.

stack down <id>...

Transitions from running to stopped. Data and containers are preserved — stack up brings it back without re-running setup.

Several ids stop together, separated by spaces or commas (stack down memory code docs or stack down memory,code,docs), in reverse dependency order. Every id is checked first, so an unknown one stops nothing. stack up and stack destroy take ids the same way (up in dependency order, destroy in reverse, each destroy confirmed on its own unless --yes). stack restart stops them all, then starts them in dependency order.

Use stack down all to stop every currently-running stacklet in reverse dependency order (dependents first, deps last).

1. hooks/on_stop.sh — stop native services (host stacklets only)
2. Write the Caddyfile without this stacklet (domain mode)
3. docker compose stop — pause containers
4. Reload Caddy (domain mode)

stack destroy <id>

Transitions to available. Removes everything — containers, data, secrets, config. Requires confirmation.

1. hooks/on_stop.sh — stop native services
2. hooks/on_destroy.sh — remove native services (host stacklets only)
3. Render .env if missing (compose needs it to parse volume defs)
4. docker compose down -v --remove-orphans — remove containers + volumes
5. Delete .env
6. Delete stacklet secrets from secrets.toml ({id}__*)
7. Delete setup-done marker
8. Delete data directory (~/{data_dir}/{id}/)
9. Write the Caddyfile without this stacklet, reload Caddy (domain mode)

Global secrets (global__ADMIN_PASSWORD) survive destroy. The user's password doesn't change when they remove a single service.

stack uninstall

Destroys all stacklets, removes Docker network, deletes runtime state (.famstack/), and removes config files (stack.toml, users.toml). The nuclear option — back to a fresh clone.

Hooks

Each lifecycle transition can trigger stacklet-specific hooks. Convention over configuration: if the file exists, it runs. If not, the step is skipped. No registration needed.

First stack up:

  on_configure ──► on_install ──► on_start ──► health ──► on_install_success ──► on_start_ready
  (config gate)    (system)       (services)   (wait)     (first-run API work)   (every-up API work)

on_configure is the gate: it collects all required settings (provider
choice, API keys, server names) and persists them to stack.toml or
secrets.toml. If it fails or is interrupted, the next stack up re-enters
on_configure and picks up where it left off. on_install reads the config
that on_configure wrote and acts on it. Both hooks should be idempotent.

Subsequent stack up:

  on_start ──► health ──► on_start_ready
  (services)   (wait)     (API work)

stack down:

  on_stop
  (services)

stack destroy:

  on_stop ──► on_destroy
              (teardown)
Hook Runs Purpose
on_configure Once Config gate. Collects all required configuration via interactive prompts and writes it to stack.toml or secrets.toml. Must be idempotent — if it set some config values but the process was interrupted, the next run should detect what's already set and only ask for what's missing. on_install only proceeds when on_configure completes without error.
on_install Once System setup. Create directories, install native software, build from source. Should be idempotent — check whether each step was already done before doing it again (e.g. brew list omlx before brew install omlx, check if binary exists before building).
on_install_success Once Obtain API tokens, seed initial data, create accounts. Runs after first healthy start.
on_start Every up Runs before containers start. Validate config, start native services. If required config is missing or invalid, raise with a clear message — the framework stops the pipeline and containers won't start.
on_start_ready Every up Runs after health checks pass. The service is healthy and accepting API calls. Seed data, sync accounts, anything that needs the service running. Must be idempotent. Its ctx.env is rendered after on_install_success, so secrets written there are in it.
on_stop Every down Stop native services. Only stops services we manage (.state/ markers).
on_destroy Once Remove native services entirely (unload plists, uninstall).
on_restore On stack backup restore Reserved — not yet invoked. Runs after the backup engine has put a stacklet's files back on disk. Owns stacklet-specific recovery: DB import, search-index rebuild, account re-seed. Photos can ship an empty stub (Immich re-indexes from the library on its own); Docs needs pg_restore + Paperless reindex. Hook signature will match the other run(ctx) hooks.

File resolution: for each hook, the runtime looks for .py first, then .sh. Only one can exist — not both. Python is preferred.

hooks/on_install.py   ← checked first (preferred)
hooks/on_install.sh   ← fallback

Once-only hooks (on_configure, on_install, on_install_success) are gated by the .famstack/{id}.setup-done marker. Created after on_install completes. Deleted by on_destroy. A future stack up runs them again from scratch.

Python hooks (run(ctx)) receive a context dict:

Key Type Description
ctx.env dict Rendered environment variables.
ctx.secret(name) callable Read a secret. ctx.secret(name, value) writes one.
ctx.step(msg) callable Print a progress line.
ctx.shell(cmd) callable Run a shell command with streaming output and error handling. The error message includes the command.
ctx.run_in_container(argv, service=None, user=None) callable Run argv in the stacklet's container (stack-<id>, or stack-<id>-<service>) with docker exec, as a list and without a shell. Returns stdout. On failure it raises with the container, the exit code and the last stderr line, never the arguments, so a password or secret can be passed.
ctx.http_post(url, body) callable HTTP POST, returns parsed JSON.
ctx.http_get(url) callable HTTP GET, returns parsed JSON.

Shell hooks receive environment variables: all rendered env vars plus FAMSTACK_DATA_DIR and FAMSTACK_DOMAIN.

Hook reference

Hook interface

All Python hooks implement run(ctx). The runtime calls it with a context object. Shell hooks receive environment variables instead.

The context object (ctx) provides:

Key Type Description
ctx["env"] dict Rendered environment variables (all templates resolved).
ctx["secret"] callable secret(name) reads a secret. secret(name, value) writes one. Lookup chain: stacklet-specific (photos__X) then global (global__X). Writes go to stacklet namespace.
ctx["step"] callable step(msg) prints a progress line to the user.
ctx["http_post"] callable http_post(url, body, content_type=..., headers=...) → parsed JSON. Form-encoded by default.
ctx["http_get"] callable http_get(url, headers=...) → parsed JSON. Pass auth explicitly: headers={"Authorization": "Bearer ..."}.

A Python hook or CLI plugin may put its stacklet's directory on sys.path and import helpers from it by bare name (from oauth import sync). The framework forgets those modules and restores sys.path when the hook or command returns, so two stacklets can each have a helper of the same name even when one process runs both, as stack restart all does.

A hook that raises stops the command and is reported as a failure. A hook that asked the admin and got a no raises stack.hooks.Cancelled instead: stack up stops with {"cancelled": <message>}, prints the message once, exits 1, and does not tell the family room that the stacklet failed to start.

from stack.hooks import Cancelled
from stack.prompt import confirm

def run(ctx):
    if not confirm("Replace the remote AI server with this Mac?", default=False):
        raise Cancelled("Cancelled. Nothing changed.")

Example hooks/on_install.py (system work with ctx.shell()):

def run(ctx):
    data_dir = ctx.env["FAMSTACK_DATA_DIR"]
    ctx.step("Creating directories...")
    ctx.shell(f"mkdir -p {data_dir}/docs/paperless/media")
    ctx.shell(f"mkdir -p {data_dir}/docs/postgres")
    ctx.shell(f"chmod 700 {data_dir}/docs/postgres")

Example hooks/on_install_success.py (API work):

def run(ctx):
    secret = ctx["secret"]
    step = ctx["step"]

    existing = secret("API_TOKEN")
    if existing:
        # Verify it still works (may be stale after destroy + up)
        try:
            ctx["http_get"](
                "http://localhost:42020/api/documents/",
                headers={"Authorization": f"Token {existing}"},
            )
            return  # still valid
        except Exception:
            step("Stored token invalid — obtaining new one")

    step("Obtaining API token...")
    data = ctx["http_post"](
        "http://localhost:42020/api/token/",
        f"username={ctx['env']['ADMIN_USER']}&password={ctx['env']['ADMIN_PASSWORD']}",
    )
    secret("API_TOKEN", data["token"])
    step("API token saved")

CLI Commands

Any .py file under cli/ (except files starting with _ and post_setup.py) becomes a subcommand.

stacklets/photos/cli/seed.py  →  stack photos seed
stacklets/messages/cli/send.py →  stack messages send

The convention:

Attribute Purpose
HELP Module-level string. Shown in stack <id> --help.
Module body Executed when the command runs. Has access to sys.argv for arguments.

Files starting with _ are private helpers (e.g., _matrix.py), not exposed as commands.

Command rules

Every stacklet adds commands, so these keep them reading as one CLI. Check a new command, or a changed one, against them.

  1. Name the thing or the action, as an admin sees it. Never the component that does the work (curator, archivist) and never when it usually runs (nightly): those change, the admin's intent does not.
  2. A thing with more than one action is a noun with verbs. stack memory wiki update, stack memory wiki clean, stack messages room list. The bare noun lists its verbs, and the noun leaves room for a later verb such as show.
  3. Flags narrow or modify, and one word means one thing everywhere. --all everything, --dry-run preview and write nothing, --yes skip the confirmation, --as <member> act as that family member. --json belongs to the framework. Flags that contradict each other are refused in one line that names both.
  4. A command that changes something applies by default. --dry-run is the opt-in preview, never the other way round.
  5. A command that asks a background worker waits until the work is done, prints each step and each item while it waits, and on timeout says how far it got. The request itself survives the timeout.
  6. A command returns a dict, {"error": ...} on failure, which exits non-zero. With --json that dict is all of stdout; what the command prints for people goes to stderr.

A command that records a choice in stack.toml calls config["set_cfg"](section, key, value), the plugin-side twin of a hook's ctx.cfg(key, value). It changes that one line in that section and leaves the admin's comments and other sections as they are. stack ai switch uses it to write [ai].

Docker Compose Conventions

Every docker-compose.yml follows a set of naming and wiring conventions. The runtime relies on these for discovery, cleanup, cross-stacklet communication, and auto-updates.

Project Name

name: stack-docs

Set explicitly. Format: stack-{stacklet_id}. This prevents Docker from deriving a project name from the directory path, which breaks when the repo is cloned to a different location.

Container Names

services:
  stack-docs-paperless:
    container_name: stack-docs-paperless

Format: stack-{stacklet_id}-{service}. Both the service key and container_name use the same value. Single-container stacklets can omit the service suffix: stack-chatai, stack-bots.

This convention enables:

  • stack ps to group containers by stacklet
  • stack uninstall to find and remove all famstack containers (docker ps -a --filter "name=^stack-")
  • Inter-container references by predictable name (http://stack-docs-paperless:8000)

Network

networks:
  stack:
    external: true

All containers join the shared stack network, declared as external. Created by stack init, removed by stack uninstall. Every service in every stacklet must include this.

Containers reference each other by container name across stacklets:

environment:
  PAPERLESS_URL: http://stack-docs-paperless:8000
  MATRIX_HOMESERVER: http://stack-messages-synapse:8008

Native macOS services (oMLX, whisper.cpp) are reached from containers via host.docker.internal.

Port Binding

ports:
  - "${PORT_BIND_IP:-127.0.0.1}:42020:8000"

The PORT_BIND_IP variable controls access scope. The runtime sets it:

  • Port mode: 0.0.0.0 (reachable from the LAN)
  • Domain mode: 127.0.0.1 (only Caddy reaches it)

A port the LAN needs in domain mode too is bound to 0.0.0.0 literally instead: infra's DNS (53) and proxy (80, 443), and Forgejo's SSH, which does not go through Caddy and is encrypted and authenticated by itself.

The host port is the stacklet's declared port from stacklet.toml. The container port is whatever the upstream service uses internally.

Volumes

volumes:
  - ${PAPERLESS_DATA_DIR}/media:/usr/src/paperless/media
  - ${PAPERLESS_DB_DATA}:/var/lib/postgresql/data

Bind mounts to {data_dir}/{stacklet_id}/. All paths come from environment variables rendered by the runtime — never hardcoded. This makes the data directory discoverable and consistent: ~/famstack-data/docs/, ~/famstack-data/photos/, etc.

Named Docker volumes are avoided. Bind mounts are explicit, visible in the filesystem, and easy to back up.

Auto-Updates (Watchtower)

labels:
  - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_ENABLE:-true}"

Every container gets this label. Watchtower (in the core stacklet) monitors labeled containers and pulls new images on the nightly schedule. Set channel = "none" in stacklet.toml to disable.

Watchtower pulls the tag the compose file names, so the tag decides how far an image can move. Pin the minor line where upstream publishes one (gotenberg:8.37), otherwise the exact version (synapse:v1.161.0), or a digest when upstream tags no release at all. Watchtower then delivers patch releases, and a minor or major version changes only with a commit. Immich keeps to semantic versioning and tags only its major line, so it is pinned to that (immich-server:v3). For PostgreSQL the major tag (postgres:16-alpine) is already the patch line: its minor releases are bug and security fixes. Named tags such as latest, main or release fail tests/unit/framework/test_compose_pins.py.

Health Checks

healthcheck:
  test: ["CMD-SHELL", "pg_isready -U ${DB_USER} -d ${DB_NAME}"]
  interval: 10s
  timeout: 5s
  retries: 5

Supporting services (databases, caches) should declare Docker health checks so depends_on with condition: service_healthy works. The main service uses [health] in stacklet.toml instead — the runtime polls it after docker compose up.

Restart Policy

restart: unless-stopped

All containers use unless-stopped. They survive Docker daemon restarts and host reboots, but stay down if explicitly stopped with stack down.

Dependencies

depends_on:
  stack-docs-db:
    condition: service_healthy
  stack-docs-redis:
    condition: service_healthy

Within a stacklet, use depends_on with health conditions. Across stacklets, use requires in stacklet.toml — the runtime enforces ordering at the CLI level.

Complete Example

name: stack-docs

services:
  stack-docs-paperless:
    container_name: stack-docs-paperless
    image: ghcr.io/paperless-ngx/paperless-ngx:3.0
    labels:
      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_ENABLE:-true}"
    networks:
      - famstack
    depends_on:
      stack-docs-db:
        condition: service_healthy
    volumes:
      - ${PAPERLESS_DATA_DIR}/data:/usr/src/paperless/data
    environment:
      PAPERLESS_DBHOST: stack-docs-db
      PAPERLESS_DBPASS: ${DB_PASSWORD}
    ports:
      - "${PORT_BIND_IP:-127.0.0.1}:42020:8000"
    restart: unless-stopped

  stack-docs-db:
    container_name: stack-docs-db
    image: postgres:16-alpine
    labels:
      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_ENABLE:-true}"
    networks:
      - famstack
    volumes:
      - ${PAPERLESS_DB_DATA}:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER}"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

networks:
  famstack:
    external: true

Port Mode vs Domain Mode

The runtime operates in one of two modes based on stack.toml:

Port mode (domain = ""):

  • Services bind to 0.0.0.0:<port>
  • URLs are http://hostname:port
  • Caddy does not start
  • Zero DNS setup required

Domain mode (domain = "home.internal"):

  • Services bind to 127.0.0.1:<port> (only Caddy reaches them)
  • URLs are http://photos.home.internal, or https:// once dns_provider is set (see TLS below)
  • The runtime assembles the Caddyfile from caddy.snippet files
  • Needs two DNS records to the server's LAN IP, *.<domain> and <domain>, because a wildcard does not cover the bare name

Caddy Snippets

A stacklet that serves something in a browser ships its routes in a caddy.snippet:

# stacklets/docs/caddy.snippet
docs.{$STACK_DOMAIN} {
    reverse_proxy stack-docs-paperless:8000
}

The domain stays the {$STACK_DOMAIN} placeholder. Caddy substitutes it from its own environment when it loads the file; the runtime never does. Backends are container names on the stack network.

In domain mode the runtime assembles the snippets of every stacklet that is up into {data_dir}/infra/Caddyfile, in stacklet order, and the infra stacklet's Caddy container mounts that file. "Up" is read from Docker like every other state: the stacklet has containers running, starting, or restarting. A crash-looping stacklet keeps its routes, because its other containers are usually still serving.

stack up, stack down and stack destroy write the file, each counting the stacklet it acts on as already in its new state, and have a running Caddy load it with caddy reload once the containers have changed. The file is written before a starting stacklet's containers exist, because infra's own Caddy reads it the moment it starts. A reload Caddy rejects leaves it serving its previous routes, and the command prints a warning. Nothing is written in port mode, or while infra is not up.

Around the snippets the assembler (lib/stack/caddy.py) emits:

  • A global options block: how certificates are obtained, or, without TLS, every site served as plain HTTP on port 80.
  • A *.{$STACK_DOMAIN} site that answers 404 for any host no running stacklet claims, a stopped stacklet's included.

Snippets describe routes only. Anything that applies to every site belongs in the assembler, not in a snippet.

Sites for services that are not stacklets go in {data_dir}/infra/Caddyfile.local. The admin writes it and the runtime only reads it, appending it after the snippets every time it writes the Caddyfile; stack up infra applies an edit. Only one proxy can own ports 80 and 443, so a hand-written Caddyfile cannot run beside this one. A local site gets the wildcard certificate like any other subdomain. A host that a running stacklet also serves makes Caddy reject the reload, and the command prints the warning. stack destroy infra deletes the file with the rest of infra's data.

TLS

[core] dns_provider ("hetzner" or "cloudflare") turns on HTTPS. The global block then makes Let's Encrypt the issuer for every certificate, through a DNS-01 challenge at that provider:

{
	cert_issuer acme {
		dns hetzner {env.DNS_API_TOKEN}
		propagation_delay 30s
		propagation_timeout 5m
	}
}

Cloudflare's block has the dns line only. DNS-01 needs nothing on the LAN to be reachable from the internet, and it is the only challenge that issues a wildcard. Caddy obtains two certificates: *.<domain>, for the catch-all site, which every subdomain site then uses, and <domain> itself, which a wildcard does not cover.

The token is the infra stacklet's secret DNS_API_TOKEN. stack infra dns-token stores or replaces it, and the first stack up infra asks for it in a terminal. It reaches Caddy as an environment variable and is read when a certificate is requested, so it appears neither in the Caddyfile nor in the config Caddy serves on its admin endpoint. infra's on_start refuses a provider the image has no plugin for, or a missing token.

With dns_provider set, the URLs the runtime renders ({url}, {<id>_url}, {home_url}) are https://. [core] https = true does the same for a reverse proxy in front of the stack that is not this one.

Secrets

Generated Secrets

Declared in [env].generate. Stored in .famstack/secrets.toml, namespaced by stacklet ID:

photos__DB_PASSWORD = "xK7mQp3JvR2nYs8LwB4dN6..."
docs__DB_PASSWORD = "aB3cD4eF5gH6iJ7kL8mN9..."
docs__API_TOKEN = "060068ace4f65db88c52da..."
global__ADMIN_PASSWORD = "muw7suf7"

Properties:

  • Auto-generated on first stack up if missing
  • Never overwritten on subsequent runs
  • Preserved across stack destroy (so re-enable reuses credentials)
  • Gitignored
  • 32-character alphanumeric for service secrets
  • 8-character lowercase+digits for admin passwords (typed on phones)

Admin Password

A single admin password is generated once and shared across all stacklets. Stored as global__ADMIN_PASSWORD in secrets.toml. Available to templates as {admin_password}.

Identity (who the admin is) lives in users.toml. Credentials (the password) live in secrets.toml. Never in the same file.

Users

users.toml defines the family members. Identity only — no passwords.

[[users]]
id = "homer"
name = "Homer"
email = "homer@home.local"
role = "admin"

[[users]]
id = "sarah"
name = "Sarah"
email = "sarah@home.local"
role = "member"
stacklets = ["photos", "documents"]
Field Description
id Username. Used in CLI commands and as default login name.
name Display name.
email Email address. Used as login for services that require one.
role admin (created on every stacklet), member, or restricted.
stacklets Which services this user gets an account on. Admins ignore this — they're created everywhere.

Global Configuration: stack.toml

One file, committed to the repo. User edits it directly.

[core]
name          = "famstack"            # product name: CLI branding and
                                      # derived paths (~/<name>-extensions).
                                      # Missing = "stack". An instance
                                      # without it whose data dir exists
                                      # as <name>-data gets it written on
                                      # the next command.
domain        = ""                    # empty = port mode
dns_provider  = ""                    # domain mode: "hetzner" or
                                      # "cloudflare" serves HTTPS with
                                      # certificates via DNS-01; empty
                                      # serves plain HTTP. The API token
                                      # is a secret (`stack infra
                                      # dns-token`), never in this file.
host          = ""                    # port-mode host override.
                                      # Empty = auto-detect LAN IP.
                                      # Set to "localhost" for
                                      # single-machine testing, or a
                                      # custom hostname for stable
                                      # access across network changes.
data_dir      = "~/famstack-data"
timezone      = "Europe/Berlin"
language      = "de"                  # document tags, UI language (de, en)
shared_bucket = "family"              # slug for the shared/institutional
                                      # bucket inside the memory vault.
                                      # Personal entities (homer, marge, …)
                                      # live at <vault>/<localpart>/. Default
                                      # "family" fits famstack; non-family
                                      # deployments override.

[updates]
schedule = "0 0 3 * * *"        # Watchtower cron (3am nightly)

[ai]
openai_url = "http://localhost:8000/v1"
openai_key = "local"
default    = "mlx-community/Qwen2.5-14B-Instruct-4bit"
whisper_url = "http://localhost:6111/v1"
language   = "en"

[backup]
# One block per destination. Engine name selects the implementation.
[backup.targets.vault]
engine   = "external-disk"
disk     = "backup-vault"
schedule = "0 2 * * *"

[backup.targets.<name>] defines one destination. <name> is the user's label for that destination (any string). The required engine field picks the implementation under stacklets/backup/engines/. Today only external-disk ships. Engine-specific fields (disk, schedule, future repository, password) live alongside engine in the same block.

Stacklets never read stack.toml directly. The runtime resolves template variables and passes everything through the rendered .env.

Runtime State: .famstack/

Gitignored. Created by stack init. Contains:

File Purpose
secrets.toml Auto-generated credentials (passwords, API tokens).
*.setup-done Marker files. Gates once-only hooks (on_install, on_install_success).

No enabled file — stacklet state is derived from Docker containers and the filesystem. See States.

Deleted entirely by stack uninstall.

Multiple Instances: STACK_DIR

One repo can power more than one stack instance. A stacklet definition (code, compose file, hooks) is shared; the instance (config, secrets, state, data) is swappable.

  • Repo root — holds stacklets/ and lib/. Discovered by walking up from the current working directory; outside any checkout, the checkout the CLI runs from. That is how the installer's global famstack command (in Homebrew's bin) works from any directory. It names only the checkout, so STACK_DIR selects the instance through it as it does through ./stack.
  • Instance dir — holds stack.toml, users.toml, .stack/. Holds the data referenced by [core].data_dir. Defaults to the repo root.

Point the CLI at a different instance with STACK_DIR:

STACK_DIR=tests/e2e/instance stack up docs

The same stacklets/ definitions apply, but config, secrets, setup markers, and data all live under that directory instead. If STACK_DIR is set to a non-existent path, the CLI fails fast rather than silently falling back.

Instances share Docker container names (stack-docs, stack-messages), so two instances cannot run concurrently on the same machine — stack down the active one before bringing another up. Useful for: dedicated test environments, sandboxes, experimenting with a clean state without touching your real household install.