feat(infra): serve domain mode through Caddy with HTTPS

docs/admin-guide.md

@@ -547,7 +547,7 @@ The wiki maintains itself. A curator sidecar (`stack-memory-curator`) watches th
 
 | | |
 |---|---|
-| Wiki port | `42070` (domain mode: `wiki.<your-domain>`) |
+| Wiki port | `42070` (domain mode: `memory.<your-domain>`; `wiki.<your-domain>` redirects there) |
 | Data | `~/famstack-data/memory/vault/` (a checkout of Forgejo `family/memory`) |
 
 Useful commands:
@@ -592,7 +592,8 @@ wiki_nightly = "03:30"           # nightly full rebuild, local time ("" disables
 
 Key things to know:
 
-- **domain**: leave empty to start. Services are reachable via `<host>:<port>`. Set a domain later for pretty URLs like `photos.home.internal` (requires wildcard DNS on your router).
+- **domain**: leave empty to start. Services are reachable via `<host>:<port>`. Set a domain later for pretty URLs like `photos.home.example.family`, served by the infra stacklet's proxy. It needs two DNS records pointing at the Mac's LAN IP, `*.<domain>` and `<domain>` itself, because the wildcard covers neither the bare name's DNS nor its certificate. Add both as DNS rewrites in AdGuard, and at your DNS provider for devices that bypass AdGuard. Domain mode makes the Mac the DNS server of your whole network and takes some networking knowledge to set up; read [stacklets/infra/README.md](../stacklets/infra/README.md) first.
+- **dns_provider**: with a domain, `"hetzner"` or `"cloudflare"` serves everything over HTTPS, with certificates Caddy obtains through the provider's API. Empty serves plain HTTP. The API token is stored with `./stack infra dns-token`, never in `stack.toml`.
 - **host**: the hostname used in port-mode URLs and hints. Empty = auto-detect the LAN IP (right for households: phones need it). Set to `localhost` for a single-machine setup, or to a fixed name like `mac-mini.local` if the LAN IP keeps shifting.
 - **data_dir**: where all persistent data lives. Back this up. Outside the git repo.
 - **language**: detected from your timezone. Controls which document categories get seeded (German or English). Change it and run `./stack restart docs` to seed missing tags.

docs/agent/ops.md

@@ -43,6 +43,7 @@ If a precondition is missing, `./stack` prints exactly what to do. Don't improvi
 | `./stack ai models` | yes | no | lists installed AI models |
 | `./stack ai connect <url>` | yes | no | points `[ai]` at an AI server elsewhere, or `local`; checks it, installs nothing, restarts nothing. Warns when the address is outside the home network |
 | `./stack setup ai` | yes | no | re-runs AI install (model swap path) |
+| `./stack infra dns-token` | yes | no | stores or replaces the DNS provider API token (HTTPS in domain mode); `./stack up infra` applies it |
 
 **Output contract:** every command returns JSON when piped or when `--json` is passed. Force human output with `--pretty`. Exit code 0 == success.
 
@@ -72,7 +73,9 @@ Refuse without explicit, scoped human approval:
 | Editing `.stack/secrets.toml` | Breaks every stacklet that depends on the changed secret. |
 | Moving `data_dir` while stacklets are running | Bind mounts break. `stack down all` first, then move, then `stack up`. |
 | Changing `stack.toml [core] language` | Re-seeds Paperless taxonomy. Orphan tags require manual cleanup. |
-| Switching `[core] domain` empty ↔ non-empty | Switches port mode ↔ domain mode. Requires wildcard DNS + Caddy understanding. |
+| Switching `[core] domain` empty ↔ non-empty | Switches port mode ↔ domain mode. Needs `*.<domain>` and `<domain>` DNS records to the Mac, and the infra stacklet up. |
+| Pointing the router's DNS at the Mac (infra) | Every device on the LAN then resolves through AdGuard; the Mac or AdGuard going down takes name resolution for the whole house with it. The admin changes the router, never an agent; note the previous DNS setting first. |
+| `./stack down infra` / `./stack destroy infra` | With the router pointing at the Mac, the house loses DNS until infra is back or the router is switched back. |
 
 ## Ports (42xxx range)
 
@@ -88,6 +91,9 @@ Refuse without explicit, scoped human approval:
 | 42060 | ai | oMLX |
 | 42062 | ai | Whisper |
 | 42070 | memory | Family wiki (Quartz) |
+| 42080 | infra | AdGuard admin (domain mode, Mac only) |
+| 42081 | infra | AdGuard setup wizard, first run (domain mode, Mac only) |
+| 53, 80, 443 | infra | DNS and the Caddy proxy (domain mode, whole LAN) |
 
 Port collisions: do not silently rebind. Surface them. The user's fix is "stop the offender" or switch to domain mode.
 

docs/stack-reference.md

@@ -202,6 +202,8 @@ Available template variables:
 | `{ai_default_model}` | `stack.toml` → `[ai].default` |
 | `{ai_tts_voice}` | Derived from `[ai].language` |
 | `{messages_server_name}` | `stack.toml` → `[messages].server_name` |
+| `{url}`, `{<id>_url}`, `{home_url}` | Public URLs: `http://<ip>:<port>` in port mode; `https://<id>.<domain>` in domain mode once `[core].dns_provider` is set, `http://` before that |
+| `{<id>__<NAME>}` | A secret from `secrets.toml`, e.g. `{docs__API_TOKEN}`, `{infra__DNS_API_TOKEN}` |
 
 ### Hints
 
@@ -512,15 +514,15 @@ effect.
     b. hooks/on_install.sh — create dirs, install deps, build
  4. Write .env to stacklet directory
  5. Bot runner discovers bots (if stacklet has bot/bot.toml)
- 6. Assemble Caddyfile (domain mode)
- 7. Build or pull Docker images
- 8. hooks/on_start.sh — start native services (host stacklets)
+ 6. hooks/on_start.sh — start native services (host stacklets)
+ 7. Write the Caddyfile, counting this stacklet as up (domain mode)
+ 8. Build or pull Docker images
  9. Start containers (docker compose up -d)
-10. Wait for health check
-11. First run only:
+10. Reload Caddy (domain mode)
+11. Wait for health check
+12. First run only:
     a. hooks/on_install_success.py — obtain tokens, seed data
-12. hooks/on_start_ready.py — service is healthy, seed data, sync accounts
-13. Reload Caddy (domain mode)
+13. hooks/on_start_ready.py — service is healthy, seed data, sync accounts
 14. Show welcome screen with URL, login, hints
 ```
 
@@ -546,7 +548,9 @@ dependency order (dependents first, deps last).
 
 ```
 1. hooks/on_stop.sh — stop native services (host stacklets only)
-2. docker compose stop — pause containers
+2. Write the Caddyfile without this stacklet (domain mode)
+3. docker compose stop — pause containers
+4. Reload Caddy (domain mode)
 ```
 
 ### `stack destroy <id>`
@@ -563,7 +567,7 @@ secrets, config. Requires confirmation.
 6. Delete stacklet secrets from secrets.toml ({id}__*)
 7. Delete setup-done marker
 8. Delete data directory (~/{data_dir}/{id}/)
-9. Reassemble Caddyfile (domain mode)
+9. Write the Caddyfile without this stacklet, reload Caddy (domain mode)
 ```
 
 Global secrets (`global__ADMIN_PASSWORD`) survive destroy. The user's
@@ -946,24 +950,95 @@ The runtime operates in one of two modes based on `stack.toml`:
 
 **Domain mode** (`domain = "home.internal"`):
 - Services bind to `127.0.0.1:<port>` (only Caddy reaches them)
-- URLs are `http://photos.home.internal`
-- Caddy assembles routes from `caddy.snippet` files
-- Requires wildcard DNS on router
+- URLs are `http://photos.home.internal`, or `https://` once
+  `dns_provider` is set (see TLS below)
+- The runtime assembles the Caddyfile from `caddy.snippet` files
+- Needs two DNS records to the server's LAN IP, `*.<domain>` and
+  `<domain>`, because a wildcard does not cover the bare name
 
 ### Caddy Snippets
 
-Each stacklet can include a `caddy.snippet` file. The runtime assembles
-all snippets into a single Caddyfile on every `stack up`.
+A stacklet that serves something in a browser ships its routes in a
+`caddy.snippet`:
 
 ```
 # stacklets/docs/caddy.snippet
-docs.{$FAMSTACK_DOMAIN} {
+docs.{$STACK_DOMAIN} {
     reverse_proxy stack-docs-paperless:8000
 }
 ```
 
-The `{$FAMSTACK_DOMAIN}` variable is set by the runtime in Caddy's
-environment.
+The domain stays the `{$STACK_DOMAIN}` placeholder. Caddy substitutes it
+from its own environment when it loads the file; the runtime never does.
+Backends are container names on the `stack` network.
+
+In domain mode the runtime assembles the snippets of every stacklet that
+is up into `{data_dir}/infra/Caddyfile`, in stacklet order, and the infra
+stacklet's Caddy container mounts that file. "Up" is read from Docker like
+every other state: the stacklet has containers running, starting, or
+restarting. A crash-looping stacklet keeps its routes, because its other
+containers are usually still serving.
+
+`stack up`, `stack down` and `stack destroy` write the file, each counting
+the stacklet it acts on as already in its new state, and have a running
+Caddy load it with `caddy reload` once the containers have changed. The
+file is written before a starting stacklet's containers exist, because
+infra's own Caddy reads it the moment it starts. A reload Caddy rejects
+leaves it serving its previous routes, and the command prints a warning.
+Nothing is written in port mode, or while infra is not up.
+
+Around the snippets the assembler (`lib/stack/caddy.py`) emits:
+
+- A global options block: how certificates are obtained, or, without TLS,
+  every site served as plain HTTP on port 80.
+- A `*.{$STACK_DOMAIN}` site that answers 404 for any host no running
+  stacklet claims, a stopped stacklet's included.
+
+Snippets describe routes only. Anything that applies to every site belongs
+in the assembler, not in a snippet.
+
+Sites for services that are not stacklets go in
+`{data_dir}/infra/Caddyfile.local`. The admin writes it and the runtime
+only reads it, appending it after the snippets every time it writes the
+Caddyfile; `stack up infra` applies an edit. Only one proxy can own ports
+80 and 443, so a hand-written Caddyfile cannot run beside this one. A local
+site gets the wildcard certificate like any other subdomain. A host that a
+running stacklet also serves makes Caddy reject the reload, and the
+command prints the warning. `stack destroy infra` deletes the file with
+the rest of infra's data.
+
+### TLS
+
+`[core] dns_provider` (`"hetzner"` or `"cloudflare"`) turns on HTTPS. The
+global block then makes Let's Encrypt the issuer for every certificate,
+through a DNS-01 challenge at that provider:
+
+```
+{
+	cert_issuer acme {
+		dns hetzner {env.DNS_API_TOKEN}
+		propagation_delay 30s
+		propagation_timeout 5m
+	}
+}
+```
+
+Cloudflare's block has the `dns` line only. DNS-01 needs nothing on the
+LAN to be reachable from the internet, and it is the only challenge that
+issues a wildcard. Caddy obtains two certificates: `*.<domain>`, for the
+catch-all site, which every subdomain site then uses, and `<domain>`
+itself, which a wildcard does not cover.
+
+The token is the infra stacklet's secret `DNS_API_TOKEN`. `stack infra
+dns-token` stores or replaces it, and the first `stack up infra` asks for
+it in a terminal. It reaches Caddy as an environment variable and is read
+when a certificate is requested, so it appears neither in the Caddyfile
+nor in the config Caddy serves on its admin endpoint. infra's `on_start`
+refuses a provider the image has no plugin for, or a missing token.
+
+With `dns_provider` set, the URLs the runtime renders (`{url}`,
+`{<id>_url}`, `{home_url}`) are `https://`. `[core] https = true` does the
+same for a reverse proxy in front of the stack that is not this one.
 
 ---
 
@@ -1036,6 +1111,12 @@ One file, committed to the repo. User edits it directly.
 ```toml
 [core]
 domain        = ""                    # empty = port mode
+dns_provider  = ""                    # domain mode: "hetzner" or
+                                      # "cloudflare" serves HTTPS with
+                                      # certificates via DNS-01; empty
+                                      # serves plain HTTP. The API token
+                                      # is a secret (`stack infra
+                                      # dns-token`), never in this file.
 host          = ""                    # port-mode host override.
                                       # Empty = auto-detect LAN IP.
                                       # Set to "localhost" for
@@ -1090,7 +1171,6 @@ Gitignored. Created by `stack init`. Contains:
 |---|---|
 | `secrets.toml` | Auto-generated credentials (passwords, API tokens). |
 | `*.setup-done` | Marker files. Gates once-only hooks (`on_install`, `on_install_success`). |
-| `caddy/conf.d/*.snippet` | Assembled Caddy snippets (domain mode). |
 
 No `enabled` file — stacklet state is derived from Docker containers
 and the filesystem. See [States](#states).

lib/stack/caddy.py

@@ -0,0 +1,103 @@
+"""The Caddyfile for domain mode, assembled from stacklet snippets.
+
+In domain mode every service is reached by name (`photos.<domain>`)
+through one reverse proxy, the Caddy container of the proxy stacklet.
+A stacklet that serves something ships its routes as a `caddy.snippet`;
+this module joins the snippets of the stacklets that are up into the one
+Caddyfile that container mounts.
+
+Snippets describe routes only. Everything that applies to the proxy as a
+whole, the catch-all for hosts no stacklet claims and how certificates
+are obtained, is emitted here, so a snippet never has to know about any
+other.
+
+With `[core] dns_provider` set, every certificate comes from Let's
+Encrypt through a DNS-01 challenge at that provider: Caddy writes a TXT
+record through the provider's API, so nothing on the LAN has to be
+reachable from the internet. That is also the only challenge that can
+issue the wildcard certificate. The `*.<domain>` site makes Caddy manage
+one, and every subdomain site uses it rather than getting its own. The
+bare domain is not covered by the wildcard, and gets a second
+certificate the same way.
+
+The domain is left as the `{$STACK_DOMAIN}` placeholder, in snippets and
+here alike. Caddy fills it in from its own environment when it loads the
+file, so the file is a function of which stacklets are up and nothing
+else.
+"""
+from __future__ import annotations
+
+# The stacklet that runs the proxy, and where it expects its config.
+STACKLET = "infra"
+CONTAINER = "stack-infra-caddy"
+CONFIG_IN_CONTAINER = "/etc/caddy/Caddyfile"
+
+RELOAD_COMMAND = ("caddy", "reload", "--config", CONFIG_IN_CONTAINER)
+
+# The admin's own sites, next to the assembled Caddyfile, appended after
+# the stacklets' snippets.
+LOCAL_FILE = "Caddyfile.local"
+
+_HEADER = """\
+# Assembled by the stack CLI from the caddy.snippet of every stacklet that
+# is up, and Caddyfile.local next to this file. It is rewritten whenever a
+# stacklet starts or stops, so an edit here does not survive: put your own
+# sites in Caddyfile.local, or change the stacklet's snippet.
+"""
+
+# Sites without a scheme listen on the HTTPS port, TLS or not. With no
+# certificates to serve, that port is moved to 80, so every snippet is
+# served as plain HTTP without having to spell out `http://` itself.
+_PLAIN_HTTP = """\
+{
+\tauto_https off
+\thttps_port 80
+}
+"""
+
+# The DNS providers whose Caddy plugin is compiled into the infra image,
+# and the settings each one's plugin asks for beyond the token. Hetzner's
+# README asks for a delay before Caddy starts checking for the record.
+DNS_PROVIDERS = {
+    "hetzner": ("propagation_delay 30s", "propagation_timeout 5m"),
+    "cloudflare": (),
+}
+
+# The token is read from Caddy's environment when a certificate is
+# requested, so it never appears in this file or in the config Caddy
+# serves on its admin endpoint.
+_TOKEN = "{env.DNS_API_TOKEN}"
+
+# A host that no running stacklet claims, including one whose stacklet is
+# stopped, gets an answer instead of a connection error.
+_UNKNOWN_HOSTS = """\
+*.{$STACK_DOMAIN} {
+\trespond "No service at {host}" 404
+}
+"""
+
+
+def _global_options(dns_provider: str) -> str:
+    """Global options: certificates through the DNS provider, or none."""
+    if not dns_provider:
+        return _PLAIN_HTTP
+    if dns_provider not in DNS_PROVIDERS:
+        raise ValueError(
+            f"dns_provider '{dns_provider}' is not supported; "
+            f"use {' or '.join(DNS_PROVIDERS)}, or leave it empty for plain HTTP")
+    lines = [f"dns {dns_provider} {_TOKEN}", *DNS_PROVIDERS[dns_provider]]
+    body = "".join(f"\t\t{line}\n" for line in lines)
+    return f"{{\n\tcert_issuer acme {{\n{body}\t}}\n}}\n"
+
+
+def assemble(snippets: list[tuple[str, str]], dns_provider: str = "") -> str:
+    """The Caddyfile for these `(source, snippet)` pairs, in order.
+
+    A source is the stacklet a snippet came from, or `LOCAL_FILE`.
+
+    Raises ValueError for a DNS provider the infra image has no plugin for.
+    """
+    parts = [_HEADER, _global_options(dns_provider), _UNKNOWN_HOSTS]
+    for source, snippet in snippets:
+        parts.append(f"# ── {source} ──\n\n{snippet.strip()}\n")
+    return "\n".join(parts)

lib/stack/cli.py

@@ -21,6 +21,7 @@
 import tomllib
 from pathlib import Path
 
+from . import caddy
 from . import docker
 from . import doctor
 from .commands import COMMANDS
@@ -255,6 +256,8 @@ def up(self, stacklet_id: str) -> dict:
             if code != 0:
                 return {"error": "Failed to start services", "output": err}
 
+        self._reload_proxy()
+
         if compose_file:
             # Health checks run first — hooks can assume the service is healthy
             template_vars = self.stack._build_template_vars()
@@ -330,13 +333,15 @@ def down(self, stacklet_id: str) -> dict:
             except ValueError:
                 pass
             code, output = docker.compose_stop(compose_file)
+            self._reload_proxy()
             result = {"stacklet": stacklet_id, "action": "down",
                       "success": code == 0, "output": output}
             if code != 0:
                 # Surface the compose error instead of the caller's generic
                 # "unknown error", which hid the real cause.
                 result["error"] = output or "docker compose stop failed"
             return result
+        self._reload_proxy()
         return {"stacklet": stacklet_id, "action": "down", "success": True}
 
     def _down_all(self) -> dict:
@@ -440,7 +445,27 @@ def destroy(self, stacklet_id: str) -> dict:
                 return {"error": "Failed to stop containers", "output": output}
             self.stack.output.step("Containers removed")
 
-        return self.stack.destroy(stacklet_id)
+        result = self.stack.destroy(stacklet_id)
+        self._reload_proxy()
+        return result
+
+    def _reload_proxy(self) -> None:
+        """Have the running proxy load the Caddyfile the framework just wrote.
+
+        Called after every change to which stacklets are up. Port mode has
+        no proxy, and a proxy that is not up reads the file when it starts.
+        A rejected reload leaves Caddy serving its previous config, so the
+        step that asked for it still succeeded; the admin is told the
+        routes did not change.
+        """
+        if not self.stack._cfg("core", "domain"):
+            return
+        if caddy.STACKLET not in self.stack.serving_ids():
+            return
+        code, err = docker.exec_in(caddy.CONTAINER, *caddy.RELOAD_COMMAND)
+        if code != 0:
+            reason = err.strip().splitlines()[-1] if err.strip() else f"exit {code}"
+            self.stack.output.warn(f"Caddy kept its previous routes: {reason}")
 
 
 # ── Stacklet id arguments ─────────────────────────────────────────────────