Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,13 @@ changing architecture.

Phase 0 approved (2026-10-28). Phase 1 (v0.1) and Phase 2 (v0.2) approved and archived
(2026-10-28, `openspec/changes/archive/2026-10-28-phase-1-stacks-and-routing` and
`…-phase-2-overview-and-budget`). siding today: `setup [--port] [--budget]`, `teardown`,
`up [--wait] [--force]`, `status [--wait]`, `ps`, `logs`, `open`, `down [--volumes]`, `gc`,
`lint`, `ui`, `skill`; routing by host and path through `siding-proxy`; a memory budget;
`just smoke-stacks`, `just build-darwin`. Next: Phase 3 (`phase-3-snapshots-and-queue`, v0.3),
starting with `/opsx:propose`.
`…-phase-2-overview-and-budget`). Phase 3 (v0.3) approved and archived (2026-10-29,
`…/2026-10-29-phase-3-snapshots-and-queue`). siding today: `setup [--port] [--budget]
[--builds]`, `teardown [--snapshots]`, `up [--wait] [--force] [--seed]`, `status [--wait]`,
`ps`, `logs`, `open`, `down [--volumes]`, `snapshot save|ls|rm`, `gc`, `lint`, `ui`, `skill`;
routing by host and path through `siding-proxy`; a memory budget with a start queue and learned
peaks; build slots; `just smoke-stacks`, `just build-darwin`. Next: Phase 4
(`phase-4-idle-and-wake`, v0.4), starting with `/opsx:propose`.

## Working agreements

Expand Down Expand Up @@ -112,7 +114,8 @@ requests and on `main`, all inside the devcontainer image.
`siding`, the user's stacks, `~/.siding`) is not development footprint. `hack/nuke.sh` never
removes it; siding's own commands and `docker compose -p siding down` do.
- Tests create stacks only as Compose projects with the prefix `siding-test-` and remove them by
their exact name. The smoke tests' proxies listen on 18080, never on 8080; the end-to-end
their exact name. Their snapshots are therefore named `siding-snap_siding-test-…` and are
removed by exact name too; `just smoke-stacks` checks that none of either is left. The smoke tests' proxies listen on 18080, never on 8080; the end-to-end
tests run their own proxy `siding-test-e2e-proxy` on the network `siding-test-e2e`, so a
developer's `siding-proxy` is never touched.
- No real secrets in the repo, in golden files or in logs; golden files may contain obvious
Expand All @@ -132,8 +135,11 @@ requests and on `main`, all inside the devcontainer image.
and `dev.siding.agent` (optional), never on networks or volumes, whose labels must not change
between two `up`s. Service labels: `dev.siding.route` (`/`, `/<path>`, `false`),
`dev.siding.port` (the HTTP port of a service with several), `dev.siding.seed`
(`volume|script|none`). Volume label `dev.siding.shared`. On `siding-proxy`:
`dev.siding.budget` (the memory budget in bytes) and `dev.siding.host-alias` (what
(`volume|script|none`), `dev.siding.seed.command` (the seed command of `script`). Volume
label `dev.siding.shared`. Snapshot volumes: `dev.siding.snapshot` (name),
`dev.siding.snapshot-volume` (the stack's volume key), `dev.siding.snapshot-created`, plus
`dev.siding.repo` and `dev.siding.machine`. On `siding-proxy`: `dev.siding.budget` (the memory
budget in bytes), `dev.siding.builds` (build slots) and `dev.siding.host-alias` (what
`host.docker.internal` resolves to in the proxy).
- Compose project `<repo>-<branch-slug>`, host `<repo>.<branch-slug>.localhost`, proxy
`siding-proxy` on the network `siding`, override `.siding/override.yml`.
Expand Down
76 changes: 69 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,9 @@
siding gives every git worktree its own Docker Compose stack, at its own `*.localhost` address,
and tells you and your coding agents whether it actually runs and what it costs.

> **Pre-alpha (v0.2).** Stacks per worktree, routing through one local proxy, the verdict of
> `siding status`, the overview `siding ui`, a memory budget, `gc` and `lint` work; expect
> **Pre-alpha (v0.3).** Stacks per worktree, routing through one local proxy, the verdict of
> `siding status`, the overview `siding ui`, a memory budget with a queue and learned peaks,
> snapshots and seeding, a build limit, `gc` and `lint` work; expect
> rough edges and changes before v1. There are no releases
> yet: build it from source (see below). The plan is in [openspec/changes/](openspec/changes)
> (one change per phase), what siding does today in [openspec/specs/](openspec/specs).
Expand All @@ -22,8 +23,10 @@ siding teardown # remove the proxy again; stacks keep running, --dr
```

Every stack is reached through this one proxy at `http://<repo>.<branch>.localhost`. Browsers
and curl resolve `*.localhost` to your machine by themselves; for other tools add the host to
`/etc/hosts`. Running `siding setup` again changes nothing unless the port differs.
resolve `*.localhost` to your machine by themselves, and so does curl on Linux. The curl of
macOS does not: there, use `curl -H "Host: shop.feature-login.localhost" http://127.0.0.1/`, or
add the host to `/etc/hosts`, as for other tools. Running `siding setup` again changes nothing
unless the port differs.

### In a worktree: start the stack

Expand Down Expand Up @@ -79,6 +82,47 @@ siding lint # exits 1 when it finds an error; --json for a Lint
| `PublishedPort` | info | published ports, which siding resets |
| `NoRoot` | info | HTTP services, but none at `/` |

### Snapshots

```sh
siding snapshot save base # copy this stack's named volumes into the snapshot "base"
siding snapshot ls # the snapshots of this repository
siding snapshot rm base # --dry-run lists it first
```

A snapshot is a copy of every named volume of a stack, kept as Docker volumes on this machine
per repository. While it copies, `save` stops the services that use those volumes, so a
database is copied in a consistent state, and starts them again. Snapshots outlive stacks:
`down --volumes` and `gc` never remove them; `siding teardown --snapshots` does.

### Seeding a new stack

```sh
siding up --seed base # on the stack's first up: its volumes start as copies of "base"
```

A new worktree's stack can start with data instead of empty. On its first `up` (the project
has no containers and no volumes yet), `--seed <snapshot>` creates its volumes as copies of the
snapshot before Compose starts it; a later `up` never seeds again. Per service, the label
`dev.siding.seed` chooses:

```yaml
services:
db:
image: postgres:17
labels:
dev.siding.seed: script # run a command once on the first up
dev.siding.seed.command: "psql -U app -f /seed/dev.sql"
cache:
image: redis:7
labels:
dev.siding.seed: none # never seeded, starts empty
# without the label, or with dev.siding.seed: volume, the volumes come from --seed
```

The seed command runs with `docker compose exec` as soon as the service is ready. If it fails,
the verdict is `Failed` with the reason `SeedFailed` and the command's output.

### Memory budget

```sh
Expand All @@ -87,9 +131,16 @@ siding up --force # start even if the stack does not fit
```

Before `up` starts a stack, siding adds the memory the other stacks use now to what this
stack will need: per service its `mem_limit` (or `deploy.resources.limits.memory`), or
256 MiB without a limit. If that exceeds the budget, `up` refuses with exit code 3 and says
how much is missing. Setting `mem_limit` on big services makes the estimate right. `siding ps`
stack will need: per service the highest use siding has seen for it in this repository, plus a
quarter and at most its limit; else its `mem_limit` (or `deploy.resources.limits.memory`); else
256 MiB. siding learns those peaks whenever it reads memory (`ps`, `ui`, `up`, a Ready `status`)
and keeps them in `~/.siding/peaks.json`, a cache you can delete at any time. If that exceeds the budget, `up` refuses with exit code 3 and says
how much is missing. Setting `mem_limit` on big services makes the estimate right.

With `siding up --wait`, a stack that does not fit waits in line instead, until another stack
frees memory or `--timeout` (3 minutes) is up; then it exits 3. Stacks start in the order they
began to wait. `siding ps` and `siding ui` show the line. A waiting `up` that you stop with
Ctrl-C, or that is killed, leaves the line. `siding ps`
shows the total against the budget.

### Overview
Expand All @@ -104,6 +155,17 @@ return the documents of `siding ps --json` and `siding gc --dry-run --json`. It
machine only, and nothing of it keeps running after Ctrl-C. After upgrading siding, run
`siding setup` once, so the proxy knows the route.

### Builds

```sh
siding setup --builds 2 # at most two stacks build images at once; "default" is half the CPUs
```

When a stack has services with `build:`, `up` takes one of these build slots while Compose
builds and starts it, and waits for a free one first (within `--timeout`). Stacks of images
only never wait. The slots are lock files under `~/.siding/builds`; a stopped or killed `up`
frees its slot at once.

### Logs, address, and taking it down

```sh
Expand Down
9 changes: 9 additions & 0 deletions cmd/siding/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ package main

import (
"context"
"errors"
"fmt"
"os"
"os/signal"
Expand Down Expand Up @@ -41,6 +42,8 @@ func main() {
Getenv: os.Getenv,
OSMachineID: stack.OSMachineID,
Stdin: os.Stdin,
PID: os.Getpid(),
Alive: processAlive,
IsLocalAddress: overview.IsLocalAddress,
IsTerminal: stdinIsTerminal,
Git: git.Client{},
Expand All @@ -56,6 +59,12 @@ func main() {
os.Exit(code)
}

// processAlive reports whether the process pid still runs: signal 0 checks without sending.
func processAlive(pid int) bool {
err := syscall.Kill(pid, 0)
return err == nil || errors.Is(err, syscall.EPERM)
}

// stdinIsTerminal reports whether stdin is a terminal, i.e. a human can answer a question.
func stdinIsTerminal() bool {
fi, err := os.Stdin.Stat()
Expand Down
7 changes: 5 additions & 2 deletions hack/smoke/stacks.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,11 @@ left="$(
docker ps -a --format '{{.Names}}'
docker network ls --format '{{.Name}}'
docker volume ls --format '{{.Name}}'
docker images --format '{{.Repository}}'
)"
if grep -q "^${SIDING_TEST_PREFIX}" <<<"$left"; then
die "objects named ${SIDING_TEST_PREFIX}* are left in the inner daemon: $(grep "^${SIDING_TEST_PREFIX}" <<<"$left" | tr '\n' ' ')"
# Snapshot volumes are named siding-snap_<repo>_..., and the tests' repositories siding-test-*.
pattern="^(${SIDING_TEST_PREFIX}|siding-snap_${SIDING_TEST_PREFIX})"
if grep -Eq "$pattern" <<<"$left"; then
die "test objects are left in the inner daemon: $(grep -E "$pattern" <<<"$left" | tr '\n' ' ')"
fi
echo "smoke-stacks: ok"
20 changes: 3 additions & 17 deletions internal/budget/budget.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,23 +45,9 @@ func ParseSize(s string) (int64, error) {
return n << shift, nil
}

// Estimate is the memory a project will need: per service and replica its memory limit, or
// ServiceDefault without one.
func Estimate(p *compose.Project) int64 {
var sum int64
for _, s := range p.Services {
per := s.MemoryLimit
if per <= 0 {
per = ServiceDefault
}
replicas := s.Replicas
if replicas < 1 {
replicas = 1
}
sum += per * int64(replicas)
}
return sum
}
// Estimate is the memory a project will need without learned peaks: per service and replica its
// memory limit, or ServiceDefault without one.
func Estimate(p *compose.Project) int64 { return EstimateLearned(p, "", nil) }

// Check is the outcome of fitting a stack into the budget.
type Check struct {
Expand Down
101 changes: 101 additions & 0 deletions internal/budget/peaks.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
package budget

import (
"encoding/json"
"fmt"
"os"
"path/filepath"

"github.com/tweinmann/siding/internal/compose"
)

// PeakMargin is added to a learned peak: one-shot stats see the use now, not the maximum.
const PeakMargin = 1.25

// Peaks is the highest memory use siding has seen per "<repository>/<service>". It is kept in
// ~/.siding/peaks.json, a losable cache.
type Peaks map[string]int64

// PeakKey is the key of a service of a repository.
func PeakKey(repo, service string) string { return repo + "/" + service }

func peaksPath(home string) string { return filepath.Join(home, ".siding", "peaks.json") }

// LoadPeaks reads the cache; a missing or broken file is an empty cache.
func LoadPeaks(home string) Peaks {
data, err := os.ReadFile(peaksPath(home))
if err != nil {
return Peaks{}
}
var p Peaks
if err := json.Unmarshal(data, &p); err != nil || p == nil {
return Peaks{}
}
return p
}

// Learn records samples, keeping the highest value per key, and writes the cache atomically. A
// sample another siding process wrote meanwhile may be lost; it is a cache.
func Learn(home string, samples Peaks) error {
if len(samples) == 0 {
return nil
}
p := LoadPeaks(home)
changed := false
for k, v := range samples {
if v > p[k] {
p[k] = v
changed = true
}
}
if !changed {
return nil
}
data, err := json.MarshalIndent(p, "", " ")
if err != nil {
return err
}
path := peaksPath(home)
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return fmt.Errorf("write learned peaks: %w", err)
}
tmp, err := os.CreateTemp(filepath.Dir(path), ".peaks-*.json")
if err != nil {
return fmt.Errorf("write learned peaks: %w", err)
}
if _, err := tmp.Write(append(data, '\n')); err != nil {
_ = tmp.Close()
_ = os.Remove(tmp.Name())
return fmt.Errorf("write learned peaks: %w", err)
}
if err := tmp.Close(); err != nil {
_ = os.Remove(tmp.Name())
return err
}
return os.Rename(tmp.Name(), path)
}

// EstimateLearned is the memory a project of repo will need: per service and replica its
// learned peak plus PeakMargin, capped at its memory limit; else the limit; else
// ServiceDefault.
func EstimateLearned(p *compose.Project, repo string, peaks Peaks) int64 {
var sum int64
for name, s := range p.Services {
per := s.MemoryLimit
if peak, ok := peaks[PeakKey(repo, name)]; ok && peak > 0 {
per = int64(float64(peak) * PeakMargin)
if s.MemoryLimit > 0 && per > s.MemoryLimit {
per = s.MemoryLimit
}
}
if per <= 0 {
per = ServiceDefault
}
replicas := s.Replicas
if replicas < 1 {
replicas = 1
}
sum += per * int64(replicas)
}
return sum
}
67 changes: 67 additions & 0 deletions internal/budget/peaks_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
package budget

import (
"os"
"path/filepath"
"testing"

"github.com/tweinmann/siding/internal/compose"
)

func TestEstimateLearned(t *testing.T) {
t.Parallel()
p := &compose.Project{Services: map[string]compose.Service{
"db": {MemoryLimit: 1024 * mib, Replicas: 1},
"web": {Replicas: 2},
"api": {MemoryLimit: 512 * mib, Replicas: 1},
}}
tests := []struct {
name string
peaks Peaks
want int64
}{
{name: "nothing learned", want: (1024 + 2*256 + 512) * mib},
{name: "a learned peak plus a quarter", peaks: Peaks{"shop/web": 400 * mib}, want: (1024 + 2*500 + 512) * mib},
{name: "capped at the limit", peaks: Peaks{"shop/db": 900 * mib}, want: (1024 + 2*256 + 512) * mib},
{name: "below the limit", peaks: Peaks{"shop/api": 200 * mib}, want: (1024 + 2*256 + 250) * mib},
{name: "another repository's peak does not count", peaks: Peaks{"blog/web": 400 * mib}, want: (1024 + 2*256 + 512) * mib},
}
for _, tt := range tests {
if got := EstimateLearned(p, "shop", tt.peaks); got != tt.want {
t.Errorf("%s: %d MiB, want %d MiB", tt.name, got/mib, tt.want/mib)
}
}
}

func TestLearn(t *testing.T) {
t.Parallel()
home := t.TempDir()
if got := LoadPeaks(home); len(got) != 0 {
t.Errorf("no file: %v", got)
}
if err := Learn(home, Peaks{"shop/web": 400 * mib}); err != nil {
t.Fatal(err)
}
if err := Learn(home, Peaks{"shop/web": 300 * mib, "shop/db": 100 * mib}); err != nil {
t.Fatal(err)
}
got := LoadPeaks(home)
if got["shop/web"] != 400*mib || got["shop/db"] != 100*mib {
t.Errorf("peaks %v", got)
}
entries, _ := os.ReadDir(filepath.Join(home, ".siding"))
if len(entries) != 1 {
t.Errorf("temporary files left: %v", entries)
}

// A broken cache is an empty one, and learning starts anew.
if err := os.WriteFile(filepath.Join(home, ".siding", "peaks.json"), []byte("{not json"), 0o644); err != nil {
t.Fatal(err)
}
if got := LoadPeaks(home); len(got) != 0 {
t.Errorf("broken file: %v", got)
}
if err := Learn(home, Peaks{"shop/web": 10 * mib}); err != nil || LoadPeaks(home)["shop/web"] != 10*mib {
t.Errorf("after a broken file: %v, %v", err, LoadPeaks(home))
}
}
Loading
Loading