feat(bin): reclaim Next.js build output from pooled copies

[tiago-peixoto]

Intent

Make Next.js build-cache reclamation structural instead of something noticed only when the disk fills.

Why now, with measured numbers: on 2026-08-18 one idle Artemis copy held a 15 GB packages/frontend/.next on a volume with 11 GB free; reclaiming that single directory took free space to 27 GB. A 2026-08-07 measurement found ~25.6 GB spread across eight copies. This is pure build output that regenerates, so it is the largest low-risk reclaim available, and it keeps coming back because nothing removes it.

Two pieces were asked for, and only two:

1. Teardown removes the copy's .next before returning it to the pool. That is where the problem is created: a copy goes back to the pool carrying gigabytes nobody will ever use again.
2. A way to sweep idle copies without disturbing a live one.

Explicit constraints the user set:

• NEVER delete from a copy whose task is running. A dev server holding that directory will simply rewrite it, and removing it mid-build is worse than leaving it alone. A copy must be PROVEN unowned before it is touched: no owning task record, plus the same clean-tree and stash checks teardown already makes.
• Report what was reclaimed rather than sweeping silently. A cleanup that says nothing is indistinguishable from one that did nothing, and the operator needs to know where the space went.
• Refuse by shape: do NOT build a daemon, a watcher, a scheduler, or a disk-pressure monitor. The direct path is teardown doing its job plus one sweep entry point a human or firstmate can run. Extra machinery is only justified by naming a concrete blocker in the direct path first. This is a deliberate architectural exclusion, not an oversight.
• Do NOT delete anything that is not regenerable build output. Never node_modules, never source, never git data. .next is the target. Other caches worth including must be NAMED with their measured size rather than folded in by pattern.

Decisions and tradeoffs made while doing the work, which a reviewer reading only the diff would not know:

• One discovery rule lives in bin/fm-next-cache-lib.sh and is shared by both surfaces so "what counts as reclaimable" is stated once. A directory qualifies only if it is named .next AND git ignores it AND its parent is a Next.js app root (next.config.* beside it, or a package.json naming next). Both proofs are deliberate: gitignore status means tracked content can never match, and the app-root proof is what makes "regenerable build output" true rather than assumed, so a gitignored directory that merely happens to be called .next is left alone.
• Verified against current Next.js documentation: distDir defaults to '.next', and cleanDistDir defaults to true, so next build already clears that directory itself on every production build, preserving only .next/cache. Removing it is the same operation the framework performs one build earlier.
• A project with a custom distDir is deliberately NOT discovered. Reading a build config to decide what to delete would make the deletion set depend on untrusted project code. Such a project keeps its cache until someone names the directory. This is the safe direction to be wrong in, and is an intentional gap, not a bug.
• Other caches were measured and deliberately left out rather than swept by pattern, as the user required: a project's .tmp scratch root at ~8.7 GB pooled holds audit output, coverage JSON, browser recordings, and review artifacts - agent work product that nothing regenerates, so it is out of scope. A dist directory measured ~102 MB pooled and is tracked in some projects. .turbo, .cache, .parcel-cache, .vite, and .output all measured zero.
• The sweep requires FOUR independent proofs before touching a copy: the pool reports it available, no task record in this home or any locally registered secondmate home names it as its worktree, its tree is clean, and it has no stashes. The pool lease is checked first on purpose: the worktree pool is shared across firstmate homes (verified: two different clones resolve to the same pool), so treehouse's lease is the only ownership signal that spans all of them. The stash check is an addition beyond what teardown does today, because a stash is unlanded work a clean tree does not show and nobody is watching an idle copy to notice it disappear.
• Every "could not determine" outcome is treated as "owned": an unreadable or unparseable pool, an unrecognized pool status, a path that is not an inspectable git worktree, or a git command that fails all skip the copy. A missing treehouse makes the whole sweep refuse with exit 2 rather than fall back to the checks it can still make.
• A copy whose ownership could not be established is ALWAYS reported by name, even though its size cannot be measured, because the same broken inspection that hides its owner also hides how much it is holding; a size-gated skip line would pass such a copy over in silence.
• One race is left open deliberately and documented in the script header: a copy can be leased between the pool reporting it available and the removal running. Closing it would need a lease this tool does not own, and the consequence is bounded - the copy was leased to start fresh work, so it loses output it was about to regenerate anyway. Losing that race costs a rebuild, not work.
• The sweep never touches the project clone under projects/, only pooled copies. Firstmate reads its clones and only crewmates change them, and measurement confirmed the clones hold no build output anyway.
• Teardown's reclaim runs after every unlanded-work refusal has passed and after the worktree's processes are reaped, so nothing can still be writing the directory it removes, and it is never fatal: the output regenerates from source, so it is never the work product a refusal protects. A test pins that ordering and fails if the reclaim is moved after the pool return.

A later user requirement, accepted and implemented: because the change makes teardown DELETE something before returning a copy, the tests must cover a copy that IS owned and a copy whose ownership CANNOT be determined, not just the clean case. Both are now covered - five owned cases and six cannot-determine cases, each asserting the build output survives.

One unrelated pre-existing bug was found and fixed in a separate commit, deliberately included rather than left broken: on stock macOS Bash 3.2 (what /usr/bin/env bash resolves to on the machines this fleet runs on), . on a missing file terminates the whole shell with status 0, ignoring both the || return 1 beside the source and the enclosing if !. A missing backend adapter therefore ended fm-teardown.sh mid-run while REPORTING SUCCESS, so a caller records a task as cleaned up when nothing was cleaned up and the safety refusal never prints. Bash 5 returns 1 as written, which is why the Linux CI lanes never saw it and tests/fm-teardown.test.sh's herdr preflight case failed only locally. fm_backend_source now tests the adapter file before sourcing it, through one helper the five backends share; the guard is read and written with eval because that file is also sourced from zsh, where ${!var} and printf -v do not mean what they mean in bash. A new case in tests/fm-backend.test.sh fails without the fix.

Three other pre-existing local test failures were investigated and deliberately NOT fixed, because they are host-environment limitations rather than code defects and would pass on CI's Linux runners: tests/fm-kimi-harness.test.sh (python3 on PATH is 3.8.20, and tomllib needs 3.11+), tests/fm-muse-harness.test.sh (macOS SIGKILLs a copied system bash, so the ancestry fixture cannot launch - verified rc=137), and tests/fm-composer-lib.test.sh (half-block glyph matching under Bash 3.2's weak multibyte support). Fixing three more unrelated subsystems would have expanded this task well past its scope.

RE-SCAFFOLD, 2026-08-18: this branch was re-created from upstream/main on the captain's explicit ruling and the work cherry-picked across unchanged. The first attempt was branched from the fork's main, so its rebase gate correctly warned that 35 unrelated fork commits (78 files) would enter the upstream pull request. The captain ruled to make that gate's premise true rather than approve or skip it, and this branch is the result: it now carries exactly four commits relative to upstream/main and nothing else. The carried patch was verified byte-identical to the pre-re-scaffold work - the only difference between the two diffs is one AGENTS.md hunk header line number, because upstream's AGENTS.md is longer than the fork's; the inserted line lands in the same paragraph, under the same anchor sentence.

A fourth commit was added during the re-scaffold and is part of this change: the suite's fixture commits inherited the host's commit.gpgsign setting, which on the upstream base made them fail to write a commit object and killed the script with exit 128 before its first assertion - a crash that prints no "not ok" line and therefore reads as a pass to anything counting failures. Every fixture git invocation now passes -c commit.gpgsign=false explicitly, so these cases depend on no personal signing key and no ambient harness setting.

Deliberately NOT included, because it would import unrelated fork divergence into an upstream pull request: upstream's tests/lib.sh has no commit.gpgsign guard, while the fork's does. On a host that signs commits by default, pristine upstream/main already fails tests/fm-teardown.test.sh (exit 128) and tests/fm-backend.test.sh (exit 1) for that reason alone, before any change here. Those are pre-existing upstream host-environment failures, verified against a clean upstream/main checkout, and fixing them belongs in its own change rather than being folded in here.

What Changed

• Add shared .next eligibility checks and a sweep command that reports or reclaims build output only from available, proven-unowned pooled copies, failing closed when ownership cannot be established.
• Reclaim qualifying Next.js build output during teardown before returning completed copies to the pool, with dry-run support, explicit outcome reporting, documentation, and comprehensive safety coverage.
• Make missing backend adapters return a refusable error instead of terminating callers on stock macOS Bash.

Risk Assessment

✅ Low: Captain, the changes now fail closed: explicit targets are report-only, deleting targets require primary-worktree identity, and duplicate pool fields are rejected.

Testing

The focused cache and backend suites passed on macOS Bash 3.2 after neutralizing the documented ambient signing setting for the backend retry; direct CLI transcripts demonstrate reported idle reclamation, teardown reclamation, source preservation, live-copy protection, and fail-closed unknown ownership.

Evidence: Cache sweep CLI evidence

Dry-run preserved output; reclaim removed only .next; in-use and undetermined copies preserved output and were reported.

OPERATOR CHECK 1: dry-run on an available, unowned pooled copy
sweep: would reclaim 8K from /private/var/folders/r8/cylyt7xd7t50y9x5wc05my380000gn/T/fm-next-cache-sweep.3OKyh7/cli-evidence/pool/1/packages/frontend/.next
sweep: would reclaim 8K from 1 copy
RESULT: build output still exists after dry-run

OPERATOR CHECK 2: reclaim the same idle pooled copy
sweep: reclaimed 8K of Next.js build output from /private/var/folders/r8/cylyt7xd7t50y9x5wc05my380000gn/T/fm-next-cache-sweep.3OKyh7/cli-evidence/pool/1/packages/frontend/.next
sweep: reclaimed 8K from 1 copy
RESULT: build output removed; Next.js config and tracked source remain

OPERATOR CHECK 3: pool reports the copy in use
sweep: skipped-as-owned /private/var/folders/r8/cylyt7xd7t50y9x5wc05my380000gn/T/fm-next-cache-sweep.3OKyh7/cli-evidence/pool/1 (in use by the pool), holding 4K
sweep: nothing to reclaim; 1 copy were skipped as owned (listed above)
RESULT: live copy build output preserved

OPERATOR CHECK 4: ownership cannot be determined
sweep: undetermined /private/var/folders/r8/cylyt7xd7t50y9x5wc05my380000gn/T/fm-next-cache-sweep.3OKyh7/cli-evidence/pool/1 (the pool did not report it available)
sweep: incomplete ownership input: one or more announced pool candidates could not be fully assessed for /private/var/folders/r8/cylyt7xd7t50y9x5wc05my380000gn/T/fm-next-cache-sweep.3OKyh7/cli-evidence/projects/app; reclamation refused
sweep: reclamation incomplete; 1 project could not be inspected
RESULT: exit=1; build_output_preserved=yes

Evidence: Teardown CLI evidence

Teardown reported reclaim, exited successfully, removed .next, and preserved tracked Next.js source.

OPERATOR CHECK: tear down a landed task copy holding Next.js build output
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
●  WATCHER DOWN - SUPERVISION IS OFF
●  1 task(s) in flight, but no live watcher process holds this home lock (last beat: 0s ago).
●  Trust the emitted supervision protocol for this harness; do not use shell & for watcher repair.
●  This is a supervision warning only; the guarded operation WILL still run.
●  repair missing watcher supervision according to the session-start block for this harness; do not use shell &.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
teardown: reclaimed 8K of Next.js build output from /private/var/folders/r8/cylyt7xd7t50y9x5wc05my380000gn/T/fm-next-cache-sweep.yiqbrg/teardown-cli-evidence/wt/packages/frontend/.next
/private/var/folders/r8/cylyt7xd7t50y9x5wc05my380000gn/T/fm-next-cache-sweep.yiqbrg/teardown-cli-evidence/project: already current
teardown task-x1 complete (window firstmate:fm-task-x1, worktree /private/var/folders/r8/cylyt7xd7t50y9x5wc05my380000gn/T/fm-next-cache-sweep.yiqbrg/teardown-cli-evidence/wt)
Backlog: task-x1 just finished. Update data/backlog.md - move task-x1 to Done, keep Done to the 10 most recent, then re-scan Queued and dispatch only work whose blockers are gone and date is due.
RESULT: exit=0; build_output_removed=yes; tracked_next_source_preserved=yes

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 5 issues found → auto-fixed (6) ✅

• 🚨 bin/fm-next-cache-lib.sh:105 - “Never delete anything that is not regenerable build output” is contradicted by checking only whether the .next directory itself is ignored. Git does not track directories, so an ignored .next can contain force-added tracked descendants; line 163 then deletes them, and a later failed pool return leaves tracked content missing. Reject candidates when git ls-files reports any descendant, treating inspection failure as non-reclaimable.
• 🚨 bin/fm-next-cache-sweep.sh:145 - “Every ‘could not determine’ outcome is treated as ‘owned’” is contradicted by this pipeline. Without pipefail, treehouse status may emit valid JSON and exit nonzero while Python exits 0, allowing an available entry to reach deletion despite a failed authoritative pool lookup. Capture and check treehouse’s status before parsing, or protect the pipeline with pipefail.
• 🚨 bin/fm-next-cache-sweep.sh:123 - “A copy must be PROVEN unowned: no owning task record” is contradicted by silently skipping missing state directories, malformed registry entries, and failed metadata reads; the caller also ignores loader failure. An unreadable registered-secondmate record can therefore be omitted and its available, clean copy swept. Make task-record enumeration an all-or-nothing proof and refuse when any local registry or candidate metadata cannot be read.
• 🚨 bin/fm-next-cache-sweep.sh:138 - The required “no owning task record names it as its worktree” proof uses literal grep -Fx path equality. On case-insensitive macOS, differently-cased paths can identify the same directory but fail this comparison; if its lease is available or released, the remaining checks pass and a live task’s cache can be removed. Compare filesystem identity, such as device and inode, at this boundary.
• ⚠️ bin/fm-next-cache-sweep.sh:292 - After a project’s pool lookup fails, RC changes but no incomplete/unknown count is recorded, so the final output can still assert “no idle copy holds Next.js build output.” That contradicts the requested honest reporting because the sweep could not establish that fact. Track incomplete projects and report an incomplete sweep instead.

🔧 Fix: Captain: fail closed on uncertain cache ownership
4 issues (2 errors, 2 warnings) still open:

• 🚨 bin/fm-next-cache-sweep.sh:167 - “No task record ... names it as its worktree” remains bypassable because stat does not follow a final symlink by default. If metadata names /pool/link and treehouse reports its target, the literal paths and lstat inodes differ, so an available, clean copy reaches deletion; a broken final symlink also returns its own inode instead of the required unknown result. Make this boundary compare resolved referents and return undetermined when a final symlink cannot be resolved.
• 🚨 bin/fm-next-cache-sweep.sh:124 - The all-or-nothing task-record proof still accepts a syntactically parseable but non-absolute local home: and then treats it as absent relative to the caller’s current directory. The shared registry contract rejects such homes as unsafe; here the intended home’s task records can be omitted and its copy swept. Before the absence exception, reject any local home that is not an absolute, resolvable registry path.
• ⚠️ bin/fm-next-cache-sweep.sh:216 - “An unreadable or unparseable pool ... skips the copy” is contradicted by silently dropping dictionary entries with no usable path. A status-0 response containing such an entry is accepted as a complete pool and can produce the absolute “no idle copy” summary. Treat a missing or non-string path as a malformed pool and make the whole project incomplete.
• ⚠️ bin/fm-next-cache-sweep.sh:313 - “A copy whose ownership could not be established is ALWAYS reported by name” is contradicted by [ -d "$wt" ] || continue. A named pool entry whose path is missing or inaccessible disappears silently instead of being reported as undetermined. Pass it through the ownership verdict so identity failure produces the existing named skip line.

🔧 Fix: Captain: fail closed on incomplete cache ownership
4 issues (2 errors, 2 warnings) still open:

• 🚨 bin/fm-next-cache-sweep.sh:235 - The required rule that an incomplete ownership check must refuse is still bypassable: both grep comparisons treat every nonzero status as “no match.” If grep is unavailable or exits 2, an exact task-owned copy can be classified free and reclaimed. Distinguish no-match (1) from inspection failure and return the undetermined verdict on the latter.
• 🚨 bin/fm-next-cache-sweep.sh:274 - The malformed-pool refusal does not reject NUL characters. Python accepts fields such as avail\u0000able, but Bash command substitution cannot represent NUL and can collapse that value to available, authorizing deletion without the pool’s exact availability proof. Reject NUL before emitting the line protocol.
• ⚠️ bin/fm-next-cache-sweep.sh:421 - The captain’s requirement that no silent set-narrowing survive remains contradicted: the old git ... || continue was only rewritten as a nested if. With one readable project and one project directory whose Git metadata cannot be inspected, the latter disappears from TARGETS and the sweep can still claim completeness. Record that directory as an incomplete project or refuse discovery.
• ⚠️ bin/fm-next-cache-sweep.sh:464 - If every attempted removal fails, RC becomes 1 but RECLAIMED, SKIPPED, and INCOMPLETE remain zero, so the final summary falsely says no idle copy holds build output even though the failed directory remains. Track removal failures and qualify the summary accordingly.

🔧 Fix: Captain: fail closed across cache ownership inputs
3 issues (2 errors, 1 warning) still open:

• 🚨 bin/fm-next-cache-sweep.sh:530 - The required “sweep never touches the project clone” and task-record ownership proofs are not established by [ -d "$wt" ] plus inode identity. git rev-parse --git-dir also succeeds for repository subdirectories: a pool response naming the project clone deletes its .next, while naming a live copy’s child directory bypasses the task record for its root. Captain, this is the same completeness class after the promised exhaustive enumeration, so the stated boundary calls for removing sweep deletion authority rather than another local patch.
• 🚨 bin/fm-next-cache-sweep.sh:698 - A valid free copy can be inspected and planned before a later malformed entry refuses the project. The plan is then discarded without reporting that copy, but INSPECTED remains incremented and the summary says “nothing to reclaim in 1 copy” even though its .next remains. This contradicts the required honest summary and “ALWAYS reported by name” rule; keep project refusal atomic but treat discarded plans as incomplete, not empty.
• ⚠️ tests/fm-next-cache-sweep.test.sh:1559 - If FM_NEXT_CACHE_TEST contains a misspelled or removed test name, every wrapper invocation silently skips and the suite exits successfully without running an assertion. Track whether the selector matched and fail when it did not, so targeted RED/green evidence cannot be vacuous.

🔧 Fix: Captain: prove pool provenance and honest sweep outcomes
2 errors still open:

• 🚨 bin/fm-next-cache-sweep.sh:845 - The required positive proof that a candidate “must not be the project clone” remains bypassable. sweep_project accepts an explicit project subdirectory because git rev-parse --git-dir succeeds there; provenance then compares the candidate clone root against that subdirectory rather than the actual project root. Thus a pool response naming the clone can pass the worktree-root/registry checks and have its .next deleted. Require the project argument itself to equal its resolved --show-toplevel before pool lookup, or derive the primary clone identity independently. This also disproves the header inventory’s claim that every relevant input was accounted for; under the captain’s stated boundary, deletion authority should be reconsidered rather than applying another local patch.
• 🚨 bin/fm-next-cache-sweep.sh:763 - The requirement that a copy whose ownership cannot be established is “ALWAYS reported by name” is contradicted by returning on the first failed candidate preflight. If a complete pool listing contains an invalid first entry followed by other named copies, those later copies receive no verdict and are never reported, although the whole project is refused. Preserve atomic refusal, but evaluate or explicitly report every remaining pool entry as refused before returning.

🔧 Fix: Correct wrong root verdict and reconcile cache candidates
2 errors still open:

• 🚨 bin/fm-next-cache-sweep.sh:1098 - The required guarantee that “the sweep never touches the project clone” remains bypassable when an explicit argument is another linked-worktree root. Such a path equals its own --show-toplevel, so it passes here; provenance then treats that linked worktree—not the repository’s primary worktree—as the clone to exclude. If the pool response names the real primary clone, it is a distinct registered root and can reach deletion. Prove the supplied project is the primary worktree or derive the primary identity independently. This is the same destructive causal theme after multiple fixes, so I recommend withholding sweep deletion authority until that boundary is proven.
• 🚨 bin/fm-next-cache-sweep.sh:615 - The requirement that ambiguous pool input refuse reclamation is contradicted by parsing JSON with Python’s default duplicate-key behavior. A response such as {"status":"in-use","status":"available","path":"…"} is accepted with the last status winning, allowing conflicting authoritative lease evidence to become available. Reject duplicate object keys during JSON decoding before emitting any candidate rows; otherwise a malformed pool can still authorize deletion.

🔧 Fix: Correct authority verdicts and reject duplicate pool fields
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• /bin/bash tests/fm-next-cache-sweep.test.sh
• /bin/bash tests/fm-backend.test.sh — relevant adapter case passed; later encountered the known ambient GPG-signing fixture issue
• GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=commit.gpgsign GIT_CONFIG_VALUE_0=false /bin/bash tests/fm-backend.test.sh
• Controlled bin/fm-next-cache-sweep.sh --dry-run and reclaim checks against an available, unowned pooled copy
• Controlled sweep checks with pool status in-use and unknown, verifying .next survived
• Controlled bin/fm-teardown.sh task-x1 check for a landed copy holding .next
• git status --short and evidence-file integrity checks

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[tiago-peixoto]

Closing in favour of a split.

The general fixes from this branch - the stock macOS Bash 3.2 defect where sourcing a missing sibling library ends a script silently with status 0, and the teardown paths that swallowed the resulting failure - now live in #2646.

The remaining content here is the Next.js build-output reclamation, which assumes a specific worktree-pool layout and is going to a fork instead of upstream.

No commits from this branch are deleted; it remains available for reference.