Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .agents/skills/secondmate-provisioning/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
name: secondmate-provisioning
description: >-
Load when creating, seeding, validating, launching, recovering, handing backlog to, pushing inherited local material into, or retiring a secondmate home, or when editing data/secondmates.md.
Load when creating, seeding, validating, launching, pausing, resuming, recovering, handing backlog to, pushing inherited local material into, or retiring a secondmate home, or when editing data/secondmates.md.
user-invocable: false
metadata:
internal: true
Expand Down Expand Up @@ -208,9 +208,15 @@ That same rule governs routing generally: a Relay-linked request whose work goes
It refuses any destination that is not a genuine seeded firstmate home with safe operational directories and a matching `.fm-secondmate-home` marker, so a move can never land in a project.
Do not hand off `local-only` items.

## Pause and resume

When asked to pause or resume a secondmate, use the existing lifecycle owner's `--pause` or `--resume` operation described in [`bin/fm-secondmate-restart.sh`](../../../bin/fm-secondmate-restart.sh).
The [operator reference](../../../docs/configuration.md#pause-and-resume) describes supported placement, safety refusals and work routing.
Check `fm_secondmate_is_paused` from the registry library before treating a stopped mate as a recovery candidate; an intentional pause survives a primary restart and must not be repaired as endpoint loss.

## Recovery

For local `kind=secondmate` meta with no window, treat the secondmate as a dead persistent direct report and respawn it with:
For unpaused local `kind=secondmate` meta with no window, treat the secondmate as a dead persistent direct report and respawn it with:

```sh
bin/fm-spawn.sh <id> --secondmate
Expand Down
5 changes: 3 additions & 2 deletions .agents/skills/updatefirstmate/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Self-update firstmate in place.
Firstmate is its own repo, behind the same no-mistakes gate as any project, so new tracked material (`AGENTS.md`, `bin/`, `.agents/skills/`, and public `skills/`) reaches `main` and then sits there until each running firstmate pulls it.
Only `AGENTS.md`, `bin/`, and `.agents/skills/` are a running firstmate instruction surface; public `skills/` is installer-facing and is not loaded by firstmate.
This skill performs that pull for the running main firstmate and every secondmate, without disturbing any in-flight work.
For intentional downtime, follow the [pause and resume contract](../../../docs/configuration.md#pause-and-resume).
Updating is consumption, not publication: it only pulls from upstream, so every home runs it regardless of role, and the firstmate-repository publication boundary (`firstmate-self-repo`) never applies to it.

Pulling the files is only half of it.
Expand Down Expand Up @@ -54,8 +55,8 @@ This touches only the firstmate repo and its own worktrees, never anything under

The two second-mate sets are disjoint and the script owns the split; do not re-derive it.
`restart-secondmates:` carries every live mate the pass left on the latest commit, whether it advanced or was already there.
A mate reaches neither set only because its home was skipped, because it has no live endpoint recorded here, or because its endpoint was positively classified as dead or missing.
A skipped genuine divergence still requires attention through its durable reconciliation record; the other two cases need no update action from you.
A mate reaches neither set because it is intentionally paused, because its home was skipped, because it has no live endpoint recorded here, or because its endpoint was positively classified as dead or missing.
A skipped genuine divergence still requires attention through its durable reconciliation record.

2. **Re-read AGENTS.md if your own instructions changed.**
When the updater printed `reread-firstmate: yes`, the tracked instruction surface (`AGENTS.md`, `bin/`, or `.agents/skills/`) just advanced under you.
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,11 @@ Launching a supported harness inside it for your primary session instantiates yo
- **Two task shapes** - ship tasks deliver authorized changes; scout tasks leave standalone investigation reports when the intake contract warrants separate research.
- **Explicit project modes** - each project ships via `no-mistakes`, `direct-PR`, or `local-only`, with an optional `+yolo` merge-autonomy flag, an optional `branch=<prefix>` override for the default `fm/` ship-branch prefix, and an optional [forge binding](docs/configuration.md#forge-selection-dataprojectsmd-forge-token) for Gerrit changes or Forgejo/Gitea pull requests.
- **Self-repo boundary** - a home runs as firstmate user/operator by default, so a firstmate bug is diagnosed and patched locally but never pushed, PRed, issued, commented, or forked upstream on its own; publishing to firstmate's own repository needs the `config/firstmate-role` maintainer opt-in or a per-task captain consent ([schema](docs/configuration.md#firstmate-role-configfirstmate-role)).
- **Optional secondmates** - opt in to persistent second mates that run from isolated firstmate homes with their own `FM_HOME`, state, projects, and session lock, either locally or as a whole home on an SSH-reachable host, with guarded updates and recovery that never turns an unavailable remote route into a local replacement.
- **Optional secondmates** - opt in to persistent second mates that run from isolated firstmate homes with their own `FM_HOME`, state, projects, and session lock, either locally or as a whole home on an SSH-reachable host, with guarded updates and recovery that never turns an unavailable remote route into a local replacement; see [pause and resume](docs/configuration.md#pause-and-resume) for intentional downtime.
- **Event-driven, zero-token supervision** - a bash watcher sleeps on the fleet and wakes the first mate only when something needs you; verified primary harnesses also get a turn-end backstop that blocks or follows up on a blind stop when work is under way and supervision is not live.
- **Optional Relay** - opt in with one local `.env` pairing token so firstmate can answer your public mentions on X and Discord alike, act on normal reversible mention requests through the same lifecycle as chat requests, acknowledge spawned work, and post up to three public-safe completion follow-ups within seven days for genuine milestones and the final outcome without changing non-Relay behavior; a final reply promised in a thread becomes durable state that is reconciled from disk, so a restart or a compacted conversation cannot lose it; dry-run preview records would-be replies and dismissals locally before go-live.
- **Strict project boundary** - the first mate is read-only over your projects except for the narrow guarded and captain-approved operations authorized by [hard rule 1](AGENTS.md#1-identity-and-prime-directives), including fleet sync's guarded safe branch pruning; crewmates make every other project change behind the configured merge authority.
- **Restart-proof** - all state lives on disk and in the active session backend (tmux by hard default, herdr, rex, or cmux when selected or auto-detected, zellij/orca/paseo/t3code/tern when explicitly selected); the next session reconciles after a restart, while ordinary supervision recovers confirmed-dead secondmate agents without waiting for one.
- **Restart-proof** - all state lives on disk and in the active session backend (tmux by hard default, herdr, rex, or cmux when selected or auto-detected, zellij/orca/paseo/t3code/tern when explicitly selected); the next session reconciles after a restart, while ordinary supervision recovers confirmed-dead, unpaused secondmate agents without waiting for one.

Full detail on every feature lives in [docs/architecture.md](docs/architecture.md).

Expand Down
6 changes: 3 additions & 3 deletions bin/backends/rex.sh
Original file line number Diff line number Diff line change
Expand Up @@ -764,7 +764,7 @@ EOF
# an unreachable server or a bad read is `unreadable`. Read-only: it never
# starts a server.
fm_backend_rex_agent_state() { # <target>
local role pid name path argv0 kind fg_kind='' seen=0 below
local role pid name path argv0 kind fg_kind='' process_seen=0 below
if ! fm_backend_rex_process_chain "$1"; then
case "$FM_BACKEND_REX_READ_CLASS" in
missing) printf 'missing' ;;
Expand All @@ -774,7 +774,7 @@ fm_backend_rex_agent_state() { # <target>
fi
while IFS=$'\t' read -r role pid name path argv0; do
[ -n "$role" ] || continue
seen=1
process_seen=1
kind=$(fm_backend_rex_process_kind "$pid" "$name" "$path" "$argv0")
if [ "$kind" = agent ]; then
printf 'alive'
Expand All @@ -784,7 +784,7 @@ fm_backend_rex_agent_state() { # <target>
done <<EOF
$FM_BACKEND_REX_CHAIN
EOF
if [ "$seen" -eq 0 ]; then
if [ "$process_seen" -eq 0 ]; then
# No foreground process: the terminal child exited (a kept-open block) or
# the block is still starting.
if [ "$FM_BACKEND_REX_LAST_EXIT" = exited ]; then
Expand Down
46 changes: 30 additions & 16 deletions bin/fm-backlog-handoff.sh
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,10 @@
# or undeletable stale marker is reported and ignored by later resumes and
# handoffs, so wake-state cleanup neither suppresses a new wake nor fails a
# completed remote handoff.
# Only a fresh handoff that moves new work grants fm-send --new-work resume
# authority. Pending wake and outbox retries, including --resume-pending, defer
# the wake while the mate is paused; received items remain queued in its backlog
# and pending wake intent is preserved until resume or another fresh handoff.
# Usage: fm-backlog-handoff.sh <secondmate-id> <item-key>...
# fm-backlog-handoff.sh --resume-pending
set -eu
Expand Down Expand Up @@ -550,8 +554,11 @@ receiver_wake_clear_confirmed() { # <secondmate-id>
fi
}

wake_secondmate_receiver() { # <secondmate-id> <correlation-id>
local id=$1 corr=$2 meta="$STATE/$1.meta" out rc=0
wake_secondmate_receiver() { # <secondmate-id> <correlation-id> [new-work]
local id=$1 corr=$2 new_work=${3:-0} meta="$STATE/$1.meta" out rc=0
local -a send_args
send_args=("$id")
[ "$new_work" = 0 ] || send_args+=(--new-work)
if [ ! -f "$meta" ] || [ -L "$meta" ]; then
printf 'error: handed off work to secondmate %s, but no live receiver endpoint is recorded; the destination backlog is durable and the receiver was not woken\n' "$id" >&2
return 1
Expand All @@ -562,7 +569,8 @@ wake_secondmate_receiver() { # <secondmate-id> <correlation-id>
}
out=$(FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$STATE" FM_ROOT_OVERRIDE="$FM_ROOT" \
FM_PENDING_REPLY_EXISTING_CORR="$corr" \
"$SCRIPT_DIR/fm-send.sh" "$id" "$RECEIVER_WAKE_MESSAGE" 2>&1) || rc=$?
"$SCRIPT_DIR/fm-send.sh" "${send_args[@]}" "$RECEIVER_WAKE_MESSAGE" 2>&1) || rc=$?
[ "$rc" -ne 75 ] || return 75
if [ "$rc" -ne 0 ]; then
[ -z "$out" ] || printf '%s\n' "$out" >&2
printf 'error: backlog delivery to secondmate %s succeeded, but its receiver wake failed; retry a tracked remote wake with --resume-pending or a later new handoff, and retry a local wake by rerunning its handoff\n' "$id" >&2
Expand All @@ -571,8 +579,8 @@ wake_secondmate_receiver() { # <secondmate-id> <correlation-id>
[ -z "$out" ] || printf '%s\n' "$out"
}

wake_pending_secondmate_receiver() { # <secondmate-id> [retain-confirmed]
local id=$1 retain=${2:-0} marker="$STATE/.backlog-handoff-$1.wake-pending" value corr rec delivered
wake_pending_secondmate_receiver() { # <secondmate-id> [retain-confirmed] [new-work]
local id=$1 retain=${2:-0} new_work=${3:-0} marker="$STATE/.backlog-handoff-$1.wake-pending" value corr rec delivered wake_rc=0
[ -e "$marker" ] || [ -L "$marker" ] || return 0
if [ ! -f "$marker" ] || [ -L "$marker" ]; then
printf 'error: receiver wake state for secondmate %s is unsafe or invalid\n' "$id" >&2
Expand Down Expand Up @@ -601,11 +609,14 @@ wake_pending_secondmate_receiver() { # <secondmate-id> [retain-confirmed]
fm_pending_reply_reconcile_delivery "$STATE" "$corr" >/dev/null 2>&1 || true
delivered=$(fm_pending_reply_get "$rec" delivered_epoch)
if [ -z "$delivered" ]; then
if [ "$new_work" = 0 ] && fm_secondmate_is_paused "$STATE" "$id"; then return 75; fi
fm_pending_reply_corr_reusable "$STATE" "$corr" "$id" || {
printf 'error: receiver wake delivery for secondmate %s is unresolved; refusing to resend correlation %s\n' "$id" "$corr" >&2
return 1
}
wake_secondmate_receiver "$id" "$corr" || return 1
wake_secondmate_receiver "$id" "$corr" "$new_work" || wake_rc=$?
[ "$wake_rc" -ne 75 ] || return 75
[ "$wake_rc" -eq 0 ] || return 1
fi
if [ "$retain" = 1 ]; then
receiver_wake_state_write "$id" "confirmed:$corr" || {
Expand All @@ -624,8 +635,8 @@ outbox_item_count() { # <path>
awk '/^- \[[ x]\] / { count++ } END { print count + 0 }' "$1"
}

remote_deliver_outbox() { # <secondmate-id> <outbox-path>
local id=$1 outbox=$2 remote_rel receive_out snapshot bytes hash generation counter counter_tmp current marker wake_rc=0 wake_state=pending
remote_deliver_outbox() { # <secondmate-id> <outbox-path> [new-work]
local id=$1 outbox=$2 new_work=${3:-0} remote_rel receive_out snapshot bytes hash generation counter counter_tmp current marker wake_rc=0 wake_state=pending
[ -f "$outbox" ] && [ ! -L "$outbox" ] || {
echo "error: pending outbox is unavailable or unsafe: $outbox" >&2
return 1
Expand Down Expand Up @@ -713,7 +724,7 @@ remote_deliver_outbox() { # <secondmate-id> <outbox-path>
}
fi
if [ "$wake_rc" -eq 0 ]; then
wake_pending_secondmate_receiver "$id" 1 || wake_rc=$?
wake_pending_secondmate_receiver "$id" 1 "$new_work" || wake_rc=$?
fi
rm -f -- "$outbox" || {
echo "error: remote backlog is durable at $id, but local outbox cleanup failed: $outbox" >&2
Expand Down Expand Up @@ -759,7 +770,7 @@ remove_interrupted_source_duplicates() { # <outbox> <keys...>
}

remote_handoff() { # <secondmate-id> <keys...>
local id=$1 outbox section main_section out_section key mv_out
local id=$1 outbox section main_section out_section key mv_out new_work=0
local -a requested to_move already missing in_flight done_items not_queued
shift
requested=("$@")
Expand Down Expand Up @@ -837,7 +848,8 @@ remote_handoff() { # <secondmate-id> <keys...>
# persist. The outbox is already authoritative in that state, so converge by
# deleting only duplicates that tasks-axi itself confirms are dependency-safe.
remove_interrupted_source_duplicates "$outbox" "${requested[@]}" || return 1
remote_deliver_outbox "$id" "$outbox" || return 1
[ "${#to_move[@]}" -eq 0 ] || new_work=1
remote_deliver_outbox "$id" "$outbox" "$new_work" || return 1
echo "handed off ${#requested[@]} item(s) to remote secondmate $id: ${requested[*]}"
[ "${#already[@]}" -eq 0 ] || echo " already staged (recovered): ${already[*]}"
warn_stale_public_commitments "$id" "${requested[@]}"
Expand Down Expand Up @@ -897,7 +909,7 @@ resume_remote_wake() { # <secondmate-id>
[ -e "$DATA/handoff/$id.outbox.md" ] || [ -L "$DATA/handoff/$id.outbox.md" ] || {
receiver_wake_clear_confirmed "$id"
receiver_wake_pending_valid "$id" || return 0
wake_pending_secondmate_receiver "$id"
wake_pending_secondmate_receiver "$id" || [ "$?" -eq 75 ]
}
}

Expand Down Expand Up @@ -1013,7 +1025,7 @@ if [ "${#TO_MOVE[@]}" -eq 0 ]; then
;;
esac
echo "nothing to move: ${ALREADY[*]:-no keys} already present in $SUB_BACKLOG"
wake_pending_secondmate_receiver "$ID" || exit 1
wake_pending_secondmate_receiver "$ID" || { [ "$?" -eq 75 ] || exit 1; }
exit 0
fi

Expand Down Expand Up @@ -1045,8 +1057,10 @@ if [ -e "$WAKE_PENDING_MARKER" ] || [ -L "$WAKE_PENDING_MARKER" ]; then
;;
*)
wake_pending_secondmate_receiver "$ID" || {
echo "error: previous receiver wake for secondmate $ID is unresolved; nothing new was moved" >&2
exit 1
[ "$?" -eq 75 ] || {
echo "error: previous receiver wake for secondmate $ID is unresolved; nothing new was moved" >&2
exit 1
}
}
;;
esac
Expand Down Expand Up @@ -1093,7 +1107,7 @@ receiver_wake_promote_prepared "$ID" "$REQUESTED_BATCH" || {
echo "error: handed off work to secondmate $ID, but durable receiver wake state could not be recorded" >&2
exit 1
}
wake_pending_secondmate_receiver "$ID" || exit 1
wake_pending_secondmate_receiver "$ID" 0 1 || exit 1
if [ "${#ALREADY[@]}" -gt 0 ]; then
echo " already present (skipped): ${ALREADY[*]}"
fi
Expand Down
1 change: 1 addition & 0 deletions bin/fm-bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -501,6 +501,7 @@ secondmate_sync() {
echo "NUDGE_SECONDMATES: secondmate ${id:-unknown}: send failed: retry target has no live secondmate metadata"
continue
}
fm_secondmate_is_paused "$STATE" "$id" && continue
meta_home=$(fm_meta_get "$meta" home)
[ -n "$meta_home" ] || meta_home=$(secondmate_registry_field "$DATA/secondmates.md" "$id" home || true)
if ! validate_secondmate_home "$id" "$meta_home"; then
Expand Down
23 changes: 15 additions & 8 deletions bin/fm-control.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,9 @@
# every uncommitted change. Interrupts first when the task reads
# busy, then submits the harness's exit command. Postcondition:
# the backend's recovery-grade classifier reports the agent gone.
# Already-stopped is success (idempotent). An endpoint that reads
# Already-stopped is success (idempotent). Exit status 3 refuses
# before stop input; status 1 leaves an attempted stop uncertain.
# An endpoint that reads
# `missing` (or, on Rex, `unreadable`, which is how a stopped Rex
# server reads without starting it) is put through the control
# plane's per-backend absence
Expand Down Expand Up @@ -170,19 +172,22 @@ esac
# drive a crewmate's lifecycle (see bin/fm-gate-refuse-lib.sh).
fm_refuse_if_gate_agent

CONTROL_FAILURE_EXIT=1
[ "${2:-}" != exit ] || CONTROL_FAILURE_EXIT=3

if [ -z "${FM_HOME+x}" ] || [ -z "${FM_HOME:-}" ]; then
echo "error: FM_HOME is not set; fm-control refuses to resolve a task without an explicit firstmate home" >&2
exit 1
exit "$CONTROL_FAILURE_EXIT"
fi
[ -d "$FM_HOME" ] || {
echo "error: FM_HOME '$FM_HOME' is not a directory" >&2
exit 1
exit "$CONTROL_FAILURE_EXIT"
}
STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}"
DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}"
[ -d "$STATE" ] || {
echo "error: state dir '$STATE' is missing; fm-control cannot resolve tasks for FM_HOME '$FM_HOME'" >&2
exit 1
exit "$CONTROL_FAILURE_EXIT"
}

# shellcheck source=bin/fm-backend.sh
Expand Down Expand Up @@ -217,7 +222,7 @@ EXIT_RETRIES=${FM_CONTROL_EXIT_RETRIES:-3}

die() { # <message>
echo "error: $1" >&2
exit 1
exit "$CONTROL_FAILURE_EXIT"
}

CONTROL_LOCK=
Expand Down Expand Up @@ -379,7 +384,7 @@ if [ -n "$(fm_meta_get "$META" remote_host)" ]; then
die "task $ID is a remotely placed secondmate on $(fm_meta_get "$META" remote_host); its agent runs outside this home, so no lifecycle action here could verify that it interrupted, stopped, or came back. Drive its lifecycle on that host, and reconcile it through the secondmate recovery path rather than this plane"
fi

fm_backend_validate_task_endpoint "$META" "$ID" || exit 1
fm_backend_validate_task_endpoint "$META" "$ID" || exit "$CONTROL_FAILURE_EXIT"
BACKEND=$FM_BACKEND_VALIDATED_BACKEND
T=$FM_BACKEND_VALIDATED_TARGET
LABEL="fm-$ID"
Expand All @@ -397,9 +402,9 @@ HARNESS=$(fm_control_harness_family "$RECORDED_HARNESS") \
fm_control_harness_supported "$HARNESS" \
|| die "task $ID records harness '${RECORDED_HARNESS:-none}', which has no verified control mechanics; fm-control refuses to guess an interrupt key or exit command"

fm_backend_validate "$BACKEND" || exit 1
fm_backend_validate "$BACKEND" || exit "$CONTROL_FAILURE_EXIT"
if [ "$BACKEND" = t3code ]; then
fm_backend_runtime_check t3code || exit 1
fm_backend_runtime_check t3code || exit "$CONTROL_FAILURE_EXIT"
fi

# --- shared helpers ---------------------------------------------------------
Expand Down Expand Up @@ -751,6 +756,7 @@ do_exit() {
# A busy agent is interrupted first before the exit command is submitted.
case "$(busy_verdict)" in
busy*)
CONTROL_FAILURE_EXIT=1
cancel=$(deliver_interrupt) || return $?
state=$(agent_state)
case "$state" in
Expand Down Expand Up @@ -800,6 +806,7 @@ do_exit() {
# interacting with the typed submit verification. A Ctrl-C over leftover
# composer text only clears it, so Ctrl-U clears the composer first and one
# more Ctrl-C follows if the agent is still alive.
CONTROL_FAILURE_EXIT=1
case "$cmd" in
C-c)
fm_backend_send_key "$BACKEND" "$T" C-u "$LABEL" \
Expand Down
Loading
Loading