diff --git a/.agents/skills/secondmate-provisioning/SKILL.md b/.agents/skills/secondmate-provisioning/SKILL.md index 300f6c6a41a..70cd7709a17 100644 --- a/.agents/skills/secondmate-provisioning/SKILL.md +++ b/.agents/skills/secondmate-provisioning/SKILL.md @@ -1,7 +1,7 @@ --- name: secondmate-provisioning description: >- - Load when creating, seeding, validating, launching, recovering, handing backlog to, pushing inherited local material into, or retiring a secondmate home, or when editing data/secondmates.md. + Load when creating, seeding, validating, launching, pausing, resuming, recovering, handing backlog to, pushing inherited local material into, or retiring a secondmate home, or when editing data/secondmates.md. user-invocable: false metadata: internal: true @@ -208,9 +208,15 @@ That same rule governs routing generally: a Relay-linked request whose work goes It refuses any destination that is not a genuine seeded firstmate home with safe operational directories and a matching `.fm-secondmate-home` marker, so a move can never land in a project. Do not hand off `local-only` items. +## Pause and resume + +When asked to pause or resume a secondmate, use the existing lifecycle owner's `--pause` or `--resume` operation described in [`bin/fm-secondmate-restart.sh`](../../../bin/fm-secondmate-restart.sh). +The [operator reference](../../../docs/configuration.md#pause-and-resume) describes supported placement, safety refusals and work routing. +Check `fm_secondmate_is_paused` from the registry library before treating a stopped mate as a recovery candidate; an intentional pause survives a primary restart and must not be repaired as endpoint loss. + ## Recovery -For local `kind=secondmate` meta with no window, treat the secondmate as a dead persistent direct report and respawn it with: +For unpaused local `kind=secondmate` meta with no window, treat the secondmate as a dead persistent direct report and respawn it with: ```sh bin/fm-spawn.sh --secondmate diff --git a/.agents/skills/updatefirstmate/SKILL.md b/.agents/skills/updatefirstmate/SKILL.md index cda8982937f..eb7a204cdfd 100644 --- a/.agents/skills/updatefirstmate/SKILL.md +++ b/.agents/skills/updatefirstmate/SKILL.md @@ -14,6 +14,7 @@ Self-update firstmate in place. Firstmate is its own repo, behind the same no-mistakes gate as any project, so new tracked material (`AGENTS.md`, `bin/`, `.agents/skills/`, and public `skills/`) reaches `main` and then sits there until each running firstmate pulls it. Only `AGENTS.md`, `bin/`, and `.agents/skills/` are a running firstmate instruction surface; public `skills/` is installer-facing and is not loaded by firstmate. This skill performs that pull for the running main firstmate and every secondmate, without disturbing any in-flight work. +For intentional downtime, follow the [pause and resume contract](../../../docs/configuration.md#pause-and-resume). Updating is consumption, not publication: it only pulls from upstream, so every home runs it regardless of role, and the firstmate-repository publication boundary (`firstmate-self-repo`) never applies to it. Pulling the files is only half of it. @@ -54,8 +55,8 @@ This touches only the firstmate repo and its own worktrees, never anything under The two second-mate sets are disjoint and the script owns the split; do not re-derive it. `restart-secondmates:` carries every live mate the pass left on the latest commit, whether it advanced or was already there. - A mate reaches neither set only because its home was skipped, because it has no live endpoint recorded here, or because its endpoint was positively classified as dead or missing. - A skipped genuine divergence still requires attention through its durable reconciliation record; the other two cases need no update action from you. + A mate reaches neither set because it is intentionally paused, because its home was skipped, because it has no live endpoint recorded here, or because its endpoint was positively classified as dead or missing. + A skipped genuine divergence still requires attention through its durable reconciliation record. 2. **Re-read AGENTS.md if your own instructions changed.** When the updater printed `reread-firstmate: yes`, the tracked instruction surface (`AGENTS.md`, `bin/`, or `.agents/skills/`) just advanced under you. diff --git a/README.md b/README.md index c44a73d0667..61d2b574a64 100644 --- a/README.md +++ b/README.md @@ -47,11 +47,11 @@ Launching a supported harness inside it for your primary session instantiates yo - **Two task shapes** - ship tasks deliver authorized changes; scout tasks leave standalone investigation reports when the intake contract warrants separate research. - **Explicit project modes** - each project ships via `no-mistakes`, `direct-PR`, or `local-only`, with an optional `+yolo` merge-autonomy flag, an optional `branch=` override for the default `fm/` ship-branch prefix, and an optional [forge binding](docs/configuration.md#forge-selection-dataprojectsmd-forge-token) for Gerrit changes or Forgejo/Gitea pull requests. - **Self-repo boundary** - a home runs as firstmate user/operator by default, so a firstmate bug is diagnosed and patched locally but never pushed, PRed, issued, commented, or forked upstream on its own; publishing to firstmate's own repository needs the `config/firstmate-role` maintainer opt-in or a per-task captain consent ([schema](docs/configuration.md#firstmate-role-configfirstmate-role)). -- **Optional secondmates** - opt in to persistent second mates that run from isolated firstmate homes with their own `FM_HOME`, state, projects, and session lock, either locally or as a whole home on an SSH-reachable host, with guarded updates and recovery that never turns an unavailable remote route into a local replacement. +- **Optional secondmates** - opt in to persistent second mates that run from isolated firstmate homes with their own `FM_HOME`, state, projects, and session lock, either locally or as a whole home on an SSH-reachable host, with guarded updates and recovery that never turns an unavailable remote route into a local replacement; see [pause and resume](docs/configuration.md#pause-and-resume) for intentional downtime. - **Event-driven, zero-token supervision** - a bash watcher sleeps on the fleet and wakes the first mate only when something needs you; verified primary harnesses also get a turn-end backstop that blocks or follows up on a blind stop when work is under way and supervision is not live. - **Optional Relay** - opt in with one local `.env` pairing token so firstmate can answer your public mentions on X and Discord alike, act on normal reversible mention requests through the same lifecycle as chat requests, acknowledge spawned work, and post up to three public-safe completion follow-ups within seven days for genuine milestones and the final outcome without changing non-Relay behavior; a final reply promised in a thread becomes durable state that is reconciled from disk, so a restart or a compacted conversation cannot lose it; dry-run preview records would-be replies and dismissals locally before go-live. - **Strict project boundary** - the first mate is read-only over your projects except for the narrow guarded and captain-approved operations authorized by [hard rule 1](AGENTS.md#1-identity-and-prime-directives), including fleet sync's guarded safe branch pruning; crewmates make every other project change behind the configured merge authority. -- **Restart-proof** - all state lives on disk and in the active session backend (tmux by hard default, herdr, rex, or cmux when selected or auto-detected, zellij/orca/paseo/t3code/tern when explicitly selected); the next session reconciles after a restart, while ordinary supervision recovers confirmed-dead secondmate agents without waiting for one. +- **Restart-proof** - all state lives on disk and in the active session backend (tmux by hard default, herdr, rex, or cmux when selected or auto-detected, zellij/orca/paseo/t3code/tern when explicitly selected); the next session reconciles after a restart, while ordinary supervision recovers confirmed-dead, unpaused secondmate agents without waiting for one. Full detail on every feature lives in [docs/architecture.md](docs/architecture.md). diff --git a/bin/backends/rex.sh b/bin/backends/rex.sh index fbc49b18f4c..27564aee49c 100644 --- a/bin/backends/rex.sh +++ b/bin/backends/rex.sh @@ -764,7 +764,7 @@ EOF # an unreachable server or a bad read is `unreadable`. Read-only: it never # starts a server. fm_backend_rex_agent_state() { # - local role pid name path argv0 kind fg_kind='' seen=0 below + local role pid name path argv0 kind fg_kind='' process_seen=0 below if ! fm_backend_rex_process_chain "$1"; then case "$FM_BACKEND_REX_READ_CLASS" in missing) printf 'missing' ;; @@ -774,7 +774,7 @@ fm_backend_rex_agent_state() { # fi while IFS=$'\t' read -r role pid name path argv0; do [ -n "$role" ] || continue - seen=1 + process_seen=1 kind=$(fm_backend_rex_process_kind "$pid" "$name" "$path" "$argv0") if [ "$kind" = agent ]; then printf 'alive' @@ -784,7 +784,7 @@ fm_backend_rex_agent_state() { # done < ... # fm-backlog-handoff.sh --resume-pending set -eu @@ -550,8 +554,11 @@ receiver_wake_clear_confirmed() { # fi } -wake_secondmate_receiver() { # - local id=$1 corr=$2 meta="$STATE/$1.meta" out rc=0 +wake_secondmate_receiver() { # [new-work] + local id=$1 corr=$2 new_work=${3:-0} meta="$STATE/$1.meta" out rc=0 + local -a send_args + send_args=("$id") + [ "$new_work" = 0 ] || send_args+=(--new-work) if [ ! -f "$meta" ] || [ -L "$meta" ]; then printf 'error: handed off work to secondmate %s, but no live receiver endpoint is recorded; the destination backlog is durable and the receiver was not woken\n' "$id" >&2 return 1 @@ -562,7 +569,8 @@ wake_secondmate_receiver() { # } out=$(FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$STATE" FM_ROOT_OVERRIDE="$FM_ROOT" \ FM_PENDING_REPLY_EXISTING_CORR="$corr" \ - "$SCRIPT_DIR/fm-send.sh" "$id" "$RECEIVER_WAKE_MESSAGE" 2>&1) || rc=$? + "$SCRIPT_DIR/fm-send.sh" "${send_args[@]}" "$RECEIVER_WAKE_MESSAGE" 2>&1) || rc=$? + [ "$rc" -ne 75 ] || return 75 if [ "$rc" -ne 0 ]; then [ -z "$out" ] || printf '%s\n' "$out" >&2 printf 'error: backlog delivery to secondmate %s succeeded, but its receiver wake failed; retry a tracked remote wake with --resume-pending or a later new handoff, and retry a local wake by rerunning its handoff\n' "$id" >&2 @@ -571,8 +579,8 @@ wake_secondmate_receiver() { # [ -z "$out" ] || printf '%s\n' "$out" } -wake_pending_secondmate_receiver() { # [retain-confirmed] - local id=$1 retain=${2:-0} marker="$STATE/.backlog-handoff-$1.wake-pending" value corr rec delivered +wake_pending_secondmate_receiver() { # [retain-confirmed] [new-work] + local id=$1 retain=${2:-0} new_work=${3:-0} marker="$STATE/.backlog-handoff-$1.wake-pending" value corr rec delivered wake_rc=0 [ -e "$marker" ] || [ -L "$marker" ] || return 0 if [ ! -f "$marker" ] || [ -L "$marker" ]; then printf 'error: receiver wake state for secondmate %s is unsafe or invalid\n' "$id" >&2 @@ -601,11 +609,14 @@ wake_pending_secondmate_receiver() { # [retain-confirmed] fm_pending_reply_reconcile_delivery "$STATE" "$corr" >/dev/null 2>&1 || true delivered=$(fm_pending_reply_get "$rec" delivered_epoch) if [ -z "$delivered" ]; then + if [ "$new_work" = 0 ] && fm_secondmate_is_paused "$STATE" "$id"; then return 75; fi fm_pending_reply_corr_reusable "$STATE" "$corr" "$id" || { printf 'error: receiver wake delivery for secondmate %s is unresolved; refusing to resend correlation %s\n' "$id" "$corr" >&2 return 1 } - wake_secondmate_receiver "$id" "$corr" || return 1 + wake_secondmate_receiver "$id" "$corr" "$new_work" || wake_rc=$? + [ "$wake_rc" -ne 75 ] || return 75 + [ "$wake_rc" -eq 0 ] || return 1 fi if [ "$retain" = 1 ]; then receiver_wake_state_write "$id" "confirmed:$corr" || { @@ -624,8 +635,8 @@ outbox_item_count() { # awk '/^- \[[ x]\] / { count++ } END { print count + 0 }' "$1" } -remote_deliver_outbox() { # - local id=$1 outbox=$2 remote_rel receive_out snapshot bytes hash generation counter counter_tmp current marker wake_rc=0 wake_state=pending +remote_deliver_outbox() { # [new-work] + local id=$1 outbox=$2 new_work=${3:-0} remote_rel receive_out snapshot bytes hash generation counter counter_tmp current marker wake_rc=0 wake_state=pending [ -f "$outbox" ] && [ ! -L "$outbox" ] || { echo "error: pending outbox is unavailable or unsafe: $outbox" >&2 return 1 @@ -713,7 +724,7 @@ remote_deliver_outbox() { # } fi if [ "$wake_rc" -eq 0 ]; then - wake_pending_secondmate_receiver "$id" 1 || wake_rc=$? + wake_pending_secondmate_receiver "$id" 1 "$new_work" || wake_rc=$? fi rm -f -- "$outbox" || { echo "error: remote backlog is durable at $id, but local outbox cleanup failed: $outbox" >&2 @@ -759,7 +770,7 @@ remove_interrupted_source_duplicates() { # } remote_handoff() { # - local id=$1 outbox section main_section out_section key mv_out + local id=$1 outbox section main_section out_section key mv_out new_work=0 local -a requested to_move already missing in_flight done_items not_queued shift requested=("$@") @@ -837,7 +848,8 @@ remote_handoff() { # # persist. The outbox is already authoritative in that state, so converge by # deleting only duplicates that tasks-axi itself confirms are dependency-safe. remove_interrupted_source_duplicates "$outbox" "${requested[@]}" || return 1 - remote_deliver_outbox "$id" "$outbox" || return 1 + [ "${#to_move[@]}" -eq 0 ] || new_work=1 + remote_deliver_outbox "$id" "$outbox" "$new_work" || return 1 echo "handed off ${#requested[@]} item(s) to remote secondmate $id: ${requested[*]}" [ "${#already[@]}" -eq 0 ] || echo " already staged (recovered): ${already[*]}" warn_stale_public_commitments "$id" "${requested[@]}" @@ -897,7 +909,7 @@ resume_remote_wake() { # [ -e "$DATA/handoff/$id.outbox.md" ] || [ -L "$DATA/handoff/$id.outbox.md" ] || { receiver_wake_clear_confirmed "$id" receiver_wake_pending_valid "$id" || return 0 - wake_pending_secondmate_receiver "$id" + wake_pending_secondmate_receiver "$id" || [ "$?" -eq 75 ] } } @@ -1013,7 +1025,7 @@ if [ "${#TO_MOVE[@]}" -eq 0 ]; then ;; esac echo "nothing to move: ${ALREADY[*]:-no keys} already present in $SUB_BACKLOG" - wake_pending_secondmate_receiver "$ID" || exit 1 + wake_pending_secondmate_receiver "$ID" || { [ "$?" -eq 75 ] || exit 1; } exit 0 fi @@ -1045,8 +1057,10 @@ if [ -e "$WAKE_PENDING_MARKER" ] || [ -L "$WAKE_PENDING_MARKER" ]; then ;; *) wake_pending_secondmate_receiver "$ID" || { - echo "error: previous receiver wake for secondmate $ID is unresolved; nothing new was moved" >&2 - exit 1 + [ "$?" -eq 75 ] || { + echo "error: previous receiver wake for secondmate $ID is unresolved; nothing new was moved" >&2 + exit 1 + } } ;; esac @@ -1093,7 +1107,7 @@ receiver_wake_promote_prepared "$ID" "$REQUESTED_BATCH" || { echo "error: handed off work to secondmate $ID, but durable receiver wake state could not be recorded" >&2 exit 1 } -wake_pending_secondmate_receiver "$ID" || exit 1 +wake_pending_secondmate_receiver "$ID" 0 1 || exit 1 if [ "${#ALREADY[@]}" -gt 0 ]; then echo " already present (skipped): ${ALREADY[*]}" fi diff --git a/bin/fm-bootstrap.sh b/bin/fm-bootstrap.sh index e1eba23e6f3..6f24b00567c 100755 --- a/bin/fm-bootstrap.sh +++ b/bin/fm-bootstrap.sh @@ -501,6 +501,7 @@ secondmate_sync() { echo "NUDGE_SECONDMATES: secondmate ${id:-unknown}: send failed: retry target has no live secondmate metadata" continue } + fm_secondmate_is_paused "$STATE" "$id" && continue meta_home=$(fm_meta_get "$meta" home) [ -n "$meta_home" ] || meta_home=$(secondmate_registry_field "$DATA/secondmates.md" "$id" home || true) if ! validate_secondmate_home "$id" "$meta_home"; then diff --git a/bin/fm-control.sh b/bin/fm-control.sh index 33013f7b06b..0e0a318d076 100755 --- a/bin/fm-control.sh +++ b/bin/fm-control.sh @@ -36,7 +36,9 @@ # every uncommitted change. Interrupts first when the task reads # busy, then submits the harness's exit command. Postcondition: # the backend's recovery-grade classifier reports the agent gone. -# Already-stopped is success (idempotent). An endpoint that reads +# Already-stopped is success (idempotent). Exit status 3 refuses +# before stop input; status 1 leaves an attempted stop uncertain. +# An endpoint that reads # `missing` (or, on Rex, `unreadable`, which is how a stopped Rex # server reads without starting it) is put through the control # plane's per-backend absence @@ -170,19 +172,22 @@ esac # drive a crewmate's lifecycle (see bin/fm-gate-refuse-lib.sh). fm_refuse_if_gate_agent +CONTROL_FAILURE_EXIT=1 +[ "${2:-}" != exit ] || CONTROL_FAILURE_EXIT=3 + if [ -z "${FM_HOME+x}" ] || [ -z "${FM_HOME:-}" ]; then echo "error: FM_HOME is not set; fm-control refuses to resolve a task without an explicit firstmate home" >&2 - exit 1 + exit "$CONTROL_FAILURE_EXIT" fi [ -d "$FM_HOME" ] || { echo "error: FM_HOME '$FM_HOME' is not a directory" >&2 - exit 1 + exit "$CONTROL_FAILURE_EXIT" } STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" [ -d "$STATE" ] || { echo "error: state dir '$STATE' is missing; fm-control cannot resolve tasks for FM_HOME '$FM_HOME'" >&2 - exit 1 + exit "$CONTROL_FAILURE_EXIT" } # shellcheck source=bin/fm-backend.sh @@ -217,7 +222,7 @@ EXIT_RETRIES=${FM_CONTROL_EXIT_RETRIES:-3} die() { # echo "error: $1" >&2 - exit 1 + exit "$CONTROL_FAILURE_EXIT" } CONTROL_LOCK= @@ -379,7 +384,7 @@ if [ -n "$(fm_meta_get "$META" remote_host)" ]; then die "task $ID is a remotely placed secondmate on $(fm_meta_get "$META" remote_host); its agent runs outside this home, so no lifecycle action here could verify that it interrupted, stopped, or came back. Drive its lifecycle on that host, and reconcile it through the secondmate recovery path rather than this plane" fi -fm_backend_validate_task_endpoint "$META" "$ID" || exit 1 +fm_backend_validate_task_endpoint "$META" "$ID" || exit "$CONTROL_FAILURE_EXIT" BACKEND=$FM_BACKEND_VALIDATED_BACKEND T=$FM_BACKEND_VALIDATED_TARGET LABEL="fm-$ID" @@ -397,9 +402,9 @@ HARNESS=$(fm_control_harness_family "$RECORDED_HARNESS") \ fm_control_harness_supported "$HARNESS" \ || die "task $ID records harness '${RECORDED_HARNESS:-none}', which has no verified control mechanics; fm-control refuses to guess an interrupt key or exit command" -fm_backend_validate "$BACKEND" || exit 1 +fm_backend_validate "$BACKEND" || exit "$CONTROL_FAILURE_EXIT" if [ "$BACKEND" = t3code ]; then - fm_backend_runtime_check t3code || exit 1 + fm_backend_runtime_check t3code || exit "$CONTROL_FAILURE_EXIT" fi # --- shared helpers --------------------------------------------------------- @@ -751,6 +756,7 @@ do_exit() { # A busy agent is interrupted first before the exit command is submitted. case "$(busy_verdict)" in busy*) + CONTROL_FAILURE_EXIT=1 cancel=$(deliver_interrupt) || return $? state=$(agent_state) case "$state" in @@ -800,6 +806,7 @@ do_exit() { # interacting with the typed submit verification. A Ctrl-C over leftover # composer text only clears it, so Ctrl-U clears the composer first and one # more Ctrl-C follows if the agent is still alive. + CONTROL_FAILURE_EXIT=1 case "$cmd" in C-c) fm_backend_send_key "$BACKEND" "$T" C-u "$LABEL" \ diff --git a/bin/fm-crew-state.sh b/bin/fm-crew-state.sh index 40bdad02d2a..d6996bb2f70 100755 --- a/bin/fm-crew-state.sh +++ b/bin/fm-crew-state.sh @@ -228,6 +228,11 @@ meta_value kind KIND meta_value harness HARNESS meta_value remote_host REMOTE_HOST [ -n "$KIND" ] || KIND=ship +# shellcheck source=bin/fm-secondmate-registry-lib.sh +. "$SCRIPT_DIR/fm-secondmate-registry-lib.sh" +if [ "$KIND" = secondmate ] && fm_secondmate_is_paused "$STATE" "$ID"; then + emit paused secondmate-pause "intentionally paused; resume explicitly or route new work" +fi # A torn-down (or never-created) worktree has no current state to read. A # remote secondmate's recorded worktree is a path on ITS host, so the local diff --git a/bin/fm-ff-lib.sh b/bin/fm-ff-lib.sh index dbc2ea407eb..aa2234c06d3 100644 --- a/bin/fm-ff-lib.sh +++ b/bin/fm-ff-lib.sh @@ -350,6 +350,7 @@ live_secondmate_meta_records() { [ -f "$meta" ] || continue grep -q '^kind=secondmate$' "$meta" 2>/dev/null || continue id=$(basename "$meta" .meta) + fm_secondmate_is_paused "$state" "$id" && continue home=$(grep '^home=' "$meta" 2>/dev/null | tail -1 | cut -d= -f2- || true) if [ -z "$home" ] && [ -n "$registry" ]; then home=$(secondmate_registry_field "$registry" "$id" home || true) diff --git a/bin/fm-pending-reply-lib.sh b/bin/fm-pending-reply-lib.sh index af39dc5b666..f31d9ac71bd 100755 --- a/bin/fm-pending-reply-lib.sh +++ b/bin/fm-pending-reply-lib.sh @@ -130,6 +130,8 @@ _FM_PENDING_REPLY_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd 2>/dev/n . "$_FM_PENDING_REPLY_LIB_DIR/fm-path-lib.sh" # shellcheck source=bin/fm-backend.sh . "$_FM_PENDING_REPLY_LIB_DIR/fm-backend.sh" +# shellcheck source=bin/fm-secondmate-registry-lib.sh +. "$_FM_PENDING_REPLY_LIB_DIR/fm-secondmate-registry-lib.sh" # Deliberately undirected: this library consumes no symbols from # bin/fm-tmux-lib.sh, so following it under ShellCheck's external-source # traversal would expand that graph for zero cross-file checks. @@ -1163,6 +1165,7 @@ fm_pending_reply_send_recovery() { # # earliest moment a correlated report could exist to race against. _fm_pending_reply_elapsed_at_least "$now" "$completed" "$grace" || return 1 task_id=$(fm_pending_reply_get "$rec" task_id) + fm_secondmate_is_paused "$state" "$task_id" && return 1 # A remote mate's report may exist and simply not have been mirrored yet. fm_pending_reply_missing_report_is_evidence "$state" "$task_id" "$completed" || return 1 # config/wait-no-turns: a mate waiting on its own open decision or blocker @@ -1201,17 +1204,28 @@ fm_pending_reply_send_recovery() { # if [ -n "${FM_PENDING_REPLY_SEND_HOOK:-}" ]; then # Hook receives: task_id message # shellcheck disable=SC2086 - if ! eval "$FM_PENDING_REPLY_SEND_HOOK" "$(printf '%q' "$task_id")" "$(printf '%q' "$msg")"; then - send_status=1 - fi + eval "$FM_PENDING_REPLY_SEND_HOOK" "$(printf '%q' "$task_id")" "$(printf '%q' "$msg")" || send_status=$? else if [ -z "$parent_home" ] || [ ! -d "$parent_home" ]; then send_status=1 - elif ! env FM_HOME="$parent_home" FM_PENDING_REPLY_EXISTING_CORR="$corr" \ - "$_FM_PENDING_REPLY_LIB_DIR/fm-send.sh" "$task_id" "$msg"; then - send_status=1 + else + env FM_HOME="$parent_home" FM_PENDING_REPLY_EXISTING_CORR="$corr" \ + "$_FM_PENDING_REPLY_LIB_DIR/fm-send.sh" "$task_id" "$msg" || send_status=$? fi fi + if [ "$send_status" = 75 ]; then + fm_lock_acquire_wait "$lock" || return 1 + if [ "$(fm_pending_reply_get "$rec" phase)" = recovery_sending ] \ + && [ "$(fm_pending_reply_get "$rec" recovery_sender_pid)" = "$sender_pid" ] \ + && [ "$(fm_pending_reply_get "$rec" recovery_sender_identity)" = "$sender_identity" ]; then + fm_pending_reply_set "$rec" recovery_sender_pid "" \ + && fm_pending_reply_set "$rec" recovery_sender_identity "" \ + && fm_pending_reply_set "$rec" recovery_attempted_epoch "" \ + && fm_pending_reply_set "$rec" phase awaiting_report || true + fi + fm_lock_release "$lock" + return 1 + fi if [ "$send_status" = 0 ]; then fm_pending_reply_finish_recovery "$state" "$corr" confirmed return $? @@ -1643,7 +1657,7 @@ fm_pending_reply_restatement_copy_same_basename() { # [secondmate-home] [allow-recovery] local state=$1 corr=$2 busy_state=$3 sm_home=${4-} allow_recovery=${5:-1} - local rec phase delivered + local rec phase delivered task_id fm_pending_reply_path "$state" "$corr" rec [ -f "$rec" ] || return 1 fm_pending_reply_reconcile_delivery "$state" "$corr" || true @@ -1660,6 +1674,8 @@ fm_pending_reply_tick_one() { # [secondmate- if fm_pending_reply_try_resolve "$state" "$corr"; then return 0 fi + fm_pending_reply_get "$rec" task_id task_id + fm_secondmate_is_paused "$state" "$task_id" && return 0 fm_pending_reply_get "$rec" phase phase case "$phase" in awaiting_report|recovery_sending) @@ -1782,6 +1798,10 @@ fm_pending_reply_tick() { # [watcher-beacon] fm_pending_reply_get "$rec" corr_id corr [ -n "$corr" ] || corr=$(basename "$rec") fm_pending_reply_get "$rec" task_id task_id + if fm_secondmate_is_paused "$state" "$task_id"; then + fm_pending_reply_try_resolve "$state" "$corr" >/dev/null 2>&1 || true + continue + fi fm_pending_reply_get "$rec" phase phase if [ "$phase" = resolved ]; then # Cheap no-op unless an escalation for this record is still open; this is diff --git a/bin/fm-pr-body-preflight.sh b/bin/fm-pr-body-preflight.sh index 43ce0ec23a2..ad529390e59 100755 --- a/bin/fm-pr-body-preflight.sh +++ b/bin/fm-pr-body-preflight.sh @@ -58,6 +58,7 @@ task_temp=$3 if [ -n "$gh_url$forgejo_url" ]; then if [ -n "$forgejo_url" ]; then + # shellcheck source=bin/fm-pr-lib.sh . "$(dirname "${BASH_SOURCE[0]}")/fm-pr-lib.sh" fm_pr_url_parse "$forgejo_url" && [ "$FM_PR_PROVIDER" = forgejo ] || { printf 'error: expected a canonical Forgejo PR URL: %s\n' "$forgejo_url" >&2 diff --git a/bin/fm-precompact-stow.sh b/bin/fm-precompact-stow.sh index 21babd384f9..05d0b341c55 100755 --- a/bin/fm-precompact-stow.sh +++ b/bin/fm-precompact-stow.sh @@ -80,6 +80,7 @@ if [ "$harness" = claude ]; then command -v claude >/dev/null 2>&1 || exit 0; fi lock="$STATE/.precompact-stow.lock" fm_lock_try_acquire "$lock" || exit 0 snapshot= +# shellcheck disable=SC2329 # Invoked by the EXIT trap below. cleanup() { [ -z "$snapshot" ] || rm -f "$snapshot" fm_lock_release "$lock" diff --git a/bin/fm-remote-secondmate-control.sh b/bin/fm-remote-secondmate-control.sh index 2bae758bb3b..9cb3a2aa5f5 100755 --- a/bin/fm-remote-secondmate-control.sh +++ b/bin/fm-remote-secondmate-control.sh @@ -4,6 +4,8 @@ # Usage: # fm-remote-secondmate-control.sh launch herdr [traceparent] # fm-remote-secondmate-control.sh relaunch +# fm-remote-secondmate-control.sh pause +# fm-remote-secondmate-control.sh resume # fm-remote-secondmate-control.sh state # fm-remote-secondmate-control.sh route # fm-remote-secondmate-control.sh send [fire-and-forget] @@ -269,7 +271,7 @@ cmd_relaunch() { FM_STATE_OVERRIDE="$CONTROL_STATE" FM_DATA_OVERRIDE="$CONTROL_DATA" \ FM_CONFIG_OVERRIDE="$TARGET_HOME/config" FM_SKIP_SECONDMATE_INHERIT=1 \ FM_SKIP_SECONDMATE_SYNC=1 \ - "$SCRIPT_DIR/fm-control.sh" "${control_args[@]}" + "$SCRIPT_DIR/fm-control.sh" "${control_args[@]}" || return $? # A parent tracking this route needs the identity the relaunch actually # produced, not the one it asked for, so it can republish its own record the # same way cmd_launch's caller already does. Reading it back from the @@ -279,6 +281,25 @@ cmd_relaunch() { print_route "$id" } +cmd_pause() { + local id=$1 + ( validate_id "$id"; validate_home "$id"; remote_endpoint_require "$id" ) || return 3 + # The endpoint belongs to the host-local parent-route home; its children + # remain in TARGET_HOME/state, which the shared pause preflight inspects. + FM_HOME="$FM_ROOT" FM_ROOT_OVERRIDE="$FM_ROOT" \ + FM_STATE_OVERRIDE="$CONTROL_STATE" FM_DATA_OVERRIDE="$CONTROL_DATA" \ + FM_CONFIG_OVERRIDE="$TARGET_HOME/config" \ + "$SCRIPT_DIR/fm-secondmate-restart.sh" --pause "$id" +} + +cmd_resume() { + local id=$1 + # A successful relaunch is the same verified transaction as normal recovery. + # Keep host-side pause intent on any failure or unknown completion. + cmd_relaunch "$@" || return $? + rm -f -- "$CONTROL_STATE/.secondmate-paused-$id" +} + cmd_send_body() { local id=$1 message=$2 delivery_mode=${3:-} rec ring_rc=0 meta meta_lock validate_id "$id" @@ -465,6 +486,8 @@ cmd_retire() { case "${1:-}" in launch) shift; [ "$#" -ge 5 ] && [ "$#" -le 6 ] || usage; cmd_launch "$@" ;; relaunch) shift; [ "$#" -eq 4 ] || usage; cmd_relaunch "$@" ;; + pause) shift; [ "$#" -eq 1 ] || usage; cmd_pause "$1" ;; + resume) shift; [ "$#" -eq 4 ] || usage; cmd_resume "$@" ;; state) shift; [ "$#" -eq 1 ] || usage; validate_id "$1"; validate_home "$1"; state_value "$1" ;; route) shift; [ "$#" -eq 1 ] || usage; cmd_route "$1" ;; send) shift; [ "$#" -ge 2 ] && [ "$#" -le 3 ] || usage; cmd_send "$@" ;; diff --git a/bin/fm-remote-secondmate-relaunch.sh b/bin/fm-remote-secondmate-relaunch.sh index 7e704d22e27..e2d28e81137 100755 --- a/bin/fm-remote-secondmate-relaunch.sh +++ b/bin/fm-remote-secondmate-relaunch.sh @@ -2,7 +2,7 @@ # Relaunch a REMOTE secondmate onto a new harness, model, or effort, then # republish this parent's own route record to match what the host confirmed. # -# Usage: fm-remote-secondmate-relaunch.sh +# Usage: fm-remote-secondmate-relaunch.sh [resume] # # bin/fm-remote-secondmate-control.sh's relaunch verb runs entirely on the # secondmate's own host and can only rewrite that host's own endpoint record; @@ -33,7 +33,9 @@ STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" die() { printf 'error: %s\n' "$1" >&2; exit 1; } usage() { sed -n '2,4p' "$0" | sed 's/^# \{0,1\}//'; exit 2; } -[ "$#" -eq 4 ] || usage +[ "$#" -ge 4 ] && [ "$#" -le 5 ] || usage +VERB=${5:-relaunch} +case "$VERB" in relaunch|resume) ;; *) usage ;; esac ID=$1 HARNESS=$2 MODEL=$3 @@ -46,8 +48,26 @@ REMOTE_HOST=$(fm_meta_get "$META" remote_host) [ -n "$REMOTE_HOST" ] \ || die "task $ID is not a remotely placed secondmate; use bin/fm-control.sh $ID relaunch instead" +if [ "$VERB" = resume ]; then + # shellcheck source=bin/fm-ff-lib.sh + . "$SCRIPT_DIR/fm-ff-lib.sh" + # shellcheck source=bin/fm-secondmate-nudge-lib.sh + . "$SCRIPT_DIR/fm-secondmate-nudge-lib.sh" + # shellcheck source=bin/fm-lease-lib.sh + . "$SCRIPT_DIR/fm-lease-lib.sh" + INHERIT_LOCK= + trap 'fm_lease_guard_release; [ -z "$INHERIT_LOCK" ] || fm_lock_release "$INHERIT_LOCK" || true' EXIT + INHERIT_LOCK=$(fm_remote_inherit_transaction_lock_path "$STATE" "$ID") || exit 1 + fm_lock_acquire_wait "$INHERIT_LOCK" || exit 1 + fm_lease_guard "$ID" "resume remote secondmate" || exit 1 + PRIMARY_COMMIT=$(primary_head_commit "$FM_ROOT") || die "cannot resolve the primary instruction commit" + GENERATION=$(fm_remote_inherit_generation_next "$STATE" "$ID") || die "cannot publish inheritance generation" + "$SCRIPT_DIR/fm-on.sh" "$ID" fm-remote-secondmate-control.sh sync "$ID" "$PRIMARY_COMMIT" &1) || { + "$VERB" "$ID" "$HARNESS" "$MODEL" "$EFFORT" &1) || { rc=$? printf '%s\n' "$RELAUNCH_OUT" >&2 exit "$rc" @@ -93,3 +113,7 @@ done < "$META" chmod 0600 "$META_TMP" mv -f -- "$META_TMP" "$META" fm_lock_release "$META_LOCK" + +if [ "$VERB" = resume ]; then + rm -f -- "$STATE/.secondmate-paused-$ID" +fi diff --git a/bin/fm-secondmate-liveness-lib.sh b/bin/fm-secondmate-liveness-lib.sh index 2a1ae3ba829..ca72bd1296f 100644 --- a/bin/fm-secondmate-liveness-lib.sh +++ b/bin/fm-secondmate-liveness-lib.sh @@ -63,6 +63,9 @@ FM_SM_LIVE_LIB_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" # shellcheck source=bin/fm-timeout-lib.sh . "$FM_SM_LIVE_LIB_DIR/fm-timeout-lib.sh" +# shellcheck source=bin/fm-secondmate-registry-lib.sh +. "$FM_SM_LIVE_LIB_DIR/fm-secondmate-registry-lib.sh" + # Per-task probe+kill+relaunch serialization. A busy lock means another # supervisor (the other sweep, or a racing tick) is mid-episode on this mate; # callers skip and let that episode finish rather than probe a moving target. @@ -142,6 +145,11 @@ fm_secondmate_liveness_probe() { # FM_SM_LIVE_STATUS=skipped FM_SM_LIVE_STATE=unknown FM_SM_LIVE_KILL=0 FM_SM_LIVE_RESUME=0 FM_SM_LIVE_CAUSE='' FM_SM_LIVE_WHERE='' FM_SM_LIVE_REASON='' FM_SM_LIVE_LINE='' local window harness remote_host remote_rc out agent_state readiness_reason route_out remote_backend + if fm_secondmate_is_paused "$STATE" "$id"; then + FM_SM_LIVE_STATUS=silent + FM_SM_LIVE_REASON="intentionally paused; use fm-secondmate-restart.sh --resume $id" + return 0 + fi window=$(fm_meta_get "$meta" window) [ -n "$window" ] || { FM_SM_LIVE_STATUS=silent; return 0; } harness=$(fm_meta_get "$meta" harness) diff --git a/bin/fm-secondmate-reconcile.sh b/bin/fm-secondmate-reconcile.sh index 9dca24702c7..086d07b9de2 100755 --- a/bin/fm-secondmate-reconcile.sh +++ b/bin/fm-secondmate-reconcile.sh @@ -89,6 +89,8 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}}" STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +# shellcheck source=bin/fm-secondmate-registry-lib.sh +. "$SCRIPT_DIR/fm-secondmate-registry-lib.sh" # shellcheck source=bin/fm-wake-lib.sh . "$SCRIPT_DIR/fm-wake-lib.sh" @@ -463,6 +465,7 @@ cmd_notify() { local id sampled_spawn_gen sampled_host expected_remote_host kind path last age now delivered_at reconcile_lock control_lock meta meta_lock did send_rc while IFS=$'\037' read -r id sampled_spawn_gen sampled_host kind; do [ -n "${id:-}" ] || continue + fm_secondmate_is_paused "$STATE" "$id" && continue path=$(nudge_path "$id") reconcile_lock="$STATE/.$id.reconcile.lock" if ! fm_lock_try_acquire "$reconcile_lock"; then diff --git a/bin/fm-secondmate-registry-lib.sh b/bin/fm-secondmate-registry-lib.sh index d2cb6555830..4c649c742fc 100644 --- a/bin/fm-secondmate-registry-lib.sh +++ b/bin/fm-secondmate-registry-lib.sh @@ -28,6 +28,12 @@ SECONDMATE_REGISTRY_MATCH_PROJECTS= SECONDMATE_REGISTRY_MATCH_REMOTE=0 SECONDMATE_REGISTRY_ERROR= +# Pause records are owned by fm-secondmate-restart.sh. Treat even an unsafe +# marker as paused so routine recovery never turns corrupt state into a launch. +fm_secondmate_is_paused() { # + [ -e "$1/.secondmate-paused-$2" ] || [ -L "$1/.secondmate-paused-$2" ] +} + secondmate_registry_lock_path() { printf '%s/.secondmate-registry.lock\n' "$1"; } secondmate_reply_lifecycle_lock_path() { printf '%s/.remote-reply-lifecycle-%s.lock\n' "$1" "$2"; } diff --git a/bin/fm-secondmate-restart-lib.sh b/bin/fm-secondmate-restart-lib.sh index 3a121c559fe..287f13cbab3 100644 --- a/bin/fm-secondmate-restart-lib.sh +++ b/bin/fm-secondmate-restart-lib.sh @@ -103,3 +103,51 @@ fm_secondmate_restart_capable() { # FM_SECONDMATE_RESTART_HARNESS=$family return 0 } + +# Caller holds the home's task-set lock through the stop, preventing a child +# spawn from publishing between this preflight and the verified agent exit. +# Like retirement's preflight, any child record still owns potentially unlanded +# work even when its endpoint is stopped or its last status says done. +fm_secondmate_pause_home_valid() { # + local home=$1 id=$2 + [ -d "$home" ] && [ ! -L "$home" ] \ + && [ -f "$home/.fm-secondmate-home" ] && [ ! -L "$home/.fm-secondmate-home" ] \ + && [ "$(cat "$home/.fm-secondmate-home")" = "$id" ] \ + && [ -d "$home/state" ] && [ ! -L "$home/state" ] \ + || { echo "error: $id has no safe seeded home to pause" >&2; return 1; } +} + +fm_secondmate_pause_safe() { # + local home=$1 id=$2 child dirty backend backlog rows count + fm_secondmate_pause_home_valid "$home" "$id" || return 1 + for child in "$home/state"/*.meta "$home/state"/*.backlog-close; do + if [ -e "$child" ] || [ -L "$child" ]; then + echo "error: cannot pause $id: in-flight or unlanded work remains at $child; finish and land it first" >&2 + return 1 + fi + done + # Ask the backlog's existing owner, including non-markdown adapters, rather + # than assuming absent child metadata proves there is no in-flight work. + # shellcheck source=bin/fm-tasks-axi-lib.sh + . "$_FM_SECONDMATE_RESTART_LIB_DIR/fm-tasks-axi-lib.sh" + # shellcheck source=bin/fm-backlog-transition-lib.sh + . "$_FM_SECONDMATE_RESTART_LIB_DIR/fm-backlog-transition-lib.sh" + backend=$(fm_tasks_axi_backend "$home") || return 1 + backlog="$home/data/backlog.md" + if [ "$backend" != markdown ] || [ -e "$backlog" ] || [ -L "$backlog" ]; then + if [ "$backend" = markdown ] && { [ ! -f "$backlog" ] || [ -L "$backlog" ]; }; then + echo "error: cannot pause $id: unsafe backlog" >&2; return 1 + fi + rows=$(fm_backlog_row_list "$home/data" --state in_flight) || { + echo "error: cannot inspect $id's in-flight backlog: $rows" >&2; return 1; + } + count=$(printf '%s\n' "$rows" | sed -n 's/^count: //p') + [ "$count" = 0 ] || { + echo "error: cannot pause $id: in-flight backlog work remains or its count is unreadable" >&2; return 1; + } + fi + dirty=$(git -C "$home" status --porcelain 2>/dev/null) \ + || { echo "error: cannot inspect $id's home before pausing" >&2; return 1; } + [ -z "$dirty" ] \ + || { echo "error: cannot pause $id: its home has uncommitted work" >&2; return 1; } +} diff --git a/bin/fm-secondmate-restart.sh b/bin/fm-secondmate-restart.sh index ceebe6455e5..fa97c73a1a5 100755 --- a/bin/fm-secondmate-restart.sh +++ b/bin/fm-secondmate-restart.sh @@ -3,6 +3,26 @@ # wiring, persisting their open records first. # # Usage: fm-secondmate-restart.sh ... [--help] +# fm-secondmate-restart.sh --pause +# fm-secondmate-restart.sh --resume +# +# Pause stops only the agent, retaining its home, backlog, route, endpoint and +# open records. It refuses any child metadata or pending backlog-close record +# (including stopped workers with unlanded work), in-flight backlog rows, or a +# dirty home. The home's task-set lock excludes child spawns through the stop. +# Unsupported stop or replacement backends refuse without stopping the agent. +# state/.secondmate-paused- is the durable parent-owned pause record. +# It is written before stopping; an uncertain stop leaves it present so liveness +# cannot silently undo the request. Ordinary convergence and nudges skip it. +# Resume runs guarded instruction and inherited-config convergence before +# launching a fresh agent through the control/spawn owners, clearing the marker only +# after success. For a local mate it renews an existing home wake-queue progress +# timestamp under the lifecycle lock, preserving its row key, ring marker, +# pending records and delivery counters; paused time cannot become stall time. +# Only explicit resume or fm-send --new-work resumes a paused mate; backlog +# handoff grants that authority only to fresh work after making it durable. +# Remote pause/resume uses fm-on and the same host-local lifecycle owners. +# Both operations are idempotent, take one exact bare id, and require FM_HOME. # # This is the executable half of /updatefirstmate's reload step. A running agent # holds AGENTS.md and every skill it has loaded frozen from launch, and no @@ -68,7 +88,10 @@ # # Exit status: 0 every named mate restarted; 3 at least one was nudged or left # unreached and every mate was still accounted for; 1 the input itself is -# unusable; 2 invalid use. +# unusable; 2 invalid use. Pause returns 3 for a safety refusal before stopping +# or 6 for a supervision-lease refusal, removing newly written intent on a +# confirmed pre-stop refusal. Uncertain stop or transport completion retains +# intent and propagates its failure status; preexisting intent is preserved. set -u SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -93,10 +116,103 @@ STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" # shellcheck source=bin/fm-secondmate-restart-lib.sh . "$SCRIPT_DIR/fm-secondmate-restart-lib.sh" +# shellcheck source=bin/fm-secondmate-registry-lib.sh +. "$SCRIPT_DIR/fm-secondmate-registry-lib.sh" # shellcheck source=bin/fm-secondmate-nudge-lib.sh . "$SCRIPT_DIR/fm-secondmate-nudge-lib.sh" # shellcheck source=bin/fm-pending-reply-lib.sh . "$SCRIPT_DIR/fm-pending-reply-lib.sh" +# shellcheck source=bin/fm-wake-lib.sh +. "$SCRIPT_DIR/fm-wake-lib.sh" + +# Explicit pause/resume is an operator lifecycle action, not an update sweep. +# Serialize with liveness before publishing the pause intent or replacing it. +case "${1:-}" in + --pause|--resume) + [ "$#" -eq 2 ] || { usage >&2; exit 2; } + action=${1#--}; id=$2 + preflight_exit=1 + [ "$action" != pause ] || preflight_exit=3 + case "$id" in ''|*[!A-Za-z0-9._-]*|.|..) echo "error: invalid secondmate id: $id" >&2; exit 2 ;; esac + # shellcheck source=bin/fm-gate-refuse-lib.sh + . "$SCRIPT_DIR/fm-gate-refuse-lib.sh" + fm_refuse_if_gate_agent + # shellcheck source=bin/fm-lease-lib.sh + . "$SCRIPT_DIR/fm-lease-lib.sh" + lifecycle_lock="$STATE/.secondmate-liveness-$id.lock" + child_set_lock= + # shellcheck disable=SC2329 # Invoked by the EXIT trap below. + cleanup_lifecycle() { + [ -z "$child_set_lock" ] || fm_lock_release "$child_set_lock" || true + fm_lease_guard_release || true + fm_lock_release "$lifecycle_lock" || true + } + fm_lock_try_acquire "$lifecycle_lock" || { echo "error: lifecycle action already in progress for $id" >&2; exit "$preflight_exit"; } + trap cleanup_lifecycle EXIT + fm_secondmate_restart_capable "$STATE/$id.meta" || { + echo "error: $id: $FM_SECONDMATE_RESTART_REASON" >&2; exit "$preflight_exit"; + } + marker="$STATE/.secondmate-paused-$id" + if [ -L "$marker" ] || { [ -e "$marker" ] && [ ! -f "$marker" ]; }; then + echo "error: unsafe pause record for $id" >&2; exit "$preflight_exit" + fi + was_paused=0 + if fm_secondmate_is_paused "$STATE" "$id"; then was_paused=1; fi + if [ "$action" = pause ]; then + fm_lease_guard "$id" "pause secondmate" || exit 1 + if [ "$FM_SECONDMATE_RESTART_PLACEMENT" = local ]; then + mate_home=$(fm_meta_get "$STATE/$id.meta" worktree) + fm_secondmate_pause_home_valid "$mate_home" "$id" || exit 3 + child_set_lock=$(fm_task_set_lock_path "$mate_home/state") || exit 3 + fm_lock_try_acquire "$child_set_lock" || { child_set_lock=; echo "error: $id's task set is busy; retry pause" >&2; exit 3; } + fm_secondmate_pause_safe "$mate_home" "$id" || exit 3 + fi + (umask 077; printf '%s\n' "$(date +%s)" > "$marker") || { + [ "$was_paused" = 1 ] || rm -f -- "$marker" + exit 3 + } + if [ "$FM_SECONDMATE_RESTART_PLACEMENT" = remote ]; then + "$SCRIPT_DIR/fm-on.sh" "$id" fm-remote-secondmate-control.sh pause "$id" selector; the # bare id is equally acceptable so a hand-run stays natural. id=${arg#fm-} + if fm_secondmate_is_paused "$STATE" "$id"; then + printf 'skipped: %s intentionally paused\n' "$id" + continue + fi case "$id" in ''|*[!A-Za-z0-9._-]*) echo "error: invalid second mate id: $arg" >&2; exit 2 ;; esac case " ${IDS[*]:-} " in *" $id "*) continue ;; esac IDS+=("$id") done -[ "${#IDS[@]}" -gt 0 ] || { usage >&2; exit 2; } +[ "${#IDS[@]}" -gt 0 ] || exit 0 # Per-mate pass state, kept as parallel indexed arrays so this stays bash-3.2 # safe. PLAN is the phase the mate reached: persist-sent, or fallback with the @@ -219,8 +339,18 @@ report_unreached() { # } restart_mate() { # - local i=$1 id restart_out restart_rc restart_reason ran_on + local i=$1 id restart_out restart_rc restart_reason ran_on lifecycle_lock id=${IDS[$i]} + lifecycle_lock="$STATE/.secondmate-liveness-$id.lock" + fm_lock_try_acquire "$lifecycle_lock" || { + report_unreached "$id" "lifecycle action already in progress" + return + } + if fm_secondmate_is_paused "$STATE" "$id"; then + printf 'skipped: %s intentionally paused\n' "$id" + fm_lock_release "$lifecycle_lock" + return + fi if [ "${PLACEMENT[i]}" = remote ]; then restart_out=$(FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$STATE" \ "$SCRIPT_DIR/fm-remote-secondmate-relaunch.sh" \ @@ -231,6 +361,7 @@ restart_mate() { # "$SCRIPT_DIR/fm-control.sh" "$id" relaunch 2>&1) restart_rc=$? fi + fm_lock_release "$lifecycle_lock" if [ "$restart_rc" -eq 0 ]; then ran_on=$(printf '%s\n' "$restart_out" | sed -n 's/^relaunched .* harness=\([^ ]*\).*/\1/p' | tail -1) [ -n "$ran_on" ] || ran_on=${HARNESS[i]} @@ -291,6 +422,7 @@ harvest_restarts() { case "$out" in restarted:*) restarted_count=$((restarted_count + 1)) ;; nudged:*) nudged_count=$((nudged_count + 1)) ;; + skipped:*) ;; *) unreached_count=$((unreached_count + 1)) ;; esac PLAN[i]="done" diff --git a/bin/fm-send.sh b/bin/fm-send.sh index 384ab173c75..41019f9d2c2 100755 --- a/bin/fm-send.sh +++ b/bin/fm-send.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # Steer a task by durable record: write the message into the task's steering # inbox and ring a constant doorbell through its backend endpoint, best-effort. -# Usage: fm-send.sh [--resolve-key ]... [--fire-and-forget ] +# Usage: fm-send.sh [--new-work] [--resolve-key ]... [--fire-and-forget ] # may be an exact task id, a legacy fm- task label resolved # through this home's state/.meta, or an explicit well-formed backend # target. fm-send refuses unresolved guesses rather than falling back to a @@ -42,6 +42,10 @@ # fire-and-forget request instead retries with its same caller-supplied delivery # id. A still-unconfirmed reply-bearing request keeps its reply expectation # preserved for the record that may have landed. +# --new-work resumes a paused secondmate through fm-secondmate-restart before +# delivery. Backlog handoff sets it after making queued work durable. Other text +# sends refuse a paused or lifecycle-busy mate with status 75 and no delivery, +# retaining maintenance retries. # Pending-reply bookkeeping trouble after a durable enqueue NEVER exits # nonzero: with the recovery marker stored the watcher reconciles it silently, # and with both the commit and the marker lost the send prints a distinct @@ -452,6 +456,12 @@ fm_send_resolve_target() { # return 1 } +SEND_LIFECYCLE_LOCK= +fm_send_cleanup() { + fm_lease_guard_release + [ -z "$SEND_LIFECYCLE_LOCK" ] || fm_lock_release "$SEND_LIFECYCLE_LOCK" +} + RAW_TARGET=$1 fm_send_resolve_target "$RAW_TARGET" || exit 1 T=$RESOLVED_TARGET @@ -467,7 +477,7 @@ if [ -n "$TARGET_META" ]; then LEASE_GUARD_TASK=$(fm_send_id_from_meta "$TARGET_META") if [ -n "$LEASE_GUARD_TASK" ]; then fm_lease_guard "$LEASE_GUARD_TASK" "steer (fm-send)" - trap 'fm_lease_guard_release' EXIT + trap fm_send_cleanup EXIT fi fi @@ -477,6 +487,7 @@ fi RESOLVE_KEYS= FIRE_AND_FORGET_ID= FIRE_AND_FORGET_SET=0 +NEW_WORK=0 fm_send_add_resolve_key() { # local k=$1 case "$k" in @@ -495,6 +506,10 @@ fm_send_add_resolve_key() { # } while :; do case "${1:-}" in + --new-work) + NEW_WORK=1 + shift + ;; --resolve-key) [ $# -ge 2 ] || { echo "error: --resolve-key requires a key" >&2 @@ -795,6 +810,7 @@ if [ "${1:-}" = "--key" ]; then exit 1 ;; esac + [ "$NEW_WORK" = 0 ] || { echo "error: --new-work requires a text request" >&2; exit 1; } key=$2 semantic_key=$(fm_send_normalize_key "$key") if [ "$TARGET_BACKEND" = remote ]; then @@ -827,6 +843,27 @@ else if [ "$TARGET_BACKEND" = remote ]; then fm_send_remote_budget || exit 1 fi + if [ "$MARK_FROM_FIRSTMATE" = 1 ]; then + # shellcheck source=bin/fm-secondmate-registry-lib.sh + . "$SCRIPT_DIR/fm-secondmate-registry-lib.sh" + fm_lease_guard_release + if [ "$NEW_WORK" = 1 ] && fm_secondmate_is_paused "$STATE" "$TARGET_TASK_ID"; then + "$SCRIPT_DIR/fm-secondmate-restart.sh" --resume "$TARGET_TASK_ID" || exit 1 + fm_send_resolve_target "$RAW_TARGET" || exit 1 + T=$RESOLVED_TARGET + fi + lifecycle_lock="$STATE/.secondmate-liveness-$TARGET_TASK_ID.lock" + fm_lock_try_acquire "$lifecycle_lock" || { + echo "error: secondmate $TARGET_TASK_ID lifecycle is busy; retry delivery" >&2; exit 75; + } + SEND_LIFECYCLE_LOCK=$lifecycle_lock + trap fm_send_cleanup EXIT + fm_lease_guard "$TARGET_TASK_ID" "steer (fm-send)" + if fm_secondmate_is_paused "$STATE" "$TARGET_TASK_ID"; then + echo "error: secondmate $TARGET_TASK_ID is intentionally paused; request not sent" >&2 + exit 75 + fi + fi # The pre-marker answer text, kept for the closing resolved note so the # durable ledger records the plain answer without marker or corr bytes. RESOLVE_ANSWER_TEXT=$MESSAGE diff --git a/bin/fm-t3-answer.sh b/bin/fm-t3-answer.sh index 5230c3455d7..7336bf9f821 100755 --- a/bin/fm-t3-answer.sh +++ b/bin/fm-t3-answer.sh @@ -43,8 +43,10 @@ if [ "$#" -eq 1 ]; then exit fi +# shellcheck source=bin/fm-gate-refuse-lib.sh . "$SCRIPT_DIR/fm-gate-refuse-lib.sh" fm_refuse_if_gate_agent +# shellcheck source=bin/fm-lease-lib.sh . "$SCRIPT_DIR/fm-lease-lib.sh" fm_lease_guard "$ID" "answer (fm-t3-answer)" trap 'fm_lease_guard_release' EXIT diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index 5e0d86f77b1..3091bfaed70 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -1231,7 +1231,8 @@ remote_secondmate_teardown() { fm_backlog_atomic_transition remove "$STATE/$ID.meta" "task record" "$STATE" || return 1 rm -f -- "$STATE/$ID.turn-ended" "$STATE/$ID.progress" \ "$(fm_wake_signal_seen_path "$STATE" "$STATE/$ID.turn-ended")" \ - "$STATE/.secondmate-relaunch-$ID" "$STATE/.secondmate-relaunch-bound-$ID" + "$STATE/.secondmate-relaunch-$ID" "$STATE/.secondmate-relaunch-bound-$ID" \ + "$STATE/.secondmate-paused-$ID" printf 'teardown %s complete (remote %s:%s)\n' "$ID" "$remote_host" "$remote_home" return "$rc" } @@ -4124,7 +4125,8 @@ rm -f "$STATE/$ID.turn-ended" "$STATE/$ID.progress" \ "$STATE/$ID.ready-timeout" \ "$STATE/$ID.polytoken-busy.sh" \ "$STATE/.$ID.branch-outcome-index" \ - "$STATE/.secondmate-relaunch-$ID" "$STATE/.secondmate-relaunch-bound-$ID" + "$STATE/.secondmate-relaunch-$ID" "$STATE/.secondmate-relaunch-bound-$ID" \ + "$STATE/.secondmate-paused-$ID" # The steering inbox (bin/fm-task-inbox-lib.sh) is runtime state for the # retired endpoint; teardown only runs after landing is confirmed, so any # leftover unhandled steer here is moot rather than unlanded work. diff --git a/bin/fm-turnend-guard.sh b/bin/fm-turnend-guard.sh index b2fb72463e0..d55385aa91c 100755 --- a/bin/fm-turnend-guard.sh +++ b/bin/fm-turnend-guard.sh @@ -137,7 +137,7 @@ record_parent_stop_state() { [ -n "$state" ] && [ -n "$id" ] && [ -n "$gen" ] && [ -x "$writer" ] || return 0 if [ "$status" -eq 2 ]; then verdict=busy - event=stop-blocked + event='stop-blocked' fi "$writer" apply "$state" "$id" "$verdict" --gen "$gen" --source claude-hook --event "$event" >/dev/null 2>&1 || true } diff --git a/bin/fm-update.sh b/bin/fm-update.sh index 8c4e9071f52..498e86ba07e 100755 --- a/bin/fm-update.sh +++ b/bin/fm-update.sh @@ -131,6 +131,7 @@ FF_STEER_WINDOWS="" secondmate_agent_may_be_alive() { # local id=$1 meta="$STATE/$1.meta" remote_host state=unreadable + fm_secondmate_is_paused "$STATE" "$id" && return 1 remote_host=$(fm_meta_get "$meta" remote_host) if [ -n "$remote_host" ]; then state=$("$SCRIPT_DIR/fm-on.sh" "$id" \ diff --git a/bin/fm-watch.sh b/bin/fm-watch.sh index bd954698909..bc1d34fa73d 100755 --- a/bin/fm-watch.sh +++ b/bin/fm-watch.sh @@ -505,6 +505,21 @@ inbox_steer_busy() { # fi } +WATCHER_SECONDMATE_TASK= + +watcher_secondmate_check() { + local task=$1 rc=0 + shift + fm_secondmate_liveness_lock "$task" || return 0 + WATCHER_SECONDMATE_TASK=$task + if ! fm_secondmate_is_paused "$STATE" "$task"; then + "$@" || rc=$? + fi + fm_secondmate_liveness_unlock "$task" + WATCHER_SECONDMATE_TASK= + return "$rc" +} + # Steering-inbox loss detection, one cheap check per recorded window per poll. # bin/fm-task-inbox-lib.sh owns delivery, busy-deferral, retry, and escalation policy. # Endpoint and busy checks precede delivery so recovery never types into a busy, @@ -513,6 +528,14 @@ inbox_steer_busy() { # # Runs for secondmates too: their pane-staleness exemption is about quiet panes # being healthy, while an unacknowledged instruction can still be a stuck steer. inbox_steer_check() { # + if [ "$(fm_meta_get "$STATE/$2.meta" kind)" = secondmate ]; then + watcher_secondmate_check "$2" inbox_steer_check_active "$@" + else + inbox_steer_check_active "$@" + fi +} + +inbox_steer_check_active() { local w=$1 task=$2 action verb rec count tail40 reason='' ring_rc backend agent_state action=$(fm_task_inbox_due_action "$STATE" "$task") || return 0 verb=${action%% *} @@ -942,11 +965,93 @@ secondmate_ring_to_drain() { # # parent alarm. Empty inbox and a fresh child beacon are not idle proof. # Receipts close the append-before-marker crash window without changing the # foreign queue. +secondmate_wake_stall_check() { + local meta=$1 task=$2 now=$3 threshold=$4 + local home queue row epoch seq row_key marker progress_marker ring_marker progress observed_at observed_key + local receipt receipt_dir notify_key queued idle reason episode_alerted already_rung w + home=$(fm_meta_get "$meta" home) + [ -n "$home" ] || return 0 + [ -f "$home/.fm-secondmate-home" ] && [ ! -L "$home/.fm-secondmate-home" ] || return 0 + [ "$(cat "$home/.fm-secondmate-home" 2>/dev/null || true)" = "$task" ] || return 0 + queue="$home/state/.wake-queue" + row=$(secondmate_oldest_queue_row "$queue") + marker="$STATE/.secondmate-wake-stall-$task" + progress_marker="$STATE/.secondmate-wake-progress-$task" + ring_marker="$STATE/.secondmate-wake-ring-$task" + receipt_dir="$STATE/.secondmate-wake-stall-receipts/$task" + if [ -z "$row" ]; then + rm -f "$marker" "$progress_marker" "$ring_marker" + if [ -e "$receipt_dir" ] || [ -L "$receipt_dir" ]; then + [ -d "$receipt_dir" ] && [ ! -L "$receipt_dir" ] || return 1 + rm -rf -- "$receipt_dir" || return 1 + fi + return 0 + fi + IFS=$'\t' read -r epoch seq _row_kind _row_key _row_payload </dev/null || true) + observed_at=${progress%%[[:space:]]*} + observed_key=${progress#*[[:space:]]} + if [ "$observed_at" = "$progress" ]; then + observed_key= + else + observed_key=${observed_key%%[[:space:]]*} + fi + case "$observed_at" in ''|*[!0-9]*) observed_at= ;; esac + case "$observed_key" in ''|*[!0-9-]*) observed_key= ;; esac + if [ -z "$observed_at" ] || [ -z "$observed_key" ] \ + || [ "$now" -lt "$observed_at" ] || [ "$row_key" != "$observed_key" ]; then + fm_wake_secondmate_progress_marker_write "$task" "$now" "$row_key" || return 1 + [ "$episode_alerted" -eq 0 ] || rm -f "$marker" || return 1 + rm -f "$ring_marker" || return 1 + return 0 + fi + [ "$episode_alerted" -eq 0 ] || return 0 + idle=$((now - observed_at)) + [ "$idle" -ge "$threshold" ] || return 0 + w=$(fm_backend_target_of_meta "$meta") + ! secondmate_in_active_turn "$w" "$idle" || return 0 + already_rung=0 + if [ -e "$ring_marker" ] || [ -L "$ring_marker" ]; then + [ -f "$ring_marker" ] && [ ! -L "$ring_marker" ] || return 1 + [ "$(cat "$ring_marker" 2>/dev/null || true)" = "$row_key" ] && already_rung=1 + fi + if [ "$already_rung" -eq 0 ] && secondmate_idle_ring_safe "$w"; then + if secondmate_ring_to_drain "$task" "$w"; then + fm_wake_secondmate_ring_marker_write "$task" "$row_key" || return 1 + fm_wake_secondmate_progress_marker_write "$task" "$now" "$row_key" || return 1 + return 0 + fi + fi + receipt="$receipt_dir/$row_key" + if [ "$(cat "$receipt" 2>/dev/null || true)" = "$row_key" ]; then + fm_wake_secondmate_stall_marker_write "$task" "$row_key" || return 1 + return 0 + fi + notify_key="secondmate-wake-loop-$task-$row_key" + reason="check: secondmate wake-loop stalled: mate=$task row=$seq idle=${idle}s" + queued=$(fm_wake_queued_keys check) + if ! printf '%s\n' "$queued" | grep -Fx "$notify_key" >/dev/null 2>&1; then + fm_wake_append check "$notify_key" "$reason" || return 1 + fi + fm_wake_secondmate_stall_receipt_write "$task" "$row_key" || return 1 + fm_wake_secondmate_stall_marker_write "$task" "$row_key" || return 1 + wake "$reason" + return 0 +} + secondmate_wake_stall_tick() { local now=$(( $(date +%s) )) threshold=$SECONDMATE_WAKE_STALL_SECS - local meta task kind remote_host home queue row epoch seq row_key marker progress_marker ring_marker progress observed_at observed_key - local receipt receipt_dir notify_key queued idle reason episode_alerted already_rung w - # Endpoint metadata admits this queue-loop check; secondmate-liveness owns registered mates whose endpoint is missing or dead. + local meta task kind remote_host for meta in "$STATE"/*.meta; do [ -e "$meta" ] || continue kind=$(fm_meta_get "$meta" kind) @@ -956,83 +1061,7 @@ secondmate_wake_stall_tick() { task=${meta##*/} task=${task%.meta} case "$task" in ''|*[!A-Za-z0-9._-]*) continue ;; esac - home=$(fm_meta_get "$meta" home) - [ -n "$home" ] || continue - [ -f "$home/.fm-secondmate-home" ] && [ ! -L "$home/.fm-secondmate-home" ] || continue - [ "$(cat "$home/.fm-secondmate-home" 2>/dev/null || true)" = "$task" ] || continue - queue="$home/state/.wake-queue" - row=$(secondmate_oldest_queue_row "$queue") - marker="$STATE/.secondmate-wake-stall-$task" - progress_marker="$STATE/.secondmate-wake-progress-$task" - ring_marker="$STATE/.secondmate-wake-ring-$task" - receipt_dir="$STATE/.secondmate-wake-stall-receipts/$task" - if [ -z "$row" ]; then - rm -f "$marker" "$progress_marker" "$ring_marker" - if [ -e "$receipt_dir" ] || [ -L "$receipt_dir" ]; then - [ -d "$receipt_dir" ] && [ ! -L "$receipt_dir" ] || return 1 - rm -rf -- "$receipt_dir" || return 1 - fi - continue - fi - IFS=$'\t' read -r epoch seq _row_kind _row_key _row_payload </dev/null || true) - observed_at=${progress%%[[:space:]]*} - observed_key=${progress#*[[:space:]]} - if [ "$observed_at" = "$progress" ]; then - observed_key= - else - observed_key=${observed_key%%[[:space:]]*} - fi - case "$observed_at" in ''|*[!0-9]*) observed_at= ;; esac - case "$observed_key" in ''|*[!0-9-]*) observed_key= ;; esac - if [ -z "$observed_at" ] || [ -z "$observed_key" ] \ - || [ "$now" -lt "$observed_at" ] || [ "$row_key" != "$observed_key" ]; then - fm_wake_secondmate_progress_marker_write "$task" "$now" "$row_key" || return 1 - [ "$episode_alerted" -eq 0 ] || rm -f "$marker" || return 1 - rm -f "$ring_marker" || return 1 - continue - fi - [ "$episode_alerted" -eq 0 ] || continue - idle=$((now - observed_at)) - [ "$idle" -ge "$threshold" ] || continue - w=$(fm_backend_target_of_meta "$meta") - ! secondmate_in_active_turn "$w" "$idle" || continue - already_rung=0 - if [ -e "$ring_marker" ] || [ -L "$ring_marker" ]; then - [ -f "$ring_marker" ] && [ ! -L "$ring_marker" ] || return 1 - [ "$(cat "$ring_marker" 2>/dev/null || true)" = "$row_key" ] && already_rung=1 - fi - if [ "$already_rung" -eq 0 ] && secondmate_idle_ring_safe "$w"; then - if secondmate_ring_to_drain "$task" "$w"; then - fm_wake_secondmate_ring_marker_write "$task" "$row_key" || return 1 - fm_wake_secondmate_progress_marker_write "$task" "$now" "$row_key" || return 1 - continue - fi - fi - receipt="$receipt_dir/$row_key" - if [ "$(cat "$receipt" 2>/dev/null || true)" = "$row_key" ]; then - fm_wake_secondmate_stall_marker_write "$task" "$row_key" || return 1 - continue - fi - notify_key="secondmate-wake-loop-$task-$row_key" - reason="check: secondmate wake-loop stalled: mate=$task row=$seq idle=${idle}s" - queued=$(fm_wake_queued_keys check) - if ! printf '%s\n' "$queued" | grep -Fx "$notify_key" >/dev/null 2>&1; then - fm_wake_append check "$notify_key" "$reason" || return 1 - fi - fm_wake_secondmate_stall_receipt_write "$task" "$row_key" || return 1 - fm_wake_secondmate_stall_marker_write "$task" "$row_key" || return 1 - wake "$reason" + watcher_secondmate_check "$task" secondmate_wake_stall_check "$meta" "$task" "$now" "$threshold" || return 1 done return 0 } @@ -1911,7 +1940,7 @@ wedge_defer_guard() { # [dialog-name] [repeat-secs] [busy-proven] local win=$1 since_file=$2 label=$3 escalation_file=$4 task=$5 hash=$6 dialog=${7-} repeat_secs=${8:-$STALE_ESCALATE_SECS} - local since age n reason interval evidence guard guard_note busy_proven=${9:-0} + local since age n reason interval guard guard_note busy_proven=${9:-0} local _wkey _wrollback_sentinel _wrb_ts _wrb_cap_reason _wrb_cap_status _wrb_rm_status local _wrb_since _wrb_now _wrb_age _wrb_cap_ts _wrb_gate _wkey=$(window_key "$win") @@ -2786,14 +2815,11 @@ pause_state_class() { # # Only a proven open hold may defer a wedge or change an eligible wait's owner. # bin/fm-captain-hold.sh owns the predicate and lifecycle identity; an unreadable # or absent record grants no deferral. Idle policy is applied separately. -CAPTAIN_CALL_IDENTITY= - task_captain_call_open() { # local task=$1 - CAPTAIN_CALL_IDENTITY= [ -n "$task" ] || return 1 - CAPTAIN_CALL_IDENTITY=$(FM_HOME="$FM_HOME" "$SCRIPT_DIR/fm-captain-hold.sh" \ - open "$task" --identity 2>/dev/null) || return 1 + FM_HOME="$FM_HOME" "$SCRIPT_DIR/fm-captain-hold.sh" \ + open "$task" --identity >/dev/null 2>&1 || return 1 return 0 } @@ -3607,6 +3633,10 @@ watcher_cleanup() { # Quietly: a stop usually lands inside the waited event wait, and bash would # report the group it kills here as a "Terminated" job line on stderr. fm_active_check_stop 2>/dev/null || cleanup_status=1 + if [ -n "$WATCHER_SECONDMATE_TASK" ]; then + fm_secondmate_liveness_unlock "$WATCHER_SECONDMATE_TASK" + WATCHER_SECONDMATE_TASK= + fi fm_check_output_cleanup fm_capture_output_cleanup fm_wait_tmp_cleanup diff --git a/docs/agent-control.md b/docs/agent-control.md index 7b512466b1f..c0856b8d268 100644 --- a/docs/agent-control.md +++ b/docs/agent-control.md @@ -11,6 +11,8 @@ The split exists because the data plane's marking is exactly right for a message A routing-marked `/quit` arrives as ordinary chat - `[fm-from-firstmate] /quit` - which the agent reasons about instead of executing. The failure repeated across harnesses and homes, and the workaround (remember to use an unmarked send for agent-control commands, and improvise the right key or command per harness) lived only in agent prose, so it failed again every time a session did not happen to recall it. +For intentional secondmate downtime, use the [secondmate pause and resume operations](configuration.md#pause-and-resume). + ## What the control plane owns `bin/fm-control-lib.sh` is the single executable owner of three capability tables, which have no side effects, so they can be read as a contract: @@ -63,7 +65,7 @@ T3 Code has no composer and its `/mcp` tools offer no session stop, so `exit` re `exit` stops an agent and preserves everything else. Removing a worktree, closing an endpoint, or discarding work stays with [`bin/fm-teardown.sh`](../bin/fm-teardown.sh), which owns the landed-work test. -**`resume` is not a verb.** +**`resume` is not a control-plane verb.** It is not deterministic across the verified adapters: codex, grok, gemini, devin, commandcode, and cortex resume only from a session id printed at exit, polytoken resumes only from a session id held in its own sessions registry, opencode continues the most recent session for the cwd, and claude, pi, pi-signed, omp, kimi, and agy have no verified general pane-resume contract. `relaunch` uses the brief on disk - not a harness-private session - as the durable instruction when the backend can prove the old agent stopped and the composer is empty; Devin on Herdr currently fails that composer check and refuses. T3 Code cannot host a fresh replacement agent in a thread, so `relaunch` refuses before stopping anything. diff --git a/docs/architecture.md b/docs/architecture.md index 768945c7e97..a5777933d93 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -41,14 +41,15 @@ The existing secondmate liveness and wake-loop guards remain independent of the Optional wedge caps and validating-run deferrals retain their upstream defaults; [`configuration.md`](configuration.md) owns those opt-ins. T3 Code adds a [native thread consult](t3code-backend.md#current-lifecycle-and-safety) before escalation. Those actionable wakes are written to a durable local queue (`state/.wake-queue`) only after generation-bound recovery evidence is published, so an interrupted watcher or handling turn can be recovered without losing the queue record. -Agent endpoint liveness and queue-consumption liveness are separate: on each poll, the primary watcher reads the oldest valid actionable row from every endpoint-recorded local secondmate home's durable wake queue without locking, consuming, or rewriting that foreign queue. +Agent endpoint liveness and queue-consumption liveness are separate: on each poll, the primary watcher reads the oldest valid actionable row from each unpaused endpoint-recorded local secondmate home's durable wake queue without locking, consuming, or rewriting that foreign queue. A queue that is draining is not stalled, so the primary times the interval since that oldest actionable row last changed rather than the age of the row itself, and rows that declare themselves a bounded external wait (`awaiting external - declared pause`) are not actionable evidence at all. Once that no-progress interval reaches `FM_SECONDMATE_WAKE_STALL_SECS` and the mate is not provably inside an active turn (an exact busy verdict, honored only while that same no-progress interval is under `FM_BUSY_TURN_MAX_SECS`, because a mate's turns end in its own home and leave no completed-turn evidence in the primary's), a mate whose semantic busy class is exactly idle, whose agent is alive, and whose composer is not pending is rung once so its own home can drain, and the parent notification is withheld until that same row stays frozen for another stall interval; unknown, busy-over-bound, and ring-unsafe endpoints keep the parent alarm, and empty inbox or a fresh child beacon is not idle proof. The primary then appends one keyed `check` wake naming the mate, row sequence, and observed idle interval; parent receipts and queued-key deduplication suppress repeats across watcher and handling crashes, one notification covers a whole no-progress episode, and any move of that position - drain progress, or the fresh rows of a queue reprovisioned under the same task id, at whatever sequence it restarts - ends that episode and starts a fresh observation interval, while empty, advancing, and declared-wait queues remain silent. Endpointless registered mates remain outside this queue scan because its preconditions can never be met for them. Dead-or-missing endpoint recovery is instead shared by two drivers over one library, `bin/fm-secondmate-liveness-lib.sh`: the session-start sweep in `bin/fm-bootstrap.sh`, and the watcher's own `FM_SECONDMATE_LIVENESS_SECS`-cadence tick during ordinary supervision. Both use the library's [recovery contract](../bin/fm-secondmate-liveness-lib.sh), including its close-proof barrier; [T3 recovery](t3code-backend.md#restart-and-liveness-behavior) owns continuation on an existing thread. -A remote route is probed read-only across its host-local boundary and is never replaced by a local endpoint, and the per-mate liveness lock keeps a concurrent sweep and tick from killing or re-probing an endpoint the other is mid-recovery on. +An unpaused remote route is probed read-only across its host-local boundary and is never replaced by a local endpoint, and the per-mate liveness lock serializes recovery, pause/resume and watcher inbox or home-queue delivery checks. +The [pause and resume reference](configuration.md#pause-and-resume) owns intentional-downtime behavior; the [lifecycle command header](../bin/fm-secondmate-restart.sh) owns renewal of the existing queue-progress observation on resume. [`bin/fm-secondmate-liveness-lib.sh`'s header](../bin/fm-secondmate-liveness-lib.sh) owns the additional per-home serialization across different mates and its bounded wait; `tests/fm-secondmate-liveness.test.sh` covers concurrent recovery and lock-wait expiry without endpoint mutation or an attempt entry. Each automatic relaunch surfaces as exactly one `check` wake plus a durable line in `state/.secondmate-relaunch-`, and a mate that exceeds `FM_SECONDMATE_LIVENESS_MAX_ATTEMPTS` ledgered attempts inside `FM_SECONDMATE_LIVENESS_WINDOW_SECS` is parked behind a bound marker and escalated once until a live probe rearms it with a full attempt budget. `tests/fm-wake-queue.test.sh` pins the no-progress notification, drain-progress reset, declared-pause exclusion, active-turn deferral, proven-idle child-first ring, busy and unknown parent-alarm paths, genuine stall after a ring, idempotence, quiet-queue, and byte-for-byte foreign-row preservation guarantees. @@ -347,7 +348,7 @@ The [`fm-backlog-handoff.sh`](../bin/fm-backlog-handoff.sh) header owns route-sp An unreachable remote host is unknown rather than dead, preserves its route and durable work, and is never failed over or relaunched locally. Idle secondmate endpoints are healthy; teardown is explicit and refuses while the secondmate home has in-flight work unless the captain has approved discard with `--force`. -Secondmate homes converge conservatively to the primary's version and declared inherited local material at launch and during locked session start. +Unpaused secondmate homes converge conservatively to the primary's version and declared inherited local material at launch and during locked session start. The [`secondmate-provisioning` skill](../.agents/skills/secondmate-provisioning/SKILL.md) owns the full guarded sync, propagation, nudge, and mid-session local-material push contract. Secondmate agents can run on a different verified harness than crewmates. diff --git a/docs/configuration.md b/docs/configuration.md index 5972a05377a..ed19ab3de81 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1290,6 +1290,24 @@ A preexisting project-bearing charter is also refused until it is re-scaffolded The lease is held under the secondmate id until explicit retirement or seed rollback returns it, so normal restarts do not free or recycle the home. Teardown of a leased home fails closed if `treehouse return` cannot release the lease; plain-clone homes with no treehouse pool slot are removed directly. +### Pause and resume + +Ask firstmate to pause an idle secondmate or resume it by name; the [lifecycle command header](../bin/fm-secondmate-restart.sh) owns direct command usage, marker, locking, retry and backend capability mechanics. +Both local homes and `host:` routes use these operations from their parent home. +Pause keeps the home, backlog, project clones and open records, while stopping the agent session. +It refuses outstanding child records, in-flight backlog work and uncommitted home changes so work awaiting landing cannot be abandoned. +The runtime must support both a verified stop and a replacement agent; see the [control-plane capability limits](agent-control.md#fail-closed-boundaries). +A confirmed refusal before stopping leaves a previously running mate unpaused; uncertain stop completion retains pause intent to prevent automatic recovery from undoing the request. +Routine liveness recovery, bootstrap convergence, update restarts, instruction and config reread nudges, and missed-reply recovery skip a paused mate across primary restarts. +Monitoring also defers pending-instruction retries and home wake-queue stall checks, preserving pending records and delivery counters without spending a recovery attempt. + +New work still follows the registered scope and normal backlog handoff. +Only explicit resume or genuinely fresh routed work resumes a paused mate. +The handoff queues fresh work durably before automatically resuming the mate; if resume fails, the work and retry record remain available. +Pending handoff and outbox retries leave it paused, with received work queued until resume. +Resume starts a fresh agent that reconciles its existing backlog and records at startup. +Resume uses the existing guarded pre-launch convergence for instructions and inherited configuration; local home wake-queue monitoring then starts a fresh observation interval so paused time cannot cause an immediate stall alert. + ### Project modes and backlog handoff Secondmate routes cover `no-mistakes` and `direct-PR` projects; `local-only` projects remain main-firstmate work. diff --git a/docs/remote-secondmates.md b/docs/remote-secondmates.md index 219aaa9b7aa..8a00b30c108 100644 --- a/docs/remote-secondmates.md +++ b/docs/remote-secondmates.md @@ -682,6 +682,10 @@ When the primary does not have a newly declared item, the older receiver's refus That item still costs one extra `absent` call on every inheritance push to a remote secondmate, on every home, including one that never selects `config/backend=t3code`: the remote job worker runs one home's commands one after another, and the lifecycle test legs that push inheritance measure about 5-12% slower; an older remote code root tolerates the call as above. The [`secondmate-provisioning` skill](../.agents/skills/secondmate-provisioning/SKILL.md#charter-and-seed) owns the placement-specific re-read contract, including changed transfers, unchanged retries, and partial or unknown completion. +### Pause and resume a remote second mate + +Use the parent-side [pause and resume operations](configuration.md#pause-and-resume) for a `host:` route; they preserve the remote home and perform lifecycle control only on its configured host. + ### Relaunch a live remote second mate A live remote second mate is restarted with `relaunch`, which runs the ordinary [control plane](agent-control.md) on that host. diff --git a/docs/scripts.md b/docs/scripts.md index e24332bbb5c..6c109398980 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -25,7 +25,7 @@ The shared no-mistakes gate lifecycle boundary is summarized in [architecture.md | `fm-captain-pane.py` | Render one Captain's Call card in a Herdr or SSH terminal and route its answer | | `fm-secondmate-reconcile.sh` | Queue Bearings reconcile requests for later supervision delivery and ask each mismatched home through its durable inbox with a per-home cooldown | | `fm-update.sh` | Guarded self-update of firstmate and local or remote secondmate homes, reconciling redundant divergence and classifying every live mate left on the target commit for restart or fallback nudge | -| `fm-secondmate-restart.sh` | Persist open conversational work, then restart eligible second mates or report the fallback outcome | +| [`fm-secondmate-restart.sh`](../bin/fm-secondmate-restart.sh) | Restart eligible second mates after persisting open conversational work; also owns [pause and resume](configuration.md#pause-and-resume) | | `fm-secondmate-restart-lib.sh` | Shared second-mate restart capability and persistence-request contract | | `fm-on.sh` | Execute one tracked Firstmate command in a configured remote secondmate home, using its job worker except for the doctor bootstrap | | `fm-remote-job-lib.sh` | Shared bounded remote job queue, worker readiness, LaunchAgent contract, and filesystem-composed PATH | diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index 3383a745521..a8a6a0081e9 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -6,6 +6,29 @@ This record contains reusable version-scoped evidence for active runtime guarant The backend guides own current setup, safety boundaries, and limitations. Exact task chronology, branch names, temporary homes, local paths, process ids, thread ids, and delivery transcripts remain in private reports or PR evidence. +## Secondmate pause lifecycle + +Synthetic regression coverage refreshed on 2026-10-29 with Bash 5.3.9 and tasks-axi 0.2.6. +The lifecycle uses the existing backend and harness capability tables rather than adding adapter-specific stop or process-detection signals. +The pause/resume boundary supports the control plane's replacement-capable tmux, Herdr and Rex paths; T3's persistent-thread lifecycle and backends without a recovery-grade stop classifier refuse before an agent is stopped. +Primary harness selection does not change this shell-level pause record or routing behavior. +Remote coverage uses SSH and session-provider fixtures, with no remote host, installed-harness turn or credentialed account exercised. + +```sh +bin/fm-test-run.sh tests/fm-secondmate-restart.test.sh tests/fm-secondmate-liveness.test.sh tests/fm-secondmate-sync.test.sh tests/fm-pending-reply.test.sh +``` + +Observed output includes: + +```text +ok - pause refuses orphan backlog work, dirty homes and concurrent child creation +ok - paused state is explicit and failed resume keeps the marker +ok - backlog handoff queues work and automatically resumes a paused receiver +ok - durable pause intent suppresses bootstrap and poll liveness for both placements +ok - paused homes skip bootstrap convergence, pending rereads and config push +ok - pause defers missed-reply recovery while preserving correlated report resolution +``` + ## Polytoken worker lifecycle Verified on 2026-10-14 with Polytoken 0.9.0 on macOS arm64 using the tmux worker path. diff --git a/tests/fm-backend-rex.test.sh b/tests/fm-backend-rex.test.sh index 9c905a7963f..62cda855f37 100644 --- a/tests/fm-backend-rex.test.sh +++ b/tests/fm-backend-rex.test.sh @@ -951,13 +951,13 @@ test_control_exit_after_a_server_stop() { touch "$dir/fake/unreachable" out=$(run_rex_control "$dir" t1 exit); rc=$? - expect_code 1 "$rc" "a server that cannot be reached or started proves nothing" + expect_code 3 "$rc" "a server that cannot be reached or started proves nothing" assert_not_contains "$out" "endpoint-gone" "exit must not claim a stop it could not prove" rm -f "$dir/fake/unreachable" touch "$dir/fake/down" out=$(run_rex_control "$dir" t2 exit); rc=$? - expect_code 1 "$rc" "a stopped server answering for a different recorded server must refuse" + expect_code 3 "$rc" "a stopped server answering for a different recorded server must refuse" assert_not_contains "$out" "endpoint-gone" "a different server's not-found proves nothing" touch "$dir/fake/down" diff --git a/tests/fm-backend-t3code.test.sh b/tests/fm-backend-t3code.test.sh index 5027baedce0..154c578e73f 100755 --- a/tests/fm-backend-t3code.test.sh +++ b/tests/fm-backend-t3code.test.sh @@ -1216,7 +1216,7 @@ test_control_exit_refused_before_any_call() { id=$(t3_uid t3exitz1) make_t3_control_task control-exit "$id" "$thread" running out=$(run_t3_control "$id" exit); rc=$? - expect_code 1 "$rc" "exit on a t3code task must refuse"$'\n'"$out" + expect_code 3 "$rc" "exit on a t3code task must refuse"$'\n'"$out" assert_contains "$out" "no session stop" "the refusal must name the missing session stop" assert_contains "$out" "'interrupt' ends the running turn" "the refusal must name the verb that works" [ -z "$(t3_dispatch_types)" ] || fail "a refused exit must send nothing to T3, not even an interrupt, got '$(t3_dispatch_types)'" diff --git a/tests/fm-backlog-handoff.test.sh b/tests/fm-backlog-handoff.test.sh index 48e81f788a9..5dbf085e656 100755 --- a/tests/fm-backlog-handoff.test.sh +++ b/tests/fm-backlog-handoff.test.sh @@ -618,7 +618,7 @@ EOF test_local_teardown_waits_for_handoff_wake() { local home="$TMP_ROOT/teardown-race-main" sub="$TMP_ROOT/teardown-race-sub" - local basebin blockbin="$TMP_ROOT/teardown-race-blockbin" handoff teardown i + local basebin blockbin="$TMP_ROOT/teardown-race-blockbin" handoff teardown i teardown_rc=0 setup_homes "$home" "$sub" printf 'project=%s\n' "$ROOT" >> "$home/state/design.meta" mkdir -p "$sub/data" "$blockbin" @@ -663,15 +663,18 @@ SH "$ROOT/bin/fm-teardown.sh" design --force > "$TMP_ROOT/teardown-race-teardown.out" 2>&1 & teardown=$! sleep 0.3 - kill -0 "$teardown" 2>/dev/null \ - || fail "local teardown bypassed the in-flight handoff lock: $(cat "$TMP_ROOT/teardown-race-teardown.out")" + if ! kill -0 "$teardown" 2>/dev/null; then + wait "$teardown" || teardown_rc=$? + [ "$teardown_rc" -ne 0 ] \ + || fail "local teardown reported success before the handoff wake completed" + fi [ -d "$sub" ] || fail "local teardown removed the receiver home before handoff wake completed" assert_grep 'teardown-race' "$sub/data/backlog.md" \ "local teardown removed routed work before handoff wake completed" touch "$TMP_ROOT/teardown-race.release" wait "$handoff" || fail "teardown-race handoff failed after releasing its wake" wait "$teardown" 2>/dev/null || true - pass "local teardown waits for the routed move and receiver wake" + pass "local teardown waits or refuses while the routed move and receiver wake are in flight" } test_local_teardown_preserves_wake_when_home_removal_fails() { @@ -691,6 +694,7 @@ test_local_teardown_preserves_wake_when_home_removal_fails() { marker="$home/state/.backlog-handoff-design.wake-pending" rec="$home/state/pending-replies/$corr" printf 'pending:%s\n' "$corr" > "$marker" + printf '1\n' > "$home/state/.secondmate-paused-design" cp -p -- "$marker" "$marker_before" cp -p -- "$rec" "$rec_before" real_rm=$(command -v rm) @@ -722,6 +726,7 @@ SH cmp -s "$rec_before" "$rec" \ || fail "failed home removal changed the pending wake correlation" assert_present "$home/state/design.meta" "failed teardown removed route metadata" + assert_present "$home/state/.secondmate-paused-design" "failed teardown removed pause intent" assert_grep '- design ' "$home/data/secondmates.md" "failed teardown removed the registry route" PATH="$fakebin:$PATH" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ @@ -733,6 +738,7 @@ SH assert_absent "$sub" "teardown retry left the receiver home" assert_absent "$marker" "teardown retry left the pending wake marker" assert_absent "$rec" "teardown retry left the pending wake correlation" + assert_absent "$home/state/.secondmate-paused-design" "teardown retry left inherited pause intent" assert_no_grep '- design ' "$home/data/secondmates.md" "teardown retry left the registry route" pass "failed local home removal preserves its wake and a retry retires both" } diff --git a/tests/fm-contributions.test.sh b/tests/fm-contributions.test.sh index 30c78719fba..3b524b7ac2d 100755 --- a/tests/fm-contributions.test.sh +++ b/tests/fm-contributions.test.sh @@ -950,7 +950,7 @@ pad_to_bytes() { # file bytes: pad with trailing JSON whitespace to an exact siz } test_batched_saved_read_keeps_reference_verdicts() { - local home i accepted=() expected actual reference input + local home i accepted=() expected actual reference home=$(new_home batched-read) # More files than one batch holds, so the read crosses batch boundaries. for i in $(seq 1 300); do diff --git a/tests/fm-control.test.sh b/tests/fm-control.test.sh index 4647c810a8f..c2c4b33af19 100755 --- a/tests/fm-control.test.sh +++ b/tests/fm-control.test.sh @@ -287,7 +287,9 @@ expect_pi_exit() { [ "$(literals "$dir")" = /quit ] \ || fail "$label should type exactly /quit, got: $(literals "$dir")" else - expect_code 1 "$rc" "$label must refuse"$'\n'"$out" + local refusal=3 + [ ! -s "$dir/fake/keys" ] || refusal=1 + expect_code "$refusal" "$rc" "$label must refuse"$'\n'"$out" [ ! -s "$dir/fake/literal" ] || fail "$label typed lifecycle text: $(literals "$dir")" fi } @@ -631,7 +633,7 @@ test_devin_stuck_picker_refuses_and_exit_types_nothing() { add_task "$dir" t1 devin devin_as "$dir" picker out=$(FM_FAKE_DEVIN_PICKER_STUCK=1 run_control "$dir" t1 exit); rc=$? - expect_code 1 "$rc" "exit must refuse while the revert picker is open"$'\n'"$out" + expect_code 3 "$rc" "exit must refuse while the revert picker is open"$'\n'"$out" [ -z "$(literals "$dir")" ] || fail "exit typed into the revert picker: $(literals "$dir")" ! grep -qx Enter "$dir/fake/keys" || fail "exit pressed Enter in the revert picker" pass "fm-control Devin: an open revert picker refuses every typed command" @@ -715,7 +717,7 @@ test_unverified_harness_is_refused() { add_task "$dir" t1 someagent alive_as "$dir" someagent out=$(run_control "$dir" t1 exit); rc=$? - expect_code 1 "$rc" "an unverified harness should refuse" + expect_code 3 "$rc" "an unverified harness should refuse" assert_contains "$out" "no verified control mechanics" "refusal should name the missing verification" [ -z "$(literals "$dir")" ] || fail "an unverified harness must receive no bytes" pass "fm-control: a harness with no verified control mechanics is refused, not guessed at" @@ -815,7 +817,7 @@ test_unverified_state_backends_refuse_stop_verbs() { ;; esac out=$(run_control "$dir" t1 exit); rc=$? - expect_code 1 "$rc" "exit on $backend should refuse"$'\n'"$out" + expect_code 3 "$rc" "exit on $backend should refuse"$'\n'"$out" assert_contains "$out" "no recovery-grade agent-state classifier" \ "the $backend refusal should name the missing stop proof" [ -z "$(literals "$dir")" ] || fail "$backend must receive no exit command" @@ -890,7 +892,7 @@ test_window_label_is_refused_with_the_exact_id() { add_task "$dir" t1 claude alive_as "$dir" claude out=$(run_control "$dir" fm-t1 exit); rc=$? - expect_code 1 "$rc" "a window label should refuse" + expect_code 3 "$rc" "a window label should refuse" assert_contains "$out" "pass the exact task id 't1'" "the refusal should name the exact id" [ -z "$(literals "$dir")" ] || fail "a refused target must receive no bytes" pass "fm-control: a legacy window label is refused and the exact task id is named" @@ -902,7 +904,7 @@ test_explicit_endpoint_is_refused() { add_task "$dir" t1 claude alive_as "$dir" claude out=$(run_control "$dir" "fmses:fm-t1" exit); rc=$? - expect_code 1 "$rc" "an explicit endpoint should refuse" + expect_code 3 "$rc" "an explicit endpoint should refuse" assert_contains "$out" "exact task id only" "the refusal should name the exact-id rule" [ -z "$(literals "$dir")" ] || fail "a refused target must receive no bytes" pass "fm-control: an explicit backend endpoint is never a control target" @@ -913,7 +915,7 @@ test_unknown_task_is_refused() { dir=$(new_case unknown) add_task "$dir" t1 claude out=$(run_control "$dir" t2 exit); rc=$? - expect_code 1 "$rc" "an unknown task should refuse" + expect_code 3 "$rc" "an unknown task should refuse" assert_contains "$out" "no task 't2'" "the refusal should name the missing task" pass "fm-control: an unrecorded task id is refused" } @@ -927,7 +929,7 @@ test_record_bound_to_another_task_is_refused() { > "$dir/home/state/t1.meta.tmp" mv "$dir/home/state/t1.meta.tmp" "$dir/home/state/t1.meta" out=$(run_control "$dir" t1 exit); rc=$? - expect_code 1 "$rc" "a record bound to another task should refuse" + expect_code 3 "$rc" "a record bound to another task should refuse" assert_contains "$out" "belongs to task other" "the refusal should name the conflicting binding" [ -z "$(literals "$dir")" ] || fail "a foreign record must receive no bytes" pass "fm-control: a record whose endpoint identity names another task is refused" @@ -960,7 +962,9 @@ test_remote_secondmate_is_refused_by_placement() { else out=$(run_control "$dir" t1 "$verb"); rc=$? fi - expect_code 1 "$rc" "$verb on a remotely placed secondmate should refuse" + local refusal=1 + [ "$verb" != exit ] || refusal=3 + expect_code "$refusal" "$rc" "$verb on a remotely placed secondmate should refuse" assert_contains "$out" "remotely placed secondmate on example.invalid" \ "the $verb refusal should name the remote placement, not blame the record" assert_not_contains "$out" "malformed" \ @@ -999,7 +1003,9 @@ test_interrupt_and_exit_lock_before_task_state_resolution() { out=$(run_control "$case_dir" t1 "$verb"); rc=$? kill "$holder" 2>/dev/null || true wait "$holder" 2>/dev/null || true - expect_code 1 "$rc" "$verb should refuse a held lifecycle lock" + local refusal=1 + [ "$verb" != exit ] || refusal=3 + expect_code "$refusal" "$rc" "$verb should refuse a held lifecycle lock" assert_contains "$out" "another lifecycle action is already running" \ "$verb should serialize before reading mutable task state" [ -z "$(literals "$case_dir")" ] || fail "contended $verb must type no command" @@ -1048,7 +1054,7 @@ test_relaunch_only_flags_are_rejected_on_other_verbs() { add_task "$dir" t1 claude alive_as "$dir" claude out=$(run_control "$dir" t1 exit --harness codex); rc=$? - expect_code 1 "$rc" "--harness should not apply to exit" + expect_code 3 "$rc" "--harness should not apply to exit" assert_contains "$out" "apply to 'relaunch' only" "the refusal should scope the flags" pass "fm-control: profile and note flags belong to relaunch only" } @@ -1087,7 +1093,7 @@ test_missing_tmux_endpoint_refuses_rather_than_claiming_a_stop() { # rather than claim a stop it cannot see, and sends nothing to an address it # cannot trust. Reclaim of a destroyed endpoint is Herdr-only # (docs/agent-control.md "Reclaiming a task whose endpoint is gone"). - expect_code 1 "$rc" "a tmux endpoint whose absence cannot be proven must refuse" + expect_code 3 "$rc" "a tmux endpoint whose absence cannot be proven must refuse" assert_not_contains "$out" "endpoint-gone" "exit must not report a stop it could not prove" [ -z "$(literals "$dir")" ] || fail "nothing may be sent into an endpoint exit cannot trust" pass "fm-control exit: an unprovable tmux endpoint refuses instead of claiming the agent stopped" @@ -1115,7 +1121,7 @@ test_ambiguous_endpoint_refuses() { add_task "$dir" t1 claude alive_as "$dir" some-unrelated-process out=$(run_control "$dir" t1 exit); rc=$? - expect_code 1 "$rc" "an unattributed endpoint should refuse" + expect_code 3 "$rc" "an unattributed endpoint should refuse" assert_contains "$out" "positively classified" "the refusal should name the missing attribution" [ -z "$(literals "$dir")" ] || fail "an unattributed endpoint must receive no bytes" pass "fm-control exit: an endpoint whose process cannot be attributed refuses" @@ -1156,7 +1162,7 @@ test_exit_refuses_an_open_background_picker() { alive_as "$dir" claude exit_picker_screen > "$dir/fake/pane" out=$(run_control "$dir" t1 exit); rc=$? - expect_code 1 "$rc" "an open exit picker should refuse"$'\n'"$out" + expect_code 3 "$rc" "an open exit picker should refuse"$'\n'"$out" assert_contains "$out" "blocked on a prompt: Claude background-task exit picker" \ "the refusal should name the dialog" assert_not_contains "$out" "Esc" "the refusal must not name a dismissal key" @@ -1231,7 +1237,7 @@ test_footer_busy_pi_on_tmux_refuses_exit() { alive_as "$dir" pi printf '────────────────────────\n\n────────────────────────\nworking Pi footer\n' > "$dir/fake/pane" out=$(FM_BUSY_REGEX='working Pi footer' run_control "$dir" t1 exit); rc=$? - expect_code 1 "$rc" "a footer-busy Pi on tmux cannot prove its composer empty" + expect_code 3 "$rc" "a footer-busy Pi on tmux cannot prove its composer empty" assert_contains "$out" "composer state is 'unknown'" "the refusal should name the unproven composer" [ -z "$(literals "$dir")" ] \ || fail "a footer-busy Pi on tmux may be blocked on a prompt and must receive no /quit" @@ -1481,6 +1487,18 @@ EOF pass "fm-control-lib: only a runtime's own recorded session has a relaunch resume form" } +test_exit_missing_home_is_a_pre_stop_refusal() { + local out rc + out=$(env -u FM_HOME "$ROOT/bin/fm-control.sh" t1 exit 2>&1); rc=$? + expect_code 3 "$rc" 'a missing home refuses exit before any stop input' + assert_contains "$out" 'FM_HOME is not set' 'the refusal must identify the missing home' + out=$(env -u FM_HOME "$ROOT/bin/fm-control.sh" t1 interrupt 2>&1); rc=$? + expect_code 1 "$rc" 'the ordinary interrupt error status remains compatible' + pass 'exit identifies even initial home validation as a pre-stop refusal' +} + +test_exit_missing_home_is_a_pre_stop_refusal + test_exit_types_each_harness_verified_command test_pi_exit_herdr_cost_footer_boundary test_pi_exit_uses_herdr_compact_proof_boundary diff --git a/tests/fm-daemon.test.sh b/tests/fm-daemon.test.sh index f4d947037b9..2160e4ddcbf 100755 --- a/tests/fm-daemon.test.sh +++ b/tests/fm-daemon.test.sh @@ -3504,7 +3504,7 @@ test_inject_msg_defers_on_unrecognized_composer_state() { test_daemon_idle_checkin_policy() { local spec kind listed dir state win key task FM_CONFIG_OVERRIDE for spec in ship:off scout:off secondmate:off secondmate:on; do - kind=${spec%%:*}; listed=${spec##*:}; task=policy-idle + kind=${spec%%:*}; listed=${spec##*:}; task='policy-idle' dir=$(make_supercase "idle-policy-$kind-$listed"); state="$dir/state" FM_CONFIG_OVERRIDE="$dir/config"; mkdir -p "$FM_CONFIG_OVERRIDE" win="sess:fm-$task"; key=$task @@ -3616,7 +3616,7 @@ test_daemon_worker_gate_retains_persistence() { test_daemon_drops_ordinary_idle_stale_marker() { local dir state task win key - dir=$(make_supercase idle-legacy-marker); state="$dir/state"; task=policy-idle + dir=$(make_supercase idle-legacy-marker); state="$dir/state"; task='policy-idle' win="sess:fm-$task"; key=$task printf 'window=%s\nkind=ship\n' "$win" > "$state/$task.meta" printf 'working: old idle status\n' > "$state/$task.status" diff --git a/tests/fm-pending-reply.test.sh b/tests/fm-pending-reply.test.sh index ba72d293e7b..22b200aedae 100755 --- a/tests/fm-pending-reply.test.sh +++ b/tests/fm-pending-reply.test.sh @@ -3360,6 +3360,95 @@ test_record_update_preserves_unreadable_input test_record_field_framing test_record_field_output_variable test_tick_without_metadata || fail "test_tick_without_metadata failed" +test_paused_mate_defers_reply_recovery_and_still_resolves_reports() { + local home state corr rec hook_log + local FM_PENDING_REPLY_SEND_HOOK=paused_recovery_hook + home=$(setup_parent paused-recovery) + state="$home/state" + hook_log="$TMP_ROOT/paused-recovery-hook.log" + export FM_PENDING_REPLY_NOW=2000 + # shellcheck disable=SC2329 # Invoked through the production test hook. + paused_recovery_hook() { printf '%s\n' "$*" >> "$hook_log"; } + export -f paused_recovery_hook + export FM_PENDING_REPLY_SEND_HOOK + corr=$(fm_pending_reply_create "$home" "$state" hibit 'open request') + fm_pending_reply_mark_delivered "$state" "$corr" + fm_pending_reply_observe_busy "$state" "$corr" busy + fm_pending_reply_observe_busy "$state" "$corr" idle + printf '1\n' > "$state/.secondmate-paused-hibit" + fm_pending_reply_tick_one "$state" "$corr" idle + fm_pending_reply_tick "$state" + if fm_pending_reply_send_recovery "$state" "$corr"; then + fail 'explicit recovery must also respect pause intent' + fi + assert_absent "$hook_log" 'routine recovery must not resume a paused mate' + rec=$(fm_pending_reply_path "$state" "$corr") + [ -z "$(fm_pending_reply_get "$rec" recovery_attempted_epoch)" ] || fail 'pause spent the one recovery attempt' + printf 'done [corr=%s]: report arrived\n' "$corr" > "$state/hibit.status" + fm_pending_reply_tick "$state" + assert_equals "$(phase_of "$state" "$corr")" resolved 'reports can settle while paused' + unset FM_PENDING_REPLY_SEND_HOOK + pass 'pause defers missed-reply recovery while preserving correlated report resolution' +} + +test_raced_recovery_deferral_preserves_the_attempt() { + local mode home state corr rec fb calls lifecycle_lock rc + local FM_PENDING_REPLY_SEND_HOOK=raced_recovery_hook + export FM_PENDING_REPLY_SEND_HOOK + for mode in pause contention resolved; do + home=$(setup_parent "recovery-deferral-$mode") + state="$home/state" + fb=$(make_stubs "$home") + fm_write_meta "$state/hibit.meta" kind=secondmate harness=claude backend=tmux \ + window=firstmate:fm-hibit endpoint_task_id=hibit "worktree=$home" + export FM_PENDING_REPLY_NOW=2000 + corr=$(fm_pending_reply_create "$home" "$state" hibit 'open request') + rec=$(fm_pending_reply_path "$state" "$corr") + fm_pending_reply_mark_delivered "$state" "$corr" + fm_pending_reply_observe_busy "$state" "$corr" busy + fm_pending_reply_observe_busy "$state" "$corr" idle + calls=0 + lifecycle_lock="$state/.secondmate-liveness-hibit.lock" + raced_recovery_hook() { + calls=$((calls + 1)) + if [ "$calls" -eq 1 ]; then + case "$mode" in + pause|resolved) printf '1\n' > "$state/.secondmate-paused-hibit" ;; + contention) fm_lock_try_acquire "$lifecycle_lock" || return 99 ;; + esac + if [ "$mode" = resolved ]; then + printf 'done [corr=%s]: report arrived\n' "$corr" > "$state/hibit.status" + fm_pending_reply_try_resolve "$state" "$corr" || return 99 + fi + fi + rc=0 + env PATH="$fb:$PATH" FM_HOME="$home" FM_SEND_LOG="$home/send.log" \ + FM_PENDING_REPLY_EXISTING_CORR="$corr" "$SEND" "$@" >/dev/null 2>&1 || rc=$? + [ "$mode" != contention ] || fm_lock_release "$lifecycle_lock" + return "$rc" + } + if fm_pending_reply_send_recovery "$state" "$corr"; then fail 'a raced recovery must defer'; fi + [ "$rc" -eq 75 ] || fail "the $mode send was not deferred before delivery" + if [ "$mode" = resolved ]; then + [ "$(phase_of "$state" "$corr")" = resolved ] || fail 'deferral resurrected a resolved report' + else + [ "$(phase_of "$state" "$corr")" = awaiting_report ] || fail 'deferral consumed the recovery phase' + [ -z "$(fm_pending_reply_get "$rec" recovery_attempted_epoch)" ] || fail 'deferral spent the one recovery attempt' + rm -f "$state/.secondmate-paused-hibit" + fm_pending_reply_tick_one "$state" "$corr" idle + [ "$(phase_of "$state" "$corr")" = recovery_sent ] || fail 'the deferred recovery did not send after resume' + [ "$calls" -eq 2 ] || fail 'the deferred recovery was skipped or duplicated' + assert_present "$state/hibit.inbox/001.msg" 'the eventual recovery must be durably delivered' + fi + done + unset FM_PENDING_REPLY_SEND_HOOK + pass 'pause and contention defer recovery without spending its attempt or reopening a report' +} + +test_raced_recovery_deferral_preserves_the_attempt + +test_paused_mate_defers_reply_recovery_and_still_resolves_reports + test_normal_correlated_reply_resolves_once test_completed_turn_no_report_triggers_one_recovery test_recovery_waits_while_the_mate_has_an_open_decision diff --git a/tests/fm-remote-backlog-handoff.test.sh b/tests/fm-remote-backlog-handoff.test.sh index 542fe1eade5..fb6b74deb53 100755 --- a/tests/fm-remote-backlog-handoff.test.sh +++ b/tests/fm-remote-backlog-handoff.test.sh @@ -344,12 +344,43 @@ assert_absent "$PARENT/data/handoff/ios.outbox.md" \ "remote receiver wake failure retained the durably received outbox" assert_present "$PARENT/state/.backlog-handoff-ios.wake-pending" \ "remote receiver wake failure was not tracked separately" +printf '1\n' > "$PARENT/state/.secondmate-paused-ios" +wakes_before=$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG") +handoff_env "$ROOT/bin/fm-backlog-handoff.sh" --resume-pending > "$TMP_ROOT/paused-wake-retry.out" 2>&1 \ + || fail "paused pending wake should defer: $(cat "$TMP_ROOT/paused-wake-retry.out")" +assert_present "$PARENT/state/.secondmate-paused-ios" "pending wake retry resumed a paused mate" +assert_present "$PARENT/state/.backlog-handoff-ios.wake-pending" "deferral lost the pending receiver wake" +[ "$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG")" -eq "$wakes_before" ] \ + || fail "pending wake retry contacted a paused receiver" +[ "$(grep -cF remote-wake-fail "$REMOTE/data/backlog.md")" -eq 1 ] \ + || fail "paused wake retry lost or duplicated the queued item" +rm -f "$PARENT/state/.secondmate-paused-ios" handoff_env "$ROOT/bin/fm-backlog-handoff.sh" --resume-pending >/dev/null \ || fail "remote receiver wake failure did not recover through resume-pending" assert_absent "$PARENT/state/.backlog-handoff-ios.wake-pending" \ "remote receiver wake recovery left separate wake state pending" pass "remote handoff releases durable work and separately retries its receiver wake" +write_backlog '- [ ] paused-outbox - queued until explicit resume (repo: alpha)' +if FM_FAKE_SSH_MODE=unreachable handoff_env "$ROOT/bin/fm-backlog-handoff.sh" ios paused-outbox > "$TMP_ROOT/paused-outbox-stage.out" 2>&1; then + fail "the unavailable transport should retain its outbox" +fi +printf '1\n' > "$PARENT/state/.secondmate-paused-ios" +wakes_before=$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG") +handoff_env "$ROOT/bin/fm-backlog-handoff.sh" --resume-pending > "$TMP_ROOT/paused-outbox-retry.out" 2>&1 \ + || fail "paused outbox receipt should complete: $(cat "$TMP_ROOT/paused-outbox-retry.out")" +assert_present "$PARENT/state/.secondmate-paused-ios" "outbox retry resumed a paused mate" +assert_absent "$PARENT/data/handoff/ios.outbox.md" "confirmed receipt did not release the paused outbox" +assert_present "$PARENT/state/.backlog-handoff-ios.wake-pending" "paused outbox retry lost wake intent" +[ "$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG")" -eq "$wakes_before" ] \ + || fail "outbox retry contacted a paused receiver" +[ "$(grep -cF paused-outbox "$REMOTE/data/backlog.md")" -eq 1 ] \ + || fail "paused outbox retry lost or duplicated its queued item" +rm -f "$PARENT/state/.secondmate-paused-ios" +handoff_env "$ROOT/bin/fm-backlog-handoff.sh" --resume-pending >/dev/null \ + || fail "the deferred outbox wake did not retry after resume" +pass "pending wake and outbox retries preserve pause and queue work durably" + # The receiver wake is a best-effort live nudge sent AFTER the backlog receipt # is durable. A wake whose remote transport is lost leaves its correlation # undelivered with delivery unknown, and the watcher's very next pending-reply @@ -479,6 +510,15 @@ esac wakes_after=$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG") [ "$wakes_after" -eq $((wakes_before + 1)) ] \ || fail "remote cleanup failure did not perform exactly one receiver wake" +printf '1\n' > "$PARENT/state/.secondmate-paused-ios" +handoff_env "$ROOT/bin/fm-backlog-handoff.sh" --resume-pending > "$TMP_ROOT/paused-confirmed-cleanup.out" 2>&1 \ + || fail "confirmed cleanup must finish while paused: $(cat "$TMP_ROOT/paused-confirmed-cleanup.out")" +assert_present "$PARENT/state/.secondmate-paused-ios" "confirmed cleanup resumed a paused mate" +assert_absent "$PARENT/data/handoff/ios.outbox.md" "confirmed cleanup retained the old outbox" +assert_absent "$PARENT/state/.backlog-handoff-ios.wake-pending" "confirmed cleanup retained a stale wake confirmation" +[ "$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG")" -eq "$wakes_after" ] \ + || fail "confirmed cleanup contacted a paused receiver" +rm -f "$PARENT/state/.secondmate-paused-ios" write_backlog '- [ ] after-cleanup - fresh work after confirmed cleanup failure (repo: alpha)' handoff_env "$ROOT/bin/fm-backlog-handoff.sh" ios after-cleanup >/dev/null \ || fail "fresh handoff did not converge an older confirmed cleanup failure" diff --git a/tests/fm-remote-secondmate-lifecycle-e2e.test.sh b/tests/fm-remote-secondmate-lifecycle-e2e.test.sh index 9d22fe7c9b2..bb1c7a598e5 100755 --- a/tests/fm-remote-secondmate-lifecycle-e2e.test.sh +++ b/tests/fm-remote-secondmate-lifecycle-e2e.test.sh @@ -31,7 +31,7 @@ PARENT_ROUTE_INBOX="$REMOTE_HOME/state/parent-route/ios.inbox" CLAIMS="$TMP_ROOT/claims" mkdir -p "$PARENT/data" "$PARENT/state" "$PARENT/config" "$PARENT/projects" "$REMOTE_ROOT" "$CLAIMS" cleanup() { - local worker_pid='' cleanup_attempt=0 + local worker_pid='' touch "$TMP_ROOT/provision.release" "$TMP_ROOT/seed.release" "$TMP_ROOT/handoff.release" \ "$TMP_ROOT/inherit.release" "$TMP_ROOT/launch.release" "$TMP_ROOT/race-clone.release" 2>/dev/null || true # A watcher leg cut short by a failed assertion is still polling the root. diff --git a/tests/fm-secondmate-liveness.test.sh b/tests/fm-secondmate-liveness.test.sh index 0a8a6b8fce0..4f500fc082a 100755 --- a/tests/fm-secondmate-liveness.test.sh +++ b/tests/fm-secondmate-liveness.test.sh @@ -840,6 +840,30 @@ test_remote_poll_probe_unreachable_preserves_route() { pass "poll probe: unreachable or inconclusive remote reads preserve the route" } +test_paused_mates_skip_local_and_remote_liveness() { + local w fb tmuxfb log out mode + w=$(new_world paused-local) + add_sm_home "$w" sm1 firstmate:fm-sm1 + fb=$(make_toolchain "$w"); tmuxfb=$(make_liveness_tmux "$w") + log="$w/calls.log"; : > "$log" + printf '1\n' > "$w/home/state/.secondmate-paused-sm1" + out=$(run_bootstrap "$tmuxfb:$fb" "$w/home" zsh "$log") + assert_not_contains "$(cat "$log")" 'kill-window' 'bootstrap must leave paused endpoints alone' + assert_not_contains "$(cat "$log")" 'new-window' 'bootstrap must not relaunch a paused mate' + assert_absent "$w/home/state/.secondmate-relaunch-sm1" 'pause must not spend a liveness relaunch attempt' + w=$(make_remote_probe_world paused-remote) + printf '1\n' > "$w/home/state/.secondmate-paused-rsm1" + for mode in poll full; do + out=$(probe_remote "$w" "$mode" FM_FAKE_REMOTE_REPLY=dead) + assert_contains "$out" 'silent|unknown|0|||' 'both liveness callers skip remote pause intent' + assert_contains "$out" 'intentionally paused' 'the skip reason names the pause' + done + assert_absent "$w/ssh.log" 'paused remote liveness must not contact the host' + pass 'durable pause intent suppresses bootstrap and poll liveness for both placements' +} + +test_paused_mates_skip_local_and_remote_liveness + test_tmux_agent_state_classifies test_tmux_agent_state_rejects_malformed_targets_before_probe test_herdr_agent_state_preserves_husk_classifier diff --git a/tests/fm-secondmate-restart.test.sh b/tests/fm-secondmate-restart.test.sh index 7fcd7a3b166..c81defec671 100755 --- a/tests/fm-secondmate-restart.test.sh +++ b/tests/fm-secondmate-restart.test.sh @@ -32,8 +32,6 @@ set -u # shellcheck source=tests/t3-fake-lib.sh . "$(dirname "${BASH_SOURCE[0]}")/t3-fake-lib.sh" -RESTART="$ROOT/bin/fm-secondmate-restart.sh" - fm_git_identity fmtest fmtest@example.com TMP_ROOT=$(fm_test_tmproot fm-secondmate-restart) mkdir -p "$TMP_ROOT" @@ -207,6 +205,10 @@ new_case() { printf 'claude' > "$dir/fake/command" printf 'claude' > "$dir/fake/becomes" make_stub "$dir" + # Launch commands are recorded by the session-provider stub, never executed. + # Keep the fixture hermetic on hosts without an installed Pi executable. + printf '#!/usr/bin/env bash\nprintf "pi 0.0.0 synthetic fixture\\n"\n' > "$dir/fakebin/pi" + chmod +x "$dir/fakebin/pi" fm_test_fake_tmux_foreground_cwd "$dir/fakebin" "$dir/fake/cwd" printf '%s\n' "$dir" } @@ -312,7 +314,7 @@ run_restart() { # FM_SECONDMATE_PERSIST_WAIT="${FM_TEST_PERSIST_WAIT:-30}" \ FM_CONTROL_POLL=0.01 FM_CONTROL_EXIT_WAIT=0.05 FM_CONTROL_LAUNCH_WAIT=0.05 \ FM_SSH_BIN="${FM_TEST_SSH_BIN:-ssh}" \ - "$RESTART" "$@" 2>&1 + "$ROOT/bin/fm-secondmate-restart.sh" "$@" 2>&1 } # --- T1: the persist request is the task subset of /stow, and it gates -------- @@ -600,6 +602,20 @@ case "${FM_FAKE_SSH_MODE:-ok}" in unreachable) exit 255 ;; esac case "${rargs[1]:-}" in + pause) + [ "${FM_FAKE_SSH_MODE:-ok}" != prestop-refusal ] || exit 3 + ;; + sync) + [ "${FM_FAKE_SSH_MODE:-ok}" != sync-fail ] || exit 1 + printf 'current: %s\n' "${rargs[3]}" + ;; + put|absent) + [ "${FM_FAKE_SSH_MODE:-ok}" != inherit-fail ] || exit 1 + if [ "${rargs[1]}" = put ] && [ -n "${FM_FAKE_REMOTE_HOME:-}" ]; then + mkdir -p "$FM_FAKE_REMOTE_HOME/$(dirname "${rargs[2]}")" + printf '%s' "$stdin_payload" > "$FM_FAKE_REMOTE_HOME/${rargs[2]}" + fi + ;; send-stdin) # Model the live remote mate: act on the instruction and report back on the # parent channel, carrying the correlation token the request embedded. @@ -609,7 +625,7 @@ case "${rargs[1]:-}" in >> "$FM_FAKE_ANSWER_STATUS" fi ;; - relaunch) + relaunch|resume) case "${FM_FAKE_SSH_MODE:-ok}" in slow-relaunch) : > "$FM_FAKE_DIR/remote-relaunch-start" @@ -1116,6 +1132,496 @@ w = { tokens: [], pass "T17 a T3 mate is routed to the re-read message by both passes, never restarted" } +# Pause/resume reuses the same executable lifecycle paths as restart. These +# cases are synthetic session-provider tests; they do not use a Herdr server. +clean_pause_home() { + local dir=$1 id=$2 home="$1/$2-home" + printf '/state/\n/data/\n/config/\n/projects/\n.fm-secondmate-home\n' > "$home/.gitignore" + git -C "$home" add -A + git -C "$home" commit -qm 'seed fixture home' +} + +test_pause_authorizes_before_publishing_intent() { + local dir out rc placement + for placement in local remote; do + dir=$(new_case "pause-lease-$placement") + if [ "$placement" = local ]; then + add_local_mate "$dir" sm1 + clean_pause_home "$dir" sm1 + else + setup_remote_case "$dir" sm1 ok + fi + printf '%s\n' "$$" > "$dir/home/state/.lock" + printf 'branch\t%s\t123\n' "$$" > "$dir/home/state/.lease-sm1" + out=$(FM_SUPERVISION_ACTOR=main run_restart "$dir" --pause sm1); rc=$? + expect_code 6 "$rc" "pause must honor $placement branch authority"$'\n'"$out" + assert_absent "$dir/home/state/.secondmate-paused-sm1" 'authorization refusal must not publish pause intent' + assert_no_grep '/exit' "$dir/fake/literal" 'authorization refusal must not stop the agent' + if [ "$placement" = remote ]; then + assert_no_grep ' pause ' "$dir/ssh.log" 'authorization refusal must not contact the host' + printf '1\n' > "$dir/home/state/.secondmate-paused-sm1" + out=$(FM_SUPERVISION_ACTOR=main run_restart "$dir" --resume sm1); rc=$? + expect_code 6 "$rc" "remote resume must honor parent lease authority"$'\n'"$out" + out=$(env FM_SUPERVISION_ACTOR=main FM_HOME="$dir/home" FM_SSH_BIN="$FM_TEST_SSH_BIN" \ + "$ROOT/bin/fm-remote-secondmate-relaunch.sh" sm1 claude default default resume 2>&1); rc=$? + expect_code 6 "$rc" "direct remote resume must honor parent lease authority"$'\n'"$out" + assert_present "$dir/home/state/.secondmate-paused-sm1" 'lease refusal retains pause' + [ ! -s "$dir/ssh.log" ] || fail 'unauthorized resume performed remote convergence or launch' + fi + done + unset FM_TEST_SSH_BIN FM_FAKE_SSH_MODE + pass 'local and remote pause authorize before publishing intent' +} + +test_pause_control_contention_keeps_the_mate_unpaused() { + local dir out rc + dir=$(new_case pause-control-contention) + add_local_mate "$dir" sm1 + clean_pause_home "$dir" sm1 + # shellcheck disable=SC2016 # Variables expand in the child shell. + out=$(env PATH="$dir/fakebin:$PATH" FM_HOME="$dir/home" FM_FAKE_DIR="$dir/fake" bash -c ' + . "$1/bin/fm-wake-lib.sh" + lock="$FM_HOME/state/.control-sm1.lock" + fm_lock_try_acquire "$lock" || exit 99 + "$1/bin/fm-secondmate-restart.sh" --pause sm1 + rc=$? + fm_lock_release "$lock" + exit "$rc" + ' _ "$ROOT" 2>&1); rc=$? + expect_code 3 "$rc" "control contention is a pre-stop refusal"$'\n'"$out" + assert_absent "$dir/home/state/.secondmate-paused-sm1" 'control contention must not publish false pause state' + assert_no_grep '/exit' "$dir/fake/literal" 'control contention must not stop the agent' + pass 'a busy control lock refuses pause without leaving false intent' +} + +test_maintenance_cannot_resume_or_race_pause() { + local dir out rc message + dir=$(new_case pause-maintenance) + add_local_mate "$dir" sm1 + clean_pause_home "$dir" sm1 + out=$(run_restart "$dir" --pause sm1); rc=$? + expect_code 0 "$rc" "pause before maintenance"$'\n'"$out" + for message in 're-read AGENTS.md and inherited config' 'persist open work for update' 'reconcile your backlog' 'repost your missed report'; do + out=$(env PATH="$dir/fakebin:$PATH" FM_HOME="$dir/home" FM_FAKE_DIR="$dir/fake" \ + "$ROOT/bin/fm-send.sh" sm1 "$message" 2>&1); rc=$? + [ "$rc" -ne 0 ] || fail 'maintenance was accepted by a paused mate' + assert_present "$dir/home/state/.secondmate-paused-sm1" 'maintenance must preserve pause' + assert_absent "$dir/home/state/sm1.inbox" 'maintenance must not queue a nudge while paused' + done + out=$(run_restart "$dir" --resume sm1); rc=$? + expect_code 0 "$rc" "resume before lifecycle exclusion check"$'\n'"$out" + # shellcheck disable=SC2016 # Variables expand in the child shell. + out=$(env PATH="$dir/fakebin:$PATH" FM_HOME="$dir/home" FM_FAKE_DIR="$dir/fake" bash -c ' + . "$1/bin/fm-wake-lib.sh" + lock="$FM_HOME/state/.secondmate-liveness-sm1.lock" + fm_lock_try_acquire "$lock" || exit 99 + "$1/bin/fm-send.sh" sm1 "re-read config" + rc=$? + fm_lock_release "$lock" + exit "$rc" + ' _ "$ROOT" 2>&1); rc=$? + [ "$rc" -ne 0 ] || fail 'maintenance bypassed a lifecycle action already in progress' + assert_contains "$out" 'lifecycle is busy' 'the maintenance send observes lifecycle ownership' + for message in "$dir/home/state/sm1.inbox"/*.msg; do + [ ! -e "$message" ] || fail 'a concurrent lifecycle action allowed final maintenance delivery' + done + pass 'maintenance preserves pause and final delivery obeys lifecycle exclusion' +} + +test_pause_after_persist_prevents_update_replacement() { + local dir out rc + dir=$(new_case pause-after-persist) + add_local_mate "$dir" sm1 + clean_pause_home "$dir" sm1 + cat > "$dir/fake/on-doorbell" <<'SH' +#!/usr/bin/env bash +( + . "$FM_TEST_ROOT/bin/fm-wake-lib.sh" + lock="$FM_HOME/state/.secondmate-liveness-sm1.lock" + fm_lock_acquire_wait "$lock" || exit 1 + fm_lock_release "$lock" + "$FM_TEST_ROOT/bin/fm-secondmate-restart.sh" --pause sm1 || exit 1 + corr=$(grep -oE 'corr=[0-9a-f]{16}' "$FM_HOME/state/sm1.inbox"/*.msg | head -1) + printf 'done [%s]: persisted\n' "$corr" >> "$FM_HOME/state/sm1.status" +) > "$FM_FAKE_DIR/concurrent-pause.log" 2>&1 & +SH + chmod +x "$dir/fake/on-doorbell" + out=$(FM_TEST_ROOT="$ROOT" run_restart "$dir" sm1); rc=$? + expect_code 0 "$rc" "update must skip a mate paused after persistence delivery"$'\n'"$out"$'\n'"$(cat "$dir/fake/concurrent-pause.log" 2>/dev/null)" + assert_contains "$out" 'skipped: sm1 intentionally paused' 'the final replacement checks pause under the lifecycle lock' + assert_present "$dir/home/state/.secondmate-paused-sm1" 'update preserves the concurrent pause' + assert_absent "$dir/home/state/sm1.control-relaunch" 'update must not replace a paused mate' + pass 'pause after persistence delivery excludes the update replacement' +} + +test_pause_refuses_live_and_stopped_unlanded_children() { + local dir out rc kind + for kind in live stopped-unlanded; do + dir=$(new_case "pause-$kind") + add_local_mate "$dir" sm1 + clean_pause_home "$dir" sm1 + printf 'kind=ship\nworktree=%s\n' "$dir/child" > "$dir/sm1-home/state/child.meta" + [ "$kind" != stopped-unlanded ] || printf 'done: awaiting landing\n' > "$dir/sm1-home/state/child.status" + out=$(run_restart "$dir" --pause sm1); rc=$? + expect_code 3 "$rc" "pause must refuse $kind child work"$'\n'"$out" + assert_contains "$out" 'in-flight or unlanded work' 'refusal names the safety boundary' + assert_absent "$dir/home/state/.secondmate-paused-sm1" 'unsafe pause must leave no intent marker' + assert_no_grep '/exit' "$dir/fake/literal" 'unsafe pause must not stop the agent' + done + pass 'pause refuses both live workers and stopped workers awaiting landing' +} + +test_explicit_pause_and_resume() { + local dir out rc + dir=$(new_case pause-resume) + add_local_mate "$dir" sm1 + clean_pause_home "$dir" sm1 + out=$(run_restart "$dir" --pause sm1); rc=$? + expect_code 0 "$rc" "safe pause succeeds"$'\n'"$out" + assert_contains "$out" 'paused: sm1' 'pause is reported' + assert_present "$dir/home/state/.secondmate-paused-sm1" 'pause survives a new command process' + assert_present "$dir/home/state/sm1.meta" 'pause retains the route' + assert_present "$dir/sm1-home" 'pause retains the home' + assert_grep '/exit' "$dir/fake/literal" 'pause stops the agent through control' + out=$(run_restart "$dir" --pause sm1); rc=$? + expect_code 0 "$rc" 'repeated pause is idempotent' + out=$(run_restart "$dir" sm1); rc=$? + expect_code 0 "$rc" 'update restart skips a paused mate' + assert_contains "$out" 'intentionally paused' 'update reports the skip' + assert_absent "$dir/home/state/sm1.inbox" 'skipped restart sends no persistence or reread nudge' + out=$(run_restart "$dir" --resume sm1); rc=$? + expect_code 0 "$rc" "explicit resume succeeds"$'\n'"$out" + assert_contains "$out" 'resumed: sm1' 'resume is reported' + assert_absent "$dir/home/state/.secondmate-paused-sm1" 'confirmed resume clears pause intent' + assert_present "$dir/home/state/sm1.control-relaunch" 'resume uses the ordinary relaunch transaction' + out=$(run_restart "$dir" --resume sm1); rc=$? + expect_code 0 "$rc" 'repeated resume is idempotent' + pass 'explicit pause and resume preserve the home and use verified lifecycle owners' +} + +resume_stall_tick() { + local dir=$1 + # shellcheck disable=SC2016 # Positional parameters expand in the child shell. + env PATH="$dir/fakebin:$PATH" FM_HOME="$dir/home" FM_FAKE_DIR="$dir/fake" \ + FM_STATE_OVERRIDE="$dir/home/state" FM_CONFIG_OVERRIDE="$dir/home/config" \ + FM_FAKE_NOW_FILE="$dir/now" FM_SECONDMATE_WAKE_STALL_SECS=60 FM_BUSY_TURN_MAX_SECS=120 \ + bash -eu -c '. "$1/bin/fm-watch.sh"; secondmate_wake_stall_tick' _ "$ROOT" 2>&1 +} + +test_resume_renews_the_retained_stall_observation() { + local dir state out rc mode real_date retained observed_at row_key rings + real_date=$(command -v date) + for mode in explicit fresh-work; do + dir=$(new_case "resume-stall-$mode") + state="$dir/home/state" + add_local_mate "$dir" sm1 + clean_pause_home "$dir" sm1 + cat > "$dir/fakebin/date" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = +%s ]; then + cat "$FM_FAKE_NOW_FILE" +else + exec "$FM_REAL_DATE" "$@" +fi +SH + chmod +x "$dir/fakebin/date" + printf '100\t7\tcheck\trouted\tcheck: pending work\n' > "$dir/sm1-home/state/.wake-queue" + cp "$dir/sm1-home/state/.wake-queue" "$dir/queue-before" + "$ROOT/bin/fm-busy-event.sh" arm "$state" sm1 >/dev/null || fail 'could not arm the fixture' + "$ROOT/bin/fm-busy-event.sh" apply "$state" sm1 idle --current-gen \ + --source claude-hook --event stop >/dev/null || fail 'could not mark the fixture idle' + printf '1000\n' > "$dir/now" + out=$(FM_REAL_DATE="$real_date" resume_stall_tick "$dir"); rc=$? + expect_code 0 "$rc" "first queue observation"$'\n'"$out" + printf '1061\n' > "$dir/now" + out=$(FM_REAL_DATE="$real_date" resume_stall_tick "$dir"); rc=$? + expect_code 0 "$rc" "drain ring before pause"$'\n'"$out" + [ "$(cat "$state/.secondmate-wake-ring-sm1")" = 100-7 ] || fail 'the watcher did not ring the retained row' + [ "$(wc -l < "$dir/fake/rings" | tr -d ' ')" = 1 ] || fail 'expected exactly one initial drain ring' + # shellcheck disable=SC2016 # Variables expand in the child shell. + env FM_HOME="$dir/home" FM_STATE_OVERRIDE="$state" bash -eu -c ' + . "$1/bin/fm-task-inbox-lib.sh" + ordinary=$(fm_task_inbox_write "$STATE" sm1 "pending ordinary steer") + fm_task_inbox_record_ring "$STATE" sm1 "$ordinary" + fm_task_inbox_record_busy "$STATE" sm1 "$ordinary" >/dev/null + fm_task_inbox_mark_retry "$STATE" sm1 "$STATE/sm1.inbox/001.msg" + ' _ "$ROOT" || fail 'could not seed retained delivery counters' + mkdir "$dir/inbox-before" + for retained in "$state/sm1.inbox/"*.msg "$state/sm1.inbox/.ring-state" \ + "$state/sm1.inbox/.busy-state" "$state/sm1.inbox/.retry-ring"; do + cp "$retained" "$dir/inbox-before/" + done + cp "$state/.secondmate-wake-ring-sm1" "$dir/ring-before" + out=$(FM_REAL_DATE="$real_date" FM_FAKE_NOW_FILE="$dir/now" run_restart "$dir" --pause sm1); rc=$? + expect_code 0 "$rc" "pause after the drain ring"$'\n'"$out" + printf '10000\n' > "$dir/now" + out=$(FM_REAL_DATE="$real_date" resume_stall_tick "$dir"); rc=$? + expect_code 0 "$rc" "long pause defers stall checks"$'\n'"$out" + [ ! -s "$state/.wake-queue" ] || fail 'pause published a stall alert' + if [ "$mode" = explicit ]; then + out=$(FM_REAL_DATE="$real_date" FM_FAKE_NOW_FILE="$dir/now" run_restart "$dir" --resume sm1); rc=$? + else + out=$(env PATH="$dir/fakebin:$PATH" FM_HOME="$dir/home" FM_FAKE_DIR="$dir/fake" \ + FM_REAL_DATE="$real_date" FM_FAKE_NOW_FILE="$dir/now" FM_SUPERVISION_ACTOR=main \ + FM_SPAWN_NO_GUARD=1 FM_CONTROL_POLL=0.01 FM_CONTROL_EXIT_WAIT=0.05 FM_CONTROL_LAUNCH_WAIT=0.05 \ + "$ROOT/bin/fm-send.sh" sm1 --new-work 'new scoped work' 2>&1); rc=$? + fi + expect_code 0 "$rc" "$mode resume after a long pause"$'\n'"$out" + assert_absent "$state/.secondmate-paused-sm1" 'successful resume clears pause intent' + IFS=$'\t' read -r observed_at row_key < "$state/.secondmate-wake-progress-sm1" + [ "$observed_at" = 10000 ] && [ "$row_key" = 100-7 ] || fail 'resume did not renew the existing observation' + for retained in "$dir/inbox-before/"* "$dir/inbox-before/".[!.]*; do + cmp "$retained" "$state/sm1.inbox/${retained##*/}" || fail 'resume changed a retained record or counter' + done + cmp "$dir/queue-before" "$dir/sm1-home/state/.wake-queue" || fail 'resume changed queued work' + cmp "$dir/ring-before" "$state/.secondmate-wake-ring-sm1" || fail 'resume reset the ring marker' + rings=$(wc -l < "$dir/fake/rings") + printf '10001\n' > "$dir/now" + out=$(FM_REAL_DATE="$real_date" resume_stall_tick "$dir"); rc=$? + expect_code 0 "$rc" "first post-resume watcher check"$'\n'"$out" + [ ! -s "$state/.wake-queue" ] || fail 'paused time caused a false post-resume stall alert' + assert_absent "$state/.secondmate-wake-stall-sm1" 'startup is given a new observation interval' + [ "$(wc -l < "$dir/fake/rings")" = "$rings" ] || fail 'resume repeated the retained drain ring' + printf '10121\n' > "$dir/now" + out=$(FM_REAL_DATE="$real_date" resume_stall_tick "$dir"); rc=$? + expect_code 0 "$rc" "genuine stall after the resumed interval"$'\n'"$out" + assert_contains "$out" 'secondmate wake-loop stalled: mate=sm1 row=7 idle=121s' 'a genuine resumed stall must still surface' + done + pass 'explicit and fresh-work resumes exclude paused time without resetting queued work or delivery counters' +} + +test_routed_backlog_resumes_paused_mate() { + command -v tasks-axi >/dev/null 2>&1 || { echo 'skip: tasks-axi not found'; return; } + local dir out rc + dir=$(new_case pause-handoff) + add_local_mate "$dir" sm1 + clean_pause_home "$dir" sm1 + printf -- '- sm1 - domain (home: %s; scope: things; projects: p; added 2026-09-03)\n' \ + "$dir/sm1-home" > "$dir/home/data/secondmates.md" + printf '# Backlog\n\n## Queued\n\n- [ ] routed - new work (repo: p; added 2026-10-29)\n\n## In flight\n\n## Done\n' > "$dir/home/data/backlog.md" + printf '# Backlog\n\n## Queued\n\n## In flight\n\n## Done\n' > "$dir/sm1-home/data/backlog.md" + out=$(run_restart "$dir" --pause sm1); rc=$? + expect_code 0 "$rc" "pause before handoff"$'\n'"$out" + out=$(env PATH="$dir/fakebin:$PATH" FM_HOME="$dir/home" FM_FAKE_DIR="$dir/fake" \ + FM_SUPERVISION_ACTOR=main \ + FM_SPAWN_NO_GUARD=1 FM_CONTROL_POLL=0.01 FM_CONTROL_EXIT_WAIT=0.05 FM_CONTROL_LAUNCH_WAIT=0.05 \ + "$ROOT/bin/fm-backlog-handoff.sh" sm1 routed 2>&1); rc=$? + expect_code 0 "$rc" "backlog handoff resumes its paused receiver"$'\n'"$out" + assert_grep 'routed -' "$dir/sm1-home/data/backlog.md" 'work is queued in the mate backlog' + assert_no_grep 'routed -' "$dir/home/data/backlog.md" 'the parent does not retain dispatchable work' + assert_absent "$dir/home/state/.secondmate-paused-sm1" 'routing automatically resumes the mate' + assert_present "$dir/home/state/sm1.control-relaunch" 'routing relaunches through the lifecycle owner' + assert_present "$dir/home/state/sm1.inbox" 'routed work reaches the ordinary durable inbox' + pass 'backlog handoff queues work and automatically resumes a paused receiver' +} + +test_remote_pause_resume_and_unknown_transport() { + local dir out rc sync_line inherit_line resume_line + dir=$(new_case remote-pause) + setup_remote_case "$dir" sm1 ok + fm_git_init_commit "$dir/primary" + mkdir -p "$dir/primary/bin" + cp "$ROOT/bin/fm-remote-secondmate-control.sh" "$ROOT/bin/fm-remote-inherit.sh" "$dir/primary/bin/" + git -C "$dir/primary" add bin + git -C "$dir/primary" commit -qm 'seed tracked remote commands' + printf 'codex\n' > "$dir/home/config/crew-harness" + export FM_FAKE_REMOTE_HOME="$dir/remote-home" + mkdir -p "$FM_FAKE_REMOTE_HOME" + out=$(run_restart "$dir" --pause sm1); rc=$? + expect_code 0 "$rc" "remote pause uses transport"$'\n'"$out" + assert_grep 'fm-remote-secondmate-control.sh pause sm1' "$dir/ssh.log" 'remote pause stays on its host' + assert_present "$dir/home/state/.secondmate-paused-sm1" 'parent retains durable remote pause intent' + rm -f "$dir/home/state/.secondmate-paused-sm1" + out=$(FM_FAKE_SSH_MODE=prestop-refusal run_restart "$dir" --pause sm1); rc=$? + expect_code 3 "$rc" 'a proven remote pre-stop refusal is distinct from lost completion' + assert_absent "$dir/home/state/.secondmate-paused-sm1" 'remote pre-stop refusal clears newly published intent' + out=$(run_restart "$dir" --pause sm1); rc=$? + expect_code 0 "$rc" 'remote pause can retry after a definite refusal' + cat > "$dir/fakebin/rm" <<'SH' +#!/usr/bin/env bash +for arg in "$@"; do + if [ -n "${FM_TEST_PAUSE_MARKER:-}" ] && [ "$arg" = "$FM_TEST_PAUSE_MARKER" ]; then + [ -e "$FM_HOME/state/.fm-lease-command.lock" ] || exit 99 + printf 'guard held\n' > "$FM_FAKE_DIR/resume-guard-held" + fi +done +exec "$FM_TEST_REAL_RM" "$@" +SH + chmod +x "$dir/fakebin/rm" + out=$(FM_SUPERVISION_ACTOR=main FM_TEST_REAL_RM="$(command -v rm)" \ + FM_TEST_PAUSE_MARKER="$dir/home/state/.secondmate-paused-sm1" \ + FM_ROOT_OVERRIDE="$dir/primary" run_restart "$dir" --resume sm1); rc=$? + expect_code 0 "$rc" "remote resume uses transport"$'\n'"$out" + assert_grep 'fm-remote-secondmate-control.sh resume sm1 claude' "$dir/ssh.log" 'remote resume passes the parent profile' + assert_absent "$dir/home/state/.secondmate-paused-sm1" 'confirmed remote resume clears intent' + assert_present "$dir/fake/resume-guard-held" 'remote resume retains lease authority through marker removal' + rm -f "$dir/fakebin/rm" + [ "$(cat "$dir/remote-home/config/crew-harness")" = codex ] || fail 'remote resume did not refresh inherited config' + sync_line=$(grep -n '^fm-remote-secondmate-control.sh sync sm1 ' "$dir/ssh.log" | head -1 | cut -d: -f1) + inherit_line=$(grep -n '^fm-remote-inherit.sh put config/crew-harness ' "$dir/ssh.log" | head -1 | cut -d: -f1) + resume_line=$(grep -n '^fm-remote-secondmate-control.sh resume sm1 ' "$dir/ssh.log" | head -1 | cut -d: -f1) + [ -n "$sync_line" ] && [ -n "$inherit_line" ] && [ "$sync_line" -lt "$resume_line" ] && [ "$inherit_line" -lt "$resume_line" ] \ + || fail 'remote resume launched before instruction and config convergence' + printf '1\n' > "$dir/home/state/.secondmate-paused-sm1" + printf 'pi\n' > "$dir/home/config/crew-harness" + : > "$dir/ssh.log" + out=$(env PATH="$dir/fakebin:$PATH" FM_ROOT_OVERRIDE="$dir/primary" FM_HOME="$dir/home" \ + FM_SSH_BIN="$FM_TEST_SSH_BIN" FM_SUPERVISION_ACTOR=main \ + "$ROOT/bin/fm-send.sh" sm1 --new-work 'new scoped work' 2>&1); rc=$? + expect_code 0 "$rc" "remote routed work converges before resume"$'\n'"$out" + [ "$(cat "$dir/remote-home/config/crew-harness")" = pi ] || fail 'routed resume did not refresh inherited config' + assert_absent "$dir/home/state/.secondmate-paused-sm1" 'routed remote work resumes the mate' + assert_grep 'fm-remote-secondmate-control.sh send-stdin sm1' "$dir/ssh.log" 'routed work reaches the host after resume' + local mode + for mode in sync-fail inherit-fail; do + printf '1\n' > "$dir/home/state/.secondmate-paused-sm1" + : > "$dir/ssh.log" + export FM_FAKE_SSH_MODE=$mode + out=$(FM_ROOT_OVERRIDE="$dir/primary" run_restart "$dir" --resume sm1); rc=$? + [ "$rc" -ne 0 ] || fail "remote resume ignored $mode" + assert_present "$dir/home/state/.secondmate-paused-sm1" 'failed convergence retains pause' + assert_no_grep 'fm-remote-secondmate-control.sh resume ' "$dir/ssh.log" 'failed convergence must prevent launch' + done + export FM_FAKE_SSH_MODE=unreachable + out=$(run_restart "$dir" --pause sm1); rc=$? + expect_code 255 "$rc" 'unknown transport is not a confirmed pause' + assert_present "$dir/home/state/.secondmate-paused-sm1" 'unknown pause completion suppresses liveness recovery' + out=$(run_restart "$dir" --resume sm1); rc=$? + [ "$rc" -ne 0 ] || fail 'unreachable resume must fail' + assert_present "$dir/home/state/.secondmate-paused-sm1" 'failed resume preserves pause intent' + unset FM_TEST_SSH_BIN FM_FAKE_SSH_MODE FM_FAKE_REMOTE_HOME + pass 'remote pause and resume use host transport and preserve intent on unknown completion' +} + +test_host_local_pause_refuses_children() { + local dir home out rc + dir=$(new_case remote-pause-child) + add_local_mate "$dir" sm1 + clean_pause_home "$dir" sm1 + home="$dir/sm1-home" + mkdir -p "$home/state/parent-route" + printf 'kind=ship\n' > "$home/state/child.meta" + cat > "$home/state/parent-route/sm1.meta" < "$FM_FAKE_DIR/herdr-called"\nexit 99\n' > "$dir/fakebin/herdr" + chmod +x "$dir/fakebin/herdr" + out=$(env PATH="$dir/fakebin:$PATH" FM_HOME="$home" FM_FAKE_DIR="$dir/fake" \ + "$ROOT/bin/fm-remote-secondmate-control.sh" pause sm1 2>&1); rc=$? + expect_code 3 "$rc" "host-local pause refuses children"$'\n'"$out" + assert_contains "$out" 'in-flight or unlanded work' 'remote safety uses the shared preflight' + assert_absent "$dir/fake/herdr-called" 'unsafe remote pause never calls Herdr' + rm -f "$home/state/child.meta" + # shellcheck disable=SC2016 # Variables expand in the child shell. + out=$(env PATH="$dir/fakebin:$PATH" FM_HOME="$home" FM_FAKE_DIR="$dir/fake" bash -c ' + . "$1/bin/fm-wake-lib.sh" + lock="$FM_HOME/state/.task-set.lock" + fm_lock_try_acquire "$lock" || exit 99 + "$1/bin/fm-remote-secondmate-control.sh" pause sm1 + rc=$? + fm_lock_release "$lock" + exit "$rc" + ' _ "$ROOT" 2>&1); rc=$? + expect_code 3 "$rc" "host task-set contention is a pre-stop refusal"$'\n'"$out" + assert_absent "$home/state/parent-route/.secondmate-paused-sm1" 'host contention must leave no new pause record' + assert_absent "$dir/fake/herdr-called" 'host contention must precede backend access' + ln -s "$home/state/missing-record" "$home/state/child.meta" + printf '1\n' > "$home/state/parent-route/.secondmate-paused-sm1" + out=$(env PATH="$dir/fakebin:$PATH" FM_HOME="$home" FM_FAKE_DIR="$dir/fake" \ + "$ROOT/bin/fm-remote-secondmate-control.sh" resume sm1 claude default default 2>&1); rc=$? + [ "$rc" -ne 0 ] || fail 'host-local resume hid its control-plane refusal' + assert_present "$home/state/parent-route/.secondmate-paused-sm1" 'host control failure preserves pause intent' + assert_absent "$dir/fake/herdr-called" 'host checkpoint refusal must precede Herdr access' + pass 'host-local pause and failed resume preserve in-flight work before backend access' +} + +test_pause_refuses_orphan_backlog_dirty_home_and_spawn_race() { + local dir out rc lock + dir=$(new_case pause-safety) + add_local_mate "$dir" sm1 + clean_pause_home "$dir" sm1 + printf '## In flight\n- [ ] orphan - still in flight (repo: p)\n\n## Queued\n\n## Done\n' > "$dir/sm1-home/data/backlog.md" + out=$(run_restart "$dir" --pause sm1); rc=$? + expect_code 3 "$rc" "orphan in-flight backlog refuses pause"$'\n'"$out" + assert_contains "$out" 'in-flight backlog work' 'orphan refusal names the backlog' + printf '## Queued\n\n## Done\n' > "$dir/sm1-home/data/backlog.md" + printf 'uncommitted\n' >> "$dir/sm1-home/AGENTS.md" + out=$(run_restart "$dir" --pause sm1); rc=$? + expect_code 3 "$rc" 'uncommitted home work refuses pause' + git -C "$dir/sm1-home" add AGENTS.md + git -C "$dir/sm1-home" commit -qm 'fixture clean' + # A concurrent spawn owns the existing home-wide task-set lock. Pause must + # refuse before checking a snapshot that could miss its unpublished child. + lock="$dir/sm1-home/state/.task-set.lock" + # shellcheck disable=SC2016 # Positional parameters expand in the child shell. + env FM_HOME="$dir/home" bash -c ' + . "$1/bin/fm-wake-lib.sh" + fm_lock_try_acquire "$2" || exit 1 + printf ready > "$3" + while [ ! -e "$4" ]; do sleep 0.1; done + fm_lock_release "$2" + ' _ "$ROOT" "$lock" "$dir/ready" "$dir/release" & + local holder=$! i + for ((i=0; i<100; i++)); do [ ! -e "$dir/ready" ] || break; sleep 0.1; done + [ -e "$dir/ready" ] || fail 'fixture lock holder did not start' + out=$(run_restart "$dir" --pause sm1); rc=$? + : > "$dir/release" + wait "$holder" + [ "$rc" -ne 0 ] || fail 'pause raced a child spawn' + assert_contains "$out" 'task set is busy' 'pause refuses a concurrently changing task set' + assert_absent "$dir/home/state/.secondmate-paused-sm1" 'all safety refusals preserve running posture' + assert_no_grep '/exit' "$dir/fake/literal" 'unsafe pauses never stop the agent' + pass 'pause refuses orphan backlog work, dirty homes and concurrent child creation' +} + +test_paused_current_state_and_failed_resume() { + local dir out rc + dir=$(new_case pause-current) + add_local_mate "$dir" sm1 + clean_pause_home "$dir" sm1 + out=$(run_restart "$dir" --pause sm1); rc=$? + expect_code 0 "$rc" "pause before current-state read"$'\n'"$out" + out=$(FM_HOME="$dir/home" "$ROOT/bin/fm-crew-state.sh" sm1 2>&1); rc=$? + expect_code 0 "$rc" "paused current state is readable"$'\n'"$out" + assert_contains "$out" 'paused' 'current state reports intentional downtime' + # Refusal from the ordinary control owner must retain the pause marker. + printf 'unverified-adapter\n' > "$dir/home/config/secondmate-harness" + out=$(run_restart "$dir" --resume sm1); rc=$? + [ "$rc" -ne 0 ] || fail 'unsupported replacement must refuse' + assert_present "$dir/home/state/.secondmate-paused-sm1" 'failed resume preserves pause intent' + pass 'paused state is explicit and failed resume keeps the marker' +} + +test_pause_authorizes_before_publishing_intent +test_pause_control_contention_keeps_the_mate_unpaused +test_maintenance_cannot_resume_or_race_pause +test_pause_after_persist_prevents_update_replacement +test_pause_refuses_orphan_backlog_dirty_home_and_spawn_race +test_paused_current_state_and_failed_resume + +test_pause_refuses_live_and_stopped_unlanded_children +test_explicit_pause_and_resume +test_resume_renews_the_retained_stall_observation +test_routed_backlog_resumes_paused_mate +test_remote_pause_resume_and_unknown_transport +test_host_local_pause_refuses_children + test_persist_gates_and_asks_only_for_open_records test_persist_precedes_restart test_working_pi_blank_composer_restarts diff --git a/tests/fm-secondmate-sync.test.sh b/tests/fm-secondmate-sync.test.sh index 40a100717dc..4fb104356ae 100755 --- a/tests/fm-secondmate-sync.test.sh +++ b/tests/fm-secondmate-sync.test.sh @@ -1425,6 +1425,43 @@ test_remote_launch_does_not_retarget_host_copy() { pass "R9 a remote launch leaves the home on the parent's commit while an ordinary spawn follows its own checkout" } +test_paused_home_skips_convergence_and_reread_retries() { + local w c1 fb out log marker + w=$(new_world paused-sync) + c1=$(head_of "$w/main") + add_sm_worktree "$w" sm-paused "$c1" + bump_primary "$w" instr + fb=$(make_fake_toolchain "$w") + log="$w/tmux.log" + printf '1\n' > "$w/home/state/.secondmate-paused-sm-paused" + mkdir -p "$w/home/state/.secondmate-nudge-pending" + marker="$w/home/state/.secondmate-nudge-pending/sm-paused.pending" + cat > "$marker" <&1) + [ "$(head_of "$w/sm-paused")" = "$c1" ] || fail 'bootstrap advanced a paused home' + assert_absent "$w/home/state/sm-paused.inbox" 'bootstrap retried a reread against a paused mate' + assert_present "$marker" 'a skipped reread remains owed until launch supersedes it' + mkdir -p "$w/home/config" + printf 'codex\n' > "$w/home/config/crew-harness" + out=$(PATH="$fb:$BASE_PATH" FM_HOME="$w/home" FM_ROOT_OVERRIDE="$w/main" \ + FM_FAKE_TMUX_LOG="$log" "$ROOT/bin/fm-config-push.sh" 2>&1) + assert_contains "$out" 'no live secondmate homes found' 'config push excludes paused discovery' + assert_absent "$w/sm-paused/config/crew-harness" 'config push does not converge a paused home' + assert_absent "$w/home/state/sm-paused.inbox" 'config push does not wake a paused mate' + pass 'paused homes skip bootstrap convergence, pending rereads and config push' +} + +test_paused_home_skips_convergence_and_reread_retries + test_ff_updated test_ff_current test_ff_dirty diff --git a/tests/fm-watch-inbox-meta.test.sh b/tests/fm-watch-inbox-meta.test.sh index 917e429e9ce..63d1dfaf77a 100644 --- a/tests/fm-watch-inbox-meta.test.sh +++ b/tests/fm-watch-inbox-meta.test.sh @@ -33,3 +33,170 @@ FM_HOME="$TMP_ROOT" FM_STATE_OVERRIDE="$TMP_ROOT/state" \ ' _ "$ROOT" pass "non-T3 inbox guard preserves no-target uncertainty and recorded busy/idle verdicts" + +FM_HOME="$TMP_ROOT/pause" FM_STATE_OVERRIDE="$TMP_ROOT/pause/state" \ + FM_CONFIG_OVERRIDE="$TMP_ROOT/pause/config" \ + FM_TASK_INBOX_GRACE_SECS=1 FM_SECONDMATE_WAKE_STALL_SECS=1 \ + bash -eu -s -- "$ROOT" <<'SH' +. "$1/bin/fm-watch.sh" +mkdir -p "$CONFIG" +touch "$CONFIG/wait-no-turns" + +fm_backend_agent_state() { printf '%s\n' "$TEST_AGENT"; } +fm_backend_busy_state() { printf 'idle\n'; } +fm_backend_capture() { printf 'quiet\n'; } +fm_backend_composer_state() { printf 'empty\n'; } +fm_backend_send_text_submit() { + if fm_secondmate_liveness_lock "$TEST_TASK"; then + printf 'unlocked\n' >> "$FM_HOME/sends" + fm_secondmate_liveness_unlock "$TEST_TASK" + return 2 + fi + printf '%s\n' "$TEST_TASK" >> "$FM_HOME/sends" +} +wake() { printf '%s\n' "$1" >> "$FM_HOME/alerts"; } + +fixture() { + TEST_TASK=$1 + mkdir -p "$FM_HOME/$TEST_TASK/state" + printf '%s\n' "$TEST_TASK" > "$FM_HOME/$TEST_TASK/.fm-secondmate-home" + printf 'kind=secondmate\nbackend=t3code\nwindow=%s\nharness=claude\nhome=%s\n' \ + "$TEST_TASK" "$FM_HOME/$TEST_TASK" > "$STATE/$TEST_TASK.meta" +} + +unchanged() { + cmp "$FM_HOME/before" "$1" + [ ! -s "$FM_HOME/sends" ] + [ ! -s "$FM_HOME/alerts" ] + [ ! -s "$STATE/.wake-queue" ] +} + +for location in local remote; do + for TEST_AGENT in dead missing alive; do + fixture "$location-$TEST_AGENT" + [ "$location" != remote ] || printf 'remote_host=synthetic\n' >> "$STATE/$TEST_TASK.meta" + rec=$(fm_task_inbox_write "$STATE" "$TEST_TASK" 'ordinary pending steer') + touch -t 202001010000 "$rec" + inbox=${rec%/*} + printf '%s\t2\t1\n' "${rec##*/}" > "$inbox/.ring-state" + printf '%s\t1\n' "${rec##*/}" > "$inbox/.busy-state" + cp -R "$inbox" "$FM_HOME/inbox-before" + touch "$STATE/.secondmate-paused-$TEST_TASK" + inbox_steer_check "$TEST_TASK" "$TEST_TASK" + diff -r "$FM_HOME/inbox-before" "$inbox" + [ ! -s "$FM_HOME/sends" ] + [ ! -s "$FM_HOME/alerts" ] + [ ! -s "$STATE/.wake-queue" ] + rm -rf "$FM_HOME/inbox-before" + rm "$rec" + rec=$(fm_task_inbox_write "$STATE" "$TEST_TASK" 'pending retry' fire-and-forget) + fm_task_inbox_mark_retry "$STATE" "$TEST_TASK" "$rec" + touch -t 202001010000 "$inbox/.retry-ring" + cp -R "$inbox" "$FM_HOME/inbox-before" + inbox_steer_check "$TEST_TASK" "$TEST_TASK" + diff -r "$FM_HOME/inbox-before" "$inbox" + rm -rf "$FM_HOME/inbox-before" + TEST_AGENT=alive + rm "$STATE/.secondmate-paused-$TEST_TASK" + inbox_steer_check "$TEST_TASK" "$TEST_TASK" + [ -f "$rec" ] + [ ! -e "$inbox/.retry-ring" ] + [ "$(cat "$FM_HOME/sends")" = "$TEST_TASK" ] + [ ! -e "$STATE/.secondmate-liveness-$TEST_TASK.lock" ] + rm "$FM_HOME/sends" + rm "$STATE/$TEST_TASK.meta" + done +done + +fixture empty +mkdir -p "$STATE/.secondmate-wake-stall-receipts/empty" +fm_wake_secondmate_progress_marker_write empty 1 100-7 +fm_wake_secondmate_ring_marker_write empty 100-7 +fm_wake_secondmate_stall_marker_write empty 100-7 +fm_wake_secondmate_stall_receipt_write empty 100-7 +for record in progress ring stall; do + cp "$STATE/.secondmate-wake-$record-empty" "$FM_HOME/$record-before" +done +cp "$STATE/.secondmate-wake-stall-receipts/empty/100-7" "$FM_HOME/receipt-before" +touch "$STATE/.secondmate-paused-empty" +secondmate_wake_stall_tick +for record in progress ring stall; do + cmp "$FM_HOME/$record-before" "$STATE/.secondmate-wake-$record-empty" +done +cmp "$FM_HOME/receipt-before" "$STATE/.secondmate-wake-stall-receipts/empty/100-7" +rm "$STATE/empty.meta" + +fixture unseen +printf '100\t7\tcheck\trouted\tcheck: routed work\n' > "$FM_HOME/unseen/state/.wake-queue" +touch "$STATE/.secondmate-paused-unseen" +secondmate_wake_stall_tick +[ ! -e "$STATE/.secondmate-wake-progress-unseen" ] +rm "$STATE/unseen.meta" + +fixture mate +TEST_AGENT=alive +rec=$(fm_task_inbox_write "$STATE" mate 'pending ordinary steer') +touch -t 202001010000 "$rec" +printf '%s\t1\t1\n' "${rec##*/}" > "$STATE/mate.inbox/.ring-state" +cp -R "$STATE/mate.inbox" "$FM_HOME/inbox-before" +printf '100\t7\tcheck\trouted\tcheck: routed work\n' > "$FM_HOME/mate/state/.wake-queue" +cp "$FM_HOME/mate/state/.wake-queue" "$FM_HOME/foreign-before" +fm_wake_secondmate_progress_marker_write mate 1 100-7 +cp "$STATE/.secondmate-wake-progress-mate" "$FM_HOME/before" +touch "$STATE/.secondmate-paused-mate" +inbox_steer_check mate mate +secondmate_wake_stall_tick +unchanged "$STATE/.secondmate-wake-progress-mate" +diff -r "$FM_HOME/inbox-before" "$STATE/mate.inbox" +cmp "$FM_HOME/foreign-before" "$FM_HOME/mate/state/.wake-queue" +[ ! -e "$STATE/.secondmate-wake-ring-mate" ] +[ ! -e "$STATE/.secondmate-wake-stall-mate" ] +[ ! -e "$STATE/.secondmate-wake-stall-receipts/mate" ] + +rm "$STATE/.secondmate-paused-mate" +( + fm_secondmate_liveness_lock mate + touch "$FM_HOME/locked" + for ((i=0; i<300; i++)); do + [ ! -f "$FM_HOME/release" ] || break + sleep 0.01 + done + fm_secondmate_liveness_unlock mate +) & +holder=$! +for ((i=0; i<300; i++)); do + [ ! -f "$FM_HOME/locked" ] || break + sleep 0.01 +done +[ -f "$FM_HOME/locked" ] +inbox_steer_check mate mate +secondmate_wake_stall_tick +unchanged "$STATE/.secondmate-wake-progress-mate" +diff -r "$FM_HOME/inbox-before" "$STATE/mate.inbox" +touch "$FM_HOME/release" +wait "$holder" + +inbox_steer_check mate mate +[ "$(cat "$FM_HOME/sends")" = mate ] +IFS=$'\t' read -r base count epoch < "$STATE/mate.inbox/.ring-state" +[ "$base" = "${rec##*/}" ] && [ "$count" = 2 ] +[ -f "$rec" ] +[ ! -e "$STATE/.secondmate-liveness-mate.lock" ] +rm "$FM_HOME/sends" +secondmate_wake_stall_tick +[ "$(cat "$FM_HOME/sends")" = mate ] +[ "$(cat "$STATE/.secondmate-wake-ring-mate")" = 100-7 ] +[ ! -e "$STATE/.secondmate-liveness-mate.lock" ] +[ ! -s "$FM_HOME/alerts" ] +[ ! -s "$STATE/.wake-queue" ] +cmp "$FM_HOME/foreign-before" "$FM_HOME/mate/state/.wake-queue" +fm_wake_secondmate_progress_marker_write mate 1 100-7 +secondmate_wake_stall_tick +grep -F 'secondmate wake-loop stalled: mate=mate row=7' "$FM_HOME/alerts" +[ -f "$STATE/.secondmate-wake-stall-mate" ] +[ -f "$STATE/.secondmate-wake-stall-receipts/mate/100-7" ] +[ ! -e "$STATE/.secondmate-liveness-mate.lock" ] +printf 'complete\n' > "$FM_HOME/complete" +SH +[ -f "$TMP_ROOT/pause/complete" ] || fail 'pause watcher scenarios did not finish' +pass "paused watcher inboxes and home queues defer without spending records or counters" diff --git a/tests/fm-watch-triage-wedge-defer.test.sh b/tests/fm-watch-triage-wedge-defer.test.sh index 0c6100770c2..9c2bd32cfe3 100644 --- a/tests/fm-watch-triage-wedge-defer.test.sh +++ b/tests/fm-watch-triage-wedge-defer.test.sh @@ -276,7 +276,7 @@ arm_validating() { # } test_wedge_threshold_defers_to_an_executing_validation_run() { - local dir state fakebin out capture window key n verdict step queued + local dir state fakebin out capture window key n verdict step window="test:fm-wedge"; key=$(printf '%s' "$window" | tr ':/.' '___') for step in running fixing; do