Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/stuck-crewmate-recovery/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ The target window's harness is recorded as `harness=` in `state/<id>.meta`.
This procedure covers ordinary `kind=ship` and `kind=scout` direct reports.
Load `secondmate-provisioning` instead for `kind=secondmate` recovery.

For a REMOTE secondmate, `fm-crew-state`'s `unknown`/`worktree gone` and `fm-send`'s `remote send failed`/`delivery unconfirmed` verdicts are unreliable and routinely false-negative; do not conclude the mate is dead or the send failed from those alone, confirm against the actual remote pane first.
For a REMOTE secondmate, `fm-crew-state` and `fm-peek` read the actual remote endpoint over `fm-on.sh`, and `fm-send` reports a delivered-with-pending-confirmation steer as delivered (their headers own the contracts); an `unknown-remote` read or unreachable-host failure means the remote state could not be read, never that the mate is dead or the send failed.
Recover a genuinely stuck remote mate only through `bin/fm-spawn.sh <id> --secondmate`, never raw herdr pane close/kill surgery, which strands the endpoint binding.

Treat the digest's endpoint result as a presence signal, not proof that the task's work or validation run is gone.
Expand Down
30 changes: 30 additions & 0 deletions bin/fm-brief.sh
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
# it carries the AGENTS.md authoring bar (widely useful knowledge only, pointers
# over copied detail) and has the crewmate add the fm-ensure-agents-md.sh
# self-governance section when a touched project AGENTS.md lacks it.
# Refuses claiming a task id whose data/<task-id>/ directory already exists.
# Refuses to overwrite an existing brief.
set -eu

Expand Down Expand Up @@ -154,6 +155,7 @@ elif [ "$MODE_SET" -eq 1 ]; then
echo "error: --mode applies only to ship briefs; a scout delivers a report and a secondmate charter is not a delivery contract" >&2
exit 1
fi
[ "${#POS[@]}" -ge 1 ] || { echo "error: task id is required" >&2; exit 1; }
ID=${POS[0]}

if [ "$KIND" = secondmate ] && [ "$HERDR_LAB" -eq 1 ]; then
Expand All @@ -166,6 +168,34 @@ if [ "$NO_PROJECTS" -eq 1 ] && [ "$KIND" != secondmate ]; then
exit 1
fi

if [ -e "$DATA/$ID" ]; then
contents=""
if [ -d "$DATA/$ID" ]; then
entries=()
shopt -s nullglob dotglob
for entry in "$DATA/$ID"/*; do
entries+=("$(basename "$entry")")
done
shopt -u nullglob dotglob
for entry in "${entries[@]}"; do
if [ -n "$contents" ]; then
contents="$contents, $entry"
else
contents="$entry"
fi
done
fi
if [ -n "$contents" ]; then
contents_desc="contains: $contents"
elif [ -d "$DATA/$ID" ]; then
contents_desc="empty directory"
else
contents_desc="existing file"
fi
echo "error: task id '$ID' already exists at $DATA/$ID ($contents_desc); choose a distinct task id (e.g. '$ID-2' or mint a new one) to preserve retained history and avoid collisions" >&2
exit 1
fi

BRIEF="$DATA/$ID/brief.md"
[ -e "$BRIEF" ] && { echo "error: $BRIEF already exists" >&2; exit 1; }
mkdir -p "$DATA/$ID"
Expand Down
57 changes: 53 additions & 4 deletions bin/fm-crew-state.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,17 @@
# fixed mapping logic, no heuristics and no LLM. Output is one stable, parseable,
# token-tight line firstmate can read every heartbeat:
#
# state: <working|parked|done|blocked|paused|failed|unknown> · source: <run-step|pane|status-log|none> · <detail>
# state: <working|parked|done|blocked|paused|failed|unknown> · source: <run-step|pane|status-log|remote-endpoint|none> · <detail>
#
# Logic, in order:
# 1. Resolve worktree + backend target + kind from state/<id>.meta.
# 1. Resolve worktree + backend target + kind from state/<id>.meta. A meta
# recording remote_host= is a remote secondmate: its worktree and endpoint
# live on that host, so the local worktree and pane reads are skipped and
# the remote host is asked for the endpoint's recovery-grade state
# (fm-on.sh + fm-remote-secondmate-control.sh state). alive falls through
# to the routed status log; dead/missing report the remote verdict; an
# unreachable or unreadable remote reports unknown-remote, never a false
# gone/dead.
# 2. Matching no-mistakes run for this crew's branch AND current code identity,
# active or terminal (from `axi status`, or the coarse `no-mistakes runs`
# fallback)? Branch name alone is not enough: a historical run on a reused
Expand Down Expand Up @@ -101,10 +108,13 @@ meta_value() { # <key>
WT=$(meta_value worktree)
KIND=$(meta_value kind)
HARNESS=$(meta_value harness)
REMOTE_HOST=$(meta_value remote_host)
[ -n "$KIND" ] || KIND=ship

# A torn-down (or never-created) worktree has no current state to read.
if [ -z "$WT" ] || [ ! -d "$WT" ]; then
# A torn-down (or never-created) worktree has no current state to read. A
# remote secondmate's recorded worktree is a path on ITS host, so the local
# probe proves nothing for it - the remote arm below reads the true source.
if [ -z "$REMOTE_HOST" ] && { [ -z "$WT" ] || [ ! -d "$WT" ]; }; then
emit unknown none "worktree gone (torn down?)"
fi

Expand Down Expand Up @@ -138,6 +148,45 @@ map_log_state() { # <line>
LOG_LINE=$(log_last_line || true)
LOG_VERB=$(status_line_verb "$LOG_LINE")

# --- remote secondmate: the true source is the remote endpoint ---------------
# A remote mate's recorded worktree and backend target live on its own host, so
# the local worktree probe above and the local pane reads below would misreport
# a healthy remote mate as gone or dead. Ask the remote host for the endpoint's
# recovery-grade state over the same fm-on.sh transport fm-send uses, then read
# current activity from the routed status log exactly as for a local
# secondmate (an idle endpoint is healthy for a secondmate either way). An
# unreachable host or unreadable endpoint is reported as unknown-remote -
# explicitly NOT proof of death - so a transport blip never reads as a torn
# down or dead mate; only the remote host's own dead/missing verdict may say
# the endpoint is actually gone.
if [ -n "$REMOTE_HOST" ]; then
if ! REMOTE_STATE=$(FM_HOME="$FM_HOME" "$SCRIPT_DIR/fm-on.sh" "$ID" \
fm-remote-secondmate-control.sh state "$ID" < /dev/null 2>/dev/null); then
REMOTE_STATE=
fi
REMOTE_STATE=$(printf '%s\n' "$REMOTE_STATE" | tail -1)
case "$REMOTE_STATE" in
alive)
if [ -n "$LOG_VERB" ]; then
LOG_STATE=$(map_log_state "$LOG_LINE")
if [ "$LOG_STATE" != unknown ]; then
emit "$LOG_STATE" status-log "$(status_line_note "$LOG_LINE")${SEP}remote endpoint alive on $REMOTE_HOST"
fi
fi
emit unknown remote-endpoint "alive on $REMOTE_HOST (an idle secondmate is healthy)"
;;
dead|missing)
emit unknown remote-endpoint "remote endpoint $REMOTE_STATE on $REMOTE_HOST"
;;
'')
emit unknown remote-endpoint "unknown-remote: $REMOTE_HOST unreachable or endpoint unreadable (not proof of death)"
;;
*)
emit unknown remote-endpoint "unknown-remote: endpoint state '$REMOTE_STATE' on $REMOTE_HOST (not proof of death)"
;;
esac
fi

# pane_readable is consulted ONLY in the no-run fallback below. The run-step path
# stays authoritative regardless of pane liveness - judge by the run-step, not the
# shell - so a finished crew whose endpoint has closed still reports its run-step
Expand Down
23 changes: 22 additions & 1 deletion bin/fm-peek.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
# Usage: fm-peek.sh <target> [lines=40]
# <target> may be an exact task id, a legacy fm-<id> task label resolved
# through this home's state/<id>.meta, or an explicit backend target.
# A selector whose meta records remote_host= is a remote secondmate: its pane
# lives on that host, so the capture routes over fm-on.sh to the host-local
# capture (fm-remote-secondmate-control.sh), clamped to that command's
# 100-line cap. An unreachable host or unreadable endpoint fails loudly naming
# the host; the local backend adapters are never asked to read a remote target.
set -eu

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
Expand All @@ -16,9 +21,25 @@ STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}"
"$SCRIPT_DIR/fm-guard.sh" || true

RAW_TARGET=$1
T=$(fm_backend_resolve_selector "$RAW_TARGET" "$STATE")
N=${2:-40}

REMOTE_META=$(fm_backend_meta_for_selector "$RAW_TARGET" "$STATE" 2>/dev/null || true)
if [ -n "$REMOTE_META" ] && [ -n "$(fm_meta_get "$REMOTE_META" remote_host)" ]; then
REMOTE_ID=${REMOTE_META##*/}
REMOTE_ID=${REMOTE_ID%.meta}
REMOTE_HOST=$(fm_meta_get "$REMOTE_META" remote_host)
case "$N" in ''|*[!0-9]*|0) N=40 ;; esac
[ "$N" -le 100 ] || N=100
if ! FM_HOME="$FM_HOME" "$SCRIPT_DIR/fm-on.sh" "$REMOTE_ID" \
fm-remote-secondmate-control.sh capture "$REMOTE_ID" "$N" < /dev/null; then
echo "error: could not read the remote pane of $REMOTE_ID on $REMOTE_HOST (host unreachable or endpoint unreadable; the mate is not thereby dead)" >&2
exit 1
fi
exit 0
fi

T=$(fm_backend_resolve_selector "$RAW_TARGET" "$STATE")

BACKEND=$(fm_backend_of_selector "$RAW_TARGET" "$T" "$STATE")
EXPECTED_LABEL=$(fm_backend_expected_label_of_selector "$RAW_TARGET" "$STATE")

Expand Down
6 changes: 6 additions & 0 deletions bin/fm-remote-secondmate-control.sh
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,12 @@ cmd_send() {
validate_id "$id"
validate_home "$id"
remote_endpoint_require "$id"
# fm-send's exit status is the delivery verdict the parent home acts on
# (0 = confirmed, 3 = delivered with the submit read-back unconfirmed, other
# nonzero = failed; see bin/fm-send.sh's header). The job worker, entrypoint,
# and ssh all preserve it, so no mapping may happen here: flattening exit 3
# into a generic failure is exactly the false-negative the parent's remote
# send path exists to avoid.
FM_HOME="$TARGET_HOME" FM_ROOT_OVERRIDE="$FM_ROOT" FM_STATE_OVERRIDE="$TARGET_HOME/state" \
"$SCRIPT_DIR/fm-send.sh" "$REMOTE_ENDPOINT_TARGET" "$message"
}
Expand Down
62 changes: 59 additions & 3 deletions bin/fm-send.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@
# submit or reports an inconclusive send. If a swallowed Enter is positively
# confirmed, fm-send exits NON-ZERO so the caller knows the steer did not land
# instead of silently leaving an unsubmitted instruction.
# Exit status contract: 0 = submit confirmed (or, for a remote secondmate
# target, delivered with confirmation pending - see the remote paragraph);
# 3 = the text was typed into the live endpoint and Enter was sent, but the
# submit read-back stayed unconfirmed (verify the pane before any resend, and
# never re-type blindly); any other nonzero = the send failed and nothing may
# be assumed delivered.
# Submission dispatches through the target's recorded backend; the tmux adapter
# shares its composer/submit core with the away-mode daemon via bin/fm-tmux-lib.sh.
# Tune with FM_SEND_RETRIES (default 3) / FM_SEND_SLEEP (0.4).
Expand All @@ -37,6 +43,20 @@
# re-sending a recovery request for an already-open expectation so a second
# record is not created. Direct unmarked captain input never creates one.
#
# Remote secondmate delivery: the send crosses fm-on.sh to a host-local leg
# (bin/fm-remote-secondmate-control.sh cmd_send) that runs this same verified
# submit against the recorded remote Herdr pane and relays its exit status
# unchanged. A leg that delivered the text into the live verified pane but
# could not synchronously confirm the submit (exit 3 - typically a busy mate
# whose harness queues the steer and keeps rendering it) is reported here as
# DELIVERED with confirmation pending: fm-send prints a non-error notice,
# exits 0, marks the pending-reply expectation delivered, and closes any
# --resolve-key decisions. Empirically that pattern is a delivered steer, a
# resend duplicates the instruction, and the parent's pending-reply
# recovery/escalation still surfaces the rare genuinely lost request. Transport
# loss (ssh exit 255, completion unknown) and every real remote failure keep
# failing loudly with the remote leg's own stderr attached.
#
# Decision closure (answerer-closes): pass --resolve-key <key> (repeatable,
# before the message) when this send answers an open keyed needs-decision: or
# blocked: record in the target task's state/<id>.status. After the submit is
Expand All @@ -63,7 +83,9 @@
# in this home's status log per status_open_decisions (bin/fm-classify-lib.sh), or
# an active captain hold for the target task. A key in neither is refused before
# sending, so a mistyped key cannot deliver an answer while silently orphaning the
# decision. A failed or unconfirmed send never closes a key; a
# decision. A failed or unconfirmed send never closes a key (a remote
# delivered-with-pending-confirmation outcome counts as delivered - see the
# remote paragraph above); a
# delivered answer whose closing append fails exits nonzero with the exact
# manual close command, leaving the decision open to re-surface (the safe
# direction). A send without the flag never closes anything: a routine steer,
Expand Down Expand Up @@ -537,12 +559,27 @@ else
# Type once, submit, verify. Only exact empty confirms delivery; every other
# verdict preserves the loud refusal boundary.
send_rc=0
REMOTE_DELIVERY_NOTICE=0
if [ "$TARGET_BACKEND" = remote ]; then
if "$SCRIPT_DIR/fm-on.sh" "$TARGET_REMOTE_ID" fm-remote-secondmate-control.sh send "$TARGET_REMOTE_ID" "$MESSAGE" < /dev/null >/dev/null; then
# The remote leg is this same script running host-locally against the
# recorded Herdr pane (cmd_send in fm-remote-secondmate-control.sh), so its
# submit verification IS the local one, and fm-on/the remote worker relay
# its exit status unchanged. Exit 3 is the delivered-unconfirmed contract
# (see this script's header) crossing the ssh boundary: the text reached
# the live verified pane and Enter was sent; only the synchronous read-back
# stayed unconfirmed. The remote stderr is held back and replayed only for
# a real failure, so a delivered outcome does not surface the inner leg's
# diagnostics as alarm.
remote_err=$("$SCRIPT_DIR/fm-on.sh" "$TARGET_REMOTE_ID" fm-remote-secondmate-control.sh send "$TARGET_REMOTE_ID" "$MESSAGE" < /dev/null 2>&1 >/dev/null) || send_rc=$?
if [ "$send_rc" -eq 0 ]; then
verdict=empty
elif [ "$send_rc" -eq 3 ]; then
verdict=empty
send_rc=0
REMOTE_DELIVERY_NOTICE=1
else
send_rc=$?
verdict=send-failed
[ -z "$remote_err" ] || printf '%s\n' "$remote_err" >&2
fi
elif verdict=$(fm_backend_send_text_submit "$TARGET_BACKEND" "$T" "$MESSAGE" "$retries" "$sleep_s" "$settle" "$EXPECTED_LABEL"); then
:
Expand Down Expand Up @@ -571,6 +608,19 @@ else
echo "error: text not sent to $T ($TARGET_BACKEND send failed; tried $RESOLUTION_TRIED)" >&2
exit 1
;;
pending)
# The text was typed into the live target and Enter was sent; only the
# submit read-back stayed unconfirmed (e.g. a busy harness queues the
# steer and keeps rendering it). That is not a proven failure, so never
# re-type the message: verify the pane instead. Exit 3 is the documented
# delivered-unconfirmed status, and the remote send leg above depends on
# it crossing the ssh boundary intact.
if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then
fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true
fi
echo "fm-send: text delivered to $T but submission is unconfirmed (verdict=pending; tried $RESOLUTION_TRIED); do not retype or blindly resend - verify with fm-peek.sh, then re-send '--key Enter' only if the composer still holds the text" >&2
exit 3
;;
*)
if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then
fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true
Expand Down Expand Up @@ -600,6 +650,12 @@ else
fm_send_close_resolved_keys "$RESOLVE_ANSWER_TEXT" || exit 1
fm_send_feed_resolved_holds "$RESOLVE_ANSWER_TEXT" || exit 1
fi
# Remote delivered-with-pending-confirmation: the outcome above is treated as
# delivered (expectation marked, keys closed), and this one non-error notice
# carries the remaining nuance so nobody re-sends the steer.
if [ "$REMOTE_DELIVERY_NOTICE" = 1 ]; then
echo "fm-send: delivered to remote secondmate $TARGET_REMOTE_ID; the remote pane accepted the text and Enter, and only the synchronous submit confirmation is still pending. This is not a failure - do not resend; the pending-reply expectation stays armed." >&2
fi
# Submit landed with exact empty. Confirmation only proves the text was
# accepted; the harness still needs a beat to spin up the
# turn before its busy footer shows. Pause so an immediate peek catches the
Expand Down
8 changes: 8 additions & 0 deletions docs/remote-secondmates.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,11 @@ Send routed requests normally:
FM_HOME=<primary-home> bin/fm-send.sh fm-<id> '<request>'
```

The [`fm-send.sh` header](../bin/fm-send.sh) owns the exact delivery-status contract.
When the verified remote endpoint accepts the text and Enter but synchronous submit confirmation remains pending, the primary reports the request as delivered rather than failed; do not resend it, because its pending-reply expectation remains armed.
`fm-peek.sh` and `fm-crew-state.sh` route remote-secondmate reads to the endpoint's host instead of consulting local worktree or backend state.
An unreachable or unreadable remote read is unknown, not evidence that the endpoint is dead.

Marked requests keep the existing correlation contract.
The remote charter appends replies to `state/parent-replies.status` in the remote home.
A process-event source performs a non-destructive, cursor-anchored delta read, fetches only referenced `data/*.md` documents through the confined reader, mirrors every content-bearing line at most once into the primary status channel, and does not carry blank separators.
Expand Down Expand Up @@ -231,6 +236,9 @@ The lifecycle test covers seeding a registered project that this machine has nev

```sh
bin/fm-test-run.sh tests/fm-on.test.sh
bin/fm-test-run.sh tests/fm-send-remote-delivery.test.sh
bin/fm-test-run.sh tests/fm-peek-remote.test.sh
bin/fm-test-run.sh tests/fm-crew-state.test.sh
bin/fm-test-run.sh tests/fm-remote-job.test.sh
bin/fm-test-run.sh tests/fm-remote-doctor.test.sh
bin/fm-test-run.sh tests/fm-project-origin.test.sh
Expand Down
2 changes: 1 addition & 1 deletion docs/tmux-backend.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ The supervisor guard selects only the detected primary harness's signature rathe
It types a message once and retries Enter only until the composer clears.
Only a proven empty composer is a positive delivery acknowledgement.
Text left in established structure remains `pending`, text in ambiguous structure remains unproven, and unreadable or unsafe state remains unknown.
`fm-send.sh` reports every unconfirmed verdict as a failure instead of retyping or assuming delivery.
`fm-send.sh` never retypes or assumes a confirmed submit for an unconfirmed verdict; its header owns the distinct delivered-unconfirmed exit status and operator response.

OpenCode 1.18.4 has one busy-queue exception.
While OpenCode is mid-turn, Enter queues the message but leaves its text visible until the turn completes.
Expand Down
Loading
Loading