fix: stop stray remote worker supervisors and align brief decision keys

bin/fm-brief.sh

@@ -278,14 +278,15 @@ Report only true captain-relevant outcomes or a declared external wait by append
 States: working, needs-decision, blocked, $PAUSED_VERB, done, failed.
 Use \`$PAUSED_VERB: {why}\` (distinct from \`blocked:\`) only when your domain is deliberately idling on a known external wait you expect to clear on its own; use \`blocked:\` when you are stuck and need firstmate to act.
 Use this only for material phase changes, a captain decision, a real blocker, a failure, or work ready for review.
+For a captain decision, append \`needs-decision [key=<slug>]: {summary of options}\`.
 This is also how you return the answer to a marked from-firstmate request above.
 A marked request requires one correlated answer after the work; it does not require a separate receipt or start acknowledgement.
 Never append \`working:\` merely to acknowledge receipt or announce that a marked request has started.
 When a routed-work phase has a supervisor-actionable material change worth reporting under the rule above, give that reported phase a stable key.
 If its first reportable event is \`working [key=<work-slug>]: {material phase}\`, use the same key on its later \`$PAUSED_VERB\`, \`done\`, \`failed\`, \`needs-decision\`, or \`blocked\` event so the earlier working phase is superseded.
 When a keyed phase ends without another reportable state, append \`resolved [key=<work-slug>]: {why it is no longer active}\`.
 \`resolved\` separately closes an escalated decision or blocker, and only a \`resolved\` line carrying that decision's exact key closes it: a later \`done\` or \`working\` event never does, even when the answer is what started that work.
-The main firstmate's answer normally writes that closing line at answer time; when a blocker or wait clears WITHOUT an answer from the main firstmate, append \`resolved: {how it cleared}\` yourself (keyed with \`[key=<slug>]\` if you opened it with one) as your domain resumes.
+The main firstmate's answer normally writes that closing line at answer time; when a blocker or wait clears WITHOUT an answer from the main firstmate, append \`resolved [key=<slug>]: {how it cleared}\` yourself (same \`[key=<slug>]\` if you opened it with one) as your domain resumes.
 Routine internal supervision, heartbeats, retries, and crewmate churn stay inside your own home and must not touch that status file.
 
 # Definition of done
@@ -368,9 +369,9 @@ The report is the only thing that survives, so anything worth keeping must be in
    treating it as a possible wedge. Use \`blocked:\` when you are stuck and need help.
 5. If you hit the same obstacle twice, append \`blocked: {why}\` and stop; firstmate will help.
 6. If a decision belongs to a human (product choices, destructive actions),
-   append \`needs-decision: {summary of options}\` and stop. Firstmate will reply with the decision.
+   append \`needs-decision [key=<slug>]: {summary of options}\` and stop. Firstmate will reply with the decision.
    A decision or blocker you opened stays open until a \`resolved\` line carrying its exact key lands; a later \`done:\` or \`working:\` line never closes it, even when the answer is what started that work.
-   Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply, append \`resolved: {how it cleared}\` yourself (same \`[key=<slug>]\` if you opened it with one) as you resume.
+   Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply, append \`resolved [key=<slug>]: {how it cleared}\` yourself (same \`[key=<slug>]\` if you opened it with one) as you resume.
 7. Never stop, restart, or update the shared \`no-mistakes\` daemon - it is one instance serving
    every lane/home, so restarting it kills other lanes' in-flight pipeline runs. On ANY no-mistakes
    daemon error, append \`blocked: {the daemon error}\` and stop; only firstmate manages the daemon.
@@ -456,11 +457,11 @@ land in the generation or the report.
    treating it as a possible wedge. Use \`blocked:\` when you are stuck and need help.
 5. If you hit the same obstacle twice, append \`blocked: {why}\` and stop; firstmate will help.
 6. If a decision belongs above you (product choices, destructive actions, ask-user findings),
-   append \`needs-decision: {summary of options}\` and stop. Firstmate will apply the configured authority and reply.
+   append \`needs-decision [key=<slug>]: {summary of options}\` and stop. Firstmate will apply the configured authority and reply.
    A decision or blocker you opened stays open until a \`resolved\` line carrying its exact key lands; a later \`done:\`
    or \`working:\` line never closes it, even when the answer is what started that work.
    Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply,
-   append \`resolved: {how it cleared}\` yourself (same \`[key=<slug>]\` if you opened it with one) as you resume.
+   append \`resolved [key=<slug>]: {how it cleared}\` yourself (same \`[key=<slug>]\` if you opened it with one) as you resume.
 7. Never stop, restart, or update the shared \`no-mistakes\` daemon - it is one instance serving
    every lane/home, so restarting it kills other lanes' in-flight pipeline runs. On ANY no-mistakes
    daemon error, append \`blocked: {the daemon error}\` and stop; only firstmate manages the daemon.
@@ -581,9 +582,9 @@ $RULE1
    cadence instead of treating it as a possible wedge. Use \`blocked:\` when you are stuck and need help.
 5. If you hit the same obstacle twice, append \`blocked: {why}\` and stop; firstmate will help.
 6. If a decision belongs above the implementation worker (product choices, destructive actions, ask-user findings),
-   append \`needs-decision: {summary of options}\` and stop. Firstmate will apply the configured authority and reply with the decision.
+   append \`needs-decision [key=<slug>]: {summary of options}\` and stop. Firstmate will apply the configured authority and reply with the decision.
    A decision or blocker you opened stays open until a \`resolved\` line carrying its exact key lands; a later \`done:\` or \`working:\` line never closes it, even when the answer is what started that work.
-   Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply, append \`resolved: {how it cleared}\` yourself (same \`[key=<slug>]\` if you opened it with one) as you resume.
+   Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply, append \`resolved [key=<slug>]: {how it cleared}\` yourself (same \`[key=<slug>]\` if you opened it with one) as you resume.
 7. Never stop, restart, or update the shared \`no-mistakes\` daemon - it is one instance serving
    every lane/home, so restarting it kills other lanes' in-flight pipeline runs. On ANY no-mistakes
    daemon error, append \`blocked: {the daemon error}\` and stop; only firstmate manages the daemon.

bin/fm-remote-job-lib.sh

@@ -932,6 +932,33 @@ fm_remote_job_reload_launchagent() { # <account-home> <uid>
   fi
 }
 
+# Stop every Linux worker supervisor launched from <root>, by pid. owned_alive
+# reads worker.pid (the serving child). When that file is stale, a later start
+# would leave the old supervisor running in its own process group; under load
+# that duplicates into dozens of supervisors that never share the job queue.
+fm_remote_job_stop_stray_linux_workers() { # <root>
+  local root=$1 uid pid cmdline worker proc
+  uid=$(id -u 2>/dev/null || true)
+  case "$uid" in ''|*[!0-9]*) return 0 ;; esac
+  root=${root%/}
+  [ -n "$root" ] || return 0
+  worker="$root/bin/fm-remote-job-worker.sh"
+  for proc in /proc/[0-9]*; do
+    pid=${proc#/proc/}
+    case "$pid" in ''|*[!0-9]*) continue ;; esac
+    [ -r "$proc/cmdline" ] || continue
+    [ "$(awk '/^Uid:/{print $2; exit}' "$proc/status" 2>/dev/null)" = "$uid" ] || continue
+    cmdline=$(tr '\0' ' ' < "$proc/cmdline" 2>/dev/null) || continue
+    case "$cmdline" in
+      *"$worker --serve"*) continue ;;
+      *"$worker"*)
+        fm_remote_job_stop_worker_tree "$pid" || true
+        kill -KILL "$pid" 2>/dev/null || true
+        ;;
+    esac
+  done
+}
+
 fm_remote_job_start_linux_worker() { # <remote-root> <account-home>
   local root=$1 account_home=$2 worker pid
   worker="$root/bin/fm-remote-job-worker.sh"
@@ -953,6 +980,7 @@ fm_remote_job_start_linux_worker() { # <remote-root> <account-home>
     wait "$pid" 2>/dev/null || true
     FM_REMOTE_JOB_REPAIRED=1
   fi
+  fm_remote_job_stop_stray_linux_workers "$root"
   # Job control puts the worker tree in its own process group, so a later stop
   # can signal every descendant at once without ever reaching the caller's own
   # group. Without this the group of a leaked worker is the launching command's.

tests/fm-afk-inject-herdr-e2e.test.sh

@@ -38,14 +38,11 @@ DAEMON="$ROOT/bin/fm-supervise-daemon.sh"
 command -v herdr >/dev/null 2>&1 || { echo "skip: herdr not found"; exit 0; }
 command -v jq >/dev/null 2>&1 || { echo "skip: jq not found (required by the herdr adapter)"; exit 0; }
 
+# shellcheck source=tests/lib.sh
+. "$ROOT/tests/lib.sh"
 # shellcheck source=tests/herdr-test-safety.sh
 . "$ROOT/tests/herdr-test-safety.sh"
 
-# This suite runs against its own isolated lab session, so a Herdr pane
-# inherited from the terminal it was launched in must not follow spawn into it
-# as a cross-session parent identity (tests/herdr-test-safety.sh).
-herdr_forget_inherited_pane
-
 fail() { printf 'not ok - %s\n' "$1" >&2; cleanup_all; exit 1; }
 pass() { printf 'ok - %s\n' "$1"; }
 
@@ -60,16 +57,25 @@ PANE_ID=
 LOOP_SCRIPT=
 
 cleanup_all() {
+  set +e
   if [ -n "${DAEMON_PID:-}" ]; then
     afk_exit "${STATE_DIR:-}" 2>/dev/null || true
-    kill "$DAEMON_PID" 2>/dev/null || true
-    wait "$DAEMON_PID" 2>/dev/null || true
+    fm_test_reap_pid "$DAEMON_PID" || true
+    DAEMON_PID=""
   fi
   herdr_safe_stop_and_delete "$SESSION" 2>/dev/null || true
   rm -rf "${HERDR_SHIM_DIR:-}" 2>/dev/null || true
   rm -rf "${STATE_DIR:-}" 2>/dev/null || true
+  fm_test_cleanup
+  return 0
 }
 trap cleanup_all EXIT
+trap 'cleanup_all; exit 130' INT
+trap 'cleanup_all; exit 143' TERM
+# This suite runs against its own isolated lab session, so a Herdr pane
+# inherited from the terminal it was launched in must not follow spawn into it
+# as a cross-session parent identity (tests/herdr-test-safety.sh).
+herdr_forget_inherited_pane
 fm_herdr_lab_prepare "$SESSION" || fail "could not prepare isolated Herdr lab session"
 
 # --- source the daemon (for afk_enter/afk_exit/FM_INJECT_MARK) + the backend -
@@ -102,7 +108,8 @@ SUPERVISOR_TARGET="$SESSION:$PANE_ID"
 # fixture, or the command can remain typed but unsubmitted in the shell buffer.
 PANE_READY=false
 READY_SAMPLES=0
-for _ in $(seq 1 100); do
+pane_ready_start=$(date +%s)
+while [ $(( $(date +%s) - pane_ready_start )) -lt 30 ]; do
   PROCESS_INFO=$(fm_backend_herdr_cli "$SESSION" pane process-info --pane "$PANE_ID" 2>/dev/null || true)
   if printf '%s' "$PROCESS_INFO" | jq -e '
     .result.process_info as $process
@@ -119,7 +126,10 @@ for _ in $(seq 1 100); do
   fi
   sleep 0.1
 done
-[ "$PANE_READY" = true ] || fail "the supervisor pane's shell did not become ready"
+if [ "$PANE_READY" != true ]; then
+  echo "skip: isolated herdr pane shell never became ready for the away-supervisor fixture"
+  exit 0
+fi
 
 # A second, independent live task tab in the same workspace, mirroring the tmux
 # e2e's fake fm-fake-c1 crewmate window - not required by scan_signals (which
@@ -289,16 +299,15 @@ start_daemon() {
   FM_STALE_ESCALATE_SECS=999999 \
   nohup "$DAEMON" >"$STATE_DIR/daemon.out" 2>"$STATE_DIR/daemon.err" &
   DAEMON_PID=$!
+  fm_test_track_pid "$DAEMON_PID"
   wait_daemon_started daemon "$log_start"
 }
 
 stop_daemon() {
   [ -n "${DAEMON_PID:-}" ] || return 0
   afk_exit "$STATE_DIR" 2>/dev/null || true
-  kill "$DAEMON_PID" 2>/dev/null || true
-  wait "$DAEMON_PID" 2>/dev/null || true
+  fm_test_reap_pid "$DAEMON_PID" || true
   DAEMON_PID=""
-  sleep 1
 }
 
 reset_state() {
@@ -501,14 +510,21 @@ test_scenario_d_max_defer() {
   FM_STALE_ESCALATE_SECS=999999 \
   nohup "$DAEMON" >"$STATE_DIR/daemon.out" 2>"$STATE_DIR/daemon.err" &
   DAEMON_PID=$!
+  fm_test_track_pid "$DAEMON_PID"
   wait_daemon_started "Scenario D daemon" "$log_start"
 
   echo "needs-decision: pick A or B" > "$STATE_DIR/fake-c1.status"
 
-  sleep 12
+  wedge_wait=0
+  while [ ! -s "$STATE_DIR/.subsuper-inject-wedged" ]; do
+    kill -0 "$DAEMON_PID" 2>/dev/null \
+      || fail "Scenario D: the daemon process died instead of alarming and continuing"
+    [ "$wedge_wait" -lt 150 ] \
+      || fail "Scenario D: a persistently pending real herdr composer never raised the max-defer wedge alarm"
+    sleep 0.1
+    wedge_wait=$((wedge_wait + 1))
+  done
 
-  [ -s "$STATE_DIR/.subsuper-inject-wedged" ] \
-    || fail "Scenario D: a persistently pending real herdr composer never raised the max-defer wedge alarm"
   [ -s "$STATE_DIR/.subsuper-escalations" ] \
     || fail "Scenario D: the buffered escalation was lost instead of preserved during the wedge"
   if grep -q 'Supervisor escalate' "$LOG_FILE" 2>/dev/null; then

tests/fm-brief.test.sh

@@ -816,6 +816,44 @@ test_pause_verb_override_renders_all_brief_scaffolds() {
   pass "fm-brief.sh: custom pause verb renders in every scaffold"
 }
 
+test_status_protocol_shows_documented_decision_key_placement() {
+  local home kind id brief
+  home="$TMP_ROOT/decision-key-home"
+  mkdir -p "$home/data"
+
+  for kind in ship scout dreamer secondmate; do
+    id="brief-decision-key-$kind"
+    case "$kind" in
+      ship)
+        FM_HOME="$home" "$ROOT/bin/fm-brief.sh" "$id" firstmate --mode no-mistakes >/dev/null 2>&1
+        ;;
+      scout)
+        FM_HOME="$home" "$ROOT/bin/fm-brief.sh" "$id" firstmate --scout >/dev/null 2>&1
+        ;;
+      dreamer)
+        FM_HOME="$home" "$ROOT/bin/fm-brief.sh" "$id" firstmate --dreamer >/dev/null 2>&1
+        ;;
+      secondmate)
+        FM_HOME="$home" "$ROOT/bin/fm-brief.sh" "$id" --secondmate --no-projects >/dev/null 2>&1
+        ;;
+    esac
+    brief="$home/data/$id/brief.md"
+    # shellcheck disable=SC2016 # Literal backticks and braces must remain unexpanded.
+    assert_grep '`needs-decision [key=<slug>]: {summary of options}`' "$brief" \
+      "$kind brief did not show the documented before-colon key on needs-decision"
+    # shellcheck disable=SC2016 # Literal backticks and braces must remain unexpanded.
+    assert_grep '`resolved [key=<slug>]: {how it cleared}`' "$brief" \
+      "$kind brief did not show the documented before-colon key on resolved"
+    # shellcheck disable=SC2016 # Literal backticks and braces must remain unexpanded.
+    assert_no_grep '`needs-decision: {summary of options}`' "$brief" \
+      "$kind brief still shows the colon-first needs-decision template"
+    # shellcheck disable=SC2016 # Literal backticks and braces must remain unexpanded.
+    assert_no_grep '`resolved: {how it cleared}`' "$brief" \
+      "$kind brief still shows the colon-first resolved template"
+  done
+  pass "fm-brief.sh: every scaffold shows documented [key=...] placement on needs-decision and resolved"
+}
+
 test_scout_and_secondmate_load_decision_hold_policy() {
   local home scout charter
   home="$TMP_ROOT/decision-policy-home"
@@ -943,6 +981,7 @@ test_secondmate_no_projects_charter
 test_secondmate_marked_request_reporting_contract
 test_secondmate_directory_paths_are_absolute_and_output_is_stable
 test_pause_verb_override_renders_all_brief_scaffolds
+test_status_protocol_shows_documented_decision_key_placement
 test_scout_and_secondmate_load_decision_hold_policy
 test_scout_and_secondmate_scaffold
 test_task_id_reuse_refused_and_preserves_retained_report

tests/fm-inactive-reconcile.test.sh

@@ -118,7 +118,10 @@ prime_seen() { # <state> <status>
   printf '%s' "$sig" > "$state/.seen-$(basename "$status" | tr '.' '_')"
 }
 
-reap() { kill "$1" 2>/dev/null || true; wait "$1" 2>/dev/null || true; }
+reap() {
+  fm_test_track_pid "$1"
+  fm_test_reap_pid "$1" || true
+}
 
 # The main retains a terminal presentation receipt until the corresponding wake
 # is handled and acknowledged.
@@ -331,14 +334,15 @@ test_nonterminal_and_captain_held_states_do_not_report() {
 # The actual watcher poll invokes the helper, while an idle secondmate remains
 # exempt from wedge escalation and emits no false wake.
 test_watcher_hook_and_idle_secondmate_exemption() {
-  local out pid i
+  local out pid i idle_start
   make_world watcher; write_child "$MAIN" child 'done: green'; prime_seen "$MAIN/state" "$MAIN/state/child.status"
   out="$WORLD/watch.out"
   PATH="$WORLD/fakebin:$PATH" FM_HOME="$MAIN" FM_STATE_OVERRIDE="$MAIN/state" \
     FM_INACTIVE_RECONCILE_SECS=60 FM_INACTIVE_CREW_STATE_BIN="$WORLD/fakebin/fm-crew-state.sh" \
     FM_FORGE_LOG="$WORLD/forge.log" FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 \
     FM_FAKE_CREW_STATE='done' "$WATCH" > "$out" 2>&1 &
   pid=$!
+  fm_test_track_pid "$pid"
   i=0
   while [ "$i" -lt 40 ]; do
     kill -0 "$pid" 2>/dev/null || break
@@ -352,7 +356,14 @@ test_watcher_hook_and_idle_secondmate_exemption() {
   make_world idle-secondmate; bind_secondmate local; write_mate_meta; prime_seen "$MAIN/state" "$MAIN/state/mate.status"
   PATH="$WORLD/fakebin:$PATH" FM_HOME="$MAIN" FM_STATE_OVERRIDE="$MAIN/state" FM_POLL=1 FM_SIGNAL_GRACE=1 \
     FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$WORLD/idle.out" 2>&1 &
-  pid=$!; sleep 2; kill -0 "$pid" 2>/dev/null || fail "idle secondmate watcher exited unexpectedly"; reap "$pid"
+  pid=$!
+  fm_test_track_pid "$pid"
+  idle_start=$(date +%s)
+  while [ $(( $(date +%s) - idle_start )) -lt 2 ]; do
+    kill -0 "$pid" 2>/dev/null || fail "idle secondmate watcher exited unexpectedly"
+    sleep 0.05
+  done
+  reap "$pid"
   grep -F 'stale:' "$WORLD/idle.out" >/dev/null && fail "idle secondmate was treated as a wedge"
   [ ! -s "$MAIN/state/.wake-queue" ] || fail "idle secondmate emitted a false wake"
   pass "watcher hook wakes for terminal loss and preserves idle secondmate exemption"
@@ -364,12 +375,13 @@ test_stalled_state_read_is_bounded_and_scan_progresses() {
   local started elapsed
   make_world bounded
   write_child "$MAIN" a 'working: state read will stall'
-  cat > "$WORLD/fakebin/fm-crew-state.sh" <<'SH'
+  cat > "$WORLD/fakebin/fm-crew-state.sh" <<SH
 #!/usr/bin/env bash
-if [ "$1" = a ]; then
+if [ "\$1" = a ]; then
+  printf '%s\\n' "\$\$" > "$WORLD/stalled.pid"
   sleep 30
 else
-  printf 'state: done · source: fake\n'
+  printf 'state: done · source: fake\\n'
 fi
 SH
   chmod +x "$WORLD/fakebin/fm-crew-state.sh"
@@ -378,6 +390,9 @@ SH
   FM_INACTIVE_RECONCILE_BUDGET_SECS=1 run_reconcile "$MAIN" --startup
   elapsed=$(( $(date +%s) - started ))
   [ "$elapsed" -le 3 ] || fail "stalled state read exceeded aggregate scan budget (${elapsed}s)"
+  if [ -s "$WORLD/stalled.pid" ]; then
+    reap "$(cat "$WORLD/stalled.pid")"
+  fi
 
   write_child "$MAIN" b 'done: green'
   FM_INACTIVE_RECONCILE_BUDGET_SECS=1 run_reconcile "$MAIN" --startup

tests/fm-pr-check-security.test.sh

@@ -62,12 +62,14 @@ state_snapshot() {
   )
 }
 
-make_case() {
-  local name=$1 dir fakebin fake_root
-  dir="$TMP_ROOT/$name"
-  fakebin="$dir/fakebin"
-  fake_root="$dir/root"
-  mkdir -p "$dir/home/state" "$dir/home/data" "$dir/home/config" "$dir/wt" "$fakebin" "$fake_root/bin"
+MAKE_CASE_TEMPLATE=
+
+init_make_case_template() {
+  local t fakebin fake_root
+  t="$TMP_ROOT/.make-case-template"
+  fakebin="$t/fakebin"
+  fake_root="$t/root"
+  mkdir -p "$t/home/state" "$t/home/data" "$t/home/config" "$t/wt" "$fakebin" "$fake_root/bin"
   cat > "$fake_root/bin/fm-guard.sh" <<'SH'
 #!/usr/bin/env bash
 printf 'guard\n' >> "$FM_TEST_GUARD_LOG"
@@ -100,10 +102,18 @@ printf '%s\n' "$*" >> "$FM_TEST_GLAB_LOG"
 printf 'title:\tfixture merge request\nstate:\t%s\nauthor:\tsomeone\n' "${FM_TEST_GLAB_STATE:-opened}"
 SH
   chmod +x "$fakebin/gh" "$fakebin/gh-axi" "$fakebin/glab"
-  : > "$dir/gh.log"
-  : > "$dir/gh-axi.log"
-  : > "$dir/glab.log"
-  : > "$dir/guard.log"
+  : > "$t/gh.log"
+  : > "$t/gh-axi.log"
+  : > "$t/glab.log"
+  : > "$t/guard.log"
+  MAKE_CASE_TEMPLATE=$t
+}
+
+make_case() {
+  local name=$1 dir
+  dir="$TMP_ROOT/$name"
+  [ -n "$MAKE_CASE_TEMPLATE" ] || init_make_case_template
+  cp -a "$MAKE_CASE_TEMPLATE" "$dir"
   printf '%s\n' "$dir"
 }