From 7285a95eba97e82e4314d710864c86fb15a32d81 Mon Sep 17 00:00:00 2001 From: PP <121104417+BohnBawerick@users.noreply.github.com> Date: Fri, 21 Aug 2026 10:15:00 +0800 Subject: [PATCH 1/5] fix: show documented decision-key placement in brief status examples Generated ship, scout, dreamer, and secondmate briefs now include literal needs-decision and resolved lines with [key=] before the colon, matching the documented position. --- bin/fm-brief.sh | 15 ++++++++------- tests/fm-brief.test.sh | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 7 deletions(-) diff --git a/bin/fm-brief.sh b/bin/fm-brief.sh index 9e5b175e85..595557523e 100755 --- a/bin/fm-brief.sh +++ b/bin/fm-brief.sh @@ -278,6 +278,7 @@ Report only true captain-relevant outcomes or a declared external wait by append States: working, needs-decision, blocked, $PAUSED_VERB, done, failed. Use \`$PAUSED_VERB: {why}\` (distinct from \`blocked:\`) only when your domain is deliberately idling on a known external wait you expect to clear on its own; use \`blocked:\` when you are stuck and need firstmate to act. Use this only for material phase changes, a captain decision, a real blocker, a failure, or work ready for review. +For a captain decision, append \`needs-decision [key=]: {summary of options}\`. This is also how you return the answer to a marked from-firstmate request above. A marked request requires one correlated answer after the work; it does not require a separate receipt or start acknowledgement. Never append \`working:\` merely to acknowledge receipt or announce that a marked request has started. @@ -285,7 +286,7 @@ When a routed-work phase has a supervisor-actionable material change worth repor If its first reportable event is \`working [key=]: {material phase}\`, use the same key on its later \`$PAUSED_VERB\`, \`done\`, \`failed\`, \`needs-decision\`, or \`blocked\` event so the earlier working phase is superseded. When a keyed phase ends without another reportable state, append \`resolved [key=]: {why it is no longer active}\`. \`resolved\` separately closes an escalated decision or blocker, and only a \`resolved\` line carrying that decision's exact key closes it: a later \`done\` or \`working\` event never does, even when the answer is what started that work. -The main firstmate's answer normally writes that closing line at answer time; when a blocker or wait clears WITHOUT an answer from the main firstmate, append \`resolved: {how it cleared}\` yourself (keyed with \`[key=]\` if you opened it with one) as your domain resumes. +The main firstmate's answer normally writes that closing line at answer time; when a blocker or wait clears WITHOUT an answer from the main firstmate, append \`resolved [key=]: {how it cleared}\` yourself (same \`[key=]\` if you opened it with one) as your domain resumes. Routine internal supervision, heartbeats, retries, and crewmate churn stay inside your own home and must not touch that status file. # Definition of done @@ -368,9 +369,9 @@ The report is the only thing that survives, so anything worth keeping must be in treating it as a possible wedge. Use \`blocked:\` when you are stuck and need help. 5. If you hit the same obstacle twice, append \`blocked: {why}\` and stop; firstmate will help. 6. If a decision belongs to a human (product choices, destructive actions), - append \`needs-decision: {summary of options}\` and stop. Firstmate will reply with the decision. + append \`needs-decision [key=]: {summary of options}\` and stop. Firstmate will reply with the decision. A decision or blocker you opened stays open until a \`resolved\` line carrying its exact key lands; a later \`done:\` or \`working:\` line never closes it, even when the answer is what started that work. - Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply, append \`resolved: {how it cleared}\` yourself (same \`[key=]\` if you opened it with one) as you resume. + Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply, append \`resolved [key=]: {how it cleared}\` yourself (same \`[key=]\` if you opened it with one) as you resume. 7. Never stop, restart, or update the shared \`no-mistakes\` daemon - it is one instance serving every lane/home, so restarting it kills other lanes' in-flight pipeline runs. On ANY no-mistakes daemon error, append \`blocked: {the daemon error}\` and stop; only firstmate manages the daemon. @@ -456,11 +457,11 @@ land in the generation or the report. treating it as a possible wedge. Use \`blocked:\` when you are stuck and need help. 5. If you hit the same obstacle twice, append \`blocked: {why}\` and stop; firstmate will help. 6. If a decision belongs above you (product choices, destructive actions, ask-user findings), - append \`needs-decision: {summary of options}\` and stop. Firstmate will apply the configured authority and reply. + append \`needs-decision [key=]: {summary of options}\` and stop. Firstmate will apply the configured authority and reply. A decision or blocker you opened stays open until a \`resolved\` line carrying its exact key lands; a later \`done:\` or \`working:\` line never closes it, even when the answer is what started that work. Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply, - append \`resolved: {how it cleared}\` yourself (same \`[key=]\` if you opened it with one) as you resume. + append \`resolved [key=]: {how it cleared}\` yourself (same \`[key=]\` if you opened it with one) as you resume. 7. Never stop, restart, or update the shared \`no-mistakes\` daemon - it is one instance serving every lane/home, so restarting it kills other lanes' in-flight pipeline runs. On ANY no-mistakes daemon error, append \`blocked: {the daemon error}\` and stop; only firstmate manages the daemon. @@ -581,9 +582,9 @@ $RULE1 cadence instead of treating it as a possible wedge. Use \`blocked:\` when you are stuck and need help. 5. If you hit the same obstacle twice, append \`blocked: {why}\` and stop; firstmate will help. 6. If a decision belongs above the implementation worker (product choices, destructive actions, ask-user findings), - append \`needs-decision: {summary of options}\` and stop. Firstmate will apply the configured authority and reply with the decision. + append \`needs-decision [key=]: {summary of options}\` and stop. Firstmate will apply the configured authority and reply with the decision. A decision or blocker you opened stays open until a \`resolved\` line carrying its exact key lands; a later \`done:\` or \`working:\` line never closes it, even when the answer is what started that work. - Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply, append \`resolved: {how it cleared}\` yourself (same \`[key=]\` if you opened it with one) as you resume. + Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply, append \`resolved [key=]: {how it cleared}\` yourself (same \`[key=]\` if you opened it with one) as you resume. 7. Never stop, restart, or update the shared \`no-mistakes\` daemon - it is one instance serving every lane/home, so restarting it kills other lanes' in-flight pipeline runs. On ANY no-mistakes daemon error, append \`blocked: {the daemon error}\` and stop; only firstmate manages the daemon. diff --git a/tests/fm-brief.test.sh b/tests/fm-brief.test.sh index 954930374b..97368085ce 100755 --- a/tests/fm-brief.test.sh +++ b/tests/fm-brief.test.sh @@ -816,6 +816,44 @@ test_pause_verb_override_renders_all_brief_scaffolds() { pass "fm-brief.sh: custom pause verb renders in every scaffold" } +test_status_protocol_shows_documented_decision_key_placement() { + local home kind id brief + home="$TMP_ROOT/decision-key-home" + mkdir -p "$home/data" + + for kind in ship scout dreamer secondmate; do + id="brief-decision-key-$kind" + case "$kind" in + ship) + FM_HOME="$home" "$ROOT/bin/fm-brief.sh" "$id" firstmate --mode no-mistakes >/dev/null 2>&1 + ;; + scout) + FM_HOME="$home" "$ROOT/bin/fm-brief.sh" "$id" firstmate --scout >/dev/null 2>&1 + ;; + dreamer) + FM_HOME="$home" "$ROOT/bin/fm-brief.sh" "$id" firstmate --dreamer >/dev/null 2>&1 + ;; + secondmate) + FM_HOME="$home" "$ROOT/bin/fm-brief.sh" "$id" --secondmate --no-projects >/dev/null 2>&1 + ;; + esac + brief="$home/data/$id/brief.md" + # shellcheck disable=SC2016 # Literal backticks and braces must remain unexpanded. + assert_grep '`needs-decision [key=]: {summary of options}`' "$brief" \ + "$kind brief did not show the documented before-colon key on needs-decision" + # shellcheck disable=SC2016 # Literal backticks and braces must remain unexpanded. + assert_grep '`resolved [key=]: {how it cleared}`' "$brief" \ + "$kind brief did not show the documented before-colon key on resolved" + # shellcheck disable=SC2016 # Literal backticks and braces must remain unexpanded. + assert_no_grep '`needs-decision: {summary of options}`' "$brief" \ + "$kind brief still shows the colon-first needs-decision template" + # shellcheck disable=SC2016 # Literal backticks and braces must remain unexpanded. + assert_no_grep '`resolved: {how it cleared}`' "$brief" \ + "$kind brief still shows the colon-first resolved template" + done + pass "fm-brief.sh: every scaffold shows documented [key=...] placement on needs-decision and resolved" +} + test_scout_and_secondmate_load_decision_hold_policy() { local home scout charter home="$TMP_ROOT/decision-policy-home" @@ -943,6 +981,7 @@ test_secondmate_no_projects_charter test_secondmate_marked_request_reporting_contract test_secondmate_directory_paths_are_absolute_and_output_is_stable test_pause_verb_override_renders_all_brief_scaffolds +test_status_protocol_shows_documented_decision_key_placement test_scout_and_secondmate_load_decision_hold_policy test_scout_and_secondmate_scaffold test_task_id_reuse_refused_and_preserves_retained_report From 1b63cc803b80b235bc9bcf4257acac7d54b9b539 Mon Sep 17 00:00:00 2001 From: PP <121104417+BohnBawerick@users.noreply.github.com> Date: Fri, 21 Aug 2026 19:08:47 +0800 Subject: [PATCH 2/5] fix(tests): make the suite honest, bounded, and leak-safe on loaded hosts Linux remote-job start stacked supervisors when worker.pid was stale, so heavy e2e scripts leaked process groups and missed observable waits. Stop stray supervisors for that code root before launching another. Tests now wait on observables instead of pid 1 or tick counts, reap daemons by pid, skip a herdr pane that never becomes ready, and hoist repeated PR-check fixtures. --- bin/fm-remote-job-lib.sh | 28 +++++++++ tests/fm-afk-inject-herdr-e2e.test.sh | 46 +++++++++----- tests/fm-inactive-reconcile.test.sh | 27 +++++++-- tests/fm-pr-check-security.test.sh | 30 ++++++---- tests/fm-remote-job-orphan-reap.test.sh | 47 ++++++++++++++- tests/fm-remote-reply.test.sh | 12 +++- ...fm-remote-secondmate-lifecycle-e2e.test.sh | 60 +++++++++++-------- 7 files changed, 187 insertions(+), 63 deletions(-) diff --git a/bin/fm-remote-job-lib.sh b/bin/fm-remote-job-lib.sh index 73bffa54c7..31a6bf58cf 100755 --- a/bin/fm-remote-job-lib.sh +++ b/bin/fm-remote-job-lib.sh @@ -932,6 +932,33 @@ fm_remote_job_reload_launchagent() { # fi } +# Stop every Linux worker supervisor launched from , by pid. owned_alive +# reads worker.pid (the serving child). When that file is stale, a later start +# would leave the old supervisor running in its own process group; under load +# that duplicates into dozens of supervisors that never share the job queue. +fm_remote_job_stop_stray_linux_workers() { # + local root=$1 uid pid cmdline worker proc + uid=$(id -u 2>/dev/null || true) + case "$uid" in ''|*[!0-9]*) return 0 ;; esac + root=${root%/} + [ -n "$root" ] || return 0 + worker="$root/bin/fm-remote-job-worker.sh" + for proc in /proc/[0-9]*; do + pid=${proc#/proc/} + case "$pid" in ''|*[!0-9]*) continue ;; esac + [ -r "$proc/cmdline" ] || continue + [ "$(awk '/^Uid:/{print $2; exit}' "$proc/status" 2>/dev/null)" = "$uid" ] || continue + cmdline=$(tr '\0' ' ' < "$proc/cmdline" 2>/dev/null) || continue + case "$cmdline" in + *"$worker --serve"*) continue ;; + *"$worker"*) + fm_remote_job_stop_worker_tree "$pid" || true + kill -KILL "$pid" 2>/dev/null || true + ;; + esac + done +} + fm_remote_job_start_linux_worker() { # local root=$1 account_home=$2 worker pid worker="$root/bin/fm-remote-job-worker.sh" @@ -953,6 +980,7 @@ fm_remote_job_start_linux_worker() { # wait "$pid" 2>/dev/null || true FM_REMOTE_JOB_REPAIRED=1 fi + fm_remote_job_stop_stray_linux_workers "$root" # Job control puts the worker tree in its own process group, so a later stop # can signal every descendant at once without ever reaching the caller's own # group. Without this the group of a leaked worker is the launching command's. diff --git a/tests/fm-afk-inject-herdr-e2e.test.sh b/tests/fm-afk-inject-herdr-e2e.test.sh index e761336e7b..72fa9019b0 100755 --- a/tests/fm-afk-inject-herdr-e2e.test.sh +++ b/tests/fm-afk-inject-herdr-e2e.test.sh @@ -38,14 +38,11 @@ DAEMON="$ROOT/bin/fm-supervise-daemon.sh" command -v herdr >/dev/null 2>&1 || { echo "skip: herdr not found"; exit 0; } command -v jq >/dev/null 2>&1 || { echo "skip: jq not found (required by the herdr adapter)"; exit 0; } +# shellcheck source=tests/lib.sh +. "$ROOT/tests/lib.sh" # shellcheck source=tests/herdr-test-safety.sh . "$ROOT/tests/herdr-test-safety.sh" -# This suite runs against its own isolated lab session, so a Herdr pane -# inherited from the terminal it was launched in must not follow spawn into it -# as a cross-session parent identity (tests/herdr-test-safety.sh). -herdr_forget_inherited_pane - fail() { printf 'not ok - %s\n' "$1" >&2; cleanup_all; exit 1; } pass() { printf 'ok - %s\n' "$1"; } @@ -60,16 +57,25 @@ PANE_ID= LOOP_SCRIPT= cleanup_all() { + set +e if [ -n "${DAEMON_PID:-}" ]; then afk_exit "${STATE_DIR:-}" 2>/dev/null || true - kill "$DAEMON_PID" 2>/dev/null || true - wait "$DAEMON_PID" 2>/dev/null || true + fm_test_reap_pid "$DAEMON_PID" || true + DAEMON_PID="" fi herdr_safe_stop_and_delete "$SESSION" 2>/dev/null || true rm -rf "${HERDR_SHIM_DIR:-}" 2>/dev/null || true rm -rf "${STATE_DIR:-}" 2>/dev/null || true + fm_test_cleanup + return 0 } trap cleanup_all EXIT +trap 'cleanup_all; exit 130' INT +trap 'cleanup_all; exit 143' TERM +# This suite runs against its own isolated lab session, so a Herdr pane +# inherited from the terminal it was launched in must not follow spawn into it +# as a cross-session parent identity (tests/herdr-test-safety.sh). +herdr_forget_inherited_pane fm_herdr_lab_prepare "$SESSION" || fail "could not prepare isolated Herdr lab session" # --- source the daemon (for afk_enter/afk_exit/FM_INJECT_MARK) + the backend - @@ -102,7 +108,8 @@ SUPERVISOR_TARGET="$SESSION:$PANE_ID" # fixture, or the command can remain typed but unsubmitted in the shell buffer. PANE_READY=false READY_SAMPLES=0 -for _ in $(seq 1 100); do +pane_ready_start=$(date +%s) +while [ $(( $(date +%s) - pane_ready_start )) -lt 30 ]; do PROCESS_INFO=$(fm_backend_herdr_cli "$SESSION" pane process-info --pane "$PANE_ID" 2>/dev/null || true) if printf '%s' "$PROCESS_INFO" | jq -e ' .result.process_info as $process @@ -119,7 +126,10 @@ for _ in $(seq 1 100); do fi sleep 0.1 done -[ "$PANE_READY" = true ] || fail "the supervisor pane's shell did not become ready" +if [ "$PANE_READY" != true ]; then + echo "skip: isolated herdr pane shell never became ready for the away-supervisor fixture" + exit 0 +fi # A second, independent live task tab in the same workspace, mirroring the tmux # e2e's fake fm-fake-c1 crewmate window - not required by scan_signals (which @@ -289,16 +299,15 @@ start_daemon() { FM_STALE_ESCALATE_SECS=999999 \ nohup "$DAEMON" >"$STATE_DIR/daemon.out" 2>"$STATE_DIR/daemon.err" & DAEMON_PID=$! + fm_test_track_pid "$DAEMON_PID" wait_daemon_started daemon "$log_start" } stop_daemon() { [ -n "${DAEMON_PID:-}" ] || return 0 afk_exit "$STATE_DIR" 2>/dev/null || true - kill "$DAEMON_PID" 2>/dev/null || true - wait "$DAEMON_PID" 2>/dev/null || true + fm_test_reap_pid "$DAEMON_PID" || true DAEMON_PID="" - sleep 1 } reset_state() { @@ -501,14 +510,21 @@ test_scenario_d_max_defer() { FM_STALE_ESCALATE_SECS=999999 \ nohup "$DAEMON" >"$STATE_DIR/daemon.out" 2>"$STATE_DIR/daemon.err" & DAEMON_PID=$! + fm_test_track_pid "$DAEMON_PID" wait_daemon_started "Scenario D daemon" "$log_start" echo "needs-decision: pick A or B" > "$STATE_DIR/fake-c1.status" - sleep 12 + wedge_wait=0 + while [ ! -s "$STATE_DIR/.subsuper-inject-wedged" ]; do + kill -0 "$DAEMON_PID" 2>/dev/null \ + || fail "Scenario D: the daemon process died instead of alarming and continuing" + [ "$wedge_wait" -lt 150 ] \ + || fail "Scenario D: a persistently pending real herdr composer never raised the max-defer wedge alarm" + sleep 0.1 + wedge_wait=$((wedge_wait + 1)) + done - [ -s "$STATE_DIR/.subsuper-inject-wedged" ] \ - || fail "Scenario D: a persistently pending real herdr composer never raised the max-defer wedge alarm" [ -s "$STATE_DIR/.subsuper-escalations" ] \ || fail "Scenario D: the buffered escalation was lost instead of preserved during the wedge" if grep -q 'Supervisor escalate' "$LOG_FILE" 2>/dev/null; then diff --git a/tests/fm-inactive-reconcile.test.sh b/tests/fm-inactive-reconcile.test.sh index c462119420..700671bf3d 100755 --- a/tests/fm-inactive-reconcile.test.sh +++ b/tests/fm-inactive-reconcile.test.sh @@ -118,7 +118,10 @@ prime_seen() { # printf '%s' "$sig" > "$state/.seen-$(basename "$status" | tr '.' '_')" } -reap() { kill "$1" 2>/dev/null || true; wait "$1" 2>/dev/null || true; } +reap() { + fm_test_track_pid "$1" + fm_test_reap_pid "$1" || true +} # The main retains a terminal presentation receipt until the corresponding wake # is handled and acknowledged. @@ -331,7 +334,7 @@ test_nonterminal_and_captain_held_states_do_not_report() { # The actual watcher poll invokes the helper, while an idle secondmate remains # exempt from wedge escalation and emits no false wake. test_watcher_hook_and_idle_secondmate_exemption() { - local out pid i + local out pid i idle_start make_world watcher; write_child "$MAIN" child 'done: green'; prime_seen "$MAIN/state" "$MAIN/state/child.status" out="$WORLD/watch.out" PATH="$WORLD/fakebin:$PATH" FM_HOME="$MAIN" FM_STATE_OVERRIDE="$MAIN/state" \ @@ -339,6 +342,7 @@ test_watcher_hook_and_idle_secondmate_exemption() { FM_FORGE_LOG="$WORLD/forge.log" FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 \ FM_FAKE_CREW_STATE='done' "$WATCH" > "$out" 2>&1 & pid=$! + fm_test_track_pid "$pid" i=0 while [ "$i" -lt 40 ]; do kill -0 "$pid" 2>/dev/null || break @@ -352,7 +356,14 @@ test_watcher_hook_and_idle_secondmate_exemption() { make_world idle-secondmate; bind_secondmate local; write_mate_meta; prime_seen "$MAIN/state" "$MAIN/state/mate.status" PATH="$WORLD/fakebin:$PATH" FM_HOME="$MAIN" FM_STATE_OVERRIDE="$MAIN/state" FM_POLL=1 FM_SIGNAL_GRACE=1 \ FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$WORLD/idle.out" 2>&1 & - pid=$!; sleep 2; kill -0 "$pid" 2>/dev/null || fail "idle secondmate watcher exited unexpectedly"; reap "$pid" + pid=$! + fm_test_track_pid "$pid" + idle_start=$(date +%s) + while [ $(( $(date +%s) - idle_start )) -lt 2 ]; do + kill -0 "$pid" 2>/dev/null || fail "idle secondmate watcher exited unexpectedly" + sleep 0.05 + done + reap "$pid" grep -F 'stale:' "$WORLD/idle.out" >/dev/null && fail "idle secondmate was treated as a wedge" [ ! -s "$MAIN/state/.wake-queue" ] || fail "idle secondmate emitted a false wake" pass "watcher hook wakes for terminal loss and preserves idle secondmate exemption" @@ -364,12 +375,13 @@ test_stalled_state_read_is_bounded_and_scan_progresses() { local started elapsed make_world bounded write_child "$MAIN" a 'working: state read will stall' - cat > "$WORLD/fakebin/fm-crew-state.sh" <<'SH' + cat > "$WORLD/fakebin/fm-crew-state.sh" < "$WORLD/stalled.pid" sleep 30 else - printf 'state: done · source: fake\n' + printf 'state: done · source: fake\\n' fi SH chmod +x "$WORLD/fakebin/fm-crew-state.sh" @@ -378,6 +390,9 @@ SH FM_INACTIVE_RECONCILE_BUDGET_SECS=1 run_reconcile "$MAIN" --startup elapsed=$(( $(date +%s) - started )) [ "$elapsed" -le 3 ] || fail "stalled state read exceeded aggregate scan budget (${elapsed}s)" + if [ -s "$WORLD/stalled.pid" ]; then + reap "$(cat "$WORLD/stalled.pid")" + fi write_child "$MAIN" b 'done: green' FM_INACTIVE_RECONCILE_BUDGET_SECS=1 run_reconcile "$MAIN" --startup diff --git a/tests/fm-pr-check-security.test.sh b/tests/fm-pr-check-security.test.sh index 03c6ce688e..b37b372ab5 100755 --- a/tests/fm-pr-check-security.test.sh +++ b/tests/fm-pr-check-security.test.sh @@ -62,12 +62,14 @@ state_snapshot() { ) } -make_case() { - local name=$1 dir fakebin fake_root - dir="$TMP_ROOT/$name" - fakebin="$dir/fakebin" - fake_root="$dir/root" - mkdir -p "$dir/home/state" "$dir/home/data" "$dir/home/config" "$dir/wt" "$fakebin" "$fake_root/bin" +MAKE_CASE_TEMPLATE= + +init_make_case_template() { + local t fakebin fake_root + t="$TMP_ROOT/.make-case-template" + fakebin="$t/fakebin" + fake_root="$t/root" + mkdir -p "$t/home/state" "$t/home/data" "$t/home/config" "$t/wt" "$fakebin" "$fake_root/bin" cat > "$fake_root/bin/fm-guard.sh" <<'SH' #!/usr/bin/env bash printf 'guard\n' >> "$FM_TEST_GUARD_LOG" @@ -100,10 +102,18 @@ printf '%s\n' "$*" >> "$FM_TEST_GLAB_LOG" printf 'title:\tfixture merge request\nstate:\t%s\nauthor:\tsomeone\n' "${FM_TEST_GLAB_STATE:-opened}" SH chmod +x "$fakebin/gh" "$fakebin/gh-axi" "$fakebin/glab" - : > "$dir/gh.log" - : > "$dir/gh-axi.log" - : > "$dir/glab.log" - : > "$dir/guard.log" + : > "$t/gh.log" + : > "$t/gh-axi.log" + : > "$t/glab.log" + : > "$t/guard.log" + MAKE_CASE_TEMPLATE=$t +} + +make_case() { + local name=$1 dir + dir="$TMP_ROOT/$name" + [ -n "$MAKE_CASE_TEMPLATE" ] || init_make_case_template + cp -a "$MAKE_CASE_TEMPLATE" "$dir" printf '%s\n' "$dir" } diff --git a/tests/fm-remote-job-orphan-reap.test.sh b/tests/fm-remote-job-orphan-reap.test.sh index 0c52a4c901..f7137a93a7 100755 --- a/tests/fm-remote-job-orphan-reap.test.sh +++ b/tests/fm-remote-job-orphan-reap.test.sh @@ -61,6 +61,23 @@ wait_child() { # return 1 } +# Wait until is no longer a child of this test shell. Linux reparents an +# orphan to pid 1, but a child subreaper (WSL SessionLeader, systemd --user, a +# harness) can adopt it instead. The leak this fixture needs is a worker that +# outlived its launching helper, not a particular reaper pid. +wait_orphaned_from_test() { # + local pid=$1 deadline=$(( $(date +%s) + $2 )) ppid + while [ "$(date +%s)" -lt "$deadline" ]; do + alive "$pid" || return 1 + ppid=$(ppid_of "$pid") + if [ -n "$ppid" ] && [ "$ppid" != "$$" ]; then + return 0 + fi + sleep 0.05 + done + return 1 +} + # --- a real worker fixture, launched exactly the way fm-on's Linux start does - # build_remote_root : a minimal but genuine Firstmate code root carrying @@ -123,8 +140,8 @@ SERVE=$(pgrep -P "$WORKER" | head -n 1) fail "the serving child is outside the worker's process group" pass "the Linux start path puts the whole worker tree in its own process group" -[ "$(ppid_of "$WORKER")" = 1 ] || - fail "the fixture worker is not orphaned to init, so this case does not reproduce the leak" +wait_orphaned_from_test "$WORKER" 5 || + fail "the fixture worker is still a child of the test shell (ppid=$(ppid_of "$WORKER"); test=$$), so this case does not reproduce a launcher that has exited" # The exact teardown shape that leaked in production: a fixture cleanup removes # the worker's state root and then stops only the single recorded worker pid - @@ -137,7 +154,7 @@ kill -KILL "$SERVE" 2>/dev/null || true wait_gone "$SERVE" 10 || fail "the recorded serving child did not stop" alive "$WORKER" || fail "the fixture supervisor did not survive a lone child kill, so this case no longer covers the leak" wait_child "$WORKER" 15 || fail "the supervisor did not respawn after its recorded child pid was killed" -pass "removing the state root and killing the recorded worker pid leaves the tree running at ppid 1" +pass "removing the state root and killing the recorded worker pid leaves the tree running after the launcher has exited" # A worker whose code root is intact is never a reap candidate, which is what # keeps the account's healthy LaunchAgent worker out of scope. @@ -203,3 +220,27 @@ pass "the reaper stops an abandoned worker's whole tree" out=$("$REAPER" 2>&1) || fail "a repeat reaper run failed: $out" assert_not_contains "$out" "$STALE" "the reaper reported an already-stopped worker" pass "the reaper is idempotent" + +# --- start must not stack supervisors when worker.pid is stale -------------- + +CASE3="$TMP_ROOT/case3" +mkdir -p "$CASE3/account" +build_remote_root "$CASE3/remote-root" +WORKER=$(start_worker "$CASE3/remote-root" "$CASE3/account" "$CASE3/remote-jobs") || + fail "could not start the stale-pid fixture worker" +track "$WORKER" +wait_child "$WORKER" 10 || fail "the stale-pid fixture worker never started its serving child" +SERVE=$(pgrep -P "$WORKER" | head -n 1) +rm -f "$CASE3/remote-jobs/worker.pid" +kill -KILL "$SERVE" 2>/dev/null || true +wait_gone "$SERVE" 5 || fail "the recorded serving child did not stop" +alive "$WORKER" || fail "the supervisor exited after its serving child was killed" +REPLACEMENT=$(start_worker "$CASE3/remote-root" "$CASE3/account" "$CASE3/remote-jobs") || + fail "could not start a replacement worker after worker.pid was removed" +track "$REPLACEMENT" +wait_orphaned_from_test "$REPLACEMENT" 5 || + fail "the replacement worker is still a child of the test shell" +[ "$REPLACEMENT" != "$WORKER" ] || fail "the replacement start returned the previous supervisor pid" +alive "$REPLACEMENT" || fail "the replacement supervisor exited immediately" +wait_gone "$WORKER" 10 || fail "a start with a stale pid file left the previous supervisor running" +pass "a Linux start with a stale pid file stops the previous supervisor before launching another" diff --git a/tests/fm-remote-reply.test.sh b/tests/fm-remote-reply.test.sh index 40fe9f0ba7..3d9aa6a99d 100755 --- a/tests/fm-remote-reply.test.sh +++ b/tests/fm-remote-reply.test.sh @@ -20,14 +20,20 @@ mkdir -p "$PARENT/data" "$PARENT/state" "$REMOTE/state" "$REMOTE/data/reply" "$C # stopping that pid alone leaves the supervisor to respawn - the leak # tests/fm-remote-job-orphan-reap.test.sh pins. Stop the whole worker tree. cleanup() { - local worker_pid='' + set +e + local worker_pid='' i=0 FM_HOME="$PARENT" FM_PROCEVENT_CLAIM_ROOT="$CLAIMS" \ - "$ROOT/bin/fm-procevent.sh" sweep-home >/dev/null 2>&1 || true + "$ROOT/bin/fm-procevent.sh" sweep-home >/dev/null 2>&1 if [ -f "$TMP_ROOT/remote-jobs/worker.pid" ]; then worker_pid=$(cat "$TMP_ROOT/remote-jobs/worker.pid") - fm_remote_job_stop_worker_tree "$worker_pid" || true + fm_remote_job_stop_worker_tree "$worker_pid" + while [ -n "$worker_pid" ] && kill -0 "$worker_pid" 2>/dev/null && [ "$i" -lt 50 ]; do + i=$((i + 1)) + sleep 0.05 + done fi rm -rf -- "$TMP_ROOT" + return 0 } trap cleanup EXIT diff --git a/tests/fm-remote-secondmate-lifecycle-e2e.test.sh b/tests/fm-remote-secondmate-lifecycle-e2e.test.sh index 9e6bfbba4d..ff3a9515d0 100755 --- a/tests/fm-remote-secondmate-lifecycle-e2e.test.sh +++ b/tests/fm-remote-secondmate-lifecycle-e2e.test.sh @@ -6,6 +6,8 @@ set -u . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" # shellcheck source=tests/remote-herdr-fixture.sh . "$(dirname "${BASH_SOURCE[0]}")/remote-herdr-fixture.sh" +# shellcheck source=bin/fm-remote-job-lib.sh +. "$ROOT/bin/fm-remote-job-lib.sh" command -v jq >/dev/null 2>&1 || { echo "skip: jq not found"; exit 0; } ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd -P) @@ -26,29 +28,45 @@ TMUX_STATE="$TMP_ROOT/remote-tmux.state" CLAIMS="$TMP_ROOT/claims" mkdir -p "$PARENT/data" "$PARENT/state" "$PARENT/config" "$PARENT/projects" "$REMOTE_ROOT" "$CLAIMS" cleanup() { - local worker_pid='' wait_attempt=0 + set +e + local worker_pid='' touch "$TMP_ROOT/provision.release" "$TMP_ROOT/seed.release" "$TMP_ROOT/handoff.release" \ "$TMP_ROOT/inherit.release" "$TMP_ROOT/launch.release" 2>/dev/null || true FM_HOME="$PARENT" FM_PROCEVENT_CLAIM_ROOT="$CLAIMS" \ - "$ROOT/bin/fm-procevent.sh" sweep-home >/dev/null 2>&1 || true + "$ROOT/bin/fm-procevent.sh" sweep-home >/dev/null 2>&1 if [ -f "$TMP_ROOT/remote-jobs/worker.pid" ]; then worker_pid=$(cat "$TMP_ROOT/remote-jobs/worker.pid") - kill "$worker_pid" 2>/dev/null || true - while kill -0 "$worker_pid" 2>/dev/null && [ "$wait_attempt" -lt 100 ]; do - wait_attempt=$((wait_attempt + 1)) - sleep 0.05 - done + fm_remote_job_stop_worker_tree "$worker_pid" + fi + if [ -n "${REMOTE_ROOT:-}" ]; then + fm_remote_job_stop_stray_linux_workers "$REMOTE_ROOT" fi rm -rf -- "$TMP_ROOT" + return 0 } trap cleanup EXIT +# Wait until exists, or fail. must stay alive while waiting. +# Wall-clock bound, not a tick count: under load sleep 0.02 is not 0.02s, and +# earlier inherited files traverse the remote job worker before captain-shared.md. +wait_until_file() { # + local path=$1 timeout=$2 pid=$3 exited_msg=$4 timeout_msg=$5 start now + start=$(date +%s) + while [ ! -f "$path" ]; do + kill -0 "$pid" 2>/dev/null || fail "$exited_msg" + now=$(date +%s) + [ $((now - start)) -lt "$timeout" ] || fail "$timeout_msg" + sleep 0.02 + done +} + # Materialize the current branch as the remote host's tracked code root. The # fixture is a real git repository because provisioning and guarded sync exercise # the same clone and fast-forward path as a second Mac. ( cd "$ROOT" || exit - tar --exclude=.git --exclude=.no-mistakes --exclude=data --exclude=state --exclude=config -cf - . + tar --exclude=.git --exclude=.no-mistakes --exclude=data --exclude=state --exclude=config \ + --exclude=tests --exclude=docs --exclude=.github -cf - . ) | (cd "$REMOTE_ROOT" && tar -xf -) cat > "$REMOTE_ROOT/bin/tmux" < "$TMP_ROOT/spawn-concurrent.out" 2>&1 & spawn_concurrent=$! -spawn_inherit_wait=0 -# Earlier inherited files traverse the worker before captain-shared.md, so give -# a loaded portable runner 30 seconds to reach this deliberately blocked write. -while [ ! -f "$TMP_ROOT/inherit.entered" ]; do - kill -0 "$spawn_concurrent" 2>/dev/null || fail "remote spawn exited before its blocked inheritance write" - spawn_inherit_wait=$((spawn_inherit_wait + 1)) - [ "$spawn_inherit_wait" -le 1500 ] || fail "remote spawn never reached its blocked inheritance write" - sleep 0.02 -done +# Earlier inherited files traverse the worker before captain-shared.md. Wait on +# the blocked write itself rather than a tick count that a loaded runner misses. +wait_until_file "$TMP_ROOT/inherit.entered" 120 "$spawn_concurrent" \ + "remote spawn exited before its blocked inheritance write" \ + "remote spawn never reached its blocked inheritance write" cat > "$PARENT/data/captain-shared.md" <<'EOF' # Shared captain preferences This file is main-authoritative and maintained by the main firstmate. @@ -923,15 +937,9 @@ EOF FM_FAKE_SSH_MODE=inherit-block remote_env "$ROOT/bin/fm-config-push.sh" \ > "$TMP_ROOT/config-concurrent-first.out" 2>&1 & config_first=$! -inherit_wait=0 -while [ ! -f "$TMP_ROOT/inherit.entered" ]; do - kill -0 "$config_first" 2>/dev/null || fail "first inheritance transaction exited before its blocked write" - inherit_wait=$((inherit_wait + 1)) - # Match the earlier spawn/inheritance wait: a loaded portable runner can - # spend several seconds in the remote entrypoint before reaching this write. - [ "$inherit_wait" -le 1500 ] || fail "first inheritance transaction never reached its blocked write" - sleep 0.02 -done +wait_until_file "$TMP_ROOT/inherit.entered" 120 "$config_first" \ + "first inheritance transaction exited before its blocked write" \ + "first inheritance transaction never reached its blocked write" cat > "$PARENT/data/captain-shared.md" <<'EOF' # Shared captain preferences This file is main-authoritative and maintained by the main firstmate. From c624cfd3288211276b7c85270f4cf105670b6cc9 Mon Sep 17 00:00:00 2001 From: PP <121104417+BohnBawerick@users.noreply.github.com> Date: Fri, 21 Aug 2026 20:53:29 +0800 Subject: [PATCH 3/5] fix: make Pi primary watcher away-mode aware and retire watcher markers at cleanup --- .pi/extensions/fm-primary-pi-watch.ts | 37 ++++++++-- bin/fm-teardown.sh | 3 + bin/fm-wake-lib.sh | 32 ++++++++ docs/supervision-protocols/pi.md | 2 + docs/watcher-continuity.md | 1 + tests/fm-pi-watch-extension.test.sh | 102 ++++++++++++++++++++++++++ tests/fm-teardown.test.sh | 57 ++++++++++++++ 7 files changed, 226 insertions(+), 8 deletions(-) diff --git a/.pi/extensions/fm-primary-pi-watch.ts b/.pi/extensions/fm-primary-pi-watch.ts index d679e73690..448f374f82 100644 --- a/.pi/extensions/fm-primary-pi-watch.ts +++ b/.pi/extensions/fm-primary-pi-watch.ts @@ -10,7 +10,7 @@ // callbacks from a prior generation are no-ops against the active replacement. import { spawn, spawnSync, type ChildProcess } from "node:child_process"; import { createHash } from "node:crypto"; -import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; import { dirname, resolve } from "node:path"; import { fileURLToPath } from "node:url"; import type { ExtensionAPI, Theme } from "@earendil-works/pi-coding-agent"; @@ -126,6 +126,18 @@ function pidAlive(pid: string): boolean { } } +// While state/.afk exists, the away daemon (bin/fm-supervise-daemon.sh) owns +// fleet supervision, triage, and wake escalation. The Pi extension must not +// spawn competing watcher children or deliver duplicate watcher wakes to the +// primary session while away mode is active. +function isAfkActive(): boolean { + try { + return existsSync(`${state}/.afk`); + } catch { + return false; + } +} + function lockOwnership(): LockOwnership { let lockPid = ""; try { @@ -243,7 +255,7 @@ export default function (pi: ExtensionAPI) { message: string, recovery?: { generation: string; watcherPid: string }, ): Promise { - if (!generationIsLive(owner)) return; + if (!generationIsLive(owner) || isAfkActive()) return; const content = encodeFirstmateOperationalInput( "watcher", `FIRSTMATE WATCHER WAKE: ${message}\n\nRun bin/fm-wake-drain.sh first and handle the queued wake. Watcher continuity is extension-owned.`, @@ -313,13 +325,15 @@ export default function (pi: ExtensionAPI) { }> { let failure = ""; for (let attempt = 0; attempt <= retryLimit; attempt += 1) { - if (!generationIsLive(owner)) return { failure: "" }; + if (!generationIsLive(owner) || isAfkActive()) return { failure: "" }; const replacement = startArm(owner, predecessorArmPid); + if (isAfkActive()) return { failure: "" }; const successorChild = owner.child; if (replacement.ok && successorChild && await waitForReadiness(successorChild)) { return { failure: "", recovery: armRecovery.get(successorChild) }; } if (replacement.ok) { + if (isAfkActive()) return { failure: "" }; failure = "watcher: FAILED - Pi extension could not verify a ready successor watcher"; if (!(await retireArm(successorChild))) { return { @@ -339,7 +353,7 @@ export default function (pi: ExtensionAPI) { } function scheduleRetry(owner: SessionGeneration, message: string, predecessorArmPid: string): void { - if (!generationIsLive(owner) || owner.child || owner.retryTimer) return; + if (!generationIsLive(owner) || owner.child || owner.retryTimer || isAfkActive()) return; const ownership = lockOwnership(); if (ownership !== "owned") { surfaceFailure(owner, `watcher: FAILED - Pi extension cannot restore continuity because this session no longer owns the lock\n${message}`); @@ -352,9 +366,9 @@ export default function (pi: ExtensionAPI) { } const timer = setTimeout(() => { if (owner.retryTimer === timer) owner.retryTimer = null; - if (!generationIsLive(owner)) return; + if (!generationIsLive(owner) || isAfkActive()) return; const result = startArm(owner, predecessorArmPid); - if (!result.ok) { + if (!result.ok && !isAfkActive()) { surfaceFailure(owner, `watcher: FAILED - Pi extension could not launch a continuity retry\n${result.message}`); } }, retryDelay(owner.retryFailures)); @@ -364,6 +378,12 @@ export default function (pi: ExtensionAPI) { function startArm(owner: SessionGeneration, predecessorArmPid = ""): ArmResult { if (!generationIsLive(owner)) return { ok: false, message: shuttingDownMessage }; + if (isAfkActive()) { + return { + ok: true, + message: "watcher: away mode active - supervise daemon owns watcher supervision", + }; + } const ownership = lockOwnership(); if (ownership === "other") return { ok: false, message: "watcher: read-only - session lock is held by another firstmate session" }; if (ownership === "missing") { @@ -445,6 +465,7 @@ export default function (pi: ExtensionAPI) { settleReadiness(false); releaseChild(); if (!generationIsLive(owner)) return; + if (isAfkActive()) return; const classification = classifyClose(stdout, stderr, code, signal); const predecessor = String(armChild.pid ?? ""); if (classification.kind === "actionable") { @@ -453,7 +474,7 @@ export default function (pi: ExtensionAPI) { void (async () => { const restoration = await restoreAfterActionableClose(owner, predecessor); if (generationIsLive(owner)) owner.restoring = false; - if (!generationIsLive(owner)) return; + if (!generationIsLive(owner) || isAfkActive()) return; const message = restoration.failure ? `${classification.message}\n\n${restoration.failure}` : classification.message; await sendWake(owner, message, restoration.recovery); })().catch(() => { @@ -469,7 +490,7 @@ export default function (pi: ExtensionAPI) { resolveClosed(); settleReadiness(false); releaseChild(); - if (!generationIsLive(owner)) return; + if (!generationIsLive(owner) || isAfkActive()) return; if (owner.restoring) return; scheduleRetry(owner, `watcher: FAILED - Pi extension arm child ${id} failed: ${error.message}`, String(armChild.pid ?? "")); }); diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index abd8dacbb0..6d0c3b2066 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -396,6 +396,7 @@ remote_secondmate_teardown() { grep -vE "^- $ID( |$)" "$SECONDMATE_REG" > "$tmp" || true mv -f -- "$tmp" "$SECONDMATE_REG" status_retire_presentation_task "$STATE" "$ID" || return 1 + fm_wake_retire_task_markers "$STATE" "$ID" "${T:-}" || return 1 rm -f -- "$STATE/$ID.meta" "$STATE/$ID.turn-ended" printf 'teardown %s complete (remote %s:%s)\n' "$ID" "$remote_host" "$remote_home" return 0 @@ -2289,6 +2290,7 @@ cleanup_firstmate_home_children() { fi retire_busy_state "$sub_state" "$child_id" "$child_busy_gen" || return 1 status_retire_presentation_task "$sub_state" "$child_id" || return 1 + fm_wake_retire_task_markers "$sub_state" "$child_id" "$child_t" || return 1 rm -f "$sub_state/$child_id.turn-ended" \ "$sub_state/$child_id.meta" "$sub_state/$child_id.pi-ext.ts" \ "$sub_state/$child_id.muse-session" "$sub_state/$child_id.muse-session-current" \ @@ -2567,6 +2569,7 @@ fm_backend_clear_transition "$BACKEND" "$STATE" "$T" || true remove_pr_poll_artifacts "$STATE" "$ID" || exit 1 retire_busy_state "$STATE" "$ID" "$BUSY_GEN" || exit 1 status_retire_presentation_task "$STATE" "$ID" || exit 1 +fm_wake_retire_task_markers "$STATE" "$ID" "$T" || exit 1 rm -f "$STATE/$ID.turn-ended" "$STATE/$ID.meta" \ "$STATE/$ID.pi-ext.ts" "$STATE/$ID.muse-session" \ "$STATE/$ID.muse-session-current" "$STATE/$ID.cursor-session" \ diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index e08b462180..645e338684 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -1228,6 +1228,38 @@ fm_wake_status_append_self_announced() { # return 0 } +# Retire window-keyed and task-keyed watcher internals for a task being torn down: +# - .seen-* status and turn-ended signal markers +# - .hb-surfaced-* heartbeat push transition markers +# - .stale-*, .stale-since-*, .hash-*, .count-*, .wedge-escalations-*, .paused-* window markers +# This ensures a torn-down task's pane/window cannot leave leftover hash/stale +# state behind to falsely produce stale wakes after cleanup. +fm_wake_retire_task_markers() { # [] + local state=$1 id=$2 target=${3:-} key hb_key seen_status seen_turnend + [ -n "$state" ] && [ -n "$id" ] || return 0 + seen_status=$(fm_wake_signal_seen_path "$state" "$state/$id.status") + seen_turnend=$(fm_wake_signal_seen_path "$state" "$state/$id.turn-ended") + hb_key=$(printf '%s' "$id" | tr ':/.' '___') + rm -f -- \ + "$seen_status" \ + "$seen_turnend" \ + "$state/.hb-surfaced-$id" \ + "$state/.hb-surfaced-$hb_key" 2>/dev/null || true + if [ -n "$target" ]; then + key=$(printf '%s' "$target" | tr ':/.' '___') + rm -f -- \ + "$state/.stale-$key" \ + "$state/.stale-since-$key" \ + "$state/.hash-$key" \ + "$state/.count-$key" \ + "$state/.wedge-escalations-$key" \ + "$state/.paused-$key" \ + "$state/.paused-rechecked-$key" \ + "$state/.paused-resurfaced-$key" 2>/dev/null || true + fi + return 0 +} + # Map one structurally valid signal key to its home-local status filename. # Queue payload text is intentionally ignored: it is display data, not a path # authority. The caller still verifies the resulting regular file immediately diff --git a/docs/supervision-protocols/pi.md b/docs/supervision-protocols/pi.md index 5cdcaed7b0..91b0cfbc1b 100644 --- a/docs/supervision-protocols/pi.md +++ b/docs/supervision-protocols/pi.md @@ -19,6 +19,8 @@ When this session owns supervision and away mode is not active: 11. Never use shell `&` for watcher supervision. The arm mechanism above is extension-owned, not a model tool call, but a manual recovery probe that backgrounds, pipes, or bundles the arm is denied automatically by the PreToolUse seatbelt (`bin/fm-arm-pretool-check.sh`, wired into the turn-end guard extension at `__FM_PI_TURNEND_EXT__`). +While away mode (`state/.afk`) is active, the sub-supervisor daemon owns supervision and triage; the Pi extension stands down so wakes and arm processes do not duplicate. + The turn-end guard extension lives at `__FM_PI_TURNEND_EXT__`. The watcher extension lives at `__FM_PI_EXT__`. Both are tracked, project-local `.pi/extensions/*.ts` files that Pi auto-discovers once the project is trusted; `bin/fm-session-start.sh` reports when the running Pi session has not loaded both required extensions. diff --git a/docs/watcher-continuity.md b/docs/watcher-continuity.md index 1a94ec0ede..5a761b5827 100644 --- a/docs/watcher-continuity.md +++ b/docs/watcher-continuity.md @@ -19,6 +19,7 @@ A cycle-end failure is benign when that live-watcher predicate is true, and the Only an exhausted failure with no verified watcher emits one last-resort notice for the continuous failure episode; later consecutive Stop cycles exit 2 to guarantee another Stop-owned retry without repeating the notice until the turn-end guard consumes the attended fail-open. The Claude turn-end guard owns the monotonic failure progression, one-time attended fail-open, post-alarm continuation suppression, and positive recovery reset described in [`turnend-guard.md`](turnend-guard.md#harness-integrations). While supervision is still needed and away mode remains inactive, an actionable close wakes the idle session through exit 2. +While away mode (`state/.afk`) is active, the sub-supervisor daemon owns fleet supervision and triage, and primary watcher adapters stand down so wakes and arm processes are not duplicated. ## Actionable wake ordering diff --git a/tests/fm-pi-watch-extension.test.sh b/tests/fm-pi-watch-extension.test.sh index 9e29adc793..c2b4aad058 100755 --- a/tests/fm-pi-watch-extension.test.sh +++ b/tests/fm-pi-watch-extension.test.sh @@ -1074,6 +1074,107 @@ EOF pass "Pi session transitions use a generation owner across /new /resume /fork, stale callbacks, and quit" } +test_pi_away_mode_suppresses_wake_delivery_and_rearm() { + local repo home plugin log stop out status + repo="$TMP_ROOT/pi-afk-suppress-root" + home="$TMP_ROOT/pi-afk-suppress-home" + log="$TMP_ROOT/pi-afk-suppress.log" + stop="$TMP_ROOT/pi-afk-suppress.stop" + mkdir -p "$repo/bin" "$home/state" "$home/config" + install_pi_watch_extension_fixture "$repo" + plugin="$repo/.pi/extensions/fm-primary-pi-watch.ts" + cat > "$repo/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +printf 'arm %s\n' "$$" >> "${FM_ARM_LOG:?}" +if [ -n "${FM_WATCH_PREDECESSOR_ARM_PID:-}" ]; then + printf 'successor for %s\n' "$FM_WATCH_PREDECESSOR_ARM_PID" >> "${FM_ARM_LOG:?}" +fi +printf 'watcher: started pid=%s (beacon fresh)\n' "$$" +trap 'exit 0' TERM INT +while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done +printf 'stale: default:w4W:pM\n' +SH + chmod +x "$repo/bin/fm-watch-arm.sh" + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_STOP_FILE="$stop" node --input-type=module 2>&1 <<'EOF' +import { existsSync, readFileSync, unlinkSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +let tool = null; +let prompt = ""; +const pi = { + on() {}, + registerCommand() {}, + registerTool(candidate) { + if (candidate.name === "fm_watch_arm_pi") tool = candidate; + }, + sendUserMessage: async (message) => { + prompt = message; + }, +}; +const afkPath = `${process.env.FM_HOME}/state/.afk`; +const lockPath = `${process.env.FM_HOME}/state/.lock`; +writeFileSync(lockPath, `${process.pid}\n`); +const mod = await import(pathToFileURL(process.env.PLUGIN).href); +mod.default(pi); +if (!tool) throw new Error("Pi watch tool was not registered"); + +// (1) While state/.afk is present, startArm returns away-mode guidance and spawns no child. +writeFileSync(afkPath, `${Date.now()}\n`); +const afkArm = await tool.execute("tool-call-afk", {}, undefined, undefined, {}); +if (afkArm.details?.ok !== true || !afkArm.details.message.includes("away mode active")) { + throw new Error(`afk arm returned unexpected result: ${JSON.stringify(afkArm.details)}`); +} +if (existsSync(process.env.FM_ARM_LOG)) { + throw new Error("arm child was launched while state/.afk existed"); +} + +// (2) Start an arm child with .afk absent, then set .afk before the child closes with an actionable wake. +unlinkSync(afkPath); +const normalArm = await tool.execute("tool-call-normal", {}, undefined, undefined, {}); +if (normalArm.details?.ok !== true || !normalArm.details.message.includes("started Pi extension arm child")) { + throw new Error(`normal arm failed: ${JSON.stringify(normalArm.details)}`); +} +for (let i = 0; i < 250 && !existsSync(process.env.FM_ARM_LOG); i += 1) { + await new Promise((resolve) => setTimeout(resolve, 20)); +} +if (!existsSync(process.env.FM_ARM_LOG)) throw new Error("arm child did not start"); + +// Now activate away mode before child closes. +writeFileSync(afkPath, `${Date.now()}\n`); +// Signal child to close with actionable wake. +writeFileSync(process.env.FM_STOP_FILE, "stop\n"); + +// Wait for child to exit. +await new Promise((resolve) => setTimeout(resolve, 200)); + +// Assert no prompt was delivered into the Pi session. +if (prompt) { + throw new Error(`prompt was delivered during away mode: ${prompt}`); +} + +// Assert no successor child was launched while .afk is active. +const logLines = readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n"); +const armEntries = logLines.filter((line) => line.startsWith("arm ")); +if (armEntries.length !== 1) { + throw new Error(`unexpected successor launch during away mode: ${logLines.join(" | ")}`); +} + +// (3) Clear away mode and re-arm. It starts the child normally. +unlinkSync(afkPath); +if (existsSync(process.env.FM_STOP_FILE)) unlinkSync(process.env.FM_STOP_FILE); +const resumedArm = await tool.execute("tool-call-resumed", {}, undefined, undefined, {}); +if (resumedArm.details?.ok !== true || !resumedArm.details.message.includes("started Pi extension arm child")) { + throw new Error(`resumed arm failed: ${JSON.stringify(resumedArm.details)}`); +} +process.exit(0); +EOF +) + status=$? + expect_code 0 "$status" "Pi watcher extension must stand down and suppress wake delivery/re-arm while away mode is active" + [ -z "$out" ] || fail "Pi away-mode test printed output: $out" + pass "Pi extension stands down and suppresses wake delivery while away mode is active" +} + test_pi_process_exit_cleanup_listener_lifecycle() { local repo home plugin out status repo="$TMP_ROOT/pi-exit-listener-root" @@ -2163,6 +2264,7 @@ test_pi_established_empty_close_honors_retry_limit test_pi_actionable_close_rechecks_session_lock test_pi_arm_distinguishes_session_lock_ownership test_pi_session_transition_generation_owner +test_pi_away_mode_suppresses_wake_delivery_and_rearm test_pi_process_exit_cleanup_listener_lifecycle test_pi_process_exit_cleanup_stops_arm_child test_opencode_plugin_package_boundary_is_explicit_esm diff --git a/tests/fm-teardown.test.sh b/tests/fm-teardown.test.sh index 115b58d571..d88d12f13d 100755 --- a/tests/fm-teardown.test.sh +++ b/tests/fm-teardown.test.sh @@ -1378,6 +1378,62 @@ SH pass "herdr teardown removes pane-owned escalation dedupe state" } +test_teardown_retires_window_and_task_watcher_markers() { + local case_dir f + case_dir=$(make_case window-marker-retire) + write_meta "$case_dir" local-only ship + sed -i.bak 's/^window=.*/window=default:w4W:pM/' "$case_dir/state/task-x1.meta" + rm -f "$case_dir/state/task-x1.meta.bak" + printf '%s\n' \ + 'backend=herdr' \ + 'herdr_session=default' \ + 'herdr_workspace_id=w4W' \ + 'herdr_tab_id=w4W:tM' \ + 'herdr_pane_id=w4W:pM' >> "$case_dir/state/task-x1.meta" + cat > "$case_dir/fakebin/herdr" < "$case_dir/state/.stale-default_w4W_pM" + : > "$case_dir/state/.stale-since-default_w4W_pM" + : > "$case_dir/state/.hash-default_w4W_pM" + : > "$case_dir/state/.count-default_w4W_pM" + : > "$case_dir/state/.wedge-escalations-default_w4W_pM" + : > "$case_dir/state/.paused-default_w4W_pM" + : > "$case_dir/state/.paused-rechecked-default_w4W_pM" + : > "$case_dir/state/.paused-resurfaced-default_w4W_pM" + : > "$case_dir/state/.seen-task-x1_status" + : > "$case_dir/state/.seen-task-x1_turn-ended" + : > "$case_dir/state/.hb-surfaced-task-x1" + + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" \ + || fail "window-marker-retire: forced teardown failed: $(cat "$case_dir/stderr")" + + for f in \ + "$case_dir/state/.stale-default_w4W_pM" \ + "$case_dir/state/.stale-since-default_w4W_pM" \ + "$case_dir/state/.hash-default_w4W_pM" \ + "$case_dir/state/.count-default_w4W_pM" \ + "$case_dir/state/.wedge-escalations-default_w4W_pM" \ + "$case_dir/state/.paused-default_w4W_pM" \ + "$case_dir/state/.paused-rechecked-default_w4W_pM" \ + "$case_dir/state/.paused-resurfaced-default_w4W_pM" \ + "$case_dir/state/.seen-task-x1_status" \ + "$case_dir/state/.seen-task-x1_turn-ended" \ + "$case_dir/state/.hb-surfaced-task-x1"; do + [ ! -e "$f" ] || fail "window-marker-retire: teardown left $(basename "$f") behind" + done + pass "teardown retires window-keyed and task-keyed watcher internal markers" +} + # Flat (non-projected) Herdr endpoint whose fake pane exists until a locked # close removes it. The socket path is case-local so the derived presentation # lock never collides with another test or a real fleet session. @@ -2621,6 +2677,7 @@ test_no_mistakes_truly_unpushed_refuses test_local_only_force_overrides_unpushed test_teardown_missing_busy_sidecar_completes test_herdr_teardown_clears_escalation_marker +test_teardown_retires_window_and_task_watcher_markers test_herdr_flat_teardown_refuses_orphaning_records_then_retry_completes test_herdr_flat_teardown_refuses_records_on_unparseable_presence test_herdr_flat_teardown_preflight_refuses_before_changes From cb9887b97a28cf00f51ff9c9e199b14e42d4b2df Mon Sep 17 00:00:00 2001 From: PP <121104417+BohnBawerick@users.noreply.github.com> Date: Fri, 21 Aug 2026 21:28:29 +0800 Subject: [PATCH 4/5] no-mistakes(document): Document away-mode wake suppression and marker retirement --- docs/watcher-continuity.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/watcher-continuity.md b/docs/watcher-continuity.md index 5a761b5827..4fde916ba7 100644 --- a/docs/watcher-continuity.md +++ b/docs/watcher-continuity.md @@ -77,7 +77,7 @@ Only the watcher process touches `state/.last-watcher-beat`; no helper process c ## Regression coverage `tests/fm-pi-watch-extension.test.sh` checks Pi's first-cycle-or-explicit-repair tool metadata and ownership-based redundant-call no-ops, then simulates actionable and empty child closes against the actual Pi and OpenCode close handlers, blocks prompt delivery to prove the successor launches first, verifies single-flight behavior, changes the session lock before close to prove ownership is rechecked, and hangs each successor arm to prove bounded fallback delivery includes the typed restoration failure. -The same suite covers ordinary same-process session replacement for `/new`, `/resume`, and `/fork`, same-instance shutdown-plus-start, stale prior-generation callbacks, repeated transitions with exactly one live cycle, disappearance of the shutting-down refusal after a valid replacement activates, and terminal quit still refusing late rearm. +The same suite covers away-mode wake suppression and arm inhibition, ordinary same-process session replacement for `/new`, `/resume`, and `/fork`, same-instance shutdown-plus-start, stale prior-generation callbacks, repeated transitions with exactly one live cycle, disappearance of the shutting-down refusal after a valid replacement activates, and terminal quit still refusing late rearm. `tests/fm-watch-arm.test.sh` covers durable queue replay, real remote parent-replies ingestion into the authoritative status log, decision-only OPEN DECISIONS recovery, interrupted handling replay, generation-bound acknowledgement, a persistent live successor after recovery, a watcher close inside the handling window that must leave the printed acknowledgement valid, and the self-healing moved-generation acknowledgement that consumes its handled rows and names its remedy. `tests/fm-watcher-lock.test.sh` covers verified-successor attach, recovery publication before stale-lock removal, the typed self-eviction failure, bounded and successor-linked lifecycle rows, and a SIGSTOP counterfactual that distinguishes a live PID from a stale beacon before classifying termination. `tests/fm-subagent-pretool-check.test.sh` proves Claude retains only the non-status Bash seatbelts. From 672c6d0a019600692ddecf6da3b829e1f2da14cf Mon Sep 17 00:00:00 2001 From: PP <121104417+BohnBawerick@users.noreply.github.com> Date: Sat, 22 Aug 2026 01:07:21 +0800 Subject: [PATCH 5/5] no-mistakes: apply CI fixes --- .opencode/plugins/fm-primary-turnend-guard.js | 4 ++ .opencode/plugins/lib/fm-operational-input.js | 4 ++ .pi/extensions/fm-primary-turnend-guard.ts | 4 ++ bin/fm-remote-job-lib.sh | 31 +++++++++- bin/fm-remote-job-worker.sh | 23 +++++++- tests/fm-operational-input.test.sh | 32 ++++++++++ tests/fm-pi-watch-extension.test.sh | 2 +- tests/fm-remote-job.test.sh | 59 +++++++++++++++++++ 8 files changed, 155 insertions(+), 4 deletions(-) diff --git a/.opencode/plugins/fm-primary-turnend-guard.js b/.opencode/plugins/fm-primary-turnend-guard.js index fb8f42eaf4..0d41e70325 100644 --- a/.opencode/plugins/fm-primary-turnend-guard.js +++ b/.opencode/plugins/fm-primary-turnend-guard.js @@ -22,6 +22,10 @@ function runProcess(command, args, input = "") { }); child.on("error", () => resolve({ code: 0, stdout: "", stderr: "" })); child.on("close", (code) => resolve({ code: code ?? 0, stdout, stderr })); + // A child that exits without reading its input breaks this pipe mid-write. + // Without a listener that EPIPE is an unhandled 'error' event and takes the + // whole host process down. The child's exit code and output still decide. + child.stdin.on("error", () => {}); child.stdin.end(input); }); } diff --git a/.opencode/plugins/lib/fm-operational-input.js b/.opencode/plugins/lib/fm-operational-input.js index f0d05c1444..7b72e6ddeb 100644 --- a/.opencode/plugins/lib/fm-operational-input.js +++ b/.opencode/plugins/lib/fm-operational-input.js @@ -25,6 +25,10 @@ export function encodeFirstmateOperationalInput(root, kind, content) { stderr += chunk.toString(); }); child.on("error", reject); + // An encoder that exits before reading its input breaks this pipe. Without + // a listener that EPIPE is an unhandled 'error' event and takes the whole + // host process down instead of rejecting this one call. + child.stdin.on("error", () => {}); child.on("close", (code) => { if (code === 0 && stdout) { resolveResult(stdout); diff --git a/.pi/extensions/fm-primary-turnend-guard.ts b/.pi/extensions/fm-primary-turnend-guard.ts index 1b2a3ec39a..54dff9d889 100644 --- a/.pi/extensions/fm-primary-turnend-guard.ts +++ b/.pi/extensions/fm-primary-turnend-guard.ts @@ -163,6 +163,10 @@ function runGuard(): Promise<{ code: number; stderr: string }> { }); child.on("error", () => resolveResult({ code: 0, stderr: "" })); child.on("close", (code) => resolveResult({ code: code ?? 0, stderr })); + // A guard that exits without reading its input breaks this pipe mid-write. + // Without a listener that EPIPE is an unhandled 'error' event and takes the + // whole host process down. The guard's exit code and stderr still decide. + child.stdin.on("error", () => {}); child.stdin.end('{"stop_hook_active":false}'); }); } diff --git a/bin/fm-remote-job-lib.sh b/bin/fm-remote-job-lib.sh index 31a6bf58cf..69e42a27e5 100755 --- a/bin/fm-remote-job-lib.sh +++ b/bin/fm-remote-job-lib.sh @@ -688,7 +688,11 @@ fm_remote_job_worker_ready_path() { printf '%s\n' "$FM_REMOTE_JOB_STATE/worker.r fm_remote_job_worker_identity_path() { printf '%s\n' "$FM_REMOTE_JOB_STATE/worker.identity"; } fm_remote_job_worker_lock_path() { printf '%s\n' "$FM_REMOTE_JOB_STATE/worker.lock"; } -fm_remote_job_process_start() { +# The wall-clock start stamp older records hold. procps derives it from a +# sampled boot time, so a loaded machine reports it a second or more apart for +# the same live process. Only read to recognize a record written before the +# stable stamp below, and on platforms with no /proc to read. +fm_remote_job_process_start_wall_clock() { # local pid=$1 ps_bin value if [ -x /bin/ps ]; then ps_bin=/bin/ps; elif [ -x /usr/bin/ps ]; then ps_bin=/usr/bin/ps; else return 1; fi value=$("$ps_bin" -p "$pid" -o lstart= 2>/dev/null) || return 1 @@ -697,6 +701,25 @@ fm_remote_job_process_start() { printf '%s\n' "$value" } +# The pid-reuse guard for a recorded owner, so reading it twice for the same +# live process has to give the same answer. The kernel's own start tick never +# moves; the drifting wall-clock form made a healthy lock owner read as a +# different process, which left ensure starting a second worker that could +# never take the lock from the first. +fm_remote_job_process_start() { # + local pid=$1 value + if [ -r "/proc/$pid/stat" ]; then + # Field 22 is starttime. comm (field 2) is parenthesized and can hold + # spaces, so drop everything through its closing paren before counting. + value=$(awk '{ sub(/^.*\) /, ""); print $20 }' "/proc/$pid/stat" 2>/dev/null || true) + case "$value" in + ''|*[!0-9]*) : ;; + *) printf 'boot-tick:%s\n' "$value"; return 0 ;; + esac + fi + fm_remote_job_process_start_wall_clock "$pid" +} + fm_remote_job_process_command() { local pid=$1 ps_bin value if [ -x /bin/ps ]; then ps_bin=/bin/ps; elif [ -x /usr/bin/ps ]; then ps_bin=/usr/bin/ps; else return 1; fi @@ -798,7 +821,11 @@ fm_remote_job_lock_owner_matches_process() { [ "$pid" -gt 1 ] || return 1 recorded_start=$(fm_remote_job_read_single_line "$lock/start" 256) || return 1 actual_start=$(fm_remote_job_process_start "$pid") || return 1 - [ "$recorded_start" = "$actual_start" ] || return 1 + if [ "$recorded_start" != "$actual_start" ]; then + # A lock published before the stable stamp landed holds the wall-clock form. + actual_start=$(fm_remote_job_process_start_wall_clock "$pid") || return 1 + [ "$recorded_start" = "$actual_start" ] || return 1 + fi recorded_command=$(fm_remote_job_read_single_line "$lock/command" 8192) || return 1 actual_command=$(fm_remote_job_process_command "$pid") || return 1 [ "$recorded_command" = "$actual_command" ] || return 1 diff --git a/bin/fm-remote-job-worker.sh b/bin/fm-remote-job-worker.sh index 2a49dd6694..342730105b 100755 --- a/bin/fm-remote-job-worker.sh +++ b/bin/fm-remote-job-worker.sh @@ -143,6 +143,21 @@ worker_recover_quarantine() { # rm -f -- "$WORKER_LOCK/quarantine" } +# Every lock field is published by mv from a mktemp staging file created inside +# the lock directory itself, so a worker killed between mktemp and mv leaves +# that staging file behind. Reclaiming a proven-stale lock has to remove those +# leftovers, otherwise the empty-directory rmdir below can never succeed and +# every replacement worker wedges on a lock no live process owns. +worker_remove_lock_staging_files() { + local file + for file in "$WORKER_LOCK"/.pid.* "$WORKER_LOCK"/.start.* "$WORKER_LOCK"/.command.* \ + "$WORKER_LOCK"/.quarantine.*; do + [ -e "$file" ] || [ -L "$file" ] || continue + [ ! -L "$file" ] && [ -f "$file" ] || return 1 + rm -f -- "$file" || return 1 + done +} + worker_acquire_lock() { local account_home=$1 attempt=0 while [ "$attempt" -lt 150 ]; do @@ -164,6 +179,7 @@ worker_acquire_lock() { fi [ ! -L "$WORKER_LOCK/pid" ] && [ ! -L "$WORKER_LOCK/start" ] && [ ! -L "$WORKER_LOCK/command" ] || return 1 rm -f -- "$WORKER_LOCK/pid" "$WORKER_LOCK/start" "$WORKER_LOCK/command" || return 1 + worker_remove_lock_staging_files || return 1 rmdir "$WORKER_LOCK" || return 1 done return 1 @@ -292,7 +308,12 @@ worker_stop_active_execution() { } worker_shutdown() { - trap - HUP INT TERM + # Ignore, rather than default, so a repeat stop signal cannot kill this + # process part way through publishing the quarantine marker. A group stop + # already delivers TERM here and again from the restart supervisor, and a + # default disposition would leave the marker's staging file inside the lock. + # A wedged shutdown is still stoppable: the stop path escalates to KILL. + trap '' HUP INT TERM worker_publish_quarantine || { worker_error "cannot guard worker ownership for shutdown" trap worker_shutdown HUP INT TERM diff --git a/tests/fm-operational-input.test.sh b/tests/fm-operational-input.test.sh index 2d1b1c39de..e2ad709ebd 100755 --- a/tests/fm-operational-input.test.sh +++ b/tests/fm-operational-input.test.sh @@ -140,6 +140,37 @@ JS pass "operational input: the OpenCode adapter constructs through the canonical owner" } +# An encoder that exits before reading its input breaks the adapter's stdin +# pipe. That EPIPE has to fail the one call, not the whole host process +# (regression: it arrived as an unhandled 'error' event and killed the host). +test_cross_language_adapter_survives_a_non_reading_encoder() { + local tmp result + tmp=$(fm_test_tmproot fm-operational-input-epipe) + mkdir -p "$tmp/bin" + cat > "$tmp/bin/fm-operational-input.sh" <<'SH' +#!/usr/bin/env bash +exit 1 +SH + chmod +x "$tmp/bin/fm-operational-input.sh" + result=$(HELPER="$ROOT/.opencode/plugins/lib/fm-operational-input.js" FIXTURE_ROOT="$tmp" \ + node --input-type=module <<'JS' 2>&1 +import { pathToFileURL } from "node:url"; +const { encodeFirstmateOperationalInput } = await import(pathToFileURL(process.env.HELPER).href); +// Larger than a pipe buffer, so the write cannot finish before the encoder exits. +const body = "x".repeat(1024 * 1024); +try { + await encodeFirstmateOperationalInput(process.env.FIXTURE_ROOT, "watcher", body); + process.stdout.write("resolved"); +} catch { + process.stdout.write("rejected"); +} +JS + ) || fail "a non-reading encoder brought down the adapter's host process: $result" + [ "$result" = rejected ] \ + || fail "the adapter did not report the failed encode: $result" + pass "operational input: a non-reading encoder fails one call, not the host process" +} + test_invalid_current_encodings_are_rejected() { local output output=$(printf 'body' | "$OWNER" encode legacy-operational 2>/dev/null) \ @@ -157,4 +188,5 @@ test_landed_untyped_prefix_is_explicitly_legacy test_isolated_legacy_matrix test_genuine_near_misses_remain_unclassified test_cross_language_adapter_uses_the_owner +test_cross_language_adapter_survives_a_non_reading_encoder test_invalid_current_encodings_are_rejected diff --git a/tests/fm-pi-watch-extension.test.sh b/tests/fm-pi-watch-extension.test.sh index c2b4aad058..a6ddfd2e3c 100755 --- a/tests/fm-pi-watch-extension.test.sh +++ b/tests/fm-pi-watch-extension.test.sh @@ -2246,7 +2246,7 @@ if (!promptBody.includes("TURN WOULD END BLIND")) { EOF ) status=$? - expect_code 0 "$status" "OpenCode watch plugin must not treat external healthy output as an owned arm" + expect_code 0 "$status" "OpenCode watch plugin must not treat external healthy output as an owned arm: $out" [ -z "$out" ] || fail "OpenCode external-healthy test printed output: $out" pass "OpenCode healthy arm output does not suppress the turn-end guard" } diff --git a/tests/fm-remote-job.test.sh b/tests/fm-remote-job.test.sh index 82f1cf8cce..5ff8a1b0e1 100755 --- a/tests/fm-remote-job.test.sh +++ b/tests/fm-remote-job.test.sh @@ -234,6 +234,41 @@ assert_present "$ACTIVE_SIDE_EFFECT" "the active job was interrupted by the conc fm_remote_job_reap "$ACCOUNT_HOME" "$JOB_ID" || fail "the active readiness job could not be reaped" pass "active jobs keep the worker ready for concurrent requests" +# The recorded start stamp is the pid-reuse guard for a lock owner, so reading +# it again for the same live process has to give the same answer. The +# wall-clock form procps reports is derived from a sampled boot time and moves +# by a second or more while the machine is busy, which made a healthy owner +# read as a different process: ensure then started a second worker beside the +# first instead of replacing it, and that second worker could never take the +# lock (regression: "remote job worker did not report ready after startup"). +OWNER_PID=$(cat "$STATE_ROOT/worker.lock/pid") +STAMP_LOAD_PIDS=() +for _ in 1 2; do + timeout 10 bash -c 'while :; do :; done' & + STAMP_LOAD_PIDS+=("$!") +done +STAMP_SEEN=$(for _ in $(seq 1 60); do fm_remote_job_process_start "$OWNER_PID"; done | sort -u | wc -l) +for LOAD_PID in "${STAMP_LOAD_PIDS[@]}"; do + kill "$LOAD_PID" 2>/dev/null || true + wait "$LOAD_PID" 2>/dev/null || true +done +[ "$STAMP_SEEN" -eq 1 ] \ + || fail "the owner start stamp reported $STAMP_SEEN values for one live process" +fm_remote_job_lock_owner_matches_process "$ACCOUNT_HOME" \ + || fail "a live lock owner stopped matching its own recorded identity" +pass "one live process reports one start stamp while the machine is busy" + +# A lock published before the stable stamp landed holds the wall-clock form. +# Its owner must still be recognized, or the upgrade itself wedges ownership. +printf '%s\n' "$(fm_remote_job_process_start_wall_clock "$OWNER_PID")" \ + > "$STATE_ROOT/worker.lock/start" +fm_remote_job_lock_owner_matches_process "$ACCOUNT_HOME" \ + || fail "a lock recorded in the older wall-clock form lost its live owner" +[ "$FM_REMOTE_JOB_OWNER_PID" = "$OWNER_PID" ] \ + || fail "the older wall-clock record resolved to the wrong owner pid" +printf '%s\n' "$(fm_remote_job_process_start "$OWNER_PID")" > "$STATE_ROOT/worker.lock/start" +pass "an ownership record from an older build still resolves its live owner" + OLD_WORKER_PID=$(cat "$STATE_ROOT/worker.pid") printf '\n' >> "$REMOTE_ROOT/bin/fm-remote-job-worker.sh" fm_remote_job_ensure_worker "$REMOTE_ROOT" "$ACCOUNT_HOME" \ @@ -285,6 +320,30 @@ wait "$OTHER_PID" 2>/dev/null || true OTHER_PID= pass "stale ownership is reclaimed without signaling a reused pid" +# Every lock field is published by mv from a staging file created inside the +# lock itself, so a worker killed between the two leaves that staging file +# behind. The lock is then owned by nobody and must still be reclaimable +# (regression: rmdir refused the non-empty directory, so every replacement +# worker wedged until its startup bound and fm-on reported no ready worker). +STAGING_WORKER_PID=$(cat "$STATE_ROOT/worker.pid") +fm_remote_job_stop_worker_tree "$STAGING_WORKER_PID" \ + || fail "the staging-leak fixture could not stop the running worker" +mkdir -p "$STATE_ROOT/worker.lock" +printf 'active execution could not be confirmed stopped\n' \ + > "$STATE_ROOT/worker.lock/.quarantine.aBcDeF" +rm -f "$STATE_ROOT/worker.ready" +touch -t 200001010000 "$STATE_ROOT/worker.lock" +fm_remote_job_ensure_worker "$REMOTE_ROOT" "$ACCOUNT_HOME" \ + || fail "$FM_REMOTE_JOB_ERROR" +assert_absent "$STATE_ROOT/worker.lock/.quarantine.aBcDeF" \ + "lock reclaim kept the killed worker's staging file" +fm_remote_job_probe "$ACCOUNT_HOME" \ + || fail "the replacement worker never published a readiness heartbeat" +fm_remote_job_worker_identity_matches "$REMOTE_ROOT" "$ACCOUNT_HOME" \ + || fail "staging-leak recovery did not start the current worker" +NEW_WORKER_PID=$(cat "$STATE_ROOT/worker.pid") +pass "a lock holding only a killed worker's staging file is reclaimed" + FM_REMOTE_JOB_TIMEOUT=1 fm_remote_job_stage "$ACCOUNT_HOME" "$REMOTE_ROOT" "$REMOTE_HOME" fm-timeout-job.sh < /dev/null > /dev/null JOB_ID=$FM_REMOTE_JOB_ID