From 5aaa759d4db3748f10515b26b14034b667ddabc5 Mon Sep 17 00:00:00 2001 From: Laszlo Horanszky <40281577+Dematom1@users.noreply.github.com> Date: Thu, 29 Oct 2026 20:18:06 -0600 Subject: [PATCH 01/56] feat(bin): add shared_quota opt-out from quota-based dispatch ranking (#7894) * feat(dispatch): add shared_quota home opt-out from quota-based ranking A home whose Claude and other provider usage draws on one combined AI budget has no meaningful per-provider quota rows, so spendPriority ranking and runway gates pick the wrong candidate. An optional top-level "shared_quota": true in config/crew-dispatch.json now opts the home out: the typed resolver makes no quota-axi call, applies no quota floor or runway gate, ranks nothing by spendPriority, and resolves the matched rule's use profiles in declared order, printing every candidate line with quota ranking off by home declaration. Firstmate's quota-array-dispatch intake honors the same declaration. Without the setting, behavior is unchanged. Non-boolean values are actionable configuration errors in both the resolver and bootstrap. * test(dispatch): cover shared-quota bypass of unmet rule and profile floors The shared_quota opt-out test picked a rule with no quota floor, so the separate rule-floor and profile-floor bypass paths stayed untested. Two cases now run against the same fixture: a rule whose model:fable floor (20%) exceeds the snapshot's 15%, and a profile whose all_models floor (50%) exceeds codex's 31%, each asserting the declared-order pick still wins with no quota-axi invocation. * no-mistakes(review): Narrow shared-quota skill opt-out to quota-only gates * no-mistakes(review): Document preferred_provider under shared_quota opt-out and test it * no-mistakes(document): Document shared_quota in schema example and verification inventory --- .agents/skills/quota-array-dispatch/SKILL.md | 9 ++ AGENTS.md | 1 + bin/fm-bootstrap.sh | 1 + bin/fm-dispatch-resolve.sh | 43 +++++++--- docs/configuration.md | 12 +++ docs/verification/dispatch-resolve.md | 4 +- tests/fm-bootstrap.test.sh | 3 + tests/fm-dispatch-resolve.test.sh | 86 ++++++++++++++++++++ 8 files changed, 147 insertions(+), 12 deletions(-) diff --git a/.agents/skills/quota-array-dispatch/SKILL.md b/.agents/skills/quota-array-dispatch/SKILL.md index edbe950428c..32e6c166f42 100644 --- a/.agents/skills/quota-array-dispatch/SKILL.md +++ b/.agents/skills/quota-array-dispatch/SKILL.md @@ -16,6 +16,14 @@ This skill is the single owner of the completion-aware profile-array selection p Do not add a daemon, opaque composite score, routing wrapper, hard-coded model-specific policy, or producer-side route recommendation. The [worker helper](../../../bin/fm-quota-choose.sh) and [typed resolver](../../../docs/configuration.md#typed-dispatch-resolution-env-fm_dispatch_provider-fm_dispatch_model) own their deterministic mapping boundaries. +## Shared-quota opt-out + +When the home's `config/crew-dispatch.json` declares a top-level `"shared_quota": true`, this home draws every provider's usage from one single shared budget, so per-provider quota rows do not describe what is left, and this skill's quota snapshot, the quota parts of the eligibility and runway gates, and `spendPriority` ranking are off by home declaration. +Make no `quota-axi` call, apply no quota row, quota floor, runway gate, or `spendPriority` rank, and choose the matched rule's first declared profile that its authoritative catalog supports and that meets the required reasoning class; catalog eligibility and reasoning-class fit still apply. +`preferred_provider` is not a quota signal, so it still applies: `preferred_provider` still narrows the pool; declared order then picks within it. +Account for every candidate as always, stating in place of quota evidence that quota ranking is off by home declaration. +The declared value must be a boolean; anything else is an actionable configuration error. + ## Worker-side quota helper The canonical shell helper for a worker that has already performed its model-selection reasoning and now needs to pick the first viable candidate is `bin/fm-quota-choose.sh`. @@ -35,6 +43,7 @@ It never removes this skill's authority, and its `ambiguous`, `escalate`, and `e ## Read the default TOON +When the home declares the [shared-quota opt-out](#shared-quota-opt-out), skip this section's quota reading and every quota-derived check built on it (quota rows, floors, runway, and `spendPriority`); catalog eligibility and reasoning-class fit still apply. Start each intake by running `quota-axi` once with no `--json`, and reuse that TOON for every candidate. Post-consolidation quota-axi (the floor owned by `bin/fm-quota-axi-lib.sh`) puts `spendPriority` in the default `quota[]` block beside `effectivePercentRemaining`, `runway`, `confidence`, `limitedBy`, and `resetsAt`. Sparse `exhaustion[]` carries finite-runway seconds only for `projected_exhaustion` and `exhausted_now`. diff --git a/AGENTS.md b/AGENTS.md index 1d4974d3e62..f208f4c29bb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -99,6 +99,7 @@ Routing precedence is an explicit per-task captain override, then the best-fit c Firstmate alone resolves a matched profile array: begin with `quota-axi`'s default TOON at that intake, using the skill's narrow TOON-then-`--json` fallback only for genuine ambiguity, evaluate every configured candidate against that current output, and choose under the skill's `preferred_provider` procedure with inspectable `spendPriority` as the one quota-perspective ranker after its eligibility, reasoning-class, and runway-feasibility gates. Account for every candidate with the catalog evidence, provider relationship, applicable quota and authentication facts, remaining uncertainty, fit and reasoning class, and the spendPriority and runway evidence used in selection; never omit a candidate, guess, fall back silently, or call the result quota-informed without them. Establish model support and provider family from that harness's own authoritative catalog, then apply the [account and scope matching rules in `quota-array-dispatch`](.agents/skills/quota-array-dispatch/SKILL.md#1-eligibility). +When the home's `config/crew-dispatch.json` declares `"shared_quota": true`, make no quota-axi call and apply no quota gate or ranking: choose by the matched rule's declared profile order and reasoning class (`preferred_provider` still narrows the pool; declared order then picks within it), accounting for every candidate with quota ranking off by home declaration (schema: `docs/configuration.md` "Crew dispatch profiles"). Missing model-level quota, a missing authentication source, unmeasurable headroom, or unmodeled authentication is disclosed uncertainty that keeps a candidate eligible, never a credential or login escalation. Only concrete contradictory evidence blocks a candidate, such as an authoritative catalog proving the model unsupported or proof that the credential selected for that surface is unusable; never infer a credential store, provider family, or quota mapping from a harness, model, or source name, and never launch another harness's CLI to judge a candidate. Preserve malformed profile configuration as an actionable error rather than selecting around it. diff --git a/bin/fm-bootstrap.sh b/bin/fm-bootstrap.sh index ecda61144eb..f650ea033e2 100755 --- a/bin/fm-bootstrap.sh +++ b/bin/fm-bootstrap.sh @@ -1237,6 +1237,7 @@ crew_dispatch_validate() { elif [(.rules // [])[]? | select(has("select") and ((.select? | type) != "string" or (.select | length) == 0))] | length > 0 then "select must be a non-empty string" elif [(.rules // [])[]? | .select? // empty | select(. != "quota-balanced")] | length > 0 then "unknown select: " + ([ (.rules // [])[]? | .select? // empty | select(. != "quota-balanced") ] | unique | join(", ")) + elif has("shared_quota") and (.shared_quota | type) != "boolean" then "shared_quota must be true or false when present" elif has("default") and ((.default | type) != "object" and (.default | type) != "array") then "default must be a profile object or non-empty profile array" elif has("default") and ((.default | type) == "array" and (.default | length) == 0) then "default needs at least one profile" elif has("default") and ([profiles(.default)[]? | select(type != "object")] | length) > 0 then "each default profile must be an object" diff --git a/bin/fm-dispatch-resolve.sh b/bin/fm-dispatch-resolve.sh index 49137ff06a6..37e95c9fa81 100755 --- a/bin/fm-dispatch-resolve.sh +++ b/bin/fm-dispatch-resolve.sh @@ -38,6 +38,17 @@ # The model never sees quota, catalogs, approvals, preferred_provider, # confidence floors, `why`, or `use`. With no rules, it returns a non-clear # result so firstmate keeps using the existing intake. +# +# Shared-quota opt-out: an optional top-level "shared_quota": true in the +# rules file declares this home draws every provider's usage from one single +# shared budget, so per-provider quota rows do not describe what is left. +# With it set, this tool makes no quota-axi call, applies no quota floor or +# runway gate, ranks nothing by spendPriority, and resolves the matched +# rule's `use` profiles in declared order; preferred_provider still narrows +# the pool; declared order then picks within it. Every candidate line still +# prints, stating quota ranking is off by home declaration. Without the +# setting, behavior is unchanged. The declared value must be a boolean; +# anything else is an actionable configuration error. # docs/configuration.md "Crew dispatch profiles" owns the declared fields and # "Typed dispatch resolution" owns this tool's operator contract. # @@ -60,6 +71,7 @@ # reason: # preferred_provider: # candidate: : provider=.. scope=.. remaining=..% spendPriority=.. runway=.. -> eligible | eligible, unranked: | not eligible: +# (with the shared-quota opt-out: : provider=.. -> eligible (quota ranking off: home declares a single shared quota)) # profile: --harness [--model ] [--effort ] (status clear only) # clear -> pass the profile line to fm-spawn.sh unless you state a reason to override # ambiguous -> confidence below the floor; decide as today from the probabilities @@ -265,6 +277,7 @@ rules_err=$(jq -r --argjson verified_harnesses "$VERIFIED_HARNESSES" --arg provi elif any((.rules // [])[]; has("select") and ((.select | type) != "string" or (.select | length) == 0)) then "select must be a non-empty string" elif any((.rules // [])[]; has("select") and .select != "quota-balanced") then "unknown select: " + ([.rules[] | select(has("select") and .select != "quota-balanced") | .select] | unique | join(", ")) + elif has("shared_quota") and (.shared_quota | type) != "boolean" then "shared_quota must be true or false when present" elif any((.rules // [])[]; has("floor") and floor_bad(.floor; true)) then "rule floor needs scope, min_percent 0..100, and provider matching ^[a-z0-9]+(-[a-z0-9]+)*\\z" elif any((.rules // [])[] | profiles(.use)[]; profile_bad(.)) then "each use profile needs harness; model, effort, and floor must be well formed, and provider must match ^[a-z0-9]+(-[a-z0-9]+)*\\z when present" elif any((.rules // [])[]; duplicate_profiles(profiles(.use))) then "each rule use must not contain duplicate harness, model, and effort profiles" @@ -481,13 +494,21 @@ jq -e --slurpfile rules "$RULES" ' (.usage.output_tokens | type) == "number"))' \ "$RESP_FILE" >/dev/null 2>&1 || emit_error "response is not a rule Choice answer" -# ---- quota evidence: one quota-axi --json snapshot ----------------------------- -command -v quota-axi >/dev/null 2>&1 || emit_error "quota-axi not installed" -quota-axi --json > "$QUOTA" 2>/dev/null || emit_error "quota-axi --json failed" -fm_quota_json_valid < "$QUOTA" || emit_error "quota-axi --json returned an invalid snapshot" +# ---- quota evidence: one quota-axi --json snapshot, skipped entirely under the +# ---- shared-quota opt-out (a stub keeps the slurpfile contract intact) --------- +SQ=false +SHARED_QUOTA=$(jq -r '.shared_quota // false' "$RULES" 2>/dev/null) || SHARED_QUOTA=false +if [ "$SHARED_QUOTA" = true ]; then + printf '{"providers":[]}\n' > "$QUOTA" + SQ=true +else + command -v quota-axi >/dev/null 2>&1 || emit_error "quota-axi not installed" + quota-axi --json > "$QUOTA" 2>/dev/null || emit_error "quota-axi --json failed" + fm_quota_json_valid < "$QUOTA" || emit_error "quota-axi --json returned an invalid snapshot" +fi # ---- resolution: declared gates + quota evidence + argmax, all in jq ------------ -RESULT=$(jq -n --arg floor "$CONFIDENCE_FLOOR" --argjson lat "$LAT_MS" --arg none_criterion "$DEFAULT_WHEN" --argjson pmap "$PMAP" \ +RESULT=$(jq -n --arg floor "$CONFIDENCE_FLOOR" --argjson lat "$LAT_MS" --arg none_criterion "$DEFAULT_WHEN" --argjson pmap "$PMAP" --argjson sq "$SQ" \ --slurpfile resp "$RESP_FILE" --slurpfile rules "$RULES" --slurpfile quota "$QUOTA" "$FM_QUOTA_ROW_JQ"' ($resp[0]) as $r | ($rules[0]) as $cfg | ($quota[0]) as $q | ($r.answers.rule) as $a | def profiles($v): if ($v | type) == "array" then $v elif ($v | type) == "object" then [$v] else [] end; @@ -517,7 +538,9 @@ RESULT=$(jq -n --arg floor "$CONFIDENCE_FLOOR" --argjson lat "$LAT_MS" --arg non $rows | map({scope, status, pct: (.effectivePercentRemaining // null), runway: (.runway.status // null), spendPriority: (.selection.spendPriority // null)}); def evaluate($c): (provider_of($c)) as $p | (lane_of($c)) as $lane | - if $p == null then {profile: $c, eligible: false, reason: "no provider family for harness \($c.harness); declare provider on the profile"} + if $sq then + {profile: $c, provider: $p, eligible: true, sq: true, reason: "ok"} + elif $p == null then {profile: $c, eligible: false, reason: "no provider family for harness \($c.harness); declare provider on the profile"} elif prov($p; $lane) == null then {profile: $c, provider: $p, eligible: true, unranked: true, reason: (if any($q.providers[]; .provider == $p) @@ -585,7 +608,7 @@ RESULT=$(jq -n --arg floor "$CONFIDENCE_FLOOR" --argjson lat "$LAT_MS" --arg non end) as $fb | (if $fb.to then $fb.to else $picked end) as $choice | (rule_at($choice)) as $rule | - (if $rule == null then "none" else floor_state($rule.floor; $rule.floor.provider; "") end) as $rule_floor_state | + (if $sq or $rule == null then "none" else floor_state($rule.floor; $rule.floor.provider; "") end) as $rule_floor_state | (if $choice != "default" and $rule == null then [] elif $rule == null then profiles($cfg.default // null) else profiles($rule.use) @@ -631,8 +654,8 @@ RESULT=$(jq -n --arg floor "$CONFIDENCE_FLOOR" --argjson lat "$LAT_MS" --arg non (if ($preferred | length) > 0 then " in preferred provider \($cfg.preferred_provider); uncertainty does not authorize fallback" else "" end)), note: $sel.note, candidates: $cands} else - ($elig | max_by(.spendPriority)) as $best | - ([$elig[] | select(.spendPriority == $best.spendPriority)] | length) as $ties | + (if $sq then ($elig | first) else ($elig | max_by(.spendPriority)) end) as $best | + (if $sq then 1 else ([$elig[] | select(.spendPriority == $best.spendPriority)] | length) end) as $ties | if $ties > 1 then $ev + {status: "escalate", reason: "genuine spendPriority tie", note: $sel.note, candidates: $cands} else $ev + {status: "clear", note: $sel.note, candidates: $cands, chosen: $best} + (if ($unranked | length) > 0 then @@ -660,7 +683,7 @@ TEXT=$(jq -r ' + (if .provider then " provider=\(.provider | flat)" else "" end) + (if .scope then " scope=\(.scope | flat) remaining=\(show(.pct))% spendPriority=\(show(.spendPriority)) runway=\(show(.runway))" else "" end) + (if (.bounds // [] | length) > 1 then " bounds=" + ([.bounds[] | "\(.scope | flat):\(show(.pct))%/\((.runway // .status) | flat)"] | join(",")) else "" end) - + " -> " + (if .unranked then "eligible, unranked: \(.reason | flat): disclosed uncertainty" elif .eligible then "eligible" else "not eligible: \(.reason | flat)" end)), + + " -> " + (if .unranked then "eligible, unranked: \(.reason | flat): disclosed uncertainty" elif .sq then "eligible (quota ranking off: home declares a single shared quota)" elif .eligible then "eligible" else "not eligible: \(.reason | flat)" end)), (if .chosen then " profile: --harness \(.chosen.profile.harness | shell_arg)" + (if .chosen.profile.model then " --model \(.chosen.profile.model | shell_arg)" else "" end) + (if .chosen.profile.effort then " --effort \(.chosen.profile.effort | shell_arg)" else "" end) else empty end)' <<<"$RESULT") || emit_error "output rendering failed" diff --git a/docs/configuration.md b/docs/configuration.md index b275fc45afd..aa3d61c142b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1721,6 +1721,7 @@ This section is the single owner of the canonical schema and its per-field seman ```json { "preferred_provider": "", + "shared_quota": false, "rules": [ { "when": "", @@ -1749,6 +1750,7 @@ This section is the single owner of the canonical schema and its per-field seman | Profile `harness` | Required in every profile. | | Profile `model` and `effort`; rule `why` | Optional. | | Top-level `preferred_provider` | Optional provider-family preference, applied after eligibility checks under `quota-array-dispatch`. | +| Top-level `shared_quota` | Optional; see "Shared-quota opt-out" below. | **Preferred provider** @@ -1759,6 +1761,15 @@ Bootstrap validates this field even without a resolver key because the agent-own An absent field preserves the existing routing behavior. The [profile-array procedure](../.agents/skills/quota-array-dispatch/SKILL.md#rank-by-spendpriority) owns family preference, uncertainty, and fallback after the existing gates. +**Shared-quota opt-out** + +An optional top-level `"shared_quota": true` declares this home draws every provider's usage from one single shared budget, such as one workplace AI spend line, so per-provider quota rows do not describe what is left. +With it set, typed dispatch resolution makes no `quota-axi` call, applies no quota floor or runway gate, ranks nothing by `spendPriority`, and resolves the matched rule's `use` profiles in declared order; every candidate line still prints, stating quota ranking is off by home declaration. +`preferred_provider` is a separate home preference, not a quota signal, so it still applies: `preferred_provider` still narrows the pool; declared order then picks within it. +Firstmate's own `quota-array-dispatch` intake honors the same declaration. +Without the setting, or with `"shared_quota": false`, behavior is unchanged. +The declared value must be a boolean; anything else is an actionable configuration error reported like any other malformed rule. + **Fields applied only by typed resolution** Rule `approval`, `min_confidence`, and `floor`, and profile `provider` and `floor` are optional declarations that only [typed dispatch resolution](#typed-dispatch-resolution-env-fm_dispatch_provider-fm_dispatch_model) applies in code; without that opt-in they are inert, and firstmate's own intake reads them as ordinary hints. @@ -1894,6 +1905,7 @@ After the answer, code applies all remaining checks and ranking: - Every applicable account-wide and model/product row from one `quota-axi --json` snapshot. - The optional `preferred_provider` restriction under the [profile-array procedure](../.agents/skills/quota-array-dispatch/SKILL.md#rank-by-spendpriority). - The numeric `spendPriority` argmax within that candidate pool, using each candidate's limiting row. +- With the home's `shared_quota: true` declaration ("Crew dispatch profiles"), the quota snapshot, floors, runway gates, and `spendPriority` argmax are skipped entirely; resolution is the matched rule's declared profile order (`preferred_provider` still narrows the pool; declared order then picks within it). The resolver prints its preferred-family decision while retaining evidence for every candidate. If preferred candidates remain eligible but none is rankable, it escalates instead of selecting another family. diff --git a/docs/verification/dispatch-resolve.md b/docs/verification/dispatch-resolve.md index 6e19798c661..45ab92dcf11 100644 --- a/docs/verification/dispatch-resolve.md +++ b/docs/verification/dispatch-resolve.md @@ -306,8 +306,8 @@ It proves that with no provider set, other providers' settings leave the origina It proves `systemone` posts the unchanged System One request to `/v1/systemone`, sends no `Authorization` header and opens no descriptor without a key, carries a key only on the descriptor, reads its settings from `.env`, exits 2 for a missing model or a missing or scheme-less base URL, withholds a never-send base URL, and returns partial responses and transport failure as `error`. It proves `openai` is off with one line without its key, exits 2 without a model, sends the same state as JSON text `input` with one named choice question carrying every option, maps the probability array back, withholds a never-send match inside the serialized input or the model name, and rejects refusals, missing or duplicated answers and option values, missing options or confidence, and non-JSON bodies before quota is read. It proves a declared `min_confidence` is checked against the rule's own probability both as the pick and as a runner-up, a picked rule below it falls to the most probable runner-up that clears its floor, is `ambiguous` when none does or two tie, and that a file without declared floors keeps the global 0.6 floor on confidence unchanged. -It proves the clear, fixed-floor ambiguous with candidate evidence, escalate (approval with candidate evidence, unverifiable rule floor, tie, nothing rankable), known rule-floor fall-through, known and unverifiable profile-floor evidence, explicit-provider and provider-ID enforcement, authoritative Agy and explicit-provider Gemini routing, partial providers, eligible unranked candidates and their clear-result note, concrete quota vetoes and profile-floor shortfalls taking precedence over uncertainty, account-wide quota veto, limiting-bound ranking, schema-6 account-row binding with schema-5 compatibility, `preferred_provider` family preference across accounts with fallback only on proven ineligibility and escalation on unknown preferred quota, missing-curl and quota-axi failures, HTTP 429 and 500, transport failure, malformed usage, zero-mass or malformed probabilities or confidence, malformed or duplicate profile, invalid selector, removed-option rejection, and out-of-range rule ID paths behave as the contract states, with configuration errors exiting 2 before any network call. -`tests/fm-bootstrap.test.sh` covers bootstrap's typesafe-key validation gate, its key-independent `preferred_provider` check, and child-environment protection; [configuration](../configuration.md#crew-dispatch-profiles-configcrew-dispatchjson) owns that gate's operator contract. +It proves the clear, fixed-floor ambiguous with candidate evidence, escalate (approval with candidate evidence, unverifiable rule floor, tie, nothing rankable), known rule-floor fall-through, known and unverifiable profile-floor evidence, explicit-provider and provider-ID enforcement, authoritative Agy and explicit-provider Gemini routing, partial providers, eligible unranked candidates and their clear-result note, concrete quota vetoes and profile-floor shortfalls taking precedence over uncertainty, account-wide quota veto, limiting-bound ranking, schema-6 account-row binding with schema-5 compatibility, `preferred_provider` family preference across accounts with fallback only on proven ineligibility and escalation on unknown preferred quota, the `shared_quota` opt-out (declared-order pick with no `quota-axi` call, unmet rule- and profile-floor bypass, `preferred_provider` narrowing, the quota-ranking-off candidate line, `false` keeping the quota-ranked path, and non-boolean rejection), missing-curl and quota-axi failures, HTTP 429 and 500, transport failure, malformed usage, zero-mass or malformed probabilities or confidence, malformed or duplicate profile, invalid selector, removed-option rejection, and out-of-range rule ID paths behave as the contract states, with configuration errors exiting 2 before any network call. +`tests/fm-bootstrap.test.sh` covers bootstrap's typesafe-key validation gate, its key-independent `preferred_provider` and `shared_quota` checks, and child-environment protection; [configuration](../configuration.md#crew-dispatch-profiles-configcrew-dispatchjson) owns that gate's operator contract. ```console $ bash tests/fm-dispatch-resolve.test.sh | tail -1 diff --git a/tests/fm-bootstrap.test.sh b/tests/fm-bootstrap.test.sh index 198089df578..3fc5cee118a 100755 --- a/tests/fm-bootstrap.test.sh +++ b/tests/fm-bootstrap.test.sh @@ -1410,6 +1410,9 @@ unsupported kimi effort is flagged^{"rules":[{"when":"kimi work","use":{"harness cursor model profile is accepted^{"rules":[{"when":"cursor work","use":{"harness":"cursor","model":"cursor-grok-4.5-high"}}]}^empty^ unsupported cursor effort is flagged^{"rules":[{"when":"cursor work","use":{"harness":"cursor","model":"cursor-grok-4.5-high","effort":"high"}}]}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - invalid effort: cursor:high array use with quota-balanced is accepted^{"rules":[{"when":"big feature","use":[{"harness":"claude","model":"claude-sonnet-5","effort":"high"},{"harness":"codex","model":"gpt-5.5","effort":"high"}],"select":"quota-balanced"}]}^empty^ +top-level shared_quota true is accepted^{"shared_quota":true,"rules":[{"when":"big feature","use":{"harness":"claude"}}]}^empty^ +top-level shared_quota false is accepted^{"shared_quota":false,"rules":[{"when":"big feature","use":{"harness":"claude"}}]}^empty^ +non-boolean shared_quota is flagged^{"shared_quota":"yes","rules":[{"when":"big feature","use":{"harness":"claude"}}]}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - shared_quota must be true or false when present array use without select is accepted^{"rules":[{"when":"big feature","use":[{"harness":"claude"},{"harness":"codex"}]}]}^empty^ one-element array use is accepted^{"rules":[{"when":"focused feature","use":[{"harness":"claude"}]}]}^empty^ default array is accepted^{"default":[{"harness":"pi","model":"anthropic/claude-sonnet-5"},{"harness":"grok"}]}^empty^ diff --git a/tests/fm-dispatch-resolve.test.sh b/tests/fm-dispatch-resolve.test.sh index 8e0376e3ec4..bb054db558f 100755 --- a/tests/fm-dispatch-resolve.test.sh +++ b/tests/fm-dispatch-resolve.test.sh @@ -1340,6 +1340,8 @@ assert_contains "$err" 'not JSON' "non-JSON rules is named" for bad in \ '{"rules":[{"when":"x","use":{"harness":"claude"},"approval":"firstmate"}]}|approval must be "captain" when present' \ '{"rules":[{"when":"x","use":{"harness":"claude"},"select":"mystery"}]}|unknown select: mystery' \ + '{"shared_quota":"yes","rules":[{"when":"x","use":{"harness":"claude"}}]}|shared_quota must be true or false when present' \ + '{"shared_quota":1,"rules":[{"when":"x","use":{"harness":"claude"}}]}|shared_quota must be true or false when present' \ '{"rules":[{"when":"x","use":{"harness":"claude"},"min_confidence":"high"}]}|min_confidence must be a number from 0 through 1 when present' \ '{"rules":[{"when":"x","use":{"harness":"claude"},"min_confidence":1.5}]}|min_confidence must be a number from 0 through 1 when present' \ '{"rules":[{"when":"x","use":{"harness":"claude"},"floor":{"scope":"model:fable","min_percent":20}}]}|rule floor needs scope, min_percent 0..100, and provider matching ^[a-z0-9]+(-[a-z0-9]+)*\z' \ @@ -1379,4 +1381,88 @@ expect_code 0 "$code" "--help exits 0" assert_contains "$out" 'Usage:' "--help prints usage" pass "configuration errors exit 2 before any network call" +# --- shared-quota opt-out: no quota call, declared-order pick ---------------- +cp "$BASE_RULES" "$RULES" +jq '.shared_quota = true' "$BASE_RULES" > "$RULES" +reset_log +write_response "$RESPONSE" rule_4 0.9 +TYPESAFE_API_KEY=$KEY run code out err "$BRIEF" --project pager +expect_code 0 "$code" "shared-quota clear exits 0" +assert_contains "$out" ' status: clear' "shared-quota clear status" +assert_contains "$out" " profile: --harness 'claude' --model 'sonnet' --effort 'high'" "shared-quota picks the first declared profile, not the argmax winner" +assert_not_contains "$out" 'spendPriority=' "shared-quota prints no quota evidence" +assert_not_contains "$out" ' remaining=' "shared-quota prints no remaining percentages" +assert_not_contains "$out" ' runway=' "shared-quota prints no runway gates" +assert_contains "$out" 'quota ranking off: home declares a single shared quota' "every candidate line states quota ranking is off" +assert_absent "$LOG/quota-axi.calls" "shared-quota never reads quota-axi" +[ "$(printf '%s\n' "$out" | grep -c 'quota ranking off')" -ge 3 ] || fail "shared-quota states the opt-out on every candidate line" +pass "shared-quota opt-out: no quota snapshot, deterministic declared-order pick, opt-out stated on every candidate" + +# --- shared-quota: an unmet rule-level floor is bypassed too ----------------- +# rule_1 declares floor model:fable >= 20% but the fixture has 15% there, so the +# quota-ranked path would fall through to default; the opt-out keeps the rule. +jq '.shared_quota = true' "$BASE_RULES" > "$RULES" +reset_log +write_response "$RESPONSE" rule_1 0.9 +TYPESAFE_API_KEY=$KEY run code out err "$BRIEF" --project pager +expect_code 0 "$code" "shared-quota unmet rule floor exits 0" +assert_contains "$out" ' status: clear' "shared-quota unmet rule floor status" +assert_contains "$out" ' note: rule matched' "shared-quota skips the unmet rule floor instead of falling through to default" +assert_not_contains "$out" 'fall through to default' "shared-quota unmet rule floor never falls through to default" +assert_contains "$out" " profile: --harness 'claude' --model 'fable' --effort 'xhigh'" "shared-quota unmet rule floor picks the rule's own declared profile" +assert_contains "$out" 'quota ranking off: home declares a single shared quota' "unmet rule floor candidate states quota ranking is off" +assert_absent "$LOG/quota-axi.calls" "shared-quota unmet rule floor never reads quota-axi" +pass "shared-quota bypasses an unmet rule-level floor: the rule's own profile wins with no quota call" + +# --- shared-quota: an unmet profile-level floor is bypassed too --------------- +# The codex profile declares floor all_models >= 50% but the fixture has 31%, so +# the quota-ranked path marks it not eligible; the opt-out resolves the declared +# order anyway. The use array is reversed so the floored profile is first. +jq '.shared_quota = true | .rules[1].use |= reverse' "$BASE_RULES" > "$RULES" +reset_log +write_response "$RESPONSE" rule_2 0.99 +TYPESAFE_API_KEY=$KEY run code out err "$BRIEF" --project pager +expect_code 0 "$code" "shared-quota unmet profile floor exits 0" +assert_contains "$out" ' status: clear' "shared-quota unmet profile floor status" +assert_contains "$out" " profile: --harness 'codex' --model 'gpt-5.6-sol'" "shared-quota unmet profile floor picks the floored profile first in declared order" +assert_not_contains "$out" 'not eligible' "shared-quota unmet profile floor marks no candidate ineligible" +assert_contains "$out" 'quota ranking off: home declares a single shared quota' "unmet profile floor candidates state quota ranking is off" +assert_absent "$LOG/quota-axi.calls" "shared-quota unmet profile floor never reads quota-axi" +pass "shared-quota bypasses an unmet profile-level floor: the first declared profile wins with no quota call" + +# --- shared-quota: preferred_provider still narrows the pool ------------------ +# The matched rule declares claude first, then two codex-family profiles; the +# home prefers codex, so the first declared codex profile wins. +jq '.shared_quota = true | .preferred_provider = "codex" | .rules[3].use = [ + {harness:"claude",model:"sonnet",effort:"high"}, + {harness:"pi",model:"openai-codex/gpt-5.6-sol",provider:"codex"}, + {harness:"codex",model:"gpt-5.6-sol"}]' "$BASE_RULES" > "$RULES" +reset_log +write_response "$RESPONSE" rule_4 0.9 +TYPESAFE_API_KEY=$KEY run code out err "$BRIEF" --project pager +expect_code 0 "$code" "shared-quota preferred_provider exits 0" +assert_contains "$out" ' status: clear' "shared-quota preferred_provider status" +assert_contains "$out" ' preferred_provider: codex: prefer eligible family candidates' "shared-quota still applies the family preference" +assert_contains "$out" " profile: --harness 'pi' --model 'openai-codex/gpt-5.6-sol'" "shared-quota preferred_provider picks the first declared preferred-family profile" +assert_absent "$LOG/quota-axi.calls" "shared-quota preferred_provider never reads quota-axi" +[ "$(printf '%s\n' "$out" | grep -c '^ candidate: ')" -eq 3 ] || fail "shared-quota preferred_provider lists every candidate" +[ "$(printf '%s\n' "$out" | grep '^ candidate: ' | grep -c 'quota ranking off: home declares a single shared quota')" -eq 3 ] || + fail "shared-quota preferred_provider states the opt-out on every candidate line" +pass "shared-quota keeps preferred_provider narrowing: the preferred family wins in declared order with no quota call" + +# restore the shared response fixture for the false-path test below +write_response "$RESPONSE" rule_4 0.9 + +# --- shared_quota false: byte-for-byte the quota-ranked path ------------------ +jq '.shared_quota = false' "$BASE_RULES" > "$RULES" +reset_log +TYPESAFE_API_KEY=$KEY run code out err "$BRIEF" --project pager +expect_code 0 "$code" "shared_quota false exits 0" +assert_contains "$out" " profile: --harness 'cursor' --model 'cursor-grok-4.6-medium'" "shared_quota false keeps the spendPriority argmax" +assert_not_contains "$out" 'quota ranking off' "shared_quota false prints no opt-out wording" +[ -f "$LOG/quota-axi.calls" ] || fail "shared_quota false still reads quota-axi" +assert_contains "$out" 'spendPriority=' "shared_quota false prints quota evidence" +pass "shared_quota false keeps the unchanged quota-ranked behavior" +cp "$BASE_RULES" "$RULES" + printf '# all fm-dispatch-resolve tests passed\n' From 3b82fd89ee88b384fe443e78aeab7e620455cc22 Mon Sep 17 00:00:00 2001 From: Ian Brown <742554+zestysoft@users.noreply.github.com> Date: Thu, 29 Oct 2026 19:19:23 -0700 Subject: [PATCH 02/56] fix(bin): distinguish recycled reaper PIDs in tombstone cleanup (#7958) * fix(bin): distinguish recycled reaper PIDs in tombstone cleanup * no-mistakes(ci): ci-2 fixed in tests/fm-watcher-lock.test.sh (test_lock_tombstone_tracks_reaper_start_time). Invariant: no fixture child may outlive a SIGKILLed runner without a bound. (1) The paused reaper's wait loop now also stops when FM_TEST_SUITE_PID is no longer alive or when SECONDS reaches FM_TEST_STUB_MAX_BLOCK_SECONDS. (2) The one sibling child in the same test, the unrelated `sleep 300`, now sleeps for FM_TEST_STUB_MAX_BLOCK_SECONDS so it uses the same bound. Verification, run serially: bash tests/fm-watcher-lock.test.sh. The new test passed on 4 of 4 runs. Red-before still holds: with the base bin/fm-wake-lib.sh from c9da5523, the test fails with 'not ok - recycled reaper pid kept an abandoned tombstone alive'. Separately, the known stand-down flake ('the replaced watcher did not stand down on its own', watch-legacy-local-time-identity) failed on every run on this host and also fails on base c9da5523 without this change, so the test file still exits 1; left unfixed as instructed. ci-1 (Full suite waiting on the full-ci maintainer label) was ignored per the user's decision --- bin/fm-wake-lib.sh | 25 +++++++++++-- tests/fm-watcher-lock.test.sh | 70 +++++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+), 3 deletions(-) diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index c6a8456bf23..0c7dd97bca5 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -1282,8 +1282,12 @@ fm_recovery_marker_handover_restore() { # /dev/null) || return 1 fm_current_pid current || return 1 @@ -1295,14 +1299,29 @@ fm_lock_reap_dead_link() { token= for tomb in "$owner".reaped.*; do [ -d "$tomb" ] || continue - if [ "${tomb##*.reaped.}" != "$current" ]; then - fm_pid_alive "${tomb##*.reaped.}" && return 1 + reaper=${tomb##*.reaped.} + pid=${reaper%%.start-*} + if [ "$pid" != "$current" ] && fm_pid_alive "$pid"; then + case "$reaper" in + *.start-*) recorded=${reaper#*.start-} ;; + *) return 1 ;; + esac + [ -n "$recorded" ] || return 1 + start=$(fm_pid_start_time "$pid" 2>/dev/null || true) + [ -n "$start" ] || return 1 + start=$(printf '%s' "$start" | od -An -v -tx1 | tr -d '[:space:]') + [ "$start" != "$recorded" ] || return 1 fi token=$tomb done [ -n "$token" ] || return 1 fi tomb="$owner.reaped.$current" + start=$(fm_pid_start_time "$current" 2>/dev/null || true) + if [ -n "$start" ]; then + start=$(printf '%s' "$start" | od -An -v -tx1 | tr -d '[:space:]') + tomb="$tomb.start-$start" + fi if [ "$token" != "$tomb" ]; then mv -- "$token" "$tomb" 2>/dev/null || return 1 fi diff --git a/tests/fm-watcher-lock.test.sh b/tests/fm-watcher-lock.test.sh index c19a244e85d..d619c9eb19f 100755 --- a/tests/fm-watcher-lock.test.sh +++ b/tests/fm-watcher-lock.test.sh @@ -627,6 +627,75 @@ test_lock_resumes_own_interrupted_steal_reap() { pass "a steal reap interrupted in this process is resumed from its own tombstone" } +# Interrupt the real cleanup at its unlink, after it publishes its tombstone. +test_lock_tombstone_tracks_reaper_start_time() ( + local dir state lockdir owner reaper tomb i rc holder_start unrelated_start unrelated + sleep "$FM_TEST_STUB_MAX_BLOCK_SECONDS" & + unrelated=$! + trap 'kill "$unrelated" 2>/dev/null || true; wait "$unrelated" 2>/dev/null || true' EXIT + # macOS start times have one-second precision. + sleep 2 + dir=$(make_case lock-reaper-pid-reuse) + state="$dir/state" + lockdir="$state/.contend.lock.steal" + owner="$lockdir.owner.fixture" + mkdir "$owner" + printf '%s\n' "$(dead_pid)" > "$owner/pid" + ln -s "$owner" "$lockdir" + FM_STATE_OVERRIDE="$state" bash -c ' + . "$1" + lock=$2 + ready=$3 + release=$4 + rm() { + if [ "${!#}" = "$lock" ]; then + : > "$ready" + while [ ! -e "$release" ] && kill -0 "$FM_TEST_SUITE_PID" 2>/dev/null \ + && [ "$SECONDS" -lt "$FM_TEST_STUB_MAX_BLOCK_SECONDS" ]; do + sleep 0.05 + done + fi + command rm "$@" + } + fm_lock_reap_dead_link "$lock" + ' _ "$LIB" "$lockdir" "$dir/reaper.ready" "$dir/release" & + reaper=$! + i=0 + while [ "$i" -lt 100 ] && [ ! -e "$dir/reaper.ready" ]; do + sleep 0.05 + i=$((i + 1)) + done + [ -e "$dir/reaper.ready" ] \ + || { kill "$reaper" 2>/dev/null || true; fail "reaper did not publish its election tombstone"; } + holder_start=$(start_time_of "$reaper") + unrelated_start=$(start_time_of "$unrelated") + [ -n "$holder_start" ] && [ -n "$unrelated_start" ] && [ "$holder_start" != "$unrelated_start" ] \ + || { kill "$reaper" 2>/dev/null || true; fail "reaper and unrelated process must have distinct starts"; } + rc=0 + FM_STATE_OVERRIDE="$state" bash -c '. "$1"; fm_lock_reap_dead_link "$2"' _ "$LIB" "$lockdir" || rc=$? + if [ "$rc" -eq 0 ] || [ ! -L "$lockdir" ]; then + : > "$dir/release" + wait "$reaper" 2>/dev/null || true + fail "contender cleared a genuinely live reaper's tombstone" + fi + kill -KILL "$reaper" 2>/dev/null || true + wait "$reaper" 2>/dev/null || true + tomb= + for tomb in "$owner".reaped.*; do [ ! -d "$tomb" ] || break; done + [ -d "$tomb" ] || fail "interrupted reaper left no tombstone" + # Model PID reuse by substituting a known unrelated live process, preserving + # whatever start-time identity the production election actually published. + mv -- "$tomb" "$owner.reaped.$unrelated${tomb#"$owner.reaped.$reaper"}" + FM_STATE_OVERRIDE="$state" bash -c '. "$1"; fm_lock_reap_dead_link "$2"' _ "$LIB" "$lockdir" \ + || fail "recycled reaper pid kept an abandoned tombstone alive" + [ ! -L "$lockdir" ] || fail "recycled-pid cleanup left the stale link" + for tomb in "$owner".reaped.*; do + [ ! -d "$tomb" ] || fail "recycled-pid cleanup left a tombstone" + done + is_live_non_zombie "$unrelated" || fail "cleanup disturbed the unrelated live process" + pass "tombstone cleanup respects a live reaper and recovers a recycled reaper pid" +) + test_lock_steal_reap_cannot_remove_successor() { # Two reapers verify the same dead steal owner. The competitor runs to # completion exactly when the first one is about to remove the link; at most @@ -1924,6 +1993,7 @@ test_msys_pid_identity_uses_proc() { pass "MSYS process identity uses compatible /proc fields" } +test_lock_tombstone_tracks_reaper_start_time || exit 1 test_wait_deadline_reaps_a_stopped_child test_singleton_start test_pid_identity_is_locale_invariant From 05196e30ead54796dfbe986db61abe9d8df9e3d2 Mon Sep 17 00:00:00 2001 From: Fabricio Tramontano Pirini Date: Thu, 29 Oct 2026 23:20:43 -0300 Subject: [PATCH 03/56] feat(bin): allow omp as a remote secondmate harness (#7939) * feat: allow omp as a remote secondmate harness * test: cover omp on the remote secondmate gates * fix: retire a remote secondmate whose busy state left with its home * docs: record omp remote retirement evidence * review: Refresh pinned remote doctor bootstrap checksum --- .../references/harness/omp.md | 1 - bin/fm-remote-doctor.sh | 2 +- bin/fm-remote-entrypoint.sh | 2 +- bin/fm-remote-secondmate-control.sh | 4 +- bin/fm-spawn.sh | 2 +- bin/fm-teardown.sh | 3 + docs/remote-secondmates.md | 3 +- docs/verification/runtime-backends.md | 22 +++ tests/fm-remote-doctor.test.sh | 9 ++ tests/fm-remote-secondmate-omp.test.sh | 127 ++++++++++++++++++ 10 files changed, 168 insertions(+), 7 deletions(-) create mode 100755 tests/fm-remote-secondmate-omp.test.sh diff --git a/.agents/skills/harness-adapters/references/harness/omp.md b/.agents/skills/harness-adapters/references/harness/omp.md index 959967e02b5..297f50bfd35 100644 --- a/.agents/skills/harness-adapters/references/harness/omp.md +++ b/.agents/skills/harness-adapters/references/harness/omp.md @@ -54,4 +54,3 @@ omp has no asynchronous Stop-hook equivalent, so the Claude auto-arm model does The Pi supervision branch does not run on omp; without the supervision host every actionable wake is delivered to main, and in a home with `config/supervision-host` and no `config/supervision-host-off` the watch extension spawns the host instead of the arm, with Claude's print mode as its headless engine ([`supervision-host.md`](../../../../../docs/supervision-host.md)). Launch a primary with plain `omp` inside the home (`FM_OMP_HARNESS=omp omp` when starting from a Claude pane); `../../../bin/fm-session-start.sh` prints `OMP_WATCH_EXTENSION: not loaded` when the running session has not loaded both tracked extensions. `FM_OMP_LIVE_E2E=1 ../../../tests/fm-omp-primary-live-e2e.test.sh` is the opt-in live guard; `../../../tests/fm-omp-harness.test.sh` is the portable regression. -A secondmate registered with `remote=1` in `data/secondmates.md`, spawned through the ordinary `../../../bin/fm-spawn.sh --secondmate` path, is refused on omp until a remote host verifies it, as is `../../../bin/fm-remote-secondmate-control.sh launch`; there is no `--remote` flag. diff --git a/bin/fm-remote-doctor.sh b/bin/fm-remote-doctor.sh index c6bea4de725..ea92f6d352b 100755 --- a/bin/fm-remote-doctor.sh +++ b/bin/fm-remote-doctor.sh @@ -68,7 +68,7 @@ FM_ROOT="${FM_ROOT_OVERRIDE:-$(CDPATH='' cd "$SCRIPT_DIR/.." && pwd -P)}" # shellcheck source=bin/fm-remote-herdr-owner-lib.sh . "$SCRIPT_DIR/fm-remote-herdr-owner-lib.sh" REQUIRED_TOOLS=(git jq herdr tasks-axi treehouse) -HARNESS_TOOLS=(claude codex opencode pi pi-signed grok kimi) +HARNESS_TOOLS=(claude codex opencode pi pi-signed grok kimi omp) OPTIONAL_TOOLS=(tmux no-mistakes gh) LAUNCH_AGENT_LABEL=dev.firstmate.herdr.fm-remote # The dedicated remote-secondmate session. The user's interactive Herdr work diff --git a/bin/fm-remote-entrypoint.sh b/bin/fm-remote-entrypoint.sh index 59e86ddf9d1..8b9c4043b70 100755 --- a/bin/fm-remote-entrypoint.sh +++ b/bin/fm-remote-entrypoint.sh @@ -31,7 +31,7 @@ set -eu PROTOCOL=1 -DOCTOR_SHA256=78efccd6cb7a0123400e49fa323292a64c8e3c7ebd3717151be69f87735302fb +DOCTOR_SHA256=b048ec093a8a1e58717b0906f269c666048b35feb90b2a62cd4ef86756573569 REAL_SOURCE=$(python3 -c 'import os, sys; print(os.path.realpath(sys.argv[1]))' "${BASH_SOURCE[0]}" 2>/dev/null) || REAL_SOURCE=$(realpath "${BASH_SOURCE[0]}" 2>/dev/null) || REAL_SOURCE=${BASH_SOURCE[0]} diff --git a/bin/fm-remote-secondmate-control.sh b/bin/fm-remote-secondmate-control.sh index 1f50e56d323..ee91f8ae152 100755 --- a/bin/fm-remote-secondmate-control.sh +++ b/bin/fm-remote-secondmate-control.sh @@ -177,7 +177,7 @@ cmd_launch() { validate_id "$id" validate_home "$id" case "$harness" in - claude|codex|opencode|pi|pi-signed|grok|kimi|cursor) ;; + claude|codex|opencode|pi|pi-signed|grok|kimi|cursor|omp) ;; *) die "unverified remote secondmate harness: $harness" ;; esac case "$effort" in -|low|medium|high|xhigh|max|ultra) ;; *) die "invalid remote secondmate effort: $effort" ;; esac @@ -249,7 +249,7 @@ cmd_relaunch() { validate_id "$id" validate_home "$id" case "$harness" in - claude|codex|opencode|pi|pi-signed|grok|kimi|cursor) ;; + claude|codex|opencode|pi|pi-signed|grok|kimi|cursor|omp) ;; *) die "unverified remote secondmate harness: $harness" ;; esac case "$effort" in -|default|low|medium|high|xhigh|max|ultra) ;; *) die "invalid remote secondmate effort: $effort" ;; esac diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index ff45aec720a..51e713efd03 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -1293,7 +1293,7 @@ spawn_remote_secondmate() { harness=$("$FM_ROOT/bin/fm-harness.sh" secondmate "$id") fi case "$harness" in - claude | codex | opencode | pi | pi-signed | grok | kimi | cursor) ;; + claude | codex | opencode | pi | pi-signed | grok | kimi | cursor | omp) ;; *) fm_lock_release "$registry_lock" || true fm_lock_release "$SPAWN_TASK_LOCK" || true diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index 4334540e43d..da43aa6410f 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -1581,6 +1581,9 @@ remove_kimi_turnend_auth() { retire_busy_state() { local state_dir=$1 id=$2 gen=${3:-} + # A remote secondmate's host-local route record lives inside the home that + # teardown has already removed, taking its busy records with it. + [ -d "$state_dir" ] || return 0 if [ -n "$gen" ]; then "$SCRIPT_DIR/fm-busy-event.sh" retire "$state_dir" "$id" --gen "$gen" elif [ -f "$state_dir/$id.busy-gen" ]; then diff --git a/docs/remote-secondmates.md b/docs/remote-secondmates.md index cd0e4bd0218..62924819313 100644 --- a/docs/remote-secondmates.md +++ b/docs/remote-secondmates.md @@ -323,7 +323,7 @@ A file at `~/.local/bin/fm-remote-entrypoint.sh` that is not Firstmate's own sym | Requirement | Tools | | --- | --- | | Always required | `git`, `jq`, `herdr`, compatible `tasks-axi`, and `treehouse` | -| At least one of | `claude`, `codex`, `opencode`, `pi`, `pi-signed`, `grok`, or `kimi` | +| At least one of | `claude`, `codex`, `opencode`, `pi`, `pi-signed`, `grok`, `kimi`, or `omp` | | Additionally required on macOS | `lsof`, so the doctor and guard can prove which process owns the session socket | ## Provision a route @@ -722,6 +722,7 @@ bin/fm-test-run.sh tests/fm-secondmate-sync.test.sh bin/fm-test-run.sh tests/fm-remote-reply.test.sh bin/fm-test-run.sh tests/fm-remote-backlog-handoff.test.sh bin/fm-test-run.sh tests/fm-remote-secondmate-lifecycle-e2e.test.sh +bin/fm-test-run.sh tests/fm-remote-secondmate-omp.test.sh bin/fm-test-run.sh tests/fm-remote-secondmate-trace-context.test.sh ``` diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index 544f8f7ef15..3e66c978288 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -3297,6 +3297,28 @@ Error: Invalid OMP profile "BAD". Profile names must match ^[a-z0-9][a-z0-9._-]{ `bin/fm-omp-profile-lib.sh` enforces the same name rule before launch, and `tests/fm-omp-harness.test.sh` pins the flag position, the profile-scoped catalog check, and the unpinned launch through a fake omp. Repeat these commands after an omp upgrade to refresh this record. +### Remote second mate on Linux + +Verified on 2026-10-28 against omp 18.13.3 (a Bun-compiled binary installed through mise) on Linux x86_64 with Herdr 0.9.1, driven from a macOS primary through a scratch home on both sides. +The agent process reads `comm=omp` with its executable inside the mise install tree, so the anchored name identifies it on Linux as on macOS and no launcher-specific identity rule is needed: + +```text +$ bin/fm-spawn.sh --secondmate # config/secondmate-harness: omp anthropic/claude-opus-5-5 high +spawned harness=omp kind=secondmate mode=secondmate yolo=off window=remote: worktree= remote= backend=herdr +$ bin/fm-on.sh fm-remote-secondmate-control.sh state +alive +$ bin/fm-remote-secondmate-relaunch.sh omp anthropic/claude-opus-5-5 high +relaunched harness=omp from=omp model=anthropic/claude-opus-5-5 effort=high backend=herdr endpoint=fm-remote:w1:p2 worktree= +# on the host, the relaunched agent process +comm=omp FM_OMP_HARNESS=omp HERDR_SESSION=fm-remote +$ bin/fm-teardown.sh +teardown complete (remote :) +``` + +Two `bin/fm-send.sh fm-` steers, one before and one after the relaunch, were moved to `handled/` by the agent and answered with correlated lines in the remote `state/parent-replies.status`; the second answer reported `bin/fm-harness.sh` resolving `omp` from the agent's own tool shell and the session lock held by the omp process. +Retirement removes the remote home, and with it the route state that holds the busy records omp arms, so the host-local teardown skips retiring those records once that state directory is gone. +`tests/fm-remote-secondmate-omp.test.sh` is the portable regression for the remote gates and retirement; repeat the commands above after an omp or Herdr upgrade to refresh this record. + ## Busy inbox escalation Verified at `2026-10-03T19:20:07Z` on commit `23b0232908a5adc7fbf7339ef48c34a091a7a799` with Claude Code `2.1.288 (Claude Code)` on Herdr `0.9.1`, protocol `22`, in a named isolated lab session through `bin/fm-herdr-lab.sh`. diff --git a/tests/fm-remote-doctor.test.sh b/tests/fm-remote-doctor.test.sh index b134a815e7b..3a6dc1a9229 100755 --- a/tests/fm-remote-doctor.test.sh +++ b/tests/fm-remote-doctor.test.sh @@ -779,6 +779,15 @@ assert_contains "$DOCTOR_OUT" 'check herdr-server=ok:' "the started server was n [ ! -s "$CASE_LAUNCHCTL_LOG" ] || fail "the linux path invoked launchctl" pass "a non-darwin host skips launch agents and starts its herdr server directly" +new_case Linux with-herdr no-gui +rm -f "$CASE_BIN/claude" +printf '#!/usr/bin/env bash\nexit 0\n' > "$CASE_BIN/omp" +chmod +x "$CASE_BIN/omp" +doctor --fix +expect_code 0 "$DOCTOR_RC" "a host whose only harness is omp was not ready after --fix" +assert_contains "$DOCTOR_OUT" 'required harness=omp:' "omp did not satisfy the at-least-one-harness requirement" +pass "omp alone satisfies the remote harness requirement" + # --- --fix may add only owned wrappers for version-manager tools ------------- new_case Linux with-herdr no-gui diff --git a/tests/fm-remote-secondmate-omp.test.sh b/tests/fm-remote-secondmate-omp.test.sh new file mode 100755 index 00000000000..44f114b15d5 --- /dev/null +++ b/tests/fm-remote-secondmate-omp.test.sh @@ -0,0 +1,127 @@ +#!/usr/bin/env bash +# tests/fm-remote-secondmate-omp.test.sh - an omp remote secondmate passes every +# gate that decides remote placement and retires cleanly: +# +# A) bin/fm-remote-secondmate-control.sh launch accepts omp and drives the +# ordinary host-local secondmate spawn onto the Herdr fm-remote session, +# recording harness=omp and the pinned model in the endpoint metadata. +# B) The relaunch verb accepts omp and reaches the ordinary control plane's own +# pre-stop checkpoint, proving its harness gate passed. +# C) bin/fm-spawn.sh's parent-side remote secondmate gate accepts omp and +# proceeds to the remote transport hop. +# D) Host-local retirement succeeds even though the busy records omp arms live +# in the route state inside the home that retirement removes. +# +# The Herdr endpoint is the stateful fixture in tests/remote-herdr-fixture.sh +# and omp is a fake that answers `models --json`, so no vendor tool is needed; +# tests/fm-omp-harness.test.sh owns the omp launch contract itself. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +# shellcheck source=tests/remote-herdr-fixture.sh +. "$(dirname "${BASH_SOURCE[0]}")/remote-herdr-fixture.sh" + +command -v jq >/dev/null 2>&1 || { echo "skip: jq not found"; exit 0; } +command -v node >/dev/null 2>&1 || { echo "skip: node not found"; exit 0; } + +TMP_ROOT=$(fm_test_tmproot fm-remote-secondmate-omp) +mkdir -p "$TMP_ROOT" +TMP_ROOT=$(cd "$TMP_ROOT" && pwd -P) +BASE_PATH=/usr/bin:/bin:/usr/sbin:/sbin + +FAKEBIN="$TMP_ROOT/fakebin" +mkdir -p "$FAKEBIN" +install_remote_herdr_fixture "$FAKEBIN-root" "$TMP_ROOT/herdr.state" "$TMP_ROOT/herdr.log" \ + "$TMP_ROOT/herdr-send-fail" "$TMP_ROOT/herdr.sock" +mv "$FAKEBIN-root/bin/herdr" "$FAKEBIN/herdr" +rmdir "$FAKEBIN-root/bin" "$FAKEBIN-root" +ln -sf "$(command -v git)" "$FAKEBIN/git" +ln -sf "$(command -v jq)" "$FAKEBIN/jq" +ln -sf "$(command -v node)" "$FAKEBIN/node" +cat > "$FAKEBIN/omp" <<'SH' +#!/usr/bin/env bash +[ "${1:-}" != models ] || printf '%s\n' '{"models":[{"provider":"anthropic","id":"claude-test","selector":"anthropic/claude-test"}]}' +exit 0 +SH +chmod +x "$FAKEBIN/omp" + +SSH_LOG="$TMP_ROOT/ssh-hops.log" +cat > "$FAKEBIN/fake-ssh" <> "$SSH_LOG" +exit 255 +SH +chmod +x "$FAKEBIN/fake-ssh" + +ID=sm-omp +HOME_DIR="$TMP_ROOT/home-$ID" +mkdir -p "$HOME_DIR/bin" "$HOME_DIR/data" +printf '# Firstmate\n' > "$HOME_DIR/AGENTS.md" +printf '%s\n' "$ID" > "$HOME_DIR/.fm-secondmate-home" +printf '# Charter\n\nStand by until steering arrives.\n' > "$HOME_DIR/data/charter.md" +git -C "$HOME_DIR" init -q + +ctl() { + PATH="$FAKEBIN:$BASE_PATH" TMUX='' FM_GATE_REFUSE_BYPASS=1 FM_SPAWN_NO_GUARD=1 \ + FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$HOME_DIR" \ + "$ROOT/bin/fm-remote-secondmate-control.sh" "$@" +} + +# --- A) host-local launch ----------------------------------------------------- + +ctl launch "$ID" omp anthropic/claude-test high herdr > "$TMP_ROOT/launch.out" 2> "$TMP_ROOT/launch.err" +expect_code 0 "$?" "host-local remote secondmate launch on omp failed"$'\n'"$(cat "$TMP_ROOT/launch.err")" +assert_grep 'harness=omp' "$TMP_ROOT/launch.out" "the printed route lost the omp harness" +assert_grep 'target=fm-remote:' "$TMP_ROOT/launch.out" "the omp endpoint target is outside fm-remote" +META="$HOME_DIR/state/parent-route/$ID.meta" +assert_grep 'harness=omp' "$META" "endpoint metadata did not record the omp harness" +assert_grep 'model=anthropic/claude-test' "$META" "endpoint metadata lost the pinned omp model" +assert_grep 'effort=high' "$META" "endpoint metadata lost the pinned omp effort" +assert_grep 'herdr_session=fm-remote' "$META" "endpoint metadata did not record the fm-remote session" +[ "$(ctl state "$ID")" = alive ] || fail "the omp endpoint did not read alive from its host-local route" +pass "A host-local launch accepts omp and records the fm-remote Herdr endpoint" + +# --- B) host-local relaunch --------------------------------------------------- + +cp "$META" "$TMP_ROOT/meta-before-relaunch" +mkdir -p "$TMP_ROOT/not-a-checkout" +sed "s|^worktree=.*|worktree=$TMP_ROOT/not-a-checkout|" "$TMP_ROOT/meta-before-relaunch" > "$META" +if ctl relaunch "$ID" omp - - > "$TMP_ROOT/relaunch.out" 2>&1; then + fail "relaunch with an unaccountable checkout should refuse in the control plane" +fi +assert_grep 'refusing to relaunch without a checkout whose unlanded work can be accounted for' \ + "$TMP_ROOT/relaunch.out" "an omp relaunch did not reach the control plane's own pre-stop checkpoint" +cp "$TMP_ROOT/meta-before-relaunch" "$META" +pass "B host-local relaunch accepts omp and reaches the control plane" + +# --- C) parent-side remote spawn ---------------------------------------------- + +PARENT="$TMP_ROOT/parent" +mkdir -p "$PARENT/data" "$PARENT/state" "$PARENT/config" "$PARENT/projects" +cat > "$PARENT/data/secondmates.md" < "$TMP_ROOT/spawn.out" 2> "$TMP_ROOT/spawn.err" \ + && fail "the parent-side spawn should surface the unreachable remote transport" +assert_grep 'readiness could not be confirmed; preserved route' "$TMP_ROOT/spawn.err" \ + "the omp remote spawn did not pass the harness gate to the transport hop"$'\n'"$(cat "$TMP_ROOT/spawn.err")" +[ -s "$SSH_LOG" ] || fail "the omp remote spawn never reached the remote transport hop" +pass "C the parent-side remote spawn gate accepts omp" + +# --- D) host-local retirement ------------------------------------------------- + +# omp arms the busy contract, so its busy records live in the route state inside +# the home that retirement removes. +[ -f "$HOME_DIR/state/parent-route/$ID.busy-gen" ] || fail "the omp launch armed no busy generation to retire" +ctl retire "$ID" > "$TMP_ROOT/retire.out" 2>&1 +expect_code 0 "$?" "host-local retirement of an omp mate failed"$'\n'"$(cat "$TMP_ROOT/retire.out")" +assert_absent "$HOME_DIR" "retirement left the omp mate's home behind" +pass "D host-local retirement removes an omp mate's home and succeeds" + +echo "ALL TESTS PASSED" From c0aa730d8bdde9ddce3d0fbb8a0cee71e0b56610 Mon Sep 17 00:00:00 2001 From: c6aw <225476801+c6aw@users.noreply.github.com> Date: Fri, 30 Oct 2026 01:16:11 -0500 Subject: [PATCH 04/56] fix(bin): name a consumed branch grant instead of a wiring bug (#7969) * fix(bin): name a consumed branch grant instead of a wiring bug When a supervision branch's own acknowledgement consumes every granted row, the drain removes the branch-eligible row snapshot while the owner record stays live. A repeat drain or acknowledgement in the same turn then refused with a message implying a broken handoff. Say instead that the grant was already consumed; the refusal and its exit status are unchanged. * no-mistakes(document): Document consumed-grant refusal in watcher continuity doc --------- Co-authored-by: c-c6aw <225476801+c-c6aw@users.noreply.github.com> --- bin/fm-wake-drain.sh | 19 ++++++++++++------- docs/watcher-continuity.md | 3 ++- tests/fm-wake-queue.test.sh | 27 +++++++++++++++++++++++++++ 3 files changed, 41 insertions(+), 8 deletions(-) diff --git a/bin/fm-wake-drain.sh b/bin/fm-wake-drain.sh index 9f2756e892e..37905ff8057 100755 --- a/bin/fm-wake-drain.sh +++ b/bin/fm-wake-drain.sh @@ -193,15 +193,20 @@ consume_actor_rows_locked() { # # A branch-actor drain or ack requires a snapshot to already exist and name at # least one row. The extension always writes a non-empty snapshot before it -# ever prompts the branch (an empty eligible set means no prompt at all), so a -# missing or empty file here means this ran outside that handoff - a wiring -# bug, never "nothing eligible" - and must fail loudly rather than silently -# draining or acking nothing. +# ever prompts the branch (an empty eligible set means no prompt at all), and +# the branch's own ack removes the snapshot once it consumes every granted row. +# So a missing file with a live owner means this turn's grant was already +# consumed, and otherwise this ran outside that handoff - a wiring bug. Either +# way this must fail loudly rather than silently draining or acking nothing. require_branch_eligible_rows() { - rows_file_valid "$ELIGIBLE_ROWS_FILE" || { + rows_file_valid "$ELIGIBLE_ROWS_FILE" && return 0 + if [ ! -e "$ELIGIBLE_ROWS_FILE" ] && [ ! -L "$ELIGIBLE_ROWS_FILE" ] \ + && fm_wake_branch_owner_matches "$ELIGIBLE_OWNER_FILE"; then + echo "wake drain: this branch's grant was already consumed by its acknowledgement; nothing is left to drain or acknowledge, so do not acknowledge again" >&2 + else echo "wake drain: no branch-eligible row snapshot at $ELIGIBLE_ROWS_FILE; refusing to guess what this actor may consume" >&2 - return 1 - } + fi + return 1 } # The highest sequence this actor has already been presented: the branch's diff --git a/docs/watcher-continuity.md b/docs/watcher-continuity.md index a7c51c639c3..1f57575f100 100644 --- a/docs/watcher-continuity.md +++ b/docs/watcher-continuity.md @@ -349,7 +349,8 @@ Under the away-posture record the exclusion lifts and a check row is offered to `fm-wake-drain.sh` never reclassifies a row itself. It filters the queue to the current actor's opaque claim before same-key deduplication, then presents and acknowledges only that actor-local view. -A missing or empty branch snapshot is refused loudly rather than read as "nothing eligible", because reaching the drain without the non-empty handoff promised by the extension is a wiring bug. +A missing or empty branch snapshot is refused loudly rather than read as "nothing eligible". +With a live branch owner, a missing snapshot means the branch's own acknowledgement already consumed the grant, and the refusal says so; otherwise reaching the drain without the non-empty handoff promised by the extension is a wiring bug. A branch acknowledgement retires the check-row receipts - inactive-outcome, inactive-reconcile notice, and secondmate stall - of exactly the granted sequences it consumes, so a branch-consumed check is never re-queued by its producer. Attended, a grant names no check row and each scan finds nothing. diff --git a/tests/fm-wake-queue.test.sh b/tests/fm-wake-queue.test.sh index 5f3cf0b519a..69868a4b9c3 100755 --- a/tests/fm-wake-queue.test.sh +++ b/tests/fm-wake-queue.test.sh @@ -1944,6 +1944,32 @@ test_branch_actor_without_eligible_snapshot_refuses() { pass "a branch-actor drain with no eligible-row snapshot refuses loudly instead of draining nothing" } +test_branch_repeat_ack_after_consumed_grant_names_the_consumed_grant() { + local dir state seq gen + dir=$(make_case branch-repeat-ack-consumed) + state="$dir/state" + append_wake "$state" signal "task-a.status" "signal: task-a" || fail "append failed" + FM_STATE_OVERRIDE="$state" "$GRANT" activate "$$" branch-repeat || fail "branch owner activation failed" + FM_STATE_OVERRIDE="$state" "$GRANT" publish branch-repeat 1 || fail "grant publication failed" + FM_STATE_OVERRIDE="$state" FM_SUPERVISION_ACTOR=branch "$DRAIN" > "$dir/drain.out" 2> "$dir/drain.err" \ + || fail "branch drain failed: $(cat "$dir/drain.err")" + seq=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through \([0-9][0-9]*\) --recovery-generation .*/\1/p' "$dir/drain.err") + gen=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--recovery-generation \([A-Za-z0-9._-][A-Za-z0-9._-]*\)$/\1/p' "$dir/drain.err") + FM_STATE_OVERRIDE="$state" FM_SUPERVISION_ACTOR=branch "$DRAIN" --ack-through "$seq" --recovery-generation "$gen" \ + || fail "first branch acknowledgement failed" + if FM_STATE_OVERRIDE="$state" FM_SUPERVISION_ACTOR=branch "$DRAIN" --ack-through "$seq" --recovery-generation "$gen" \ + >/dev/null 2>"$dir/repeat.err"; then + fail "a repeat branch acknowledgement after its grant was consumed must still refuse" + fi + grep -q "already consumed" "$dir/repeat.err" \ + || fail "the repeat acknowledgement did not say the grant was already consumed: $(cat "$dir/repeat.err")" + if grep -q "wiring" "$dir/repeat.err"; then + fail "the repeat acknowledgement still blamed wiring: $(cat "$dir/repeat.err")" + fi + FM_STATE_OVERRIDE="$state" "$GRANT" deactivate "$$" branch-repeat || fail "branch owner deactivation failed" + pass "a repeat branch acknowledgement after its own ack consumed the grant says the grant was already consumed" +} + test_wake_publish_requires_atomic_recovery_evidence() { local dir state fakebin real_mv rc out dir=$(make_case wake-publish-recovery-evidence) @@ -3754,6 +3780,7 @@ test_main_ack_leaves_a_row_that_arrived_after_its_drain_unclaimed test_actor_filter_precedes_same_key_deduplication test_main_reclaims_a_grant_whose_branch_owner_exited test_branch_actor_without_eligible_snapshot_refuses +test_branch_repeat_ack_after_consumed_grant_names_the_consumed_grant test_wake_publish_requires_atomic_recovery_evidence test_recovery_mint_and_delivery_log_avoid_sibling_subst test_large_sequences_stay_exact_decimal_identifiers From b83dc59e095d859cc12030836223350d64393d6f Mon Sep 17 00:00:00 2001 From: WTF Am I Doing ? Date: Fri, 30 Oct 2026 02:17:10 -0400 Subject: [PATCH 05/56] fix: preserve literal ampersands in spawn and supervision paths (#7906) --- bin/fm-spawn.sh | 3 ++ bin/fm-supervision-instructions.sh | 3 ++ .../fm-spawn-compact-adviser-disable.test.sh | 32 +++++++++++++++++++ tests/fm-supervision-instructions.test.sh | 29 +++++++++++++++++ 4 files changed, 67 insertions(+) diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 51e713efd03..1cceea164a0 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -702,6 +702,9 @@ # Local spawns never pass it and resolve their own carrier exactly as before. set -eu +# Template replacements must stay literal on Bash 5.2+ and stock macOS Bash. +shopt -u patsub_replacement 2>/dev/null || true + SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" usage() { diff --git a/bin/fm-supervision-instructions.sh b/bin/fm-supervision-instructions.sh index f70c4a943d2..771fde1ce92 100755 --- a/bin/fm-supervision-instructions.sh +++ b/bin/fm-supervision-instructions.sh @@ -12,6 +12,9 @@ # unchanged. set -eu +# Template replacements must stay literal on Bash 5.2+ and stock macOS Bash. +shopt -u patsub_replacement 2>/dev/null || true + SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" FM_ROOT="${FM_ROOT_OVERRIDE:-$REPO_ROOT}" diff --git a/tests/fm-spawn-compact-adviser-disable.test.sh b/tests/fm-spawn-compact-adviser-disable.test.sh index f9b7f7482d1..10849684b42 100755 --- a/tests/fm-spawn-compact-adviser-disable.test.sh +++ b/tests/fm-spawn-compact-adviser-disable.test.sh @@ -382,6 +382,38 @@ SH pass "a compound raw launch-command still starts its agent with the compact-adviser switch on" } +test_launch_preserves_literal_home_and_brief_paths() { + local name rec id out status seen want brief + for name in normal literal; do + id="path-$name-a1" + rec=$(make_case "path-$name" codex "$id") + read_case "$rec" + if [ "$name" = literal ]; then + mv "$HOME_DIR" "$CASE_DIR/R&D \"double\" 'single' home" + HOME_DIR="$CASE_DIR/R&D \"double\" 'single' home" + fi + out=$(run_case_spawn "$id" "$PROJ_DIR" --mode no-mistakes --yolo off) + status=$? + expect_code 0 "$status" "$name path spawn should succeed: $out" + # Capture the environment and final prompt received by the fake agent. + cat > "$FAKEBIN_DIR/codex" <<'SH' +#!/bin/sh +printf '%s\n' "$FM_TASK_INBOX" +for arg do :; done +printf '%s\n' "$arg" +SH + chmod +x "$FAKEBIN_DIR/codex" + seen=$(emitted_launch_env "$FAKEBIN_DIR" "$LAUNCH_LOG" "$PANE_LOG") \ + || fail "$name path: the emitted launch failed to run" + brief=$("$ROOT/bin/fm-operational-input.sh" encode launch-brief < "$HOME_DIR/data/$id/launch-brief.md") + want="$(cd "$HOME_DIR/state" && pwd -P)/$id.inbox +$brief" + assert_equals "$want" "$seen" "$name path: launch changed the home or brief bytes" + done + pass "launch preserves literal ampersands, quotes and spaces in home and brief paths" +} + +test_launch_preserves_literal_home_and_brief_paths test_ship_allowlist_absent test_ship_allowlist_enabled test_launch_command_carries_the_switch_without_the_pane_export diff --git a/tests/fm-supervision-instructions.test.sh b/tests/fm-supervision-instructions.test.sh index 699ad35816b..830966e9f45 100755 --- a/tests/fm-supervision-instructions.test.sh +++ b/tests/fm-supervision-instructions.test.sh @@ -342,6 +342,35 @@ test_pi_snippet_uses_effective_extension_path() { pass "pi supervision snippet renders the effective extension path" } +test_literal_paths_in_rendered_snippets() { + local name home harness out quoted_config + for name in normal 'R&D "double" '\''single'\'' home'; do + home="$TMP_ROOT/$name" + mkdir -p "$home/config" "$home/state" + : > "$home/config/supervision-host-off" + quoted_config="'$(printf '%s' "$home/config/x-mode.env" | sed "s/'/'\\\\''/g")'" + for harness in pi pi-signed omp codex grok cursor opencode claude; do + out=$(FM_HOME="$home" FM_ROOT_OVERRIDE="$home" "$RENDER" --harness "$harness" --x-mode 1) + assert_not_contains "$out" '__FM_' "$harness: literal path leaked a placeholder" + case "$harness" in + pi|pi-signed) + assert_contains "$out" "$home/.pi/extensions/fm-primary-pi-watch.ts" "Pi watch path changed" + assert_contains "$out" "$home/.pi/extensions/fm-primary-turnend-guard.ts" "Pi turn-end path changed" + ;; + omp) + assert_contains "$out" "$home/.omp/extensions/fm-primary-omp-watch.ts" "omp watch path changed" + assert_contains "$out" "$home/.omp/extensions/fm-primary-turnend-guard.ts" "omp turn-end path changed" + ;; + grok) + assert_contains "$out" "[ -f $quoted_config ] && . $quoted_config; exec bin/fm-watch-arm.sh" "quoted config path changed" + ;; + esac + done + done + pass "snippets preserve literal ampersands, quotes and spaces in paths across primary harnesses" +} + +test_literal_paths_in_rendered_snippets test_supervision_host_protocol_on_a_claude_home_unless_off test_supervision_host_protocol_on_every_arm_owner test_selected_harness_block_only From 7647d6ba10ff33082f473576c241e15dcbf45cbe Mon Sep 17 00:00:00 2001 From: Michael Date: Fri, 30 Oct 2026 11:20:50 +0100 Subject: [PATCH 06/56] feat(bin): read dispatch provider keys from a declared credential source (#5409) * feat(dispatch): read Typesafe key from Keychain * no-mistakes(review): Share Keychain key lookup with bootstrap; hard-code service name * no-mistakes(review): Use one readable-secret Keychain check for bootstrap and resolver * no-mistakes(document): Point AGENTS key row to owner; record Keychain bootstrap evidence * feat(dispatch): read provider keys from a declared credential source bin/fm-dispatch-resolve.sh took the typesafe.ai and OpenRouter keys only from the environment or $FM_HOME/.env, so a home that keeps its key in the macOS Keychain needed a wrapper to supply it, and the documented direct call reported off. Let the resolver consume an optional, provider-keyed declaration in the effective config directory, config/dispatch-credential-sources.json. It names where a key lives and never holds one. macos-keychain is the first adapter (service, optional account), kept behind a small schema and adapter registry in bin/fm-credential-source-lib.sh so the shared surface binds to no one OS or provider. - Precedence is the environment, then .env, then the declared source, which is read only when neither supplied a key and only for the selected provider. - No entry and no key stays off, exactly as before. An unreadable or invalid declaration exits 2 with a line that names the problem and never a declared value. - A declared source that this host cannot run, that yields no key, or that does not answer within FM_CREDENTIAL_SOURCE_TIMEOUT ends as a structured error outcome with nothing sent, so it is not mistaken for an unconfigured resolver. - The key is captured into the same private variable as before and still reaches curl only as an fd 3 header. The adapter puts only the service and account on argv and discards the tool's stderr. - --supports-credential-source answers supported or unsupported without reading a declaration or running the credential tool, so an installer can refuse an unsupported activation before it writes anything. The resolver tests cover the mapping, effective-home and provider isolation, precedence, the three failure kinds, and secret containment with a stubbed security and uname. Closes #7879 * no-mistakes(review): Gate bootstrap typed checks on declared source; fix timeout * no-mistakes(review): Document storing the declared Keychain item without access dialog * no-mistakes(document): Clarify that only a typesafe declaration keeps resolver on * no-mistakes(document): Merge dispatch key-source precedence into one configuration sentence --------- Co-authored-by: Kun Chen --- bin/fm-bootstrap.sh | 12 +- bin/fm-credential-source-lib.sh | 139 ++++++++++ bin/fm-dispatch-resolve.sh | 120 ++++++-- docs/configuration.md | 68 ++++- docs/scripts.md | 1 + docs/verification/dispatch-resolve.md | 9 +- tests/fm-bootstrap.test.sh | 33 ++- tests/fm-dispatch-resolve.test.sh | 385 +++++++++++++++++++++++++- 8 files changed, 719 insertions(+), 48 deletions(-) create mode 100644 bin/fm-credential-source-lib.sh diff --git a/bin/fm-bootstrap.sh b/bin/fm-bootstrap.sh index f650ea033e2..da625e34117 100755 --- a/bin/fm-bootstrap.sh +++ b/bin/fm-bootstrap.sh @@ -211,6 +211,8 @@ DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" . "$SCRIPT_DIR/fm-remote-readiness-lib.sh" # shellcheck source=bin/fm-timeout-lib.sh disable=SC1091 . "$SCRIPT_DIR/fm-timeout-lib.sh" +# shellcheck source=bin/fm-credential-source-lib.sh disable=SC1091 +. "$SCRIPT_DIR/fm-credential-source-lib.sh" # Shared secondmate endpoint probe + guarded relaunch; the watcher's poll tick # drives the same library so session start and ordinary supervision recover # from identical evidence through an identical path. @@ -1148,7 +1150,7 @@ EOF } crew_dispatch_validate() { - local file err verified_harnesses typed_key typed_active=false + local file cred err verified_harnesses typed_key typed_active=false file="$CONFIG/crew-dispatch.json" [ -f "$file" ] || return 0 if ! command -v jq >/dev/null 2>&1; then @@ -1161,7 +1163,13 @@ crew_dispatch_validate() { fi typed_key=$TYPESAFE_API_KEY_PRIVATE [ -n "$typed_key" ] || typed_key=$(fmx_env_get TYPESAFE_API_KEY "$FM_HOME/.env") - [ -z "$typed_key" ] || typed_active=true + cred="$CONFIG/dispatch-credential-sources.json" + if [ -n "$typed_key" ]; then + typed_active=true + elif [ -f "$cred" ] && fm_credsrc_check "$cred" typesafe openrouter >/dev/null && + fm_credsrc_declares "$cred" typesafe; then + typed_active=true + fi if $typed_active; then verified_harnesses=$(fm_control_harnesses | jq -Rsc 'split("\n") | map(select(length > 0))') else diff --git a/bin/fm-credential-source-lib.sh b/bin/fm-credential-source-lib.sh new file mode 100644 index 00000000000..a58f4fe56a0 --- /dev/null +++ b/bin/fm-credential-source-lib.sh @@ -0,0 +1,139 @@ +# shellcheck shell=bash +# Declared credential sources for provider keys. +# Usage: . bin/fm-credential-source-lib.sh +# +# This file is the single owner of the credential-source declaration: a JSON +# object keyed by provider whose values say WHERE that provider's key lives and +# never hold the key itself, for example +# {"typesafe": {"source": "macos-keychain", "service": "typesafe-api-key"}} +# bin/fm-dispatch-resolve.sh reads it only after the environment and .env have +# supplied no key, and bin/fm-bootstrap.sh only checks whether it declares a +# typesafe source; docs/configuration.md "Typed dispatch resolution" owns the +# file's location and the operator contract. +# +# A source is an adapter: FM_CREDSRC_SCHEMA names each adapter and its fields, +# fm_credsrc_supported says whether this host can run it, and fm_credsrc_read +# fetches the key through it. macos-keychain is the first. A further adapter +# adds one schema entry and one arm to those two functions; nothing here knows +# which provider is asking, so the shared surface binds to no one OS or vendor. +# +# Secret containment: a key leaves fm_credsrc_read on stdout for the caller's +# command substitution and nowhere else. It is never an argument, an +# environment variable, a file, or a log line. An adapter passes only the +# fields that identify the item to its tool, and discards the tool's stderr +# because that can echo what it was given. fm_credsrc_supported never runs the +# tool, so asking whether a source is available retrieves no credential. + +# shellcheck source=bin/fm-timeout-lib.sh +. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/fm-timeout-lib.sh" + +# Adapter name -> the fields it takes. A declaration holds "source" plus +# exactly these fields, so no field exists for a credential value to sit in. +FM_CREDSRC_SCHEMA='{"macos-keychain":{"required":["service"],"optional":["account"]}}' + +# Seconds one adapter read may take, so an access prompt nobody answers cannot +# hold the caller. +FM_CREDSRC_TIMEOUT=5 + +# fm_credsrc_supported +# Succeed when names an adapter this host can run. It checks the host +# and that the adapter's tool is on PATH, and runs nothing else. +fm_credsrc_supported() { + case "${1:-}" in + macos-keychain) + [ "$(uname -s 2>/dev/null)" = Darwin ] && command -v security >/dev/null 2>&1 + ;; + *) return 1 ;; + esac +} + +# fm_credsrc_check ... +# Succeed, printing nothing, when is one JSON object whose keys are all +# among the named providers and whose values are well-formed declarations. +# Otherwise print the first problem on one line and fail. A problem names the +# provider, the field, or the allowed values; it never prints a declared value. +fm_credsrc_check() { + local file=$1 providers problem + shift + providers=$(printf '%s\n' "$@" | jq -Rsc 'split("\n") | map(select(length > 0))') || return 1 + problem=$(jq -rs --argjson schema "$FM_CREDSRC_SCHEMA" --argjson providers "$providers" ' + def flat: explode | map(if . < 32 or . == 127 then 63 else . end) | implode; + def plain: type == "string" and length > 0 and (explode | all(. >= 32 and . != 127)); + def entry($name; $d): + if ($d | type) != "object" then "\($name) must be an object with a source" + elif ($d | has("source") | not) then "\($name) needs a source" + elif ($d.source | type) != "string" or ($schema | has($d.source) | not) then + "\($name) source must be one of \($schema | keys | join(", "))" + else + $schema[$d.source] as $f + | (["source"] + $f.required + $f.optional) as $allowed + | ([$d | keys_unsorted[] | select(. as $k | $allowed | index($k) | not)]) as $extra + | ([$f.required[] | . as $k | select(($d[$k] | plain) | not)]) as $bad_required + | ([$f.optional[] | . as $k | select($d | has($k)) | select(($d[$k] | plain) | not)]) as $bad_optional + | if ($extra | length) > 0 then + "\($name) has unknown field \($extra[0] | flat) (allowed: \($allowed | join(", "))); credential values do not belong in this file" + elif ($bad_required | length) > 0 then + "\($name) \($bad_required[0]) must be a non-empty string without control characters" + elif ($bad_optional | length) > 0 then + "\($name) \($bad_optional[0]) must be a non-empty string without control characters when present" + else empty end + end; + if length != 1 then "must hold exactly one JSON object" + elif (.[0] | type) != "object" then "top-level value must be an object keyed by provider" + else + .[0] as $doc + | ([$doc | keys_unsorted[] | select(. as $k | $providers | index($k) | not)]) as $unknown + | if ($unknown | length) > 0 then + "unknown provider \($unknown[0] | flat) (expected \($providers | join(" or ")))" + else first(($doc | to_entries[] | entry(.key; .value)), empty) + end + end' "$file" 2>/dev/null) || problem='not JSON' + if [ -z "$problem" ]; then + return 0 + fi + printf '%s\n' "$problem" + return 1 +} + +# fm_credsrc_declares +# Succeed when declares a source for . +fm_credsrc_declares() { + jq -e --arg p "$2" 'has($p)' "$1" >/dev/null 2>&1 +} + +# fm_credsrc_field +# Print one declared field of 's source, or nothing when it is absent. +# Call only on a file fm_credsrc_check accepted, which keeps every value on one +# printable line. +fm_credsrc_field() { + jq -r --arg p "$2" --arg f "$3" '.[$p][$f] // empty' "$1" +} + +# fm_credsrc_read +# Print the key 's declared source yields. Status 0 means a non-empty +# read; fm_timed_out accepts the status when the source did not answer within +# its bound; any other status means the source holds no readable key. Call only +# on a file fm_credsrc_check accepted and fm_credsrc_supported agreed to run. +fm_credsrc_read() { + local src service account + src=$(fm_credsrc_field "$1" "$2" source) + case "$src" in + macos-keychain) + service=$(fm_credsrc_field "$1" "$2" service) + account=$(fm_credsrc_field "$1" "$2" account) + fm_credsrc_macos_keychain "$service" "$account" + ;; + *) return 1 ;; + esac +} + +# macos-keychain: one generic-password read, bounded by FM_CREDSRC_TIMEOUT. +# Only the service and, when declared, the account reach argv; the key comes +# back on stdout. +fm_credsrc_macos_keychain() { # [] + if [ -n "${2:-}" ]; then + fm_run_timed "$FM_CREDSRC_TIMEOUT" security find-generic-password -s "$1" -a "$2" -w 2>/dev/null /dev/null [--project ] +# fm-dispatch-resolve.sh --supports-credential-source # # Opt-in gate: FM_DISPATCH_PROVIDER and FM_DISPATCH_MODEL select the provider # and model from the environment or $FM_HOME/.env (environment wins). With no @@ -13,9 +14,33 @@ # (http:// or https://) and posts to /v1/systemone; its optional # FM_DISPATCH_API_KEY, when absent, means no Authorization header. The openai # provider maps the System One request to OpenAI's Decisions API and maps its -# answer back before validation. The selected key lives in a non-exported -# shell variable and reaches curl through fd 3, never argv, logs, or child -# environments. +# answer back before validation. The typesafe and openrouter keys come from the +# environment, else $FM_HOME/.env, else the credential source declared for that +# provider (below); the openai and systemone keys come from the environment or +# $FM_HOME/.env only. The selected key lives in a non-exported shell variable +# and reaches curl through fd 3, never argv, logs, or child environments. +# +# Credential source declaration: the optional $CONFIG/dispatch-credential-sources.json +# ($CONFIG is FM_CONFIG_OVERRIDE, else $FM_HOME/config) is a JSON object keyed +# by provider (typesafe or openrouter). Each value says where that provider's +# key lives and holds no key: {"source": "macos-keychain", "service": "", +# "account": ""}, where account is optional and no other field is +# accepted. bin/fm-credential-source-lib.sh owns the schema and its adapters; +# macos-keychain reads the generic-password item with `security +# find-generic-password -s [-a ] -w`, so only the service +# and account reach argv. The file is read only when neither the environment +# nor .env supplied a key, and only the selected provider's entry is used. +# With no entry for that provider and no key the tool is off as above. An +# unreadable or invalid file exits 2 naming the problem, never a value. A +# declared source this host cannot run, or that yields no key within +# 5 seconds, is an error outcome ending "nothing sent", with no network or +# quota call, so it is not mistaken for an unconfigured resolver. +# +# Capability probe: --supports-credential-source prints "supported" +# and exits 0, or prints "unsupported" and exits 1, for a source name such as +# macos-keychain on this host. It reads no declaration, retrieves no +# credential, and runs no credential tool, so a configuration installer can +# refuse an unsupported activation before it writes anything. # # What it does when on with at least one rule: one POST with the project name and the brief's # `## Captain's intent` and `## Firstmate spec` sections, tagged when it is a @@ -76,12 +101,14 @@ # clear -> pass the profile line to fm-spawn.sh unless you state a reason to override # ambiguous -> confidence below the floor; decide as today from the probabilities # escalate -> the rule requires captain approval, no candidate is rankable, or a genuine tie -# error -> API, network, response, or quota-axi failure; decide as today +# error -> API, network, response, quota-axi, or declared credential source failure; decide as today # Every outcome exits 0 so an intake is never blocked by this tool. # Exit 2 only for a usage or configuration error (unreadable brief, an -# existing unreadable rules file, malformed rules, missing jq, an unknown -# provider, a selected non-typesafe provider without a model, or systemone -# without a valid base URL), which is actionable, never selected around. +# existing unreadable rules file, malformed rules, an unreadable or invalid +# credential source declaration, missing jq, an unknown provider, a selected +# non-typesafe provider without a model, or systemone without a valid base +# URL), which is actionable, never selected around. +# The capability probe is the one other exit: 1 for an unsupported source. # # Environment: # FM_DISPATCH_PROVIDER, FM_DISPATCH_MODEL, FM_DISPATCH_BASE_URL, @@ -121,6 +148,8 @@ CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" . "$SCRIPT_DIR/fm-timing-lib.sh" # shellcheck source=bin/fm-brief-heading-lib.sh . "$SCRIPT_DIR/fm-brief-heading-lib.sh" +# shellcheck source=bin/fm-credential-source-lib.sh +. "$SCRIPT_DIR/fm-credential-source-lib.sh" CONFIDENCE_FLOOR=0.6 TS_MODEL=jev-latest @@ -138,6 +167,12 @@ no_rules() { printf 'dispatch-resolve:\n status: escalate\n reason: no rules to match\n' exit 0 } +emit_error() { + local reason=$1 + echo "dispatch-resolve: error ($reason)" >&2 + printf 'dispatch-resolve:\n status: error\n reason: %s\n' "$reason" + exit 0 +} usage() { awk ' NR == 1 { next } @@ -148,9 +183,14 @@ usage() { BRIEF='' PROJECT='' RULES_PATH="$CONFIG/crew-dispatch.json" RULES='' NEVER_SEND_PATH="$CONFIG/dispatch-never-send" +CRED_PATH="$CONFIG/dispatch-credential-sources.json" while [ $# -gt 0 ]; do case "$1" in --project) [ $# -ge 2 ] || die "--project needs a value"; PROJECT=$2; shift 2 ;; + --supports-credential-source) + [ $# -ge 2 ] || die "--supports-credential-source needs a value" + if fm_credsrc_supported "$2"; then echo supported; exit 0; fi + echo unsupported; exit 1 ;; -h|--help) usage; exit 0 ;; -*) die "unknown flag $1" ;; *) [ -z "$BRIEF" ] || die "one brief file only"; BRIEF=$1; shift ;; @@ -158,6 +198,47 @@ while [ $# -gt 0 ]; do done # ---- opt-in gate --------------------------------------------------------------- +# declared_key : set the variable from the source +# $CRED_PATH declares for . Returns 1 when none is declared, which +# leaves the off gate in charge. Past that point nothing is mistaken for an +# unconfigured resolver: an unreadable or invalid declaration is a +# configuration error, and a declared source this host cannot run or that +# yields no key is an error outcome. Nothing is sent in either case. +declared_key() { + local provider=$1 src problem value rc + [ -e "$CRED_PATH" ] || [ -L "$CRED_PATH" ] || return 1 + [ -r "$CRED_PATH" ] || die "credential source declaration not readable: $CRED_PATH" + command -v jq >/dev/null 2>&1 || die "jq required" + problem=$(fm_credsrc_check "$CRED_PATH" typesafe openrouter) \ + || die "malformed credential source declaration: $CRED_PATH - $problem" + fm_credsrc_declares "$CRED_PATH" "$provider" || return 1 + src=$(fm_credsrc_field "$CRED_PATH" "$provider" source) + fm_credsrc_supported "$src" \ + || emit_error "credential source $src declared for $provider is not supported on this host; nothing sent" + value=$(fm_credsrc_read "$CRED_PATH" "$provider"); rc=$? + if [ "$rc" -ne 0 ] || [ -z "$value" ]; then + if fm_timed_out "$rc"; then + emit_error "credential source $src declared for $provider did not answer in time; nothing sent" + fi + emit_error "credential source $src declared for $provider is unavailable; nothing sent" + fi + printf -v "$2" '%s' "$value" +} + +# select_key : the selected provider's +# key is the environment copy already in the variable, else KEY_NAME in +# $FM_HOME/.env, else the source declared for the provider. With none of them +# the tool is off, exactly as it was before declarations existed. +select_key() { + local provider=$1 name=$2 var=$3 + [ -n "${!var}" ] || printf -v "$var" '%s' "$(fmx_env_get "$name" "$FM_HOME/.env")" + [ -n "${!var}" ] || declared_key "$provider" "$var" + if [ -z "${!var}" ]; then + echo "dispatch-resolve: off ($name absent from the environment and $FM_HOME/.env)" >&2 + exit 0 + fi +} + DISPATCH_PROVIDER=${FM_DISPATCH_PROVIDER:-} [ -n "$DISPATCH_PROVIDER" ] || DISPATCH_PROVIDER=$(fmx_env_get FM_DISPATCH_PROVIDER "$FM_HOME/.env") DISPATCH_MODEL=${FM_DISPATCH_MODEL:-} @@ -166,27 +247,15 @@ case "$DISPATCH_PROVIDER" in '') DISPATCH_PROVIDER=typesafe DISPATCH_MODEL=$TS_MODEL - [ -n "$TYPESAFE_API_KEY_PRIVATE" ] || TYPESAFE_API_KEY_PRIVATE=$(fmx_env_get TYPESAFE_API_KEY "$FM_HOME/.env") - if [ -z "$TYPESAFE_API_KEY_PRIVATE" ]; then - echo "dispatch-resolve: off (TYPESAFE_API_KEY absent from the environment and $FM_HOME/.env)" >&2 - exit 0 - fi + select_key typesafe TYPESAFE_API_KEY TYPESAFE_API_KEY_PRIVATE ;; typesafe) [ -n "$DISPATCH_MODEL" ] || DISPATCH_MODEL=$TS_MODEL - [ -n "$TYPESAFE_API_KEY_PRIVATE" ] || TYPESAFE_API_KEY_PRIVATE=$(fmx_env_get TYPESAFE_API_KEY "$FM_HOME/.env") - if [ -z "$TYPESAFE_API_KEY_PRIVATE" ]; then - echo "dispatch-resolve: off (TYPESAFE_API_KEY absent from the environment and $FM_HOME/.env)" >&2 - exit 0 - fi + select_key typesafe TYPESAFE_API_KEY TYPESAFE_API_KEY_PRIVATE ;; openrouter) [ -n "$DISPATCH_MODEL" ] || die "FM_DISPATCH_MODEL required when FM_DISPATCH_PROVIDER=openrouter" - [ -n "$OPENROUTER_API_KEY_PRIVATE" ] || OPENROUTER_API_KEY_PRIVATE=$(fmx_env_get OPENROUTER_API_KEY "$FM_HOME/.env") - if [ -z "$OPENROUTER_API_KEY_PRIVATE" ]; then - echo "dispatch-resolve: off (OPENROUTER_API_KEY absent from the environment and $FM_HOME/.env)" >&2 - exit 0 - fi + select_key openrouter OPENROUTER_API_KEY OPENROUTER_API_KEY_PRIVATE ;; systemone) [ -n "$DISPATCH_MODEL" ] || die "FM_DISPATCH_MODEL required when FM_DISPATCH_PROVIDER=systemone" @@ -322,13 +391,6 @@ done < <(jq -r ' RULE_COUNT=$(jq -r '(.rules // []) | length' "$RULES") -emit_error() { - local reason=$1 - echo "dispatch-resolve: error ($reason)" >&2 - printf 'dispatch-resolve:\n status: error\n reason: %s\n' "$reason" - exit 0 -} - if [ "$RULE_COUNT" -eq 0 ]; then no_rules fi diff --git a/docs/configuration.md b/docs/configuration.md index aa3d61c142b..750be788a21 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1826,9 +1826,10 @@ See [`docs/examples/crew-dispatch.json`](examples/crew-dispatch.json) for a star - When the file exists, bootstrap validates it with `jq`. - Valid files stay silent by default; with `FM_BOOTSTRAP_VERBOSE_FACTS=1`, bootstrap emits `BOOTSTRAP_INFO: crew dispatch active config/crew-dispatch.json`, one `BOOTSTRAP_INFO:` fact per rule, and one fact for the optional default profile set. - Malformed JSON, malformed rules, an empty or malformed profile array, a malformed `preferred_provider`, an unverified harness, or an effort value unsupported by that harness is reported as `CREW_DISPATCH: invalid config/crew-dispatch.json - ...`. -- Bootstrap enables resolver-only checks for `approval`, `min_confidence`, `floor`, and present `provider` declarations when `TYPESAFE_API_KEY` is non-empty in the environment or the home's `.env`, regardless of `FM_DISPATCH_PROVIDER`. - An OpenRouter key alone does not enable these bootstrap checks; the resolver validates those declarations itself before any request. - Without a typesafe key, bootstrap retains its pre-existing verified-harness baseline. +- Bootstrap enables resolver-only checks for `approval`, `min_confidence`, `floor`, and present `provider` declarations when `TYPESAFE_API_KEY` is non-empty in the environment or the home's `.env`, or when the home's valid `config/dispatch-credential-sources.json` names a `typesafe` source, regardless of `FM_DISPATCH_PROVIDER`. + Bootstrap only checks that declaration; it never reads the key or runs the source's tool. + An OpenRouter key or OpenRouter declaration alone does not enable these bootstrap checks; the resolver validates those declarations itself before any request. + Without a typesafe key or a declared typesafe source, bootstrap retains its pre-existing verified-harness baseline. - Missing `jq` is reported through the normal `MISSING: jq` install-consent flow. - While the file remains present, no crewmate or scout spawn may proceed without an explicit resolved harness; malformed configuration must be reported and corrected rather than selected around. @@ -1843,11 +1844,12 @@ Secondmate homes inherit this file from the primary, so a secondmate's own crewm Each setting comes from the calling environment or the home's gitignored `.env` through `bin/fm-env-lib.sh`; a non-empty environment value wins. OpenRouter, `systemone`, and OpenAI each require their explicit `FM_DISPATCH_PROVIDER` value and a non-empty `FM_DISPATCH_MODEL`; an omitted model is an actionable configuration error, and the presence of a provider's key never selects that provider by itself. `systemone` also requires `FM_DISPATCH_BASE_URL`, an `http://` or `https://` base such as a local llama.cpp server or a hosted System One-compatible service, and posts to `/v1/systemone`; a missing or scheme-less base URL is an actionable configuration error. -With no provider setting, the original typesafe.ai behavior is unchanged: `jev-latest` is used when `TYPESAFE_API_KEY` is available, `FM_DISPATCH_MODEL` is ignored, and the resolver is off otherwise. +With no provider setting, the original typesafe.ai behavior is unchanged: `jev-latest` is used when `TYPESAFE_API_KEY` is available, `FM_DISPATCH_MODEL` is ignored, and the resolver is off when no key is available and no `typesafe` credential source is declared. Explicit `FM_DISPATCH_PROVIDER=typesafe` also uses `jev-latest` when no model is set. -The selected provider's key must be non-empty in the environment or `.env`, with the environment taking precedence; `systemone` is the exception, where an absent `FM_DISPATCH_API_KEY` means the request carries no `Authorization` header. +The selected provider needs a non-empty key from the environment, else `.env`, else, for `typesafe` and `openrouter` only, the credential source the home declares for that provider; `systemone` is the exception, where an absent `FM_DISPATCH_API_KEY` means the request carries no `Authorization` header. Off means one `dispatch-resolve: off` line on stderr, nothing on stdout, exit 0, and no network call, so firstmate dispatches exactly as it does without the tool. +A declared credential source that cannot supply its key is not off; see "Credential source declaration" below. This section is the single owner of the tool's operator contract; the script header owns its exact flags and output lines, and "Crew dispatch profiles" above owns the declared top-level, rule, and profile fields it applies. Rules come only from the effective home's `config/crew-dispatch.json`; `FM_CONFIG_OVERRIDE` selects the config directory for tests and specialized setup like the other scripts. @@ -1941,12 +1943,12 @@ No qualifying option, or two equally probable qualifying options, produces `ambi | `clear` | A `profile:` line ready for `fm-spawn.sh`. | | `ambiguous` | Confidence below the floor with no runner-up taken. | | `escalate` | An approval-gated rule, unverifiable rule floor, nothing rankable, or a genuine tie. | -| `error` | API, network, malformed response metadata, rendering, or quota-axi failure. | +| `error` | API, network, malformed response metadata, rendering, quota-axi failure, or a declared credential source that cannot supply its key. | Every result above exits 0. - Response probabilities must contain exactly every offered choice, use numeric values from 0 through 1, and sum to approximately 1 within 0.01. -- Only a usage or configuration error exits 2: an unreadable brief, an existing but unreadable or malformed canonical rules file, a missing `jq`, an unknown provider, a non-typesafe provider without a model, or `systemone` without a valid base URL, each reported and never selected around. +- Only a usage or configuration error exits 2: an unreadable brief, an existing but unreadable or malformed canonical rules file, an unreadable or invalid credential source declaration, a missing `jq`, an unknown provider, a non-typesafe provider without a model, or `systemone` without a valid base URL, each reported and never selected around. - Missing `curl` is a normal structured `error` outcome with exit 0 so firstmate uses today's routing. **Firstmate retains the dispatch decision** @@ -1958,9 +1960,57 @@ Explicit captain overrides retain precedence over the resolver's suggestion. Firstmate passes its profile line unless it states a reason to override, such as the brief's reasoning class or an eligible-unranked-candidate note; every non-clear result returns to the full existing intake. +**Credential source declaration (config/dispatch-credential-sources.json)** + +The optional local, gitignored `config/dispatch-credential-sources.json` tells the resolver where a provider's key lives when neither the environment nor `.env` supplies one. +It names a source and never holds a key. +It is read from the effective config directory, which is `FM_CONFIG_OVERRIDE` when set and `$FM_HOME/config` otherwise, so each home reads only its own file. +It is not among the settings `bin/fm-config-inherit-lib.sh` propagates, so a secondmate home that wants a declared source keeps its own file. +A home without the file behaves exactly as it did before the file existed. +It covers the `typesafe` and `openrouter` keys only; the `systemone` and OpenAI keys still come from the environment or `.env`. + +The file is one JSON object keyed by provider, `typesafe` or `openrouter`, and each value says where that provider's key is: + +```json +{ + "typesafe": { "source": "macos-keychain", "service": "typesafe-api-key", "account": "me" } +} +``` + +`source` is required and names the adapter, and `macos-keychain` is the first. +It requires `service`, accepts an optional `account`, and reads that generic-password item with `security find-generic-password -s [-a ] -w`. +Store the item with `security add-generic-password -s [-a ] -w`, which prompts for the key and lets `security` read it back without an access dialog. +Only the service and account reach `security`'s argument list, and the key comes back on a pipe. +No other field is accepted, so the file has nowhere to put a key. +`bin/fm-credential-source-lib.sh` owns the schema and the adapters, so a further source adds an adapter there rather than a special case in the resolver. + +The resolver reads the file only after the environment and `.env` have supplied no key for the selected provider, and it uses only that provider's entry. +A declaration for one provider never feeds another provider and never selects a provider. +Three outcomes stay distinct: + +- **Off.** No entry for the selected provider and no key keeps today's behavior: the `dispatch-resolve: off` line, nothing on stdout, and exit 0. +- **Configuration error.** An unreadable file, malformed JSON, an unknown provider, an unknown or missing field, or a value that is not a plain non-empty string exits 2 with one line that names the problem. + The check covers the whole file, so an invalid entry for a provider that is not selected also exits 2. + The line names providers and fields and never prints a declared value. +- **Unavailable source.** An entry whose source this host cannot run, or that yields no key, ends as the `error` outcome with exit 0 and a reason ending `nothing sent`, and makes no network or quota call. + The reason says whether the host cannot run the source, the source holds no readable key, or it did not answer in time, and it never relays the tool's own output. + A source that does not answer within 5 seconds ends the same way, so an unanswered Keychain access prompt cannot hold an intake. + +A configuration installer can check a source before it writes anything: + +```sh +bin/fm-dispatch-resolve.sh --supports-credential-source macos-keychain +``` + +The probe prints `supported` and exits 0, or prints `unsupported` and exits 1, for a source name on the host it runs on. +`unsupported` covers an unknown name, a host the adapter does not run on, and a missing tool. +The probe reads no declaration and no credential and runs no credential tool. +An installer should treat any status other than 0 as a refusal, because a resolver that predates the probe rejects the flag with exit 2. + **Key handling and provider settings** - The resolver copies every environment-provided key into a non-exported private variable and unsets `TYPESAFE_API_KEY`, `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, and `FM_DISPATCH_API_KEY` before launching child processes; bootstrap also protects the typesafe key. +- A key read from a declared source is captured into the same private variable, so it is never exported either. - The resolver sends only the selected key to `curl` as a header read from a file descriptor, never on argv, and never prints, logs, or writes any key. - The typesafe.ai endpoint is `https://api.typesafe.ai/v1/systemone`, the OpenRouter endpoint is `https://openrouter.ai/api/alpha/decisions`, the OpenAI endpoint is `https://api.openai.com/v1/decisions`, and the `systemone` endpoint is `/v1/systemone`. - The default confidence floor remains 0.6; request timeouts are 5 seconds for typesafe.ai and 10 seconds for OpenRouter, `systemone`, and OpenAI. @@ -3205,8 +3255,8 @@ FM_DISPATCH_PROVIDER= # typesafe, openrouter, systemone, or openai; absent pre FM_DISPATCH_MODEL= # explicit model for the selected provider; required for openrouter, systemone, and openai FM_DISPATCH_BASE_URL= # http(s) base of a System One-compatible host; required for systemone FM_DISPATCH_API_KEY= # optional systemone key; absent sends no Authorization header -TYPESAFE_API_KEY= # typesafe dispatch key, from the environment or .env -OPENROUTER_API_KEY= # OpenRouter dispatch key, from the environment or .env +TYPESAFE_API_KEY= # typesafe dispatch key, from the environment, .env, or the declared credential source ("Typed dispatch resolution") +OPENROUTER_API_KEY= # OpenRouter dispatch key, from the environment, .env, or the declared credential source OPENAI_API_KEY= # OpenAI dispatch key, from the environment or .env FMX_DISCORD_REPLY_MAX_CHARS=1900 # Discord reply per-message split budget; values below 50 clamp to 50, values above 2000 reset to 1900 FMX_X_THREAD_MAX=25 # maximum messages in one auto-split reply thread diff --git a/docs/scripts.md b/docs/scripts.md index be9c93f39d0..5f2c29d99ec 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -124,6 +124,7 @@ The shared no-mistakes gate lifecycle boundary is summarized in [architecture.md | `fm-tasks-axi-lib.sh` | Shared backlog-backend selector and `tasks-axi` compatibility probe | | `fm-backlog-transition-lib.sh` | Pair task-record changes with their backlog transitions and replay interrupted closes | | `fm-quota-axi-lib.sh` | Shared `quota-axi` compatibility floor and quota snapshot schema validation | +| `fm-credential-source-lib.sh` | Single owner of the declared credential-source schema and its adapters, read by `fm-dispatch-resolve.sh` and checked by `fm-bootstrap.sh` | | `fm-quota-choose.sh` | Choose the first candidate with known positive quota from an ordered harness:model list | | `fm-vendor-auth-probe.sh`| Run one hard-bounded, non-destructive authentication probe of a named vendor CLI and report the fact | | `fm-wake-drain.sh` | Present and acknowledge the current actor's claimed wake rows alongside status, outcome-backstop, decision, divergence, supervision-host outcome, recovery, and supervision checks | diff --git a/docs/verification/dispatch-resolve.md b/docs/verification/dispatch-resolve.md index 45ab92dcf11..19b4409dd4d 100644 --- a/docs/verification/dispatch-resolve.md +++ b/docs/verification/dispatch-resolve.md @@ -302,12 +302,19 @@ It proves the key is absent from child environments, never appears on `curl` arg It proves the default typesafe.ai request keeps its fixed endpoint and model, carries only the project, the brief's task sections read by the shared brief-heading parser with a scout line only for a scout brief and never a ship brief's delivery mode (or the whole brief when it has neither section), and rule Choice with one option per rule plus the fixed neutral none option, and never carries `why`, `use`, or quota. It also proves OpenRouter requires an explicit provider and model, accepts the environment or `.env` key, sends the same Choice request to the Decisions API, preserves clear and ambiguous output, withholds a never-send match, and handles malformed responses and HTTP errors. The fake `curl` and `quota-axi` prove that the OpenRouter key stays out of their argv and environments. +It proves the credential source declaration with a fake `security` standing in for the macOS Keychain and a fake `uname`, so no case reads the real Keychain. +The declared service and optional account become the whole lookup, the environment and then `.env` win before the declared source is read, and an invalid declaration is not read at all when a key is already present. +Each provider reads only its own entry, a declaration never selects a provider, and only the effective config directory's declaration is consulted, whether the other one belongs to another home or sits under `FM_HOME/config` while `FM_CONFIG_OVERRIDE` selects a different directory. +An absent entry with no key stays off, and every invalid shape exits 2 with one line that names the problem and never a declared value. +A missing or empty Keychain item, a Keychain that refuses, a host that is not macOS, a missing tool, and a source that never answers each end as an `error` outcome with exit 0 that makes no network or quota call, and the unanswered read is stopped rather than left running. +The fake `curl` and `quota-axi` also prove the declared key appears in no child environment and in no file the tool holds while the request is in flight, and the key is on no argv, output, or request body. +The capability probe answers `supported` or `unsupported` without reading a declaration or running `security`. It proves that with no provider set, other providers' settings leave the original off line unchanged byte for byte, with no network or quota call. It proves `systemone` posts the unchanged System One request to `/v1/systemone`, sends no `Authorization` header and opens no descriptor without a key, carries a key only on the descriptor, reads its settings from `.env`, exits 2 for a missing model or a missing or scheme-less base URL, withholds a never-send base URL, and returns partial responses and transport failure as `error`. It proves `openai` is off with one line without its key, exits 2 without a model, sends the same state as JSON text `input` with one named choice question carrying every option, maps the probability array back, withholds a never-send match inside the serialized input or the model name, and rejects refusals, missing or duplicated answers and option values, missing options or confidence, and non-JSON bodies before quota is read. It proves a declared `min_confidence` is checked against the rule's own probability both as the pick and as a runner-up, a picked rule below it falls to the most probable runner-up that clears its floor, is `ambiguous` when none does or two tie, and that a file without declared floors keeps the global 0.6 floor on confidence unchanged. It proves the clear, fixed-floor ambiguous with candidate evidence, escalate (approval with candidate evidence, unverifiable rule floor, tie, nothing rankable), known rule-floor fall-through, known and unverifiable profile-floor evidence, explicit-provider and provider-ID enforcement, authoritative Agy and explicit-provider Gemini routing, partial providers, eligible unranked candidates and their clear-result note, concrete quota vetoes and profile-floor shortfalls taking precedence over uncertainty, account-wide quota veto, limiting-bound ranking, schema-6 account-row binding with schema-5 compatibility, `preferred_provider` family preference across accounts with fallback only on proven ineligibility and escalation on unknown preferred quota, the `shared_quota` opt-out (declared-order pick with no `quota-axi` call, unmet rule- and profile-floor bypass, `preferred_provider` narrowing, the quota-ranking-off candidate line, `false` keeping the quota-ranked path, and non-boolean rejection), missing-curl and quota-axi failures, HTTP 429 and 500, transport failure, malformed usage, zero-mass or malformed probabilities or confidence, malformed or duplicate profile, invalid selector, removed-option rejection, and out-of-range rule ID paths behave as the contract states, with configuration errors exiting 2 before any network call. -`tests/fm-bootstrap.test.sh` covers bootstrap's typesafe-key validation gate, its key-independent `preferred_provider` and `shared_quota` checks, and child-environment protection; [configuration](../configuration.md#crew-dispatch-profiles-configcrew-dispatchjson) owns that gate's operator contract. +`tests/fm-bootstrap.test.sh` covers bootstrap's typesafe-key and declared-typesafe-source validation gate, which never runs `security`, its key-independent `preferred_provider` and `shared_quota` checks, and child-environment protection; [configuration](../configuration.md#crew-dispatch-profiles-configcrew-dispatchjson) owns that gate's operator contract. ```console $ bash tests/fm-dispatch-resolve.test.sh | tail -1 diff --git a/tests/fm-bootstrap.test.sh b/tests/fm-bootstrap.test.sh index 3fc5cee118a..915f98fb580 100755 --- a/tests/fm-bootstrap.test.sh +++ b/tests/fm-bootstrap.test.sh @@ -1361,7 +1361,7 @@ test_preferred_provider_validation() { } test_crew_dispatch_validation() { - local label body expect mode case_dir fakebin out child_env n + local label body expect mode case_dir fakebin out child_env n cred declaration n=0 while IFS='^' read -r label body mode expect; do [ -n "$label" ] || continue @@ -1495,6 +1495,35 @@ ROWS [ -z "$out" ] || fail "typed resolution should add verified Gemini crewmate routing, got: $out" rm -f "$case_dir/home/.env" + cat > "$fakebin/security" <<'SH' +#!/usr/bin/env bash +printf '%s\n' "$*" >> "${FM_TEST_SECURITY_LOG:?}" +printf '%s\n' declared-source-secret +SH + chmod +x "$fakebin/security" + : > "$case_dir/security.log" + cred="$case_dir/home/config/dispatch-credential-sources.json" + printf '%s\n' '{"typesafe":{"source":"macos-keychain","service":"typesafe-api-key"}}' > "$cred" + out=$(PATH="$fakebin:$BASE_PATH" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ + FM_TEST_SECURITY_LOG="$case_dir/security.log" FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + [ -z "$out" ] || fail "a declared typesafe source should add verified Gemini crewmate routing, got: $out" + printf '%s\n' '{"rules":[{"when":"hard design","approval":"Captain","use":{"harness":"claude"}}]}' > "$case_dir/home/config/crew-dispatch.json" + out=$(PATH="$fakebin:$BASE_PATH" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ + FM_TEST_SECURITY_LOG="$case_dir/security.log" FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + [ "$out" = 'CREW_DISPATCH: invalid config/crew-dispatch.json - approval must be "captain" when present' ] \ + || fail "a declared typesafe source must activate resolver-field validation, got: $out" + printf '%s\n' '{"rules":[{"when":"gemini work","use":{"harness":"gemini","model":"gemini-3.8-flash-high","provider":"google"}}]}' > "$case_dir/home/config/crew-dispatch.json" + for declaration in \ + '{"openrouter":{"source":"macos-keychain","service":"openrouter-api-key"}}' \ + '{"typesafe":{"source":"macos-keychain","service":"typesafe-api-key","key":"inline"}}'; do + printf '%s\n' "$declaration" > "$cred" + out=$(PATH="$fakebin:$BASE_PATH" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ + FM_TEST_SECURITY_LOG="$case_dir/security.log" FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + [ "$out" = 'CREW_DISPATCH: invalid config/crew-dispatch.json - unverified harness: gemini' ] \ + || fail "a declaration that is not a valid typesafe source must keep the verified-harness baseline ($declaration), got: $out" + done + [ ! -s "$case_dir/security.log" ] || fail "bootstrap must never run the credential tool, got: $(cat "$case_dir/security.log")" + rm -f "$cred" : > "$case_dir/child-env.log" out=$(PATH="$fakebin:$BASE_PATH" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ TYPESAFE_API_KEY=test-key FM_TEST_CHILD_ENV_LOG="$case_dir/child-env.log" \ @@ -1503,7 +1532,7 @@ ROWS child_env=$(cat "$case_dir/child-env.log") [ -n "$child_env" ] || fail "bootstrap child environment probe did not run" assert_not_contains "$child_env" 'secret-present' "bootstrap children never inherit the typesafe key" - pass "bootstrap gates resolver fields and additive harnesses on the typed key" + pass "bootstrap gates resolver fields and additive harnesses on the typed key or a declared typesafe source" } test_bootstrap_reporting diff --git a/tests/fm-dispatch-resolve.test.sh b/tests/fm-dispatch-resolve.test.sh index bb054db558f..7dd60ad9323 100755 --- a/tests/fm-dispatch-resolve.test.sh +++ b/tests/fm-dispatch-resolve.test.sh @@ -4,9 +4,10 @@ # Drives the public argv and environment interface with a fake curl on PATH # that records argv, the request body it read from stdin, and the header it # read from file descriptor 3, and answers with a canned provider response. -# A fake quota-axi serves the selected schema-5 fixture. No case touches the -# network, and the absent-key case proves the tool makes no call -# at all. +# A fake quota-axi serves the selected schema-5 fixture. A fake security and a +# fake uname stand in for the macOS Keychain, so no case reads the real one. +# No case touches the network, and the absent-key case proves the tool makes +# no call at all. set -u # shellcheck source=tests/lib.sh @@ -17,16 +18,27 @@ TMP_ROOT=$(fm_test_tmproot fm-dispatch-resolve) HOME_DIR="$TMP_ROOT/home" FAKEBIN=$(fm_fakebin "$TMP_ROOT") NO_CURL_BIN="$TMP_ROOT/no-curl-bin" +NO_SECURITY_BIN="$TMP_ROOT/no-security-bin" +NO_JQ_BIN="$TMP_ROOT/no-jq-bin" LOG="$TMP_ROOT/log" BRIEF="$TMP_ROOT/brief.md" BASE_RULES="$TMP_ROOT/rules.json" RULES="$HOME_DIR/config/crew-dispatch.json" QUOTA="$TMP_ROOT/quota.json" BASE_PATH=$PATH -mkdir -p "$HOME_DIR/config" "$LOG" "$NO_CURL_BIN" +mkdir -p "$HOME_DIR/config" "$LOG" "$NO_CURL_BIN" "$NO_SECURITY_BIN" "$NO_JQ_BIN" for command_name in bash chmod cp dirname jq mktemp rm; do ln -s "$(command -v "$command_name")" "$NO_CURL_BIN/$command_name" + ln -s "$(command -v "$command_name")" "$NO_SECURITY_BIN/$command_name" done +# The minimal PATHs that simulate one missing command: no security, or no jq. +# Both keep the fake macOS uname, and the no-jq one keeps the fake security. +for command_name in bash chmod cp dirname mktemp rm; do + ln -s "$(command -v "$command_name")" "$NO_JQ_BIN/$command_name" +done +ln -s "$FAKEBIN/uname" "$NO_SECURITY_BIN/uname" +ln -s "$FAKEBIN/uname" "$NO_JQ_BIN/uname" +ln -s "$FAKEBIN/security" "$NO_JQ_BIN/security" cat > "$BRIEF" <<'MD' # Task @@ -119,6 +131,16 @@ if [ -n "${TYPESAFE_API_KEY+x}" ] || [ -n "${TYPESAFE_API_KEY_PRIVATE+x}" ] || else printf 'curl:clean\n' >> "${CHILD_ENV_LOG:?}" fi +# With a canary file named, also look for the canary's VALUE in this process's +# environment and in the directories the test lists, where the tool's own +# files live while the request is in flight. +if [ -n "${FAKE_CANARY_FILE:-}" ]; then + canary=$(cat "$FAKE_CANARY_FILE") + env | grep -qF -- "$canary" && printf 'curl:canary-in-env\n' >> "${CHILD_ENV_LOG:?}" + for dir in ${FAKE_CANARY_DIRS:-}; do + grep -rqF -- "$canary" "$dir" 2>/dev/null && printf 'curl:canary-in-file\n' >> "${CHILD_ENV_LOG:?}" + done +fi out='' while [ $# -gt 0 ]; do case "$1" in @@ -151,6 +173,9 @@ if [ -n "${TYPESAFE_API_KEY+x}" ] || [ -n "${TYPESAFE_API_KEY_PRIVATE+x}" ] || else printf 'quota-axi:clean\n' >> "${CHILD_ENV_LOG:?}" fi +if [ -n "${FAKE_CANARY_FILE:-}" ] && env | grep -qF -- "$(cat "$FAKE_CANARY_FILE")"; then + printf 'quota-axi:canary-in-env\n' >> "${CHILD_ENV_LOG:?}" +fi printf '%s\n' "$*" >> "${QUOTA_AXI_CALLS:?}" [ "${FAKE_QUOTA_FAIL:-0}" = 1 ] && exit 1 [ "${1:-}" = --json ] || exit 2 @@ -158,8 +183,60 @@ cat "${QUOTA_AXI_FIXTURE:?}" SH chmod +x "$FAKEBIN/quota-axi" +cat > "$FAKEBIN/security" <<'SH' +#!/usr/bin/env bash +# Fake security: the Keychain stand-in. It answers only an exact +# `find-generic-password -s [-a ] -w` query, from the item +# file FAKE_SECURITY_STORE/, and records argv and whether a key +# variable reached its environment. FAKE_SECURITY_EXIT fails the query after +# printing the item to stdout and stderr as a decoy, and FAKE_SECURITY_HANG +# names a file to receive its pid before it stops answering. +set -u +printf '%s\n' "$*" >> "${FAKE_SECURITY_LOG:?}" +if [ -n "${TYPESAFE_API_KEY+x}" ] || [ -n "${TYPESAFE_API_KEY_PRIVATE+x}" ] || + [ -n "${OPENROUTER_API_KEY+x}" ] || [ -n "${OPENROUTER_API_KEY_PRIVATE+x}" ]; then + printf 'security:secret-present\n' >> "${CHILD_ENV_LOG:?}" +else + printf 'security:clean\n' >> "${CHILD_ENV_LOG:?}" +fi +[ "${1:-}" = find-generic-password ] || exit 2 +shift +service='' wants_password=0 +while [ $# -gt 0 ]; do + case "$1" in + -s) [ $# -ge 2 ] || exit 2; service=$2; shift 2 ;; + -a) { [ $# -ge 2 ] && [ -n "$2" ]; } || exit 2; shift 2 ;; + -w) wants_password=1; shift ;; + *) exit 2 ;; + esac +done +[ -n "$service" ] && [ "$wants_password" = 1 ] || exit 2 +if [ -n "${FAKE_SECURITY_HANG:-}" ]; then + printf '%s\n' "$$" > "$FAKE_SECURITY_HANG" + exec sleep 60 +fi +item="${FAKE_SECURITY_STORE:?}/$service" +if [ -n "${FAKE_SECURITY_EXIT:-}" ]; then + if [ -r "$item" ]; then + cat "$item" + printf 'security: could not use the item holding %s\n' "$(cat "$item")" >&2 + fi + exit "$FAKE_SECURITY_EXIT" +fi +[ -r "$item" ] || exit 44 +cat "$item" +SH +chmod +x "$FAKEBIN/security" + +cat > "$FAKEBIN/uname" <<'SH' +#!/usr/bin/env bash +# Fake uname: the host is macOS unless FAKE_UNAME names another system. +printf '%s\n' "${FAKE_UNAME:-Darwin}" +SH +chmod +x "$FAKEBIN/uname" + RESPONSE="$TMP_ROOT/response.json" -export FAKE_CURL_LOG="$LOG" FAKE_CURL_RESPONSE="$RESPONSE" QUOTA_AXI_CALLS="$LOG/quota-axi.calls" QUOTA_AXI_FIXTURE="$QUOTA" CHILD_ENV_LOG="$LOG/child-env" +export FAKE_CURL_LOG="$LOG" FAKE_CURL_RESPONSE="$RESPONSE" QUOTA_AXI_CALLS="$LOG/quota-axi.calls" QUOTA_AXI_FIXTURE="$QUOTA" CHILD_ENV_LOG="$LOG/child-env" FAKE_SECURITY_LOG="$LOG/security.argv" reset_log() { rm -rf "$LOG" @@ -188,6 +265,18 @@ run_without_curl() { printf -v "$__err" '%s' "$(cat "$TMP_ROOT/stderr")" } +# run_on_path [args...]: the tool on a +# caller-built PATH, for the cases that simulate a missing command. +run_on_path() { + local __path=$1 __exit=$2 __out=$3 __err=$4 _out _code + shift 4 + _out=$(PATH="$__path" FM_HOME="$HOME_DIR" "$TOOL" "$@" 2> "$TMP_ROOT/stderr") + _code=$? + printf -v "$__exit" '%s' "$_code" + printf -v "$__out" '%s' "$_out" + printf -v "$__err" '%s' "$(cat "$TMP_ROOT/stderr")" +} + KEY='test-key-9f1c2d3e-never-on-argv' code='' out='' err='' @@ -437,6 +526,292 @@ expect_code 2 "$code" "unknown provider is a configuration error" assert_contains "$err" 'unknown FM_DISPATCH_PROVIDER' "unknown provider is actionable" pass "OpenRouter ambiguous, malformed, HTTP, and missing-model paths are handled" +# --- credential source declaration: where a key lives, never the key ----------- +# $CONFIG/dispatch-credential-sources.json names a source per provider. It is +# read only when neither the environment nor .env supplied a key. The fake +# Keychain holds one item file per service name. +SECRET='kc-typesafe-4d7a1e90-never-on-argv' +KC_OR_SECRET='kc-openrouter-8b21f6c3-never-on-argv' +EFFECTIVE_SECRET='kc-effective-c07e5a12-never-on-argv' +STORE="$TMP_ROOT/keychain" +CRED="$HOME_DIR/config/dispatch-credential-sources.json" +mkdir -p "$STORE" +printf '%s\n' "$SECRET" > "$STORE/typesafe-svc" +printf '%s\n' "$KC_OR_SECRET" > "$STORE/openrouter-svc" +printf '%s\n' "$EFFECTIVE_SECRET" > "$STORE/effective-svc" +: > "$STORE/empty-svc" +export FAKE_SECURITY_STORE="$STORE" + +# declare_source []: add or replace one +# provider's macos-keychain entry, keeping the other providers' entries. +declare_source() { + local file=$1 provider=$2 service=$3 account=${4:-} + [ -f "$file" ] || printf '{}\n' > "$file" + jq --arg p "$provider" --arg s "$service" --arg a "$account" \ + '.[$p] = ({source: "macos-keychain", service: $s} + (if $a == "" then {} else {account: $a} end))' \ + "$file" > "$file.next" && mv "$file.next" "$file" +} + +# mapping: the declared service and optional account become the lookup +reset_log +write_response "$RESPONSE" rule_4 0.9 +declare_source "$CRED" typesafe typesafe-svc mike +run code out err "$BRIEF" --project pager +expect_code 0 "$code" "a declared source resolves" +assert_contains "$out" ' status: clear' "a declared source turns the tool on" +assert_equals 'find-generic-password -s typesafe-svc -a mike -w' "$(cat "$LOG/security.argv")" "the declared service and account are the whole Keychain lookup" +assert_equals "Authorization: Bearer $SECRET" "$(cat "$LOG/header")" "the declared key reaches curl on the fd header" +rm -f "$CRED" +declare_source "$CRED" typesafe typesafe-svc +reset_log +run code out err "$BRIEF" --project pager +assert_contains "$out" ' status: clear' "a declaration without an account resolves" +assert_equals 'find-generic-password -s typesafe-svc -w' "$(cat "$LOG/security.argv")" "an omitted account is omitted from the lookup" +pass "mapping: a declared macos-keychain service and optional account become the lookup, and the key rides the fd header" + +# precedence: environment, then .env, then the declared source +reset_log +TYPESAFE_API_KEY=env-wins run code out err "$BRIEF" --project pager +assert_equals 'Authorization: Bearer env-wins' "$(cat "$LOG/header")" "the environment key wins over a declared source" +assert_absent "$LOG/security.argv" "the environment key wins before the declared source is read" +printf '%s\n' 'TYPESAFE_API_KEY=dotenv-wins' > "$HOME_DIR/.env" +reset_log +run code out err "$BRIEF" --project pager +assert_equals 'Authorization: Bearer dotenv-wins' "$(cat "$LOG/header")" ".env key wins over a declared source" +assert_absent "$LOG/security.argv" ".env key wins before the declared source is read" +rm -f "$HOME_DIR/.env" "$CRED" +printf '%s\n' '{"typesafe":' > "$CRED" +reset_log +TYPESAFE_API_KEY=$KEY run code out err "$BRIEF" --project pager +expect_code 0 "$code" "a key from the environment never consults an invalid declaration" +assert_contains "$out" ' status: clear' "an unread declaration cannot break a configured resolver" +assert_absent "$LOG/security.argv" "an invalid declaration is not read when the environment has the key" +rm -f "$CRED" +pass "precedence: environment, then .env, then the declared source, which is read only when needed" + +# provider isolation: each provider reads only its own entry, and an entry never selects a provider +rm -f "$CRED" +declare_source "$CRED" typesafe typesafe-svc +reset_log +write_openrouter_response rule_4 0.9 +FM_DISPATCH_PROVIDER=openrouter FM_DISPATCH_MODEL=typesafe/jev-1.13 FAKE_CURL_RESPONSE=$OR_RESPONSE \ + run code out err "$BRIEF" --project pager +expect_code 0 "$code" "a typesafe declaration leaves OpenRouter off" +assert_equals '' "$out" "an OpenRouter run never uses a typesafe declaration" +assert_contains "$err" 'dispatch-resolve: off (OPENROUTER_API_KEY absent from the environment and' "OpenRouter without its own declaration is off" +assert_absent "$LOG/security.argv" "a typesafe declaration is never read for OpenRouter" +assert_absent "$LOG/argv" "OpenRouter without a key never calls curl" +rm -f "$CRED" +declare_source "$CRED" openrouter openrouter-svc +reset_log +run code out err "$BRIEF" --project pager +assert_equals '' "$out" "an OpenRouter declaration neither selects OpenRouter nor feeds typesafe" +assert_contains "$err" 'dispatch-resolve: off (TYPESAFE_API_KEY absent from the environment and' "typesafe without its own declaration is off" +assert_absent "$LOG/security.argv" "an OpenRouter declaration is never read for typesafe" +declare_source "$CRED" typesafe typesafe-svc +reset_log +FM_DISPATCH_PROVIDER=openrouter FM_DISPATCH_MODEL=typesafe/jev-1.13 FAKE_CURL_RESPONSE=$OR_RESPONSE \ + run code out err "$BRIEF" --project pager +assert_contains "$out" ' status: clear' "OpenRouter resolves from its own declared source" +assert_equals 'find-generic-password -s openrouter-svc -w' "$(cat "$LOG/security.argv")" "OpenRouter reads only its own declared service" +assert_equals "Authorization: Bearer $KC_OR_SECRET" "$(cat "$LOG/header")" "OpenRouter's declared key reaches curl, not typesafe's" +assert_contains "$(cat "$LOG/argv")" 'https://openrouter.ai/api/alpha/decisions' "OpenRouter's declared key goes to OpenRouter's endpoint" +reset_log +write_response "$RESPONSE" rule_4 0.9 +FM_DISPATCH_PROVIDER=typesafe run code out err "$BRIEF" --project pager +assert_equals 'find-generic-password -s typesafe-svc -w' "$(cat "$LOG/security.argv")" "typesafe reads only its own declared service" +assert_equals "Authorization: Bearer $SECRET" "$(cat "$LOG/header")" "typesafe's declared key reaches curl, not OpenRouter's" +assert_contains "$(cat "$LOG/argv")" 'https://api.typesafe.ai/v1/systemone' "typesafe's declared key goes to the typesafe.ai endpoint" +rm -f "$CRED" +pass "provider isolation: each provider reads only its own declared service, and a declaration never selects a provider" + +# effective-home isolation: the effective config directory's declaration, and no other +declare_source "$CRED" typesafe typesafe-svc +OTHER_HOME="$TMP_ROOT/other-home" +mkdir -p "$OTHER_HOME/config" +cp "$BASE_RULES" "$OTHER_HOME/config/crew-dispatch.json" +PRIMARY_HOME=$HOME_DIR +HOME_DIR=$OTHER_HOME +reset_log +run code out err "$BRIEF" --project pager +HOME_DIR=$PRIMARY_HOME +assert_equals '' "$out" "a declaration in another home is not consulted" +assert_contains "$err" "dispatch-resolve: off (TYPESAFE_API_KEY absent from the environment and $OTHER_HOME/.env)" "a home with no declaration is off" +assert_absent "$LOG/security.argv" "another home's declaration is never read" +EFFECTIVE_CONFIG="$TMP_ROOT/effective-config" +mkdir -p "$EFFECTIVE_CONFIG" +cp "$BASE_RULES" "$EFFECTIVE_CONFIG/crew-dispatch.json" +declare_source "$EFFECTIVE_CONFIG/dispatch-credential-sources.json" typesafe effective-svc +reset_log +FM_CONFIG_OVERRIDE=$EFFECTIVE_CONFIG run code out err "$BRIEF" --project pager +assert_contains "$out" ' status: clear' "an overridden config directory supplies its own declaration" +assert_equals 'find-generic-password -s effective-svc -w' "$(cat "$LOG/security.argv")" "only the effective directory's declaration is read" +assert_equals "Authorization: Bearer $EFFECTIVE_SECRET" "$(cat "$LOG/header")" "the effective directory's key is the one sent" +rm -f "$EFFECTIVE_CONFIG/dispatch-credential-sources.json" +reset_log +FM_CONFIG_OVERRIDE=$EFFECTIVE_CONFIG run code out err "$BRIEF" --project pager +assert_equals '' "$out" "the home's own config declaration is ignored when another directory is effective" +assert_absent "$LOG/security.argv" "a declaration outside the effective directory is never read" +rm -f "$CRED" +pass "effective-home isolation: only the effective config directory's declaration is consulted" + +# failure distinctions: off, a configuration error, and an unavailable source are three different things +printf '{}\n' > "$CRED" +reset_log +run code out err "$BRIEF" --project pager +expect_code 0 "$code" "a declaration with no entries is off" +assert_equals '' "$out" "off prints nothing on stdout" +assert_contains "$err" 'dispatch-resolve: off (TYPESAFE_API_KEY absent from the environment and' "off says so on stderr" +assert_absent "$LOG/security.argv" "an empty declaration never reaches the Keychain" +assert_absent "$LOG/argv" "off never calls curl" + +for bad in \ + '{"typesafe":|not JSON' \ + '|must hold exactly one JSON object' \ + '[]|top-level value must be an object keyed by provider' \ + '{"typsafe":{"source":"macos-keychain","service":"s"}}|unknown provider typsafe (expected typesafe or openrouter)' \ + '{"typesafe":"kc-planted-value"}|typesafe must be an object with a source' \ + '{"typesafe":{"service":"s"}}|typesafe needs a source' \ + '{"typesafe":{"source":"kc-planted-value","service":"s"}}|typesafe source must be one of macos-keychain' \ + '{"typesafe":{"source":"macos-keychain"}}|typesafe service must be a non-empty string without control characters' \ + '{"typesafe":{"source":"macos-keychain","service":""}}|typesafe service must be a non-empty string without control characters' \ + '{"typesafe":{"source":"macos-keychain","service":7}}|typesafe service must be a non-empty string without control characters' \ + '{"typesafe":{"source":"macos-keychain","service":"a\nb"}}|typesafe service must be a non-empty string without control characters' \ + '{"typesafe":{"source":"macos-keychain","service":"s","account":""}}|typesafe account must be a non-empty string without control characters when present' \ + '{"typesafe":{"source":"macos-keychain","service":"s","key":"kc-planted-value"}}|typesafe has unknown field key (allowed: source, service, account); credential values do not belong in this file' \ + '{"openrouter":{"source":"macos-keychain","service":"s","token":"kc-planted-value"}}|openrouter has unknown field token (allowed: source, service, account); credential values do not belong in this file'; do + printf '%s\n' "${bad%%|*}" > "$CRED" + reset_log + run code out err "$BRIEF" --project pager + expect_code 2 "$code" "an invalid declaration exits 2: ${bad#*|}" + assert_contains "$err" "malformed credential source declaration: $CRED - ${bad#*|}" "the problem is named: ${bad#*|}" + assert_not_contains "$err" 'kc-planted-value' "an invalid declaration never echoes a declared value" + assert_equals '' "$out" "an invalid declaration prints nothing on stdout: ${bad#*|}" + assert_equals '1' "$(grep -c . <<<"$err")" "an invalid declaration is one diagnostic line: ${bad#*|}" + assert_absent "$LOG/security.argv" "an invalid declaration never reaches the Keychain: ${bad#*|}" + assert_absent "$LOG/argv" "an invalid declaration never reaches the network: ${bad#*|}" +done +rm -f "$CRED" +ln -s "$TMP_ROOT/missing-declaration.json" "$CRED" +reset_log +run code out err "$BRIEF" --project pager +expect_code 2 "$code" "a broken declaration symlink exits 2" +assert_contains "$err" "credential source declaration not readable: $CRED" "an unreadable declaration is named" +rm -f "$CRED" +declare_source "$CRED" typesafe typesafe-svc +reset_log +run_on_path "$NO_JQ_BIN" code out err "$BRIEF" --project pager +expect_code 2 "$code" "a declaration cannot be read without jq" +assert_contains "$err" 'error: jq required' "missing jq is named" +assert_absent "$LOG/security.argv" "missing jq never reaches the Keychain" + +# expect_unavailable