Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 31 additions & 2 deletions accelerators/coral/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -291,10 +291,38 @@ def tool_object_detect(args: dict) -> dict:
return {"objects": objects, "count": len(objects), "model": MODELS[key]["file"], "image_width": w, "image_height": h}


def _out_path(args: dict, image: str, suffix: str) -> str:
"""Where a tool may write its PNG: the caller's out_path, or <image stem><suffix> beside the image.

Writing anywhere new is the caller's choice; replacing a file that is not already a PNG never is (a source
image, a config or a record clobbered with mask bytes is data loss). An earlier output is replaced. The
same rule the Axera sidecar and the fleet caller's landing apply.
"""
out = args.get("out_path")
if out is not None and (not isinstance(out, str) or not out):
raise ValueError({"error": "bad_request", "detail": "out_path must be a non-empty string"})
if not out:
out = os.path.splitext(image)[0] + suffix
d = os.path.dirname(os.path.abspath(out))
if not os.path.isdir(d) or not os.access(d, os.W_OK):
raise ValueError({"error": "out_path_unwritable", "out_path": out})
if os.path.lexists(out):
try:
with open(out, "rb") as fh:
is_png = fh.read(8) == b"\x89PNG\r\n\x1a\n"
except OSError:
is_png = False
if os.path.islink(out) or not os.path.isfile(out) or not is_png:
raise ValueError({"error": "out_path_exists", "out_path": out,
"detail": "an existing file that is not a PNG is never overwritten; choose a new path"})
return out


def tool_semantic_segment(args: dict) -> dict:
image = args.get("image_path")
if not isinstance(image, str) or not image:
raise ValueError({"error": "bad_request", "detail": "image_path required"})
out_path = _out_path(args, image, ".segmask.png") # refused before the TPU runs, never after
key = "deeplab-pascal"
it = _interpreter(key)
d = it.get_input_details()[0]
Expand All @@ -308,8 +336,9 @@ def tool_semantic_segment(args: dict) -> dict:
t = t[0]
mask = t.argmax(-1).astype(np.uint8) if t.ndim == 3 else t.astype(np.uint8)
labels = _load_labels(key)
out_path = args.get("out_path") or (os.path.splitext(image)[0] + ".segmask.png")
Image.fromarray(mask, mode="L").resize((w, h), Image.NEAREST).save(out_path)
tmp = out_path + ".part.png"
Image.fromarray(mask, mode="L").resize((w, h), Image.NEAREST).save(tmp, format="PNG")
os.replace(tmp, out_path)
ids, counts = np.unique(mask, return_counts=True)
classes = [{"class_id": int(i), "label": labels[int(i)] if int(i) < len(labels) else str(int(i)), "pixels": int(c)}
for i, c in zip(ids, counts)]
Expand Down
10 changes: 10 additions & 0 deletions changes/unreleased/fix-accel-independent-and-fallback.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
### Fixed - an accelerator answers while the node's GPUs are leased, and a shared accelerator tool falls back to the next device

- **Why.** Two rules kept accelerators idle while callers waited, both seen 2026-11-11. A text lease on a node's GPUs made `/fleet/dispatch` refuse every forwarded accelerator call with `node leased` for the whole measurement window, although an `accel` job runs on an NPU or TPU and never on a leased card. And a shared tool (`offload_object_detect`, `offload_classify_image`, `offload_image_embed`, `offload_semantic_segment`) had one owner and no second choice (ADR 0068: "nothing falls back at call time"), so with the owner's node off or refusing the tool deferred while two other devices that serve the same name were reachable.
- **What it does: the lease.** A GPU lease never refuses an `accel` job (`fleetnode.textLeaseAgainst`). Every other task type keeps its reading, and the lease is published in health as before.
- **What it does: the fallback.** The owner of a shared tool is tried first. When it cannot take the call (a defer, or a sidecar refusal about the device such as a missing model or a disabled NPU), the call runs on the next device that serves the name, in the listed order, through that device's own tool row and with only the arguments its schema declares. The answer carries `fallback: {from, reason, served_by}`; when no device answers, the owner's defer carries `fallback_tried`. A refusal of the caller's input is the answer and is not retried elsewhere. Nothing is remembered between calls. Both surfaces (the MCP tools and the agent loop) use one function, `agent.AccelFallback`, and a node running a forwarded job still cannot forward again.
- **Embeddings.** `offload_image_embed` and `offload_text_embed` move to another device only when the caller passes `any_space: true`, because that vector is in another space; without it the defer carries `fallback_withheld`. The argument is added to the schema only on a box where a fallback device exists.
- **Safety.** One security pass ran on the change; both findings are fixed here. A fallback could hand `out_path` to a device whose sidecar checks nothing: the Coral's `semantic_segment` overwrote any path it was given, the caller's own image included. The fallback now applies the lane's rule before any device runs (`out_path` is never the input image, its directory must exist, an existing target is replaced only when it is a regular PNG file; otherwise the defer carries `fallback_withheld`), and the Coral sidecar applies the same rule itself and writes through a temporary file. A fallback device that answers JSON `null` is a defer, not a crash.
- **What a caller sees.** A tool with a fallback device carries a `FALLBACK:` sentence in its description (kept by the compact tools profile), and `offload_status.accelerators.<id>.fallback_for` lists the names a device answers for another owner. A box with one device, or with no shared name, advertises exactly what it did.
- **Check.** Two tests, one per incident: an `accel` dispatch under a text lease is admitted while an image job on the same node is still refused, and through the MCP surface a tool whose owner has no node to reach runs on the next device with the owner named, an embedding stays put without `any_space`, and `any_space` never reaches a sidecar.
- **Docs.** ADR 0102 (supersedes one consequence of ADR 0068), `docs/systems/accelerators.md`, `docs/systems/gpu-lease.md`, the operator guide's lease paragraph.
3 changes: 2 additions & 1 deletion docs/OPERATOR-GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2330,7 +2330,8 @@ HAND-EDITED, while UNSTAMPED (every config rendered before 0.123.0) prints as a
re-rendering with `install render`, never by hand-editing the file.

**Lease in health (0.113.16).** `/fleet/health` carries `lease` while the node's GPU lease is held; a text lease makes the
node ineligible for new delegated work (and its dispatch answers 503, re-placeable). `gpu reserve --drain --unload-seat` and
node ineligible for new delegated work (and its dispatch answers 503, re-placeable). An `accel` job is the exception: it runs
on an NPU or TPU, never on a leased card, so a lease never refuses it (ADR 0102). `gpu reserve --drain --unload-seat` and
`gpu release --warm-seat` are the maintenance verbs — see docs/systems/gpu-lease.md.

**A held card is a place in line (0.115.2).** `gpu reserve` QUEUES behind a current holder for `--wait` (default 8h;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ without an entry is decided exactly as before.
so moving it changes the space a caller receives, and the tool description says which device serves it.
- Nothing is chosen per call and nothing falls back at call time. If the named device's node is down, the
call defers (ADR 0038) instead of running on the other device, which matches every other accelerator
lane.
lane. **Superseded by [ADR 0102](0102-an-accelerator-answers-through-a-gpu-lease-and-a-shared-tool-falls-back.md):**
the named device is tried first, and the next device that serves the name answers when it cannot.

## Related

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
---
status: Accepted
date: "2026-11-11"
---

# An accelerator answers through a GPU lease, and a shared tool falls back to the next device

## Context

Two rules kept accelerators idle while callers waited. Both were seen on 2026-11-11, the day the AX650-class
NPU card ([ADR 0101](0101-axera-ax650-npu-card-as-an-accelerator.md)) took four shared tools on the delegator.

**A reservation of the GPUs refused the NPU.** A text lease reserves a node's cards for a measurement, and
`/fleet/dispatch` refused every task type but an agent contract while one was held, because "the cards it
will use are chosen later, and unknown is every card". An `accel` job uses none of them: it runs on a TPU or
an NPU through a loopback sidecar, and on the NPU card in its own memory. For the hours a measurement held
the node's GPUs, every forwarded accelerator call was refused with `node leased`, with the device idle.

**A shared tool had one device and no second choice.** A shared tool name has one owner per box
([ADR 0037](0037-a-capability-name-has-one-owner-per-box.md)), and the operator may name it
([ADR 0068](0068-the-operator-may-name-the-owner-of-a-shared-accelerator-tool.md)). ADR 0068 also decided
that "nothing is chosen per call and nothing falls back at call time". With detection, classification,
embedding and semantic segmentation handed to the NPU card on one node, those four tools deferred whenever
that node was off or refusing, while a Coral and an RK3588 NPU that serve the same names were reachable and
idle. The operator's standing order is that the harness adapts to the work.

## Decision

**1. A GPU lease never refuses an `accel` job.** `fleetnode.textLeaseAgainst` answers "not held" for the
`accel` task type. Every other task type keeps its reading: an agent contract by the seats it could run on,
the rest by the whole node. The lease is still published in `/fleet/health` unchanged.

**2. The owner of a shared tool is the first choice, not the only one.** When the owner cannot take a call,
the call runs on the next device that serves the same name, in the listed order (local devices, then
`fleet_accelerators`), and the result says so. This supersedes ADR 0068's "nothing falls back at call time";
the rest of ADR 0068 and ADR 0037 stand: one registered tool per name, the owner's description, schema and
model, the same owner on both surfaces.

- **What "cannot take the call" means** (`agent.AccelCannotServe`). A defer: the node is off, no node
advertises the device, the dispatch was refused, the sidecar did not start. Or a sidecar refusal that is
about the device: a model that is missing, fails its digest or does not load, a disabled or absent NPU, an
internal error. A refusal of what the caller sent (`bad_request`, `image_missing`, `image_unreadable`,
`out_path_exists`, `out_path_unwritable`, `forbidden`, `not_found`, `unknown_model`) is an answer: another
device would refuse the same input.
- **Each fallback device runs its own tool.** Its own table row gives the sidecar tool, the required
argument and the schema. The caller sent the owner's arguments; only the ones that device's schema
declares travel to it.
- **The result is attributable.** A fallback answer carries
`fallback: {from, reason, served_by}`. When no device serves the call, the owner's answer comes back with
`fallback_tried: [{device, reason}]`.
- **An embedding does not move silently.** A vector from another device is in another space, and nothing in
the numbers says so. `offload_image_embed` and `offload_text_embed` fall back only when the caller passes
`any_space: true`; without it the owner's defer comes back with `fallback_withheld`, naming the devices
that could have answered. The argument is added to the tool's schema only on a box where a fallback device
exists, and it never reaches a sidecar.
- **`out_path` is checked before it travels.** The owner applied its own rule to `out_path` (the NPU card's
sidecar, or the caller-side landing of a fleet device). A fallback device's sidecar may apply none, and an
agent seat can pass any path, so the fallback applies the lane's rule itself before any device runs:
`out_path` is never the input image, its directory must exist, and an existing target is replaced only
when it is a regular file that is already a PNG. Otherwise the owner's answer comes back with
`fallback_withheld`. The Coral sidecar's `semantic_segment` gains the same rule at the sink, where it had
none (found by the security pass on this change).
- **Nothing is remembered between calls.** There is no "owner is down" state and no health pre-check: every
call tries the owner first. A node that is off costs the roster probe (about 2 s) before the fallback runs.
- **Where it lives.** At tool registration on both surfaces, through one function
(`agent.AccelFallback`): `mcpserver.handleAccelChain` and `agent.accelLaneTools`. A node running a
forwarded job calls its local lane directly (`pipeline.runAccelTask`), so a call cannot forward twice.
- **Status.** A device's `offload_status.accelerators` row gains `fallback_for`, the names it answers when
their owner cannot. `serves` and `owns` are unchanged. A tool with a fallback says so in its description.

## Consequences

- A measurement lease on a node's GPUs and accelerator work on the same node run side by side. The lease
holder's GPU numbers are untouched; the sidecar's pre- and post-processing does use host CPU for a few
milliseconds per call.
- A fallback answer comes from another model: a different detector, classifier or segmenter, with that
device's defaults for any argument the caller did not send. `fallback.served_by` and the result's `model`
say which one ran, and a caller that needs the owner's model reads them.
- A box with one device, or with no shared name, advertises exactly the tools it did before: descriptions and
schemas are extended as text and only where a fallback device exists.
- A call whose input is refused is not retried elsewhere; a call the owner fails for a device reason costs
one more call per fallback device before it defers.

## Alternatives considered

- **Keep the refusal and let the operator release the lease.** That is the rule that failed: the lease was
doing its job for the GPUs and had no claim on the NPU.
- **Scope the exemption to devices with their own memory.** The RK3588 NPU shares host RAM, but a lease
reserves GPUs, not RAM, and its declared RAM figure is for the holder's own load. One rule for every
accelerator is the one a reader can predict.
- **Route by health before the call.** It needs a second source of truth about the owner and goes stale
between the probe and the call. Trying the owner first reads the same health through `pickNode` and fails
in about 2 s.
- **Let an embedding fall back like the rest.** A similarity index built from two spaces is wrong without
any error, and two of the three devices can share a dimension.
- **Let the caller name a device per call.** It would work, and it is the device-suffixed tool ADR 0037
rejected: a caller would have to know the hardware to ask for a capability.

## Related code

- `internal/fleetnode/leases.go` (`textLeaseAgainst`), `internal/fleetnode/accel_task_test.go`
- `internal/agent/accelfallback.go`, `internal/agent/acceltools.go` (`accelLaneTools`)
- `internal/mcpserver/acceltools.go` (`accelFallbackPlan`, `handleAccelChain`, `accelStatus`),
`internal/mcpserver/accelfallback_test.go`

## Related docs

- [systems/accelerators.md](../../systems/accelerators.md)
- [ADR 0037](0037-a-capability-name-has-one-owner-per-box.md),
[ADR 0038](0038-accelerator-work-travels-to-the-box-that-has-the-device.md),
[ADR 0068](0068-the-operator-may-name-the-owner-of-a-shared-accelerator-tool.md),
[ADR 0101](0101-axera-ax650-npu-card-as-an-accelerator.md)
1 change: 1 addition & 0 deletions docs/architecture/decisions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ code, treat everything else as background.
| [0099](0099-frontier-updates-floors-never-pins.md) | Accepted | Frontier updates: floors, never pins. Versions are never pinned: the installers and `local-offload update` resolve the newest complete build from the publisher and verify its digest; the only version literals are floors in `setup/floors.json`; a node keeps its current and previous generation behind a live path and rolls back on a failed gate; a lint keeps every other file clean; supersedes the second-build doctrine of [0081](0081-the-embeddinggemma2-entry-may-run-from-a-second-llama-build.md) |
| [0100](0100-the-memory-wall-mapped-onto-the-fleet.md) | Proposed | The memory wall mapped onto the fleet (NVIDIA's five: NVFP4, WideEP, Dynamo, Vera, CMX; tiers G1-G4 plus G3.5): the reuse instrument reads vLLM's `cached_tokens`; llama.cpp slot files become the G3 tier on every one-slot seat at build b11506 or later, keyed node-side over the byte-exact prefix, saved on a prefix's second sighting, restored and judged by the next completion's cache share, and saved across a tool wait of 240 s or more so a seat that unloads mid-run restores instead of re-prefilling; nvfp4 KV beside a cache server is refused; WideEP, disaggregation, KV-aware routing, CMX hardware and tool-wait GPU-to-host offload are ruled out or already covered, each with its reason |
| [0101](0101-axera-ax650-npu-card-as-an-accelerator.md) | Proposed | An AX650-class NPU on an M.2 card is the fourth accelerator, `axera-ax650`: fleet-visible, detected from `/proc/bus/pci/devices` (`1f4b0650` + driver `ax_pcie_dev_host`), its own 4 GB (2944 MiB CMM) never host RAM or VRAM, ten licence-clean publisher-compiled tools (RF-DETR, SigLIP2, DeepLabV3+, MobileSAM, Depth Anything 3, RTMPose, PP-OCRv5) with `offload_ocr` `engine:"npu"` going Hailo, then Axera, then fleet; ownership handed to it per tool by `accelerator_tool_owners`; telemetry read through the runtime library in a child process (never the single-instance vendor CLI); CPU pinned to its top step at boot; face, person re-id, low-light, speech and an NPU LLM seat not owned (later slices) |
| [0102](0102-an-accelerator-answers-through-a-gpu-lease-and-a-shared-tool-falls-back.md) | Accepted | An accelerator answers through a GPU lease, and a shared tool falls back: a text lease never refuses an `accel` job (the device is not a card a lease names); the owner of a shared tool is the first choice and the next device that serves the name answers when it cannot (a defer or a device-side refusal, never a refusal of the caller's input), each fallback through its own table row with only the arguments it takes, the result carrying `fallback{from,reason,served_by}`; an embedding moves only with `any_space:true`; nothing is remembered between calls; supersedes ADR 0068's "nothing falls back at call time" |


## Lifecycle
Expand Down
Loading
Loading