Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
d2d2388
refactor: record every turn's tool effects on a per-turn ledger
AlexanderZ-Band Oct 4, 2026
794fd5d
fix!: judge every turn with band-sdk-core's turn outcome
AlexanderZ-Band Oct 4, 2026
89fa7a1
fix: let a cancel reach a turn whose decision resolves in the same tick
AlexanderZ-Band Oct 4, 2026
db6d010
test: have the delegating peer ask a question, not forward a bare token
AlexanderZ-Band Oct 4, 2026
87c56e0
test: end event-only and remember-only smoke turns deliberately
AlexanderZ-Band Oct 4, 2026
388e828
fix: keep the plain-text warning in the base prompt; vary the copilot…
AlexanderZ-Band Oct 4, 2026
f667f5f
fix: answer a rejected OpenCode question so the model can still reply
AlexanderZ-Band Oct 4, 2026
b6b86a7
fix: judge Slack-inbound turns and deduped replies; address review
AlexanderZ-Band Oct 4, 2026
10d4e90
test: align turn-outcome coverage with the TypeScript SDK
AlexanderZ-Band Oct 5, 2026
9cafdc2
fix: address the cross-SDK and ACP-boundary reviews
AlexanderZ-Band Oct 5, 2026
2741048
fix: fall back when a missing-reply report fails and honor the report…
AlexanderZ-Band Oct 5, 2026
f091057
fix: report a turn whenever its own failure post did not land
AlexanderZ-Band Oct 5, 2026
c4653ee
Merge origin/main into fix/fixpython-judge-every-turn-with-cores-turn…
AlexanderZ-Band Oct 5, 2026
37cef25
fix: preserve successful turn effects and complete adapter replies
bandzalkin Oct 5, 2026
7eb311c
refactor: judge Letta tool effects from complete responses
bandzalkin Oct 5, 2026
4f3e718
revert: judge Letta tool effects in stream order
AlexanderZ-Band Oct 5, 2026
97a5250
refactor: match each Letta tool return to its pending call
AlexanderZ-Band Oct 5, 2026
a53ee8e
Merge origin/main into fix/fixpython-judge-every-turn-with-cores-turn…
AlexanderZ-Band Oct 5, 2026
3d27277
test: custom tools record their effect on a room-bound endpoint's turn
AlexanderZ-Band Oct 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,16 @@ text. `tests/framework_conformance/test_tool_text_drift.py` is the drift
guard; an adapter that builds its own tool schemas sets `advertised_arg_text`
on its test config so the guard can read what the model actually sees.

## Turn Outcome

`SimpleAdapter.run_judged_turn` (called by `on_event`) judges every turn with band-sdk-core's rule: a turn
that replied, declined via `band_no_reply`, did real work, was settled by the
adapter, or already reported a failure completes; anything else is reported
once and marked FAILED. Adapters never judge turns themselves. Post an
adapter's own messages with `send_notice` (never `send_message`, which counts as
the reply) and relay a model's final text only through `relay_reply`. See
[docs/turn-outcome.md](docs/turn-outcome.md).

## Adapter Constructor Shape

Every adapter is built as `XAdapter(config: XAdapterConfig | None = None, *,
Expand Down Expand Up @@ -94,7 +104,7 @@ confined to an explicit AST-enforced allowlist
`OpencodeAdapter` maps each Band room to a session on a running `opencode
serve`. Band tools are never gated behind approval, unlike other tool
calls — see [docs/adapters/opencode.md](docs/adapters/opencode.md) for this
and three more invariants that are easy to break.
and four more invariants that are easy to break.

## ACP (Agent Client Protocol) Integration

Expand Down
22 changes: 15 additions & 7 deletions docs/acp.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,15 @@ assert adapter.config.command == ("codex-acp",)
- **Narration is live and ordered.** `ACPCollectingClient` streams finalized chunks to
`RoomTurnEmitter` as they arrive, so a Band tool's own room post (a remote band-mcp
posts over REST mid-turn) lands between its `tool_call` and `tool_result`.
- **Assistant text is held to turn close** and relayed only if no completed call settled
the reply (`turn_replied_in_room`, `settles_turn_reply`). Detection matches the
`tool_call` title, because tools may run out-of-process. Narrated names are canonicalized
(`canonicalize_mcp_tool_name`) so Copilot's `band-` prefix never reaches the room.
- **`emit=` never gates** chunk recording (the reply decision must see the whole turn),
the held text, or the closing `task` event. That event is resume state:
- **Assistant text is held to turn close** and relayed as one reply through `relay_reply`,
unless the turn already replied or declined. Injected Band tools record their own effect
on `tools.turn`. With `inject_band_tools=False` an external band-mcp runs out of process,
so the emitter records each completed call's `turn_effect` from the `tool_call` title
instead, and only in that mode, so no call is counted twice. Narrated names are
canonicalized (`canonicalize_mcp_tool_name`) so Copilot's `band-` prefix never reaches
the room.
- **`emit=` never gates** that recording, the held text, or the closing `task` event.
That event is resume state:
`ACPClientHistoryConverter` reads `acp_client_session_id` / `acp_client_room_id` from it
to `session/load` after a restart.
- **Approved permissions are silent.** Only a denied request posts a synthetic
Expand Down Expand Up @@ -76,7 +79,12 @@ failure fails that room turn visibly instead of falling back.
- **Cursor:** question, plan and permission decisions default to `manual`, resolved by a
room participant with `/cursor <word> <token>`. Cursor omits the session id on its
extension notifications, so the adapter holds a turn lock and binds them to that turn's
session; Cursor turns are serialized.
session; Cursor turns are serialized. Decision prompts, timeout notices and `/cursor`
replies post through `send_notice`, so they never count as the model's reply, and a
`/cursor` message settles its own turn without reaching the agent. A turn parked on a
decision releases its message early (`tools.turn.detach()`), so it is judged when the
ACP turn completes normally; a failed or cancelled turn is never reported as a missing
reply.
The adapter does not pick a plan/agent mode itself; a caller selects one through
`resolve_session_config`, which reads each session's advertised catalog before the
first prompt. The live `backends` lane pins the Cursor CLI and passes `E2E_CURSOR_API_KEY`
Expand Down
12 changes: 11 additions & 1 deletion docs/adapters/opencode.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
`opencode serve`: room messages become prompts, and the server's SSE stream is
relayed back as room messages, tool narration, and error events.

Four invariants are easy to break and expensive to rediscover:
Five invariants are easy to break and expensive to rediscover:

- **Band tools are never gated.** A `permission.asked` naming one of the
adapter's own registered tools is auto-approved with `always` in *every*
Expand All @@ -24,9 +24,19 @@ Four invariants are easy to break and expensive to rediscover:
- **The model is told the current `chat_id` every turn.** The band MCP tools'
schemas require it, so without the per-turn Room Context block the platform
tools are uncallable.
- **A declined ask must hand the turn back to the model.** OpenCode ends the
turn on a bare permission reject and on its question-reject route, so the model
never replies and the turn is reported as a missing reply. A permission reject
carries feedback, and a rejected question (by the room, `auto_reject`, or a
timeout) is answered with a decline instead of rejected; either way the model
sees the refusal and answers the room.

`turn_timeout_s` bounds *compute*: time parked on a manual approval is excluded,
since the ask carries its own `approval_wait_timeout_s` expiry.
A manual ask also releases the turn's delivery early (it is marked processed
while the human decides), so the adapter judges that turn itself when OpenCode
finishes it: a turn that ends without a reply then posts one missing-reply
failure. A turn cancelled by cleanup or an interrupt posts nothing.

`variant` is an opaque, provider- and model-specific name the running server
owns: it may select reasoning effort or be a custom provider setting, so read
Expand Down
265 changes: 265 additions & 0 deletions docs/turn-outcome.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,265 @@
# Turn outcome

Every message pushed to an agent @mentions it, so every turn owes the room an
answer: a reply, or a deliberate decision not to reply.
`SimpleAdapter.run_judged_turn`, which `on_event` and the Slack adapter's
inbound path both call, judges each turn with one rule from `band-sdk-core`,
shared with the TypeScript SDK, and tells the platform honestly whether the
delivery succeeded.

## The rule

A turn is **complete** when it:

- replied (`band_send_message`, `band_send_room_file`)
- declined on purpose (`band_no_reply`)
- did real work (any Band tool whose effect is `act`, such as
`band_add_participant`)
- was settled by the adapter itself (a control reply, a busy notice)
- already reported a failure (`send_failure`)

Anything else is a **missing reply**. Fetching state or narrating through
`band_send_event` only observes, so it never completes a turn. A missing reply
is reported once, as an `error` event from provider `band-runtime` carrying
core's text, and the delivery is marked FAILED. There is no nudge and no
automatic retry: the SDK reports what happened instead of claiming success.

Every Band tool call records its effect on the turn's ledger, `tools.turn`,
whichever path made the call:

```python
import asyncio

from band.testing.fake_tools import FakeAgentTools


async def main() -> None:
tools = FakeAgentTools()

await tools.send_event("Looking that up", message_type="thought")
assert not tools.turn.complete # narration only observes

await tools.no_reply("FYI only")
assert tools.turn.complete
assert tools.turn.replied # the model's final text is not relayed


asyncio.run(main())
```

A turn that ends without completing is reported, then raised as
`TurnResultAlreadyReported` so the runtime marks the delivery FAILED without a
second report:

```python
import asyncio
from datetime import UTC, datetime

import band_sdk_core

from band.core.protocols import TurnResultAlreadyReported
from band.core.simple_adapter import SimpleAdapter
from band.core.types import AgentInput, HistoryProvider, PlatformMessage
from band.testing.fake_tools import FakeAgentTools, reported_failures


class Forgetful(SimpleAdapter[HistoryProvider]):
async def on_message(
self,
msg,
tools,
history,
participants_msg,
contacts_msg,
*,
is_session_bootstrap,
room_id,
):
await tools.get_participants()


async def main() -> None:
tools = FakeAgentTools()
msg = PlatformMessage(
id="m1",
room_id="r1",
content="@bot hi",
sender_id="u1",
sender_type="User",
sender_name="Alice",
message_type="text",
metadata={},
created_at=datetime.now(UTC),
)
inp = AgentInput(
msg=msg,
tools=tools,
history=HistoryProvider(raw=[]),
participants_msg=None,
contacts_msg=None,
is_session_bootstrap=True,
room_id="r1",
)
try:
await Forgetful().on_event(inp)
except TurnResultAlreadyReported:
pass
else:
raise AssertionError("a turn that did nothing must be reported")

assert [f["message"] for f in reported_failures(tools)] == [
band_sdk_core.missing_reply_message()
]


asyncio.run(main())
```

## Declining with `band_no_reply`

`band_no_reply` is in every adapter's tool set, and `BASE_INSTRUCTIONS` tells
the model to call it when the latest message needs no answer: it was addressed
to someone else, it is an FYI or an acknowledgement, or someone already answered
it. The call is local only; its `reason` goes to the agent log.

## Custom tools: `@declares_turn_effect`

An undeclared custom tool counts as `observe`: the SDK cannot tell a lookup from
an action, so the turn still owes a reply. A tool that completes the turn says
so:

```python
import asyncio

from pydantic import BaseModel

from band.runtime.custom_tools import declares_turn_effect, execute_custom_tool
from band.runtime.tools import TurnEffect
from band.testing.fake_tools import FakeAgentTools


class FileTicketInput(BaseModel):
"""File a support ticket."""

title: str


@declares_turn_effect(TurnEffect.ACT)
async def file_ticket(args: FileTicketInput) -> str:
return f"filed: {args.title}"


async def main() -> None:
tools = FakeAgentTools()
await execute_custom_tool(
(FileTicketInput, file_ticket), {"title": "Login broken"}, turn=tools.turn
)
assert tools.turn.complete


asyncio.run(main())
```

`REPLY` declares a tool that delivers the answer itself, and `DECLINE` one
whose silence is the answer; both also stop a relaying adapter from posting the
model's final text.

A declared effect is recorded only on success. A handler that raises or returns
an explicit failure (`{"ok": False}`, or a string starting with `Error:` or
`Error executing `, case-insensitively) records nothing: the turn still owes an
answer, and a failed reply or decline does not suppress the final-text relay.
Other return values, including `None` from a side-effect-only handler, count as
successful.

**Limits:** a custom tool only records on the turn it is handed. Where a tool
runs with `turn=None` (not bound to a room, as in a standalone MCP server), it
cannot complete any turn. A framework-native tool (rather than an
`(InputModel, handler)` pair) records its declared effect only where the
adapter observes it: Strands and Pydantic AI do; Agno tools and ready-made
LangChain tools count as `observe`.

## Adapter posts: `send_notice` and `settle()`

A post through `send_message` or `deliver_reply` counts as the turn's reply. An
adapter's own message (an approval prompt, a busy notice, a `/status` reply)
goes through `tools.send_notice`, which posts the same way but records nothing,
so it can never stand in for the model's answer. When the adapter consumes a
message without running the model, or ends the turn with its own text, it calls
`tools.turn.settle()`:

```python
import asyncio

from band.testing.fake_tools import FakeAgentTools


async def main() -> None:
tools = FakeAgentTools()

await tools.send_notice(
"Still working on the previous request.", mentions=["@alice"]
)
assert not tools.turn.complete # a notice is never the reply

tools.turn.settle()
assert tools.turn.complete


asyncio.run(main())
```

`relay_reply(tools, text, mentions)` from `band.core.delivery` is the one gate
for a model's final text: it posts through `deliver_reply` only when no tool
replied or declined this turn. It is a fallback, not a channel to steer
toward: the base prompt still says plain text is never delivered, because
without that line models more often send their closing narration as a second
`band_send_message` (measured live on gemini-2.5-flash).

Comment thread
AlexanderZ-Band marked this conversation as resolved.
Parlant joins the non-preamble final segments from one event batch before
relaying, so recording the fallback reply cannot suppress a later segment of
that same answer.

For external Letta MCP servers, each successful grouped tool return is matched
to its call by `tool_call_id` and records that core tool's turn effect. Failed
returns record nothing; self-hosted tools record their own effects.

`tests/framework_conformance/test_reply_boundary.py` pins every `send_message`,
`deliver_reply` and `relay_reply` call outside the tool implementations, per
file with the reason it carries the model's words, so a new one fails until it
is justified.

## Detached turns

An adapter whose turn outlives `on_message` (one parked on a human approval)
calls `tools.turn.detach()` when it releases the turn early. `run_judged_turn` then
leaves the verdict to the adapter, which reports at the turn's normal end. A
cancelled turn reports nothing:

```python
import asyncio

from band.core.turn import judge_detached_turn
from band.testing.fake_tools import FakeAgentTools, reported_failures


async def main() -> None:
tools = FakeAgentTools()
tools.turn.judged = True # set by SimpleAdapter.run_judged_turn
tools.turn.detach() # released while waiting for an approval

# ...later, on the turn's normal completion path (never a ``finally``):
await judge_detached_turn(tools, room_id="room-1")

assert len(reported_failures(tools)) == 1


asyncio.run(main())
```

## Exempt adapters

`SimpleAdapter.judges_turns` is `False` where the turn is not the model's to
answer through Band tools: `A2AAdapter`, `A2AGatewayAdapter`,
`CrewAIFlowAdapter`, `BandACPServerAdapter`, `ParlantAdapter` (its engine owns
its replies) and a LangGraph adapter built from a static `graph=` (it never
gets Band tools). Synthetic contact-hub turns are never judged.
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ classifiers = [

dependencies = [
"band-client-rest==0.0.41",
"band-sdk-core==2.7.0",
"band-sdk-core==2.8.0",
"phoenix-channels-python-client>=0.2.4",
"python-dotenv>=1.2.2",
"pydantic>=2.0",
Expand Down
Loading
Loading