Discuss connected service operation selection and tool catalog boundaries

[louis4li]

Background

A channel runtime run hit this catalog budget failure while preparing the initial LLM turn:

Aevatar.AI.Abstractions.ToolProviders.AgentTurnToolCatalogException: Final tool catalog exceeds its typed safety budget (schema_bytes=123316/49152, connected_reads=341/2147483647, connected_writes=151/2147483647)

The current feature/integrate path appears to treat selected connected-service API surface as final LLM tool catalog material:

ChannelRuntimeConfigProof
  -> resolve tool set refs
  -> discover tools
  -> select visible route tools
  -> select connected operations by service selector / endpoint selector
  -> union all selected names
  -> build AgentTurnToolCatalog
  -> proof budget check

The failure is not caused by connected read/write count limits. The path uses the ordinary catalog budget, so read/write limits are effectively unbounded here. The failure is caused by total canonical schema bytes: 123 KB selected vs 48 KB budget.

The design question is broader than the immediate exception: should a connected service OpenAPI surface be converted into LLM tools first, or should we use a lighter operation-selection layer before any execution boundary is exposed to the model?

For example, a Google Workspace service may not be one of many services for a given agent, but a single service can still contain many endpoints across Drive, Calendar, Gmail, etc. Treating the whole service as a tool catalog is likely to exceed context and safety budgets.

Goals

• Keep the agent able to use broad connected services.
• Avoid oversized per-turn LLM tool catalogs.
• Preserve authorization, risk policy, approval, audit, and validation boundaries.
• Avoid silently losing the right endpoint when filtering.
• Support both read-heavy discovery flows and sensitive write/destructive flows.
• Fit Aevatar command/event/actor architecture rather than making provider-specific tool calling the only execution abstraction.

Non-goals

• Raising the schema budget as the primary fix.
• Passing raw OpenAPI specs to the LLM on every turn.
• Letting the model construct arbitrary URLs, methods, headers, or unvalidated request bodies.
• Treating service authorization as equivalent to per-turn execution authorization.

Current design: expand selected API operations as tools

Shape

service selector / endpoint selector
  -> discovered IAgentTool operations
  -> selectedNames
  -> AgentTurnToolCatalog
  -> LLM sees endpoint tools directly

Benefits

• Simple mental model: selected endpoint equals callable tool.
• Tool schemas are strong guidance for the model.
• Existing tool-call infrastructure already handles structured invocation.
• Catalog proof can digest and budget the final selected tools.

Problems

• Service-level selection can expand into hundreds of tools.
• Schema bytes grow with endpoint count and parameter detail.
• Similar endpoints increase model choice noise.
• Wide write/destructive tool exposure increases risk even if approval exists later.
• Runtime may fail only when building the reply-generation catalog.
• Budget enforcement acts as a final reject instead of guiding catalog reduction.

Option 1: exact-tool materialization after operation selection

Shape

OpenAPI / NyxID catalog spec
  -> operation index cards
  -> rank/filter by current user turn
  -> select small operation set
  -> materialize only selected operations as exact typed tools
  -> LLM calls exact tools

The operation index would be much lighter than full tool schemas. Example card fields:

service_slug
operation_id
method + path
tag/group
summary
short description
x-aevatar-tool metadata
readOnly / destructive / requiresApproval
input field names
schema digest
required auth scopes

Benefits

• Keeps final tool catalog small and strongly typed.
• Preserves model-friendly per-endpoint argument schemas.
• Allows broad service authorization while narrowing per-turn execution.
• Existing tool infrastructure and catalog proof can remain useful.
• Write operations can be capped separately, e.g. max one write operation per turn.

Risks / tradeoffs

• Endpoint selection can miss the right operation.
• Runtime has to support dynamic tool materialization.
• Requires an operation index and selector path before catalog construction.

Mitigations for missed endpoints

• Use broad candidate recall first, e.g. top 20-50 operation cards.
• Only materialize a smaller execution set, e.g. reads <= 3, writes <= 1.
• Add a small search_connected_service_operations capability so the agent can request more candidates if current exact tools are insufficient.
• Prefer precision over recall for write/destructive operations.
• Return structured diagnostics when selection confidence is low and ask a clarifying question.

Option 2: one generic invoke tool with server-side validation

Shape

The LLM sees one generic invocation tool:

invoke_connected_service_operation({
  service_slug,
  operation_id,
  arguments
})

The operation id must come from a server-selected operation set for the current turn.

Required server checks

• operation_id is in the current selected operation set.
• Agent/channel is authorized for the service and operation.
• Arguments validate against the OpenAPI request schema.
• Method/path/query/body/header are compiled server-side, not by the model.
• Risk policy and approval requirements are enforced before execution.
• Results are recorded as auditable operation facts.

Benefits

• Final LLM tool catalog stays tiny.
• Avoids per-endpoint tool materialization.
• Works for large and changing OpenAPI surfaces.
• Can still use operation index cards for endpoint selection.

Risks / tradeoffs

• Argument quality may be worse because the LLM does not get a native per-endpoint tool schema.
• arguments can become an untyped bag unless validation is strict.
• The model may hallucinate operation ids unless constrained to selected candidates.
• More validation and repair logic moves into Aevatar runtime.

Mitigation

Show compact selected-operation input contracts in prompt/context:

operation_id: calendar.events.create
required inputs:
  - calendarId: string
  - summary: string
  - start.dateTime: RFC3339 datetime
  - end.dateTime: RFC3339 datetime
optional inputs:
  - attendees[].email
  - location
risk: write
requires_approval: true

If validation fails, return a structured validation error and let the agent repair arguments or ask the user for missing data.

Option 3: no LLM tool call; use structured command/event invocation

Shape

The model does not call provider tools. It emits a structured operation request, which the actor/runtime turns into a command/event:

{
  "kind": "connected_service_operation_request",
  "service_slug": "google-workspace",
  "operation_id": "calendar.events.create",
  "arguments": {
    "calendarId": "primary",
    "summary": "Team sync",
    "start": { "dateTime": "..." },
    "end": { "dateTime": "..." }
  }
}

Runtime validates and dispatches this as a typed command through the normal actor/event path.

Benefits

• Better aligned with Aevatar actor, command, event, continuation, approval, retry, and audit model.
• Avoids coupling connected-service execution to LLM-provider tool-calling semantics.
• Can use Protobuf command envelopes for durable operation requests and results.
• Keeps the LLM as a planner/argument drafter, not the execution authority.

Risks / tradeoffs

• Need robust structured-output parsing and repair.
• Less native schema guidance than exact LLM tools.
• More application/runtime protocol design is required.
• Must still solve endpoint selection and argument validation.

Option 4: skill-driven service workflows

Shape

Skills represent stable tasks such as:

schedule a meeting
search recent email
summarize Drive documents
send a draft email

A skill can use operation index lookup and operation invocation internally.

Benefits

• Good for stable, repeated, business-level workflows.
• Reduces per-turn endpoint-selection burden.
• Lets the skill encode multi-step service usage safely.
• Can hide OpenAPI complexity from the main agent.

Risks / tradeoffs

• Not suitable for every endpoint as a separate skill.
• Skill authoring can lag behind new service surface.
• Still needs a generic operation-selection/invocation substrate for long-tail API usage.

Option 5: require endpoint allowlists at registration/publish time

Shape

A channel registration that selects a connected service must either:

• provide explicit endpoint names, or
• opt into runtime operation selection.

Publish/validate can compute expected endpoint count, write count, and estimated schema bytes before sealing.

Benefits

• Catches oversized service selections before runtime.
• Makes owner intent explicit.
• Strongest control for sensitive agents.

Risks / tradeoffs

• Less ergonomic for broad services.
• Static allowlists may miss legitimate long-tail requests.
• Needs a good UX for browsing/selecting endpoints.

Suggested direction

My preference is a layered design rather than choosing only one mechanism:

1. Operation Index for broad recall
2. Runtime Operation Selector for per-turn narrowing
3. Server-side selected operation set as the authority
4. Invocation through either exact tools or a generic command/tool
5. Strict OpenAPI validation, risk policy, approval, audit, and actor/event execution

Short-term pragmatic fix:

• Stop treating service-level selectors as full endpoint expansion into the final tool catalog.
• If endpoint_names is empty, run operation selection or return disambiguation instead of selecting every endpoint.
• Enforce small connected-operation budgets before catalog construction.
• Catch catalog-over-budget inside the channel materializer and return diagnostics instead of letting reply generation fail with an exception.
• Add logs/diagnostics for selected endpoint count, read/write count, schema bytes, and largest schemas.

Medium-term design:

• Build an OpenAPI-derived operation index for NyxID catalog specs.
• Add a search_connected_service_operations capability or internal selector API.
• Select broad candidates from cards, then narrow to a small execution set.
• For read operations, allow iterative retrieval if the first selection misses.
• For write/destructive operations, prioritize precision and require approval.

Execution boundary preference:

• For fastest safe migration, use either exact-tool materialization after selection or one generic invoke tool with strict server-side validation.
• For long-term Aevatar architecture, prefer a structured OperationInvocationCommand / event pipeline. Tool calling can remain a provider-facing convenience, but connected-service execution should not depend on all endpoints being materialized as LLM tools.

In short: OpenAPI is the service capability catalog, operation index is the selection layer, and the per-turn execution boundary should be small, authorized, validated, and auditable. Full service APIs should not be converted into LLM tools before selection.

[alexuvlab]

option 1 feels less likely to become weird six months later. the generic invoke tool looks tidy until you’re trying to work out why it picked calendar.events.patch instead of update. i’d rather let it search first, then give it a few real tools.