Repository navigation
[Enhancement]: Account for LLM calls made inside native routing plugins #1002
Description
Affected area
- Native plugins
- Observability and ATIF
Problem
A native routing plugin can make judge or classifier calls before returning one answer. Relay
prices the answer, but those internal calls are not part of the parent call's cost summary.
Relay can already price the internal calls if the plugin emits each one as an LLM scope. I tested
that on current main: ATIF correctly summed a $0.11 routing call and a $0.00045 answer to
$0.11045. The problem is that ATIF also turned both calls into normal source="agent" steps.
The routing call was never visible to the agent, so it should not appear as another agent message.
The other existing paths do not close the gap:
- A metric mark avoids the extra ATIF step, but Relay does not price it or include it in ATIF totals.
- Calling
LlmNext::calltwice reaches the provider twice, but Relay still emits one LLM start and
end and prices only the returned response. - Optimization contributions from request intercepts arrive before the routing calls have run.
Desired behavior
Provide a supported way for a native execution plugin to report internal LLM work so Relay can:
- price it with the existing pricing catalog;
- keep answer cost and internal-call cost separate;
- include both in an observed total when usage and pricing are available; and
- keep internal calls out of the ATIF conversation while retaining their cost in aggregate metrics.
If usage or pricing is missing, the total should remain partial rather than treating the call as
free. Calculating a savings baseline is separate from this issue.
Switchyard still owns emitting the model, provider, usage, and timing for its calls. This issue is
only about the Relay surface and export behavior needed to account for them.
Related: Relay #389,
Relay #814,
Switchyard #528, and
Switchyard #629.
Activity
willkill07 commentedon Sep 8, 2026
First, the overall issue seems inflated -- Did you do Child LLM scopes or Call-scoped optimization evidence? This if the issue evaluated only one, then please restrict this issue's scope. There's a hell of a lot going on here.
Two important distinctions also have to be made here:
a. Option 1: an abstraction over what the agent interacts with
b. Option 2: an exact provider request+response regardless of ownership/visibility semantics
a. Option 1: only what the agent sees.
b. Option 2: every fine-grained detail of the total runtime.
In general, Option 2 would inflate the number of events and send each new event to every registered subscriber. Does this make sense? For the scope of this issue and for observability overall, I believe Option 1 is the most truthful. In my opinion, child LLM scopes with ATIF should simply be ignored. But for accurate cost reconciliation, we must have child costs get aggregated to the parent. This is a limitation of ATIF. But if you think about it, there is no clear correct answer here.
This is confusing to me. Switchyard can create the scopes exactly when it needs to. We should not allow plugins to alter the source of truth of events. If they not have any sort of event recorder or callback, then they should consider adding one rather than emit static events. Realtime events and the ability to react instantly are probably things they will need eventually.
Runtime contract and binding impact
If this is true, then this issue should live under Switchyard? It is unclear as to what changes there are to the runtime contract and binding impact itself here. I was hoping this section would be more definitive.
Details that need to be settled
Most of these are not Relay issues -- they are Switchyard issues. And again, in the scope of ATIF, Router LLM calls should not be present in ATIF due to opinionated perspective of Option 1 I declared above.
r.e. Cached input. Switchyard's normalized
input_tokensexcludes cached tokens. -- this makes sense simply because by default I would imagine a different model is in use and that there would be no caching. But really, this is dependent on whatever Switchyard does. They need to account for appropriate token mix when determining their costs. Relay will report whatever is given.afourniernv commentedon Sep 8, 2026
Thanks, this helped clarify the boundary. I was trying to make ATIF carry routing detail that fits better in OpenTelemetry.
The remaining Switchyard work is tracked in NVIDIA-NeMo/Switchyard#299.