th-bfc65e: OTel gen_ai agent spans on the production streaming path (SMOODEV-2328)

[brentrager]

Problem

The reference server drives every real turn through runner::run_streaming_turn (WS service + lambda), which emitted no gen_ai.* spans — only the secondary KnowledgeChatRuntime::run_turn was instrumented. So in production, nothing reached the observability studio despite OTLP being wired.

Epic SMOODEV-2328 (agent observability): every agent turn + tool call in the Rust core should flow via OTLP → otel-collector → ClickHouse → the smooai observability studio.

What changed

Both turn paths now emit the identical span shape.

Turn gen_ai.chat span — adds:

• gen_ai.agent.name (smooth-agent-chat)
• smooai.org_id (streaming path) — matches the monorepo TS chat handler's attribute exactly, so the studio groups Rust + TS turns by org
• (already present: system / model / conversation.id / aggregated token usage)

Tool gen_ai.tool child span — adds:

• gen_ai.tool.call.arguments, redacted via new telemetry::redact_tool_arguments (scrubs secret-named JSON keys, caps length)
• otel.status_code=ERROR + message on failure
• (already present: tool name / latency / is_error)

Streaming-path spans are materialized in the main body from the collected AgentEvent stream (not the spawned translator task) so they flow under the process-global OTLP subscriber.

Audit result (already done vs added)

• ✅ Already wired: OTLP exporter in telemetry.rs, gated on OTEL_EXPORTER_OTLP_ENDPOINT; init_telemetry() called in both server + lambda main.rs. Confirmed end-to-end, unchanged.
• ✅ Already present: gen_ai.chat + gen_ai.tool on the non-streaming run_turn.
• 🆕 Added: all instrumentation on the production run_streaming_turn; gen_ai.agent.name + smooai.org_id; redacted tool arguments + error status; server-side telemetry test.

Not in scope (flagged)

• No per-LLM-call inference span (chat {model} with per-call gen_ai.usage.* + finish-reason). The engine's AgentEvent stream only reports usage once (aggregated, on Completed) and carries no finish-reason. A real inference span needs a smooth-operator-core change to emit per-call usage/finish-reason → separate core→server release. Do not publish core unilaterally.
• Pre-existing broken tests (not touched here): smooth-operator-server/tests/{suggested_replies,interactions}.rs reference TurnRequest.identity_intake, removed by commit 85e5643. They fail to compile on main today (likely unnoticed — rust/ tests lack PR CI). Worth a follow-up pearl.

Tests

• smooth-operator/tests/telemetry.rs extended: asserts gen_ai.agent.name + gen_ai.tool.call.arguments.
• New smooth-operator-server/tests/telemetry.rs: drives a real run_streaming_turn, asserts smooai.org_id + redacted tool args on the streaming path.
• Redaction unit tests in telemetry.rs.
• cargo fmt --check, cargo clippy (clean on changed files), and all runtime + server telemetry/lib/integration tests pass (validated against the core commit main targets — see note below).

Build note: the local sibling smooth-operator-core checkout is ahead of what this repo's main targets (added an images field), so validation used an isolated core worktree at the commit main compiles against. Changeset added (@smooai/smooth-operator patch).

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: 4c29158

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@smooai/smooth-operator	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR