fix(crewai): add provider metadata, route tools to metadata, drop eager serialization#587
Merged
Merged
Conversation
…er serialization Align the CrewAI integration with .agents/skills/sdk-integrations/SKILL.md. - Add metadata.provider on crewai.llm spans, read from LLM.provider on the event source. Every llm span must carry both metadata.model AND metadata.provider per the instrumentation spec. - Route tool definitions into metadata.tools (both LLMCallStartedEvent tools and AgentExecutionStartedEvent tools). Tools no longer leak into input. - Drop eager _try_to_dict / _normalize_output / _normalize_tools passes in kickoff/task/agent/llm/tool output paths. bt_json at log time already handles Pydantic v2/v1 and dataclasses, so the eager pass is wasted work. - Narrow _agent_metadata's agent_llm read: previously ``str(llm)`` was a fallback, which dumps the pydantic repr including api_key / api_base / client_params. Now the allowlist reads only the model name. - Extend the existing direct-event tests: assert metadata.provider on the kickoff/llm tree and add a positive+negative check that tools route to metadata (not input). No new cassettes — the file docstring documents why VCR is impractical for CrewAI. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ns them) Previous behavior only skipped token metrics when the LiteLLM integration was patched. That missed the native-provider paths — CrewAI 1.x routes gpt-* to a native openai client, claude-* to a native anthropic client, etc. When those integrations are patched (as ``auto_instrument()`` does by default), they emit a leaf span with tokens and ``crewai.llm`` double-counted at every ancestor in the trace-tree rollup. The sdk-integrations SKILL is explicit here: "Do not add 'if OpenAI is patched, skip metrics' checks — define clear ownership instead." CrewAI is always an orchestration layer that delegates to a provider SDK, so the clear rule is: ``crewai.llm`` never owns tokens. Same pattern pydantic_ai already uses for its wrapper spans. - Drop ``_litellm_owns_leaf_span`` + ``_TOKEN_NAME_MAP`` / ``_TOKEN_PREFIX_MAP``. - ``on_llm_completed`` no longer computes ``extra_metrics``; timing and ``time_to_first_token`` remain. - Drop the now-unused ``extra_metrics`` parameter from ``_end_span`` / ``_end_span_by_event_id``. - Replace the two conditional token tests with a single ``test_llm_never_emits_token_metrics`` that asserts no token key (``tokens``, ``prompt_tokens``, ``completion_tokens``, ``prompt_cached_tokens``, ``completion_reasoning_tokens``) leaks onto ``crewai.llm`` regardless of which provider integrations are patched. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The module docstring already documents the wrapper-only token rule and the allowlist rationale; per-callsite restatements added noise. Keeps the load-bearing WHY comments (str(llm) API-key leak; ``list(dict)`` yielding names only) and drops the rest. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Abhijeet Prasad (AbhiPrasad)
approved these changes
Jul 17, 2026
Abhijeet Prasad (AbhiPrasad)
enabled auto-merge (squash)
July 17, 2026 16:57
6 tasks
starfolkai Bot
pushed a commit
that referenced
this pull request
Jul 17, 2026
…ization Align the LiteLLM integration with .agents/skills/sdk-integrations/SKILL.md. Audit against the SKILL surfaced four issues: - **`metadata.provider` was the framework name.** SKILL §Completion-style explicitly says LiteLLM must set `metadata.provider` to the underlying provider (openai / cohere / openrouter / ...), not "litellm". Fixed by routing every span through `_resolve_provider(model)` which calls `litellm.get_llm_provider(model)`; falls back to `"litellm"` only if the lookup fails. `@lru_cache(maxsize=256)` on the resolver so the hot path doesn't re-dispatch through litellm's routing table per call. - **Excess serialization.** Every non-streaming wrapper called `_try_to_dict(response)` on the full pydantic response then indexed into it — but `bt_json` already serializes at log time. Replaced with direct attribute reads via a shared `_get_field(obj, key, default)` helper that works on both dicts and SDK objects (openai / anthropic / cohere had all been open-coding this). Streaming paths left intact. - **Dead code.** `_is_litellm_patched` was only used by crewai until #587 moved crewai to the "wrapper never owns tokens" pattern; no remaining callers anywhere. Removed the helper and its two subprocess tests. - **Excessive comments.** Trimmed the huge `wrap_litellm` / `_is_litellm_patched` docstrings and the "wrapt wrapper for litellm.X" one-liner on every wrapper. Also lifted the rerank output cap (`results[:100]`) into `integrations/utils.RERANK_OUTPUT_MAX_RESULTS` and documented the "rerank exception to no-silent-caps" rule in the SKILL — cohere carries the same magic number and can adopt the shared constant in a follow-up. ## Test plan - [x] Updated existing VCR-backed tests to assert the underlying provider (`openai`, `cohere`, `openrouter`) instead of `"litellm"`. No cassette re-recording (HTTP requests unchanged; span-shape assertions only). - [x] Added positive `metadata["provider"] == "openrouter"` assertion to the existing OpenRouter cassette test as coverage for the new resolver. - [x] Dropped the two `_is_litellm_patched` subprocess tests alongside the helper. - [x] **No new mocks/fakes, no cassette re-recording.** - [x] `ruff check` + `ruff format --check` on touched files — clean. - [ ] Reviewer to run: `cd py && nox -s "test_litellm(latest)"` and `cd py && nox -s "test_litellm(1.74.0)"` (author's box has a broken Python 3.14 install). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Abhijeet Prasad (AbhiPrasad)
pushed a commit
that referenced
this pull request
Jul 17, 2026
…ization (#594) ## Summary Align the LiteLLM integration with `.agents/skills/sdk-integrations/SKILL.md`. Audit turned up four issues: - **Wrong `metadata.provider`.** SKILL §Completion-style explicitly says LiteLLM must set `metadata.provider` to the underlying provider (`openai` / `cohere` / `openrouter` / …), not the framework name. Fixed by routing every span through `_resolve_provider(model)` which calls `litellm.get_llm_provider(model)`; falls back to `"litellm"` only when the lookup fails. `@lru_cache(maxsize=256)` on the resolver so the hot path doesn't re-dispatch through litellm's routing table per call. - **Excess serialization.** Every non-streaming wrapper called `_try_to_dict(response)` on the full pydantic response then indexed into it — but `bt_json` already serializes at log time. Replaced with direct attribute reads via a new shared `_get_field(obj, key, default)` helper in `integrations/utils.py` that works on both dicts and SDK objects (openai / anthropic / cohere had all been open-coding this pattern). Streaming paths left intact. - **Dead code.** `_is_litellm_patched` was only used by crewai until #587 moved crewai to the "wrapper never owns tokens" pattern; no remaining callers anywhere. Removed the helper and its two subprocess tests. - **Excessive comments.** Trimmed the multi-example `wrap_litellm` / `_is_litellm_patched` docstrings and the `"""wrapt wrapper for litellm.X."""` one-liner on every wrapper. Also lifted the rerank output cap (`results[:100]`) into `integrations/utils.RERANK_OUTPUT_MAX_RESULTS` and documented the "rerank exception to no-silent-caps" rule in the SKILL. Cohere carries the same magic number and can adopt the shared constant in a follow-up. ## Test plan - [x] Updated existing VCR-backed tests to assert the underlying provider (`openai`, `cohere`, `openrouter`) instead of `"litellm"`. **No cassette re-recording** — HTTP requests unchanged, span-shape assertions only. - [x] Added positive `metadata["provider"] == "openrouter"` assertion to the existing OpenRouter cassette test as coverage for the new resolver. - [x] Dropped the two `_is_litellm_patched` subprocess tests alongside the helper. - [x] **No new mocks/fakes, no cassette re-recording.** - [x] `ruff check` + `ruff format --check` on touched files — clean. - [ ] Reviewer: `cd py && nox -s "test_litellm(latest)"` and `cd py && nox -s "test_litellm(1.74.0)"` (author's box has a broken Python 3.14 install). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- sfk:created-approved-by --> Created by abhijeet <!-- sfk:slack-thread --> [Slack thread](https://starfolkai.slack.com/archives/C0AQDETAVT3/p1784318916727759?thread_ts=1784318916.727759&cid=C0AQDETAVT3) --------- Co-authored-by: Starfolk <noreply@starfolk.ai> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Align the CrewAI integration with
.agents/skills/sdk-integrations/SKILL.md. Audit turned up five issues:metadata.provideroncrewai.llmspans — spec requires everyllmspan to carry bothmetadata.modelandmetadata.provider. Added_provider_from_source(), which reads theproviderstring off the emitting CrewAILLMobject ("openai"/"anthropic"/"bedrock"/ ...).inputinstead ofmetadata.tools— spec says tool definitions live inmetadata.tools, notinput. Moved both the LLM-calltoolsand the agenttoolsontometadata.crewai.llm— the previous rule only skipped tokens when the LiteLLM integration was patched, but CrewAI 1.x routesgpt-*to a nativeopenaiclient,claude-*to a nativeanthropicclient, etc. When those integrations are patched (asauto_instrument()does by default), they emit a leaf span with tokens andcrewai.llmdouble-counted at every ancestor in the trace-tree rollup. The SKILL is explicit here: "Do not add 'if OpenAI is patched, skip metrics' checks — define clear ownership instead." CrewAI is always an orchestration layer that delegates to a provider SDK, so the clear rule is:crewai.llmnever owns tokens. Same patternpydantic_aialready uses for its wrapper spans. Dropped_litellm_owns_leaf_span+ the token-name/prefix maps + theextra_metricsplumbing on_end_spanaccordingly._try_to_dict/_normalize_output/_normalize_toolscalls across kickoff, task, agent, LLM, and tool output paths.bt_json._to_bt_safeat log time already handles Pydantic v2 (model_dump), Pydantic v1 (dict), and dataclasses; the eager pre-pass was wasted work (same pattern as fix(claude_agent_sdk): add provider metadata, drop deep-copy serialization, allowlist hook fields #585 for claude_agent_sdk and fix(anthropic): drop redundant bt_safe_deep_copy passes in tracing #583 for anthropic).All other metadata extractors (
_causal_metadata,_crew_metadata,_task_metadata,_LLM_CONFIG_FIELDS) were already allowlist-based — no changes needed there.Test plan
test_kickoff_llm_event_tree_parents_and_shapeto emit with a realLLMsource and assertmetadata.provider == "openai".test_llm_tools_route_to_metadata_not_input(positive-in-metadata + negative-not-in-input).test_llm_never_emits_token_metricsthat asserts no token key (tokens,prompt_tokens,completion_tokens,prompt_cached_tokens,completion_reasoning_tokens) leaks ontocrewai.llmregardless of which provider integrations are patched.OpenAICompletionmodel_post_initinteraction bug). Direct-event tests remain the source of truth here; the LiteLLMmock_responsesmoke test still exercises the fullcrew.kickoff()path.cd py && BRAINTRUST_TEST_PACKAGE_VERSION=latest pytest src/braintrust/integrations/crewai/test_crewai.py -v— 17 passedcd py && nox -s pylint— successpre-commit runon the changed files — passed (ruff format, ruff check, codespell, EOF/trailing-whitespace)🤖 Generated with Claude Code
Created by abhijeet
Slack thread