Skip to content

fix(crewai): add provider metadata, route tools to metadata, drop eager serialization#587

Merged
Abhijeet Prasad (AbhiPrasad) merged 4 commits into
mainfrom
fix/crewai-skill-alignment
Jul 17, 2026
Merged

fix(crewai): add provider metadata, route tools to metadata, drop eager serialization#587
Abhijeet Prasad (AbhiPrasad) merged 4 commits into
mainfrom
fix/crewai-skill-alignment

Conversation

@starfolkai

@starfolkai starfolkai Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Align the CrewAI integration with .agents/skills/sdk-integrations/SKILL.md. Audit turned up five issues:

  • Missing metadata.provider on crewai.llm spans — spec requires every llm span to carry both metadata.model and metadata.provider. Added _provider_from_source(), which reads the provider string off the emitting CrewAI LLM object ("openai" / "anthropic" / "bedrock" / ...).
  • Tools in input instead of metadata.tools — spec says tool definitions live in metadata.tools, not input. Moved both the LLM-call tools and the agent tools onto metadata.
  • Token double-counting on crewai.llm — the previous rule only skipped tokens when the LiteLLM integration was patched, but CrewAI 1.x routes gpt-* to a native openai client, claude-* to a native anthropic client, etc. When those integrations are patched (as auto_instrument() does by default), they emit a leaf span with tokens and crewai.llm double-counted at every ancestor in the trace-tree rollup. The SKILL is explicit here: "Do not add 'if OpenAI is patched, skip metrics' checks — define clear ownership instead." CrewAI is always an orchestration layer that delegates to a provider SDK, so the clear rule is: crewai.llm never owns tokens. Same pattern pydantic_ai already uses for its wrapper spans. Dropped _litellm_owns_leaf_span + the token-name/prefix maps + the extra_metrics plumbing on _end_span accordingly.
  • Excess serialization — dropped eager _try_to_dict / _normalize_output / _normalize_tools calls across kickoff, task, agent, LLM, and tool output paths. bt_json._to_bt_safe at log time already handles Pydantic v2 (model_dump), Pydantic v1 (dict), and dataclasses; the eager pre-pass was wasted work (same pattern as fix(claude_agent_sdk): add provider metadata, drop deep-copy serialization, allowlist hook fields #585 for claude_agent_sdk and fix(anthropic): drop redundant bt_safe_deep_copy passes in tracing #583 for anthropic).

All other metadata extractors (_causal_metadata, _crew_metadata, _task_metadata, _LLM_CONFIG_FIELDS) were already allowlist-based — no changes needed there.

Test plan

  • Extended test_kickoff_llm_event_tree_parents_and_shape to emit with a real LLM source and assert metadata.provider == "openai".
  • Added test_llm_tools_route_to_metadata_not_input (positive-in-metadata + negative-not-in-input).
  • Replaced the two conditional token tests with a single test_llm_never_emits_token_metrics that asserts no token key (tokens, prompt_tokens, completion_tokens, prompt_cached_tokens, completion_reasoning_tokens) leaks onto crewai.llm regardless of which provider integrations are patched.
  • No new cassettes. The test file docstring documents why VCR is impractical for CrewAI (pytest-vcr + httpcore + CrewAI's native OpenAICompletion model_post_init interaction bug). Direct-event tests remain the source of truth here; the LiteLLM mock_response smoke test still exercises the full crew.kickoff() path.
  • cd py && BRAINTRUST_TEST_PACKAGE_VERSION=latest pytest src/braintrust/integrations/crewai/test_crewai.py -v — 17 passed
  • cd py && nox -s pylint — success
  • pre-commit run on the changed files — passed (ruff format, ruff check, codespell, EOF/trailing-whitespace)

🤖 Generated with Claude Code

Created by abhijeet

Slack thread

starfolkbot and others added 4 commits July 17, 2026 16:22
…er serialization

Align the CrewAI integration with .agents/skills/sdk-integrations/SKILL.md.

- Add metadata.provider on crewai.llm spans, read from LLM.provider on the
  event source. Every llm span must carry both metadata.model AND
  metadata.provider per the instrumentation spec.
- Route tool definitions into metadata.tools (both LLMCallStartedEvent
  tools and AgentExecutionStartedEvent tools). Tools no longer leak into
  input.
- Drop eager _try_to_dict / _normalize_output / _normalize_tools passes
  in kickoff/task/agent/llm/tool output paths. bt_json at log time
  already handles Pydantic v2/v1 and dataclasses, so the eager pass is
  wasted work.
- Narrow _agent_metadata's agent_llm read: previously ``str(llm)`` was a
  fallback, which dumps the pydantic repr including api_key / api_base /
  client_params. Now the allowlist reads only the model name.
- Extend the existing direct-event tests: assert metadata.provider on
  the kickoff/llm tree and add a positive+negative check that tools
  route to metadata (not input). No new cassettes — the file docstring
  documents why VCR is impractical for CrewAI.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ns them)

Previous behavior only skipped token metrics when the LiteLLM integration
was patched. That missed the native-provider paths — CrewAI 1.x routes
gpt-* to a native openai client, claude-* to a native anthropic client,
etc. When those integrations are patched (as ``auto_instrument()`` does
by default), they emit a leaf span with tokens and ``crewai.llm``
double-counted at every ancestor in the trace-tree rollup.

The sdk-integrations SKILL is explicit here: "Do not add 'if OpenAI is
patched, skip metrics' checks — define clear ownership instead." CrewAI
is always an orchestration layer that delegates to a provider SDK, so
the clear rule is: ``crewai.llm`` never owns tokens. Same pattern
pydantic_ai already uses for its wrapper spans.

- Drop ``_litellm_owns_leaf_span`` + ``_TOKEN_NAME_MAP`` / ``_TOKEN_PREFIX_MAP``.
- ``on_llm_completed`` no longer computes ``extra_metrics``; timing and
  ``time_to_first_token`` remain.
- Drop the now-unused ``extra_metrics`` parameter from ``_end_span`` /
  ``_end_span_by_event_id``.
- Replace the two conditional token tests with a single
  ``test_llm_never_emits_token_metrics`` that asserts no token key
  (``tokens``, ``prompt_tokens``, ``completion_tokens``,
  ``prompt_cached_tokens``, ``completion_reasoning_tokens``) leaks onto
  ``crewai.llm`` regardless of which provider integrations are patched.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The module docstring already documents the wrapper-only token rule and
the allowlist rationale; per-callsite restatements added noise. Keeps
the load-bearing WHY comments (str(llm) API-key leak; ``list(dict)``
yielding names only) and drops the rest.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) merged commit 0b054a6 into main Jul 17, 2026
82 checks passed
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) deleted the fix/crewai-skill-alignment branch July 17, 2026 16:57
starfolkai Bot pushed a commit that referenced this pull request Jul 17, 2026
…ization

Align the LiteLLM integration with .agents/skills/sdk-integrations/SKILL.md.
Audit against the SKILL surfaced four issues:

- **`metadata.provider` was the framework name.** SKILL §Completion-style
  explicitly says LiteLLM must set `metadata.provider` to the underlying
  provider (openai / cohere / openrouter / ...), not "litellm". Fixed by
  routing every span through `_resolve_provider(model)` which calls
  `litellm.get_llm_provider(model)`; falls back to `"litellm"` only if the
  lookup fails. `@lru_cache(maxsize=256)` on the resolver so the hot path
  doesn't re-dispatch through litellm's routing table per call.
- **Excess serialization.** Every non-streaming wrapper called
  `_try_to_dict(response)` on the full pydantic response then indexed
  into it — but `bt_json` already serializes at log time. Replaced with
  direct attribute reads via a shared `_get_field(obj, key, default)`
  helper that works on both dicts and SDK objects (openai / anthropic /
  cohere had all been open-coding this). Streaming paths left intact.
- **Dead code.** `_is_litellm_patched` was only used by crewai until #587
  moved crewai to the "wrapper never owns tokens" pattern; no remaining
  callers anywhere. Removed the helper and its two subprocess tests.
- **Excessive comments.** Trimmed the huge `wrap_litellm` /
  `_is_litellm_patched` docstrings and the "wrapt wrapper for litellm.X"
  one-liner on every wrapper.

Also lifted the rerank output cap (`results[:100]`) into
`integrations/utils.RERANK_OUTPUT_MAX_RESULTS` and documented the
"rerank exception to no-silent-caps" rule in the SKILL — cohere carries
the same magic number and can adopt the shared constant in a follow-up.

## Test plan

- [x] Updated existing VCR-backed tests to assert the underlying
      provider (`openai`, `cohere`, `openrouter`) instead of `"litellm"`.
      No cassette re-recording (HTTP requests unchanged; span-shape
      assertions only).
- [x] Added positive `metadata["provider"] == "openrouter"` assertion
      to the existing OpenRouter cassette test as coverage for the new
      resolver.
- [x] Dropped the two `_is_litellm_patched` subprocess tests alongside
      the helper.
- [x] **No new mocks/fakes, no cassette re-recording.**
- [x] `ruff check` + `ruff format --check` on touched files — clean.
- [ ] Reviewer to run: `cd py && nox -s "test_litellm(latest)"` and
      `cd py && nox -s "test_litellm(1.74.0)"` (author's box has a
      broken Python 3.14 install).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Abhijeet Prasad (AbhiPrasad) pushed a commit that referenced this pull request Jul 17, 2026
…ization (#594)

## Summary

Align the LiteLLM integration with
`.agents/skills/sdk-integrations/SKILL.md`. Audit turned up four issues:

- **Wrong `metadata.provider`.** SKILL §Completion-style explicitly says
LiteLLM must set `metadata.provider` to the underlying provider
(`openai` / `cohere` / `openrouter` / …), not the framework name. Fixed
by routing every span through `_resolve_provider(model)` which calls
`litellm.get_llm_provider(model)`; falls back to `"litellm"` only when
the lookup fails. `@lru_cache(maxsize=256)` on the resolver so the hot
path doesn't re-dispatch through litellm's routing table per call.
- **Excess serialization.** Every non-streaming wrapper called
`_try_to_dict(response)` on the full pydantic response then indexed into
it — but `bt_json` already serializes at log time. Replaced with direct
attribute reads via a new shared `_get_field(obj, key, default)` helper
in `integrations/utils.py` that works on both dicts and SDK objects
(openai / anthropic / cohere had all been open-coding this pattern).
Streaming paths left intact.
- **Dead code.** `_is_litellm_patched` was only used by crewai until
#587 moved crewai to the "wrapper never owns tokens" pattern; no
remaining callers anywhere. Removed the helper and its two subprocess
tests.
- **Excessive comments.** Trimmed the multi-example `wrap_litellm` /
`_is_litellm_patched` docstrings and the `"""wrapt wrapper for
litellm.X."""` one-liner on every wrapper.

Also lifted the rerank output cap (`results[:100]`) into
`integrations/utils.RERANK_OUTPUT_MAX_RESULTS` and documented the
"rerank exception to no-silent-caps" rule in the SKILL. Cohere carries
the same magic number and can adopt the shared constant in a follow-up.

## Test plan

- [x] Updated existing VCR-backed tests to assert the underlying
provider (`openai`, `cohere`, `openrouter`) instead of `"litellm"`. **No
cassette re-recording** — HTTP requests unchanged, span-shape assertions
only.
- [x] Added positive `metadata["provider"] == "openrouter"` assertion to
the existing OpenRouter cassette test as coverage for the new resolver.
- [x] Dropped the two `_is_litellm_patched` subprocess tests alongside
the helper.
- [x] **No new mocks/fakes, no cassette re-recording.**
- [x] `ruff check` + `ruff format --check` on touched files — clean.
- [ ] Reviewer: `cd py && nox -s "test_litellm(latest)"` and `cd py &&
nox -s "test_litellm(1.74.0)"` (author's box has a broken Python 3.14
install).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- sfk:created-approved-by -->
Created by abhijeet

<!-- sfk:slack-thread -->
[Slack
thread](https://starfolkai.slack.com/archives/C0AQDETAVT3/p1784318916727759?thread_ts=1784318916.727759&cid=C0AQDETAVT3)

---------

Co-authored-by: Starfolk <noreply@starfolk.ai>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants