Skip to content

fix(agentscope): improve data attached to spans#581

Merged
Abhijeet Prasad (AbhiPrasad) merged 5 commits into
mainfrom
fix/agentscope-skill-compliance
Jul 16, 2026
Merged

fix(agentscope): improve data attached to spans#581
Abhijeet Prasad (AbhiPrasad) merged 5 commits into
mainfrom
fix/agentscope-skill-compliance

Conversation

@starfolkai

@starfolkai starfolkai Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Align the AgentScope integration with the SDK integration instrumentation requirements and replace mock-heavy coverage with cassette-backed tests across the supported AgentScope 1.x and 2.x matrix.

Changes

Span shape and metadata

  • Move model tool definitions, tool_choice, and structured-output configuration out of LLM span input and into metadata.
  • Normalize provider names to canonical pricing slugs (openai, anthropic, google, dashscope, ollama, and trinity).
  • Keep request metadata to an explicit configuration allowlist instead of forwarding arbitrary kwargs that may contain credentials or transport details.
  • Materialize inline multimodal message content through the shared _normalize_chat_messages helper.
  • Continue creating integration spans through the local start_span() helper so every span carries agentscope-auto instrumentation provenance.

Metrics and streaming

  • Parse usage through the shared OpenAI-style usage helpers, including nested cached/reasoning token details when reported.
  • Derive total tokens from prompt and completion tokens when AgentScope does not expose a total directly.
  • Record time_to_first_token and keep streaming spans open until the returned iterator is consumed.
  • Add a version-gated AgentScope 1.x patcher for OpenAIChatModel._parse_openai_stream_completion_response. AgentScope 1.x consumes OpenAI's terminal usage-only chunk without yielding it, so the patcher captures those metrics without changing the user-visible stream and passes them to the outer model span through a ContextVar.
  • Leave AgentScope 2.x on its native path because it surfaces the terminal usage chunk itself.

Errors and lifecycle behavior

  • Log exception instances directly with span.log(error=exc) while preserving provider exception propagation.
  • Verify repeated setup calls do not add another wrapper layer, including compatibility with AgentScope 1.x's own decorated methods.

Tests and cassettes

  • Replace three fake/mock streaming tests with real VCR-backed AgentScope coverage.
  • Add per-version streaming cassettes and assertions for accumulated output, TTFT, prompt/completion/total tokens, and one LLM span per model call.
  • Add per-version provider-error cassettes and verify the exception originates from OpenAI, preventing missing-cassette/VCR exceptions from falsely satisfying the propagation test.
  • Tighten existing span assertions for canonical provider metadata, token metrics, tool placement, parent/child relationships, and instrumentation provenance.
  • Extend the auto-instrument subprocess test to assert metadata.provider == "openai".

Test plan

  • nox -s "test_agentscope(latest)" -- --vcr-record=none
  • nox -s "test_agentscope(1.0.0)" -- --vcr-record=none
  • Focused recording and playback for streaming and provider-error tests on both matrix versions
  • nox -s pylint
  • Targeted pre-commit hooks, including stale-cassette validation

Created by abhijeet

Slack thread

starfolkbot and others added 5 commits July 16, 2026 17:00
- move tools/tool_choice/structured_model from LLM span input into
  metadata.tools (per SKILL span design)
- log the Exception instance instead of str(exc) so Braintrust receives
  the raw error, not a pre-formatted string
- record time_to_first_token for streaming model calls
- replace class-name-stripping with an explicit provider map
  (openai / anthropic / google / dashscope / ollama / trinity)
- materialize inline multimodal message parts via _normalize_chat_messages
- restrict metadata kwarg pass-through to an explicit config allowlist
  to avoid leaking credentials passed as kwargs
- drop fake-based streaming tests, tighten cassette assertions
  (provider, metrics, metadata.tools), and add new VCR tests for
  streaming and error propagation (requires cassette re-record)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…l plumbing

- _extract_metrics now delegates to _parse_openai_usage_metrics + _try_to_dict
  (via the shared helper), so pydantic Usage objects normalize correctly and
  we pick up cached/reasoning token subkeys for free
- _is_supported_metric_value replaces the inline bool/numeric guard
- introduce _kw_or_pos to collapse four "kwargs.get else args[idx]" stanzas
- iterate kwargs (not the allowlist) for _METADATA_CONFIG_KEYS extraction
- drop unnecessary try/except around getattr in _field_value
- _deferred_stream_trace now takes an on_first_chunk callback so tool streams
  no longer accept an unused time_to_first_token param; inline the final-chunk
  logger closure in _model_call_wrapper
- short-circuit _capture_openai_stream_usage_wrapper: only run _extract_metrics
  when the chunk actually carries a usage field (OpenAI only sets it on the
  terminal chunk)
- add _run_agent test helper to remove duplicated
  "agent.reply if HAS_AGENT_REPLY_API else agent(...)" conditionals

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@AbhiPrasad Abhijeet Prasad (AbhiPrasad) changed the title fix(agentscope): align integration with sdk-integrations SKILL fix(agentscope): improve data attached to spans Jul 16, 2026
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) marked this pull request as ready for review July 16, 2026 18:11
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) merged commit 64cc920 into main Jul 16, 2026
82 checks passed
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) deleted the fix/agentscope-skill-compliance branch July 16, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants