From 57450ebb274fb6d09075d0d97957d250bbcc349d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stefan=20Br=C3=B6nner?= Date: Sun, 9 Aug 2026 07:06:32 +0200 Subject: [PATCH] docs: restore showcase and fix example links Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e9e4fe78-6589-4a59-a9ac-137849f90421 --- docs/demo/hero-report.html | 580 ++++++++++-------- docs/getting-started/ab-testing-servers.md | 2 +- docs/getting-started/comparing.md | 4 +- docs/getting-started/iterations.md | 2 +- docs/getting-started/skills.md | 2 +- docs/how-to/complete-example.md | 59 +- docs/how-to/test-cli-tools.md | 2 +- tests/integration/README.md | 138 ++--- tests/showcase/conftest.py | 28 + .../concise.md} | 0 .../detailed.md} | 0 .../friendly.md} | 0 tests/showcase/test_hero.py | 236 +++++++ 13 files changed, 651 insertions(+), 402 deletions(-) create mode 100644 tests/showcase/conftest.py rename tests/showcase/{prompts/concise.agent.md => instructions/concise.md} (100%) rename tests/showcase/{prompts/detailed.agent.md => instructions/detailed.md} (100%) rename tests/showcase/{prompts/friendly.agent.md => instructions/friendly.md} (100%) create mode 100644 tests/showcase/test_hero.py diff --git a/docs/demo/hero-report.html b/docs/demo/hero-report.html index e9a675f..5c36199 100644 --- a/docs/demo/hero-report.html +++ b/docs/demo/hero-report.html @@ -306,7 +306,9 @@ .px-3 { padding-left: 0.75rem; padding-right: 0.75rem; } .px-5 { padding-left: 1.25rem; padding-right: 1.25rem; } .px-6 { padding-left: 1.5rem; padding-right: 1.5rem; } +.py-0\.5 { padding-top: 0.125rem; padding-bottom: 0.125rem; } .py-1 { padding-top: 0.25rem; padding-bottom: 0.25rem; } +.py-1\.5 { padding-top: 0.375rem; padding-bottom: 0.375rem; } .py-3 { padding-top: 0.75rem; padding-bottom: 0.75rem; } .py-4 { padding-top: 1rem; padding-bottom: 1rem; } .py-8 { padding-top: 2rem; padding-bottom: 2rem; } @@ -467,6 +469,7 @@ .hover\:border-white\/20:hover { border-color: rgba(255, 255, 255, 0.2); } .hover\:border-primary\/30:hover { border-color: rgba(64, 81, 181, 0.3); } .hover\:border-primary\/50:hover { border-color: rgba(64, 81, 181, 0.5); } +.hover\:bg-primary\/5:hover { background-color: rgba(64, 81, 181, 0.05); } .hover\:text-text-light:hover { color: var(--color-text-light); } .hover\:text-blue-400:hover { color: var(--color-blue); } .hover\:text-accent:hover { color: var(--color-accent); } @@ -990,7 +993,6 @@ /* ==================== Overlay ==================== */ #overlay.active { display: flex !important; } -#diagram-hover-popup.active { display: block !important; } #overlay-mermaid svg { max-width: 100%; max-height: 100%; @@ -998,20 +1000,20 @@ height: auto; min-height: 60vh; } -

Core banking tests β€” parametrized across benchmark agents.

βœ“All Passed
February 23, 2026 at 11:17 AMπŸ“„ test_hero.py18 tests225.4s830–3,146 tokπŸ§ͺ $0.0454πŸ€– $0.0484πŸ’° $0.0938
πŸ€–AI Analysis
-
Recommended for Deploy
-
default
-
Achieves a 100% pass rate across the largest test surface (12 tests) with reliable tool-first behavior and consistent multi-turn handling, at a total cost of $0.023589.
+

Demonstrate direct MCP tool selection.

⚠1 Model Missing Pricing
βœ“All Passed
August 09, 2026 at 07:05 AMπŸ“„ test_hero.py9 tests134.9s26,414–64,146 tokπŸ§ͺ $0.3717πŸ’° $0.3717
πŸ€–AI Analysis
+
Recommended Configuration
+
banking-basic
+
Completes the direct MCP balance lookup with correct tool grounding and concise final output. Cost is unavailable because claude-opus-5 has no pricing data in this run.
100%Pass Rate
-
$0.023589Total Cost
-
17,014Tokens
+
N/ACost
+
41,396Tokens
-
18
+
9
Total Tests
@@ -1019,136 +1021,229 @@
Failures
-
4
-
Eval Configs
+
9
+
Agents
-
3.3
+
5.2
Avg Turns
-

Comparative Analysis

+

Comparative Analysis

Why the winner wins

    -
  • Coverage advantage: The winner (default) passed 12 tests versus 2 tests each for concise, friendly, and detailed, demonstrating robustness across core banking, multi‑turn sessions, advisory styles, and skill-augmented advice.
    • -
  • Tool discipline: Consistently follows β€œtool-first” instructions (balances before advice, transfers via tools, graceful handling of insufficient funds) with no deviations.
    • -
  • Session reliability: Correctly carries context across multi-turn workflows (balance β†’ transfer) without redundant clarifications or missed tool calls.
    • -
  • Cost-effective at scale: While per-test cost is not directly comparable across configs due to different test counts, the winner maintains low total cost while handling the widest scenario set.
    • +
  • Correct minimal orchestration: banking-basic used exactly the required banking-get_balance MCP tool and answered from the tool output without extra probing.
    • +
  • Clean grounding behavior: The assistant did not infer or fabricate the balance; it returned the formatted $1,500.00 value from the tool response.
    • +
  • Lowest-complexity passing path: All configurations passed, but this one demonstrates the most direct tool-to-answer path for a single-account query.

Notable patterns

    -
  • Concise prompts minimize tokens: concise produces the lowest token usage per advisory response while still calling the necessary tools.
    • -
  • Detailed prompts expand reasoning, not tool usage: detailed adds explanatory depth without increasing tool coverageβ€”useful for UX, not for operational accuracy.
    • -
  • Model parity on correctness: Both gpt-5-mini and gpt-4.1 achieve perfect pass rates; differences are primarily in verbosity and cost, not correctness.
    • +
  • All banking scenarios passed: Every eval grounded its answer in MCP output before responding.
    • +
  • Advice tasks consistently used balance data: Both model-comparison evals called banking-get_all_balances before recommending a savings action.
    • +
  • Transaction-history lookups were used only for broader advice: The concise, detailed, and friendly account-review scenarios added banking-get_transactions, which was appropriate for spending guidance.
    • +
  • Error handling was strong: The insufficient-funds scenario still called banking-transfer as requested, then used balances to explain the shortfall.

Alternatives

    -
  • concise: Lowest token footprint and cost on advisory tests; trade-off is reduced explanation depth. Suitable for cost-sensitive deployments focused on quick guidance.
    • -
  • friendly: Strong UX tone with moderate token cost; useful for consumer-facing experiences where encouragement matters.
    • -
  • detailed: Rich explanations with higher token cost; best for educational contexts. Not necessary for transactional banking flows.
    • +
  • banking-workflow: Best demonstration of coordinated tool use because it performed a transfer and then verified updated balances.
    • +
  • banking-with-skill: Best for domain-specific financial guidance; it loaded financial-advisor and produced an emergency-fund target table.
    • +
  • banking-gpt-5.6-sol: Passed the same account-advice task as banking-claude-opus-4.8 with fewer tokens in the observed run.
-

πŸ”§ MCP Tool Feedback

-

MCPServerStdio

-

Overall assessment: High discoverability and correct usage. Tools are clearly named and consistently invoked correctly across models and prompts.

+

πŸ€– Model Comparison

- - - - + + + + - - - - + + + + + + + + + + - - - - + + + + - - - - + + + + - - - - + + + + - - - - + + + + - - - - + + + +
ToolStatusCallsIssuesCapabilityclaude-opus-5claude-opus-4.8gpt-5.6-sol
get_balanceβœ…ManyWorking wellMCP balance toolsβœ… Reliableβœ… Reliableβœ… Reliable
Transfer workflowβœ… CorrectNot testedNot tested
get_all_balancesβœ…ManyWorking wellError recoveryβœ… CorrectNot testedNot tested
transferβœ…ManyWorking well, including error handlingFinancial advice groundingβœ… Reliableβœ… Reliableβœ… Reliable
get_transactionsβœ…FewWorking wellResponse styleDetailed and user-facingDetailed, includes follow-up offerConcise and action-oriented
deposit⚠️0Not covered by testsCost per testN/A$0.229461$0.142236
withdraw⚠️0Not covered by testsAvg turnsObserved passingObserved passingObserved passing
-

Note: deposit and withdraw were not called, but no tests asserted their usage. This is a coverage observation, not a failure.

-

πŸ“‹ MCP Prompt Template Feedback

-

default (clear)

+

claude-opus-5

+
+

Verdict: Strong default for tool-grounded banking workflows when pricing is available externally and detailed responses are acceptable.

+
+

Strengths

    -
  • Description clarity: Clear and explicit about mandatory tool usage.
    • -
  • Arguments: N/A (instructional prompt).
    • -
  • Issue: None observed.
    • -
  • Suggested change: None.
    • +
  • Correctly chains MCP calls for transfer verification.
    • +
  • Handles rejected operations without losing task context.
    • +
  • Produces polished, user-readable financial explanations.
-

πŸ€– Custom Agent Feedback

-

default (effective)

+

Weaknesses

    -
  • Description match: Matches expected behavior exactlyβ€”transactional, accurate, tool-driven.
    • -
  • Token count: 17,014
    • -
  • Behavioral impact: Strongly primes immediate tool invocation and avoids hallucination.
    • -
  • Gap analysis: None.
    • -
  • Suggested change: None.
    • +
  • Pricing was unavailable in this run, so deployment cost cannot be assessed from report data.
    • +
  • Some advice scenarios used substantially more tokens than the simpler direct lookup.
-

concise (effective)

+

claude-opus-4.8

+
+

Verdict: Reliable on the account-review task, but more expensive than gpt-5.6-sol for the same tested scenario.

+
+

Strengths

    -
  • Description match: Behavior is brief, direct, and action-oriented.
    • -
  • Token count: 2,982
    • -
  • Behavioral impact: Language like β€œBe brief and direct” suppresses verbosity while preserving correct tool calls.
    • -
  • Gap analysis: None.
    • -
  • Suggested change: None.
    • +
  • Correctly checked balances before giving advice.
    • +
  • Produced a concrete recommendation with a clear checking-buffer rationale.
-

detailed (effective)

+

Weaknesses

    -
  • Description match: Provides comprehensive explanations and context as instructed.
    • -
  • Token count: 4,284
    • -
  • Behavioral impact: Phrases like β€œexplain the reasoning” and β€œconsider multiple perspectives” increase response length without affecting correctness.
    • -
  • Gap analysis: None.
    • -
  • Suggested change: None.
    • +
  • Added a follow-up β€œWant me to make this transfer?” despite the user only asking for a recommendation.
-

friendly (effective)

+

gpt-5.6-sol

+
+

Verdict: Best observed choice for concise balance-grounded advice among models with available pricing data.

+
+

Strengths

    -
  • Description match: Encouraging tone with celebratory language.
    • -
  • Token count: 3,774
    • -
  • Behavioral impact: Positive framing increases verbosity slightly but does not interfere with tool usage.
    • -
  • Gap analysis: None.
    • -
  • Suggested change: None.
    • +
  • Used the correct balance tool.
    • +
  • Returned a compact recommendation without unnecessary follow-up orchestration.
    • +
  • Used fewer tokens than banking-claude-opus-4.8 on the same task.
-

πŸ“š Skill Feedback

-

financial-advisor (positive)

+

Weaknesses

    -
  • Usage rate: Referenced in all skill-augmented advisory responses.
    • -
  • Token cost: Included in advisory tests (up to ~3k tokens total).
    • -
  • Problem: Noneβ€”skill content directly shaped advice structure and priorities.
    • -
  • Suggested change: None. The β€œAlways call get_all_balances first” rule is especially effective.
    • +
  • Only one banking task was tested, so broader workflow and error-recovery behavior are not shown here.
+

πŸ”§ Tool Usage

+

Tool Proficiency Matrix

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ToolTotal CallsSuccessIssues
banking-get_all_balances8βœ… 8/8β€”
banking-get_balance1βœ… 1/1β€”
banking-transfer2βœ… 2/2One expected insufficient-funds rejection
banking-get_transactions3βœ… 3/3Returned empty history, handled correctly
skill1βœ… 1/1β€”
+

Tool Selection Issues

+

No incorrect tool selections were observed. The agent chose narrow lookup (get_balance) for a single-account question, broad lookup (get_all_balances) for advice and verification, and transfer when explicitly asked to move funds or test insufficient funds.

+

Efficiency Analysis

+ + + + + + + + + + + + + + + + + + + + + + + + + +
MetricValueAssessment
Avg tools per test1.7βœ… Efficient
Unnecessary tool calls0No clear wasted calls in the provided conversations
Failed tool calls0The insufficient-funds transfer was an expected domain rejection, not an agent/tool failure
+

πŸ“š Skill Impact

+ + + + + + + + + + + + + + + + + + + +
SkillTests WithTests WithoutDeltaToken Cost
financial-advisor1/1Not directly pairedNot measurable64,146 tokens in skill-backed run
+
+

Assessment: The skill improved specificity for emergency-fund guidance by injecting the 3–6 month principle and producing a useful target table. The provided data does not include a matched no-skill emergency-fund test, so impact cannot be quantified beyond the observed successful behavior.

+

πŸ’‘ Optimizations

@@ -1162,245 +1257,213 @@

πŸ’‘ Optimizations

- + - + - + - +
1Trim unused tool response fieldsCompact MCP response envelope recommended10–15% token reductionLower tool-response token usage on every MCP call
2Align advisory verbosity to prompt intentKeep model-comparison prompts action-only suggestion5–10% cost reductionFewer follow-up offers in recommendation-only tasks
-

1. Trim unused tool response fields (recommended)

+

1. Compact MCP response envelope (recommended)

    -
  • Current: Tool responses include both raw and formatted values plus explanatory messages.
    • -
  • Change: Remove fields not used in agent responses (e.g., duplicate message strings when balances are already restated).
    • -
  • Impact: 10–15% cost reduction from smaller tool payloads.
    • +
  • Current: Tool output appears both as structured JSON and again inside an escaped {"result":"..."} wrapper.
    • +
  • Change: Return one canonical compact JSON object, not a JSON string embedded inside another JSON object.
    • +
  • Impact: Reduces repeated response tokens on every banking tool call.
-

2. Align advisory verbosity to prompt intent (suggestion)

+

2. Keep model-comparison prompts action-only (suggestion)

    -
  • Current: friendly and detailed sometimes exceed practical length for banking dashboards.
    • -
  • Change: Add an optional instruction line: β€œEnd with a 3-bullet action summary.”
    • -
  • Impact: 5–10% cost reduction with improved scannability.
    • +
  • Current: banking-claude-opus-4.8 answered correctly but added β€œWant me to make this $500 transfer for you?”
    • +
  • Change: Add this exact sentence to the default system prompt: When the user asks for advice or a recommendation, do not offer to execute a transaction unless they explicitly ask you to perform one.
    • +
  • Impact: Shorter final answers and less risk of overstepping from advice into transaction initiation.

πŸ“¦ Tool Response Optimization

-

transfer (from MCPServerStdio)

+

banking MCP tools

    -
  • Current response size: ~80–100 tokens per call
    • -
  • Issues found: Redundant fields (amount_formatted + numeric amount; verbose message)
    • -
  • Suggested optimization: Keep numeric values and new balances only.
    • -
  • Estimated savings: ~30 tokens per call (~30–35% reduction)
    • +
  • Current response size: Duplicated across the visible tool payload and an escaped result string.
    • +
  • Issues found: Responses repeat the same data in two formats, increasing token use and making the transcript harder for the agent to parse.
    • +
  • Suggested optimization: Return only compact structured JSON.
    • +
  • Estimated savings: Meaningful per banking tool call because the escaped wrapper duplicates nearly the entire payload.

Example current vs optimized:

// Current
 {
-  "transaction_id": "TX0001",
-  "type": "transfer",
-  "from_account": "checking",
-  "to_account": "savings",
-  "amount": 100.0,
-  "amount_formatted": "$100.00",
-  "new_balance_from": 1400.0,
-  "new_balance_to": 3100.0,
-  "message": "Successfully transferred $100.00 from checking to savings."
-}
-
-// Optimized
+  "result": "{\"accounts\": {\"checking\": {\"balance\": 1500.0, \"formatted\": \"$1,500.00\"}, \"savings\": {\"balance\": 3000.0, \"formatted\": \"$3,000.00\"}}, \"total\": 4500.0, \"total_formatted\": \"$4,500.00\"}"
+}
+
+
// Optimized
 {
-  "transaction_id": "TX0001",
-  "from": "checking",
-  "to": "savings",
-  "amount": 100.0,
-  "new_from": 1400.0,
-  "new_to": 3100.0
-}
-

πŸ† Eval Leaderboard

EvalTestsPass RateTokensCostDuration
πŸ₯‡
default
12/12100%17,014$0.0236133.8s
πŸ₯ˆ
concise
2/2100%2,982$0.00477624.5s
πŸ₯‰
friendly
2/2100%3,774$0.00769033.1s
4
detailed
2/2100%4,284$0.00932634.0s
Compare:(Click to swap agents)

πŸ“‹ Test Results

18 / 18 tests
πŸ”—Core banking tests β€” parametrized across benchmark agents.(3 tests)
3/3
0/0
β–Ό
βœ…Check account balance.
Total 2,043 tokΒ·Total $0.002522Β·Ξ” +0%Β·Ξ” +0%Β·Ξ” +0%
default:βœ…19.4s
concise:β€”

πŸ† Eval Leaderboard

EvalTestsPass RateTokensCostDuration
πŸ₯‡
banking-basic
1/1100%41,396N/A9.5s
πŸ₯ˆ
banking-concise
1/1100%62,911N/A13.0s
πŸ₯‰
banking-detailed
1/1100%42,367N/A21.3s
4
banking-error-handling
1/1100%62,924N/A19.3s
5
banking-friendly
1/1100%42,300N/A13.8s
6
banking-with-skill
1/1100%64,146N/A20.1s
7
banking-workflow
1/1100%63,056N/A17.5s
8
banking-gpt-5.6-sol
1/1100%26,414$0.14229.9s
9
banking-claude-opus-4.8
1/1100%41,699$0.229510.5s
Compare:(Click to swap agents)

πŸ“‹ Test Results

9 / 9 tests
πŸ“‹Demonstrate direct MCP tool selection.(1 tests)
1/1
0/0
β–Ό
βœ…Retrieve one account balance.
Total 41,396 tokΒ·Total N/AΒ·Ξ” +0%Β·Ξ” +0%Β·Ξ” +0%
banking-basic:βœ…9.5s
banking-concise:β€”
βœ…Transfer funds between accounts.
Total 2,273 tokΒ·Total $0.003012Β·Ξ” +0%Β·Ξ” +0%Β·Ξ” +0%
default:βœ…14.5s
concise:β€”
πŸ“‹Demonstrate coordinated MCP tool calls.(1 tests)
0/0
0/0
β–Ό
βœ…Transfer money and verify the updated balances.
banking-basic:β€”
banking-concise:β€”
βœ…Handle insufficient funds gracefully.
Total 2,760 tokΒ·Total $0.003981Β·Ξ” +0%Β·Ξ” +0%Β·Ξ” +0%
default:βœ…34.6s
concise:β€”
πŸ“‹Compare the configured Copilot models on the same banking task.(1 tests)
0/0
0/0
β–Ό
βœ…Require every model to ground its advice in account data.
banking-basic:β€”
banking-concise:β€”
πŸ”—Multi-turn session: savings transfer workflow.(2 tests)
2/2
0/0
β–Ό
βœ…First turn: check account balances.
Total 2,159 tokΒ·Total $0.002707Β·Ξ” +0%Β·Ξ” +0%Β·Ξ” +0%
default:βœ…14.3s
concise:β€”
βœ…Second turn: transfer based on previous context.
Total 2,856 tokΒ·Total $0.003543Β·Ξ” +0%Β·Ξ” +0%Β·Ξ” +0%
default:βœ…15.3s
concise:β€”
πŸ“‹Compare how different prompt styles affect responses.(1 tests)
0/0
1/1
β–Ό
βœ…Compare advisory styles across prompts.
Total 2,982 tokΒ·Total $0.004776Β·Ξ” +0%Β·Ξ” +0%Β·Ξ” +0%
default:β€”
concise:βœ…24.5s
πŸ“‹Demonstrate domain guidance supplied by a skill.(1 tests)
0/0
0/0
β–Ό
βœ…Use the financial-advisor skill to ground savings guidance.
banking-basic:β€”
banking-concise:β€”
πŸ“‹Test how skills improve advice quality.(1 tests)
1/1
0/0
β–Ό
βœ…Eval with financial advisor skill gives better advice.
Total 4,923 tokΒ·Total $0.007823Β·Ξ” +0%Β·Ξ” +0%Β·Ξ” +0%
default:βœ…35.7s
concise:β€”
πŸ“‹Demonstrate recovery from a rejected banking operation.(1 tests)
0/0
0/0
β–Ό
βœ…Explain an insufficient-funds error after calling the transf…
banking-basic:β€”
banking-concise:β€”