+
Recommended Configuration
+
banking-basic
+
Completes the direct MCP balance lookup with correct tool grounding and concise final output. Cost is unavailable because claude-opus-5 has no pricing data in this run.
100%Pass Rate
-
$0.023589Total Cost
-
17,014Tokens
+
N/ACost
+
41,396Tokens
@@ -1019,136 +1021,229 @@
Failures
-
4
-
Eval Configs
+
9
+
Agents
-
Comparative Analysis
+
Comparative Analysis
Why the winner wins
-- Coverage advantage: The winner (
default) passed 12 tests versus 2 tests each for concise, friendly, and detailed, demonstrating robustness across core banking, multiβturn sessions, advisory styles, and skill-augmented advice.
-- Tool discipline: Consistently follows βtool-firstβ instructions (balances before advice, transfers via tools, graceful handling of insufficient funds) with no deviations.
-- Session reliability: Correctly carries context across multi-turn workflows (balance β transfer) without redundant clarifications or missed tool calls.
-- Cost-effective at scale: While per-test cost is not directly comparable across configs due to different test counts, the winner maintains low total cost while handling the widest scenario set.
+- Correct minimal orchestration:
banking-basic used exactly the required banking-get_balance MCP tool and answered from the tool output without extra probing.
+- Clean grounding behavior: The assistant did not infer or fabricate the balance; it returned the formatted
$1,500.00 value from the tool response.
+- Lowest-complexity passing path: All configurations passed, but this one demonstrates the most direct tool-to-answer path for a single-account query.
Notable patterns
-- Concise prompts minimize tokens:
concise produces the lowest token usage per advisory response while still calling the necessary tools.
-- Detailed prompts expand reasoning, not tool usage:
detailed adds explanatory depth without increasing tool coverageβuseful for UX, not for operational accuracy.
-- Model parity on correctness: Both
gpt-5-mini and gpt-4.1 achieve perfect pass rates; differences are primarily in verbosity and cost, not correctness.
+- All banking scenarios passed: Every eval grounded its answer in MCP output before responding.
+- Advice tasks consistently used balance data: Both model-comparison evals called
banking-get_all_balances before recommending a savings action.
+- Transaction-history lookups were used only for broader advice: The concise, detailed, and friendly account-review scenarios added
banking-get_transactions, which was appropriate for spending guidance.
+- Error handling was strong: The insufficient-funds scenario still called
banking-transfer as requested, then used balances to explain the shortfall.
Alternatives
-- concise: Lowest token footprint and cost on advisory tests; trade-off is reduced explanation depth. Suitable for cost-sensitive deployments focused on quick guidance.
-- friendly: Strong UX tone with moderate token cost; useful for consumer-facing experiences where encouragement matters.
-- detailed: Rich explanations with higher token cost; best for educational contexts. Not necessary for transactional banking flows.
+- banking-workflow: Best demonstration of coordinated tool use because it performed a transfer and then verified updated balances.
+- banking-with-skill: Best for domain-specific financial guidance; it loaded
financial-advisor and produced an emergency-fund target table.
+- banking-gpt-5.6-sol: Passed the same account-advice task as
banking-claude-opus-4.8 with fewer tokens in the observed run.
-
π§ MCP Tool Feedback
-
MCPServerStdio
-
Overall assessment: High discoverability and correct usage. Tools are clearly named and consistently invoked correctly across models and prompts.
+
π€ Model Comparison
-| Tool |
-Status |
-Calls |
-Issues |
+Capability |
+claude-opus-5 |
+claude-opus-4.8 |
+gpt-5.6-sol |
-| get_balance |
-β
|
-Many |
-Working well |
+MCP balance tools |
+β
Reliable |
+β
Reliable |
+β
Reliable |
+
+
+| Transfer workflow |
+β
Correct |
+Not tested |
+Not tested |
-| get_all_balances |
-β
|
-Many |
-Working well |
+Error recovery |
+β
Correct |
+Not tested |
+Not tested |
-| transfer |
-β
|
-Many |
-Working well, including error handling |
+Financial advice grounding |
+β
Reliable |
+β
Reliable |
+β
Reliable |
-| get_transactions |
-β
|
-Few |
-Working well |
+Response style |
+Detailed and user-facing |
+Detailed, includes follow-up offer |
+Concise and action-oriented |
-| deposit |
-β οΈ |
-0 |
-Not covered by tests |
+Cost per test |
+N/A |
+$0.229461 |
+$0.142236 |
-| withdraw |
-β οΈ |
-0 |
-Not covered by tests |
+Avg turns |
+Observed passing |
+Observed passing |
+Observed passing |
-
Note: deposit and withdraw were not called, but no tests asserted their usage. This is a coverage observation, not a failure.
-
π MCP Prompt Template Feedback
-
default (clear)
+
claude-opus-5
+
+Verdict: Strong default for tool-grounded banking workflows when pricing is available externally and detailed responses are acceptable.
+
+
Strengths
-- Description clarity: Clear and explicit about mandatory tool usage.
-- Arguments: N/A (instructional prompt).
-- Issue: None observed.
-- Suggested change: None.
+- Correctly chains MCP calls for transfer verification.
+- Handles rejected operations without losing task context.
+- Produces polished, user-readable financial explanations.
-
π€ Custom Agent Feedback
-
default (effective)
+
Weaknesses
-- Description match: Matches expected behavior exactlyβtransactional, accurate, tool-driven.
-- Token count: 17,014
-- Behavioral impact: Strongly primes immediate tool invocation and avoids hallucination.
-- Gap analysis: None.
-- Suggested change: None.
+- Pricing was unavailable in this run, so deployment cost cannot be assessed from report data.
+- Some advice scenarios used substantially more tokens than the simpler direct lookup.
-
concise (effective)
+
claude-opus-4.8
+
+Verdict: Reliable on the account-review task, but more expensive than gpt-5.6-sol for the same tested scenario.
+
+
Strengths
-- Description match: Behavior is brief, direct, and action-oriented.
-- Token count: 2,982
-- Behavioral impact: Language like βBe brief and directβ suppresses verbosity while preserving correct tool calls.
-- Gap analysis: None.
-- Suggested change: None.
+- Correctly checked balances before giving advice.
+- Produced a concrete recommendation with a clear checking-buffer rationale.
-
detailed (effective)
+
Weaknesses
-- Description match: Provides comprehensive explanations and context as instructed.
-- Token count: 4,284
-- Behavioral impact: Phrases like βexplain the reasoningβ and βconsider multiple perspectivesβ increase response length without affecting correctness.
-- Gap analysis: None.
-- Suggested change: None.
+- Added a follow-up βWant me to make this transfer?β despite the user only asking for a recommendation.
-
friendly (effective)
+
gpt-5.6-sol
+
+Verdict: Best observed choice for concise balance-grounded advice among models with available pricing data.
+
+
Strengths
-- Description match: Encouraging tone with celebratory language.
-- Token count: 3,774
-- Behavioral impact: Positive framing increases verbosity slightly but does not interfere with tool usage.
-- Gap analysis: None.
-- Suggested change: None.
+- Used the correct balance tool.
+- Returned a compact recommendation without unnecessary follow-up orchestration.
+- Used fewer tokens than
banking-claude-opus-4.8 on the same task.
-
π Skill Feedback
-
financial-advisor (positive)
+
Weaknesses
-- Usage rate: Referenced in all skill-augmented advisory responses.
-- Token cost: Included in advisory tests (up to ~3k tokens total).
-- Problem: Noneβskill content directly shaped advice structure and priorities.
-- Suggested change: None. The βAlways call
get_all_balances firstβ rule is especially effective.
+- Only one banking task was tested, so broader workflow and error-recovery behavior are not shown here.
+
π§ Tool Usage
+
Tool Proficiency Matrix
+
+
+
+| Tool |
+Total Calls |
+Success |
+Issues |
+
+
+
+
+banking-get_all_balances |
+8 |
+β
8/8 |
+β |
+
+
+banking-get_balance |
+1 |
+β
1/1 |
+β |
+
+
+banking-transfer |
+2 |
+β
2/2 |
+One expected insufficient-funds rejection |
+
+
+banking-get_transactions |
+3 |
+β
3/3 |
+Returned empty history, handled correctly |
+
+
+skill |
+1 |
+β
1/1 |
+β |
+
+
+
+
Tool Selection Issues
+
No incorrect tool selections were observed. The agent chose narrow lookup (get_balance) for a single-account question, broad lookup (get_all_balances) for advice and verification, and transfer when explicitly asked to move funds or test insufficient funds.
+
Efficiency Analysis
+
+
+
+| Metric |
+Value |
+Assessment |
+
+
+
+
+| Avg tools per test |
+1.7 |
+β
Efficient |
+
+
+| Unnecessary tool calls |
+0 |
+No clear wasted calls in the provided conversations |
+
+
+| Failed tool calls |
+0 |
+The insufficient-funds transfer was an expected domain rejection, not an agent/tool failure |
+
+
+
+
π Skill Impact
+
+
+
+| Skill |
+Tests With |
+Tests Without |
+Delta |
+Token Cost |
+
+
+
+
+financial-advisor |
+1/1 |
+Not directly paired |
+Not measurable |
+64,146 tokens in skill-backed run |
+
+
+
+
+Assessment: The skill improved specificity for emergency-fund guidance by injecting the 3β6 month principle and producing a useful target table. The provided data does not include a matched no-skill emergency-fund test, so impact cannot be quantified beyond the observed successful behavior.
+
π‘ Optimizations
@@ -1162,245 +1257,213 @@ π‘ Optimizations
| 1 |
-Trim unused tool response fields |
+Compact MCP response envelope |
recommended |
-10β15% token reduction |
+Lower tool-response token usage on every MCP call |
| 2 |
-Align advisory verbosity to prompt intent |
+Keep model-comparison prompts action-only |
suggestion |
-5β10% cost reduction |
+Fewer follow-up offers in recommendation-only tasks |
-
1. Trim unused tool response fields (recommended)
+
1. Compact MCP response envelope (recommended)
-- Current: Tool responses include both raw and formatted values plus explanatory messages.
-- Change: Remove fields not used in agent responses (e.g., duplicate
message strings when balances are already restated).
-- Impact: 10β15% cost reduction from smaller tool payloads.
+- Current: Tool output appears both as structured JSON and again inside an escaped
{"result":"..."} wrapper.
+- Change: Return one canonical compact JSON object, not a JSON string embedded inside another JSON object.
+- Impact: Reduces repeated response tokens on every banking tool call.
-
2. Align advisory verbosity to prompt intent (suggestion)
+
2. Keep model-comparison prompts action-only (suggestion)
-- Current:
friendly and detailed sometimes exceed practical length for banking dashboards.
-- Change: Add an optional instruction line: βEnd with a 3-bullet action summary.β
-- Impact: 5β10% cost reduction with improved scannability.
+- Current:
banking-claude-opus-4.8 answered correctly but added βWant me to make this $500 transfer for you?β
+- Change: Add this exact sentence to the default system prompt:
When the user asks for advice or a recommendation, do not offer to execute a transaction unless they explicitly ask you to perform one.
+- Impact: Shorter final answers and less risk of overstepping from advice into transaction initiation.
π¦ Tool Response Optimization
-
transfer (from MCPServerStdio)
+
banking MCP tools
-- Current response size: ~80β100 tokens per call
-- Issues found: Redundant fields (
amount_formatted + numeric amount; verbose message)
-- Suggested optimization: Keep numeric values and new balances only.
-- Estimated savings: ~30 tokens per call (~30β35% reduction)
+- Current response size: Duplicated across the visible tool payload and an escaped
result string.
+- Issues found: Responses repeat the same data in two formats, increasing token use and making the transcript harder for the agent to parse.
+- Suggested optimization: Return only compact structured JSON.
+- Estimated savings: Meaningful per banking tool call because the escaped wrapper duplicates nearly the entire payload.
Example current vs optimized:
// Current
{
- "transaction_id": "TX0001",
- "type": "transfer",
- "from_account": "checking",
- "to_account": "savings",
- "amount": 100.0,
- "amount_formatted": "$100.00",
- "new_balance_from": 1400.0,
- "new_balance_to": 3100.0,
- "message": "Successfully transferred $100.00 from checking to savings."
-}
-
-// Optimized
+ "result": "{\"accounts\": {\"checking\": {\"balance\": 1500.0, \"formatted\": \"$1,500.00\"}, \"savings\": {\"balance\": 3000.0, \"formatted\": \"$3,000.00\"}}, \"total\": 4500.0, \"total_formatted\": \"$4,500.00\"}"
+}
+
+
// Optimized
{
- "transaction_id": "TX0001",
- "from": "checking",
- "to": "savings",
- "amount": 100.0,
- "new_from": 1400.0,
- "new_to": 3100.0
-}
-