Skip to content

Session 15: budget aware routing for irregular travel operations - #14

Open
kumariAstha wants to merge 1 commit into
theschoolofai:mainfrom
kumariAstha:kumariastha/s15-travel-ops
Open

kumariAstha wants to merge 1 commit into
theschoolofai:mainfrom
kumariAstha:kumariastha/s15-travel-ops

Conversation

@kumariAstha

Copy link
Copy Markdown

Session 15: Budget-Aware Routing for Irregular Travel Operations

What this PR adds

  • Workload: 15 airline disruption recovery tasks (5 trivial / 5 moderate / 5 hard) in proofs/tasks/travel_ops.jsonl
  • Ladder: cerebras/zai-glm-4.7 (economy) → gemini/gemini-3.1-flash-lite (standard) → openrouter/openai/gpt-4.1 (frontier)
  • Budget policy: $0.03 ceiling, downgrade at 50% pressure, refuse at 90%, max 40 calls/run, max 5 calls/node
  • Adversarial proof: proofs/p8_adversarial_budget.py — ladder climb, budget exhaustion, per-node ceiling, provider failure gap

Key findings

Strategy Resolved Cost/Resolved Task
A: Always frontier 12/15 (80%) $0.000653
B: Always cheapest 2/15 (13%) $0.000172
C: Budget-aware cascade 13/15 (87%) $0.000267

The cascade resolves the most tasks at 59% lower cost per resolved task than always-frontier. It also resolves one task (t15) that frontier fails — the standard tier produces more focused output that the judge scores higher.

Break-even resolution rate: 26.2%. Signature failure mode not observed — economy model was too weak (13% resolution) for the cheap-model trap to appear.

Where the policy fails

  • t02, t09: failed across all strategies. Rate-limiting + empty answer extraction on bare calls. Same tasks succeed through the full agent runtime.
  • t15: frontier fails, cascade succeeds — verbose frontier output scored lower than concise standard output.
  • Known gap: provider-side token consumption on failed calls is not metered.

Evidence

Full analysis in the README evidence section. Proof outputs in proofs/out/ and evidence/.

Reproduce

All commands in the README. Requires API keys for Cerebras, Gemini, OpenRouter, Groq, Nvidia (all free tier).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant