S15 Parts 2 & 3: a ticket-triage policy, measured against always-frontier and attacked - Dheeraj Hegde

[Dheeraj-Hegde]

Builds a new task class, its ladder and its budget policy on top of Part 1's
runtime, measures the policy against an always-frontier baseline, and then
attacks it. Nothing in s15code/ is edited — the whole submission is one new
task file, one new config directory, two proof scripts and evidence under
proofs/out/. The main README.md has the full sections; two
standalone review docs are added for convenience:

• README_PART2.md — build a policy and measure it
• README_PART3.md — attack your own budget

Headline numbers

Part 2 — 18 IT-ticket-triage tasks
(proofs/tasks/tickets.jsonl) through the
Part 2 ladder and policy (config/part2/):

strategy	resolved	cost / call	cost / resolved task	total spend
A always-frontier	18 / 18 (100 %)	$0.014731	$0.014731	$0.26516
B always-cheapest, ≤ 2 tries	6 / 18 (33 %)	$0.001005	$0.005026	$0.03015
C budget-aware cascade (this policy)	18 / 18 (100 %)	$0.005501	$0.011003	$0.19805

• C beats A by 25.3 % on cost per resolved task at identical 100 %
  resolution.
• Ladder projected spread: 28.3×. Break-even resolution rate B vs A with
  2 attempts: 12.8 %. B sits at 33 % — above break-even, so B really is
  cheaper per resolved on the tasks it does resolve, it just leaves 12/18
  tickets unaddressed.
• Wrong case, quantified: on tk13_share_drive (hard), C spent
  +22.8 % vs A ($0.02726 vs $0.02219) for the identical outcome, because
  it burned two low-tier calls before landing on frontier. Full analysis in
  README_PART2 §6.
• One Jaeger trace with per-span cost: trace id
  96fac5e7b2e0e3dd912292062959c625, 10 spans, s15.cost across every
  provider-call span sums to $0.03257800 — exactly the ledger spent.
  proofs/out/part2_trace_with_costs.json

Part 3 — same policy under an unbounded planner ($0.002 ceiling):

	before control (extrapolated)	after control (measured)
Spend	~$11.33 over 10 000 rounds	$0.001133, 0.01 % of that
Admitted calls	unbounded	1
Refusals	0	199 — each a visible BudgetRefused node failure with a refusal_log entry naming which projection failed which threshold

All six p3 invariants pass; every refusal is attributed to principal
you/part2-attacker, reasoned, and countable in both the ledger and the
OTel span export.

What is in this PR

category	files
Task class	proofs/tasks/tickets.jsonl
Policy config	config/part2/tiers.yaml · budgets.yaml · pricing.yaml · evals.yaml
Helper scripts	proofs/_pick_tickets.py · proofs/capture_part2_trace.py
Evidence (Part 2)	p1_cost_per_task_part2.json · part2_trace_with_costs.json · _p1_part2.log · _part2_trace.log
Evidence (Part 3)	p3_denial_of_wallet_part2.json · _p3_part2.log
Docs	README.md (Part 2 & Part 3 sections added) · README_PART2.md · README_PART3.md

Nothing under s15code/ is touched. The whole delta is data + config + docs

• proof outputs.

How to reproduce from a fresh checkout

git clone <this repo> ; cd S15Code ; uv sync
# optional local Jaeger for the trace round-trip
docker run --rm -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one:latest

$env:S15_OTEL_EXPORTER_ENDPOINT="http://127.0.0.1:4318/v1/traces"
$env:S15_OTEL_SERVICE_NAME="s15code-part2"

# Part 2 — measurement + one Jaeger trace with per-span costs
uv run python proofs/p1_cost_per_task.py `
    --tasks proofs/tasks/tickets.jsonl --config-dir config/part2 `
    --principal "you/part2" --offline --label part2
uv run python proofs/capture_part2_trace.py

# Part 3 — adversarial
$task = "Adversarial: an unbounded planner will earn one more node from every outcome, forever, even after refusals. Return JSON {category,severity,first_step}."
uv run python proofs/p3_denial_of_wallet.py `
    --task $task --budget 0.002 --config-dir config/part2 `
    --principal "you/part2-attacker" --offline --label part2