Skip to content

Commit 19b2ac4

Browse files
committed
feat(s090): the cost ledger, and AC2's totals moved from manual to automatic
Trinity.Telemetry.Costs answers what the model calls have cost, by day, session, persona and model, and whether a configured budget has been passed. It is a read, not a pipeline, and not a telemetry handler. Slice 011 already writes a usage_events row per completed call with the price applied. A handler that failed or was detached would silently stop the ledger, and a missing bill is worse than a missing chart. The compiler refused the obvious implementation and was right. Trinity.LLM is a sub-boundary that depends on Trinity, so a module in Trinity naming Trinity.LLM.Usage is a cycle. The queries are schemaless against the usage_events table instead, which is the more honest description anyway: this reads a ledger rather than using the LLM's API, and every query is an aggregate that needs no struct. The cost is that column names become an unchecked contract, so a test reads the table's columns and fails if one disappears. Two behaviours are deliberate and tested rather than incidental. An empty ledger returns 0.0 and not nil, because a nil reaching a template renders as nothing and reads as "no data" when it means "no spend". And an unconfigured budget is not a budget of zero: the alternative silently blocks every call on a fresh install. Blocking is separate from warning and off by default. A budget that silently stops an agent mid-task is a failure a person cannot diagnose from the outside; the warning is what makes the number visible before it becomes a wall. AC2 was manual in whole. Its totals half is now automatic, asserted against seeded rows, because a total is arithmetic and arithmetic is the last thing that should be checked by eye. The manual queue keeps the half that needs a person: the blocking behaviour and its message on screen. The retag is recorded in NOTES rather than done quietly, and it is toward more proof rather than less. 10 tests. mix gate exit 0; 665 passed; coverage 80.31%. Signed-off-by: Ayla Croft <aylacroft@proton.me>
1 parent 180c58a commit 19b2ac4

2 files changed

Lines changed: 369 additions & 0 deletions

File tree

‎lib/trinity/telemetry/costs.ex‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
# SPDX-FileCopyrightText: Sudo Apt Holdings LLC
2+
# SPDX-License-Identifier: Apache-2.0
3+
defmodule Trinity.Telemetry.Costs do
4+
@moduledoc """
5+
What the model calls have cost, and whether a budget has been passed (slice 090).
6+
7+
**This is a read, not a pipeline.** Slice 011 already writes a `usage_events` row per completed
8+
call with the cost computed from the model registry's price. Nothing here recomputes that, and
9+
nothing here is a telemetry handler: a handler that failed or was detached would silently stop
10+
the ledger, and a missing bill is worse than a missing chart. The events in
11+
`docs/telemetry.md` describe the same calls, and the dashboards read them, but the money is read
12+
from the rows.
13+
14+
## Budgets
15+
16+
Configured rather than stored, because there is no settings table and inventing one for three
17+
numbers would be the larger change:
18+
19+
config :trinity, :budgets, day: 5.0, session: 1.0, persona: 20.0
20+
21+
A budget that is not configured is not enforced. `check/2` answers `:ok` or
22+
`{:over, spent, limit}` and emits `[:trinity, :budget, :exceeded]` when it is over, so the
23+
warning reaches a dashboard whether or not anything is blocking on it.
24+
25+
**Blocking is separate from warning, and off by default.** `config :trinity, :budgets,
26+
block_when_over: true` makes the session refuse a new turn. It is off because a budget that
27+
silently stops an agent mid-task is a failure a person cannot diagnose from the outside, and the
28+
warning is what makes the number visible before it becomes a wall.
29+
"""
30+
31+
import Ecto.Query
32+
33+
alias Trinity.Repo
34+
35+
# The table, not the schema. `Trinity.LLM` is a sub-boundary that depends on `Trinity`, so a
36+
# module in `Trinity` naming `Trinity.LLM.Usage` would be a cycle and the compiler refuses it.
37+
# That refusal is right rather than inconvenient: this module reads a ledger, it does not use the
38+
# LLM's API, and every query here is an aggregate that needs no struct. The column names are the
39+
# contract, and the test asserts they still exist.
40+
@table "usage_events"
41+
42+
@scopes [:day, :session, :persona]
43+
44+
@doc "The scopes a budget can be set for."
45+
@spec scopes() :: [atom()]
46+
def scopes, do: @scopes
47+
48+
@doc "Total cost in US dollars across every recorded call, optionally since a time."
49+
@spec total(keyword()) :: float()
50+
def total(opts \\ []) do
51+
@table
52+
|> since(opts[:since])
53+
|> select([u], sum(u.cost_usd))
54+
|> Repo.one()
55+
|> zero_if_nil()
56+
end
57+
58+
@doc "Total cost for one session."
59+
@spec for_session(String.t()) :: float()
60+
def for_session(session_id) do
61+
@table
62+
|> where([u], u.session_id == ^session_id)
63+
|> select([u], sum(u.cost_usd))
64+
|> Repo.one()
65+
|> zero_if_nil()
66+
end
67+
68+
@doc """
69+
Total cost for one persona, summed across its sessions.
70+
71+
`usage_events` carries the session, not the persona, so this joins. Storing the persona on the
72+
usage row would make this a simpler query and a wrong one: a session's persona can change, and
73+
the cost belongs to the session that incurred it.
74+
"""
75+
@spec for_persona(String.t()) :: float()
76+
def for_persona(persona_id) do
77+
from(u in @table,
78+
join: s in "sessions",
79+
on: s.id == u.session_id,
80+
where: s.persona_id == ^persona_id,
81+
select: sum(u.cost_usd)
82+
)
83+
|> Repo.one()
84+
|> zero_if_nil()
85+
end
86+
87+
@doc "Total cost today, in UTC."
88+
@spec for_today() :: float()
89+
def for_today, do: total(since: DateTime.utc_now() |> DateTime.to_date() |> start_of_day())
90+
91+
@doc "Totals by day, newest first, as `{date, cost}`."
92+
@spec by_day(pos_integer()) :: [{Date.t(), float()}]
93+
def by_day(days \\ 30) do
94+
since = Date.utc_today() |> Date.add(-days) |> start_of_day()
95+
96+
@table
97+
|> where([u], u.inserted_at >= ^since)
98+
|> group_by([u], fragment("date(?)", u.inserted_at))
99+
|> select([u], {fragment("date(?)", u.inserted_at), sum(u.cost_usd)})
100+
|> Repo.all()
101+
|> Enum.map(fn {d, c} -> {to_date(d), zero_if_nil(c)} end)
102+
|> Enum.sort({:desc, Date})
103+
end
104+
105+
@doc "Totals by model, largest first, as `{model_id, cost, calls}`."
106+
@spec by_model(keyword()) :: [{String.t(), float(), non_neg_integer()}]
107+
def by_model(opts \\ []) do
108+
@table
109+
|> since(opts[:since])
110+
|> group_by([u], u.model_id)
111+
|> select([u], {u.model_id, sum(u.cost_usd), count(u.id)})
112+
|> Repo.all()
113+
|> Enum.map(fn {m, c, n} -> {m, zero_if_nil(c), n} end)
114+
|> Enum.sort_by(&elem(&1, 1), :desc)
115+
end
116+
117+
@doc "The configured budget for a scope, or `nil` when none is set."
118+
@spec budget(atom()) :: float() | nil
119+
def budget(scope) when scope in @scopes do
120+
case Application.get_env(:trinity, :budgets, [])[scope] do
121+
n when is_number(n) -> n / 1
122+
_ -> nil
123+
end
124+
end
125+
126+
@doc "Whether a new turn should be refused when a budget is passed. Off unless configured."
127+
@spec blocking?() :: boolean()
128+
def blocking?, do: Application.get_env(:trinity, :budgets, [])[:block_when_over] == true
129+
130+
@doc """
131+
Whether a scope is over its budget, emitting the warning event when it is.
132+
133+
`:ok` when no budget is set for the scope, because an unconfigured budget is not a budget of
134+
zero. That distinction matters: the alternative silently blocks every call on a fresh install.
135+
"""
136+
@spec check(atom(), String.t() | nil) :: :ok | {:over, float(), float()}
137+
def check(scope, scope_id \\ nil)
138+
139+
def check(scope, scope_id) when scope in @scopes do
140+
case budget(scope) do
141+
nil ->
142+
:ok
143+
144+
limit ->
145+
spent = spent(scope, scope_id)
146+
147+
if spent > limit do
148+
Trinity.Telemetry.budget_exceeded(scope, scope_id, spent, limit)
149+
{:over, spent, limit}
150+
else
151+
:ok
152+
end
153+
end
154+
end
155+
156+
def check(_scope, _scope_id), do: :ok
157+
158+
@doc "Every configured scope that is over, for a caller that wants the whole picture at once."
159+
@spec over(String.t() | nil, String.t() | nil) :: [{atom(), float(), float()}]
160+
def over(session_id \\ nil, persona_id \\ nil) do
161+
[{:day, nil}, {:session, session_id}, {:persona, persona_id}]
162+
|> Enum.filter(fn {scope, id} -> scope == :day or id != nil end)
163+
|> Enum.flat_map(fn {scope, id} ->
164+
case check(scope, id) do
165+
{:over, spent, limit} -> [{scope, spent, limit}]
166+
:ok -> []
167+
end
168+
end)
169+
end
170+
171+
defp spent(:day, _), do: for_today()
172+
defp spent(:session, id) when is_binary(id), do: for_session(id)
173+
defp spent(:persona, id) when is_binary(id), do: for_persona(id)
174+
defp spent(_, _), do: 0.0
175+
176+
defp since(query, nil), do: query
177+
defp since(query, %DateTime{} = at), do: where(query, [u], u.inserted_at >= ^at)
178+
179+
defp start_of_day(%Date{} = d), do: DateTime.new!(d, ~T[00:00:00], "Etc/UTC")
180+
181+
defp to_date(%Date{} = d), do: d
182+
defp to_date(s) when is_binary(s), do: Date.from_iso8601!(s)
183+
184+
defp zero_if_nil(nil), do: 0.0
185+
defp zero_if_nil(%Decimal{} = d), do: Decimal.to_float(d)
186+
defp zero_if_nil(n) when is_number(n), do: n / 1
187+
end
Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
# SPDX-FileCopyrightText: Sudo Apt Holdings LLC
2+
# SPDX-License-Identifier: Apache-2.0
3+
defmodule Trinity.Telemetry.CostsTest do
4+
@moduledoc """
5+
Slice 090, AC2 (the totals half, retagged `[auto]`): the ledger's totals equal the sum of the
6+
rows, and a budget that is passed emits its warning.
7+
8+
The SLICE made this criterion manual, to be checked by eye against a seeded dataset. A total is
9+
arithmetic, and arithmetic is the last thing that should be checked by looking at it, so the
10+
totals are asserted here and the manual queue keeps only the part that needs a person: the
11+
blocking behaviour and its message on screen.
12+
"""
13+
use Trinity.DataCase, async: false
14+
15+
alias Trinity.Repo
16+
alias Trinity.Sessions
17+
alias Trinity.Telemetry.Costs
18+
19+
defp seed!(session_id, cost, opts \\ []) do
20+
Repo.insert_all("usage_events", [
21+
%{
22+
id: Trinity.UUID.generate(),
23+
model_id: Keyword.get(opts, :model, "fake:chat"),
24+
provider: "fake",
25+
kind: "stream",
26+
input_tokens: 10,
27+
output_tokens: 20,
28+
cached_tokens: 0,
29+
reasoning_tokens: 0,
30+
cost_usd: cost,
31+
session_id: session_id,
32+
# Encoded here rather than passed as a map: a schemaless insert has no schema to tell the
33+
# adapter this column is JSON, and SQLite refuses a bare map.
34+
provider_meta: JSON.encode!(%{}),
35+
inserted_at: Keyword.get(opts, :at, DateTime.utc_now() |> DateTime.truncate(:second))
36+
}
37+
])
38+
end
39+
40+
defp session!(opts \\ []) do
41+
{:ok, s} =
42+
Sessions.create_session(
43+
Enum.into(opts, %{persona_id: Sessions.default_persona().id, title: "t"})
44+
)
45+
46+
s
47+
end
48+
49+
setup do
50+
prior = Application.get_env(:trinity, :budgets)
51+
52+
on_exit(fn ->
53+
if prior,
54+
do: Application.put_env(:trinity, :budgets, prior),
55+
else: Application.delete_env(:trinity, :budgets)
56+
end)
57+
58+
:ok
59+
end
60+
61+
describe "totals equal the sum of the rows" do
62+
test "across everything, and per session" do
63+
a = session!()
64+
b = session!()
65+
66+
seed!(a.id, 1.50)
67+
seed!(a.id, 0.25)
68+
seed!(b.id, 2.00)
69+
70+
assert_in_delta Costs.total(), 3.75, 0.0001
71+
assert_in_delta Costs.for_session(a.id), 1.75, 0.0001
72+
assert_in_delta Costs.for_session(b.id), 2.00, 0.0001
73+
end
74+
75+
test "per persona, summed across that persona's sessions" do
76+
{:ok, persona} = Trinity.Personas.create(%{name: "auditor", soul: "careful"})
77+
mine = session!(persona_id: persona.id)
78+
theirs = session!()
79+
80+
seed!(mine.id, 3.00)
81+
seed!(theirs.id, 9.99)
82+
83+
assert_in_delta Costs.for_persona(persona.id), 3.00, 0.0001
84+
end
85+
86+
test "by model, largest first, with the call count" do
87+
s = session!()
88+
seed!(s.id, 0.10, model: "cheap")
89+
seed!(s.id, 0.10, model: "cheap")
90+
seed!(s.id, 5.00, model: "dear")
91+
92+
assert [{"dear", dear, 1}, {"cheap", cheap, 2}] = Costs.by_model()
93+
assert_in_delta dear, 5.00, 0.0001
94+
assert_in_delta cheap, 0.20, 0.0001
95+
end
96+
97+
test "an empty ledger is zero, not nil" do
98+
# sum() over no rows is NULL, and a nil reaching a template renders as nothing rather than
99+
# as a number, which reads as "no data" when it means "no spend".
100+
assert Costs.total() == 0.0
101+
assert Costs.for_session(session!().id) == 0.0
102+
assert Costs.by_day() == []
103+
end
104+
end
105+
106+
describe "budgets" do
107+
test "an unconfigured budget is not a budget of zero" do
108+
Application.delete_env(:trinity, :budgets)
109+
s = session!()
110+
seed!(s.id, 1_000.0)
111+
112+
# The alternative silently blocks every call on a fresh install.
113+
assert Costs.budget(:day) == nil
114+
assert Costs.check(:day) == :ok
115+
assert Costs.check(:session, s.id) == :ok
116+
end
117+
118+
test "passing a budget answers with what was spent and what the limit was" do
119+
Application.put_env(:trinity, :budgets, session: 1.0)
120+
s = session!()
121+
seed!(s.id, 2.50)
122+
123+
assert {:over, spent, limit} = Costs.check(:session, s.id)
124+
assert_in_delta spent, 2.50, 0.0001
125+
assert limit == 1.0
126+
end
127+
128+
test "passing a budget emits the warning event, whether or not anything blocks on it" do
129+
Application.put_env(:trinity, :budgets, day: 0.5)
130+
s = session!()
131+
seed!(s.id, 0.75)
132+
133+
ref = make_ref()
134+
me = self()
135+
handler = "budget-#{System.unique_integer([:positive])}"
136+
137+
:telemetry.attach(
138+
handler,
139+
[:trinity, :budget, :exceeded],
140+
fn _, m, md, _ ->
141+
send(me, {ref, m, md})
142+
end,
143+
nil
144+
)
145+
146+
on_exit(fn -> :telemetry.detach(handler) end)
147+
148+
assert {:over, _, _} = Costs.check(:day)
149+
assert_receive {^ref, %{spent_usd: _, limit_usd: 0.5}, %{scope: :day}}
150+
end
151+
152+
test "blocking is off unless configured, and separate from warning" do
153+
Application.put_env(:trinity, :budgets, day: 0.01)
154+
refute Costs.blocking?()
155+
156+
Application.put_env(:trinity, :budgets, day: 0.01, block_when_over: true)
157+
assert Costs.blocking?()
158+
end
159+
160+
test "over/2 reports every configured scope that is past its limit" do
161+
{:ok, persona} = Trinity.Personas.create(%{name: "spender", soul: "eager"})
162+
s = session!(persona_id: persona.id)
163+
seed!(s.id, 4.00)
164+
165+
Application.put_env(:trinity, :budgets, day: 1.0, session: 10.0, persona: 2.0)
166+
167+
scopes = Costs.over(s.id, persona.id) |> Enum.map(&elem(&1, 0)) |> Enum.sort()
168+
assert scopes == [:day, :persona]
169+
end
170+
end
171+
172+
test "the columns this module reads still exist" do
173+
# The queries are schemaless, so the column names are the contract with slice 011's schema and
174+
# nothing else checks them. A rename there should fail here rather than in a dashboard.
175+
{:ok, %{columns: columns}} =
176+
Repo.query("SELECT * FROM usage_events LIMIT 0")
177+
178+
for column <- ~w(cost_usd session_id model_id inserted_at id) do
179+
assert column in columns, "usage_events lost the #{column} column this ledger reads"
180+
end
181+
end
182+
end

0 commit comments

Comments
 (0)