Skip to content

Commit b6304db

Browse files
committed
test(s126): red first, an allow the owner issued is dropped and never consumed
Three failing tests, committed before any fix, by name: AC1: an allow clicked on the requested broadcast, with no wait for approval_wait, consumes the row and runs the call -> consumed_at is nil after status "allowed", decision "once" AC3/AC4: the chain reads ask then allow, both on the same scope, and neither is a timeout -> the chain carries ["ask"] and nothing else. The owner answered and the chain does not carry the answer. AC5: at the instant the requested broadcast is observable, the session can already accept a decision -> decided straight through the gate, no LiveView involved at all, and the grant is still not applied The difference from every approval test that passes is one line those tests have and a browser does not: collect(id, &match?({:state, :approval_wait}, &1)) The card is rendered from the gate's :requested broadcast, which is emitted from inside the tool task before the Session has returned from start_tools. So the suite has only ever exercised the ordering that works, and the synchronisation it relies on is not something any real caller can perform. AC5 is the one that rules out "the LiveView should wait": it decides through Permissions.decide_request/3 with no view in the loop and fails the same way. Note on reading the failing output: these runs also log a DBConnection ownership crash from the tool task. That is a consequence, not the defect. The assertions on status and decision pass before consumed_at fails, so the decision is recorded and never consumed; the ownership report is in-flight work colliding with sandbox teardown after the assertion has already failed. The existing approval_live_test.exs produces none of it and stays green. Signed-off-by: Ayla Croft <aylacroft@proton.me>
1 parent 033d056 commit b6304db

1 file changed

Lines changed: 136 additions & 0 deletions

File tree

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
# SPDX-FileCopyrightText: Sudo Apt Holdings LLC
2+
# SPDX-License-Identifier: Apache-2.0
3+
defmodule TrinityWeb.ApprovalResumeTest do
4+
@moduledoc """
5+
Slice 126: an allow the owner issued always consumes its approval and runs the call.
6+
7+
The invariant, set by the owner before work started: an allow either consumes the row, runs the
8+
tool and leaves the allow receipt on the same chain as the ask, or it fails closed with a receipt
9+
naming why it could not be consumed. `consumed_at: nil` after a granted allow is a defect and
10+
never a timeout.
11+
12+
Every approval test before this one calls `collect(id, &match?({:state, :approval_wait}, &1))`
13+
before clicking. That is a synchronisation the product does not have: the card is rendered from the
14+
gate's `:requested` broadcast, which is emitted from inside the tool task, before the Session has
15+
returned from `start_tools`. So the suite only ever exercised the ordering that works. These tests
16+
deliberately do not wait.
17+
"""
18+
use TrinityWeb.ConnCase, async: false
19+
@moduletag :capture_log
20+
21+
import Phoenix.LiveViewTest
22+
import Trinity.SessionCase, only: [script_deltas: 2, collect: 3]
23+
24+
alias Trinity.Factory
25+
alias Trinity.LLM.Providers.Fake
26+
alias Trinity.Permissions
27+
alias Trinity.Permissions.Approval
28+
alias Trinity.Receipts
29+
30+
setup do
31+
Fake.clear()
32+
on_exit(fn -> Trinity.SessionCase.stop_all_sessions() end)
33+
row = Factory.session!()
34+
:ok = Trinity.Sessions.subscribe(row.id)
35+
:ok = Permissions.subscribe(row.id)
36+
{:ok, id: row.id}
37+
end
38+
39+
@args %{"path" => "/home/me/notes/a.md", "text" => "hi"}
40+
41+
defp write_turn do
42+
Fake.scripts([
43+
[
44+
{:tool_call_start, "c1", "write_note"},
45+
{:tool_call_end, "c1", @args},
46+
{:usage, %{input_tokens: 1, output_tokens: 1}},
47+
{:done, :tool_calls}
48+
],
49+
script_deltas(2, "written ")
50+
])
51+
end
52+
53+
defp send_message(view, text),
54+
do: view |> form("#composer", %{"content" => text}) |> render_submit()
55+
56+
test "AC1: an allow clicked on the requested broadcast, with no wait for approval_wait, consumes the row and runs the call",
57+
%{conn: conn, id: id} do
58+
write_turn()
59+
{:ok, view, _} = live(conn, ~p"/s/#{id}")
60+
send_message(view, "write it")
61+
62+
# The only synchronisation a browser has: the card is on the page.
63+
assert_receive {:approval, :requested, %Approval{id: aid}}, 5_000
64+
assert has_element?(view, "#approval-#{aid}")
65+
66+
view |> element("#approval-#{aid} button", "Allow once") |> render_click()
67+
68+
_ = collect(id, &match?({:state, :idle}, &1), 15_000)
69+
70+
approval = Permissions.get_approval(aid)
71+
assert approval.status == "allowed"
72+
assert approval.decision == "once"
73+
74+
assert approval.consumed_at,
75+
"the owner allowed this call and the grant was never consumed. consumed_at is nil, " <>
76+
"which is the signature of the decision being dropped rather than applied: the click " <>
77+
"landed before the session had left tool_wait"
78+
79+
html = render(view)
80+
81+
assert html =~ "wrote 2 bytes",
82+
"the approved call did not run. Rendered instead: " <>
83+
(html |> String.replace(~r/<[^>]*>/, " ") |> String.replace(~r/\s+/, " "))
84+
85+
refute html =~ "timed out",
86+
"an allow the owner issued was reported as a timeout"
87+
end
88+
89+
test "AC3/AC4: the chain reads ask then allow, both on the same scope, and neither is a timeout",
90+
%{conn: conn, id: id} do
91+
write_turn()
92+
{:ok, view, _} = live(conn, ~p"/s/#{id}")
93+
send_message(view, "write it")
94+
95+
assert_receive {:approval, :requested, %Approval{id: aid}}, 5_000
96+
view |> element("#approval-#{aid} button", "Allow once") |> render_click()
97+
_ = collect(id, &match?({:state, :idle}, &1), 15_000)
98+
99+
rows = Receipts.list("session:" <> id)
100+
decisions = Enum.filter(rows, &(&1.kind == "decision"))
101+
102+
outcomes =
103+
for r <- decisions, {:ok, body} <- [JSON.decode(r.signed_payload)] do
104+
get_in(body, ["decision", "outcome"])
105+
end
106+
107+
assert "ask" in outcomes, "no ask receipt on the session chain: #{inspect(outcomes)}"
108+
109+
assert "allow" in outcomes,
110+
"the ask was receipted and the allow was not, on chain #{inspect(outcomes)}. The owner " <>
111+
"answered and the chain does not carry the answer"
112+
113+
assert Enum.find_index(outcomes, &(&1 == "ask")) <
114+
Enum.find_index(outcomes, &(&1 == "allow")),
115+
"the allow is not after the ask on the chain: #{inspect(outcomes)}"
116+
end
117+
118+
test "AC5: at the instant the requested broadcast is observable, the session can already accept a decision",
119+
%{conn: conn, id: id} do
120+
write_turn()
121+
{:ok, view, _} = live(conn, ~p"/s/#{id}")
122+
send_message(view, "write it")
123+
124+
assert_receive {:approval, :requested, %Approval{id: aid}}, 5_000
125+
126+
# Decide straight through the gate, with no LiveView in the way at all, at the first instant
127+
# the request is observable by anyone. Nothing in the product may require the caller to know
128+
# what state the Session is in.
129+
assert {:ok, _} = Permissions.decide_request(aid, :once, decided_by: "test")
130+
131+
_ = collect(id, &match?({:state, :idle}, &1), 15_000)
132+
133+
assert Permissions.get_approval(aid).consumed_at,
134+
"a decision made through the gate at the first observable instant was not applied"
135+
end
136+
end

0 commit comments

Comments
 (0)