diff --git a/specs/010-search-page-answers/contracts/answer_endpoint.md b/specs/010-search-page-answers/contracts/answer_endpoint.md index 34093ee..8735bc7 100644 --- a/specs/010-search-page-answers/contracts/answer_endpoint.md +++ b/specs/010-search-page-answers/contracts/answer_endpoint.md @@ -75,6 +75,48 @@ At most **12** citations are sent. Measured against the live endpoint this cap i binding on ordinary questions, so treat it as the most relevant few rather than the complete set. +### Some answers have no citations at all, by construction + +A `citation` carries `st_id`, a Reactome stable identifier, and only the Reactome +and disease-variant collections have them. The userguide collection indexes +documentation pages whose metadata is a page URL, so a userguide-routed question +produces **zero citations** -- measured 2026-09-18: "How do I use the pathway +browser?" returned 0 citations and 452 tokens. + +That is a good answer with no sources, not a failure. Two things follow for a +caller: + +- Render nothing rather than an empty heading. The website's panel keeps its + heading and list inside one conditional for this reason. +- It correlates with the **fastest** answers -- the userguide question is also + the 3.3s-to-first-token one -- so it is common rather than exotic, and it is + precisely what the four questions anyone would naturally test with fail to + produce. + +**Citations always precede prose**, and that is structural rather than +incidental: citations are emitted only on `on_retriever_end`, and token events +are gated on that same event, because retrieval completing is how the answer's +tokens are told apart from the query expander's. So sources are complete at the +moment the first token arrives, and a panel may render them before any prose. + +#### Open: citing userguide pages (not scheduled) + +Proposed by the website session 2026-09-18 and agreed in shape, not scheduled. +A documentation URL does not fit `{st_id, display_name}`, and **a fabricated +`R-` identifier is not an option** -- it would resolve to nothing, or worse to +the wrong entity. The shape that works is an optional sibling, exactly one of the +two present: + +``` +{"st_id": "R-HSA-8862803", "display_name": "..."} # unchanged +{"url": "/userguide/pathway-browser", "display_name": "..."} # new +``` + +A consumer that understands only `st_id` keeps working by skipping what it does +not recognise. Worth doing when this contract is next opened; an answer that is +honest about having no sources is much better than an invented one, so this is +not a blocker. + ### What `token` text contains **Markdown, never HTML.** Headings and lists appear; anchors do not. The answer diff --git a/specs/010-search-page-answers/tasks.md b/specs/010-search-page-answers/tasks.md index 835abd7..687c9a3 100644 --- a/specs/010-search-page-answers/tasks.md +++ b/specs/010-search-page-answers/tasks.md @@ -11,15 +11,15 @@ website repo entirely. Speed is Phase 5 and does not gate the handover. ## Phase 1: Setup -- [ ] T001 Create branch `010-search-page-answers`; confirm `.specify/feature.json` points at specs/010-search-page-answers -- [ ] T002 [P] Record today's baseline in specs/010-search-page-answers/quickstart.md: p50 15.2s, p90 22.4s over the 15 sweep questions, so Phase 5 has a before +- [x] T001 Create branch `010-search-page-answers`; confirm `.specify/feature.json` points at specs/010-search-page-answers +- [x] T002 [P] Record today's baseline in specs/010-search-page-answers/quickstart.md: p50 15.2s, p90 22.4s over the 15 sweep questions, so Phase 5 has a before ## Phase 2: Foundational (blocks the endpoint) -- [ ] T003 Pin the captcha middleware's current behaviour in tests/api/test_middleware_scope.py: every path under CHAINLIT_URI redirects to the captcha page except the existing allowlist -- [ ] T004 Give `AgentGraph` one home the FastAPI app can reach, in bin/chat-fastapi.py via lifespan, so the endpoint and Chainlit share one instance (SC-003) rather than paying 51.5s of startup twice -- [ ] T005 Add a streaming surface to src/agent/graph.py using the compiled graph's `astream_events`, yielding answer tokens and retrieved documents -- [ ] T006 Test that T005 yields more than one token for a real question — not that it returns 200. A surface that completes without streaming is the failure this must catch +- [x] T003 Pin the captcha middleware's current behaviour — done in tests/util/test_captcha_scope.py (6 tests) rather than the tests/api/test_middleware_scope.py named here, because the decision moved into src/util/captcha_scope.py where it can be tested without an app +- [x] T004 Give `AgentGraph` one home the FastAPI app can reach, in bin/chat-fastapi.py via lifespan, so the endpoint and Chainlit share one instance (SC-003) rather than paying 51.5s of startup twice +- [x] T005 Add a streaming surface to src/agent/graph.py using the compiled graph's `astream_events`, yielding answer tokens and retrieved documents +- [x] T006 Test that T005 yields more than one token for a real question — not that it returns 200. A surface that completes without streaming is the failure this must catch ## Phase 3: User Story 1 — a verified person sees an answer forming (P1) @@ -44,9 +44,10 @@ state. - [x] T016 [US2] Test that no model call happens for a refused request in tests/api/test_answer_endpoint.py, by asserting on a patched graph rather than on timing (SC-002) - [x] T017 [P] [US2] Rate limit per token as a backstop in src/util/rate_limit.py; the budget is the website's, enforced before the call reaches here (FR-008). 30 per 10 minutes, keyed on `sub`/`jti` when D1 provides one and a token hash until then (PR #237) - [x] T017a [US2] Stop paying for a discarded web search: the endpoint took `enable_postprocess` at its default, so every answer ran a Tavily search that `astream_answer` has no event to return (PR #237) -- [x] T021 [US2] Enforce `aud` on the caller token (asked for by the website, D1); the code refused every token carrying one, since PyJWT rejects `aud` when no audience is expected (PR #240) -- [x] T022 Rename human_token -> caller_token everywhere; D1 established the token asserts caller identity, not humanity (PR #240) -- [x] T023 Answer the website's cancellation question: a client hang-up raises CancelledError inside the answer generator and produces nothing further, so they need not cancel upstream (PR #240) +- [x] T025 [US2] Enforce `aud` on the caller token (asked for by the website, D1); the code refused every token carrying one, since PyJWT rejects `aud` when no audience is expected (PR #240) +- [x] T026 Rename human_token -> caller_token everywhere; D1 established the token asserts caller identity, not humanity (PR #240) +- [x] T027 Answer the website's cancellation question: a client hang-up raises CancelledError inside the answer generator and produces nothing further, so they need not cancel upstream (PR #240) +- [ ] T028 Decide whether userguide answers cite their pages. Needs an optional `url` sibling to `st_id` in the citation event -- agreed in shape with the website, not scheduled. A fabricated stable id is not an option - [ ] T020a Decide what to do about non-reproducible retrieval: three runs of one question shared only 4 of 19 citations (Jaccard 0.26) because query expansion is itself a model call. Affects what FR-007 can cache - [x] T018 [US2] Return `state: failed` with no partial answer on any internal error, so the page renders no panel (FR-006) - [x] T011a [US1] Strip inline HTML anchors from the token stream in src/util/anchor_strip.py; the contract promises prose without them and the chat prompt emits them, split across ~20 fragments (PR #236) @@ -58,14 +59,14 @@ state. - [x] T019 Measure first-token and completion separately across the tracked questions; publish the distribution, not one question. Measured 2026-09-18, two runs each: first token p50 9.6s / p90 12.2s (n=26), completion p50 10.4s / p90 18.1s (n=30). The earlier "36.1s to first token" came from one question and does not reproduce (PR #238) - [x] T019a Run preprocessing in two rounds instead of four sequential calls in src/agent/profiles/react_to_me.py; the base class already overlapped, and this override discarded it (PR #238) - [ ] T020 Reduce query expansion from 5 variants, measuring recall with bin/retrieval_baseline — its own call plus a 5x retrieval fan-out -- [ ] T020b Establish whether the four preprocessing calls must be sequential, and whether a search-page question needs all of them. They cost ~16s before retrieval starts and produce 36 tokens between them — the largest block in front of the first answer token -- [ ] T021 Land spec 009 collection routing and re-measure +- [ ] T020c Establish whether a search-page question needs all four preprocessing calls. The sequential half of this is answered and done (T019a): they run in two rounds and cost 2.6s at the median, not the ~16s recorded here, which never reproduced +- [x] T021 Land spec 009 collection routing and re-measure — landed and live (`state["active_sources"][0]` selects at retrieval time), and re-measured 2026-09-18: it is the main reason first-token fell from the ~36s once recorded to 9.6s - [ ] T022 Re-assess FR-005 against the result and say plainly whether 2s/10s is reachable ## Phase 6: Handover -- [ ] T023 Tell the website session the endpoint exists, with a curl that streams, and what it does not yet do -- [ ] T024 [P] Update specs/010-search-page-answers/spec.md status and record the measured first-token and completion times +- [x] T023 Tell the website session the endpoint exists, with a curl that streams, and what it does not yet do — done 2026-09-18, including the nginx user-agent 403 and the measured latency; they have built against it and a token they minted verified +- [x] T024 [P] Update specs/010-search-page-answers/spec.md status and record the measured first-token and completion times — first token p50 9.6s / p90 12.2s, completion p50 10.4s / p90 18.1s ## Dependencies