Skip to content

Audit the answered state, and say what has not been tested #34

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Robbie1977 merged 2 commits into from
Aug 19, 2026
Merged

Audit the answered state, and say what has not been tested #34

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e644e6e
Audit the answered state, and say what has not been tested
Robbie1977 Aug 19, 2026
081dbb4
Add task battery results for e644e6e [skip ci]
github-actions[bot] Aug 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 56 additions & 5 deletions app/accessibility/page.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,15 +34,19 @@ export default function AccessibilityPage() {

<section style={{ marginTop: '28px' }}>
<h2 style={{ color: '#fff' }}>What We Do to Ensure Accessibility</h2>
<p style={{ color: '#b8b8b8', lineHeight: 1.6 }}>
The service is built to meet the following, and each is checked automatically on every
change &mdash; see &ldquo;How we tested this service&rdquo; below.
</p>
<ul style={{ lineHeight: 1.7 }}>
<li>Full keyboard navigation throughout the chat interface</li>
<li>Keyboard navigation throughout the chat interface</li>
<li>Skip-to-content link for keyboard and screen reader users</li>
<li>Proper ARIA landmarks and live regions for dynamic content</li>
<li>ARIA landmarks, and a live region for content that arrives as it is generated</li>
<li>Sufficient colour contrast ratios (minimum 4.5:1 for text)</li>
<li>Visible focus indicators for interactive elements</li>
<li>Semantic HTML structure with appropriate heading hierarchy</li>
<li>Alternative text for images</li>
<li>Accessible form inputs with associated labels</li>
<li>Alternative text on images</li>
<li>Form inputs with associated labels</li>
<li>No time-limited content</li>
<li>No flashing content</li>
</ul>
Expand All @@ -53,6 +57,31 @@ export default function AccessibilityPage() {
<ul style={{ lineHeight: 1.7 }}>
<li>Network graph visualisations (SVG) convey information visually that may not be fully available to screen reader users, though graph titles and labels are provided as text.</li>
<li>AI-generated content may occasionally produce complex formatting that is not optimally structured for assistive technology.</li>
<li>
Alternative text for images in an answer is derived from Virtual Fly Brain&rsquo;s own
data, so it names the structure shown rather than describing the image. Automated
checking confirms that alternative text is present; it cannot confirm that it is
useful, and that has not yet been assessed by a person.
</li>
<li>
Announcement of an answer as it is generated has not been verified with a screen
reader. The interface uses a live region for this, but whether the announcement is
timely and not repetitive is not something automated checking can establish.
</li>
</ul>
</section>

<section style={{ marginTop: '28px' }}>
<h2 style={{ color: '#fff' }}>What We Have Not Yet Tested</h2>
<p style={{ color: '#b8b8b8', lineHeight: 1.6 }}>
We would rather say what has not been checked than imply that it has. The following are
outstanding, and we expect to complete them before the service is publicly launched:
</p>
<ul style={{ lineHeight: 1.7 }}>
<li>Testing with screen readers (JAWS, NVDA and VoiceOver) by a person</li>
<li>Testing with voice recognition software</li>
<li>Reflow and readability at 400% zoom with a long answer containing tables</li>
<li>Whether generated link text and alternative text make sense out of context</li>
</ul>
</section>

Expand Down Expand Up @@ -87,10 +116,32 @@ export default function AccessibilityPage() {
</p>
</section>

<section style={{ marginTop: '28px' }}>
<h2 style={{ color: '#fff' }}>How We Tested This Service</h2>
<p style={{ color: '#b8b8b8', lineHeight: 1.6 }}>
Every page of this service &mdash; the chat interface, this statement, the privacy
notice and the terms of use &mdash; is tested automatically against WCAG 2.2 Level AA
using axe-core, driven through a real browser. The chat interface is tested twice: once
as it first loads, and once with an answer displayed, so that the answer text, result
tables, image gallery, citations, response identifier and feedback controls are covered
rather than only the empty page. The test runs on every proposed change to the service
and a change is not accepted while any violation is outstanding.
</p>
<p style={{ color: '#b8b8b8', lineHeight: 1.6 }}>
Automated testing of this kind detects somewhere between a third and a half of WCAG
issues. It is a floor rather than a certificate: judgements about focus order,
meaningful sequence, error identification and the usefulness of alternative text still
require a person, which is why the section above says what has not yet been tested.
Testing was carried out by the Virtual Fly Brain team at the University of Edinburgh.
</p>
</section>

<section style={{ marginTop: '28px' }}>
<h2 style={{ color: '#fff' }}>Preparation of This Statement</h2>
<p style={{ color: '#b8b8b8', lineHeight: 1.6 }}>
This statement was prepared on 26 March 2026. It was last reviewed on 26 March 2026.
This statement was first prepared on 26 March 2026 and was last reviewed on 19 August
2026. We review it at least every 12 months, and whenever the service changes in a way
that affects the statements made here.
</p>
</section>

Expand Down
10 changes: 8 additions & 2 deletions app/page.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -685,7 +685,9 @@ function ResponseIdentifier({ responseId }) {
.catch(() => setCopied('select and copy'))
}, [responseId])
return (
<div style={{ marginTop: '8px', fontSize: '0.68em', color: '#6f6f6f', display: 'flex', alignItems: 'baseline', gap: '6px', flexWrap: 'wrap' }}>
// #8a8a8a, not #6f6f6f: at 0.68em this is small text, so WCAG 2.2 AA wants
// 4.5:1 and the old value measured 3.94:1 against the answer background.
<div style={{ marginTop: '8px', fontSize: '0.68em', color: '#8a8a8a', display: 'flex', alignItems: 'baseline', gap: '6px', flexWrap: 'wrap' }}>
<span>Response ID:</span>
<code style={{ fontFamily: 'ui-monospace, SFMono-Regular, Menlo, monospace', color: '#9a9a9a', userSelect: 'all' }}>
{responseId}
Expand Down Expand Up @@ -774,7 +776,11 @@ const ChatMessage = memo(function ChatMessage({
<td style={{ padding: '4px 8px', verticalAlign: 'middle' }}>
<a href={r.reportUrl} target="_blank" rel="noopener noreferrer" title={`Open ${r.name} in VFB (new tab)`} style={{ color: '#9ecbff', textDecoration: 'none' }}>{r.name}</a>
{Array.isArray(r.tags) && r.tags.length > 0 && (
<div style={{ color: '#777', fontSize: '0.85em', marginTop: '2px' }}>{r.tags.join(' · ')}</div>
// #8a8a8a, not #777: these tags render at about 11px
// inside a table row, so they are small text and
// #777 measured 4.42:1 — just under the 4.5:1 that
// WCAG 2.2 AA requires.
<div style={{ color: '#8a8a8a', fontSize: '0.85em', marginTop: '2px' }}>{r.tags.join(' · ')}</div>
)}
</td>
</tr>
Expand Down
98 changes: 94 additions & 4 deletions scripts/a11y-audit.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,84 @@ const PAGES = ['/', '/privacy', '/accessibility', '/terms']
// requirement and must not be mixed into a compliance number.
const CONFORMANCE_TAGS = ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22aa']

// ── The answered state ───────────────────────────────────────────────────────
//
// Auditing `/` on load tests an empty shell: a heading, an input and a send
// button. Everything a reader actually spends time in — the answer bubble, the
// result table, the image gallery and its data-derived alt text, the inline
// citations, the response identifier and its copy button, the feedback controls
// — exists only after a question has been answered, and none of it was covered.
// That is also the surface most likely to be wrong, because it is assembled from
// model output rather than written by hand.
//
// The response is STUBBED rather than live, for two reasons. CI has no ELM
// credential on every run, so a real question would skip or fail there and the
// audit would quietly stop covering the answered state. And an accessibility
// check wants a fixed DOM: a live answer varies run to run, so a violation would
// appear and disappear without the interface having changed. The stub is served
// through the real SSE parser and the real renderers, so what axe sees is the
// production DOM — only the words are fixed.
//
// Keep this fixture representative. If a new element type starts appearing in
// answers, add it here or it goes untested.
const ANSWERED_FIXTURE = {
response: [
'The **medulla** (FBbt_00003748) is the second optic neuropil of the adult brain.',
'',
'It receives input from the lamina and projects to the lobula complex.',
'',
'See [Fischbach & Dittrich, 1989](https://doi.org/10.1007/BF00218858) for the classical description.'
].join('\n'),
tables: [{
title: 'Neurons with synaptic terminals in the medulla',
rows: [
{ name: 'Tm3 (FlyWire)', thumbnail: 'https://www.virtualflybrain.org/data/VFB/i/0000/0001/thumbnail.png', reportUrl: 'https://www.virtualflybrain.org/reports/VFB_00000001', tags: ['Tm3', 'FlyWire'] },
{ name: 'Mi1 (FlyWire)', thumbnail: 'https://www.virtualflybrain.org/data/VFB/i/0000/0002/thumbnail.png', reportUrl: 'https://www.virtualflybrain.org/reports/VFB_00000002', tags: ['Mi1', 'FlyWire'] }
],
queryUrl: 'https://www.virtualflybrain.org/reports/FBbt_00003748'
}],
images: [
{ id: 'VFB_00000001', label: 'Tm3 neuron aligned to JRC2018Unisex', thumbnail: 'https://www.virtualflybrain.org/data/VFB/i/0000/0001/thumbnail.png' }
],
followOns: [
{ kind: 'ask', text: 'Which neurons receive output from the medulla?' },
{ kind: 'vfb', text: 'Open medulla in VFB', url: 'https://www.virtualflybrain.org/reports/FBbt_00003748', title: 'Open in Virtual Fly Brain' }
],
sources: [{ title: 'Fischbach & Dittrich 1989', url: 'https://doi.org/10.1007/BF00218858' }],
terms: [{ id: 'FBbt_00003748', label: 'medulla' }],
requestId: 'a11y-audit-request',
responseId: 'a11y-audit-response'
}

const SSE_FIXTURE =
`event: status\ndata: ${JSON.stringify({ message: 'Resolving terms' })}\n\n` +
`event: delta\ndata: ${JSON.stringify({ text: 'The medulla is the second optic neuropil.' })}\n\n` +
`event: result\ndata: ${JSON.stringify(ANSWERED_FIXTURE)}\n\n`

// Drive the real UI to the answered state: intercept the chat call, type a
// question, send it, and wait for the response identifier — which is rendered
// only once a `result` event has been applied, so it is the honest signal that
// the finished DOM is present.
async function renderAnsweredState(page) {
await page.route('**/api/chat', route => route.fulfill({
status: 200,
headers: { 'content-type': 'text/event-stream', 'cache-control': 'no-cache' },
body: SSE_FIXTURE
}))
// The rate-limit poll fires after a result and is irrelevant here; stub it so
// a missing backend cannot leave the page in an error state mid-audit.
await page.route('**/api/rate-info', route => route.fulfill({
status: 200, headers: { 'content-type': 'application/json' },
body: JSON.stringify({ used: 1, limit: 50, remaining: 49 })
}))

await page.goto(`${BASE}/`, { waitUntil: 'networkidle', timeout: 45000 })
await page.getByPlaceholder('Ask about Drosophila neuroanatomy...').fill('What is the medulla?')
await page.getByLabel('Send message').click()
await page.getByText('Response ID:').waitFor({ timeout: 20000 })
await page.waitForTimeout(400)
}

const results = []
// The sandbox ships Chromium at a fixed path and blocks the download step, so
// point at it rather than letting Playwright fetch its own.
Expand All @@ -38,13 +116,25 @@ const browser = await chromium.launch(executablePath ? { executablePath } : {})

const context = await browser.newContext({ viewport: { width: 1280, height: 900 } })

for (const path of PAGES) {
// Each target is a label plus how to get the page into the state being audited.
// The four plain loads are the shell states; the last is the answered state.
const TARGETS = [
...PAGES.map(path => ({
path,
prepare: async page => {
await page.goto(`${BASE}${path}`, { waitUntil: 'networkidle', timeout: 45000 })
// The chat page renders its shell immediately but settles a moment later.
await page.waitForTimeout(1200)
}
})),
{ path: '/ (answered)', prepare: renderAnsweredState }
]

for (const { path, prepare } of TARGETS) {
const page = await context.newPage()
let error = null
try {
await page.goto(`${BASE}${path}`, { waitUntil: 'networkidle', timeout: 45000 })
// The chat page renders its shell immediately but settles a moment later.
await page.waitForTimeout(1200)
await prepare(page)
} catch (e) {
error = e.message
}
Expand Down
Loading