[Feature Request] Would an optional nightly check for saved turns before and after Retry be useful? #6941

🥰 Feature Description

Would you be open to an optional nightly browser test that checks what NextChat actually saves when someone uses Stop or Retry?

A reply can look fine while the chat is still open, but what matters is what the user sees after they reload the page.

Stop: If someone stops a reply halfway, the part they already saw should still be there after reload, and nothing past that point should appear.

Retry: NextChat replaces the old answer with a new one. The test would first check that the original answer, whether finished or cut off partway, was actually saved. Then it would click Retry, reload, and confirm the new answer replaced it.

That first step is important. If we only check that the new answer shows up, we can't tell whether the original was ever saved.

🧐 Proposed Solution

I maintain StreamTrust (https://github.com/santoshsibyala/streamtrust, MIT). Its NextChat preset drives the app in Playwright and reads the active session from IndexedDB. It covers four checks: a completed reply survives reload, Stop keeps the partial reply without late text, and both Retry paths (after a finished or an interrupted answer) replace the saved original.

The adapter is generated in the workflow with npx streamtrust@0.6.0 init nextchat, so nothing needs to be committed to NextChat beyond the workflow file.

📝 Additional Information

At NextChat commit 1d3dc33, all four checks passed three times locally and on Ubuntu. The local simulation of this CI workflow took 1 min 21 sec including setup. It uses Node 22, Yarn Classic, Playwright Chromium and a local mock provider, and needs no model key or database service. Ubuntu run: https://github.com/santoshsibyala/streamtrust/actions/runs/42577562171

After installing NextChat's Yarn dependencies, generate the adapter and install Playwright alongside it. The Action then tests the checked-out revision:

- name: Generate the NextChat adapter
  run: npx --yes streamtrust@0.6.0 init nextchat

- name: Install Playwright for the adapter
  working-directory: nextchat
  run: |
    npm install --prefix . --no-save --package-lock=false playwright@1.64.1
    ./node_modules/.bin/playwright install --with-deps chromium

- uses: santoshsibyala/streamtrust-action@v1
  with:
    adapter: nextchat/adapter.mjs
    version: '0.6.0'
    timeout-ms: '900000'
  env:
    STREAMTRUST_NEXTCHAT_ROOT: ${{ github.workspace }}
    STREAMTRUST_NEXTCHAT_REF: ${{ github.sha }}

Would you welcome a PR for an optional nightly workflow? If it's not a fit, no problem. I'm happy to keep running it downstream instead.