Skip to content

test(promotion): stand the backend up and drive the primary journey on it - #616

Open
HomenShum wants to merge 2 commits into
mainfrom
test/live-backend-observability
Open

test(promotion): stand the backend up and drive the primary journey on it#616
HomenShum wants to merge 2 commits into
mainfrom
test/live-backend-observability

Conversation

@HomenShum

Copy link
Copy Markdown
Owner

Supersedes #615, which failed three style checks and carried packages/mcp-local/.mcpregistry_registry_token in its tree.

Wave 3 could not observe this product. Its cold reader wrote: "You cannot observe the half of the system that matters" — every product route rendered "Convex backend not configured" from a clean clone, so validation, orchestration, the model call, persistence and cancellation were readable and unit-tested but never runnable.

This stands up an isolated Convex dev deployment (never production), drives the primary journey end to end against it, and commits the captures plus their producers.

Conditions moved with evidence: 1, 3, 4, 5, 6, 9. Conditions 2, 7 and 8 stay FAIL — the audits ran and found real problems, which are recorded rather than smoothed over.

Rebased onto main, which removes the expired MCP publish token from the tree and widens the ignore pattern that let it in.

🤖 Generated with Claude Code

HomenShum and others added 2 commits August 13, 2026 20:41
Nine of the twelve promotion conditions are judged on what a browser shows,
and all nine were UNVERIFIED for one reason: every product route rendered
"Convex backend not configured" (defect D1). This provisions an isolated
Convex dev deployment, drives J1, J2 and J4 end to end against it, and
commits both the captures and the scripts that regenerate them.

Two things were actually broken behind that door, and neither names itself:

- @erquhart/convex-oss-stats imports @convex-dev/crons and declares it in
  neither dependencies nor peerDependencies. With package-lock.json
  gitignored, a fresh install can land a tree without it and the very first
  `convex dev` push dies on "Could not resolve
  @convex-dev/crons/convex.config". Declared directly.

- A missing OPTIONAL model key failed the ENTIRE deploy. Convex analyses
  every backend module on every push; coordinatorAgent.ts builds
  DEFAULT_MODEL (kimi-k2.6, OpenRouter) at module scope because a Convex
  function must be a module-level export; and buildLanguageModel threw at
  construction for an unconfigured provider. So `convex dev` failed with
  "InvalidModules: Failed to analyze domains/agents/digestAgent.js" unless
  you had an OpenRouter account -- for a provider /redesign/chat never
  calls. The error now lives on doGenerate/doStream instead, so an
  unconfigured provider fails the call that needs the key rather than the
  deploy that does not. The same shape is fixed for the google branch, and
  modelResolver.test.ts gates it: construct DEFAULT_MODEL with the key
  deleted, assert construction succeeds and the call still rejects.

New producers, each writing a committed artifact:

- scripts/capture-live-journey.mjs -- J1 (ask, stream, sealed packet),
  J2 (the permanent link replays instead of re-running, proven by
  getLatestOwnedRun().runId being unchanged), J4 (cancel, honest terminal
  state, keep working), the server-side validation error path, and the same
  journey at 375. It also reads the run row and its stream events back out
  of Convex, so "it streamed" and "it persisted" are separate claims.
- scripts/audit-web-quality.mjs -- Lighthouse 13.4.1 + @axe-core/cli 4.13.0
  against the production build under vite preview.
- scripts/review-web-interface-guidelines.mjs -- DOM measurements for the
  Web Interface Guidelines review written up in
  promotion/evidence/wig-review/REVIEW.md. A review, not a score; the two
  measure different things and are committed separately.

Scorecard: 1/12 -> 7/12, and UNVERIFIED is gone from the table. Conditions
1, 3, 4, 5, 6 and 9 move on committed evidence. Conditions 2, 7, 8, 10 and
11 FAIL on measured numbers, which is the point -- three new defects were
found by running the thing: D5 (a rejected prompt shows no reason on
screen), D6 (LCP 10.8 s plus four guideline deviations), D7 (the same
question returned 3, 2, 1, 0, 1 and 0 grounded sources across six runs).

Setup is written down in docs/START_HERE.md "Before Step 1" and
docs/codebase/CONCERNS.md C7b, including both traps above, so the next cold
reader does not rediscover them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gitguardian

gitguardian Bot commented Aug 14, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
36087749 Triggered Generic Password a4e9cd1 scripts/capture-live-journey.mjs View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@vercel

vercel Bot commented Aug 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
nodebench-ai Ready Ready Preview Aug 14, 2026 4:32am

Request Review

@github-actions

Copy link
Copy Markdown

PR size advisory

This PR adds 19309 lines of substantive change. CONTRIBUTING.md defines a soft limit of ~400 LOC.

If the PR is genuinely cohesive (e.g. an architecture map, a generated migration, a deletion of a dead module), no action is needed. Otherwise consider:

  • splitting into 2-3 PRs along independent concerns
  • pre-discussing the architecture change in a GitHub Discussion before merge

This is advisory — it does not block the merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant