Acceptance suite - #295
Draft
raphiz wants to merge 4 commits into
Draft
Acceptance suite#295raphiz wants to merge 4 commits into
raphiz wants to merge 4 commits into
Conversation
Black-box tests over HTTP against the built image. A case is a vitest test that says what to do and asserts nothing; the harness records every request, response payload and log line and pins all of it under cases/<name>/golden/. Catches what unit tests on the host cannot: rendering drift from a playwright or base-image bump, wire-format changes, and broken or newly noisy logging. Goldens are docker-only, so pixels can be byte-exact. Volatile log fields (durations, ports, workspace paths) become placeholders so the field itself stays pinned while its value stops mattering. browserVersion stays verbatim - it is how a playwright bump announces itself. Readiness waits for the server's own "Server listening" line rather than probing a route, so no harness request pollutes a log golden. Regenerate with `npm run test:acceptance -- -u`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Mx7PVadB4KQULCMucVCFu
build-docker-image.yml pushed to ghcr.io with no test gate at all. Merge it into test.yml so `needs:` can do the gating - across workflows it cannot. One job builds, runs the acceptance suite, and only then pushes, so the image that reaches the registry is byte-for-byte the one that was just verified. The push steps are guarded by ref rather than by trigger; `on: [push]` already fires for tags, which is what the old `create` trigger covered. For Renovate automerge to actually wait on this, `docker-image` has to be added to the required status checks in branch protection. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Mx7PVadB4KQULCMucVCFu
test.yml no longer only tests - it builds and publishes the image too, so the name was misleading. Job names are unchanged, so required status checks still match. src/integrationtest.test.js is fully covered by the acceptance suite now: its render-then-check round trip is cases/plain and cases/form, and its differing-reference case is cases/mismatch. The acceptance versions assert byte-exact PNGs and an exact numDiffPixels instead of `length > 0` and `> 0`, and they do it against the real image - where MHTML rendering is actually reproducible. testdata/template.mhtml had no other user. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Mx7PVadB4KQULCMucVCFu
Every entry is exactly one HTTP request, so "steps" was vaguer than the thing it named - and `steps[i].step` stuttered. The request is now flattened into the entry, which avoids the `requests[i].request` stutter in turn, and the explicit number is gone since the array already orders them and the payload descriptors carry the filename. Only the six transcript.json files change; PNG and log goldens are byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Mx7PVadB4KQULCMucVCFu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.