fix(rerun): emit partial stdout on TimeoutError in single-FE rerun --wait#1
Open
Awad-de wants to merge 62 commits into
Open
fix(rerun): emit partial stdout on TimeoutError in single-FE rerun --wait#1Awad-de wants to merge 62 commits into
Awad-de wants to merge 62 commits into
Conversation
Owner
Author
|
My Discord |
…rd (TestSprite#37) assertNotLocal lowercased the hostname but did not strip a trailing dot, so http://localhost. (the FQDN form of localhost, RFC 6761) and http://localhost%2e bypassed the host === 'localhost' loopback check. IP literals are already dot-normalized by the WHATWG URL parser, so only named hosts were affected. Strips one trailing dot before the comparison. Adds 4 regression tests (3 blocked variants + 1 public-FQDN no-false-positive).
* fix(skill-nudge): require complete Codex managed section * docs(skill-nudge): document helper contracts --------- Co-authored-by: ahndohun <19940813+ahndohun@users.noreply.github.com>
TestSprite#36) project create/update validated --name with the action handler's if (!name) check, which a whitespace-only string passes (a non-empty string is truthy). The blank name was then sent verbatim, creating a junk-named project. The sibling est create already rejects this via the requireString whitespace guard (dogfood P1 fix #1); this aligns project create/update with that behavior. Adds 2 regression tests.
…Sprite#14) Only `test` and `project` validated the global `--output` flag. The `auth`, `usage`, `agent`, and `init` command groups resolved it with `globals.output ?? 'text'`, so an unrecognised value (e.g. a typo like `--output josn`) was silently coerced to text mode instead of being rejected. A coding agent that asked for `--output json` then received a human-readable text payload and failed to parse it as JSON, with no signal as to why. Extract the validation into a shared `resolveOutputMode` helper in `lib/output.js` and route every command group's `resolveCommonOptions` through it. Invalid values now throw a typed VALIDATION_ERROR (exit 5) with an actionable message everywhere. This also unifies the error wording, which previously differed between `test` ("Flag `--output` is invalid: must be one of: json, text.") and `project` ("--output must be one of: json, text").
…prite#34) * fix(test): reject empty code get --out and strip code-file BOM When test code get --out receives an empty inline body, closeOutputFile left a zero-byte file with exit 0 — scripts and agents treated that as a successful download. Abort the sink, unlink any artifact, and surface VALIDATION_ERROR instead. Plan/steps JSON already strip a leading UTF-8 BOM from PowerShell 5.1 files; apply the same strip to --code-file reads so uploaded test source is not corrupted by an invisible U+FEFF prefix. * fix(test): rebase onto v0.2.0 and harden empty-code --out cleanup - Resolve rebase conflicts: keep atomic temp-file --out writes from main while rejecting empty inline code with VALIDATION_ERROR (exit 5) - abortOutputFile: wait for stream close before unlinking tmpPath - BOM test: use .py code-file (assertPythonCodeFile from v0.2.0) - CI: build before test + fileParallelism false (dist/ race flake)
Co-authored-by: cmdr-chara <249489759+cmdr-chara@users.noreply.github.com>
…run and run --all (TestSprite#128) test run, test create, create-batch, plan put, code put, update, and delete all print the auto-minted idempotency key to stderr under --output json (as well as --verbose / --debug) so JSON-mode automation can capture the key and replay a retry safely. test rerun and test run --all minted a key but only echoed it under --debug / --verbose, so CI flows using --output json silently lost it. Align both paths with the shared guard used by every other minting site, and cover the JSON-mode emission (and the text-mode silence) with regression tests. Co-authored-by: ahndohun <19940813+ahndohun@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…alls (TestSprite#129) The codex managed-section branch already runs inspectTargetPath during --dry-run ([P2]) so a planted symlink is refused the same way the real install refuses it. The own-file targets (claude, cursor, cline, antigravity) skipped that guard in dry-run and reported the write as successful, so 'agent install --dry-run' could claim success for an install that would actually exit 5. Run the same guard in the own-file dry-run branch and cover both the symlinked-target and symlinked-parent cases with regression tests. Co-authored-by: ahndohun <19940813+ahndohun@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ared deadline (TestSprite#130) runTestRunAll computes each member's poll budget as Math.max(1, ceil(batchDeadlineMs - now)), so a run whose turn arrives after the shared --timeout deadline still gets a fresh >=1s poll. With --max-concurrency bounding the fan-out, a batch could overshoot the documented shared deadline and report a late 'passed' for a member that should have been reported as 'timeout' (exit 7). Guard the poll helper the same way the create-batch --run --wait path already does: if the shared deadline is exhausted before a member's poll starts, return the existing timeout-shaped member result without calling pollRunUntilTerminal. Regression test drives the clock past the deadline during the first member's poll and asserts the second member is never polled and reports 'timeout'. Co-authored-by: ahndohun <19940813+ahndohun@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…rd (TestSprite#37) assertNotLocal lowercased the hostname but did not strip a trailing dot, so http://localhost. (the FQDN form of localhost, RFC 6761) and http://localhost%2e bypassed the host === 'localhost' loopback check. IP literals are already dot-normalized by the WHATWG URL parser, so only named hosts were affected. Strips one trailing dot before the comparison. Adds 4 regression tests (3 blocked variants + 1 public-FQDN no-false-positive).
…TestSprite#17) `parseRequestTimeoutFlag` was copy-pasted byte-for-byte into five command files (auth, project, usage, init, test). Every copy silently returned `undefined` for an invalid value, so an explicit `--request-timeout 30s` (a natural "30 seconds" typo) resolved to undefined and the command ran with the default 120s deadline — the operator believed they had set a timeout but had not, with no signal. Hoist a single definition into client-factory.ts (next to resolveRequestTimeoutMs and the REQUEST_TIMEOUT_* constants) and make the flag strict: a non-numeric, zero, or negative value now throws a typed VALIDATION_ERROR (exit 5), consistent with every other validated flag (--page-size, --output, --type). Positive out-of-range values are still accepted and clamped by resolveRequestTimeoutMs, and the TESTSPRITE_REQUEST_TIMEOUT_MS env-var path stays lenient by design (a stray global env var should not hard-fail every command). Adds unit coverage for parseRequestTimeoutFlag and a subprocess regression that `--request-timeout 30s` exits 5 instead of falling back to 120s.
…wait
Apply the fix in src/commands/test.ts. When the overall --timeout polling
deadline is exceeded on a single FE rerun, emit {runId, status:"running"}
to stdout before exit 7.
Co-authored-by: Cursor <cursoragent@cursor.com>
Awad-de
force-pushed
the
fix/rerun-wait-timeout-stdout-pr
branch
from
121b257 to
cff19ae
Compare
…test create) (TestSprite#39) * fix(test): reject whitespace-only --name in test update (parity with test create) * style: run prettier on src/commands/test.ts
…stSprite#12) * feat(cli): respect NO_COLOR environment variable per no-color.org * fix(ticker): treat empty NO_COLOR as color-enabled per no-color.org
…prite#48) * fix(paginate): break loop on empty items with non-null nextToken paginate() in pagination.ts entered an infinite loop when the API returned { items: [], nextToken: 'cursor' }. The loop only checked nextToken !== null, never whether items was empty. Any filtered list command returning zero results would hang indefinitely. Add an early-exit guard: if the page contains zero items, break regardless of nextToken. Includes 5 unit tests covering the bug reproduction, maxItems cap, single-page, and consecutive empty pages. Closes TestSprite#35 Signed-off-by: Aldo Rizona <aldorizona10@gmail.com> * style(test): reflow pagination.test.ts to satisfy prettier format:check gate Pure mechanical prettier reflow of two vi.fn(async () => ({...})) callbacks in test/pagination.test.ts. No logic change — only line wrapping to satisfy the format:check CI gate (npm run format:check). All other gates already pass. Coverage remains >=80% on all 4 metrics (lines/statements/functions/branches). * fix(paginate): bound cursor loops without dropping empty pages --------- Signed-off-by: Aldo Rizona <aldorizona10@gmail.com>
…Sprite#166) A successful (200) response whose body is not JSON — a misconfigured endpoint, a proxy / captive-portal / login page returning HTML with a success status, or an empty body — caused the OK path to call raw `response.json()`, whose SyntaxError escaped to the top-level handler. That produced an opaque exit 1 and, under --output json, a bare `{"error":"<parse message>"}` that breaks the typed-envelope contract every other error honors (the non-OK path already reads defensively via safeReadJson). Wrap the OK-path parse failure in a typed INTERNAL ApiError (exit 1 unchanged) that carries the requestId, names the likely cause, and points the operator at their endpoint config; details include the HTTP status, content-type, and underlying parse message. Abort/timeout errors mid-read keep their existing classification. Fixes TestSprite#94
TestSprite#33) pollAccepted in runTestRerun hardcoded exitCode:1 on ApiError, causing the auth-escalation find(r => r.error?.exitCode === 3) to always return undefined -- auth failures silently exited 1 instead of 3. - preserve err.exitCode in pollAccepted (mirrors runTestRunAll fix) - add auth escalation block before generic exit-1 throw - bound initial chunk idempotency key to <=256 chars (mirrors retry path)
* feat(agent): add kiro as an install target
Adds kiro as an own-file agent target on the current multi-skill model: AgentTarget union, a pathFor('kiro') case landing at .kiro/skills/<skill>/SKILL.md, and a TARGETS entry (experimental, own-file, wrapSkill frontmatter like claude/antigravity). Kiro installs both default skills (testsprite-verify + testsprite-onboard). Updates the --target help string, README/DOCUMENTATION target lists and counts, unit + command tests (six keys, list 12 rows, five own-file targets, 10 dry-run would-write lines, renderForTarget + content-integrity coverage), and regenerates the agent/setup help snapshots. Rebuilt on current main.
* test(e2e): include kiro in target matrix guards and multi-target install
The Local E2E Tests CI job failed because two matrix-coverage guards hardcoded the target set (claude, antigravity, cursor, cline, codex) and did not include the new kiro target. Add kiro (own-file, between cline and codex to match TARGETS order) to both guards, and add kiro to the multi-target install e2e so the target is actually exercised end-to-end.
…nd test wait (TestSprite#153) Apply the fix in src/commands/ (the compiled CLI path). When the overall --timeout polling deadline is exceeded, emit {runId, status:"running"} to stdout before exit 7 so JSON agents can chain into test wait. Co-authored-by: Contributor <contributor@testsprite.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…ressions (TestSprite#168) * feat(test): add "test diff <run-a> <run-b>" to isolate run-to-run regressions * test(diff): cover runDiff --dry-run branch (offline canned sample)
… files in --out dir (TestSprite#162) commitBundle's stale-file sweep removed EVERY directory entry not part of the fresh bundle, so 'test failure get --out <dir>' / 'test artifact get --out <dir>' pointed at a pre-existing, populated directory silently deleted the user's unrelated files (exit 0, no warning) — on the very first write, not just re-commits. Scope the sweep to entries the bundle format owns (result.json, failure.json, video.mp4, meta.json, steps, .tmp, .partial, code.<ext>). Stale bundle files are still cleaned — an old video.mp4 when the new bundle has no video, a code.py when the new bundle writes code.ts — but foreign files and directories are never touched. Fixes TestSprite#159 Co-authored-by: Kshitij Bhardwaj <tothemoon202154@outlook.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…al without --all (TestSprite#163) Two silent-footgun gaps in test rerun's flag validation: 1. Explicit test IDs combined with --all silently discarded the listed IDs — the --all branch resolves the full project test set and overwrites them — so 'rerun test_abc --all' dispatched a batch rerun of EVERY test in the project, burning rerun/auto-heal credits with no error. Both siblings already guard this exact ambiguity (test run's positional+--all guard, delete-batch's ids+--all guard). 2. --status <list> and --skip-terminal without --all were silently ignored — including INVALID --status values, which were never validated — while the same misuse of rerun's own --filter (and delete-batch's --status) exits 5. All three narrowing filters now share the same guard. Both reject with VALIDATION_ERROR (exit 5) before any network dispatch. Fixes TestSprite#160 Co-authored-by: Kshitij Bhardwaj <tothemoon202154@outlook.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…e#171) The 'testsprite usage' command help text listed three examples but omitted the --debug flag (a global option useful for diagnosing auth and network issues). It also didn't document the exit codes, which matters for CI/CD scripts that gate on the return value. This PR adds: - A --debug example showing what it traces (HTTP method/path, request id, latency) — useful when debugging 'auth error' or 'transport failure' messages. - An explicit exit-codes section (0 success, 3 auth error, 10 network failure) so users scripting against the CLI know what to expect. Pure documentation improvement — no behavioral change, no new deps. Tested: existing unit tests pass (npm test). Co-authored-by: MOAAMN SAYED <moaamnsayed560@gmail.com>
…t pure) (TestSprite#31) Interactive prompts (`prompt.ts` — the API-key prompt during `setup`/`auth configure`, the target prompt during `agent install`) wrote the question and masking to STDOUT, and the "Configuring profile …" prelude defaulted to stdout too. On the interactive path that mixes UI text into stdout — and under `--output json` it breaks the contract that stdout is a single JSON document, so a consumer doing `JSON.parse(stdout)` fails. Default both to stderr: prompts and informational preludes are interactive UI, not result data. stdout now carries only the command's result (the §8.1 stdout-purity principle the repo already enforces elsewhere). stderr is still the user's TTY, so prompts remain visible; the secret is still never echoed. Callers that inject explicit streams are unaffected. Adds regression tests: promptText writes the question to stderr by default, and the configure prelude lands on stderr (not the result stdout).
* block artifact download redirects * redact artifact download urls --------- Co-authored-by: merlinsantiago982-cmd <merlinsantiago982-cmd@users.noreply.github.com>
* fix(test): validate artifact run id default path * fix(test): reject windows dot artifact run ids * fix(test): reject windows dot-suffix artifact run ids * style(test): format artifact run id guard --------- Co-authored-by: Lexiie <28455136+Lexiie@users.noreply.github.com>
…TestSprite#11) * feat(cli): add runtime Node.js version check with clear error message * refactor(version-guard): extract to a documented module tested against the real implementation
Windsurf (Cascade) reads workspace rules from `.windsurf/rules/*.md`. Add it as an own-file agent target so `testsprite agent install --target windsurf` (and `setup --agent windsurf`) installs the TestSprite skills into a Windsurf project. Reworked onto the v0.2.0 multi-skill agent-targets API (pathFor / SKILLS / DEFAULT_SKILLS). Rule files use Cascade frontmatter with `trigger: model_decision` — the equivalent of the Cursor `.mdc` `alwaysApply: false` mode (description shown up front; full body pulled in on relevance). Budget handling: a `.windsurf/rules/*.md` file caps at ~12 K characters and Cascade silently truncates beyond it, which would cut the full ~22 KB verify skill in half. The windsurf target therefore renders the COMPACT body per skill (new `compactBody` flag + `compactBodyFor`): a skill that ships a trimmed codex asset (`testsprite-verify` → ~5 KB) uses it, while a skill whose codex contribution is only a one-liner (`testsprite-onboard`, ~6.5 KB full) keeps its full body — both land well under the cap. `agent.ts` and `renderForTarget` select the same body so installed bytes match the render. Everything else derives from the TARGETS map automatically (agent list, the setup --agent choices, skill-nudge install detection). Updated the hardcoded help strings, the --help snapshot, the agent-targets/agent unit tests (incl. Cascade-frontmatter and per-skill budget tests), the e2e matrix guards / content-integrity (gated on compactBody), and the README/DOCUMENTATION target lists (incl. the --force own-file list).
… CI proxies (TestSprite#169) * feat(cli): honor HTTPS_PROXY/HTTP_PROXY/NO_PROXY behind corporate and CI proxies * fix(proxy): degrade to default dispatcher when proxy agent init fails instead of crashing startup * fix(proxy): pin undici to ^7.16.0 for Node 20 compatibility (8.x requires Node >=22.19)
…m check, opt-out, CI-safe) (TestSprite#181)
…e#132) * feat(cli): add 'test flaky' repeat-run flaky-test detector * fix(flaky): cap --runs at 10 per maintainer scope (TestSprite#115) Rescope the flaky detector's --runs bound from 1-100 to 1-10 as requested in the TestSprite#115 triage: uncapped FE replays amplify free executions. Updates the MAX_FLAKY_RUNS constant (which drives the validation, error message, and --runs help text), docs, changelog, and the runs-bound tests. Regenerates the help snapshot, which also adds the previously-missing 'test flaky' entry. * test(snapshot): refresh flaky help snapshot after rebase onto main Rebasing onto current main (which added the global --request-timeout option, TestSprite#17) changes the 'Global options' line rendered in the test flaky --help output. Regenerate the snapshot so the help snapshot test stays green on CI. * docs(changelog): resolve leftover merge-conflict markers (keep JUnit + flaky 1-10)
* fix(config): treat empty env vars as unset in loadConfig * test: address review — dry-run whoami uses offline client; fix whitespace env key expectation
* fix(http): clear per-attempt timeout timers * fix(http): preserve timeout race classification
Bumps [marocchino/sticky-pull-request-comment](https://github.com/marocchino/sticky-pull-request-comment) from 2.9.4 to 3.0.5. - [Release notes](https://github.com/marocchino/sticky-pull-request-comment/releases) - [Commits](marocchino/sticky-pull-request-comment@7737449...5770ad5) --- updated-dependencies: - dependency-name: marocchino/sticky-pull-request-comment dependency-version: 3.0.5 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…an-out (TestSprite#154) * fix(batch): classify RequestTimeoutError as timeout in --all --wait fan-out When test run --all --wait or test rerun --all --wait hit a per-request timeout during batch fan-out polling, RequestTimeoutError rejected the fan-out before out.print(). Classify it as status:'timeout' in pollFreshAccepted and pollAccepted so stdout always lists every dispatched runId. Co-authored-by: Cursor <cursoragent@cursor.com> * Remove unused readFileSync import from test file --------- Co-authored-by: Contributor <contributor@testsprite.com> Co-authored-by: Cursor <cursoragent@cursor.com>
… invocation (TestSprite#178) * feat(test): make "test wait" variadic — attach to several runs in one invocation * fix(wait): honor the shared deadline for queued members and hint only resumable runs
* feat(agent): add GitHub Copilot as an install target Adds `copilot` to the agent-install targets. GitHub Copilot reads path-specific custom instructions from `.github/instructions/*.instructions.md` (VS Code / Visual Studio / Copilot Chat), with YAML frontmatter carrying an `applyTo` glob. The skill installs to `.github/instructions/testsprite-verify.instructions.md` (and the onboard skill alongside) with `applyTo: '**'` so the guidance attaches to every request in the repo. Because `applyTo: '**'` is always-on (Copilot has no on-demand 'model decides' mode like Cursor/Windsurf), the target renders the COMPACT body — the same reasoning behind windsurf's compact render — keeping the always-injected context small (~6 KB vs the ~23 KB full body). Slots into the existing TARGETS machinery, so `agent list`, `setup --agent`, and skill-nudge install-detection pick it up automatically. Updates the AgentTarget union, pathFor, TARGETS, help/docs, unit + e2e matrix guards, and the help snapshot. Fixes TestSprite#193 * fix(agent): include Kiro in the agent command description The top-level `agent` command description listed the other targets but omitted Kiro (a pre-existing gap), leaving it inconsistent with the `--target` help text and the docs. Align the description with the `--target` list so every supported target is named.
…#183) * feat(cli): add "testsprite doctor" environment diagnostic One-shot preflight: checks CLI version, Node runtime, profile, API endpoint, credentials, live connectivity (GET /me), and verify-skill install. Prints an OK/WARN/FAIL report and exits non-zero when any check fails, so it gates a CI step or agent preflight. Reuses the real resolution helpers; the API key is never printed. Fixes TestSprite#73 * fix(doctor): address review findings - Node check reuses the CLI runtime guard (shouldRejectNodeVersion) instead of a hardcoded major floor, so the verdict matches what the entrypoint enforces at startup; the precise 20.19+/22.13+/24+ engines are enforced by npm engine-strict. - --request-timeout raises a validation error on malformed input instead of silently defaulting, matching the other commands. - The --output json test asserts the API key never appears in the JSON path (distinct from the text renderer already covered).
…te#185) * feat(cli): graceful termination signals + broken-pipe guard Install handlers for SIGINT/SIGTERM/SIGHUP that print a one-line explanation (any started run keeps executing server-side; resume with `testsprite test list` or `testsprite test wait <runId>`) and exit with the conventional 128+signum code (SIGINT -> 130). Also guard EPIPE on stdout/stderr so piping to a reader that closes early (`... | head`) exits cleanly instead of dumping a raw `write EPIPE` stack. process and streams are injectable, so both are unit-tested without spawning a subprocess or sending a real signal. Fixes TestSprite#75 * fix(interrupt): flush the signal message synchronously before exit A signal handler calls process.exit() immediately after writing the interrupt hint. When stderr is a pipe, an async process.stderr.write() may not flush before the process terminates, so the hint could be lost. The default stderr writer now uses fs.writeSync (best-effort, guarded against EPIPE) so the hint is reliably emitted. Added a test mocking fs.writeSync to assert the synchronous write on the default path.
… or backend skeleton (TestSprite#180)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes an asymmetry in
testsprite test rerun --wait(single FE rerun path):when the overall
--timeoutpolling deadline is exceeded, the CLI now emits apartial
{ runId, status: "running" }object to stdout before throwingexit 7 — matching the behavior already present for
RequestTimeoutErrorandthe recently fixed
test run --wait/test waitpaths.Before:
TimeoutError→ throwApiErrorimmediately → stdout empty in--output jsonmode → AI agents must scrape stderr to recover the runId.After:
TimeoutError→out.print({ runId, status: "running" })→ throw→ caller can programmatically chain into
testsprite test wait <runId>.Related issue
Hackathon CLI Improvement — agent recovery on polling timeout (complements
the
test run --wait/test waitTimeoutError fix; this covers theremaining single-FE
test rerun --waitpath).Type of change
Checklist
mainbranch.(
feat(...),fix(...),docs(...), …).npm run lintandnpm run format:checkpass.npm run typecheckpasses.npm testpasses and coverage stays at or above the 80% gate.credentials required).
README.md/DOCUMENTATION.mdwhererelevant. (N/A — internal error-handling parity fix, no CLI surface change.)
Notes for reviewers
Root cause
In
runTestRerun(single FE rerun, no BE closure), theTimeoutErrorcatchblock at ~line 6132 called
ticker.finalize()then threwApiErrorwithoutany
out.print(). The adjacentRequestTimeoutErrorbranch already emitteda partial run to stdout — this was simply a missed mirror.
Scope
test rerun --waitTimeoutErrorpath only (~12 lines).into the batch result object on timeout;
test run --wait/test waitwere fixed in a prior PR.
Test added
[finding-4]intest.rerun.spec.ts:TimeoutErrorviatimeoutSeconds: 0(immediate deadline){ runId, status: "running" }Verification
npm test → 1571 passed npm run typecheck → clean npm run lint:fix → clean