Discord Username / User ID
JerryNee / jerry_nee
What does this improvement do?
testsprite test flaky documents auth failures as exit code 3, but the current rerun trigger path records most API errors as non-fatal flaky attempts. If the rerun trigger returns AUTH_REQUIRED or AUTH_FORBIDDEN, the command can emit a stability report and exit 1 instead of surfacing the standard auth error envelope / exit 3. That makes CI and agents misdiagnose a credentials problem as a flaky or failing test.
Details / implementation notes
Observed in current main:
src/commands/test.ts documents test flaky exit code 3 for auth errors in the command help.- In
runFlaky, the client.triggerRerun(...) catch block only rethrows NOT_FOUND. Every other ApiError is pushed into attempts as { outcome: "error", failureKind: code }.
flakyExitCode then maps the resulting non-stable report to exit 1.
Expected behavior:
AUTH_REQUIRED and AUTH_FORBIDDEN from the rerun trigger should propagate as normal auth errors (exit 3), matching the help text and the rest of the CLI's run/wait behavior.- In
--output json, credentials failures should produce the standard error envelope, not a flaky stability report.
NOT_FOUND for a missing replayable run should keep the current exit-4 behavior.
Suggested implementation:
- In the rerun-trigger catch block, rethrow fatal API errors before recording a non-fatal attempt error. At minimum this should include
AUTH_REQUIRED and AUTH_FORBIDDEN.
- Add focused coverage in
src/commands/test.flaky.spec.ts for auth errors during the rerun trigger.
- Keep transient per-attempt behavior unchanged for genuinely non-fatal trigger/transport blips.
Confirmations
Discord Username / User ID
JerryNee / jerry_nee
What does this improvement do?
testsprite test flakydocuments auth failures as exit code 3, but the current rerun trigger path records most API errors as non-fatal flaky attempts. If the rerun trigger returnsAUTH_REQUIREDorAUTH_FORBIDDEN, the command can emit a stability report and exit 1 instead of surfacing the standard auth error envelope / exit 3. That makes CI and agents misdiagnose a credentials problem as a flaky or failing test.Details / implementation notes
Observed in current main:
src/commands/test.tsdocumentstest flakyexit code 3 for auth errors in the command help.runFlaky, theclient.triggerRerun(...)catch block only rethrowsNOT_FOUND. Every otherApiErroris pushed intoattemptsas{ outcome: "error", failureKind: code }.flakyExitCodethen maps the resulting non-stable report to exit 1.Expected behavior:
AUTH_REQUIREDandAUTH_FORBIDDENfrom the rerun trigger should propagate as normal auth errors (exit 3), matching the help text and the rest of the CLI's run/wait behavior.--output json, credentials failures should produce the standard error envelope, not a flaky stability report.NOT_FOUNDfor a missing replayable run should keep the current exit-4 behavior.Suggested implementation:
AUTH_REQUIREDandAUTH_FORBIDDEN.src/commands/test.flaky.spec.tsfor auth errors during the rerun trigger.Confirmations