Skip to content

Support --format json for commands creating a run - #921

Merged
DavertMik merged 3 commits into
2.xfrom
feat/format-json-run-output
Aug 31, 2026
Merged

Support --format json for commands creating a run#921
DavertMik merged 3 commits into
2.xfrom
feat/format-json-run-output

Conversation

@DavertMik

@DavertMik DavertMik commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

--format json previously only worked for run --filter-list, where it prints the matching test IDs. For the commands that create a run the value was ignored — start always printed the bare run id, and run printed nothing at all to stdout.

Now --format json prints the created run on stdout:

{"runId":"a1b2c3d4","runUrl":"https://app.testomat.io/projects/demo/runs/a1b2c3d4","runPublicUrl":"https://app.testomat.io/p/a1b2c3d4"}
RUN_URL=$(npx @testomatio/reporter start --format json | jq -r .runUrl)

Covered paths:

  • start
  • run with no command (previously printed nothing to stdout, even with --format id)
  • run --remote <profile>
  • run "<command>" — printed before the runner is spawned, so it is the first stdout line

runUrl / runPublicUrl are omitted when Testomat.io did not return them.

JSON logs

With --format json the logs are machine-readable too — the CLI sets TESTOMATIO_LOG_JSON=1 and the logger prints one JSON object per line on stderr instead of [TESTOMATIO] prefixed text, so a failing run can be diagnosed without scraping text:

{"status":403,"method":"POST","url":"https://app.testomat.io/api/reporter","error":"Project API Token is invalid","response":{"message":"Project API Token is invalid"},"request":{"api_key":"<hidden>","kind":"manual","status":"scheduled"},"level":"error","message":"⚠️ Request to Testomat.io failed: ..."}
{"level":"error","message":"Error creating Testomat.io report (see details above), please check if your API key is valid. Skipping report"}
{"level":"error","message":"Failed to create run on Testomat.io."}

The JSON fields are built from the same values as the text message (no output is parsed), and the API token stays hidden. TESTOMATIO_LOG_JSON=1 works outside the CLI as well, for reporters running inside a test framework.

Token masking is done by the logger

hideTestomatioToken moved to src/utils/hide_token.js and is applied inside the logger — to the rendered message and to the serialized JSON fields — instead of per call site. Any log call that receives a raw request error is covered: util.format inspects the error and prints its config, request body included. A bare tstmt_… is masked as well, not only the quoted forms.

Other fixes in this branch

  • run without a command exited 0 when the run was not created — with --format json that is empty stdout plus a success code, indistinguishable from success. It exits 1 like start does, and both read the run id from the pipe store rather than from env vars.
  • A request body over 1000 characters is cut and is no longer valid JSON, so the JSON log adds "requestTruncated": true instead of silently making request a string.
  • Report and artifact status lines in client.js, pipe/html.js and pipe/csv.js used console.log directly, so they landed on stdout and polluted captured output. They go through log.info / log.warn / log.error now — unchanged for normal runs (INFO level, stdout), suppressed in machine-readable mode.

Behavior change to note

The error paths of the Testomat.io pipe used console.log / console.warn directly, bypassing the logger: the "⚠️ Request to Testomat.io failed" block and the "create an issue" hint went to stdout. They now go through log.error / log.warn, so they are printed to stderr with the [TESTOMATIO] prefix in text mode as well. This also fixes the failed-request blob leaking into RUN_ID=$(reporter start --format id) on a 403.

Backward compatibility

  • Any --format value other than json still prints the bare run id, so RUN_ID=$(reporter start --format id) is unchanged.
  • run --filter-list --format json keeps its current meaning (JSON array of test IDs) — no run is created there.

Notes

  • For run "<command>" the runner inherits stdout, so the docs recommend capturing the whole output and slicing the first line instead of piping into head (piping closes stdout under the running tests).
  • --format json request/response fields: response is an object, request is an object unless it was truncated (then a string plus requestTruncated: true).

Tests

  • formatRunOutput unit tests in tests/unit/pipe_utils_test.js
  • JSON logger tests in tests/unit/logger_test.js
  • End-to-end CLI tests against the mock server for all four run-creating paths, plus a 403 failure asserting JSON error lines on stderr and an empty stdout, in tests/unit/cli_start_remote_test.js
  • hideTestomatioToken tests in tests/unit/hide_token_test.js, including the escaped-JSON form printed by inspected errors
  • npm run lint, npm run build and the full unit suite (578 passing) are green

Docs updated in docs/cli.md (including the --format reference section, which claimed the format value was "not significant" for start and that --format only applied together with --filter-list) and docs/configuration.md for TESTOMATIO_LOG_JSON.

🤖 Generated with Claude Code

DavertMik and others added 3 commits August 23, 2026 02:47
--format json now prints the created run as {runId, runUrl, runPublicUrl}
for `start`, `run` without a command, `run --remote` and `run <command>`.
Any other format value keeps printing the bare run id, so existing
`--format id` scripts are unaffected, and `run --filter-list --format json`
keeps printing the array of matching test ids.

`run` without a command previously printed nothing to stdout even with
--format id.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Errors were still reported as [TESTOMATIO] prefixed text while the run
itself was printed as JSON. The logger now renders every message as one
JSON object per line when TESTOMATIO_LOG_JSON=1, which the CLI sets for
--format json. Failed requests to Testomat.io add their data as fields
(status, method, url, error, response, request) built from the same
values as the text message, with the API token hidden as before.

Error paths of the Testomat.io pipe went through console.log/console.warn
directly, so they bypassed the logger and printed to stdout: the failed
request details and the "create an issue" hint polluted the output
captured from --format commands. They are logged with log.error/log.warn
now, which prints them to stderr.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…created

The token was masked only where the failed request is formatted, so any
log call which received a raw request error printed it: util.format
inspects the error and prints its config, including the request body.
Masking moved to src/utils/hide_token.js and is applied by the logger to
every message and to the JSON fields, so it covers callers which don't
mask themselves. A bare `tstmt_...` is now masked too, not only the
quoted forms.

`run` without a command exited 0 when the run was not created, which with
--format json is indistinguishable from success: empty stdout, exit 0. It
exits 1 as `start` does, and both read the run id from the pipe store.

A cut request body is not valid JSON, so the JSON log marks it with
requestTruncated instead of silently turning `request` into a string.

Report and artifact status lines used console.log directly, so they went
to stdout and polluted the output captured from --format commands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@DavertMik
DavertMik merged commit 6ee4b6f into 2.x Aug 31, 2026
9 checks passed
@DavertMik
DavertMik deleted the feat/format-json-run-output branch August 31, 2026 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants