From 059612ea20b309ad22127a3ca7d805af2058e590 Mon Sep 17 00:00:00 2001 From: DavertMik Date: Sun, 23 Aug 2026 02:47:23 +0300 Subject: [PATCH 1/3] Support --format json for commands creating a run --format json now prints the created run as {runId, runUrl, runPublicUrl} for `start`, `run` without a command, `run --remote` and `run `. Any other format value keeps printing the bare run id, so existing `--format id` scripts are unaffected, and `run --filter-list --format json` keeps printing the array of matching test ids. `run` without a command previously printed nothing to stdout even with --format id. Co-Authored-By: Claude Opus 5 (1M context) --- docs/cli.md | 58 ++++++++++++++++++++------ src/bin/cli.js | 23 ++++++++--- src/utils/pipe_utils.js | 22 ++++++++++ tests/unit/cli_start_remote_test.js | 64 +++++++++++++++++++++++++++++ tests/unit/pipe_utils_test.js | 39 +++++++++++++++++- 5 files changed, 187 insertions(+), 19 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 95888184..901c2277 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -29,7 +29,7 @@ npx @testomatio/reporter [options] Starts a new test run and returns its ID. This requires an API key to be set in the `TESTOMATIO` environment variable. -With `--format id` (or any `--format`), `start` prints **only the run id to `stdout`** (the banner and progress logs go to `stderr`), so it is safe to capture: `RUN_ID=$(npx @testomatio/reporter start --format id)`. It exits non-zero if the run could not be created. +With `--format id`, `start` prints **only the run id to `stdout`** (the banner and progress logs go to `stderr`), so it is safe to capture: `RUN_ID=$(npx @testomatio/reporter start --format id)`. Use `--format json` to get the run id together with its URLs as a JSON object. It exits non-zero if the run could not be created. **Usage:** @@ -56,7 +56,7 @@ npx @testomatio/reporter start --filter "testomatio:tag-name=smoke" - `--env-file `: Load environment variables from a specific env file. If none specified, it will look for `.env` file. - `--kind `: Specify run type: `automated`, `manual`, `mixed`, or `detect`. Determines how the test run is categorized in Testomat.io. See [Detecting the run kind](#11-detecting-the-run-kind). - `--filter `: Scope the prepared run to the tests matching the filter (same syntax as [`run --filter`](#31-filter-pipes)). The run is created with that test list but **not** executed — useful to prepare a run and launch it later on CI (see [Prepare a run, then launch it on CI](#34-prepare-a-run-then-launch-it-on-ci)). -- `--format `: Print **only the run id** to `stdout` (banner and logs go to `stderr`) so it can be captured: `RUN_ID=$(npx @testomatio/reporter start --format id)`. +- `--format `: Machine-readable output on `stdout` (banner and logs go to `stderr`): `id` prints the bare run id so it can be captured — `RUN_ID=$(npx @testomatio/reporter start --format id)` — and `json` prints `{"runId", "runUrl", "runPublicUrl"}`. - `--warn`: Exit `0` instead of `1` when the filter matches no tests — the warning is still printed. Use in pipelines where an empty scope is a normal outcome (e.g. a PR touching no mapped files). The run is created as **scheduled**, not running: nothing has been executed yet. Testomat.io promotes it to *running* as soon as the first test result is reported, or when you launch it by hand. @@ -128,7 +128,7 @@ Alias for this command – `test`, e.g. `npx @testomatio/reporter test [options] - `-c, --command `: Test runner command (required). - `--filter `: [Filter executed tests](./pipes/testomatio.md#filter-tests) by tag, label, jira, plan. - `--filter-list `: Print the list of tests matching the filter without running them. Useful for inspecting which tests would run, or for piping IDs into another command. See [Coverage Pipe](./pipes/coverage.md#machine-readable-output-with---format) for examples. -- `--format `: Machine-readable output format for `--filter-list`. Supported values: `grep`, `json`, `newline`, `ids`. When set, the CLI banner is suppressed and informational logs go to `stderr` so `stdout` stays clean for piping. +- `--format `: Machine-readable output on `stdout`. With `--filter-list` it selects how the matched test IDs are encoded (`grep`, `json`, `newline`, `ids`); otherwise it prints the run the command creates — `id` for the bare run id, `json` for `{"runId", "runUrl", "runPublicUrl"}`. When set, the CLI banner is suppressed and informational logs go to `stderr` so `stdout` stays clean for piping. - `--env-file `: Load environment variables from a specific env file. - `--kind `: Specify run type: `automated`, `manual`, `mixed`, or `detect`. Determines how the test run is categorized in Testomat.io. `detect` needs a run scoped to a list of tests to resolve from, which `run --filter` does not create — see [Detecting the run kind](#11-detecting-the-run-kind). - `--remote `: Trigger the run on a CI profile configured on the Testomat.io project (e.g. `github`, `gitlab`, `jenkins`) instead of executing tests locally. The CLI creates the run on Testomat.io, asks the backend to dispatch the named CI workflow, and exits. Equivalent to setting [`TESTOMATIO_CI_PROFILE`](./configuration.md#testomatio_ci_profile). @@ -394,7 +394,7 @@ For more details about debug files, see the [Debug Pipe documentation](pipes/deb ## The `--format` flag -`--format` switches a command into **machine-readable mode**: `stdout` carries only the requested data so it can be captured or piped, while the banner and progress logs are routed to `stderr`. Two commands support it — [`run --filter-list`](#3-run) and [`start`](#1-start) — and machine-readable mode behaves the same way for both. +`--format` switches a command into **machine-readable mode**: `stdout` carries only the requested data so it can be captured or piped, while the banner and progress logs are routed to `stderr`. It is supported by [`start`](#1-start) and [`run`](#3-run), and machine-readable mode behaves the same way for both. **What machine-readable mode does (regardless of command or value):** @@ -407,7 +407,7 @@ This is what makes `$( … )` capture and `|` piping reliable — without `--for ### With `run --filter-list` -Prints the IDs of the tests matching the filter **without running them**. `--format` only takes effect together with `--filter-list`; the value selects the encoding: +Prints the IDs of the tests matching the filter **without running them**. The value selects the encoding: | Value | Output | Example | | --------- | ----------------------------- | -------------------- | @@ -427,30 +427,64 @@ GREP=$(npx @testomatio/reporter run --filter-list "coverage:file=coverage.yml" - npx @testomatio/reporter run --filter-list "coverage:file=coverage.yml" --format json > affected-tests.json ``` +With `--filter-list` no run is created, so `--format json` prints the array of matching test IDs — not the run object described below. + Only the `testomatio:` and `coverage:` filter pipes are supported (see [3.1 Filter pipes](#31-filter-pipes)). The [Coverage Pipe docs](./pipes/coverage.md#machine-readable-output-with---format) cover the formats in more detail. -### With `start` +### With a command that creates a run + +`start` always creates a run, and so does `run` when it is not listing tests with `--filter-list`. For those invocations `--format` selects how the created run is printed to `stdout`: -Prints **only the new run id** to `stdout`, so it can be captured directly: +| Value | Output | +| ------ | ---------------------------------------------------------- | +| `id` | the bare run id (default for any value other than `json`) | +| `json` | a JSON object with the run id and its URLs | ```bash RUN_ID=$(npx @testomatio/reporter start --format id) echo "$RUN_ID" # e.g. a1b2c3d4 + +npx @testomatio/reporter start --format json +# {"runId":"a1b2c3d4","runUrl":"https://app.testomat.io/projects/demo/runs/a1b2c3d4","runPublicUrl":"https://app.testomat.io/p/a1b2c3d4"} +``` + +`runUrl` and `runPublicUrl` are omitted when Testomat.io did not return them. Pick single fields with any JSON tool: + +```bash +RUN_URL=$(npx @testomatio/reporter start --format json | jq -r .runUrl) ``` -`start` emits a single value (the run id), so for `start` the format **value is not significant** — `--format id` is the conventional choice, but any value turns on machine-readable mode. `start` exits non-zero if the run could not be created, so `RUN_ID` is set only on success. It combines with `--kind` and `--filter`: +`start` exits non-zero if the run could not be created, so `RUN_ID` is set only on success. It combines with `--kind` and `--filter`: ```bash RUN_ID=$(npx @testomatio/reporter start --kind manual --format id) RUN_ID=$(npx @testomatio/reporter start --filter "testomatio:tag-name=smoke" --format id) ``` +The same output is printed by every `run` invocation that creates a run: + +```bash +# create a run without executing tests +RUN_ID=$(npx @testomatio/reporter run --format id) + +# trigger a run on a CI profile +npx @testomatio/reporter run --remote github --format json + +# executing tests: the runner inherits stdout, so the run data is the FIRST line. +# Capture it all, then slice — piping into `head` closes stdout under the running tests. +OUT=$(npx @testomatio/reporter run "npx playwright test" --format json) +RUN=$(printf '%s\n' "$OUT" | head -1) +``` + ### Quick reference -| Command | Accepted `--format` values | `stdout` contains | Needs | -| ------------------- | -------------------------------- | ---------------------- | -------------------- | -| `run --filter-list` | `ids`, `grep`, `json`, `newline` | the matching test IDs | `--filter-list` | -| `start` | any (use `id`) | the new run id | — | +| Command | Accepted `--format` values | `stdout` contains | +| -------------------------- | -------------------------------- | --------------------------------------- | +| `run --filter-list` | `ids`, `grep`, `json`, `newline` | the matching test IDs, no run created | +| `start` | `id`, `json` | the new run | +| `run` (no command) | `id`, `json` | the new run | +| `run --remote ` | `id`, `json` | the run triggered on CI | +| `run ""` | `id`, `json` | the new run on the first line, then the runner output (capture, do not pipe) | ## Environment Variables diff --git a/src/bin/cli.js b/src/bin/cli.js index 5f0acd2d..6dc52d6c 100755 --- a/src/bin/cli.js +++ b/src/bin/cli.js @@ -16,7 +16,7 @@ import { filesize as prettyBytes } from 'filesize'; import dotenv from 'dotenv'; import Replay from '../replay.js'; import { log } from '../utils/log.js'; -import { formatFilterListIds } from '../utils/pipe_utils.js'; +import { formatFilterListIds, formatRunOutput } from '../utils/pipe_utils.js'; import fs from 'fs'; import path from 'path'; @@ -53,7 +53,7 @@ program .description('Start a new run and return its ID') .option('--kind ', 'Specify run type: automated, manual, mixed, or detect') .option('--filter ', 'Scope the prepared run to tests matching the filter (no execution)') - .option('--format ', 'Machine-readable output: print only the run id to stdout (e.g. --format id)') + .option('--format ', 'Machine-readable output: the run id (--format id) or run details (--format json)') .option('--warn', 'Exit 0 instead of 1 when the filter matches no tests (warn only)') .action(async opts => { cleanLatestRunId(); @@ -93,8 +93,8 @@ program const plannedTests = (client.pipeStore.preparedTestIds || []).map(id => ({ test_id: id, title: id })); await client.updateRunStatus('pending', { tests: plannedTests }); - // stdout carries ONLY the run id so it can be captured: RUN_ID=$(reporter start) - console.log(runId); + // stdout carries ONLY the run data so it can be captured: RUN_ID=$(reporter start) + console.log(formatRunOutput({ ...client.pipeStore, runId }, opts.format)); process.exit(0); }); @@ -129,7 +129,10 @@ program .argument('[command]', 'Test runner command') .option('--filter ', 'Additional execution filter') .option('--filter-list ', 'Get a list of all tests by filter before running') - .option('--format ', 'Machine-readable output format for --filter-list (grep, json, newline, ids)') + .option( + '--format ', + 'Machine-readable output: test ids for --filter-list (grep, json, newline, ids), or the run created (id, json)', + ) .option('--kind ', 'Specify run type: automated, manual, mixed, or detect') .option('--remote ', 'Trigger run on the named Testomat.io CI profile instead of executing locally') .option( @@ -230,6 +233,8 @@ program log.info(`🚀 CI build triggered on profile ${pc.cyan(opts.remote)}`); log.info(`📊 Report URL: ${pc.magenta(client.pipeStore.runUrl)}`); + const remoteOutput = formatRunOutput(client.pipeStore, opts.format); + if (opts.format && remoteOutput) console.log(remoteOutput); return process.exit(0); } @@ -252,6 +257,8 @@ program log.info( `No command passed, so you need to run tests yourself:`); log.info( `TESTOMATIO_RUN=${runId} `); } + const runOutput = formatRunOutput({ ...client.pipeStore, runId }, opts.format); + if (opts.format && runOutput) console.log(runOutput); } else { log.info( '⚠️ No API key provided. Cannot create run without TESTOMATIO key.'); process.exit(1); @@ -288,7 +295,11 @@ program } if (apiKey) { - await client.createRun(createRunParams).then(runTests); + await client.createRun(createRunParams); + // the runner inherits stdout, so the run data is printed first, on its own line + const createdOutput = formatRunOutput(client.pipeStore, opts.format); + if (opts.format && createdOutput) console.log(createdOutput); + await runTests(); } else { await runTests(); } diff --git a/src/utils/pipe_utils.js b/src/utils/pipe_utils.js index 76411a15..f6f45278 100644 --- a/src/utils/pipe_utils.js +++ b/src/utils/pipe_utils.js @@ -304,6 +304,27 @@ function plannedTestsLabel(tests, testsCount) { return `**${knownTestsCount}** tests and **${suitesCount}** suites planned`; } +/** + * Format the created run for machine-readable output of `start` and `run`. + * `json` prints an object with the run details, any other format prints the bare run id. + * + * @param {{runId?: string, runUrl?: string, runPublicUrl?: string}} store - Pipe store of the client. + * @param {string} [format] - Value of the CLI `--format` option. + * @returns {string} Empty string if there is no run id. + */ +function formatRunOutput(store, format) { + const runId = store?.runId; + if (!runId) return ''; + + if (format !== 'json') return runId; + + const output = { runId }; + if (store.runUrl) output.runUrl = store.runUrl; + if (store.runPublicUrl) output.runPublicUrl = store.runPublicUrl; + + return JSON.stringify(output); +} + export { updateFilterType, parseFilterParams, @@ -317,6 +338,7 @@ export { plannedTestsLabel, parsePipeOptions, formatFilterListIds, + formatRunOutput, getObjectSize, splitTestsIntoChunks, }; diff --git a/tests/unit/cli_start_remote_test.js b/tests/unit/cli_start_remote_test.js index edd90999..0880e0a7 100644 --- a/tests/unit/cli_start_remote_test.js +++ b/tests/unit/cli_start_remote_test.js @@ -69,6 +69,18 @@ describe('cli start / run --remote', () => { expect(stdout.trim()).to.equal('startrun123'); }); + it('with --format json prints the run details as JSON on stdout', async () => { + server.on(replyRun('startrun456')); + + const { code, stdout } = await runCli(['start', '--format', 'json']); + + expect(code).to.equal(0); + const output = JSON.parse(stdout.trim()); + expect(output.runId).to.equal('startrun456'); + expect(output.runUrl).to.equal(`${TESTOMATIO_URL}/projects/demo/runs/startrun456`); + expect(output.runPublicUrl).to.equal(`${TESTOMATIO_URL}/p/startrun456`); + }); + it('exits non-zero when the run is not created', async () => { server.on(replyRun('ignored', 500)); @@ -79,6 +91,47 @@ describe('cli start / run --remote', () => { }); }); + describe('run (no command)', () => { + it('with --format id prints ONLY the run id on stdout', async () => { + server.on(replyRun('createdrun1')); + + const { code, stdout } = await runCli(['run', '--format', 'id']); + + expect(code).to.equal(0); + expect(stdout.trim()).to.equal('createdrun1'); + }); + + it('with --format json prints the run details as JSON on stdout', async () => { + server.on(replyRun('createdrun2')); + + const { code, stdout } = await runCli(['run', '--format', 'json']); + + expect(code).to.equal(0); + const output = JSON.parse(stdout.trim()); + expect(output.runId).to.equal('createdrun2'); + expect(output.runUrl).to.equal(`${TESTOMATIO_URL}/projects/demo/runs/createdrun2`); + expect(output.runPublicUrl).to.equal(`${TESTOMATIO_URL}/p/createdrun2`); + }); + }); + + describe('run ', () => { + it('with --format json prints the run as the first stdout line, before the runner output', async () => { + server.on(replyRun('execrun1')); + server.on({ + method: 'PUT', + path: '/api/reporter/execrun1', + reply: { status: 200, headers: { 'content-type': 'application/json' }, body: JSON.stringify({}) }, + }); + + const { code, stdout } = await runCli(['run', 'echo hello', '--format', 'json']); + + expect(code).to.equal(0); + const [firstLine, ...rest] = stdout.trim().split('\n'); + expect(JSON.parse(firstLine).runId).to.equal('execrun1'); + expect(rest.join('\n')).to.include('hello'); + }); + }); + describe('run --remote', () => { it('reports success and exits 0 when CI launch succeeds', async () => { server.on(replyRun('ciRun1')); @@ -89,6 +142,17 @@ describe('cli start / run --remote', () => { expect(stdout).to.include('CI build triggered'); }); + it('with --format json prints the triggered run as JSON on stdout', async () => { + server.on(replyRun('ciRun3')); + + const { code, stdout } = await runCli(['run', '--remote', 'github', '--format', 'json']); + + expect(code).to.equal(0); + const output = JSON.parse(stdout.trim()); + expect(output.runId).to.equal('ciRun3'); + expect(output.runUrl).to.equal(`${TESTOMATIO_URL}/projects/demo/runs/ciRun3`); + }); + it('exits non-zero and does NOT report success when CI launch fails', async () => { server.on(replyRun('ciRun2', 400)); diff --git a/tests/unit/pipe_utils_test.js b/tests/unit/pipe_utils_test.js index 301be613..1890469e 100644 --- a/tests/unit/pipe_utils_test.js +++ b/tests/unit/pipe_utils_test.js @@ -1,5 +1,10 @@ import { expect } from 'chai'; -import { formatFilterListIds, getObjectSize, splitTestsIntoChunks } from '../../src/utils/pipe_utils.js'; +import { + formatFilterListIds, + formatRunOutput, + getObjectSize, + splitTestsIntoChunks, +} from '../../src/utils/pipe_utils.js'; describe('formatFilterListIds', () => { const ids = ['t1234abcd', 't5678efgh', 'tabcdef01']; @@ -97,6 +102,38 @@ describe('formatFilterListIds', () => { }); }); +describe('formatRunOutput', () => { + const store = { + runId: 'run123', + runUrl: 'https://app.testomat.io/projects/demo/runs/run123', + runPublicUrl: 'https://app.testomat.io/p/run123', + }; + + it('returns empty string when there is no run id', () => { + expect(formatRunOutput({}, 'json')).to.equal(''); + expect(formatRunOutput(undefined, 'id')).to.equal(''); + }); + + it('prints the bare run id for non-json formats', () => { + expect(formatRunOutput(store, 'id')).to.equal('run123'); + expect(formatRunOutput(store, undefined)).to.equal('run123'); + }); + + it('prints run details as JSON for the json format', () => { + const output = JSON.parse(formatRunOutput(store, 'json')); + expect(output).to.deep.equal({ + runId: 'run123', + runUrl: 'https://app.testomat.io/projects/demo/runs/run123', + runPublicUrl: 'https://app.testomat.io/p/run123', + }); + }); + + it('omits urls the pipe did not provide', () => { + const output = JSON.parse(formatRunOutput({ runId: 'run123' }, 'json')); + expect(output).to.deep.equal({ runId: 'run123' }); + }); +}); + describe('splitTestsIntoChunks', () => { // build a test whose serialized size is roughly `bytes` const makeTest = (i, bytes = 100) => ({ From f256a66be7d35fcfbd4c5372fa58c6d4343a4a84 Mon Sep 17 00:00:00 2001 From: DavertMik Date: Sun, 23 Aug 2026 14:46:32 +0300 Subject: [PATCH 2/3] Print logs as JSON when --format json is used Errors were still reported as [TESTOMATIO] prefixed text while the run itself was printed as JSON. The logger now renders every message as one JSON object per line when TESTOMATIO_LOG_JSON=1, which the CLI sets for --format json. Failed requests to Testomat.io add their data as fields (status, method, url, error, response, request) built from the same values as the text message, with the API token hidden as before. Error paths of the Testomat.io pipe went through console.log/console.warn directly, so they bypassed the logger and printed to stdout: the failed request details and the "create an issue" hint polluted the output captured from --format commands. They are logged with log.error/log.warn now, which prints them to stderr. Co-Authored-By: Claude Opus 5 (1M context) --- docs/cli.md | 18 ++++++ docs/configuration.md | 16 +++++ src/bin/cli.js | 2 + src/pipe/testomatio.js | 49 +++++++++++----- src/utils/log.js | 71 ++++++++++++++++++++--- tests/unit/cli_start_remote_test.js | 34 +++++++++++ tests/unit/logger_test.js | 74 +++++++++++++++++++++++- tests/unit/pipes/testomatio_pipe_test.js | 16 +++-- 8 files changed, 253 insertions(+), 27 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 901c2277..ef6d192f 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -405,6 +405,24 @@ For more details about debug files, see the [Debug Pipe documentation](pipes/deb This is what makes `$( … )` capture and `|` piping reliable — without `--format`, the banner and `[TESTOMATIO]` logs are interleaved on `stdout`. +**With `--format json` the logs are machine-readable too.** Every reporter message is printed to `stderr` as one JSON object per line instead of prefixed text, so a failing run can be diagnosed without scraping text: + +```bash +npx @testomatio/reporter start --format json 2>errors.log +``` + +```json +{"level":"error","message":"Error creating Testomat.io report (see details above), please check if your API key is valid. Skipping report"} +``` + +A failed request to Testomat.io adds its data as fields — `status`, `method`, `url`, `error`, `response` and `request` (with the API token hidden): + +```json +{"status":403,"method":"POST","url":"https://app.testomat.io/api/reporter","error":"Project API Token is invalid","level":"error","message":"⚠️ Request to Testomat.io failed: ..."} +``` + +The same output is enabled outside the CLI with [`TESTOMATIO_LOG_JSON=1`](./configuration.md#testomatio_log_json), which the CLI also passes to the test runner it spawns. + ### With `run --filter-list` Prints the IDs of the tests matching the filter **without running them**. The value selects the encoding: diff --git a/docs/configuration.md b/docs/configuration.md index aa968fb7..4c170c98 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -338,6 +338,22 @@ TESTOMATIO_LOG_LEVEL=ERROR npm test > 📖 See [Log Level Control](./log-level.md) for more details. > 🐛 For detailed debugging, use the `DEBUG` environment variable with the debug package specified. +#### `TESTOMATIO_LOG_JSON` + +Print every `[TESTOMATIO]` message as a JSON object instead of prefixed text, one object per line: + +```json +{"level":"error","message":"Error creating Testomat.io report ..."} +``` + +Failed API requests add their data as fields (`status`, `method`, `url`, `error`, `response`, `request`) — the API token is hidden there as it is in the text output. + +``` +TESTOMATIO_LOG_JSON=1 npm test +``` + +The CLI sets this automatically for [`--format json`](./cli.md#the---format-flag), so both the output and the logs of a run are machine-readable. + #### `TESTOMATIO_UPDATE_CODE` Sends the `code` of your tests to Testomat.io on each run. (If not enabled (default) assumes the code is pushed using [check-tests](https://github.com/testomatio/check-tests#cli)). diff --git a/src/bin/cli.js b/src/bin/cli.js index 6dc52d6c..724d0741 100755 --- a/src/bin/cli.js +++ b/src/bin/cli.js @@ -43,6 +43,8 @@ program if (subOpts.filterList || subOpts.format) { process.env.TESTOMATIO_LOG_STDERR = '1'; process.env.TESTOMATIO_LOG_LEVEL ||= 'WARN'; + // with --format json the logs are machine-readable too: one JSON object per line on stderr + if (subOpts.format === 'json') process.env.TESTOMATIO_LOG_JSON = '1'; } else { console.log(pc.cyan(pc.bold(` 🤩 Testomat.io Reporter v${version}`))); } diff --git a/src/pipe/testomatio.js b/src/pipe/testomatio.js index 50cfb664..f71ace86 100644 --- a/src/pipe/testomatio.js +++ b/src/pipe/testomatio.js @@ -380,16 +380,15 @@ class TestomatioPipe { process.env.runId = this.runId; debug('Run created', this.runId); } catch (err) { - if (!this.apiKey) console.error('Testomat.io API key is not set'); + if (!this.apiKey) log.error('Testomat.io API key is not set'); const errorText = err.response?.data?.message || err.message; debug('Error creating run', err); - console.log(APP_PREFIX, errorText || err); + log.error(errorText || err); if (err.response?.status === 403) this.#disablePipe(); this.#logFailedResponse(err); - console.error( - APP_PREFIX, + log.error( 'Error creating Testomat.io report (see details above), please check if your API key is valid. Skipping report', ); printCreateIssue(); @@ -409,7 +408,7 @@ class TestomatioPipe { this.reportingCanceledDueToReqFailures = true; let errorMessage = `⚠️ ${process.env.TESTOMATIO_MAX_REQUEST_FAILURES}`; errorMessage += ' requests were failed, reporting to Testomat aborted.'; - console.warn(`${APP_PREFIX} ${pc.yellow(errorMessage)}`); + log.warn(pc.yellow(errorMessage)); } return cancelReporting; } @@ -557,7 +556,7 @@ class TestomatioPipe { const errorMessage = pc.red( `⚠️ Due to request failures, ${this.notReportedTestsCount} test(s) were not reported to Testomat.io`, ); - console.warn(`${APP_PREFIX} ${errorMessage}`); + log.warn(errorMessage); } const { status } = params; @@ -621,7 +620,7 @@ class TestomatioPipe { ); } } catch (err) { - console.log(APP_PREFIX, 'Error updating status, skipping...', err); + log.error('Error updating status, skipping...', err); this.#logFailedResponse(err); printCreateIssue(); } @@ -665,18 +664,29 @@ class TestomatioPipe { message += `\t${pc.bold('response: ')}${pc.gray(responseBody)}\n`; - const requestBody = hideTestomatioToken(stringify(error.response?.config?.data)); + let requestBody = hideTestomatioToken(stringify(error.response?.config?.data)); if (process.env.DEBUG || process.env.TESTOMATIO_DEBUG || requestBody.length < 1000) { // full body message += `\t${pc.bold('request: ')}${pc.gray(requestBody)}\n`; } else { // cut body - const requestBodyCut = requestBody.slice(0, 1000); - message += `\t${pc.bold('request: ')}${pc.gray(`${requestBodyCut}...`)}\n`; + requestBody = `${requestBody.slice(0, 1000)}...`; + message += `\t${pc.bold('request: ')}${pc.gray(requestBody)}\n`; message += '\trequest body is cut, run with TESTOMATIO_DEBUG=1 to see full body\n'; } - console.log(message); + // the JSON line is built from the same values as the text message, with the token already hidden + log.errorWithFields( + { + status: statusCode, + method, + url, + error: apiMessage || statusText || undefined, + response: parseIfJson(responseBody), + request: parseIfJson(requestBody), + }, + message, + ); if (error.response?.data?.message?.includes('could not be matched')) { this.hasUnmatchedTests = true; @@ -693,8 +703,7 @@ function printCreateIssue() { if (registeredErrorHints) return; registeredErrorHints = true; process.on('exit', () => { - console.log( - APP_PREFIX, + log.error( 'There was an error reporting to Testomat.io.\n', pc.yellow( 'If you think this is a bug please create an issue: https://github.com/testomatio/reporter/issues/new.', @@ -723,6 +732,20 @@ function hideTestomatioToken(data) { * @param {{ pretty: boolean }} opts * @returns {string} */ +/** + * Turn a JSON string back into an object for structured logs; keeps the string if it is not JSON. + * + * @param {string} data + * @returns {any} + */ +function parseIfJson(data) { + try { + return JSON.parse(data); + } catch { + return data; + } +} + function stringify(anything, opts = { pretty: false }) { return typeof anything === 'string' ? anything : JSON.stringify(anything, null, opts.pretty ? 2 : undefined); } diff --git a/src/utils/log.js b/src/utils/log.js index 5db5a1fa..b868a81b 100644 --- a/src/utils/log.js +++ b/src/utils/log.js @@ -1,5 +1,8 @@ +import { format as formatArgs, stripVTControlCharacters } from 'util'; import { APP_PREFIX } from '../constants.js'; +const stripColors = stripVTControlCharacters || (str => str?.replace(/\x1b\[[0-9;]*m/g, '') || ''); + /** * Log levels for the Testomat.io reporter. * A message is logged if its level is <= the current log level. @@ -35,16 +38,44 @@ export function shouldLog(messageLevel) { return messageLevel <= getLogLevel() || !!process.env.TESTOMATIO_DEBUG; } +/** + * Check if logs should be printed as JSON lines instead of [TESTOMATIO] prefixed text. + * Enabled by the CLI for `--format json` so the whole output is machine-readable. + * @returns {boolean} + */ +export function isJsonOutput() { + return process.env.TESTOMATIO_LOG_JSON === '1'; +} + +/** + * Render a log message as a single JSON line, e.g. `{"level":"error","message":"..."}`. + * Colors are stripped, so the message stays readable after parsing. + * @param {string} level - Log level name + * @param {any[]} args - Arguments as passed to the log function + * @param {Object} [fields] - Extra fields to add to the JSON object + * @returns {string} + */ +function jsonLine(level, args, fields = {}) { + const message = stripColors(formatArgs(...args)).trim(); + return JSON.stringify({ ...fields, level, message }); +} + /** * Log an info message with [TESTOMATIO] prefix. * Only logs when TESTOMATIO_LOG_LEVEL is INFO. * @param {...any} args - Arguments to log */ export function info(...args) { - if (shouldLog(LOG_LEVELS.INFO)) { - const fn = process.env.TESTOMATIO_LOG_STDERR === '1' ? console.error : console.log; - fn(APP_PREFIX, ...args); + if (!shouldLog(LOG_LEVELS.INFO)) return; + + let fn = console.log; + if (process.env.TESTOMATIO_LOG_STDERR === '1') fn = console.error; + + if (isJsonOutput()) { + fn(jsonLine('info', args)); + return; } + fn(APP_PREFIX, ...args); } /** @@ -53,9 +84,13 @@ export function info(...args) { * @param {...any} args - Arguments to log */ export function warn(...args) { - if (shouldLog(LOG_LEVELS.WARN)) { - console.warn(APP_PREFIX, ...args); + if (!shouldLog(LOG_LEVELS.WARN)) return; + + if (isJsonOutput()) { + console.warn(jsonLine('warn', args)); + return; } + console.warn(APP_PREFIX, ...args); } /** @@ -64,9 +99,29 @@ export function warn(...args) { * @param {...any} args - Arguments to log */ export function error(...args) { - if (shouldLog(LOG_LEVELS.ERROR)) { - console.error(APP_PREFIX, ...args); + if (!shouldLog(LOG_LEVELS.ERROR)) return; + + if (isJsonOutput()) { + console.error(jsonLine('error', args)); + return; + } + console.error(APP_PREFIX, ...args); +} + +/** + * Log an error which carries structured data, e.g. a failed API request. + * The fields are added to the JSON line; in text mode only the message is printed. + * @param {Object} fields - Extra fields to add to the JSON object + * @param {...any} args - Arguments to log as text + */ +export function errorWithFields(fields, ...args) { + if (!shouldLog(LOG_LEVELS.ERROR)) return; + + if (isJsonOutput()) { + console.error(jsonLine('error', args, fields)); + return; } + console.error(APP_PREFIX, ...args); } /** @@ -82,6 +137,8 @@ export const log = { info, warn, error, + errorWithFields, + isJsonOutput, getLogLevel, shouldLog, LOG_LEVELS, diff --git a/tests/unit/cli_start_remote_test.js b/tests/unit/cli_start_remote_test.js index 0880e0a7..573e24fc 100644 --- a/tests/unit/cli_start_remote_test.js +++ b/tests/unit/cli_start_remote_test.js @@ -81,6 +81,40 @@ describe('cli start / run --remote', () => { expect(output.runPublicUrl).to.equal(`${TESTOMATIO_URL}/p/startrun456`); }); + it('with --format json reports failures as JSON lines on stderr, keeping stdout empty', async () => { + server.on({ + method: 'POST', + path: '/api/reporter', + reply: { + status: 403, + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ message: 'Project API Token is invalid' }), + }, + }); + + const { code, stdout, stderr } = await runCli(['start', '--format', 'json']); + + expect(code).to.equal(1); + expect(stdout.trim()).to.equal(''); + + // the failed request is reported with its data, not as a formatted text blob + const failure = stderr + .split('\n') + .filter(line => line.startsWith('{')) + .map(line => JSON.parse(line)) + .find(entry => entry.status === 403); + + expect(failure).to.exist; + expect(failure.level).to.equal('error'); + expect(failure.method).to.equal('POST'); + expect(failure.url).to.include('/api/reporter'); + expect(failure.error).to.equal('Project API Token is invalid'); + expect(failure.response).to.deep.equal({ message: 'Project API Token is invalid' }); + expect(failure.request.api_key).to.equal(''); + + expect(stderr).to.not.include('[TESTOMATIO] '); + }); + it('exits non-zero when the run is not created', async () => { server.on(replyRun('ignored', 500)); diff --git a/tests/unit/logger_test.js b/tests/unit/logger_test.js index 8200c68a..8fbb38ed 100644 --- a/tests/unit/logger_test.js +++ b/tests/unit/logger_test.js @@ -1,5 +1,6 @@ import { expect } from 'chai'; -import { log, info, warn, error, LOG_LEVELS } from '../../src/utils/log.js'; +import pc from 'picocolors'; +import { log, info, warn, error, errorWithFields, LOG_LEVELS } from '../../src/utils/log.js'; describe('Logger Utility', () => { let originalEnv; @@ -230,6 +231,77 @@ describe('Logger Utility', () => { }); }); + describe('JSON output (TESTOMATIO_LOG_JSON=1)', () => { + let calls; + let originalError; + let originalWarn; + + beforeEach(() => { + process.env.TESTOMATIO_LOG_JSON = '1'; + calls = []; + originalError = console.error; + originalWarn = console.warn; + console.error = (...args) => calls.push(args); + console.warn = (...args) => calls.push(args); + }); + + afterEach(() => { + console.error = originalError; + console.warn = originalWarn; + delete process.env.TESTOMATIO_LOG_JSON; + }); + + it('prints errors as a single JSON object with level and message', () => { + error('something went wrong'); + + expect(calls.length).to.equal(1); + expect(JSON.parse(calls[0][0])).to.deep.equal({ level: 'error', message: 'something went wrong' }); + }); + + it('prints warnings as JSON and strips colors from the message', () => { + warn(pc.yellow('be careful')); + + expect(JSON.parse(calls[0][0])).to.deep.equal({ level: 'warn', message: 'be careful' }); + }); + + it('keeps the [TESTOMATIO] prefix out of the JSON output', () => { + error('plain message'); + + expect(calls[0][0]).to.not.include('[TESTOMATIO]'); + }); + + it('formats multiple arguments into one message, as the text logger does', () => { + error('failed:', { status: 403 }); + + expect(JSON.parse(calls[0][0]).message).to.equal('failed: { status: 403 }'); + }); + + it('adds structured fields of errorWithFields to the JSON object', () => { + errorWithFields({ status: 403, url: 'https://app.testomat.io/api/reporter' }, 'Request failed'); + + expect(JSON.parse(calls[0][0])).to.deep.equal({ + status: 403, + url: 'https://app.testomat.io/api/reporter', + level: 'error', + message: 'Request failed', + }); + }); + + it('never lets a field override the log level', () => { + errorWithFields({ level: 'info' }, 'still an error'); + + expect(JSON.parse(calls[0][0]).level).to.equal('error'); + }); + + it('prints prefixed text when JSON output is disabled', () => { + delete process.env.TESTOMATIO_LOG_JSON; + errorWithFields({ status: 403 }, 'Request failed'); + + expect(calls[0][0]).to.include('[TESTOMATIO]'); + expect(calls[0][1]).to.equal('Request failed'); + }); + }); + describe('LOG_LEVELS constant', () => { it('should have all expected log levels', () => { expect(LOG_LEVELS).to.have.property('ERROR', 0); diff --git a/tests/unit/pipes/testomatio_pipe_test.js b/tests/unit/pipes/testomatio_pipe_test.js index c7855bfa..41aee169 100644 --- a/tests/unit/pipes/testomatio_pipe_test.js +++ b/tests/unit/pipes/testomatio_pipe_test.js @@ -1297,6 +1297,8 @@ describe('TestomatioPipe', () => { let pipe; let consoleLogOutput; let originalRequest; + let originalLog; + let originalError; beforeEach(() => { process.env.TESTOMATIO_URL = TESTOMATIO_URL; @@ -1311,21 +1313,23 @@ describe('TestomatioPipe', () => { pipe.runId = 'test-run-123'; consoleLogOutput = []; - const originalLog = console.log; + originalLog = console.log; + originalError = console.error; + // errors are logged to stderr, so both streams are captured console.log = (...args) => { consoleLogOutput.push(args.join(' ')); }; + console.error = (...args) => { + consoleLogOutput.push(args.join(' ')); + }; // Store original request method to restore later originalRequest = pipe.client.request; - - return () => { - console.log = originalLog; - }; }); afterEach(() => { - console.log = global.console.log; + console.log = originalLog; + console.error = originalError; delete process.env.TESTOMATIO_URL; // Restore original request method pipe.client.request = originalRequest; From 32547cf7c863a3d55497a63314c29ec6e2778db6 Mon Sep 17 00:00:00 2001 From: DavertMik Date: Sun, 23 Aug 2026 16:25:25 +0300 Subject: [PATCH 3/3] Hide the API token in every log, fail the run command when no run is created The token was masked only where the failed request is formatted, so any log call which received a raw request error printed it: util.format inspects the error and prints its config, including the request body. Masking moved to src/utils/hide_token.js and is applied by the logger to every message and to the JSON fields, so it covers callers which don't mask themselves. A bare `tstmt_...` is now masked too, not only the quoted forms. `run` without a command exited 0 when the run was not created, which with --format json is indistinguishable from success: empty stdout, exit 0. It exits 1 as `start` does, and both read the run id from the pipe store. A cut request body is not valid JSON, so the JSON log marks it with requestTruncated instead of silently turning `request` into a string. Report and artifact status lines used console.log directly, so they went to stdout and polluted the output captured from --format commands. Co-Authored-By: Claude Opus 5 (1M context) --- docs/cli.md | 4 +-- docs/configuration.md | 2 +- src/bin/cli.js | 15 ++++++--- src/client.js | 8 ++--- src/pipe/csv.js | 9 +++--- src/pipe/html.js | 17 +++++----- src/pipe/testomatio.js | 48 ++++++++++++----------------- src/utils/hide_token.js | 12 ++++++++ src/utils/log.js | 25 ++++++++++----- tests/unit/cli_start_remote_test.js | 10 ++++++ tests/unit/hide_token_test.js | 30 ++++++++++++++++++ tests/unit/logger_test.js | 27 ++++++++++++++++ 12 files changed, 147 insertions(+), 60 deletions(-) create mode 100644 src/utils/hide_token.js create mode 100644 tests/unit/hide_token_test.js diff --git a/docs/cli.md b/docs/cli.md index ef6d192f..44f49767 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -415,7 +415,7 @@ npx @testomatio/reporter start --format json 2>errors.log {"level":"error","message":"Error creating Testomat.io report (see details above), please check if your API key is valid. Skipping report"} ``` -A failed request to Testomat.io adds its data as fields — `status`, `method`, `url`, `error`, `response` and `request` (with the API token hidden): +A failed request to Testomat.io adds its data as fields — `status`, `method`, `url`, `error`, `response` and `request`. The API token is hidden in every log, whichever field or message it appears in. A request body longer than 1000 characters is cut, which makes `request` a string instead of an object and adds `"requestTruncated": true`: ```json {"status":403,"method":"POST","url":"https://app.testomat.io/api/reporter","error":"Project API Token is invalid","level":"error","message":"⚠️ Request to Testomat.io failed: ..."} @@ -472,7 +472,7 @@ npx @testomatio/reporter start --format json RUN_URL=$(npx @testomatio/reporter start --format json | jq -r .runUrl) ``` -`start` exits non-zero if the run could not be created, so `RUN_ID` is set only on success. It combines with `--kind` and `--filter`: +Every command listed here exits non-zero and prints nothing to `stdout` if the run could not be created, so a captured variable is set only on success. `start` combines with `--kind` and `--filter`: ```bash RUN_ID=$(npx @testomatio/reporter start --kind manual --format id) diff --git a/docs/configuration.md b/docs/configuration.md index 4c170c98..11e8ef62 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -346,7 +346,7 @@ Print every `[TESTOMATIO]` message as a JSON object instead of prefixed text, on {"level":"error","message":"Error creating Testomat.io report ..."} ``` -Failed API requests add their data as fields (`status`, `method`, `url`, `error`, `response`, `request`) — the API token is hidden there as it is in the text output. +Failed API requests add their data as fields (`status`, `method`, `url`, `error`, `response`, `request`). The API token is hidden in every message and field, in JSON as well as in text output. ``` TESTOMATIO_LOG_JSON=1 npm test diff --git a/src/bin/cli.js b/src/bin/cli.js index 724d0741..3c1a8f18 100755 --- a/src/bin/cli.js +++ b/src/bin/cli.js @@ -84,7 +84,7 @@ program await client.createRun(createRunParams); - const runId = client.pipeStore.runId || process.env.runId; + const runId = client.pipeStore.runId; if (!runId) { log.error(pc.red('Failed to create run on Testomat.io.')); process.exit(1); @@ -96,7 +96,7 @@ program await client.updateRunStatus('pending', { tests: plannedTests }); // stdout carries ONLY the run data so it can be captured: RUN_ID=$(reporter start) - console.log(formatRunOutput({ ...client.pipeStore, runId }, opts.format)); + console.log(formatRunOutput(client.pipeStore, opts.format)); process.exit(0); }); @@ -252,15 +252,20 @@ program if (apiKey) { await client.createRun(createRunParams); - const runId = process.env.TESTOMATIO_RUN || process.env.runId; + + const runId = client.pipeStore.runId; + if (!runId) { + log.error(pc.red('Failed to create run on Testomat.io.')); + process.exit(1); + } + if (client.pipeStore.runUrl) log.info( `📊 Report URL: ${pc.magenta(client.pipeStore.runUrl)}`); if (opts.kind !== 'manual') { log.info( `No command passed, so you need to run tests yourself:`); log.info( `TESTOMATIO_RUN=${runId} `); } - const runOutput = formatRunOutput({ ...client.pipeStore, runId }, opts.format); - if (opts.format && runOutput) console.log(runOutput); + if (opts.format) console.log(formatRunOutput(client.pipeStore, opts.format)); } else { log.info( '⚠️ No API key provided. Cannot create run without TESTOMATIO key.'); process.exit(1); diff --git a/src/client.js b/src/client.js index f752f782..48a10f92 100644 --- a/src/client.js +++ b/src/client.js @@ -187,7 +187,7 @@ class Client { } } catch (err) { - console.error(APP_PREFIX, 'Error in uploadStepArtifacts for testRid', testRid, ':', err); + log.error('Error in uploadStepArtifacts for testRid', testRid, ':', err.message || err); throw err; } } @@ -225,7 +225,7 @@ class Client { try { await this.uploadStepArtifacts(steps, rid); } catch (err) { - console.log(APP_PREFIX, 'Failed to upload step artifacts:', err); + log.error('Failed to upload step artifacts:', err.message || err); } const uploadedFiles = []; @@ -423,7 +423,7 @@ class Client { const pathPadding = Math.max(...failedUploads.map(upload => upload.relativePath.length)) + 1; failedUploads.forEach(upload => { - console.log( + log.info( ` ${pc.gray('|')} 🔴 ${upload.relativePath.padEnd(pathPadding)} ${pc.gray( `| ${upload.sizePretty.padStart(filesizeStrMaxLength)} |`, )}`, @@ -439,7 +439,7 @@ class Client { })); const pathPadding = Math.max(...skippedUploads.map(upload => upload.relativePath.length)) + 1; skippedUploads.forEach(upload => { - console.log( + log.info( ` ${pc.gray('|')} 🟡 ${upload.relativePath.padEnd(pathPadding)} ${pc.gray( `| ${upload.sizePretty.padStart(filesizeStrMaxLength)} |`, )}`, diff --git a/src/pipe/csv.js b/src/pipe/csv.js index 4d812c79..d9b19e90 100644 --- a/src/pipe/csv.js +++ b/src/pipe/csv.js @@ -6,6 +6,7 @@ import pc from 'picocolors'; import merge from 'lodash.merge'; import { isSameTest, getCurrentDateTime, ansiRegExp } from '../utils/utils.js'; import { CSV_HEADERS } from '../constants.js'; +import { log } from '../utils/log.js'; const debug = createDebugMessages('@testomatio/reporter:pipe:csv'); /** @@ -76,11 +77,11 @@ class CsvPipe { this.checkExportDir(); if (!this.outputFile) { - console.log(pc.yellow(`⚠️ CSV file is not set, ignoring`)); + log.warn(pc.yellow(`⚠️ CSV file is not set, ignoring`)); return; } - console.log(pc.yellow(`⏳ The test results will be added to the csv. It will take some time...`)); + log.info(pc.yellow(`⏳ The test results will be added to the csv. It will take some time...`)); try { // Create csv writer object @@ -91,7 +92,7 @@ class CsvPipe { // Save csv file based on the current data return await writer.writeRecords(data); } catch (e) { - console.log('Unknown csv error: ', e); + log.error('Unknown csv error: ', e); } } @@ -135,7 +136,7 @@ class CsvPipe { // Save results based on the default headers if (this.isEnabled) { await this.saveToCsv(this.results, CSV_HEADERS); - console.log(pc.green(`🗃️ Recording completed! You can check the result in file = ${this.outputFile}`)); + log.info(pc.green(`🗃️ Recording completed! You can check the result in file = ${this.outputFile}`)); } } diff --git a/src/pipe/html.js b/src/pipe/html.js index 583b25d5..fdd9e4e1 100644 --- a/src/pipe/html.js +++ b/src/pipe/html.js @@ -8,6 +8,7 @@ import { marked } from 'marked'; import fileUrl from 'file-url'; import { fileSystem, isSameTest, ansiRegExp, formatStep, transformEnvVarToBoolean } from '../utils/utils.js'; import { HTML_REPORT } from '../constants.js'; +import { log } from '../utils/log.js'; import { fileURLToPath } from 'node:url'; const debug = createDebugMessages('@testomatio/reporter:pipe:html'); @@ -144,14 +145,14 @@ class HtmlPipe { debug('HTML tests data:', tests); if (!outputPath) { - console.log(pc.yellow(`🚨 HTML export path is not set, ignoring...`)); + log.warn(pc.yellow(`🚨 HTML export path is not set, ignoring...`)); return; } - console.log(pc.yellow(`⏳ The test results will be added to the HTML report. It will take some time...`)); + log.info(pc.yellow(`⏳ The test results will be added to the HTML report. It will take some time...`)); if (msg) { - console.log(pc.blue(msg)); + log.info(pc.blue(msg)); } const aggregatedTests = aggregateTestRetries(tests); @@ -295,9 +296,9 @@ class HtmlPipe { debug('HTML tests data:', fileUrlPath); - console.log(pc.green(`📊 The HTML report was successfully generated. Full filepath: ${fileUrlPath}`)); + log.info(pc.green(`📊 The HTML report was successfully generated. Full filepath: ${fileUrlPath}`)); } else { - console.log(pc.red(`🚨 Failed to generate the HTML report.`)); + log.error(pc.red(`🚨 Failed to generate the HTML report.`)); } } @@ -309,7 +310,7 @@ class HtmlPipe { */ #generateHTMLReport(data, templatePath = '') { if (!templatePath) { - console.log(pc.red(`🚨 HTML template not found. Report generation is impossible!`)); + log.error(pc.red(`🚨 HTML template not found. Report generation is impossible!`)); return; } @@ -320,8 +321,8 @@ class HtmlPipe { return template(data); } catch (e) { - console.log(pc.red('❌ Oops! An unknown error occurred when generating an HTML report')); - console.log(pc.red(e)); + log.error(pc.red('❌ Oops! An unknown error occurred when generating an HTML report')); + log.error(pc.red(e)); } } diff --git a/src/pipe/testomatio.js b/src/pipe/testomatio.js index f71ace86..e3f16d89 100644 --- a/src/pipe/testomatio.js +++ b/src/pipe/testomatio.js @@ -18,6 +18,7 @@ import { getGitCommitSha, } from '../utils/utils.js'; import { parseFilterParams, generateFilterRequestParams, setS3Credentials } from '../utils/pipe_utils.js'; +import { hideTestomatioToken } from '../utils/hide_token.js'; import { config } from '../config.js'; import { log } from '../utils/log.js'; @@ -620,7 +621,7 @@ class TestomatioPipe { ); } } catch (err) { - log.error('Error updating status, skipping...', err); + log.error('Error updating status, skipping...', err.message || err); this.#logFailedResponse(err); printCreateIssue(); } @@ -665,28 +666,31 @@ class TestomatioPipe { message += `\t${pc.bold('response: ')}${pc.gray(responseBody)}\n`; let requestBody = hideTestomatioToken(stringify(error.response?.config?.data)); + let requestTruncated = false; if (process.env.DEBUG || process.env.TESTOMATIO_DEBUG || requestBody.length < 1000) { // full body message += `\t${pc.bold('request: ')}${pc.gray(requestBody)}\n`; } else { // cut body + requestTruncated = true; requestBody = `${requestBody.slice(0, 1000)}...`; message += `\t${pc.bold('request: ')}${pc.gray(requestBody)}\n`; message += '\trequest body is cut, run with TESTOMATIO_DEBUG=1 to see full body\n'; } // the JSON line is built from the same values as the text message, with the token already hidden - log.errorWithFields( - { - status: statusCode, - method, - url, - error: apiMessage || statusText || undefined, - response: parseIfJson(responseBody), - request: parseIfJson(requestBody), - }, - message, - ); + const fields = { + status: statusCode, + method, + url, + error: apiMessage || statusText || undefined, + response: parseIfJson(responseBody), + request: parseIfJson(requestBody), + }; + // a cut body is no longer valid JSON, so consumers are told why `request` is a string + if (requestTruncated) fields.requestTruncated = true; + + log.errorWithFields(fields, message); if (error.response?.data?.message?.includes('could not be matched')) { this.hasUnmatchedTests = true; @@ -713,18 +717,6 @@ function printCreateIssue() { }); } -/** - * Removes Testomatio token from string data - * - * @param {string} data - * @returns {string} - */ -function hideTestomatioToken(data) { - return (typeof data === 'string' ? data : '') - .replace(/"api_key"\s*:\s*"[^"]+"/g, '"api_key": ""') - .replace(/"(tstmt_[^"]+)"/g, '"tstmt_***"'); -} - /** * Stringifies provided data * @@ -732,6 +724,10 @@ function hideTestomatioToken(data) { * @param {{ pretty: boolean }} opts * @returns {string} */ +function stringify(anything, opts = { pretty: false }) { + return typeof anything === 'string' ? anything : JSON.stringify(anything, null, opts.pretty ? 2 : undefined); +} + /** * Turn a JSON string back into an object for structured logs; keeps the string if it is not JSON. * @@ -746,8 +742,4 @@ function parseIfJson(data) { } } -function stringify(anything, opts = { pretty: false }) { - return typeof anything === 'string' ? anything : JSON.stringify(anything, null, opts.pretty ? 2 : undefined); -} - export default TestomatioPipe; diff --git a/src/utils/hide_token.js b/src/utils/hide_token.js new file mode 100644 index 00000000..2e243377 --- /dev/null +++ b/src/utils/hide_token.js @@ -0,0 +1,12 @@ +/** + * Hides the Testomat.io API token in any data which is about to be printed or logged. + * Applied at the logger level, so a raw error object with a request body can't leak the token. + * + * @param {string} data + * @returns {string} The data with every token replaced, empty string if data is not a string. + */ +export function hideTestomatioToken(data) { + if (typeof data !== 'string') return ''; + + return data.replace(/"api_key"\s*:\s*"[^"]+"/g, '"api_key": ""').replace(/tstmt_[\w-]+/g, 'tstmt_***'); +} diff --git a/src/utils/log.js b/src/utils/log.js index b868a81b..11fdfdab 100644 --- a/src/utils/log.js +++ b/src/utils/log.js @@ -1,7 +1,6 @@ import { format as formatArgs, stripVTControlCharacters } from 'util'; import { APP_PREFIX } from '../constants.js'; - -const stripColors = stripVTControlCharacters || (str => str?.replace(/\x1b\[[0-9;]*m/g, '') || ''); +import { hideTestomatioToken } from './hide_token.js'; /** * Log levels for the Testomat.io reporter. @@ -56,8 +55,18 @@ export function isJsonOutput() { * @returns {string} */ function jsonLine(level, args, fields = {}) { - const message = stripColors(formatArgs(...args)).trim(); - return JSON.stringify({ ...fields, level, message }); + const message = stripVTControlCharacters(formatArgs(...args)).trim(); + return hideTestomatioToken(JSON.stringify({ ...fields, level, message })); +} + +/** + * Render the arguments of a log function as text, with the API token hidden. + * Errors and other objects are formatted the way console does it. + * @param {any[]} args - Arguments as passed to the log function + * @returns {string} + */ +function textLine(args) { + return hideTestomatioToken(formatArgs(...args)); } /** @@ -75,7 +84,7 @@ export function info(...args) { fn(jsonLine('info', args)); return; } - fn(APP_PREFIX, ...args); + fn(APP_PREFIX, textLine(args)); } /** @@ -90,7 +99,7 @@ export function warn(...args) { console.warn(jsonLine('warn', args)); return; } - console.warn(APP_PREFIX, ...args); + console.warn(APP_PREFIX, textLine(args)); } /** @@ -105,7 +114,7 @@ export function error(...args) { console.error(jsonLine('error', args)); return; } - console.error(APP_PREFIX, ...args); + console.error(APP_PREFIX, textLine(args)); } /** @@ -121,7 +130,7 @@ export function errorWithFields(fields, ...args) { console.error(jsonLine('error', args, fields)); return; } - console.error(APP_PREFIX, ...args); + console.error(APP_PREFIX, textLine(args)); } /** diff --git a/tests/unit/cli_start_remote_test.js b/tests/unit/cli_start_remote_test.js index 573e24fc..7901497d 100644 --- a/tests/unit/cli_start_remote_test.js +++ b/tests/unit/cli_start_remote_test.js @@ -113,6 +113,7 @@ describe('cli start / run --remote', () => { expect(failure.request.api_key).to.equal(''); expect(stderr).to.not.include('[TESTOMATIO] '); + expect(stderr).to.not.include('faketoken'); }); it('exits non-zero when the run is not created', async () => { @@ -146,6 +147,15 @@ describe('cli start / run --remote', () => { expect(output.runUrl).to.equal(`${TESTOMATIO_URL}/projects/demo/runs/createdrun2`); expect(output.runPublicUrl).to.equal(`${TESTOMATIO_URL}/p/createdrun2`); }); + + it('exits non-zero when the run is not created', async () => { + server.on(replyRun('ignored', 400)); + + const { code, stdout } = await runCli(['run', '--format', 'json']); + + expect(code).to.equal(1); + expect(stdout.trim()).to.equal(''); + }); }); describe('run ', () => { diff --git a/tests/unit/hide_token_test.js b/tests/unit/hide_token_test.js new file mode 100644 index 00000000..8634b331 --- /dev/null +++ b/tests/unit/hide_token_test.js @@ -0,0 +1,30 @@ +import { expect } from 'chai'; +import { hideTestomatioToken } from '../../src/utils/hide_token.js'; + +describe('hideTestomatioToken', () => { + it('hides the api_key value of a JSON body', () => { + expect(hideTestomatioToken('{"api_key":"tstmt_secret123","title":"x"}')).to.equal( + '{"api_key": "","title":"x"}', + ); + }); + + it('hides a token inside an escaped JSON body, as inspected errors print it', () => { + const inspected = 'config: { data: \'{\\"api_key\\":\\"tstmt_secret123\\"}\' }'; + + expect(hideTestomatioToken(inspected)).to.not.include('tstmt_secret123'); + expect(hideTestomatioToken(inspected)).to.include('tstmt_***'); + }); + + it('hides a bare token, wherever it appears', () => { + expect(hideTestomatioToken('key=tstmt_secret-123 used')).to.equal('key=tstmt_*** used'); + }); + + it('keeps data without a token unchanged', () => { + expect(hideTestomatioToken('nothing to hide here')).to.equal('nothing to hide here'); + }); + + it('returns an empty string for non-string data', () => { + // @ts-ignore - the logger may pass anything + expect(hideTestomatioToken({ api_key: 'tstmt_secret123' })).to.equal(''); + }); +}); diff --git a/tests/unit/logger_test.js b/tests/unit/logger_test.js index 8fbb38ed..1ae4d4d3 100644 --- a/tests/unit/logger_test.js +++ b/tests/unit/logger_test.js @@ -293,6 +293,26 @@ describe('Logger Utility', () => { expect(JSON.parse(calls[0][0]).level).to.equal('error'); }); + it('hides the API token of a raw error object', () => { + const requestError = new Error('Request failed'); + // @ts-ignore - mimics an error of the http client + requestError.response = { + config: { data: '{"api_key":"tstmt_secrettoken123","title":"x"}' }, + }; + + error('Error updating status, skipping...', requestError); + + expect(calls[0][0]).to.not.include('tstmt_secrettoken123'); + expect(calls[0][0]).to.include('tstmt_***'); + }); + + it('hides the API token added as a structured field', () => { + errorWithFields({ request: { api_key: 'tstmt_secrettoken123' } }, 'Request failed'); + + expect(calls[0][0]).to.not.include('tstmt_secrettoken123'); + expect(JSON.parse(calls[0][0]).request.api_key).to.equal(''); + }); + it('prints prefixed text when JSON output is disabled', () => { delete process.env.TESTOMATIO_LOG_JSON; errorWithFields({ status: 403 }, 'Request failed'); @@ -300,6 +320,13 @@ describe('Logger Utility', () => { expect(calls[0][0]).to.include('[TESTOMATIO]'); expect(calls[0][1]).to.equal('Request failed'); }); + + it('hides the API token in text output as well', () => { + delete process.env.TESTOMATIO_LOG_JSON; + error('token: tstmt_secrettoken123'); + + expect(calls[0].join(' ')).to.not.include('tstmt_secrettoken123'); + }); }); describe('LOG_LEVELS constant', () => {