diff --git a/docs/cli.md b/docs/cli.md index 95888184..44f49767 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -29,7 +29,7 @@ npx @testomatio/reporter [options] Starts a new test run and returns its ID. This requires an API key to be set in the `TESTOMATIO` environment variable. -With `--format id` (or any `--format`), `start` prints **only the run id to `stdout`** (the banner and progress logs go to `stderr`), so it is safe to capture: `RUN_ID=$(npx @testomatio/reporter start --format id)`. It exits non-zero if the run could not be created. +With `--format id`, `start` prints **only the run id to `stdout`** (the banner and progress logs go to `stderr`), so it is safe to capture: `RUN_ID=$(npx @testomatio/reporter start --format id)`. Use `--format json` to get the run id together with its URLs as a JSON object. It exits non-zero if the run could not be created. **Usage:** @@ -56,7 +56,7 @@ npx @testomatio/reporter start --filter "testomatio:tag-name=smoke" - `--env-file `: Load environment variables from a specific env file. If none specified, it will look for `.env` file. - `--kind `: Specify run type: `automated`, `manual`, `mixed`, or `detect`. Determines how the test run is categorized in Testomat.io. See [Detecting the run kind](#11-detecting-the-run-kind). - `--filter `: Scope the prepared run to the tests matching the filter (same syntax as [`run --filter`](#31-filter-pipes)). The run is created with that test list but **not** executed — useful to prepare a run and launch it later on CI (see [Prepare a run, then launch it on CI](#34-prepare-a-run-then-launch-it-on-ci)). -- `--format `: Print **only the run id** to `stdout` (banner and logs go to `stderr`) so it can be captured: `RUN_ID=$(npx @testomatio/reporter start --format id)`. +- `--format `: Machine-readable output on `stdout` (banner and logs go to `stderr`): `id` prints the bare run id so it can be captured — `RUN_ID=$(npx @testomatio/reporter start --format id)` — and `json` prints `{"runId", "runUrl", "runPublicUrl"}`. - `--warn`: Exit `0` instead of `1` when the filter matches no tests — the warning is still printed. Use in pipelines where an empty scope is a normal outcome (e.g. a PR touching no mapped files). The run is created as **scheduled**, not running: nothing has been executed yet. Testomat.io promotes it to *running* as soon as the first test result is reported, or when you launch it by hand. @@ -128,7 +128,7 @@ Alias for this command – `test`, e.g. `npx @testomatio/reporter test [options] - `-c, --command `: Test runner command (required). - `--filter `: [Filter executed tests](./pipes/testomatio.md#filter-tests) by tag, label, jira, plan. - `--filter-list `: Print the list of tests matching the filter without running them. Useful for inspecting which tests would run, or for piping IDs into another command. See [Coverage Pipe](./pipes/coverage.md#machine-readable-output-with---format) for examples. -- `--format `: Machine-readable output format for `--filter-list`. Supported values: `grep`, `json`, `newline`, `ids`. When set, the CLI banner is suppressed and informational logs go to `stderr` so `stdout` stays clean for piping. +- `--format `: Machine-readable output on `stdout`. With `--filter-list` it selects how the matched test IDs are encoded (`grep`, `json`, `newline`, `ids`); otherwise it prints the run the command creates — `id` for the bare run id, `json` for `{"runId", "runUrl", "runPublicUrl"}`. When set, the CLI banner is suppressed and informational logs go to `stderr` so `stdout` stays clean for piping. - `--env-file `: Load environment variables from a specific env file. - `--kind `: Specify run type: `automated`, `manual`, `mixed`, or `detect`. Determines how the test run is categorized in Testomat.io. `detect` needs a run scoped to a list of tests to resolve from, which `run --filter` does not create — see [Detecting the run kind](#11-detecting-the-run-kind). - `--remote `: Trigger the run on a CI profile configured on the Testomat.io project (e.g. `github`, `gitlab`, `jenkins`) instead of executing tests locally. The CLI creates the run on Testomat.io, asks the backend to dispatch the named CI workflow, and exits. Equivalent to setting [`TESTOMATIO_CI_PROFILE`](./configuration.md#testomatio_ci_profile). @@ -394,7 +394,7 @@ For more details about debug files, see the [Debug Pipe documentation](pipes/deb ## The `--format` flag -`--format` switches a command into **machine-readable mode**: `stdout` carries only the requested data so it can be captured or piped, while the banner and progress logs are routed to `stderr`. Two commands support it — [`run --filter-list`](#3-run) and [`start`](#1-start) — and machine-readable mode behaves the same way for both. +`--format` switches a command into **machine-readable mode**: `stdout` carries only the requested data so it can be captured or piped, while the banner and progress logs are routed to `stderr`. It is supported by [`start`](#1-start) and [`run`](#3-run), and machine-readable mode behaves the same way for both. **What machine-readable mode does (regardless of command or value):** @@ -405,9 +405,27 @@ For more details about debug files, see the [Debug Pipe documentation](pipes/deb This is what makes `$( … )` capture and `|` piping reliable — without `--format`, the banner and `[TESTOMATIO]` logs are interleaved on `stdout`. +**With `--format json` the logs are machine-readable too.** Every reporter message is printed to `stderr` as one JSON object per line instead of prefixed text, so a failing run can be diagnosed without scraping text: + +```bash +npx @testomatio/reporter start --format json 2>errors.log +``` + +```json +{"level":"error","message":"Error creating Testomat.io report (see details above), please check if your API key is valid. Skipping report"} +``` + +A failed request to Testomat.io adds its data as fields — `status`, `method`, `url`, `error`, `response` and `request`. The API token is hidden in every log, whichever field or message it appears in. A request body longer than 1000 characters is cut, which makes `request` a string instead of an object and adds `"requestTruncated": true`: + +```json +{"status":403,"method":"POST","url":"https://app.testomat.io/api/reporter","error":"Project API Token is invalid","level":"error","message":"⚠️ Request to Testomat.io failed: ..."} +``` + +The same output is enabled outside the CLI with [`TESTOMATIO_LOG_JSON=1`](./configuration.md#testomatio_log_json), which the CLI also passes to the test runner it spawns. + ### With `run --filter-list` -Prints the IDs of the tests matching the filter **without running them**. `--format` only takes effect together with `--filter-list`; the value selects the encoding: +Prints the IDs of the tests matching the filter **without running them**. The value selects the encoding: | Value | Output | Example | | --------- | ----------------------------- | -------------------- | @@ -427,30 +445,64 @@ GREP=$(npx @testomatio/reporter run --filter-list "coverage:file=coverage.yml" - npx @testomatio/reporter run --filter-list "coverage:file=coverage.yml" --format json > affected-tests.json ``` +With `--filter-list` no run is created, so `--format json` prints the array of matching test IDs — not the run object described below. + Only the `testomatio:` and `coverage:` filter pipes are supported (see [3.1 Filter pipes](#31-filter-pipes)). The [Coverage Pipe docs](./pipes/coverage.md#machine-readable-output-with---format) cover the formats in more detail. -### With `start` +### With a command that creates a run + +`start` always creates a run, and so does `run` when it is not listing tests with `--filter-list`. For those invocations `--format` selects how the created run is printed to `stdout`: -Prints **only the new run id** to `stdout`, so it can be captured directly: +| Value | Output | +| ------ | ---------------------------------------------------------- | +| `id` | the bare run id (default for any value other than `json`) | +| `json` | a JSON object with the run id and its URLs | ```bash RUN_ID=$(npx @testomatio/reporter start --format id) echo "$RUN_ID" # e.g. a1b2c3d4 + +npx @testomatio/reporter start --format json +# {"runId":"a1b2c3d4","runUrl":"https://app.testomat.io/projects/demo/runs/a1b2c3d4","runPublicUrl":"https://app.testomat.io/p/a1b2c3d4"} +``` + +`runUrl` and `runPublicUrl` are omitted when Testomat.io did not return them. Pick single fields with any JSON tool: + +```bash +RUN_URL=$(npx @testomatio/reporter start --format json | jq -r .runUrl) ``` -`start` emits a single value (the run id), so for `start` the format **value is not significant** — `--format id` is the conventional choice, but any value turns on machine-readable mode. `start` exits non-zero if the run could not be created, so `RUN_ID` is set only on success. It combines with `--kind` and `--filter`: +Every command listed here exits non-zero and prints nothing to `stdout` if the run could not be created, so a captured variable is set only on success. `start` combines with `--kind` and `--filter`: ```bash RUN_ID=$(npx @testomatio/reporter start --kind manual --format id) RUN_ID=$(npx @testomatio/reporter start --filter "testomatio:tag-name=smoke" --format id) ``` +The same output is printed by every `run` invocation that creates a run: + +```bash +# create a run without executing tests +RUN_ID=$(npx @testomatio/reporter run --format id) + +# trigger a run on a CI profile +npx @testomatio/reporter run --remote github --format json + +# executing tests: the runner inherits stdout, so the run data is the FIRST line. +# Capture it all, then slice — piping into `head` closes stdout under the running tests. +OUT=$(npx @testomatio/reporter run "npx playwright test" --format json) +RUN=$(printf '%s\n' "$OUT" | head -1) +``` + ### Quick reference -| Command | Accepted `--format` values | `stdout` contains | Needs | -| ------------------- | -------------------------------- | ---------------------- | -------------------- | -| `run --filter-list` | `ids`, `grep`, `json`, `newline` | the matching test IDs | `--filter-list` | -| `start` | any (use `id`) | the new run id | — | +| Command | Accepted `--format` values | `stdout` contains | +| -------------------------- | -------------------------------- | --------------------------------------- | +| `run --filter-list` | `ids`, `grep`, `json`, `newline` | the matching test IDs, no run created | +| `start` | `id`, `json` | the new run | +| `run` (no command) | `id`, `json` | the new run | +| `run --remote ` | `id`, `json` | the run triggered on CI | +| `run ""` | `id`, `json` | the new run on the first line, then the runner output (capture, do not pipe) | ## Environment Variables diff --git a/docs/configuration.md b/docs/configuration.md index aa968fb7..11e8ef62 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -338,6 +338,22 @@ TESTOMATIO_LOG_LEVEL=ERROR npm test > 📖 See [Log Level Control](./log-level.md) for more details. > 🐛 For detailed debugging, use the `DEBUG` environment variable with the debug package specified. +#### `TESTOMATIO_LOG_JSON` + +Print every `[TESTOMATIO]` message as a JSON object instead of prefixed text, one object per line: + +```json +{"level":"error","message":"Error creating Testomat.io report ..."} +``` + +Failed API requests add their data as fields (`status`, `method`, `url`, `error`, `response`, `request`). The API token is hidden in every message and field, in JSON as well as in text output. + +``` +TESTOMATIO_LOG_JSON=1 npm test +``` + +The CLI sets this automatically for [`--format json`](./cli.md#the---format-flag), so both the output and the logs of a run are machine-readable. + #### `TESTOMATIO_UPDATE_CODE` Sends the `code` of your tests to Testomat.io on each run. (If not enabled (default) assumes the code is pushed using [check-tests](https://github.com/testomatio/check-tests#cli)). diff --git a/src/bin/cli.js b/src/bin/cli.js index 5f0acd2d..3c1a8f18 100755 --- a/src/bin/cli.js +++ b/src/bin/cli.js @@ -16,7 +16,7 @@ import { filesize as prettyBytes } from 'filesize'; import dotenv from 'dotenv'; import Replay from '../replay.js'; import { log } from '../utils/log.js'; -import { formatFilterListIds } from '../utils/pipe_utils.js'; +import { formatFilterListIds, formatRunOutput } from '../utils/pipe_utils.js'; import fs from 'fs'; import path from 'path'; @@ -43,6 +43,8 @@ program if (subOpts.filterList || subOpts.format) { process.env.TESTOMATIO_LOG_STDERR = '1'; process.env.TESTOMATIO_LOG_LEVEL ||= 'WARN'; + // with --format json the logs are machine-readable too: one JSON object per line on stderr + if (subOpts.format === 'json') process.env.TESTOMATIO_LOG_JSON = '1'; } else { console.log(pc.cyan(pc.bold(` 🤩 Testomat.io Reporter v${version}`))); } @@ -53,7 +55,7 @@ program .description('Start a new run and return its ID') .option('--kind ', 'Specify run type: automated, manual, mixed, or detect') .option('--filter ', 'Scope the prepared run to tests matching the filter (no execution)') - .option('--format ', 'Machine-readable output: print only the run id to stdout (e.g. --format id)') + .option('--format ', 'Machine-readable output: the run id (--format id) or run details (--format json)') .option('--warn', 'Exit 0 instead of 1 when the filter matches no tests (warn only)') .action(async opts => { cleanLatestRunId(); @@ -82,7 +84,7 @@ program await client.createRun(createRunParams); - const runId = client.pipeStore.runId || process.env.runId; + const runId = client.pipeStore.runId; if (!runId) { log.error(pc.red('Failed to create run on Testomat.io.')); process.exit(1); @@ -93,8 +95,8 @@ program const plannedTests = (client.pipeStore.preparedTestIds || []).map(id => ({ test_id: id, title: id })); await client.updateRunStatus('pending', { tests: plannedTests }); - // stdout carries ONLY the run id so it can be captured: RUN_ID=$(reporter start) - console.log(runId); + // stdout carries ONLY the run data so it can be captured: RUN_ID=$(reporter start) + console.log(formatRunOutput(client.pipeStore, opts.format)); process.exit(0); }); @@ -129,7 +131,10 @@ program .argument('[command]', 'Test runner command') .option('--filter ', 'Additional execution filter') .option('--filter-list ', 'Get a list of all tests by filter before running') - .option('--format ', 'Machine-readable output format for --filter-list (grep, json, newline, ids)') + .option( + '--format ', + 'Machine-readable output: test ids for --filter-list (grep, json, newline, ids), or the run created (id, json)', + ) .option('--kind ', 'Specify run type: automated, manual, mixed, or detect') .option('--remote ', 'Trigger run on the named Testomat.io CI profile instead of executing locally') .option( @@ -230,6 +235,8 @@ program log.info(`🚀 CI build triggered on profile ${pc.cyan(opts.remote)}`); log.info(`📊 Report URL: ${pc.magenta(client.pipeStore.runUrl)}`); + const remoteOutput = formatRunOutput(client.pipeStore, opts.format); + if (opts.format && remoteOutput) console.log(remoteOutput); return process.exit(0); } @@ -245,13 +252,20 @@ program if (apiKey) { await client.createRun(createRunParams); - const runId = process.env.TESTOMATIO_RUN || process.env.runId; + + const runId = client.pipeStore.runId; + if (!runId) { + log.error(pc.red('Failed to create run on Testomat.io.')); + process.exit(1); + } + if (client.pipeStore.runUrl) log.info( `📊 Report URL: ${pc.magenta(client.pipeStore.runUrl)}`); if (opts.kind !== 'manual') { log.info( `No command passed, so you need to run tests yourself:`); log.info( `TESTOMATIO_RUN=${runId} `); } + if (opts.format) console.log(formatRunOutput(client.pipeStore, opts.format)); } else { log.info( '⚠️ No API key provided. Cannot create run without TESTOMATIO key.'); process.exit(1); @@ -288,7 +302,11 @@ program } if (apiKey) { - await client.createRun(createRunParams).then(runTests); + await client.createRun(createRunParams); + // the runner inherits stdout, so the run data is printed first, on its own line + const createdOutput = formatRunOutput(client.pipeStore, opts.format); + if (opts.format && createdOutput) console.log(createdOutput); + await runTests(); } else { await runTests(); } diff --git a/src/client.js b/src/client.js index f752f782..48a10f92 100644 --- a/src/client.js +++ b/src/client.js @@ -187,7 +187,7 @@ class Client { } } catch (err) { - console.error(APP_PREFIX, 'Error in uploadStepArtifacts for testRid', testRid, ':', err); + log.error('Error in uploadStepArtifacts for testRid', testRid, ':', err.message || err); throw err; } } @@ -225,7 +225,7 @@ class Client { try { await this.uploadStepArtifacts(steps, rid); } catch (err) { - console.log(APP_PREFIX, 'Failed to upload step artifacts:', err); + log.error('Failed to upload step artifacts:', err.message || err); } const uploadedFiles = []; @@ -423,7 +423,7 @@ class Client { const pathPadding = Math.max(...failedUploads.map(upload => upload.relativePath.length)) + 1; failedUploads.forEach(upload => { - console.log( + log.info( ` ${pc.gray('|')} 🔴 ${upload.relativePath.padEnd(pathPadding)} ${pc.gray( `| ${upload.sizePretty.padStart(filesizeStrMaxLength)} |`, )}`, @@ -439,7 +439,7 @@ class Client { })); const pathPadding = Math.max(...skippedUploads.map(upload => upload.relativePath.length)) + 1; skippedUploads.forEach(upload => { - console.log( + log.info( ` ${pc.gray('|')} 🟡 ${upload.relativePath.padEnd(pathPadding)} ${pc.gray( `| ${upload.sizePretty.padStart(filesizeStrMaxLength)} |`, )}`, diff --git a/src/pipe/csv.js b/src/pipe/csv.js index 4d812c79..d9b19e90 100644 --- a/src/pipe/csv.js +++ b/src/pipe/csv.js @@ -6,6 +6,7 @@ import pc from 'picocolors'; import merge from 'lodash.merge'; import { isSameTest, getCurrentDateTime, ansiRegExp } from '../utils/utils.js'; import { CSV_HEADERS } from '../constants.js'; +import { log } from '../utils/log.js'; const debug = createDebugMessages('@testomatio/reporter:pipe:csv'); /** @@ -76,11 +77,11 @@ class CsvPipe { this.checkExportDir(); if (!this.outputFile) { - console.log(pc.yellow(`⚠️ CSV file is not set, ignoring`)); + log.warn(pc.yellow(`⚠️ CSV file is not set, ignoring`)); return; } - console.log(pc.yellow(`⏳ The test results will be added to the csv. It will take some time...`)); + log.info(pc.yellow(`⏳ The test results will be added to the csv. It will take some time...`)); try { // Create csv writer object @@ -91,7 +92,7 @@ class CsvPipe { // Save csv file based on the current data return await writer.writeRecords(data); } catch (e) { - console.log('Unknown csv error: ', e); + log.error('Unknown csv error: ', e); } } @@ -135,7 +136,7 @@ class CsvPipe { // Save results based on the default headers if (this.isEnabled) { await this.saveToCsv(this.results, CSV_HEADERS); - console.log(pc.green(`🗃️ Recording completed! You can check the result in file = ${this.outputFile}`)); + log.info(pc.green(`🗃️ Recording completed! You can check the result in file = ${this.outputFile}`)); } } diff --git a/src/pipe/html.js b/src/pipe/html.js index 583b25d5..fdd9e4e1 100644 --- a/src/pipe/html.js +++ b/src/pipe/html.js @@ -8,6 +8,7 @@ import { marked } from 'marked'; import fileUrl from 'file-url'; import { fileSystem, isSameTest, ansiRegExp, formatStep, transformEnvVarToBoolean } from '../utils/utils.js'; import { HTML_REPORT } from '../constants.js'; +import { log } from '../utils/log.js'; import { fileURLToPath } from 'node:url'; const debug = createDebugMessages('@testomatio/reporter:pipe:html'); @@ -144,14 +145,14 @@ class HtmlPipe { debug('HTML tests data:', tests); if (!outputPath) { - console.log(pc.yellow(`🚨 HTML export path is not set, ignoring...`)); + log.warn(pc.yellow(`🚨 HTML export path is not set, ignoring...`)); return; } - console.log(pc.yellow(`⏳ The test results will be added to the HTML report. It will take some time...`)); + log.info(pc.yellow(`⏳ The test results will be added to the HTML report. It will take some time...`)); if (msg) { - console.log(pc.blue(msg)); + log.info(pc.blue(msg)); } const aggregatedTests = aggregateTestRetries(tests); @@ -295,9 +296,9 @@ class HtmlPipe { debug('HTML tests data:', fileUrlPath); - console.log(pc.green(`📊 The HTML report was successfully generated. Full filepath: ${fileUrlPath}`)); + log.info(pc.green(`📊 The HTML report was successfully generated. Full filepath: ${fileUrlPath}`)); } else { - console.log(pc.red(`🚨 Failed to generate the HTML report.`)); + log.error(pc.red(`🚨 Failed to generate the HTML report.`)); } } @@ -309,7 +310,7 @@ class HtmlPipe { */ #generateHTMLReport(data, templatePath = '') { if (!templatePath) { - console.log(pc.red(`🚨 HTML template not found. Report generation is impossible!`)); + log.error(pc.red(`🚨 HTML template not found. Report generation is impossible!`)); return; } @@ -320,8 +321,8 @@ class HtmlPipe { return template(data); } catch (e) { - console.log(pc.red('❌ Oops! An unknown error occurred when generating an HTML report')); - console.log(pc.red(e)); + log.error(pc.red('❌ Oops! An unknown error occurred when generating an HTML report')); + log.error(pc.red(e)); } } diff --git a/src/pipe/testomatio.js b/src/pipe/testomatio.js index 50cfb664..e3f16d89 100644 --- a/src/pipe/testomatio.js +++ b/src/pipe/testomatio.js @@ -18,6 +18,7 @@ import { getGitCommitSha, } from '../utils/utils.js'; import { parseFilterParams, generateFilterRequestParams, setS3Credentials } from '../utils/pipe_utils.js'; +import { hideTestomatioToken } from '../utils/hide_token.js'; import { config } from '../config.js'; import { log } from '../utils/log.js'; @@ -380,16 +381,15 @@ class TestomatioPipe { process.env.runId = this.runId; debug('Run created', this.runId); } catch (err) { - if (!this.apiKey) console.error('Testomat.io API key is not set'); + if (!this.apiKey) log.error('Testomat.io API key is not set'); const errorText = err.response?.data?.message || err.message; debug('Error creating run', err); - console.log(APP_PREFIX, errorText || err); + log.error(errorText || err); if (err.response?.status === 403) this.#disablePipe(); this.#logFailedResponse(err); - console.error( - APP_PREFIX, + log.error( 'Error creating Testomat.io report (see details above), please check if your API key is valid. Skipping report', ); printCreateIssue(); @@ -409,7 +409,7 @@ class TestomatioPipe { this.reportingCanceledDueToReqFailures = true; let errorMessage = `⚠️ ${process.env.TESTOMATIO_MAX_REQUEST_FAILURES}`; errorMessage += ' requests were failed, reporting to Testomat aborted.'; - console.warn(`${APP_PREFIX} ${pc.yellow(errorMessage)}`); + log.warn(pc.yellow(errorMessage)); } return cancelReporting; } @@ -557,7 +557,7 @@ class TestomatioPipe { const errorMessage = pc.red( `⚠️ Due to request failures, ${this.notReportedTestsCount} test(s) were not reported to Testomat.io`, ); - console.warn(`${APP_PREFIX} ${errorMessage}`); + log.warn(errorMessage); } const { status } = params; @@ -621,7 +621,7 @@ class TestomatioPipe { ); } } catch (err) { - console.log(APP_PREFIX, 'Error updating status, skipping...', err); + log.error('Error updating status, skipping...', err.message || err); this.#logFailedResponse(err); printCreateIssue(); } @@ -665,18 +665,32 @@ class TestomatioPipe { message += `\t${pc.bold('response: ')}${pc.gray(responseBody)}\n`; - const requestBody = hideTestomatioToken(stringify(error.response?.config?.data)); + let requestBody = hideTestomatioToken(stringify(error.response?.config?.data)); + let requestTruncated = false; if (process.env.DEBUG || process.env.TESTOMATIO_DEBUG || requestBody.length < 1000) { // full body message += `\t${pc.bold('request: ')}${pc.gray(requestBody)}\n`; } else { // cut body - const requestBodyCut = requestBody.slice(0, 1000); - message += `\t${pc.bold('request: ')}${pc.gray(`${requestBodyCut}...`)}\n`; + requestTruncated = true; + requestBody = `${requestBody.slice(0, 1000)}...`; + message += `\t${pc.bold('request: ')}${pc.gray(requestBody)}\n`; message += '\trequest body is cut, run with TESTOMATIO_DEBUG=1 to see full body\n'; } - console.log(message); + // the JSON line is built from the same values as the text message, with the token already hidden + const fields = { + status: statusCode, + method, + url, + error: apiMessage || statusText || undefined, + response: parseIfJson(responseBody), + request: parseIfJson(requestBody), + }; + // a cut body is no longer valid JSON, so consumers are told why `request` is a string + if (requestTruncated) fields.requestTruncated = true; + + log.errorWithFields(fields, message); if (error.response?.data?.message?.includes('could not be matched')) { this.hasUnmatchedTests = true; @@ -693,8 +707,7 @@ function printCreateIssue() { if (registeredErrorHints) return; registeredErrorHints = true; process.on('exit', () => { - console.log( - APP_PREFIX, + log.error( 'There was an error reporting to Testomat.io.\n', pc.yellow( 'If you think this is a bug please create an issue: https://github.com/testomatio/reporter/issues/new.', @@ -704,18 +717,6 @@ function printCreateIssue() { }); } -/** - * Removes Testomatio token from string data - * - * @param {string} data - * @returns {string} - */ -function hideTestomatioToken(data) { - return (typeof data === 'string' ? data : '') - .replace(/"api_key"\s*:\s*"[^"]+"/g, '"api_key": ""') - .replace(/"(tstmt_[^"]+)"/g, '"tstmt_***"'); -} - /** * Stringifies provided data * @@ -727,4 +728,18 @@ function stringify(anything, opts = { pretty: false }) { return typeof anything === 'string' ? anything : JSON.stringify(anything, null, opts.pretty ? 2 : undefined); } +/** + * Turn a JSON string back into an object for structured logs; keeps the string if it is not JSON. + * + * @param {string} data + * @returns {any} + */ +function parseIfJson(data) { + try { + return JSON.parse(data); + } catch { + return data; + } +} + export default TestomatioPipe; diff --git a/src/utils/hide_token.js b/src/utils/hide_token.js new file mode 100644 index 00000000..2e243377 --- /dev/null +++ b/src/utils/hide_token.js @@ -0,0 +1,12 @@ +/** + * Hides the Testomat.io API token in any data which is about to be printed or logged. + * Applied at the logger level, so a raw error object with a request body can't leak the token. + * + * @param {string} data + * @returns {string} The data with every token replaced, empty string if data is not a string. + */ +export function hideTestomatioToken(data) { + if (typeof data !== 'string') return ''; + + return data.replace(/"api_key"\s*:\s*"[^"]+"/g, '"api_key": ""').replace(/tstmt_[\w-]+/g, 'tstmt_***'); +} diff --git a/src/utils/log.js b/src/utils/log.js index 5db5a1fa..11fdfdab 100644 --- a/src/utils/log.js +++ b/src/utils/log.js @@ -1,4 +1,6 @@ +import { format as formatArgs, stripVTControlCharacters } from 'util'; import { APP_PREFIX } from '../constants.js'; +import { hideTestomatioToken } from './hide_token.js'; /** * Log levels for the Testomat.io reporter. @@ -35,16 +37,54 @@ export function shouldLog(messageLevel) { return messageLevel <= getLogLevel() || !!process.env.TESTOMATIO_DEBUG; } +/** + * Check if logs should be printed as JSON lines instead of [TESTOMATIO] prefixed text. + * Enabled by the CLI for `--format json` so the whole output is machine-readable. + * @returns {boolean} + */ +export function isJsonOutput() { + return process.env.TESTOMATIO_LOG_JSON === '1'; +} + +/** + * Render a log message as a single JSON line, e.g. `{"level":"error","message":"..."}`. + * Colors are stripped, so the message stays readable after parsing. + * @param {string} level - Log level name + * @param {any[]} args - Arguments as passed to the log function + * @param {Object} [fields] - Extra fields to add to the JSON object + * @returns {string} + */ +function jsonLine(level, args, fields = {}) { + const message = stripVTControlCharacters(formatArgs(...args)).trim(); + return hideTestomatioToken(JSON.stringify({ ...fields, level, message })); +} + +/** + * Render the arguments of a log function as text, with the API token hidden. + * Errors and other objects are formatted the way console does it. + * @param {any[]} args - Arguments as passed to the log function + * @returns {string} + */ +function textLine(args) { + return hideTestomatioToken(formatArgs(...args)); +} + /** * Log an info message with [TESTOMATIO] prefix. * Only logs when TESTOMATIO_LOG_LEVEL is INFO. * @param {...any} args - Arguments to log */ export function info(...args) { - if (shouldLog(LOG_LEVELS.INFO)) { - const fn = process.env.TESTOMATIO_LOG_STDERR === '1' ? console.error : console.log; - fn(APP_PREFIX, ...args); + if (!shouldLog(LOG_LEVELS.INFO)) return; + + let fn = console.log; + if (process.env.TESTOMATIO_LOG_STDERR === '1') fn = console.error; + + if (isJsonOutput()) { + fn(jsonLine('info', args)); + return; } + fn(APP_PREFIX, textLine(args)); } /** @@ -53,9 +93,13 @@ export function info(...args) { * @param {...any} args - Arguments to log */ export function warn(...args) { - if (shouldLog(LOG_LEVELS.WARN)) { - console.warn(APP_PREFIX, ...args); + if (!shouldLog(LOG_LEVELS.WARN)) return; + + if (isJsonOutput()) { + console.warn(jsonLine('warn', args)); + return; } + console.warn(APP_PREFIX, textLine(args)); } /** @@ -64,9 +108,29 @@ export function warn(...args) { * @param {...any} args - Arguments to log */ export function error(...args) { - if (shouldLog(LOG_LEVELS.ERROR)) { - console.error(APP_PREFIX, ...args); + if (!shouldLog(LOG_LEVELS.ERROR)) return; + + if (isJsonOutput()) { + console.error(jsonLine('error', args)); + return; + } + console.error(APP_PREFIX, textLine(args)); +} + +/** + * Log an error which carries structured data, e.g. a failed API request. + * The fields are added to the JSON line; in text mode only the message is printed. + * @param {Object} fields - Extra fields to add to the JSON object + * @param {...any} args - Arguments to log as text + */ +export function errorWithFields(fields, ...args) { + if (!shouldLog(LOG_LEVELS.ERROR)) return; + + if (isJsonOutput()) { + console.error(jsonLine('error', args, fields)); + return; } + console.error(APP_PREFIX, textLine(args)); } /** @@ -82,6 +146,8 @@ export const log = { info, warn, error, + errorWithFields, + isJsonOutput, getLogLevel, shouldLog, LOG_LEVELS, diff --git a/src/utils/pipe_utils.js b/src/utils/pipe_utils.js index 76411a15..f6f45278 100644 --- a/src/utils/pipe_utils.js +++ b/src/utils/pipe_utils.js @@ -304,6 +304,27 @@ function plannedTestsLabel(tests, testsCount) { return `**${knownTestsCount}** tests and **${suitesCount}** suites planned`; } +/** + * Format the created run for machine-readable output of `start` and `run`. + * `json` prints an object with the run details, any other format prints the bare run id. + * + * @param {{runId?: string, runUrl?: string, runPublicUrl?: string}} store - Pipe store of the client. + * @param {string} [format] - Value of the CLI `--format` option. + * @returns {string} Empty string if there is no run id. + */ +function formatRunOutput(store, format) { + const runId = store?.runId; + if (!runId) return ''; + + if (format !== 'json') return runId; + + const output = { runId }; + if (store.runUrl) output.runUrl = store.runUrl; + if (store.runPublicUrl) output.runPublicUrl = store.runPublicUrl; + + return JSON.stringify(output); +} + export { updateFilterType, parseFilterParams, @@ -317,6 +338,7 @@ export { plannedTestsLabel, parsePipeOptions, formatFilterListIds, + formatRunOutput, getObjectSize, splitTestsIntoChunks, }; diff --git a/tests/unit/cli_start_remote_test.js b/tests/unit/cli_start_remote_test.js index edd90999..7901497d 100644 --- a/tests/unit/cli_start_remote_test.js +++ b/tests/unit/cli_start_remote_test.js @@ -69,6 +69,53 @@ describe('cli start / run --remote', () => { expect(stdout.trim()).to.equal('startrun123'); }); + it('with --format json prints the run details as JSON on stdout', async () => { + server.on(replyRun('startrun456')); + + const { code, stdout } = await runCli(['start', '--format', 'json']); + + expect(code).to.equal(0); + const output = JSON.parse(stdout.trim()); + expect(output.runId).to.equal('startrun456'); + expect(output.runUrl).to.equal(`${TESTOMATIO_URL}/projects/demo/runs/startrun456`); + expect(output.runPublicUrl).to.equal(`${TESTOMATIO_URL}/p/startrun456`); + }); + + it('with --format json reports failures as JSON lines on stderr, keeping stdout empty', async () => { + server.on({ + method: 'POST', + path: '/api/reporter', + reply: { + status: 403, + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ message: 'Project API Token is invalid' }), + }, + }); + + const { code, stdout, stderr } = await runCli(['start', '--format', 'json']); + + expect(code).to.equal(1); + expect(stdout.trim()).to.equal(''); + + // the failed request is reported with its data, not as a formatted text blob + const failure = stderr + .split('\n') + .filter(line => line.startsWith('{')) + .map(line => JSON.parse(line)) + .find(entry => entry.status === 403); + + expect(failure).to.exist; + expect(failure.level).to.equal('error'); + expect(failure.method).to.equal('POST'); + expect(failure.url).to.include('/api/reporter'); + expect(failure.error).to.equal('Project API Token is invalid'); + expect(failure.response).to.deep.equal({ message: 'Project API Token is invalid' }); + expect(failure.request.api_key).to.equal(''); + + expect(stderr).to.not.include('[TESTOMATIO] '); + expect(stderr).to.not.include('faketoken'); + }); + it('exits non-zero when the run is not created', async () => { server.on(replyRun('ignored', 500)); @@ -79,6 +126,56 @@ describe('cli start / run --remote', () => { }); }); + describe('run (no command)', () => { + it('with --format id prints ONLY the run id on stdout', async () => { + server.on(replyRun('createdrun1')); + + const { code, stdout } = await runCli(['run', '--format', 'id']); + + expect(code).to.equal(0); + expect(stdout.trim()).to.equal('createdrun1'); + }); + + it('with --format json prints the run details as JSON on stdout', async () => { + server.on(replyRun('createdrun2')); + + const { code, stdout } = await runCli(['run', '--format', 'json']); + + expect(code).to.equal(0); + const output = JSON.parse(stdout.trim()); + expect(output.runId).to.equal('createdrun2'); + expect(output.runUrl).to.equal(`${TESTOMATIO_URL}/projects/demo/runs/createdrun2`); + expect(output.runPublicUrl).to.equal(`${TESTOMATIO_URL}/p/createdrun2`); + }); + + it('exits non-zero when the run is not created', async () => { + server.on(replyRun('ignored', 400)); + + const { code, stdout } = await runCli(['run', '--format', 'json']); + + expect(code).to.equal(1); + expect(stdout.trim()).to.equal(''); + }); + }); + + describe('run ', () => { + it('with --format json prints the run as the first stdout line, before the runner output', async () => { + server.on(replyRun('execrun1')); + server.on({ + method: 'PUT', + path: '/api/reporter/execrun1', + reply: { status: 200, headers: { 'content-type': 'application/json' }, body: JSON.stringify({}) }, + }); + + const { code, stdout } = await runCli(['run', 'echo hello', '--format', 'json']); + + expect(code).to.equal(0); + const [firstLine, ...rest] = stdout.trim().split('\n'); + expect(JSON.parse(firstLine).runId).to.equal('execrun1'); + expect(rest.join('\n')).to.include('hello'); + }); + }); + describe('run --remote', () => { it('reports success and exits 0 when CI launch succeeds', async () => { server.on(replyRun('ciRun1')); @@ -89,6 +186,17 @@ describe('cli start / run --remote', () => { expect(stdout).to.include('CI build triggered'); }); + it('with --format json prints the triggered run as JSON on stdout', async () => { + server.on(replyRun('ciRun3')); + + const { code, stdout } = await runCli(['run', '--remote', 'github', '--format', 'json']); + + expect(code).to.equal(0); + const output = JSON.parse(stdout.trim()); + expect(output.runId).to.equal('ciRun3'); + expect(output.runUrl).to.equal(`${TESTOMATIO_URL}/projects/demo/runs/ciRun3`); + }); + it('exits non-zero and does NOT report success when CI launch fails', async () => { server.on(replyRun('ciRun2', 400)); diff --git a/tests/unit/hide_token_test.js b/tests/unit/hide_token_test.js new file mode 100644 index 00000000..8634b331 --- /dev/null +++ b/tests/unit/hide_token_test.js @@ -0,0 +1,30 @@ +import { expect } from 'chai'; +import { hideTestomatioToken } from '../../src/utils/hide_token.js'; + +describe('hideTestomatioToken', () => { + it('hides the api_key value of a JSON body', () => { + expect(hideTestomatioToken('{"api_key":"tstmt_secret123","title":"x"}')).to.equal( + '{"api_key": "","title":"x"}', + ); + }); + + it('hides a token inside an escaped JSON body, as inspected errors print it', () => { + const inspected = 'config: { data: \'{\\"api_key\\":\\"tstmt_secret123\\"}\' }'; + + expect(hideTestomatioToken(inspected)).to.not.include('tstmt_secret123'); + expect(hideTestomatioToken(inspected)).to.include('tstmt_***'); + }); + + it('hides a bare token, wherever it appears', () => { + expect(hideTestomatioToken('key=tstmt_secret-123 used')).to.equal('key=tstmt_*** used'); + }); + + it('keeps data without a token unchanged', () => { + expect(hideTestomatioToken('nothing to hide here')).to.equal('nothing to hide here'); + }); + + it('returns an empty string for non-string data', () => { + // @ts-ignore - the logger may pass anything + expect(hideTestomatioToken({ api_key: 'tstmt_secret123' })).to.equal(''); + }); +}); diff --git a/tests/unit/logger_test.js b/tests/unit/logger_test.js index 8200c68a..1ae4d4d3 100644 --- a/tests/unit/logger_test.js +++ b/tests/unit/logger_test.js @@ -1,5 +1,6 @@ import { expect } from 'chai'; -import { log, info, warn, error, LOG_LEVELS } from '../../src/utils/log.js'; +import pc from 'picocolors'; +import { log, info, warn, error, errorWithFields, LOG_LEVELS } from '../../src/utils/log.js'; describe('Logger Utility', () => { let originalEnv; @@ -230,6 +231,104 @@ describe('Logger Utility', () => { }); }); + describe('JSON output (TESTOMATIO_LOG_JSON=1)', () => { + let calls; + let originalError; + let originalWarn; + + beforeEach(() => { + process.env.TESTOMATIO_LOG_JSON = '1'; + calls = []; + originalError = console.error; + originalWarn = console.warn; + console.error = (...args) => calls.push(args); + console.warn = (...args) => calls.push(args); + }); + + afterEach(() => { + console.error = originalError; + console.warn = originalWarn; + delete process.env.TESTOMATIO_LOG_JSON; + }); + + it('prints errors as a single JSON object with level and message', () => { + error('something went wrong'); + + expect(calls.length).to.equal(1); + expect(JSON.parse(calls[0][0])).to.deep.equal({ level: 'error', message: 'something went wrong' }); + }); + + it('prints warnings as JSON and strips colors from the message', () => { + warn(pc.yellow('be careful')); + + expect(JSON.parse(calls[0][0])).to.deep.equal({ level: 'warn', message: 'be careful' }); + }); + + it('keeps the [TESTOMATIO] prefix out of the JSON output', () => { + error('plain message'); + + expect(calls[0][0]).to.not.include('[TESTOMATIO]'); + }); + + it('formats multiple arguments into one message, as the text logger does', () => { + error('failed:', { status: 403 }); + + expect(JSON.parse(calls[0][0]).message).to.equal('failed: { status: 403 }'); + }); + + it('adds structured fields of errorWithFields to the JSON object', () => { + errorWithFields({ status: 403, url: 'https://app.testomat.io/api/reporter' }, 'Request failed'); + + expect(JSON.parse(calls[0][0])).to.deep.equal({ + status: 403, + url: 'https://app.testomat.io/api/reporter', + level: 'error', + message: 'Request failed', + }); + }); + + it('never lets a field override the log level', () => { + errorWithFields({ level: 'info' }, 'still an error'); + + expect(JSON.parse(calls[0][0]).level).to.equal('error'); + }); + + it('hides the API token of a raw error object', () => { + const requestError = new Error('Request failed'); + // @ts-ignore - mimics an error of the http client + requestError.response = { + config: { data: '{"api_key":"tstmt_secrettoken123","title":"x"}' }, + }; + + error('Error updating status, skipping...', requestError); + + expect(calls[0][0]).to.not.include('tstmt_secrettoken123'); + expect(calls[0][0]).to.include('tstmt_***'); + }); + + it('hides the API token added as a structured field', () => { + errorWithFields({ request: { api_key: 'tstmt_secrettoken123' } }, 'Request failed'); + + expect(calls[0][0]).to.not.include('tstmt_secrettoken123'); + expect(JSON.parse(calls[0][0]).request.api_key).to.equal(''); + }); + + it('prints prefixed text when JSON output is disabled', () => { + delete process.env.TESTOMATIO_LOG_JSON; + errorWithFields({ status: 403 }, 'Request failed'); + + expect(calls[0][0]).to.include('[TESTOMATIO]'); + expect(calls[0][1]).to.equal('Request failed'); + }); + + it('hides the API token in text output as well', () => { + delete process.env.TESTOMATIO_LOG_JSON; + error('token: tstmt_secrettoken123'); + + expect(calls[0].join(' ')).to.not.include('tstmt_secrettoken123'); + }); + }); + describe('LOG_LEVELS constant', () => { it('should have all expected log levels', () => { expect(LOG_LEVELS).to.have.property('ERROR', 0); diff --git a/tests/unit/pipe_utils_test.js b/tests/unit/pipe_utils_test.js index 301be613..1890469e 100644 --- a/tests/unit/pipe_utils_test.js +++ b/tests/unit/pipe_utils_test.js @@ -1,5 +1,10 @@ import { expect } from 'chai'; -import { formatFilterListIds, getObjectSize, splitTestsIntoChunks } from '../../src/utils/pipe_utils.js'; +import { + formatFilterListIds, + formatRunOutput, + getObjectSize, + splitTestsIntoChunks, +} from '../../src/utils/pipe_utils.js'; describe('formatFilterListIds', () => { const ids = ['t1234abcd', 't5678efgh', 'tabcdef01']; @@ -97,6 +102,38 @@ describe('formatFilterListIds', () => { }); }); +describe('formatRunOutput', () => { + const store = { + runId: 'run123', + runUrl: 'https://app.testomat.io/projects/demo/runs/run123', + runPublicUrl: 'https://app.testomat.io/p/run123', + }; + + it('returns empty string when there is no run id', () => { + expect(formatRunOutput({}, 'json')).to.equal(''); + expect(formatRunOutput(undefined, 'id')).to.equal(''); + }); + + it('prints the bare run id for non-json formats', () => { + expect(formatRunOutput(store, 'id')).to.equal('run123'); + expect(formatRunOutput(store, undefined)).to.equal('run123'); + }); + + it('prints run details as JSON for the json format', () => { + const output = JSON.parse(formatRunOutput(store, 'json')); + expect(output).to.deep.equal({ + runId: 'run123', + runUrl: 'https://app.testomat.io/projects/demo/runs/run123', + runPublicUrl: 'https://app.testomat.io/p/run123', + }); + }); + + it('omits urls the pipe did not provide', () => { + const output = JSON.parse(formatRunOutput({ runId: 'run123' }, 'json')); + expect(output).to.deep.equal({ runId: 'run123' }); + }); +}); + describe('splitTestsIntoChunks', () => { // build a test whose serialized size is roughly `bytes` const makeTest = (i, bytes = 100) => ({ diff --git a/tests/unit/pipes/testomatio_pipe_test.js b/tests/unit/pipes/testomatio_pipe_test.js index c7855bfa..41aee169 100644 --- a/tests/unit/pipes/testomatio_pipe_test.js +++ b/tests/unit/pipes/testomatio_pipe_test.js @@ -1297,6 +1297,8 @@ describe('TestomatioPipe', () => { let pipe; let consoleLogOutput; let originalRequest; + let originalLog; + let originalError; beforeEach(() => { process.env.TESTOMATIO_URL = TESTOMATIO_URL; @@ -1311,21 +1313,23 @@ describe('TestomatioPipe', () => { pipe.runId = 'test-run-123'; consoleLogOutput = []; - const originalLog = console.log; + originalLog = console.log; + originalError = console.error; + // errors are logged to stderr, so both streams are captured console.log = (...args) => { consoleLogOutput.push(args.join(' ')); }; + console.error = (...args) => { + consoleLogOutput.push(args.join(' ')); + }; // Store original request method to restore later originalRequest = pipe.client.request; - - return () => { - console.log = originalLog; - }; }); afterEach(() => { - console.log = global.console.log; + console.log = originalLog; + console.error = originalError; delete process.env.TESTOMATIO_URL; // Restore original request method pipe.client.request = originalRequest;