diff --git a/.changeset/streamlined-onboarding-report.md b/.changeset/streamlined-onboarding-report.md index f9f5b32e..fe08728e 100644 --- a/.changeset/streamlined-onboarding-report.md +++ b/.changeset/streamlined-onboarding-report.md @@ -3,3 +3,5 @@ --- `taskless detect --json` now reports ast-grep and Vale configurations as linters (Taskless's own configuration under `.taskless/` excluded), the command-line tools on the host's `PATH` (`gh`, `git`, `jq`, and now `glab` for GitLab repositories) under `hostTools`, and an `agent` section of commands an agent may choose to run to check whether a source is usable, such as `gh auth status`. Detection still runs nothing and stays offline. `taskless info` lists `glab` among its tools too. + +New `taskless report` publishes an unlisted Taskless report to tskl.es: a page showing the rules in this repository, what they were drawn from, and what they are projected to catch. With no flags it prints a draft of what the CLI knows and the facts an agent still has to supply; `--from ` checks there are at least three working Taskless rules, withholds any fixture that looks like it holds a secret or contact detail, validates against the tskl.es schema, publishes, and records the report in `.taskless/reports/` (gitignored, since each record holds the URL that deletes it); `--delete ` takes a report down. The new `report` agent topic walks an agent through it. Migration 11 creates `.taskless/reports/`. diff --git a/.taskless/.gitignore b/.taskless/.gitignore index 0fe05ba6..779df1e5 100644 --- a/.taskless/.gitignore +++ b/.taskless/.gitignore @@ -3,3 +3,4 @@ /.vale.ini /.sgconfig.yml /.tmp-* +/reports/ diff --git a/.taskless/taskless.json b/.taskless/taskless.json index 91a2aa71..b2e9d0f8 100644 --- a/.taskless/taskless.json +++ b/.taskless/taskless.json @@ -1,5 +1,5 @@ { - "version": 10, + "version": 11, "install": { "targets": { ".taskless": { diff --git a/openspec/changes/streamlined-onboarding-report/design.md b/openspec/changes/streamlined-onboarding-report/design.md index 1db7a565..c439e5a4 100644 --- a/openspec/changes/streamlined-onboarding-report/design.md +++ b/openspec/changes/streamlined-onboarding-report/design.md @@ -47,7 +47,7 @@ agentCatches[tier] = round(Σ rule.fixed × 30 / d) `contextTokens` and `callsPerCatch` are agent estimates from this user's session habits (how large their sessions run, how many extra turns a missed issue costs). tskl.es exposes them as sliders, so the published page lets a reader adjust them, and turns them into its token figures itself. The other inputs (`cloudComplexity`, `pricePerMillionTokens`, `minutesPerComment`, `reviewerHourlyUsd`) are CLI constants, matching the schema's example values until there is a reason to differ. -The gate is `≥ 3` Taskless rules: `cli` tier, or `legacy` rules the agent wrote into the repository's own linter, with tests neither `fail` nor `refused`. Cloud candidates are carried in addition and never count. The projection does not gate: the page lets a reader adjust its inputs, so a threshold on it would gate on a number nobody sees. The gate lives in the CLI so it has one home. +The gate is `≥ 3` Taskless rules: `cli` tier, or `legacy` rules the agent wrote into the repository's own linter, and in either tier only those whose tests are neither `fail` nor `refused`. Cloud candidates are carried in addition and never count. The projection does not gate: the page lets a reader adjust its inputs, so a threshold on it would gate on a number nobody sees. The gate lives in the CLI so it has one home. ### D4. Tiers and engines @@ -79,7 +79,7 @@ For validation, the decision is zod 4's `z.fromJSONSchema()` over the vendored d ### D8. The record and the delete credential -`.taskless/reports/.json` holds `{ url, deleteUrl, expiresAt }` exactly as returned. The delete credential is the `deleteUrl` tskl.es returns once; the user's "delete key" is that value. It never reaches stdout or telemetry, and the directory is gitignored by migration 11. `report delete` reads the record and sends an HTTP `DELETE` to `deleteUrl`, the URL tskl.es returns for exactly that. +`.taskless/reports/.json` holds `{ url, deleteUrl, expiresAt }` exactly as returned. The delete credential is the `deleteUrl` tskl.es returns once; the user's "delete key" is that value. It never reaches stdout or telemetry, and the directory is gitignored by migration 11. `report --delete` reads the record and sends an HTTP `DELETE` to `deleteUrl`, per the schema's `x-taskless.delete` contract: 204 on success (caches may serve the page for up to a day), or 404 `not_available`, which is not retryable and gets its own code, `REPORT_NOT_AVAILABLE`, because its remedy differs from a failure's. The record survives a 404, since a report published moments ago can answer 404 before it has propagated. ### D9. Detect: one table for tools, presence, and hints diff --git a/openspec/changes/streamlined-onboarding-report/proposal.md b/openspec/changes/streamlined-onboarding-report/proposal.md index ebe7e961..f61e7fba 100644 --- a/openspec/changes/streamlined-onboarding-report/proposal.md +++ b/openspec/changes/streamlined-onboarding-report/proposal.md @@ -8,7 +8,7 @@ Onboarding is a conversation today: the agent asks which sources to mine, asks a - **Onboarding is always anonymous.** It authors only rules the CLI can write logged out (`legacy`, `sg`, `vale`). Candidates the agent judges are better solved by a runtime rule are not authored; they go into the report as `cloud` tier. - **Clarifying questions only when stuck.** The agent asks the user anything further only when it is unsure what rules to write AND the report gate refuses for fewer than three Taskless rules (`sg`, `vale`, or `legacy` rules it wrote). Cloud-tier candidates are carried in addition and do not count. - **Onboarding completes itself.** The agent runs `onboard --mark-complete` once the rules are written, without a confirmation question, and then runs the report recipe. In parallel it sends onboarding feedback through `rule-feedback`, answering as the respondent. `onboard` no longer carries the survey invite. -- **New `taskless report` command.** With no flags it prints a draft: everything the CLI can know about this repository's rules, plus `missing[]`, the facts only the agent can supply. `taskless report --from ` takes the agent's facts, computes the projection, enforces the gate, redacts, validates against the vendored ReportV1 schema, publishes to `https://tskl.es/api/v1/reports`, records the result under `.taskless/reports/`, and prints a brief templated rule summary with the link. `taskless report delete` removes a published report using the recorded delete credential. +- **New `taskless report` command.** With no flags it prints a draft: everything the CLI can know about this repository's rules, plus `missing[]`, the facts only the agent can supply. `taskless report --from ` takes the agent's facts, computes the projection, enforces the gate, redacts, validates against the vendored ReportV1 schema, publishes to `https://tskl.es/api/v1/reports`, records the result under `.taskless/reports/`, and prints a brief templated rule summary with the link. `taskless report --delete` removes a published report using the recorded delete credential. - **New `report` agent topic**: check what exists, supply what is missing, generate, delete on request. Onboarding hands off to it, and it serves "regenerate my Taskless report" later. - **ReportV1 is vendored** from `https://tskl.es/api/v1/schema` by a generate script, the same way the cloud API schema is. - **Migration 11** creates `.taskless/reports/` and adds `/reports/` to `.taskless/.gitignore`, since each record holds a credential that deletes a page anyone with the link can view. @@ -48,4 +48,4 @@ Diffs 3 and 4 may pass ~1200 lines; that is accepted for this change, since the - **Code**: `packages/cli/src/commands/` (new `report.ts`, `onboard.ts`), `packages/cli/src/agent/` (`onboard.md` v6, new `report.md`, `rule-feedback.md`), `packages/cli/src/detect/` (`scan.ts`, `host-tools.ts`, detect schema), `packages/cli/src/survey/constants.ts`, `packages/cli/src/filesystem/migrations/` (new 0011), `packages/cli/src/rules/report.ts` (renamed), `packages/cli/scripts/` (new schema fetch), `packages/cli/src/generated/` (vendored ReportV1), the skill and `tskl` command bodies, `packages/cli/src/prompts/` (topic classification). - **Network**: a new outbound call to `tskl.es` (publish and delete), separate from the Taskless API and requiring no auth. - **Privacy**: rule fixtures leave the machine on a page viewable by anyone with the link. tskl.es performs no redaction of its own; the agent and the CLI are solely responsible, and withhold rather than partially mask. -- **External dependency**: `report delete` sends an HTTP `DELETE` to the `deleteUrl` tskl.es returns on publish. The tskl.es side is adding that support; the schema's `x-taskless` block does not describe it yet. +- **External dependency**: `report --delete` sends an HTTP `DELETE` to the `deleteUrl` tskl.es returns on publish. The tskl.es side is adding that support; the schema's `x-taskless` block does not describe it yet. diff --git a/openspec/changes/streamlined-onboarding-report/specs/cli-report/spec.md b/openspec/changes/streamlined-onboarding-report/specs/cli-report/spec.md index 2f976647..bef5e2d2 100644 --- a/openspec/changes/streamlined-onboarding-report/specs/cli-report/spec.md +++ b/openspec/changes/streamlined-onboarding-report/specs/cli-report/spec.md @@ -6,7 +6,7 @@ Defines `taskless report`: the command that turns a repository's Taskless rules, ### Requirement: Report subcommand exists with a draft mode, a publish mode, and delete -The CLI SHALL provide a `taskless report` subcommand accepting the standard `--dir`/`-d` working-directory flag. Invoked with no mode flag it SHALL print a draft. Invoked with `--from ` it SHALL publish. `taskless report delete ` SHALL delete a previously published report. The subcommand SHALL require no authentication in any mode, and SHALL bootstrap `.taskless/` before reading or writing under it. +The CLI SHALL provide a `taskless report` subcommand accepting the standard `--dir`/`-d` working-directory flag. Invoked with no mode flag it SHALL print a draft. Invoked with `--from ` it SHALL publish. `taskless report --delete ` SHALL delete a previously published report. `--from` and `--delete` SHALL be mutually exclusive; supplying both SHALL exit non-zero with `INVALID_INPUT`. The subcommand SHALL require no authentication in any mode, and SHALL bootstrap `.taskless/` before reading or writing under it. #### Scenario: Report is registered @@ -51,10 +51,11 @@ The draft SHALL NOT perform any network call. The facts file SHALL be validated against a published facts schema before anything else happens. The `report` recipe SHALL embed that schema. -#### Scenario: Facts override nothing the CLI knows +#### Scenario: Facts cannot set what the CLI knows -- **WHEN** the facts file names a rule id that exists under `.taskless/rules/` and gives it a different engine -- **THEN** the published report SHALL carry the engine the CLI established +- **WHEN** the facts file gives a rule under `.taskless/rules/` an engine, a language, or a test status +- **THEN** the command SHALL refuse the field as unknown under `REPORT_INVALID` +- **AND** a report that publishes SHALL carry the engine, language, and test status the CLI established #### Scenario: Malformed facts are refused before any other work @@ -84,7 +85,7 @@ The CLI SHALL compute `projection.perMonth` itself from the merged facts and SHA ### Requirement: Publishing is gated on a meaningful report -`taskless report --from` SHALL refuse to publish unless the merged report holds at least three Taskless rules. A Taskless rule is one in the `cli` tier, or one in the `legacy` tier that the agent authored into the repository's own linter, whose test status is neither `fail` nor `refused`. Rules in the `cloud` tier SHALL NOT count toward the three; they are carried in addition. The projection SHALL NOT gate publishing. A refusal SHALL exit non-zero with `REPORT_INSUFFICIENT`, SHALL state how many of the required three exist, and SHALL perform no network call. +`taskless report --from` SHALL refuse to publish unless the merged report holds at least three Taskless rules. A Taskless rule is one in the `cli` tier, or one in the `legacy` tier that the agent authored into the repository's own linter; either way, its test status SHALL be neither `fail` nor `refused`. Rules in the `cloud` tier SHALL NOT count toward the three; they are carried in addition. The projection SHALL NOT gate publishing. A refusal SHALL exit non-zero with `REPORT_INSUFFICIENT`, SHALL state how many of the required three exist, and SHALL perform no network call. #### Scenario: Too few local rules @@ -126,6 +127,11 @@ tskl.es performs no redaction of its own. Before publishing, the CLI SHALL scan - **THEN** the command SHALL exit non-zero with `REPORT_REDACTION` naming that rule's title - **AND** SHALL perform no network call +#### Scenario: Every agent-supplied rule string is screened + +- **WHEN** the facts give a legacy rule a `tool` or `language`, or a cloud candidate a `language` or `id`, containing an email address or a host name +- **THEN** the command SHALL exit non-zero with `REPORT_REDACTION` naming that field + #### Scenario: The agent's counts are carried - **WHEN** the facts declare `person: 2` and the CLI matches no person names @@ -190,14 +196,27 @@ On a 201 response the CLI SHALL write `.taskless/reports/.jso ### Requirement: A published report can be deleted from its record -`taskless report delete ` SHALL read the named record under `.taskless/reports/`, delete the report by sending an HTTP `DELETE` to the recorded `deleteUrl`, the URL tskl.es returned for that purpose, and on success remove the record and print a one-line confirmation. A record that does not exist SHALL exit non-zero with `REPORT_NOT_FOUND`. A failed delete SHALL leave the record in place. +`taskless report --delete ` SHALL read the named record under `.taskless/reports/`, delete the report by sending an HTTP `DELETE` to the recorded `deleteUrl`, the URL tskl.es returned for that purpose, and on a 204 remove the record and print a one-line confirmation that says cached copies may be served for up to a day, as the tskl.es delete contract states. A 404 (`not_available`: never issued, already deleted, expired, or not yet propagated) SHALL exit non-zero with `REPORT_NOT_AVAILABLE`, SHALL NOT be retried, and SHALL keep the record. A record named outside `.taskless/reports/`, or not a `.json` file, SHALL exit non-zero with `INVALID_INPUT` without reading or removing it. A record that does not exist SHALL exit non-zero with `REPORT_NOT_FOUND`. A failed delete SHALL leave the record in place. #### Scenario: Deleting a report -- **WHEN** a user runs `taskless report delete 2026-10-13_142200.json` and the service confirms the deletion +- **WHEN** a user runs `taskless report --delete 2026-10-13_142200.json` and the service confirms the deletion - **THEN** the record SHALL be removed - **AND** the command SHALL print a confirmation and exit 0 +#### Scenario: A report tskl.es no longer has + +- **WHEN** the service answers the delete with 404 +- **THEN** the command SHALL exit non-zero with `REPORT_NOT_AVAILABLE` +- **AND** SHALL send no second request +- **AND** the record SHALL remain on disk + +#### Scenario: A record outside the records directory is refused + +- **WHEN** a user runs `taskless report --delete ../notes.json` +- **THEN** the command SHALL exit non-zero with `INVALID_INPUT` +- **AND** SHALL send no request and remove no file + #### Scenario: A failed delete keeps the credential - **WHEN** the service refuses or cannot be reached @@ -206,7 +225,7 @@ On a 201 response the CLI SHALL write `.taskless/reports/.jso ### Requirement: The report recipe drives the flow -The CLI SHALL embed a `report` recipe following the canonical recipe template. It SHALL instruct the agent to: run `taskless report` and read `missing`; supply each missing fact from its session, writing the facts to `.taskless/.tmp-report.json`; estimate context tokens and calls per catch from how this user's sessions actually run; run `taskless report --from` that path; on `REPORT_INSUFFICIENT`, ask the user only what would let it author the rules or establish the facts the refusal names, then retry; on success, show the CLI's summary as printed and tell the user the report can be deleted after review if they ask; delete the facts file; and on a request to delete, run `taskless report delete` with the record. The recipe SHALL embed the facts schema and SHALL state that tskl.es does not redact, so anything the agent cannot vouch for is to be withheld. +The CLI SHALL embed a `report` recipe following the canonical recipe template. It SHALL instruct the agent to: run `taskless report` and read `missing`; supply each missing fact from its session, writing the facts to `.taskless/.tmp-report.json`; estimate context tokens and calls per catch from how this user's sessions actually run; run `taskless report --from` that path; on `REPORT_INSUFFICIENT`, ask the user only what would let it author the rules or establish the facts the refusal names, then retry; on success, show the CLI's summary as printed and tell the user the report can be deleted after review if they ask; delete the facts file; and on a request to delete, run `taskless report --delete` with the record. The recipe SHALL embed the facts schema and SHALL state that tskl.es does not redact, so anything the agent cannot vouch for is to be withheld. #### Scenario: The recipe asks questions only on refusal diff --git a/openspec/changes/streamlined-onboarding-report/tasks.md b/openspec/changes/streamlined-onboarding-report/tasks.md index 8a594af2..b649ace0 100644 --- a/openspec/changes/streamlined-onboarding-report/tasks.md +++ b/openspec/changes/streamlined-onboarding-report/tasks.md @@ -14,19 +14,19 @@ ## 3. taskless report (PR 3) -- [ ] 3.1 Add `scripts/fetch-report-schema.ts` and a `generate:report` script vendoring `src/generated/report-v1.schema.json` in the style of `fetch-api-schema.ts`. Validate with `z.fromJSONSchema()` over the vendored document (D6). Verify with a test that validates the schema's `examples[0]` and rejects each counter-example. -- [ ] 3.2 Add migration `0011` creating `.taskless/reports/` with its stub and adding `/reports/` to `.taskless/.gitignore`. Verify migration tests: upgrade from version 10, fresh scaffold, and idempotence with existing records. -- [ ] 3.3 Build the draft: producer, scope from the `origin` remote, one entry per rule directory with tier, engine, language, test status and first fail/pass fixture, default projection inputs, and `missing[]`. Verify `taskless report` against fixture repositories (sg+vale rules; GitHub origin; no origin) with a snapshot test, and that it makes no network call. -- [ ] 3.4 Define the facts zod schema (history, window, rule facts, `legacy[]`, `cloud[]`, estimates, agent redaction counts, withheld fixtures) and the merge in which CLI-established fields win. Verify tests for an engine conflict, an unknown `perMonth` key refused as `REPORT_INVALID`, and cloud candidates published as `cloud`/`runtime`/`not-run`. -- [ ] 3.5 Implement the projection (D3) and the gate. Verify tests: the 90-day/30-fixes → 10 normalization, two local rules → `REPORT_INSUFFICIENT` naming 2 of 3, one `sg` plus two agent-written `legacy` rules passing, a near-zero projection not refusing, and a failing rule not counted. -- [ ] 3.6 Implement redaction (D5): the five pattern classes, withholding a whole fixture on a hit or when over length, `REPORT_REDACTION` for an agent-supplied string, summed counts, and `policyVersion`. Verify tests per class and that no withheld fixture byte appears in the payload. -- [ ] 3.7 Validate against the vendored schema and the 262,144-byte cap before sending. Verify tests that an invalid or oversized payload exits non-zero and that the stubbed client is never called. -- [ ] 3.8 Implement the publish client (D7): private retention, a 502 retried at most twice, no retry on timeout (`REPORT_PUBLISH_UNKNOWN`), 429 → `REPORT_RATE_LIMITED`, 400 → `REPORT_INVALID` with the service's details, base URL overridable for tests. Verify each path against a local test server. -- [ ] 3.9 On 201, write `.taskless/reports/.json` with `{ url, deleteUrl, expiresAt }` and print the templated summary (rule title and tier per line, headline projection, URL, expiry, the deletable-after-review sentence); `--json` prints URL, expiry, record path, and summary. Verify tests that exactly one record is written and that the delete credential appears on stdout in neither form. -- [ ] 3.10 Implement `taskless report delete ` as an HTTP `DELETE` to the recorded `deleteUrl`: remove the record on success, keep it on failure, `REPORT_NOT_FOUND` for a missing record. Verify against the local test server. -- [ ] 3.11 Emit `cli_report_drafted`, `cli_report_published` (per-tier counts and gate outcome), `cli_report_refused` (code), and `cli_report_deleted`. Verify a telemetry test that no event carries a URL, credential, title, or fixture. -- [ ] 3.12 Write `agent/report.md` (check, supply what `missing` names, estimate from session habits, generate, handle `REPORT_INSUFFICIENT` with questions only about the rules it lacks, show the summary, mention deletion, delete the facts file, delete on request), embedding the facts JSON Schema. Register the topic in the agent index and classify it in the prompts topic list. Verify `taskless agent report` opens with `# Topic: report`, contains the schema, and the topic-membership and "recipes name only commands that exist" checks pass. -- [ ] 3.13 Extend the changeset with the `report` command. Run `pnpm typecheck` and `pnpm lint`; both pass. +- [x] 3.1 Add `scripts/fetch-report-schema.ts` and a `generate:report` script vendoring `src/generated/report-v1.schema.json` in the style of `fetch-api-schema.ts`. Validate with `z.fromJSONSchema()` over the vendored document (D6). Verify with a test that validates the schema's `examples[0]` and rejects each counter-example. +- [x] 3.2 Add migration `0011` creating `.taskless/reports/` with its stub and adding `/reports/` to `.taskless/.gitignore`. Verify migration tests: upgrade from version 10, fresh scaffold, and idempotence with existing records. +- [x] 3.3 Build the draft: producer, scope from the `origin` remote, one entry per rule directory with tier, engine, language, test status and first fail/pass fixture, default projection inputs, and `missing[]`. Verify `taskless report` against fixture repositories (sg+vale rules; GitHub origin; no origin) with a snapshot test, and that it makes no network call. +- [x] 3.4 Define the facts zod schema (history, window, rule facts, `legacy[]`, `cloud[]`, estimates, agent redaction counts, withheld fixtures) and the merge in which CLI-established fields win. Verify tests for an engine conflict, an unknown `perMonth` key refused as `REPORT_INVALID`, and cloud candidates published as `cloud`/`runtime`/`not-run`. +- [x] 3.5 Implement the projection (D3) and the gate. Verify tests: the 90-day/30-fixes → 10 normalization, two local rules → `REPORT_INSUFFICIENT` naming 2 of 3, one `sg` plus two agent-written `legacy` rules passing, a near-zero projection not refusing, and a failing rule not counted. +- [x] 3.6 Implement redaction (D5): the five pattern classes, withholding a whole fixture on a hit or when over length, `REPORT_REDACTION` for an agent-supplied string, summed counts, and `policyVersion`. Verify tests per class and that no withheld fixture byte appears in the payload. +- [x] 3.7 Validate against the vendored schema and the 262,144-byte cap before sending. Verify tests that an invalid or oversized payload exits non-zero and that the stubbed client is never called. +- [x] 3.8 Implement the publish client (D7): private retention, a 502 retried at most twice, no retry on timeout (`REPORT_PUBLISH_UNKNOWN`), 429 → `REPORT_RATE_LIMITED`, 400 → `REPORT_INVALID` with the service's details, base URL overridable for tests. Verify each path against a local test server. +- [x] 3.9 On 201, write `.taskless/reports/.json` with `{ url, deleteUrl, expiresAt }` and print the templated summary (rule title and tier per line, headline projection, URL, expiry, the deletable-after-review sentence); `--json` prints URL, expiry, record path, and summary. Verify tests that exactly one record is written and that the delete credential appears on stdout in neither form. +- [x] 3.10 Implement `taskless report --delete ` as an HTTP `DELETE` to the recorded `deleteUrl`: remove the record on success, keep it on failure, `REPORT_NOT_FOUND` for a missing record. Verify against the local test server. +- [x] 3.11 Emit `cli_report_drafted`, `cli_report_published` (per-tier counts and gate outcome), `cli_report_refused` (code), and `cli_report_deleted`. Verify a telemetry test that no event carries a URL, credential, title, or fixture. +- [x] 3.12 Write `agent/report.md` (check, supply what `missing` names, estimate from session habits, generate, handle `REPORT_INSUFFICIENT` with questions only about the rules it lacks, show the summary, mention deletion, delete the facts file, delete on request), embedding the facts JSON Schema. Register the topic in the agent index and classify it in the prompts topic list. Verify `taskless agent report` opens with `# Topic: report`, contains the schema, and the topic-membership and "recipes name only commands that exist" checks pass. +- [x] 3.13 Extend the changeset with the `report` command. Run `pnpm typecheck` and `pnpm lint`; both pass. ## 4. Onboard v6, skill, and feedback (PR 4) diff --git a/packages/cli/package.json b/packages/cli/package.json index 77484bce..ec94102b 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -16,6 +16,7 @@ "generate:api:schema": "tsx scripts/fetch-api-schema.ts", "generate:api:types": "openapi-typescript src/generated/api-v2.schema.json -o src/generated/api-v2.d.ts", "generate:ast-grep-schema": "tsx scripts/fetch-ast-grep-schema.ts", + "generate:report-schema": "tsx scripts/fetch-report-schema.ts", "generate:rule-hash-vectors": "tsx scripts/fetch-rule-hash-vectors.ts", "generate:vale-schema": "tsx scripts/generate-vale-schema.ts", "prebuild": "tsx scripts/fetch-rule-hash-vectors.ts", diff --git a/packages/cli/scripts/fetch-report-schema.ts b/packages/cli/scripts/fetch-report-schema.ts new file mode 100644 index 00000000..93222522 --- /dev/null +++ b/packages/cli/scripts/fetch-report-schema.ts @@ -0,0 +1,53 @@ +import { packageFile, writeJsonArtifact } from "./artifacts"; + +/** + * Vendor the ReportV1 schema tskl.es publishes, which `taskless report` + * validates every payload against before it is sent. + * + * Committed for the reason `fetch-api-schema.ts` commits the API document: a + * change to the contract arrives as a readable diff of the document itself, + * including the `x-taskless` block that carries the publish URL, the size + * limit, and the retry contract the client reads from it. + * + * Never on the build path, and it fails loudly rather than keeping the old + * bytes: a refresh that silently kept them would report "no contract change" + * when what happened was "no contract reached". + */ + +const SOURCE_URL = + process.env.TASKLESS_REPORT_SCHEMA_URL ?? "https://tskl.es/api/v1/schema"; +const OUTPUT_PATH = packageFile("src", "generated", "report-v1.schema.json"); + +console.log("Fetching the tskl.es ReportV1 schema..."); +console.log(` URL: ${SOURCE_URL}`); + +let response: Response; +try { + response = await fetch(SOURCE_URL); +} catch (error) { + const message = error instanceof Error ? error.message : String(error); + throw new Error(`Failed to reach ${SOURCE_URL}: ${message}`); +} +if (!response.ok) { + throw new Error(`HTTP ${String(response.status)} fetching ${SOURCE_URL}`); +} + +let schema: Record; +try { + schema = (await response.json()) as Record; +} catch (error) { + const message = error instanceof Error ? error.message : String(error); + throw new Error(`Response from ${SOURCE_URL} was not valid JSON: ${message}`); +} + +// A 200 carrying some other JSON document must not replace the contract: the +// failure would surface a step later as every report refused, with nothing +// pointing back here. +if (schema.title !== "ReportV1" || typeof schema.properties !== "object") { + throw new Error( + `Response from ${SOURCE_URL} is not the ReportV1 schema (title ${JSON.stringify(schema.title)})` + ); +} + +await writeJsonArtifact(OUTPUT_PATH, schema); +console.log(` Wrote ${OUTPUT_PATH}`); diff --git a/packages/cli/src/agent/report.md b/packages/cli/src/agent/report.md new file mode 100644 index 00000000..a5bc41cc --- /dev/null +++ b/packages/cli/src/agent/report.md @@ -0,0 +1,127 @@ +# Topic: report (CLI v%(CLI_VERSION)s / topic v1) + +## Goal +Publish a Taskless report: an unlisted page on tskl.es that shows what +this repository's Taskless rules catch, built from what the CLI can +measure plus what you learned in this session. Then show the user the +CLI's short summary and the link. The same recipe regenerates a report +later and deletes one on request. + +## Preconditions +- At least three working Taskless rules: `sg` or `vale` rules under + `.taskless/rules/`, or rules you wrote into the repository's own + linter. The CLI enforces this and tells you when it is not met. +- No auth required. Publishing does not log in and is not tied to an + account. +- tskl.es does NOT redact anything. Whatever you put in the facts file, + and every fixture the CLI publishes, is visible to anyone with the + link. The CLI withholds what its patterns catch; anything you cannot + vouch for beyond that is yours to withhold. + +## Steps + +1. **See what the CLI already knows.** Run: + + ``` + %(TASKLESS_CLI)s report + ``` + + It prints a JSON draft: the repository scope, every rule under + `.taskless/rules/` with its engine, language, test status and first + fixtures, and `missing`, the facts only you can supply. Read + `missing`; it is the checklist for the next step. + +2. **Supply what is missing.** Write + `.taskless/.tmp-report.json` following the schema below. Every entry + comes from this session or from sources you can reach now, never + from guesswork you present as measurement: + + - `window`: the span of history you mined, as dates. + - `history`: what that window held. Merged pull or merge requests, + review comments, the ones an agent helped write, and how many + recurring review patterns were rule-shaped versus judgment calls. + Zeros are honest when there was nothing to mine. + - `rules`: one entry per rule the draft listed, with a title a + reader understands, how many review comments and pull requests + asked for what it enforces, and how often the pattern was fixed in + the window. Set `fixedBasis` to `measured` only when you counted. + - `legacy`: rules you wrote into the repository's own linter, which + the CLI cannot see. + - `cloud`: runtime-shaped candidates you judged better solved by a + runtime rule and did not author. + - `estimates`: `contextTokens` is how large this user's sessions + typically run; `callsPerCatch` is how many extra agent turns a + missed issue costs them before it is fixed. Estimate from what you + have actually seen of how this user works, not from a best case. + The page lets a reader adjust both. + - `redaction`: what you removed yourself, by kind, and any fixture + you choose not to publish. Names of people and customers are + yours to judge; the CLI's patterns cannot see them. + +3. **Generate.** Run: + + ``` + %(TASKLESS_CLI)s report --from .taskless/.tmp-report.json + ``` + + On failure it prints `Error (): `. Handle the code + from the table under `## Errors`. + `REPORT_INSUFFICIENT` is the only one that may involve the user: if + you are unsure what further rules to write, ask the user only what + would let you write the rules the message says are missing, write + them, and run this step again. Otherwise write the rules without + asking. + +4. **Show the result.** On success the CLI prints a short summary: the + rules, the projection, the link, and when it expires. Show it to the + user exactly as printed. Then tell them, in one line, that the + report can be deleted after they have reviewed it, and that they + only need to ask. + +5. **Clean up.** Delete `.taskless/.tmp-report.json`. The CLI kept a + record of the report in `.taskless/reports/`, which is gitignored + because it holds the URL that deletes the page. Do not print or + paste that file's contents. + +6. **Delete on request.** If the user asks to delete the report, run: + + ``` + %(TASKLESS_CLI)s report --delete + ``` + + `` is the file name the summary named, under + `.taskless/reports/`. The record is removed only once tskl.es + confirms the deletion. Tell the user cached copies of the page may + be served for up to a day afterwards. + +## Facts schema + +The `--from` JSON file conforms to: + +```json +%(INPUT_SCHEMA)s +``` + +## Errors + +| code | meaning | fix | +|--------------------------|-----------------------------------------------------------------|---------------------------------------------------------------------------| +| `REPORT_INSUFFICIENT` | fewer than three working Taskless rules | write the missing rules (step 3), then retry | +| `REPORT_INVALID` | the facts file, or the assembled report, failed validation | the message names each field; fix the facts file and retry | +| `REPORT_REDACTION` | a title or label you wrote looks like it holds a secret or contact detail | rewrite the field the message names without it | +| `REPORT_TOO_LARGE` | the report is over tskl.es's size limit | withhold long fixtures or drop cloud candidates, then retry | +| `REPORT_RATE_LIMITED` | tskl.es is limiting publishes from this address | wait a minute, then retry | +| `REPORT_PUBLISH_FAILED` | tskl.es could not store the report; nothing was published | retry later | +| `REPORT_PUBLISH_UNKNOWN` | publishing timed out; a report may exist | tell the user; do not retry on your own, since a retry may duplicate it | +| `REPORT_NOT_FOUND` | no record by that name in `.taskless/reports/` | list the directory and use a file name from it | +| `REPORT_NOT_AVAILABLE` | tskl.es has no report for that delete URL | it was already deleted, expired, or has not propagated yet; do not retry | +| `REPORT_DELETE_FAILED` | tskl.es did not delete the report | the record is kept; retry later | + +## See Also + +- `%(TASKLESS_CLI)s agent onboard`: the onboarding flow that ends by + publishing a report +- `%(TASKLESS_CLI)s agent route`: how to author a rule, when the report + needs more of them +- `%(TASKLESS_CLI)s agent ci`: run the rules in CI, so they keep + catching what the report shows diff --git a/packages/cli/src/commands/names.ts b/packages/cli/src/commands/names.ts index 91dac585..28ab4a1c 100644 --- a/packages/cli/src/commands/names.ts +++ b/packages/cli/src/commands/names.ts @@ -19,6 +19,7 @@ export const SUBCOMMAND_NAMES = [ "info", "init", "onboard", + "report", "rule", "share", "update", diff --git a/packages/cli/src/commands/report.ts b/packages/cli/src/commands/report.ts new file mode 100644 index 00000000..bcd55ab2 --- /dev/null +++ b/packages/cli/src/commands/report.ts @@ -0,0 +1,186 @@ +import { resolve } from "node:path"; +import process from "node:process"; + +import { defineCommand } from "citty"; + +import { ensureTasklessDirectory } from "../filesystem/directory"; +import { assembleReport } from "../report/assemble"; +import { buildDraft } from "../report/draft"; +import { reportFactsSchema } from "../schemas/report-facts"; +import { + deletePublishedReport, + encodeReport, + publishReport, +} from "../report/publish"; +import { readRecord, removeRecord, writeRecord } from "../report/records"; +import { reportSchemaIssues } from "../report/schema"; +import { renderSummary } from "../report/summary"; +import type { ReportV1 } from "../report/types"; +import { getTelemetry } from "../telemetry"; +import { writeJsonError, type CLIErrorCode } from "../types/errors"; +import { CLIError } from "../util/cli-error"; +import { readJsonInput } from "../util/json-input"; +import { getCliPrefix } from "../util/package-manager"; + +/** Per-tier rule counts: what telemetry may know about a report. */ +function tierCounts(report: ReportV1): Record { + const counts = { cli: 0, legacy: 0, cloud: 0 }; + for (const rule of report.rules) counts[rule.tier] += 1; + return counts; +} + +export const reportCommand = defineCommand({ + meta: { + name: "report", + description: + "Draft, publish, or delete a Taskless report on tskl.es (an unlisted page showing what your rules catch)", + }, + args: { + dir: { + type: "string", + alias: "d", + description: "Working directory", + }, + from: { + type: "string", + description: + "Publish: path to the facts JSON the agent wrote, e.g. .taskless/.tmp-report.json", + }, + delete: { + type: "string", + description: + "Delete a published report by its record in .taskless/reports/, e.g. 2026-10-13_142200.json", + }, + json: { + type: "boolean", + description: + "Print JSON: the publish result, or the { ok:false, code, message } envelope on error", + default: false, + }, + }, + async run({ args }) { + const cwd = resolve(args.dir ?? process.cwd()); + const telemetry = await getTelemetry(cwd); + + /** Report a failure once, in the requested form, and stop. */ + function fail(error: CLIError): never { + const code: CLIErrorCode = error.code ?? "INTERNAL_ERROR"; + if (args.json) { + writeJsonError(code, error.message); + } else { + // The code is printed so a recipe can branch on it without `--json`. + console.error(`Error (${code}): ${error.message}`); + } + telemetry.capture("cli_report_refused", { code }); + process.exitCode = 1; + throw new CLIError(error.message, code, { reported: true }); + } + + async function guarded(work: () => Promise): Promise { + try { + return await work(); + } catch (error) { + if (error instanceof CLIError) fail(error); + throw error; + } + } + + if (args.from !== undefined && args.delete !== undefined) { + fail( + new CLIError( + "--from publishes and --delete deletes; pass one or the other.", + "INVALID_INPUT" + ) + ); + } + + await ensureTasklessDirectory(cwd, { + onNotice: (message: string) => { + if (!args.json) console.error(message); + }, + }); + + if (args.delete !== undefined) { + const record = args.delete; + await guarded(async () => { + const { path, published } = await readRecord(cwd, record); + // The record goes only once the service confirms: it holds the only + // credential that can try again. + await deletePublishedReport(published.deleteUrl); + await removeRecord(path); + telemetry.capture("cli_report_deleted"); + if (args.json) { + console.log(JSON.stringify({ ok: true, deleted: published.url })); + } else { + // Per tskl.es, caches may serve the page for up to a day after. + console.log( + `Deleted the report at ${published.url}. Cached copies may be served for up to a day.` + ); + } + }); + return; + } + + if (args.from === undefined) { + const draft = await buildDraft(cwd); + telemetry.capture("cli_report_drafted", { + rules: draft.rules.length, + }); + console.log(JSON.stringify(draft, null, 2)); + return; + } + + const factsPath = resolve(cwd, args.from); + await guarded(async () => { + let facts; + try { + facts = await readJsonInput(factsPath, reportFactsSchema); + } catch (error) { + throw new CLIError( + error instanceof Error ? error.message : String(error), + "REPORT_INVALID" + ); + } + + const now = new Date(); + const report = assembleReport(await buildDraft(cwd), facts, now); + + const issues = reportSchemaIssues(report); + if (issues.length > 0) { + throw new CLIError( + `The assembled report does not match the tskl.es schema:\n ${issues.join("\n ")}`, + "REPORT_INVALID" + ); + } + const body = encodeReport(report); + + const published = await publishReport(body); + const record = await writeRecord(cwd, published, now); + telemetry.capture("cli_report_published", { + ...tierCounts(report), + withheldFixtures: report.redaction.withheldFixtures, + }); + + const summary = renderSummary({ + report, + url: published.url, + expiresAt: published.expiresAt, + record, + invocation: getCliPrefix(), + }); + if (args.json) { + console.log( + JSON.stringify({ + ok: true, + url: published.url, + expiresAt: published.expiresAt, + record: `.taskless/reports/${record}`, + summary, + }) + ); + } else { + console.log(summary); + } + }); + }, +}); diff --git a/packages/cli/src/filesystem/migrate.ts b/packages/cli/src/filesystem/migrate.ts index 937fec37..2e9d57ce 100644 --- a/packages/cli/src/filesystem/migrate.ts +++ b/packages/cli/src/filesystem/migrate.ts @@ -17,6 +17,7 @@ import ignoreScratchFiles from "./migrations/0007-ignore-scratch-files"; import dropBasedOnStyles from "./migrations/0008-drop-based-on-styles"; import uniqueRuleIds from "./migrations/0009-unique-rule-ids"; import adoptStrayTests from "./migrations/0010-adopt-stray-tests"; +import reportsDirectory from "./migrations/0011-reports-directory"; const migrations: Migrations = { "1": init, @@ -29,6 +30,7 @@ const migrations: Migrations = { "8": dropBasedOnStyles, "9": uniqueRuleIds, "10": adoptStrayTests, + "11": reportsDirectory, }; /** Global flag that downgrades a too-new scaffold from an error to a skip. */ diff --git a/packages/cli/src/filesystem/migrations/0011-reports-directory.ts b/packages/cli/src/filesystem/migrations/0011-reports-directory.ts new file mode 100644 index 00000000..09cc7d2f --- /dev/null +++ b/packages/cli/src/filesystem/migrations/0011-reports-directory.ts @@ -0,0 +1,46 @@ +import { mkdir, writeFile } from "node:fs/promises"; +import { dirname, join } from "node:path"; + +import { addToGitignore } from "../gitignore"; +import type { Migration } from "../types"; + +/** + * Create `.taskless/reports/`, where `taskless report` records each report it + * publishes, and keep it out of git. + * + * Each record holds the delete URL tskl.es returns exactly once: the only way + * to take down a page anyone with its link can view. Committed, it would hand + * that to everyone who can read the repository, so the directory is ignored + * whole rather than file by file. + * + * The stub says what the directory is for to someone who finds it, since a + * directory that git ignores has no history to explain it. Written only when + * absent, so a rerun never touches records or an edited stub. + * + * Anchored with a leading `/`, as `0007` anchors `/.tmp-*`: a rule directory is + * free to hold something named `reports`. + */ +const STUB = `# Taskless reports + +One JSON file per report \`taskless report\` published to tskl.es, holding +its link, when it expires, and the URL that deletes it. + +This directory is gitignored: the delete URL is the only way to take a +report down early, and anyone holding it can. Delete a report with +\`taskless report --delete \`. +`; + +const migration: Migration = async (directory) => { + const reports = join(directory, "reports"); + await mkdir(reports, { recursive: true }); + await writeFile(join(reports, "README.md"), STUB, { + encoding: "utf8", + flag: "wx", + }).catch((error: unknown) => { + if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; + }); + // `addToGitignore` takes the project root and appends `.taskless` itself. + await addToGitignore(dirname(directory), ["/reports/"]); +}; + +export default migration; diff --git a/packages/cli/src/generated/report-v1.schema.json b/packages/cli/src/generated/report-v1.schema.json new file mode 100644 index 00000000..601afbdd --- /dev/null +++ b/packages/cli/src/generated/report-v1.schema.json @@ -0,0 +1,655 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "schemaVersion": { + "type": "string", + "const": "1" + }, + "generatedAt": { + "type": "string", + "format": "date-time", + "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$" + }, + "producer": { + "type": "object", + "properties": { + "kind": { + "type": "string", + "const": "cli" + }, + "cliVersion": { + "type": "string", + "minLength": 1, + "maxLength": 64 + } + }, + "required": ["kind", "cliVersion"], + "additionalProperties": false + }, + "scope": { + "oneOf": [ + { + "type": "object", + "properties": { + "provider": { + "type": "string", + "const": "github" + }, + "owner": { + "type": "string", + "maxLength": 39, + "pattern": "^[\\dA-Za-z](?:[\\dA-Za-z]|-(?=[\\dA-Za-z]))*$" + }, + "repo": { + "type": "string", + "maxLength": 100, + "pattern": "^(?!\\.{1,2}$)[\\w.-]+$" + }, + "url": { + "type": "string", + "maxLength": 200, + "pattern": "^https:\\/\\/github\\.com\\/[\\dA-Za-z-]+\\/[\\w.-]+$" + }, + "retention": { + "type": "string", + "enum": ["public", "private"] + }, + "window": { + "type": "object", + "properties": { + "from": { + "type": "string", + "format": "date", + "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))$" + }, + "to": { + "type": "string", + "format": "date", + "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))$" + } + }, + "required": ["from", "to"], + "additionalProperties": false + } + }, + "required": ["provider", "owner", "repo", "url", "window"], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "provider": { + "type": "string", + "const": "none" + }, + "label": { + "type": "string", + "minLength": 1, + "maxLength": 100 + }, + "window": { + "type": "object", + "properties": { + "from": { + "type": "string", + "format": "date", + "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))$" + }, + "to": { + "type": "string", + "format": "date", + "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))$" + } + }, + "required": ["from", "to"], + "additionalProperties": false + } + }, + "required": ["provider", "window"], + "additionalProperties": false + } + ] + }, + "history": { + "type": "object", + "properties": { + "mergedPrs": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "reviewComments": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "agentAssistedPrs": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "patterns": { + "type": "object", + "properties": { + "ruleShaped": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "judgment": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + } + }, + "required": ["ruleShaped", "judgment"], + "additionalProperties": false + } + }, + "required": [ + "mergedPrs", + "reviewComments", + "agentAssistedPrs", + "patterns" + ], + "additionalProperties": false + }, + "rules": { + "minItems": 1, + "maxItems": 200, + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string", + "maxLength": 128, + "pattern": "^[\\dA-Za-z][\\w.-]*$" + }, + "title": { + "type": "string", + "minLength": 1, + "maxLength": 200 + }, + "tier": { + "type": "string", + "enum": ["cli", "cloud", "legacy"] + }, + "engine": { + "type": "string", + "enum": ["sg", "vale", "runtime"] + }, + "tool": { + "type": "string", + "minLength": 1, + "maxLength": 64, + "pattern": "^[\\w@./+-]+$" + }, + "language": { + "type": "string", + "minLength": 1, + "maxLength": 40 + }, + "origin": { + "type": "object", + "properties": { + "reviewComments": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "prs": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + } + }, + "required": ["reviewComments", "prs"], + "additionalProperties": false + }, + "fixed": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "fixedBasis": { + "type": "string", + "enum": ["measured", "estimated"] + }, + "complexity": { + "anyOf": [ + { + "type": "number", + "exclusiveMinimum": 0, + "maximum": 100 + }, + { + "type": "null" + } + ] + }, + "tests": { + "type": "object", + "properties": { + "status": { + "type": "string", + "enum": ["pass", "fail", "refused", "not-run"] + }, + "fail": { + "anyOf": [ + { + "type": "object", + "properties": { + "code": { + "type": "string", + "minLength": 1, + "maxLength": 4000 + } + }, + "required": ["code"], + "additionalProperties": false + }, + { + "type": "null" + } + ] + }, + "pass": { + "anyOf": [ + { + "type": "object", + "properties": { + "code": { + "type": "string", + "minLength": 1, + "maxLength": 4000 + } + }, + "required": ["code"], + "additionalProperties": false + }, + { + "type": "null" + } + ] + }, + "withheld": { + "maxItems": 2, + "type": "array", + "items": { + "type": "string", + "enum": ["fail", "pass"] + } + } + }, + "required": ["status", "fail", "pass", "withheld"], + "additionalProperties": false + } + }, + "required": [ + "id", + "title", + "tier", + "language", + "fixed", + "fixedBasis", + "complexity", + "tests" + ], + "additionalProperties": false + } + }, + "projection": { + "type": "object", + "properties": { + "perMonth": { + "type": "object", + "properties": { + "commentsPrevented": { + "type": "object", + "properties": { + "cli": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "cloud": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "legacy": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + } + }, + "required": ["cli", "cloud"], + "additionalProperties": false + }, + "agentCatches": { + "type": "object", + "properties": { + "cli": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "cloud": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "legacy": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + } + }, + "required": ["cli", "cloud"], + "additionalProperties": false + } + }, + "required": ["agentCatches"], + "additionalProperties": false + }, + "inputs": { + "type": "object", + "properties": { + "contextTokens": { + "type": "integer", + "exclusiveMinimum": 0, + "maximum": 10000000 + }, + "callsPerCatch": { + "type": "number", + "exclusiveMinimum": 0, + "maximum": 1000 + }, + "cloudComplexity": { + "type": "number", + "exclusiveMinimum": 0, + "maximum": 100 + }, + "pricePerMillionTokens": { + "type": "number", + "minimum": 0, + "maximum": 1000 + }, + "minutesPerComment": { + "type": "number", + "minimum": 0, + "maximum": 600 + }, + "reviewerHourlyUsd": { + "type": "number", + "minimum": 0, + "maximum": 10000 + } + }, + "required": [ + "contextTokens", + "callsPerCatch", + "cloudComplexity", + "pricePerMillionTokens", + "minutesPerComment", + "reviewerHourlyUsd" + ], + "additionalProperties": false + } + }, + "required": ["perMonth", "inputs"], + "additionalProperties": false + }, + "redaction": { + "type": "object", + "properties": { + "policyVersion": { + "type": "string", + "minLength": 1, + "maxLength": 16 + }, + "counts": { + "type": "object", + "properties": { + "secret": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "email": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "person": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "customer": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "host": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "ip": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "phone": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + }, + "text": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + } + }, + "required": [ + "secret", + "email", + "person", + "customer", + "host", + "ip", + "phone", + "text" + ], + "additionalProperties": false + }, + "withheldFixtures": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000 + } + }, + "required": ["policyVersion", "counts", "withheldFixtures"], + "additionalProperties": false + } + }, + "required": [ + "schemaVersion", + "generatedAt", + "producer", + "scope", + "rules", + "projection", + "redaction" + ], + "additionalProperties": false, + "$id": "https://tskl.es/api/v1/schema", + "title": "ReportV1", + "description": "A Taskless Report, as a CLI publishes it to tskl.es. Every object rejects unknown keys.", + "examples": [ + { + "schemaVersion": "1", + "generatedAt": "2026-10-12T21:04:00Z", + "producer": { + "kind": "cli", + "cliVersion": "0.13.0" + }, + "scope": { + "provider": "github", + "owner": "northwind-labs", + "repo": "crm-web", + "url": "https://github.com/northwind-labs/crm-web", + "retention": "private", + "window": { + "from": "2026-07-14", + "to": "2026-10-12" + } + }, + "history": { + "mergedPrs": 418, + "reviewComments": 2746, + "agentAssistedPrs": 255, + "patterns": { + "ruleShaped": 19, + "judgment": 4 + } + }, + "rules": [ + { + "id": "use-api-client", + "title": "Call the API through the shared apiClient", + "tier": "cli", + "engine": "sg", + "language": "TypeScript", + "origin": { + "reviewComments": 142, + "prs": 96 + }, + "fixed": 31, + "fixedBasis": "measured", + "complexity": null, + "tests": { + "status": "pass", + "fail": { + "code": "const res = await fetch(`/api/accounts/${id}`)" + }, + "pass": { + "code": "const account = await apiClient.get(`/accounts/${id}`)" + }, + "withheld": [] + } + }, + { + "id": "require-tenant-on-routes", + "title": "New API routes call requireTenant()", + "tier": "cloud", + "engine": "runtime", + "language": "TypeScript", + "origin": { + "reviewComments": 88, + "prs": 63 + }, + "fixed": 19, + "fixedBasis": "estimated", + "complexity": 4, + "tests": { + "status": "not-run", + "fail": { + "code": "export const GET = route(async (req) => {\n return db.deals.findMany()\n})" + }, + "pass": null, + "withheld": ["pass"] + } + } + ], + "projection": { + "perMonth": { + "commentsPrevented": { + "cli": 186, + "cloud": 105 + }, + "agentCatches": { + "cli": 116, + "cloud": 62 + } + }, + "inputs": { + "contextTokens": 150000, + "callsPerCatch": 20, + "cloudComplexity": 3, + "pricePerMillionTokens": 0.5, + "minutesPerComment": 4, + "reviewerHourlyUsd": 120 + } + }, + "redaction": { + "policyVersion": "1", + "counts": { + "secret": 9, + "email": 31, + "person": 0, + "customer": 4, + "host": 2, + "ip": 0, + "phone": 0, + "text": 1 + }, + "withheldFixtures": 1 + } + } + ], + "x-taskless": { + "publish": { + "method": "POST", + "url": "https://tskl.es/api/v1/reports", + "maxBytes": 262144, + "rateLimitPerIp": { + "limit": 10, + "periodSeconds": 60 + }, + "response": { + "status": 201, + "body": { + "url": "The report on tskl.es. Anyone with it can view it.", + "deleteUrl": "The only way to delete the report before it expires. Returned once, here.", + "expiresAt": "30 days after publishing, or 183 for public retention." + } + }, + "errors": { + "validation_error": { + "status": 400, + "description": "The body is not JSON, fails the schema, or fails a consistency check; `details` names each failure. Not retryable as sent." + }, + "payload_too_large": { + "status": 413, + "description": "The body is larger than `maxBytes`, by its declared length or the bytes read. Refused before parsing." + }, + "rate_limited": { + "status": 429, + "description": "Too many publishes from this address. Retry after a minute." + }, + "publish_failed": { + "status": 502, + "description": "The report was valid but could not be rendered or stored. Nothing was published: retry. Retry only on this error, never after a timeout, which may still publish." + } + } + }, + "delete": { + "method": "DELETE", + "url": "The `deleteUrl` from the publish response.", + "response": { + "status": 204, + "body": "None. The report, its JSON and its card are deleted; caches may keep serving them for up to a day." + }, + "errors": { + "not_available": { + "status": 404, + "description": "No report answers to this key or delete token: it was never issued, was deleted, expired, or has not propagated yet. Not retryable." + } + } + } + } +} diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index 99b2a49a..ea91fc1e 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -14,6 +14,7 @@ import { validateCommand, verifyCommand } from "./commands/verify"; import { infoCommand } from "./commands/info"; import { type SubcommandName } from "./commands/names"; import { onboardCommand } from "./commands/onboard"; +import { reportCommand } from "./commands/report"; import { ruleCommand } from "./commands/rules"; import { shareCommand } from "./commands/share"; import { @@ -56,6 +57,7 @@ const subCommands = { feedback: feedbackCommand, auth: authCommand, onboard: onboardCommand, + report: reportCommand, rule: ruleCommand, share: shareCommand, validate: validateCommand, diff --git a/packages/cli/src/prompts/index.ts b/packages/cli/src/prompts/index.ts index 8e8946c0..2d27f10f 100644 --- a/packages/cli/src/prompts/index.ts +++ b/packages/cli/src/prompts/index.ts @@ -104,6 +104,7 @@ export const INTERNAL_TOPICS = [ "init", "onboard", "recover-rule", + "report", "rule", "rule-feedback", "rule-meta", diff --git a/packages/cli/src/prompts/recipes.ts b/packages/cli/src/prompts/recipes.ts index c8496bb2..946674b8 100644 --- a/packages/cli/src/prompts/recipes.ts +++ b/packages/cli/src/prompts/recipes.ts @@ -13,6 +13,7 @@ import { generalInputSchema, ruleInputSchema, } from "../schemas/feedback"; +import { reportFactsSchema } from "../schemas/report-facts"; import { AST_GREP_VERSION, VALE_VERSION, @@ -89,6 +90,7 @@ const TOPIC_INPUT_SCHEMAS: Record = { "rule-feedback": ruleInputSchema, feedback: generalInputSchema, "bug-report": bugInputSchema, + report: reportFactsSchema, }; /** Agent-fill marker used when the caller does not supply a real value. */ diff --git a/packages/cli/src/report/assemble.ts b/packages/cli/src/report/assemble.ts new file mode 100644 index 00000000..3d03c94d --- /dev/null +++ b/packages/cli/src/report/assemble.ts @@ -0,0 +1,358 @@ +import { CLIError } from "../util/cli-error"; +import type { ReportDraft } from "./draft"; +import type { ReportFacts } from "../schemas/report-facts"; +import { + REDACTION_POLICY_VERSION, + hasHits, + scanText, + type PatternKind, +} from "./redact"; +import { REPORT_FIXTURE_MAX_LENGTH } from "./schema"; +import type { + PerTier, + RedactionKind, + ReportRule, + ReportScope, + ReportTests, + ReportV1, + TestStatus, + Tier, +} from "./types"; + +/** + * Build the report from the CLI's draft and the agent's facts. + * + * The CLI's values win every conflict: the facts carry only what the agent + * alone knows, and a fact about something the CLI measured (an engine, a test + * status) never reaches the payload. The projection is computed here and + * nowhere else. + */ + +/** How many Taskless rules a report needs before it is worth publishing. */ +export const MINIMUM_TASKLESS_RULES = 3; + +/** A Taskless rule counts toward the gate unless its tests say it is broken. */ +const DISQUALIFYING_STATUSES: ReadonlySet = new Set([ + "fail", + "refused", +]); + +const ZERO_COUNTS: Record = { + secret: 0, + email: 0, + person: 0, + customer: 0, + host: 0, + ip: 0, + phone: 0, + text: 0, +}; + +interface Candidate { + rule: Omit; + status: TestStatus; + fail: string | null; + pass: string | null; +} + +function invalid(message: string): CLIError { + return new CLIError(message, "REPORT_INVALID"); +} + +/** Pair each draft rule with its facts, and add the legacy and cloud rules. */ +function mergeRules(draft: ReportDraft, facts: ReportFacts): Candidate[] { + const factsById = new Map(facts.rules.map((rule) => [rule.id, rule])); + const localIds = new Set(draft.rules.map((rule) => rule.id)); + + const unknown = facts.rules + .map((rule) => rule.id) + .filter((id) => !localIds.has(id)); + if (unknown.length > 0) { + throw invalid( + `facts name rules that are not under .taskless/rules/: ${unknown.join(", ")}. List a rule written into another linter under "legacy", and a candidate you did not write under "cloud".` + ); + } + const uncovered = draft.rules + .map((rule) => rule.id) + .filter((id) => !factsById.has(id)); + if (uncovered.length > 0) { + throw invalid( + `facts are missing for these rules: ${uncovered.join(", ")}. Every rule \`taskless report\` lists needs a "rules" entry with its title, origin, and fix count.` + ); + } + + const candidates: Candidate[] = draft.rules.map((local) => { + // Checked above: every draft rule has facts. + const ruleFacts = factsById.get(local.id)!; + return { + rule: { + id: local.id, + title: ruleFacts.title, + tier: local.tier, + engine: local.engine, + language: local.language, + origin: ruleFacts.origin, + fixed: ruleFacts.fixed, + fixedBasis: ruleFacts.fixedBasis, + complexity: null, + }, + status: local.tests.status, + fail: local.tests.fail, + pass: local.tests.pass, + }; + }); + + for (const legacy of facts.legacy) { + candidates.push({ + rule: { + id: legacy.id, + title: legacy.title, + tier: "legacy", + tool: legacy.tool, + language: legacy.language, + origin: legacy.origin, + fixed: legacy.fixed, + fixedBasis: legacy.fixedBasis, + complexity: null, + }, + // The CLI cannot run another linter's rule, so it does not claim it + // passed. Agent-written legacy rules still count toward the gate. + status: "not-run", + fail: legacy.fail ?? null, + pass: legacy.pass ?? null, + }); + } + + for (const cloud of facts.cloud) { + candidates.push({ + rule: { + id: cloud.id, + title: cloud.title, + tier: "cloud", + engine: "runtime", + language: cloud.language, + origin: cloud.origin, + fixed: cloud.fixed, + fixedBasis: cloud.fixedBasis, + complexity: cloud.complexity, + }, + status: "not-run", + fail: cloud.fail ?? null, + pass: cloud.pass ?? null, + }); + } + + const seen = new Set(); + const duplicates = new Set(); + for (const { rule } of candidates) { + if (seen.has(rule.id)) duplicates.add(rule.id); + seen.add(rule.id); + } + if (duplicates.size > 0) { + throw invalid( + `rule ids must be unique across the report: ${[...duplicates].join(", ")}` + ); + } + return candidates; +} + +/** Refuse a report with fewer than three working Taskless rules. */ +function enforceGate(candidates: Candidate[]): void { + const counted = candidates.filter( + ({ rule, status }) => + (rule.tier === "cli" || rule.tier === "legacy") && + !DISQUALIFYING_STATUSES.has(status) + ).length; + if (counted < MINIMUM_TASKLESS_RULES) { + throw new CLIError( + `A report needs at least ${String(MINIMUM_TASKLESS_RULES)} working Taskless rules, and this repository has ${String(counted)} of ${String(MINIMUM_TASKLESS_RULES)}. Rules under .taskless/rules/ whose tests fail do not count, and cloud candidates are carried in addition rather than counted.`, + "REPORT_INSUFFICIENT" + ); + } +} + +/** + * Refuse any agent-written string a pattern matches. The CLI cannot rewrite + * prose, and the agent can, so it is told which field to fix. + */ +function screenStrings(facts: ReportFacts, label: string | undefined): void { + const fields: [string, string][] = []; + if (label !== undefined) fields.push(["label", label]); + // Every agent-supplied string mergeRules copies onto a published rule. Ids + // count: the schema allows dots, so an id can be a host name. + for (const group of ["rules", "legacy", "cloud"] as const) { + for (const rule of facts[group]) { + fields.push( + [`${group}[${rule.id}].id`, rule.id], + [`${group}[${rule.id}].title`, rule.title] + ); + } + } + for (const rule of facts.legacy) { + fields.push( + [`legacy[${rule.id}].tool`, rule.tool], + [`legacy[${rule.id}].language`, rule.language] + ); + } + for (const rule of facts.cloud) { + fields.push([`cloud[${rule.id}].language`, rule.language]); + } + const flagged = fields + .filter(([, text]) => hasHits(scanText(text))) + .map(([field]) => field); + if (flagged.length > 0) { + throw new CLIError( + `These fields hold something that looks like a secret, an email address, a host, an IP address, or a phone number, and tskl.es does not redact: ${flagged.join(", ")}. Rewrite them without it.`, + "REPORT_REDACTION" + ); + } +} + +interface RedactionTally { + counts: Record; + withheldFixtures: number; +} + +/** Decide each fixture: publish it, or withhold it whole. */ +function redactFixtures( + candidates: Candidate[], + facts: ReportFacts, + tally: RedactionTally +): ReportRule[] { + const agentWithheld = new Set( + facts.redaction.withheld.map(({ id, side }) => `${id}:${side}`) + ); + + return candidates.map(({ rule, status, fail, pass }) => { + const tests: ReportTests = { status, fail: null, pass: null, withheld: [] }; + for (const [side, code] of [ + ["fail", fail], + ["pass", pass], + ] as const) { + if (code === null) continue; + const hits = scanText(code); + for (const [kind, n] of Object.entries(hits) as [PatternKind, number][]) { + tally.counts[kind] += n; + } + const withhold = + agentWithheld.has(`${rule.id}:${side}`) || + hasHits(hits) || + code.length > REPORT_FIXTURE_MAX_LENGTH; + if (withhold) { + tests.withheld.push(side); + tally.withheldFixtures += 1; + } else { + tests[side] = { code }; + } + } + return { ...rule, tests }; + }); +} + +/** Days in an inclusive window, at least one. */ +function windowDays(window: ReportFacts["window"]): number { + const from = Date.parse(`${window.from}T00:00:00Z`); + const to = Date.parse(`${window.to}T00:00:00Z`); + if (to < from) { + throw invalid( + `window.to (${window.to}) is before window.from (${window.from})` + ); + } + return Math.round((to - from) / 86_400_000) + 1; +} + +/** Sum a per-rule figure by tier, normalized to a 30-day month. */ +function perMonth( + rules: ReportRule[], + days: number, + pick: (rule: ReportRule) => number +): PerTier { + const totals: Record = { cli: 0, cloud: 0, legacy: 0 }; + for (const rule of rules) totals[rule.tier] += pick(rule); + const scale = (n: number) => Math.round((n * 30) / days); + return { + cli: scale(totals.cli), + cloud: scale(totals.cloud), + ...(totals.legacy > 0 ? { legacy: scale(totals.legacy) } : {}), + }; +} + +function buildScope(draft: ReportDraft, facts: ReportFacts): ReportScope { + const window = { from: facts.window.from, to: facts.window.to }; + const { scope } = draft; + if ( + scope.provider === "github" && + scope.owner !== undefined && + scope.repo !== undefined && + scope.url !== undefined + ) { + return { + provider: "github", + owner: scope.owner, + repo: scope.repo, + url: scope.url, + // Always unlisted. Public retention is a separate decision with its own + // consent, and this command does not offer it. + retention: "private", + window, + }; + } + return { + provider: "none", + label: facts.label ?? scope.label ?? "Repository", + window, + }; +} + +export function assembleReport( + draft: ReportDraft, + facts: ReportFacts, + now: Date = new Date() +): ReportV1 { + const candidates = mergeRules(draft, facts); + enforceGate(candidates); + + const scope = buildScope(draft, facts); + screenStrings(facts, scope.provider === "none" ? scope.label : undefined); + + const tally: RedactionTally = { + counts: { ...ZERO_COUNTS }, + withheldFixtures: 0, + }; + for (const [kind, n] of Object.entries(facts.redaction.counts)) { + tally.counts[kind as RedactionKind] += n; + } + const rules = redactFixtures(candidates, facts, tally); + + const days = windowDays(facts.window); + return { + schemaVersion: "1", + // Seconds precision: the schema's date-time pattern allows fractions, but a + // report has no use for them. + generatedAt: `${now.toISOString().slice(0, 19)}Z`, + producer: draft.producer, + scope, + history: facts.history, + rules, + projection: { + perMonth: { + commentsPrevented: perMonth( + rules, + days, + (rule) => rule.origin?.reviewComments ?? 0 + ), + agentCatches: perMonth(rules, days, (rule) => rule.fixed), + }, + inputs: { + contextTokens: facts.estimates.contextTokens, + callsPerCatch: facts.estimates.callsPerCatch, + ...draft.inputs, + }, + }, + redaction: { + policyVersion: REDACTION_POLICY_VERSION, + counts: tally.counts, + withheldFixtures: tally.withheldFixtures, + }, + }; +} diff --git a/packages/cli/src/report/draft.ts b/packages/cli/src/report/draft.ts new file mode 100644 index 00000000..504bd2f6 --- /dev/null +++ b/packages/cli/src/report/draft.ts @@ -0,0 +1,240 @@ +import { readFile, readdir } from "node:fs/promises"; +import { basename, join } from "node:path"; + +import { parseDocument } from "yaml"; + +import { + listRuleIds, + ruleFilePath, + ruleTestsDirectory, +} from "../rules/engines"; +import { isMissingDirectory } from "../rules/errno"; +import { verifyOneRule } from "../rules/inspect"; +import { resolveRepositoryContext } from "../util/git-remote"; +import { CLI_VERSION } from "../version"; +import type { ProjectionInputs, TestStatus } from "./types"; + +/** + * Everything about a report the CLI can establish on its own, without the + * network and without the agent. + * + * `taskless report` prints this, and `taskless report --from` builds on it, so + * the two can never disagree about what a rule's engine, language or test + * status is: the CLI's answer is the one published, whatever the facts say. + */ + +/** A rule under `.taskless/rules/`, as the CLI sees it. */ +export interface LocalRule { + id: string; + engine: "sg" | "vale" | "runtime"; + tier: "cli" | "cloud"; + language: string; + tests: { + status: TestStatus; + /** First fixture the rule flags, or null when it has none. */ + fail: string | null; + /** First fixture the rule accepts, or null when it has none. */ + pass: string | null; + }; +} + +export interface DraftScope { + provider: "github" | "none"; + owner?: string; + repo?: string; + url?: string; + /** For `provider: "none"`: the default label, the directory name. */ + label?: string; +} + +export interface ReportDraft { + producer: { kind: "cli"; cliVersion: string }; + scope: DraftScope; + rules: LocalRule[]; + inputs: Omit; + /** The facts the agent must still supply, by facts-file key. */ + missing: string[]; +} + +/** + * The projection inputs the agent does not estimate. + * + * The values tskl.es's own example uses. The page exposes every input as a + * slider, so these are starting points a reader adjusts, not claims. + */ +export const DEFAULT_PROJECTION_INPUTS: ReportDraft["inputs"] = { + cloudComplexity: 3, + pricePerMillionTokens: 0.5, + minutesPerComment: 4, + reviewerHourlyUsd: 120, +}; + +/** + * What a Vale rule's language is reported as. Vale lints prose, not a + * programming language, and the schema requires a language for every rule. + */ +const VALE_LANGUAGE = "Prose"; + +/** What a runtime rule's checks are written in. */ +const RUNTIME_LANGUAGE = "TypeScript"; + +async function readFirstFile(directory: string): Promise { + let entries; + try { + entries = await readdir(directory, { withFileTypes: true }); + } catch (error) { + if (isMissingDirectory(error)) return null; + throw error; + } + const first = entries + .filter((entry) => entry.isFile()) + .map((entry) => entry.name) + .toSorted((a, b) => a.localeCompare(b))[0]; + if (first === undefined) return null; + const raw = await readFile(join(directory, first), "utf8"); + const content = raw.trim(); + return content === "" ? null : content; +} + +/** The first `invalid:` and `valid:` case across an sg rule's test files. */ +async function sgFixtures( + cwd: string, + id: string +): Promise<{ fail: string | null; pass: string | null }> { + const directory = ruleTestsDirectory(cwd, "sg", id); + let entries: string[]; + try { + entries = await readdir(directory); + } catch (error) { + if (isMissingDirectory(error)) return { fail: null, pass: null }; + throw error; + } + const names = entries + .filter((name) => /\.ya?ml$/.test(name)) + .toSorted((a, b) => a.localeCompare(b)); + let fail: string | null = null; + let pass: string | null = null; + for (const name of names) { + const source = await readFile(join(directory, name), "utf8"); + const parsed: unknown = parseDocument(source).toJS(); + if (typeof parsed !== "object" || parsed === null) continue; + const document = parsed as { valid?: unknown; invalid?: unknown }; + fail ??= firstString(document.invalid); + pass ??= firstString(document.valid); + } + return { fail, pass }; +} + +function firstString(value: unknown): string | null { + if (!Array.isArray(value)) return null; + const first: unknown = value.find( + (item) => typeof item === "string" && item.trim() !== "" + ); + return typeof first === "string" ? first.trim() : null; +} + +/** An sg rule's `language:`, as ast-grep spells it. */ +async function sgLanguage(cwd: string, id: string): Promise { + try { + const source = await readFile(ruleFilePath(cwd, "sg", id), "utf8"); + const parsed: unknown = parseDocument(source).toJS(); + const language = (parsed as { language?: unknown } | null)?.language; + if (typeof language === "string" && language !== "") return language; + } catch { + // An unreadable rule is `validate`'s finding to report. Here it only + // costs the language, which falls back below. + } + return "Unknown"; +} + +async function testStatus( + cwd: string, + engine: "sg" | "vale", + id: string +): Promise { + const verification = await verifyOneRule(cwd, { engine, ruleId: id }); + if (verification.refused !== undefined) return "refused"; + return verification.ok && verification.ran ? "pass" : "fail"; +} + +/** Every rule under `.taskless/rules/`, with its tests run. */ +export async function collectLocalRules(cwd: string): Promise { + const rules: LocalRule[] = []; + for (const id of await listRuleIds(cwd, "sg")) { + rules.push({ + id, + engine: "sg", + tier: "cli", + language: await sgLanguage(cwd, id), + tests: { + status: await testStatus(cwd, "sg", id), + ...(await sgFixtures(cwd, id)), + }, + }); + } + for (const id of await listRuleIds(cwd, "vale")) { + const tests = ruleTestsDirectory(cwd, "vale", id); + rules.push({ + id, + engine: "vale", + tier: "cli", + language: VALE_LANGUAGE, + tests: { + status: await testStatus(cwd, "vale", id), + fail: await readFirstFile(join(tests, "fail")), + pass: await readFirstFile(join(tests, "pass")), + }, + }); + } + // A runtime rule runs only where Taskless Cloud blessed it, and a report is + // published logged out, so it is reported as the cloud tier, untested. + for (const id of await listRuleIds(cwd, "runtime")) { + rules.push({ + id, + engine: "runtime", + tier: "cloud", + language: RUNTIME_LANGUAGE, + tests: { status: "not-run", fail: null, pass: null }, + }); + } + return rules; +} + +export async function resolveDraftScope(cwd: string): Promise { + const { repositoryUrl } = await resolveRepositoryContext(cwd); + if (repositoryUrl !== null) { + // `resolveRepositoryContext` returns the canonical + // `https://github.com/{owner}/{repo}`, so both are positional. + const segments = repositoryUrl.split("/"); + const owner = segments[3]; + const repo = segments[4]; + if (owner !== undefined && repo !== undefined) { + return { provider: "github", owner, repo, url: repositoryUrl }; + } + } + return { provider: "none", label: basename(cwd) }; +} + +/** The facts-file keys the agent must supply for this draft. */ +function missingFacts(rules: LocalRule[]): string[] { + return [ + "window", + "history", + "estimates", + ...rules.map((rule) => `rules[${rule.id}]`), + ]; +} + +export async function buildDraft(cwd: string): Promise { + const [scope, rules] = await Promise.all([ + resolveDraftScope(cwd), + collectLocalRules(cwd), + ]); + return { + producer: { kind: "cli", cliVersion: CLI_VERSION }, + scope, + rules, + inputs: DEFAULT_PROJECTION_INPUTS, + missing: missingFacts(rules), + }; +} diff --git a/packages/cli/src/report/publish.ts b/packages/cli/src/report/publish.ts new file mode 100644 index 00000000..9f0d794f --- /dev/null +++ b/packages/cli/src/report/publish.ts @@ -0,0 +1,167 @@ +import process from "node:process"; + +import { CLIError } from "../util/cli-error"; +import { REPORT_MAX_BYTES, REPORT_PUBLISH_URL } from "./schema"; +import type { ReportV1 } from "./types"; + +/** + * The tskl.es client: publish, and delete. + * + * The retry contract is tskl.es's, from the schema's `x-taskless` block: retry + * only `publish_failed` (502), which promises nothing was published, and never + * after a timeout, which may still publish. A rate limit is surfaced rather + * than retried, because the limit is per address and a loop only extends it. + */ + +/** What tskl.es returns on a 201. */ +export interface PublishedReport { + url: string; + deleteUrl: string; + expiresAt: string; +} + +/** Attempts on a 502, the first included. */ +const MAX_ATTEMPTS = 3; + +const TIMEOUT_MS = 30_000; + +/** The publish endpoint, overridable so tests can stand up their own. */ +function publishUrl(): string { + return process.env.TASKLESS_REPORT_URL ?? REPORT_PUBLISH_URL; +} + +function backoffMs(attempt: number): number { + return process.env.TASKLESS_REPORT_BACKOFF_MS === undefined + ? 1000 * attempt + : Number(process.env.TASKLESS_REPORT_BACKOFF_MS); +} + +/** The body as text, cut to something a terminal can show. */ +async function bodyText(response: Response): Promise { + const text = await response.text().catch(() => ""); + return text.length > 2000 ? `${text.slice(0, 2000)}…` : text; +} + +/** Throws `REPORT_TOO_LARGE` when the body would exceed tskl.es's limit. */ +export function encodeReport(report: ReportV1): string { + const body = JSON.stringify(report); + const bytes = Buffer.byteLength(body, "utf8"); + if (bytes > REPORT_MAX_BYTES) { + throw new CLIError( + `The report is ${String(bytes)} bytes and tskl.es accepts at most ${String(REPORT_MAX_BYTES)}. Withhold long fixtures or drop cloud candidates, then retry.`, + "REPORT_TOO_LARGE" + ); + } + return body; +} + +function isPublished(value: unknown): value is PublishedReport { + if (typeof value !== "object" || value === null) return false; + const record = value as Record; + return ( + typeof record.url === "string" && + typeof record.deleteUrl === "string" && + typeof record.expiresAt === "string" + ); +} + +export async function publishReport(body: string): Promise { + for (let attempt = 1; ; attempt += 1) { + let response: Response; + try { + response = await fetch(publishUrl(), { + method: "POST", + headers: { "content-type": "application/json" }, + body, + signal: AbortSignal.timeout(TIMEOUT_MS), + }); + } catch (error) { + // A timeout or a dropped connection: the request may have arrived, so + // the report may exist. Retrying could publish it twice. + const reason = error instanceof Error ? error.message : String(error); + throw new CLIError( + `Publishing did not complete (${reason}), and a report may have been published anyway. Re-running may create a duplicate; an unclaimed report expires on its own in 30 days.`, + "REPORT_PUBLISH_UNKNOWN" + ); + } + + if (response.status === 201) { + let parsed: unknown; + try { + parsed = await response.json(); + } catch { + parsed = null; + } + if (!isPublished(parsed)) { + throw new CLIError( + "tskl.es accepted the report but did not return its link and delete URL. Nothing was recorded.", + "REPORT_PUBLISH_UNKNOWN" + ); + } + return parsed; + } + if (response.status === 502 && attempt < MAX_ATTEMPTS) { + await new Promise((resolve) => setTimeout(resolve, backoffMs(attempt))); + continue; + } + if (response.status === 502) { + throw new CLIError( + `tskl.es could not store the report after ${String(MAX_ATTEMPTS)} attempts. Nothing was published; retry later.`, + "REPORT_PUBLISH_FAILED" + ); + } + if (response.status === 429) { + throw new CLIError( + "tskl.es is rate limiting publishes from this address. Wait a minute, then retry.", + "REPORT_RATE_LIMITED" + ); + } + if (response.status === 400 || response.status === 413) { + throw new CLIError( + `tskl.es refused the report: ${await bodyText(response)}`, + "REPORT_INVALID" + ); + } + throw new CLIError( + `tskl.es answered ${String(response.status)}: ${await bodyText(response)}`, + "REPORT_PUBLISH_FAILED" + ); + } +} + +/** + * Delete a published report through the URL tskl.es returned for it. + * + * The contract is the schema's `x-taskless.delete`: a `DELETE` to that URL + * answers 204 on success, or 404 `not_available` when no report answers to it + * (never issued, already deleted, expired, or not yet propagated), which is + * not retryable. A 404 is its own code because its remedy differs from a + * failure's: retrying cannot help, and the record may simply be stale. + */ +export async function deletePublishedReport(deleteUrl: string): Promise { + let response: Response; + try { + response = await fetch(deleteUrl, { + method: "DELETE", + signal: AbortSignal.timeout(TIMEOUT_MS), + }); + } catch (error) { + const reason = error instanceof Error ? error.message : String(error); + throw new CLIError( + `Could not reach tskl.es to delete the report: ${reason}`, + "NETWORK_ERROR" + ); + } + if (response.status === 404) { + throw new CLIError( + "tskl.es has no report for this delete URL: it was already deleted, has expired, or was published moments ago and has not propagated yet. Retrying will not help; the record was kept, and can be removed by hand once you are sure the report is gone.", + "REPORT_NOT_AVAILABLE" + ); + } + if (!response.ok) { + throw new CLIError( + `tskl.es did not delete the report (${String(response.status)}): ${await bodyText(response)}`, + "REPORT_DELETE_FAILED" + ); + } +} diff --git a/packages/cli/src/report/records.ts b/packages/cli/src/report/records.ts new file mode 100644 index 00000000..bd9b595a --- /dev/null +++ b/packages/cli/src/report/records.ts @@ -0,0 +1,113 @@ +import { mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import { basename, dirname, join, resolve } from "node:path"; + +import { CLIError } from "../util/cli-error"; +import type { PublishedReport } from "./publish"; + +/** + * The local record of a published report: `.taskless/reports/.json`. + * + * It holds the delete URL tskl.es returns exactly once, which is the only way + * to take the page down before it expires. That is why the directory is + * gitignored (migration 11) and why nothing here prints the delete URL: a + * credential that reaches a terminal reaches scrollback, logs and screenshots. + */ + +export const REPORTS_DIRECTORY = "reports"; + +export function reportsDirectory(cwd: string): string { + return join(cwd, ".taskless", REPORTS_DIRECTORY); +} + +/** `YYYY-MM-DD_HHMMSS`, UTC, so records sort by when they were published. */ +export function recordStamp(now: Date): string { + const iso = now.toISOString(); + return `${iso.slice(0, 10)}_${iso.slice(11, 19).replaceAll(":", "")}`; +} + +/** Write the record and return its file name. */ +export async function writeRecord( + cwd: string, + published: PublishedReport, + now: Date +): Promise { + const directory = reportsDirectory(cwd); + await mkdir(directory, { recursive: true }); + const name = `${recordStamp(now)}.json`; + await writeFile( + join(directory, name), + `${JSON.stringify( + { + url: published.url, + deleteUrl: published.deleteUrl, + expiresAt: published.expiresAt, + }, + null, + 2 + )}\n`, + { encoding: "utf8", flag: "wx" } + ); + return name; +} + +/** + * A record named by file name, or by a path, absolute or relative to the + * project, that lands directly in `.taskless/reports/`. Anything else is + * refused: a successful delete removes the file, and `--delete` has no + * business removing one outside the records directory. + */ +function recordPath(cwd: string, record: string): string { + const directory = resolve(reportsDirectory(cwd)); + const path = + record === basename(record) + ? join(directory, record) + : resolve(cwd, record); + if (dirname(path) !== directory || !path.endsWith(".json")) { + throw new CLIError( + `${record} is not a report record. Name a file in .taskless/reports/, e.g. 2026-10-13_142200.json.`, + "INVALID_INPUT" + ); + } + return path; +} + +export async function readRecord( + cwd: string, + record: string +): Promise<{ path: string; published: PublishedReport }> { + const path = recordPath(cwd, record); + let raw: string; + try { + raw = await readFile(path, "utf8"); + } catch { + throw new CLIError( + `No report record at ${path}. Records live in .taskless/reports/.`, + "REPORT_NOT_FOUND" + ); + } + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch { + throw new CLIError(`${path} is not valid JSON.`, "INVALID_INPUT"); + } + const value = parsed as Partial | null; + if (typeof value?.deleteUrl !== "string" || typeof value.url !== "string") { + throw new CLIError( + `${path} has no delete URL, so the report cannot be deleted from here.`, + "INVALID_INPUT" + ); + } + return { + path, + published: { + url: value.url, + deleteUrl: value.deleteUrl, + expiresAt: typeof value.expiresAt === "string" ? value.expiresAt : "", + }, + }; +} + +export async function removeRecord(path: string): Promise { + await rm(path); +} diff --git a/packages/cli/src/report/redact.ts b/packages/cli/src/report/redact.ts new file mode 100644 index 00000000..fd3f454c --- /dev/null +++ b/packages/cli/src/report/redact.ts @@ -0,0 +1,104 @@ +import type { RedactionKind } from "./types"; + +/** + * The CLI's redaction pass over everything a report would publish. + * + * tskl.es performs no redaction of its own, so what this misses is published. + * Two rules follow from that: + * + * - **Withhold, never mask.** A fixture that matches is dropped whole. A + * half-masked snippet reads as broken code on a page meant to show a rule + * working, and a masked secret still leaks its shape and length. + * - **Patterns only for what a pattern can know.** Secrets, email addresses, + * host names, IP addresses and phone numbers have a shape. People's and + * customers' names do not, so those are the agent's to judge and declare. + * + * Bump {@link REDACTION_POLICY_VERSION} whenever a pattern changes, so a + * published report says which pass it went through. + */ + +export const REDACTION_POLICY_VERSION = "1"; + +/** The kinds this pass can detect. */ +export type PatternKind = Extract< + RedactionKind, + "secret" | "email" | "host" | "ip" | "phone" +>; + +const SECRET_PATTERNS: readonly RegExp[] = [ + /\bAKIA[\dA-Z]{16}\b/g, + /\bgh[opsur]_[\dA-Za-z]{36,}\b/g, + /\bgithub_pat_\w{40,}\b/g, + /\bxox[abprs]-[\dA-Za-z-]{10,}\b/g, + /\b(?:sk|rk)_live_[\dA-Za-z]{16,}\b/g, + /\bAIza[\w-]{35}\b/g, + /\beyJ[\w-]{10,}\.[\w-]{10,}\.[\w-]{10,}\b/g, + /-----BEGIN [A-Z ]*PRIVATE KEY-----/g, + /\b(?:api[_-]?key|secret|token|passw(?:or)?d)\s*[:=]\s*["'][^"'\s]{8,}["']/gi, +]; + +const EMAIL = /\b[\w.%+-]+@[\dA-Za-z.-]+\.[A-Za-z]{2,}\b/g; + +/** + * A dotted host under a TLD a real or internal host is likely to use. Narrow + * on purpose: code is full of `a.b` member access, and a pattern loose enough + * to catch every host would withhold every fixture. `.app`, `.dev`, `.ai` and + * `.co` are real TLDs left out for that reason: `this.app` and `config.dev` + * are code, and withholding them would cost more fixtures than it protects. + */ +const HOST = + /\b(?:[\dA-Za-z-]+\.)+(?:com|net|org|io|cloud|internal|corp|local|lan)\b/gi; + +/** Names that are hosts in form but documentation by definition. */ +const RESERVED_HOST = /(?:^|\.)(?:example\.(?:com|net|org)|localhost)$/i; + +const IPV4 = + /\b(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\b/g; + +/** Loopback and unspecified addresses identify nothing. */ +const HARMLESS_IP = new Set(["127.0.0.1", "0.0.0.0"]); + +/** Requires separators, so a long number in code is not read as a phone. */ +const PHONE = /(?:\+\d{1,3}[\s.-])?\(?\b\d{3}\)?[\s.-]\d{3}[\s.-]\d{4}\b/g; + +function countMatches(pattern: RegExp, text: string): string[] { + return [...text.matchAll(pattern)].map((match) => match[0]); +} + +/** Every pattern hit in `text`, by kind. A kind with no hit is absent. */ +export function scanText(text: string): Partial> { + const hits: Partial> = {}; + const add = (kind: PatternKind, n: number) => { + if (n > 0) hits[kind] = (hits[kind] ?? 0) + n; + }; + + for (const pattern of SECRET_PATTERNS) { + add("secret", countMatches(pattern, text).length); + } + + const emails = countMatches(EMAIL, text); + add("email", emails.length); + // An email's domain is a host, but it has already been counted once. + let withoutEmails = text; + for (const email of emails) { + withoutEmails = withoutEmails.replaceAll(email, " "); + } + + add( + "host", + countMatches(HOST, withoutEmails).filter( + (host) => !RESERVED_HOST.test(host) + ).length + ); + add( + "ip", + countMatches(IPV4, text).filter((ip) => !HARMLESS_IP.has(ip)).length + ); + add("phone", countMatches(PHONE, text).length); + return hits; +} + +/** Whether `text` has any hit at all. */ +export function hasHits(hits: Partial>): boolean { + return Object.keys(hits).length > 0; +} diff --git a/packages/cli/src/report/schema.ts b/packages/cli/src/report/schema.ts new file mode 100644 index 00000000..589992c3 --- /dev/null +++ b/packages/cli/src/report/schema.ts @@ -0,0 +1,65 @@ +import { z } from "zod"; + +import reportSchemaDocument from "../generated/report-v1.schema.json"; + +/** + * ReportV1, as tskl.es publishes it, and the publish contract it carries. + * + * The vendored document is the single source for both. Validation is built + * from it with `z.fromJSONSchema()` rather than hand-written, so a refresh of + * the document (`pnpm --filter @taskless/cli generate:report-schema`) is the + * whole change when tskl.es moves, and a payload this CLI accepts is one + * tskl.es's own schema accepts. The publish URL, the size limit and the error + * vocabulary are read from its `x-taskless` block for the same reason. + */ + +const validator = z.fromJSONSchema( + reportSchemaDocument as unknown as Parameters[0] +); + +const publishContract = ( + reportSchemaDocument as unknown as { + "x-taskless": { publish: { url: string; maxBytes: number } }; + } +)["x-taskless"].publish; + +/** Where a report is published, unless overridden for tests. */ +export const REPORT_PUBLISH_URL: string = publishContract.url; + +/** The largest body tskl.es reads, in bytes. */ +export const REPORT_MAX_BYTES: number = publishContract.maxBytes; + +/** + * The longest fixture tskl.es accepts, read from the schema's own limit on + * `rules[].tests.fail.code`, so a refreshed limit needs no edit here. + */ +export const REPORT_FIXTURE_MAX_LENGTH: number = ( + reportSchemaDocument as unknown as { + properties: { + rules: { + items: { + properties: { + tests: { + properties: { + fail: { + anyOf: [{ properties: { code: { maxLength: number } } }]; + }; + }; + }; + }; + }; + }; + }; + } +).properties.rules.items.properties.tests.properties.fail.anyOf[0].properties + .code.maxLength; + +/** Every way a payload fails the vendored schema, one `path: message` each. */ +export function reportSchemaIssues(payload: unknown): string[] { + const result = validator.safeParse(payload); + if (result.success) return []; + return result.error.issues.map( + (issue) => + `${issue.path.length > 0 ? issue.path.join(".") : "(root)"}: ${issue.message}` + ); +} diff --git a/packages/cli/src/report/summary.ts b/packages/cli/src/report/summary.ts new file mode 100644 index 00000000..ba582876 --- /dev/null +++ b/packages/cli/src/report/summary.ts @@ -0,0 +1,56 @@ +import type { PerTier, ReportRule, ReportV1 } from "./types"; + +/** + * The brief rule summary printed after a publish. + * + * A CLI template rather than prose the agent writes, so every report reads the + * same and the recipe only has to say "show it". The page on tskl.es carries + * the visual treatment; this is the companion a terminal can hold. + * + * The delete URL is deliberately absent. The summary names the record the CLI + * can delete from, never the credential in it. + */ + +function tierLabel(rule: ReportRule): string { + switch (rule.tier) { + case "cli": { + return `Taskless ${rule.engine === "vale" ? "Vale" : "ast-grep"} rule`; + } + case "legacy": { + return `rule in ${rule.tool ?? "your linter"}`; + } + case "cloud": { + return "Taskless Cloud candidate"; + } + } +} + +function total(perTier: PerTier): number { + return perTier.cli + perTier.cloud + (perTier.legacy ?? 0); +} + +export function renderSummary(options: { + report: ReportV1; + url: string; + expiresAt: string; + record: string; + invocation: string; +}): string { + const { report, url, expiresAt, record, invocation } = options; + const { perMonth } = report.projection; + const lines = [ + "Taskless report published", + "", + `Rules (${String(report.rules.length)}):`, + ...report.rules.map((rule) => ` - ${rule.title} (${tierLabel(rule)})`), + "", + `Projected each month: ${String(total(perMonth.commentsPrevented))} review comments prevented, ${String(total(perMonth.agentCatches))} catches before review.`, + "", + `View it: ${url}`, + `Expires: ${expiresAt.slice(0, 10)}`, + "", + "The report is unlisted: anyone with the link can view it.", + `It can be deleted after review. Ask, or run \`${invocation} report --delete ${record}\`.`, + ]; + return lines.join("\n"); +} diff --git a/packages/cli/src/report/types.ts b/packages/cli/src/report/types.ts new file mode 100644 index 00000000..0a1d9fdc --- /dev/null +++ b/packages/cli/src/report/types.ts @@ -0,0 +1,98 @@ +/** + * The ReportV1 payload, as this CLI builds it. + * + * Declared by hand because `z.fromJSONSchema()` validates without inferring a + * static type. The vendored schema stays the authority: every payload is + * validated against it before it is sent, and the tests validate payloads + * built from real rule directories, so a type that drifts from the schema + * fails there rather than at tskl.es. + */ + +export type Tier = "cli" | "cloud" | "legacy"; + +export type TestStatus = "pass" | "fail" | "refused" | "not-run"; + +export interface ReportTests { + status: TestStatus; + fail: { code: string } | null; + pass: { code: string } | null; + withheld: ("fail" | "pass")[]; +} + +export interface ReportRule { + id: string; + title: string; + tier: Tier; + engine?: "sg" | "vale" | "runtime"; + tool?: string; + language: string; + origin?: { reviewComments: number; prs: number }; + fixed: number; + fixedBasis: "measured" | "estimated"; + complexity: number | null; + tests: ReportTests; +} + +export interface ReportWindow { + from: string; + to: string; +} + +export type ReportScope = + | { + provider: "github"; + owner: string; + repo: string; + url: string; + retention: "private"; + window: ReportWindow; + } + | { provider: "none"; label: string; window: ReportWindow }; + +export interface PerTier { + cli: number; + cloud: number; + legacy?: number; +} + +export interface ProjectionInputs { + contextTokens: number; + callsPerCatch: number; + cloudComplexity: number; + pricePerMillionTokens: number; + minutesPerComment: number; + reviewerHourlyUsd: number; +} + +export type RedactionKind = + | "secret" + | "email" + | "person" + | "customer" + | "host" + | "ip" + | "phone" + | "text"; + +export interface ReportV1 { + schemaVersion: "1"; + generatedAt: string; + producer: { kind: "cli"; cliVersion: string }; + scope: ReportScope; + history?: { + mergedPrs: number; + reviewComments: number; + agentAssistedPrs: number; + patterns: { ruleShaped: number; judgment: number }; + }; + rules: ReportRule[]; + projection: { + perMonth: { commentsPrevented: PerTier; agentCatches: PerTier }; + inputs: ProjectionInputs; + }; + redaction: { + policyVersion: string; + counts: Record; + withheldFixtures: number; + }; +} diff --git a/packages/cli/src/schemas/report-facts.ts b/packages/cli/src/schemas/report-facts.ts new file mode 100644 index 00000000..92e0968a --- /dev/null +++ b/packages/cli/src/schemas/report-facts.ts @@ -0,0 +1,193 @@ +import { z } from "zod"; + +/** + * What the agent knows that the CLI cannot: the facts `taskless report --from` + * reads. + * + * NOT A REPORTV1. The CLI builds the report; the agent supplies only what its + * session taught it. Every object is strict, so a field the CLI owns — most + * pointedly `perMonth`, which the CLI computes — is refused rather than + * quietly merged, and the agent hears that it tried to set something it does + * not own. + * + * Embedded into the `report` recipe as JSON Schema, so the recipe and this + * file cannot describe two different shapes. + */ + +const count = z.number().int().min(0); + +const isoDate = z.iso.date().describe("A calendar date, YYYY-MM-DD"); + +const ruleId = z + .string() + .regex(/^[\dA-Za-z][\w.-]*$/) + .max(128) + .describe( + "The rule's id: its directory name, or one you choose for a rule outside .taskless/" + ); + +const origin = z + .object({ + reviewComments: count.describe( + "Review comments in the window that asked for what this rule enforces" + ), + prs: count.describe("Pull or merge requests those comments were on"), + }) + .strict(); + +const outcome = { + title: z + .string() + .min(1) + .max(200) + .describe( + "What the rule enforces, in a phrase a reader of the report understands" + ), + origin, + fixed: count.describe( + "Times in the window the pattern was fixed after review: the catches this rule would have made" + ), + fixedBasis: z + .enum(["measured", "estimated"]) + .describe("`measured` when counted from history, `estimated` otherwise"), +}; + +const fixture = z + .string() + .min(1) + .describe( + "A short code sample; withheld if it holds anything you cannot vouch for" + ); + +/** Facts about a rule already under `.taskless/rules/`. */ +const localRuleFacts = z.object({ id: ruleId, ...outcome }).strict(); + +/** A rule the agent wrote into the repository's own linter. */ +const legacyRuleFacts = z + .object({ + id: ruleId, + ...outcome, + tool: z + .string() + .regex(/^[\w@./+-]+$/) + .max(64) + .describe("The linter it lives in, e.g. eslint, ruff, rubocop"), + language: z.string().min(1).max(40), + fail: fixture.optional().describe("Code the rule flags"), + pass: fixture.optional().describe("Code the rule accepts"), + }) + .strict(); + +/** A runtime-shaped candidate the agent did not author. */ +const cloudCandidateFacts = z + .object({ + id: ruleId, + ...outcome, + language: z.string().min(1).max(40), + complexity: z + .number() + .gt(0) + .max(100) + .nullable() + .describe("Rough calls per check for a runtime rule, or null if unknown"), + fail: fixture.optional().describe("Code the rule would flag"), + pass: fixture.optional().describe("Code the rule would accept"), + }) + .strict(); + +export const reportFactsSchema = z + .object({ + window: z + .object({ from: isoDate, to: isoDate }) + .strict() + .describe("The span of history you mined"), + label: z + .string() + .min(1) + .max(100) + .optional() + .describe( + "A name for the repository when it has no GitHub origin; defaults to its directory name" + ), + history: z + .object({ + mergedPrs: count, + reviewComments: count, + agentAssistedPrs: count.describe( + "Merged pull requests an agent helped write, as far as you can tell" + ), + patterns: z + .object({ + ruleShaped: count.describe( + "Recurring review patterns a rule could enforce" + ), + judgment: count.describe( + "Recurring review patterns that need human judgment" + ), + }) + .strict(), + }) + .strict() + .describe( + "What the window's history held. Zeros are honest when there was nothing to mine" + ), + rules: z + .array(localRuleFacts) + .describe("One entry per rule the draft listed under `rules`"), + legacy: z + .array(legacyRuleFacts) + .default([]) + .describe("Rules you wrote into the repository's own linter"), + cloud: z + .array(cloudCandidateFacts) + .default([]) + .describe("Runtime-shaped candidates you did not author"), + estimates: z + .object({ + contextTokens: z + .number() + .int() + .gt(0) + .max(10_000_000) + .describe( + "Tokens in a typical session of this user's, from what you have seen" + ), + callsPerCatch: z + .number() + .gt(0) + .max(1000) + .describe( + "Extra agent turns a missed issue costs this user before it is fixed" + ), + }) + .strict(), + redaction: z + .object({ + counts: z + .object({ + secret: count, + email: count, + person: count, + customer: count, + host: count, + ip: count, + phone: count, + text: count, + }) + .partial() + .strict() + .default({}) + .describe("What you removed or withheld yourself, by kind"), + withheld: z + .array( + z.object({ id: ruleId, side: z.enum(["fail", "pass"]) }).strict() + ) + .default([]) + .describe("Fixtures you chose not to publish"), + }) + .strict() + .default({ counts: {}, withheld: [] }), + }) + .strict(); + +export type ReportFacts = z.infer; diff --git a/packages/cli/src/types/errors.ts b/packages/cli/src/types/errors.ts index 78a75409..a6d691b8 100644 --- a/packages/cli/src/types/errors.ts +++ b/packages/cli/src/types/errors.ts @@ -92,6 +92,22 @@ export type CLIErrorCode = // migrate. | "SCAFFOLD_MANIFEST_UNREADABLE" | "SCAFFOLD_CONFLICT" + // `taskless report`. INSUFFICIENT is the gate (fewer than three working + // Taskless rules) and is the one a recipe answers by asking the user; + // REDACTION names an agent-written field to rewrite; PUBLISH_UNKNOWN means a + // timeout, after which a report may exist and a retry may duplicate it. + | "REPORT_INSUFFICIENT" + | "REPORT_INVALID" + | "REPORT_REDACTION" + | "REPORT_TOO_LARGE" + | "REPORT_RATE_LIMITED" + | "REPORT_PUBLISH_FAILED" + | "REPORT_PUBLISH_UNKNOWN" + | "REPORT_NOT_FOUND" + // tskl.es answered 404 `not_available` to a delete: no report answers to + // that URL. Distinct from DELETE_FAILED because retrying cannot help. + | "REPORT_NOT_AVAILABLE" + | "REPORT_DELETE_FAILED" | "INTERNAL_ERROR"; /** diff --git a/packages/cli/test/migrate-reports-directory.test.ts b/packages/cli/test/migrate-reports-directory.test.ts new file mode 100644 index 00000000..4b5f614c --- /dev/null +++ b/packages/cli/test/migrate-reports-directory.test.ts @@ -0,0 +1,107 @@ +import { execFile } from "node:child_process"; +import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { promisify } from "node:util"; + +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { ensureTasklessDirectory } from "../src/filesystem/directory"; +import { LATEST_SCHEMA_VERSION } from "../src/filesystem/migrate"; +import migration from "../src/filesystem/migrations/0011-reports-directory"; + +/** + * Migration 0011 creates `.taskless/reports/` and ignores it, because each + * record there holds the URL that deletes a published report. + */ +describe("migration 0011 creates and ignores the reports directory", () => { + let directory: string; + let taskless: string; + + beforeEach(async () => { + directory = await mkdtemp(join(tmpdir(), "tskl-0011-")); + taskless = join(directory, ".taskless"); + }); + + afterEach(async () => { + await rm(directory, { recursive: true, force: true }); + }); + + async function gitignoreLines(): Promise { + const content = await readFile(join(taskless, ".gitignore"), "utf8"); + return content.split("\n").filter(Boolean); + } + + it("upgrades a version-10 scaffold", async () => { + await mkdir(taskless, { recursive: true }); + await writeFile( + join(taskless, "taskless.json"), + JSON.stringify({ version: 10, install: {} }), + "utf8" + ); + await writeFile( + join(taskless, ".gitignore"), + ".env.local.json\n/sgconfig.yml\n/.tmp-*\n", + "utf8" + ); + + await ensureTasklessDirectory(directory, { onNotice: () => {} }); + + expect(await gitignoreLines()).toContain("/reports/"); + await expect( + readFile(join(taskless, "reports", "README.md"), "utf8") + ).resolves.toContain("taskless report --delete"); + const manifest = JSON.parse( + await readFile(join(taskless, "taskless.json"), "utf8") + ) as { version: number }; + expect(manifest.version).toBe(LATEST_SCHEMA_VERSION); + expect(LATEST_SCHEMA_VERSION).toBeGreaterThanOrEqual(11); + }); + + it("leaves a fresh scaffold in the same state", async () => { + await ensureTasklessDirectory(directory, { onNotice: () => {} }); + + expect(await gitignoreLines()).toContain("/reports/"); + await expect( + readFile(join(taskless, "reports", "README.md"), "utf8") + ).resolves.toBeTypeOf("string"); + }); + + it("is idempotent and leaves existing records alone", async () => { + await mkdir(join(taskless, "reports"), { recursive: true }); + const record = join(taskless, "reports", "2026-10-13_142200.json"); + await writeFile(record, '{"url":"u","deleteUrl":"d"}\n', "utf8"); + await writeFile(join(taskless, "reports", "README.md"), "edited\n", "utf8"); + await writeFile(join(taskless, ".gitignore"), "/reports/\n", "utf8"); + + await migration(taskless); + await migration(taskless); + + const lines = await gitignoreLines(); + expect(lines.filter((line) => line === "/reports/")).toHaveLength(1); + await expect(readFile(record, "utf8")).resolves.toBe( + '{"url":"u","deleteUrl":"d"}\n' + ); + await expect( + readFile(join(taskless, "reports", "README.md"), "utf8") + ).resolves.toBe("edited\n"); + }); + + it("keeps a record out of git", async () => { + await ensureTasklessDirectory(directory, { onNotice: () => {} }); + await writeFile( + join(taskless, "reports", "2026-10-13_142200.json"), + "{}", + "utf8" + ); + + const run = promisify(execFile); + await run("git", ["init", "-q"], { cwd: directory }); + const { stdout } = await run( + "git", + ["check-ignore", ".taskless/reports/2026-10-13_142200.json"], + { cwd: directory } + ); + expect(stdout.trim()).toBe(".taskless/reports/2026-10-13_142200.json"); + }); +}); diff --git a/packages/cli/test/prompts.test.ts b/packages/cli/test/prompts.test.ts index 9a8d1f81..19460b2b 100644 --- a/packages/cli/test/prompts.test.ts +++ b/packages/cli/test/prompts.test.ts @@ -631,6 +631,8 @@ describe("prompts entry carries no CLI runtime", () => { // `zod` and `src/survey/constants.ts`, which is pure data like // `../rules/capabilities` and imports nothing at all. "../schemas/feedback", + // The `report` recipe's facts schema. It imports `zod` and nothing else. + "../schemas/report-facts", "./recipes.js", ]); diff --git a/packages/cli/test/report-telemetry.test.ts b/packages/cli/test/report-telemetry.test.ts new file mode 100644 index 00000000..d397a2eb --- /dev/null +++ b/packages/cli/test/report-telemetry.test.ts @@ -0,0 +1,136 @@ +import { createServer, type Server } from "node:http"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import type { AddressInfo } from "node:net"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { stringify } from "yaml"; + +import { ensureTasklessDirectory } from "../src/filesystem/directory"; + +// The report command's events, captured by mocking the module it imports, as +// feedback-command.test.ts does. +const capture = vi.fn(); +vi.mock("../src/telemetry", () => ({ + getTelemetry: vi.fn(() => + Promise.resolve({ capture, shutdown: () => Promise.resolve() }) + ), + isTelemetryEnabled: () => true, + shutdownTelemetry: () => Promise.resolve(), +})); + +const { reportCommand } = await import("../src/commands/report"); + +interface RunnableCommand { + run: (context: { + args: Record; + rawArgs: string[]; + }) => Promise; +} + +const DELETE_PATH = "/api/v1/reports/abc/secret-key"; +const TITLE = "Call the API through the shared client"; + +describe("report telemetry", () => { + let cwd: string; + let server: Server; + let base: string; + + beforeEach(async () => { + cwd = await mkdtemp(join(tmpdir(), "tskl-report-telemetry-")); + await ensureTasklessDirectory(cwd, { onNotice: () => {} }); + for (const id of ["alpha", "beta", "gamma"]) { + const directory = join(cwd, ".taskless", "rules", "sg", id); + await mkdir(join(directory, ".tests"), { recursive: true }); + await writeFile( + join(directory, `${id}.yml`), + stringify({ + id, + language: "TypeScript", + severity: "error", + message: `no ${id}`, + rule: { pattern: `${id}($$$)` }, + }) + ); + await writeFile( + join(directory, ".tests", `${id}-test.yml`), + stringify({ id, valid: ["const a = 1;"], invalid: [`${id}(x);`] }) + ); + } + server = createServer((request, response) => { + request.resume(); + request.on("end", () => { + response.writeHead(request.method === "POST" ? 201 : 204, { + "content-type": "application/json", + }); + response.end( + request.method === "POST" + ? JSON.stringify({ + url: `${base}/r/abc`, + deleteUrl: `${base}${DELETE_PATH}`, + expiresAt: "2026-11-12T00:00:00Z", + }) + : "" + ); + }); + }); + await new Promise((resolve) => + server.listen(0, "127.0.0.1", resolve) + ); + base = `http://127.0.0.1:${String((server.address() as AddressInfo).port)}`; + vi.stubEnv("TASKLESS_REPORT_URL", `${base}/api/v1/reports`); + capture.mockClear(); + vi.spyOn(console, "log").mockImplementation(() => {}); + vi.spyOn(console, "error").mockImplementation(() => {}); + }); + + afterEach(async () => { + vi.restoreAllMocks(); + vi.unstubAllEnvs(); + process.exitCode = undefined; + await new Promise((resolve) => server.close(resolve)); + await rm(cwd, { recursive: true, force: true }); + }); + + it("carries counts, never the link, the delete URL, a title or a fixture", async () => { + const factsPath = join(cwd, ".taskless", ".tmp-report.json"); + await writeFile( + factsPath, + JSON.stringify({ + window: { from: "2026-07-14", to: "2026-10-11" }, + history: { + mergedPrs: 1, + reviewComments: 1, + agentAssistedPrs: 0, + patterns: { ruleShaped: 3, judgment: 0 }, + }, + rules: ["alpha", "beta", "gamma"].map((id) => ({ + id, + title: TITLE, + origin: { reviewComments: 1, prs: 1 }, + fixed: 1, + fixedBasis: "measured", + })), + estimates: { contextTokens: 1000, callsPerCatch: 2 }, + }), + "utf8" + ); + + await (reportCommand as RunnableCommand).run({ + args: { dir: cwd, from: factsPath, json: false }, + rawArgs: [], + }); + + const published = capture.mock.calls.find( + ([event]) => event === "cli_report_published" + ); + expect(published?.[1]).toMatchObject({ cli: 3, legacy: 0, cloud: 0 }); + + const everything = JSON.stringify(capture.mock.calls); + expect(everything).not.toContain("/r/abc"); + expect(everything).not.toContain(DELETE_PATH); + expect(everything).not.toContain(TITLE); + expect(everything).not.toContain("alpha(x)"); + }); +}); diff --git a/packages/cli/test/report.test.ts b/packages/cli/test/report.test.ts new file mode 100644 index 00000000..92cf55b5 --- /dev/null +++ b/packages/cli/test/report.test.ts @@ -0,0 +1,722 @@ +import { execFile } from "node:child_process"; +import { createServer, type Server } from "node:http"; +import { + mkdir, + mkdtemp, + readFile, + readdir, + rm, + writeFile, +} from "node:fs/promises"; +import type { AddressInfo } from "node:net"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { promisify } from "node:util"; + +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { stringify } from "yaml"; + +import reportSchemaDocument from "../src/generated/report-v1.schema.json"; +import { assembleReport } from "../src/report/assemble"; +import { + DEFAULT_PROJECTION_INPUTS, + type LocalRule, + type ReportDraft, +} from "../src/report/draft"; +import { + reportFactsSchema, + type ReportFacts, +} from "../src/schemas/report-facts"; +import { deletePublishedReport, publishReport } from "../src/report/publish"; +import { scanText } from "../src/report/redact"; +import { reportSchemaIssues } from "../src/report/schema"; +import { renderSummary } from "../src/report/summary"; +import { CLIError } from "../src/util/cli-error"; +import { builtCli } from "./support/built-cli"; + +const execFileAsync = promisify(execFile); + +function localRule(id: string, overrides: Partial = {}): LocalRule { + return { + id, + engine: "sg", + tier: "cli", + language: "TypeScript", + tests: { status: "pass", fail: `bad_${id}()`, pass: `good_${id}()` }, + ...overrides, + }; +} + +function draftOf(rules: LocalRule[]): ReportDraft { + return { + producer: { kind: "cli", cliVersion: "0.12.0" }, + scope: { + provider: "github", + owner: "acme", + repo: "widgets", + url: "https://github.com/acme/widgets", + }, + rules, + inputs: DEFAULT_PROJECTION_INPUTS, + missing: [], + }; +} + +function factsFor( + ids: string[], + overrides: Partial> = {} +): ReportFacts { + return reportFactsSchema.parse({ + window: { from: "2026-07-14", to: "2026-10-11" }, + history: { + mergedPrs: 400, + reviewComments: 2000, + agentAssistedPrs: 200, + patterns: { ruleShaped: 10, judgment: 3 }, + }, + rules: ids.map((id) => ({ + id, + title: `Rule ${id}`, + origin: { reviewComments: 9, prs: 6 }, + fixed: 10, + fixedBasis: "measured", + })), + estimates: { contextTokens: 150_000, callsPerCatch: 20 }, + ...overrides, + }); +} + +function codeOf(run: () => unknown): string | undefined { + try { + run(); + } catch (error) { + if (error instanceof CLIError) return error.code; + throw error; + } + return undefined; +} + +describe("the vendored ReportV1 schema", () => { + it("accepts its own example and rejects an unknown key", () => { + const example = (reportSchemaDocument as { examples: unknown[] }) + .examples[0]; + expect(reportSchemaIssues(example)).toEqual([]); + expect( + reportSchemaIssues({ ...(example as object), surprise: true }) + ).not.toEqual([]); + }); +}); + +describe("the vendored delete contract", () => { + // `deletePublishedReport` is written against this contract. A refresh that + // changes it must fail here rather than leave the client calling a method or + // reading a status tskl.es no longer uses. + it("is a DELETE to the deleteUrl: 204 on success, 404 when nothing answers", () => { + const contract = ( + reportSchemaDocument as unknown as { + "x-taskless": { + delete: { + method: string; + response: { status: number }; + errors: { not_available: { status: number } }; + }; + }; + } + )["x-taskless"].delete; + expect(contract.method).toBe("DELETE"); + expect(contract.response.status).toBe(204); + expect(contract.errors.not_available.status).toBe(404); + }); +}); + +describe("assembleReport", () => { + const three = ["a", "b", "c"]; + + it("builds a payload the vendored schema accepts", () => { + const report = assembleReport( + draftOf(three.map((id) => localRule(id))), + factsFor(three) + ); + expect(reportSchemaIssues(report)).toEqual([]); + expect(report.scope).toMatchObject({ + provider: "github", + retention: "private", + }); + }); + + it("normalizes per-tier counts to a 30-day month over the window", () => { + // 2026-07-14 through 2026-10-11 inclusive is 90 days; three rules fixed + // 10 times each is 30, so 10 a month. + const report = assembleReport( + draftOf(three.map((id) => localRule(id))), + factsFor(three) + ); + expect(report.projection.perMonth.agentCatches.cli).toBe(10); + expect(report.projection.inputs).toMatchObject({ + contextTokens: 150_000, + callsPerCatch: 20, + }); + }); + + it("refuses with too few local rules, cloud candidates notwithstanding", () => { + const facts = factsFor(["a", "b"], { + cloud: ["x", "y", "z", "w"].map((id) => ({ + id, + title: `Cloud ${id}`, + origin: { reviewComments: 1, prs: 1 }, + fixed: 1, + fixedBasis: "estimated", + language: "TypeScript", + complexity: 3, + })), + }); + let message = ""; + try { + assembleReport(draftOf([localRule("a"), localRule("b")]), facts); + } catch (error) { + expect((error as CLIError).code).toBe("REPORT_INSUFFICIENT"); + message = (error as Error).message; + } + expect(message).toContain("2 of 3"); + }); + + it("counts legacy rules the agent wrote", () => { + const facts = factsFor(["a"], { + legacy: ["l1", "l2"].map((id) => ({ + id, + title: `Legacy ${id}`, + origin: { reviewComments: 1, prs: 1 }, + fixed: 1, + fixedBasis: "measured", + tool: "eslint", + language: "TypeScript", + })), + }); + const report = assembleReport(draftOf([localRule("a")]), facts); + expect(report.rules.filter((rule) => rule.tier === "legacy")).toHaveLength( + 2 + ); + expect(reportSchemaIssues(report)).toEqual([]); + }); + + it("does not let a failing rule count", () => { + const rules = [ + localRule("a"), + localRule("b"), + localRule("c", { + tests: { status: "fail", fail: null, pass: null }, + }), + ]; + expect(codeOf(() => assembleReport(draftOf(rules), factsFor(three)))).toBe( + "REPORT_INSUFFICIENT" + ); + }); + + it("does not gate on a small projection", () => { + const facts = factsFor(three, { + estimates: { contextTokens: 1, callsPerCatch: 0.1 }, + }); + expect(() => + assembleReport(draftOf(three.map((id) => localRule(id))), facts) + ).not.toThrow(); + }); + + it("publishes cloud candidates as runtime, untested", () => { + const facts = factsFor(three, { + cloud: [ + { + id: "tenant-check", + title: "Routes call requireTenant()", + origin: { reviewComments: 4, prs: 3 }, + fixed: 2, + fixedBasis: "estimated", + language: "TypeScript", + complexity: 4, + fail: "export const GET = route(() => db.all())", + }, + ], + }); + const report = assembleReport( + draftOf(three.map((id) => localRule(id))), + facts + ); + expect( + report.rules.find((rule) => rule.id === "tenant-check") + ).toMatchObject({ + tier: "cloud", + engine: "runtime", + tests: { status: "not-run" }, + }); + }); + + it("refuses facts for a rule the CLI does not have", () => { + expect( + codeOf(() => + assembleReport( + draftOf(three.map((id) => localRule(id))), + factsFor([...three, "ghost"]) + ) + ) + ).toBe("REPORT_INVALID"); + }); + + it("refuses a per-month figure in the facts", () => { + const result = reportFactsSchema.safeParse({ + ...factsFor(three), + perMonth: { agentCatches: { cli: 999, cloud: 0 } }, + }); + expect(result.success).toBe(false); + }); + + it("withholds a fixture holding a secret, whole", () => { + const secret = "const key = 'AKIAIOSFODNN7EXAMPLE';"; + const rules = [ + localRule("a", { tests: { status: "pass", fail: secret, pass: "ok()" } }), + localRule("b"), + localRule("c"), + ]; + const report = assembleReport(draftOf(rules), factsFor(three)); + const a = report.rules.find((rule) => rule.id === "a"); + expect(a?.tests.fail).toBeNull(); + expect(a?.tests.withheld).toEqual(["fail"]); + expect(report.redaction.counts.secret).toBe(1); + expect(report.redaction.withheldFixtures).toBe(1); + expect(JSON.stringify(report)).not.toContain("AKIAIOSFODNN7EXAMPLE"); + }); + + it("refuses an email address in a title", () => { + const facts = factsFor(three); + facts.rules[0]!.title = "Ask jane@acme.io before merging"; + expect( + codeOf(() => + assembleReport(draftOf(three.map((id) => localRule(id))), facts) + ) + ).toBe("REPORT_REDACTION"); + }); + + it("refuses a match in a legacy or cloud rule's other strings, naming the field", () => { + const legacy = { + id: "l1", + title: "Legacy l1", + origin: { reviewComments: 1, prs: 1 }, + fixed: 1, + fixedBasis: "measured" as const, + tool: "eslint", + language: "TypeScript", + }; + const cloud = { + id: "c1", + title: "Cloud c1", + origin: { reviewComments: 1, prs: 1 }, + fixed: 1, + fixedBasis: "estimated" as const, + language: "TypeScript", + complexity: 3, + }; + const cases: [string, Partial][] = [ + [ + "legacy[l1].tool", + { legacy: [{ ...legacy, tool: "jane@acme.io" }], cloud: [] }, + ], + [ + "legacy[l1].language", + { legacy: [{ ...legacy, language: "TS for api.acme.io" }], cloud: [] }, + ], + [ + "cloud[c1].language", + { legacy: [], cloud: [{ ...cloud, language: "jane@acme.io" }] }, + ], + [ + "cloud[api.acme.io].id", + { legacy: [], cloud: [{ ...cloud, id: "api.acme.io" }] }, + ], + ]; + for (const [field, extra] of cases) { + let refusal: unknown; + try { + assembleReport( + draftOf(three.map((id) => localRule(id))), + factsFor(three, extra) + ); + } catch (error) { + refusal = error; + } + expect((refusal as CLIError).code, field).toBe("REPORT_REDACTION"); + expect((refusal as Error).message, field).toContain(field); + } + }); + + it("carries the agent's own redaction counts and withholdings", () => { + const facts = factsFor(three, { + redaction: { + counts: { person: 2 }, + withheld: [{ id: "b", side: "pass" }], + }, + }); + const report = assembleReport( + draftOf(three.map((id) => localRule(id))), + facts + ); + expect(report.redaction.counts.person).toBe(2); + expect(report.rules.find((rule) => rule.id === "b")?.tests).toMatchObject({ + pass: null, + withheld: ["pass"], + }); + }); +}); + +describe("scanText", () => { + it("ignores documentation hosts and ordinary member access", () => { + expect( + scanText( + "fetch('https://api.example.com'); this.app.listen(); res.json()" + ) + ).toEqual({}); + }); + + it("finds a real host, an IP, and a phone number", () => { + expect(scanText("db.internal 10.2.3.4 call (415) 555-0100")).toMatchObject({ + host: 1, + ip: 1, + phone: 1, + }); + }); +}); + +describe("renderSummary", () => { + it("names each rule and the link, and never the delete URL", () => { + const report = assembleReport( + draftOf(["a", "b", "c"].map((id) => localRule(id))), + factsFor(["a", "b", "c"]) + ); + const summary = renderSummary({ + report, + url: "https://tskl.es/r/abc", + expiresAt: "2026-11-12T00:00:00Z", + record: "2026-10-13_142200.json", + invocation: "npx @taskless/cli", + }); + expect(summary).toContain("Rule a (Taskless ast-grep rule)"); + expect(summary).toContain("https://tskl.es/r/abc"); + expect(summary).toContain("can be deleted after review"); + expect(summary).not.toContain("delete/"); + }); +}); + +interface FakeResponse { + status: number; + body?: unknown; +} + +/** + * A stand-in for tskl.es. `answer` decides each response from the request and + * the fake's own base URL; every request is recorded. + */ +async function fakeTsklEs( + answer: ( + request: { method: string; url: string }, + base: string + ) => FakeResponse +): Promise<{ + server: Server; + url: string; + requests: Array<{ method: string; url: string; body: string }>; +}> { + const requests: Array<{ method: string; url: string; body: string }> = []; + let base = ""; + const server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => (body += chunk.toString())); + request.on("end", () => { + const seen = { method: request.method ?? "", url: request.url ?? "" }; + requests.push({ ...seen, body }); + const next = answer(seen, base); + response.writeHead(next.status, { "content-type": "application/json" }); + response.end(next.body === undefined ? "" : JSON.stringify(next.body)); + }); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const { port } = server.address() as AddressInfo; + base = `http://127.0.0.1:${String(port)}`; + return { server, url: base, requests }; +} + +/** Answers from a fixed script, in order. */ +function scripted(responses: FakeResponse[]) { + return () => responses.shift() ?? { status: 500 }; +} + +describe("publishReport", () => { + const original = { ...process.env }; + let server: Server | undefined; + + afterEach(async () => { + process.env = { ...original }; + if (server) await new Promise((resolve) => server!.close(resolve)); + server = undefined; + }); + + const created = { + status: 201, + body: { + url: "https://tskl.es/r/abc", + deleteUrl: "https://tskl.es/api/v1/reports/abc/key", + expiresAt: "2026-11-12T00:00:00Z", + }, + }; + + it("retries a 502 and succeeds", async () => { + const fake = await fakeTsklEs(scripted([{ status: 502 }, created])); + server = fake.server; + process.env.TASKLESS_REPORT_URL = `${fake.url}/api/v1/reports`; + process.env.TASKLESS_REPORT_BACKOFF_MS = "1"; + + await expect(publishReport("{}")).resolves.toMatchObject({ + url: "https://tskl.es/r/abc", + }); + expect(fake.requests).toHaveLength(2); + }); + + it("surfaces a rate limit without retrying", async () => { + const fake = await fakeTsklEs(scripted([{ status: 429 }, created])); + server = fake.server; + process.env.TASKLESS_REPORT_URL = `${fake.url}/api/v1/reports`; + + await expect(publishReport("{}")).rejects.toMatchObject({ + code: "REPORT_RATE_LIMITED", + }); + expect(fake.requests).toHaveLength(1); + }); + + it("does not retry a dropped connection", async () => { + // A port nothing listens on: the request fails before any response, which + // is the case where a report may or may not exist. + const fake = await fakeTsklEs(scripted([])); + const closedUrl = `${fake.url}/api/v1/reports`; + await new Promise((resolve) => fake.server.close(resolve)); + process.env.TASKLESS_REPORT_URL = closedUrl; + + await expect(publishReport("{}")).rejects.toMatchObject({ + code: "REPORT_PUBLISH_UNKNOWN", + }); + }); +}); + +/** The report records under `.taskless/reports/`, the stub README excluded. */ +async function recordsIn(cwd: string): Promise { + const names = await readdir(join(cwd, ".taskless", "reports")); + return names.filter((name) => name.endsWith(".json")); +} + +describe("deletePublishedReport", () => { + let server: Server | undefined; + + afterEach(async () => { + if (server) await new Promise((resolve) => server!.close(resolve)); + server = undefined; + }); + + it("resolves on a 204", async () => { + const fake = await fakeTsklEs(scripted([{ status: 204 }])); + server = fake.server; + await expect( + deletePublishedReport(`${fake.url}/api/v1/reports/abc/key`) + ).resolves.toBeUndefined(); + expect(fake.requests[0]).toMatchObject({ method: "DELETE" }); + }); + + it("reports a 404 as not available, once", async () => { + const fake = await fakeTsklEs(scripted([{ status: 404 }, { status: 204 }])); + server = fake.server; + await expect( + deletePublishedReport(`${fake.url}/api/v1/reports/abc/key`) + ).rejects.toMatchObject({ code: "REPORT_NOT_AVAILABLE" }); + // Not retryable, per the contract. + expect(fake.requests).toHaveLength(1); + }); +}); + +/** End to end, through the built CLI, against a fake tskl.es. */ +describe("taskless report", () => { + const binPath = builtCli(); + let cwd: string; + let server: Server | undefined; + + async function run( + args: string[], + env: Record = {} + ): Promise<{ stdout: string; stderr: string; exitCode: number }> { + try { + const { stdout, stderr } = await execFileAsync( + "node", + [binPath, ...args, "-d", cwd], + { + env: { + ...process.env, + DO_NOT_TRACK: "1", + TASKLESS_TELEMETRY_DISABLED: "1", + ...env, + }, + } + ); + return { stdout, stderr, exitCode: 0 }; + } catch (error) { + const failure = error as { + stdout?: string; + stderr?: string; + code?: number; + }; + return { + stdout: failure.stdout ?? "", + stderr: failure.stderr ?? "", + exitCode: failure.code ?? 1, + }; + } + } + + async function sgRule(id: string): Promise { + const directory = join(cwd, ".taskless", "rules", "sg", id); + await mkdir(join(directory, ".tests"), { recursive: true }); + await writeFile( + join(directory, `${id}.yml`), + stringify({ + id, + language: "TypeScript", + severity: "error", + message: `no ${id}`, + rule: { pattern: `${id}($$$)` }, + }) + ); + await writeFile( + join(directory, ".tests", `${id}-test.yml`), + stringify({ id, valid: ["const a = 1;"], invalid: [`${id}(x);`] }) + ); + } + + beforeEach(async () => { + cwd = await mkdtemp(join(tmpdir(), "tskl-report-")); + await run(["init"]); + for (const id of ["alpha", "beta", "gamma"]) await sgRule(id); + }); + + afterEach(async () => { + if (server) await new Promise((resolve) => server!.close(resolve)); + server = undefined; + await rm(cwd, { recursive: true, force: true }); + }); + + it("drafts offline, listing every rule and what is missing", async () => { + const { stdout, exitCode } = await run(["report"], { + TASKLESS_REPORT_URL: "http://127.0.0.1:9/unreachable", + }); + expect(exitCode).toBe(0); + const draft = JSON.parse(stdout) as ReportDraft; + expect(draft.rules.map((rule) => rule.id)).toEqual([ + "alpha", + "beta", + "gamma", + ]); + expect(draft.rules.every((rule) => rule.tests.status === "pass")).toBe( + true + ); + expect(draft.scope.provider).toBe("none"); + expect(draft.missing).toContain("history"); + expect(draft.missing).toContain("rules[alpha]"); + }); + + it("publishes, records, summarizes, and deletes", async () => { + const fake = await fakeTsklEs((request, base) => + request.method === "POST" + ? { + status: 201, + body: { + url: "https://tskl.es/r/abc", + deleteUrl: `${base}/api/v1/reports/abc/secret-key`, + expiresAt: "2026-11-12T00:00:00Z", + }, + } + : { status: 204 } + ); + server = fake.server; + const deleteUrl = `${fake.url}/api/v1/reports/abc/secret-key`; + + const factsPath = join(cwd, ".taskless", ".tmp-report.json"); + await writeFile( + factsPath, + JSON.stringify(factsFor(["alpha", "beta", "gamma"])), + "utf8" + ); + const env = { TASKLESS_REPORT_URL: `${fake.url}/api/v1/reports` }; + + const published = await run(["report", "--from", factsPath], env); + expect(published.exitCode).toBe(0); + expect(published.stdout).toContain("https://tskl.es/r/abc"); + expect(published.stdout).toContain("can be deleted after review"); + expect(published.stdout).not.toContain("secret-key"); + + const sent = JSON.parse(fake.requests[0]!.body) as Record; + expect(reportSchemaIssues(sent)).toEqual([]); + + const records = await recordsIn(cwd); + expect(records).toHaveLength(1); + const record = JSON.parse( + await readFile(join(cwd, ".taskless", "reports", records[0]!), "utf8") + ) as { deleteUrl: string }; + expect(record.deleteUrl).toBe(deleteUrl); + + const deleted = await run(["report", "--delete", records[0]!], env); + expect(deleted.exitCode).toBe(0); + expect(fake.requests.at(-1)).toMatchObject({ + method: "DELETE", + url: "/api/v1/reports/abc/secret-key", + }); + await expect(recordsIn(cwd)).resolves.toEqual([]); + }); + + it("refuses too few rules without touching the network", async () => { + await rm(join(cwd, ".taskless", "rules", "sg", "gamma"), { + recursive: true, + }); + const factsPath = join(cwd, ".taskless", ".tmp-report.json"); + await writeFile( + factsPath, + JSON.stringify(factsFor(["alpha", "beta"])), + "utf8" + ); + const { stderr, exitCode } = await run(["report", "--from", factsPath], { + TASKLESS_REPORT_URL: "http://127.0.0.1:9/unreachable", + }); + expect(exitCode).not.toBe(0); + expect(stderr).toContain("REPORT_INSUFFICIENT"); + }); + + it("reports a missing record", async () => { + const { stderr, exitCode } = await run(["report", "--delete", "nope.json"]); + expect(exitCode).not.toBe(0); + expect(stderr).toContain("REPORT_NOT_FOUND"); + }); + + it("refuses a record outside .taskless/reports/", async () => { + for (const record of [ + "../outside.json", + "/tmp/outside.json", + "notes.txt", + ]) { + const { stderr, exitCode } = await run(["report", "--delete", record]); + expect(exitCode, record).not.toBe(0); + expect(stderr, record).toContain("INVALID_INPUT"); + } + }); + + it("refuses --from and --delete together", async () => { + const { stderr, exitCode } = await run([ + "report", + "--from", + "x.json", + "--delete", + "y.json", + ]); + expect(exitCode).not.toBe(0); + expect(stderr).toContain("INVALID_INPUT"); + }); +});