diff --git a/.gitignore b/.gitignore index f4b5928..63584a0 100644 --- a/.gitignore +++ b/.gitignore @@ -105,3 +105,6 @@ Pods/ .factory/ .jules/ .remember/ + +# tgrep trigram indexes (per-repo, rebuilt via `tgrep index .`) +.tgrep/ diff --git a/README.md b/README.md index 60402e6..ba06b7c 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,7 @@ pi install -l npm:@groeponline/pi-tools | `find` (spawns `fd`) | `fffind` (FFF `fileSearch`) | Fuzzy matching, frecency ranking, git-aware, pre-indexed | | `grep` (spawns `rg`) | `ffgrep` (FFF `grep`) | SIMD-accelerated, frecency-ordered, mmap-cached, no subprocess | | *(none)* | `fff-multi-grep` (FFF `multiGrep`, opt-in) | OR-logic multi-pattern search via Aho-Corasick | +| *(none)* | `tgrep` (external [tgrep](https://github.com/microsoft/tgrep) binary) | Trigram-indexed exact search when a workspace `.tgrep/` index exists | | `@` file autocomplete (fd-backed) | `@` file autocomplete (FFF-backed, default) | Fuzzy ranking from the FFF index and frecency | ### Modes @@ -54,20 +55,22 @@ Three operating modes, switchable at runtime with `/fff-mode`: | `tools-only` | Only tool injection. Keeps pi's native editor autocomplete. | | `override` | Replaces pi's built-in `grep` and `find` with FFF implementations. With `PI_FFF_MULTIGREP=1`, also registers `multi_grep`. | -Set `PI_FFF_MULTIGREP=1` to opt in to `fff-multi-grep` (or `multi_grep` in `override` mode). Without it, only `ffgrep` and `fffind` are registered. +Set `PI_FFF_MULTIGREP=1` to opt in to `fff-multi-grep` (or `multi_grep` in `override` mode). Without it, only `ffgrep` and `fffind` are registered. `tgrep` is registered independently of mode when the binary and a `.tgrep/` index are found. -Env vars: `PI_FFF_MODE`, `FFF_FRECENCY_DB`, `FFF_HISTORY_DB`. Flags: `--fff-mode`, `--fff-frecency-db`, `--fff-history-db`. The databases default to your existing fff.nvim ones when present, otherwise `~/.pi/agent/fff/`. +Env vars: `PI_FFF_MODE`, `FFF_FRECENCY_DB`, `FFF_HISTORY_DB`, `TGREP_BIN`, `TGREP_TIME_BUDGET_MS`. Flags: `--fff-mode`, `--fff-frecency-db`, `--fff-history-db`. Config (`~/.pi/agent/pi-tools.json`): `enableTgrep` (default true), `tgrepBinPath`, `tgrepTimeBudgetMs` (default 30000). The databases default to your existing fff.nvim ones when present, otherwise `~/.pi/agent/fff/`. ### Agent-facing tools - `ffgrep`. Content search. Accepts `path`, `exclude` (comma, space, or array; leading `!` optional), `caseSensitive`, `context`, and cursor pagination. Auto-detects regex, falls back to fuzzy on zero exact matches, rejects `.*`-style wildcard-only patterns up front. - `fffind`. Path and filename search. Matches the whole repo-relative path, not just the filename. Frecency-aware. The weak-match detector flags scattered fuzzy noise before it floods the agent's context. +- `tgrep`. Exact literal/symbol search through an external tgrep binary. Registered only when the binary resolves (`TGREP_BIN` → `tgrepBinPath` → `PATH`) and the workspace has a `.tgrep/` index. After your own edits, use `ffgrep`. ### Commands - `/fff-mode [tools-and-ui | tools-only | override]`. Show or switch the mode. - `/fff-health`. Picker, frecency, and git integration status. - `/fff-rescan`. Force a rescan. +- `/tgrep-status`. tgrep index and server status for the workspace. Source: [`packages/pi-tools/`](./packages/pi-tools/). Full documentation: [`packages/pi-tools/README.md`](./packages/pi-tools/README.md). diff --git a/docs/compatibility.md b/docs/compatibility.md index 0e31bc6..710b8e7 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -8,21 +8,22 @@ ## Tool surface -The extension registers two custom tools and one completion surface (names resolved from -mode; see below): +The extension registers two FFF tools, an optional `tgrep` tool, and one completion surface +(FFF names resolved from mode; see below): | Tool | Purpose | |---|---| | `fffind` | Typo-resistant file discovery + frecency-ranked access | | `ffgrep` | SIMD content search | +| `tgrep` | Trigram-indexed exact content search via an external binary (registered when the binary and a workspace `.tgrep/` index are found) | -Source of truth: `packages/pi-tools/src/index.ts:52-53` (`grep: "ffgrep", find: "fffind"`), +Source of truth: `packages/pi-tools/src/index.ts:60` (`FFF_TOOL_NAMES`: `grep: "ffgrep", find: "fffind"`), `tgrep` via `queueTgrepTool` / `queueTool(() => TGREP_TOOL_NAME, …)` gated on `resolveTgrepBinary` plus `hasTgrepIndex` (`src/tgrep.ts`) at session start, registered via `queueTool(() => toolNames.grep, …)` / `queueTool(() => toolNames.find, …)`. ### Parameters Parameter shapes, defaults and allowed values are defined in the tool schemas in -`packages/pi-tools/src/index.ts` (`grepSchema`, `findSchema`, `multiGrepSchema`). +`packages/pi-tools/src/index.ts` (`grepSchema`, `findSchema`, `multiGrepSchema`, `tgrepSchema`). `pi-tools.schema.json` documents **config keys only** (mode, DB paths, scan toggles), not tool parameters. Compatibility surface = the parameter **names**, **types**, **enums**, and **defaults** for both tools, plus their pagination behavior: @@ -38,12 +39,18 @@ tool parameters. Compatibility surface = the parameter **names**, **types**, **e The file picker UI exists only in `tools-and-ui`; the `@`-mention completion surface is disabled only in `tools-only` (`shouldEnableMentions`: `currentMode !== 'tools-only'`). A tool disappearing when mode changes is expected; a mode value being removed is breaking. +- **Exit codes (`tgrep`):** exit `0` returns `file:line:col:text` rows, exit `1` reports + `No matches found` (not a failure), exit `2` throws with the stderr cause. A leading + `[tgrep: ...]` line always carries the binary's stderr freshness warning. Output + truncates at `TGREP_OUTPUT_MAX_BYTES` with a narrowing hint. + `tgrep` is mode-independent: it keeps its name in every mode and is gated on + binary availability, a workspace `.tgrep/` directory, and `enableTgrep`. ### Config precedence Config values are resolved in this priority order -(`packages/pi-tools/src/index.ts:272` `getConfigValue`, read at startup -`resolveStartupConfig`): +(`packages/pi-tools/src/index.ts` `getConfigValue`, read at startup +`resolveStartupConfig`; `tgrep` binary and index are resolved at session start): ``` flag (--fff-mode / --fff-frecency-db / --fff-history-db / …) @@ -61,6 +68,9 @@ Concrete defaults: | history DB | `--fff-history-db` | `FFF_HISTORY_DB` | `config.historyDbPath` | (platform db path) | | root scan | `--fff-enable-root-scan` | `FFF_ENABLE_ROOT_SCAN` | `config.enableFsRootScanning` | `false` | | home scan | `--fff-enable-home-scan` | `FFF_ENABLE_HOME_SCAN` | `config.enableHomeDirScanning` | `true` | +| tgrep binary | — (no CLI flag by design) | `TGREP_BIN` | `config.tgrepBinPath` | `PATH` lookup | +| tgrep toggle | — (config file only) | — | `config.enableTgrep` | `true` | +| tgrep time budget | — (no CLI flag by design) | `TGREP_TIME_BUDGET_MS` | `config.tgrepTimeBudgetMs` | `30000` | Mode valid values (`packages/pi-tools/src/config.ts:8` `VALID_MODES`): `tools-and-ui`, `tools-only`, `override`. diff --git a/packages/pi-tools/CHANGELOG.md b/packages/pi-tools/CHANGELOG.md index f134ee8..1dd8633 100644 --- a/packages/pi-tools/CHANGELOG.md +++ b/packages/pi-tools/CHANGELOG.md @@ -4,6 +4,8 @@ ### Added +- Added the `tgrep` tool: trigram-indexed exact content search through an external [tgrep](https://github.com/microsoft/tgrep) binary. Literal by default, `file:line:col:text` output, exit 1 reported as no-match. Registered at session start only when the binary resolves (`TGREP_BIN`, `tgrepBinPath`, or `PATH`), the workspace has a `.tgrep/` index, and `enableTgrep` is not `false`; mode-independent. Prompt guidelines steer on capability (exact/literal vs fuzzy/frecency), not repository size. Optional `tgrepTimeBudgetMs` / `TGREP_TIME_BUDGET_MS` (default 30s). Output truncation stays on a UTF-8 character boundary; mid-run abort reports `Operation aborted`. `--no-index` is not exposed; after edits use `ffgrep`. +- Added the `/tgrep-status` command showing tgrep index and server status for the workspace. - Added `ffgrep.maxMatchesPerFile` to keep a single generated or vendored file from dominating a result page. - Added `ffgrep.compact` for deterministic `path:line:match` output without context blocks. diff --git a/packages/pi-tools/README.md b/packages/pi-tools/README.md index 38dbb82..c7807df 100644 --- a/packages/pi-tools/README.md +++ b/packages/pi-tools/README.md @@ -84,6 +84,26 @@ Use `fffind` for **paths**. Use `ffgrep` when you know text that should occur in Use a concrete substring, identifier, or expression. A wildcard-only expression such as `.*` is rejected because it is not an efficient way to read an entire file. Keep the default grouped output when context matters; use `compact: true` when the next action only needs stable path-and-line references. Set `maxMatchesPerFile` when a generated or vendored file could otherwise dominate the page. Both options are additive and leave existing defaults unchanged. +### `tgrep` + +`tgrep` searches file content through an external [tgrep](https://github.com/microsoft/tgrep) binary: trigram-indexed exact search with a client/server architecture. The tool is registered only when the binary is found, the session cwd has a `.tgrep/` index, and `enableTgrep` is not disabled; it keeps the name `tgrep` in every mode. Prefer `tgrep` for exact literals and symbols; prefer `ffgrep` for fuzzy, typo-tolerant, frecency-ranked search. After `tgrep index .` or `tgrep serve .`, reload the session to expose the tool. + +| Parameter | Type | Description | +| --- | --- | --- | +| `pattern` | string | Literal text by default; set `literal: false` for a regular expression. Patterns like `serve` cannot parse as subcommands. | +| `path` | string, optional | Directory or file to search, relative to the workspace; default is the workspace root. Globs go in `glob`. | +| `glob` | string or string array, optional | Repeatable file glob filter such as `*.{ts,tsx}`. | +| `fileType` | string or string array, optional | Repeatable file type filter such as `rust`, `py`, or `js`. | +| `literal` | boolean, optional | Treats the pattern as literal text; default is `true`. | +| `caseSensitive` | boolean, optional | Forces case-sensitive matching; default is smart-case. | +| `wholeWord` | boolean, optional | Matches whole words only. | +| `filesOnly` | boolean, optional | Prints only filenames with matches. | +| `count` | boolean, optional | Prints the match count per file. | +| `context` | number, optional | Context lines before and after a match; range 0–20. | +| `maxCount` | number, optional | Limits matches per file. | + +Output is `file:line:col:text` rows. Exit code 1 (no match) is reported as `No matches found`, not as a failure. A leading `[tgrep: ...]` line carries the binary's stderr freshness warning. Only index-safe flags are forwarded; full-scan forcers (`--hidden`, `--no-ignore`, `-u`, `-a`, `--encoding`, `--no-index`) are excluded by design. After your own edits, use `ffgrep` because the index lags watcher events. + ### Optional multi-pattern search Set `PI_FFF_MULTIGREP=1` before starting Pi to enable the experimental `fff-multi-grep` tool. It searches for **any** of several literal patterns in one request and is useful when an agent must check known naming variants together. @@ -105,6 +125,7 @@ The tool is opt-in while its interaction pattern is evaluated. Do not depend on | `/fff-mode [tools-and-ui \| tools-only \| override]` | Shows the current mode or records a mode for the current session. | | `/fff-health` | Displays the engine version, mode, Git integration, index status, persistence status, and active scan progress. | | `/fff-rescan` | Requests a new file scan for the active workspace. | +| `/tgrep-status` | Shows tgrep index and server status for the workspace. | ## Persistent configuration @@ -127,9 +148,14 @@ Create `pi-tools.json` in Pi’s agent directory. The default location is `~/.pi | `historyDbPath` | string | Auto-resolved | Location for query-selection history. | | `enableFsRootScanning` | boolean | `false` | Explicitly allows scans started from `/`. | | `enableHomeDirScanning` | boolean | `true` | Allows scanning when Pi starts in the home directory. | +| `enableTgrep` | boolean | `true` | Registers the `tgrep` tool when the binary and a `.tgrep/` index are found. | +| `tgrepBinPath` | string | PATH lookup | Explicit path to the `tgrep` binary. | +| `tgrepTimeBudgetMs` | number | `30000` | Child-process time budget in milliseconds. `TGREP_TIME_BUDGET_MS` overrides this. | Malformed configuration, unknown fields, and invalid values prevent the extension from loading and identify the configuration path in the error. `/fff-mode` changes session state only; it does not edit this file. +The `tgrep` binary resolves as `TGREP_BIN` environment variable, then `tgrepBinPath`, then a `tgrep` executable on `PATH`. An explicit path that is set but not executable disables the tool instead of falling back, so a typo surfaces instead of silently changing the search backend. Binary and `.tgrep/` index are resolved at session start, when the workspace cwd is known. Installing tgrep or building an index mid-session requires `/reload` before the tool appears. + ## Database resolution Frecency and history paths resolve independently in the following order: diff --git a/packages/pi-tools/pi-tools.schema.json b/packages/pi-tools/pi-tools.schema.json index ea6b63c..92f54dc 100644 --- a/packages/pi-tools/pi-tools.schema.json +++ b/packages/pi-tools/pi-tools.schema.json @@ -36,6 +36,22 @@ "type": "boolean", "default": true, "description": "Allows indexing when pi is launched from the home directory." + }, + "enableTgrep": { + "type": "boolean", + "default": true, + "description": "Registers the tgrep trigram-index search tool when the tgrep binary is found and the workspace has a .tgrep index." + }, + "tgrepBinPath": { + "type": "string", + "minLength": 1, + "description": "Explicit path to the tgrep binary; the TGREP_BIN environment variable takes precedence over config.tgrepBinPath, then PATH is used as a fallback." + }, + "tgrepTimeBudgetMs": { + "type": "integer", + "minimum": 1, + "default": 30000, + "description": "Maximum milliseconds for a tgrep child process. TGREP_TIME_BUDGET_MS overrides this value." } } } diff --git a/packages/pi-tools/src/config.ts b/packages/pi-tools/src/config.ts index f0d2569..ed914f2 100644 --- a/packages/pi-tools/src/config.ts +++ b/packages/pi-tools/src/config.ts @@ -16,6 +16,9 @@ export interface FffConfig { historyDbPath?: string; enableFsRootScanning?: boolean; enableHomeDirScanning?: boolean; + enableTgrep?: boolean; + tgrepBinPath?: string; + tgrepTimeBudgetMs?: number; } const CONFIG_KEYS = new Set([ @@ -25,8 +28,15 @@ const CONFIG_KEYS = new Set([ "historyDbPath", "enableFsRootScanning", "enableHomeDirScanning", + "enableTgrep", + "tgrepBinPath", + "tgrepTimeBudgetMs", ]); +/** + * Loads and validates pi-tools.json, falling back to the legacy filename. + * Returns an empty config when neither file exists and throws for unreadable or invalid files. + */ export function loadConfig(agentDir = piDataDir()): FffConfig { let configPath = join(agentDir, CONFIG_FILE_NAME); const legacyConfigPath = join(agentDir, LEGACY_CONFIG_FILE_NAME); @@ -75,8 +85,11 @@ export function loadConfig(agentDir = piDataDir()): FffConfig { validateString(configPath, parsed, "$schema"); validateString(configPath, parsed, "frecencyDbPath"); validateString(configPath, parsed, "historyDbPath"); + validateString(configPath, parsed, "tgrepBinPath"); validateBoolean(configPath, parsed, "enableFsRootScanning"); validateBoolean(configPath, parsed, "enableHomeDirScanning"); + validateBoolean(configPath, parsed, "enableTgrep"); + validatePositiveInteger(configPath, parsed, "tgrepTimeBudgetMs"); return parsed as FffConfig; } @@ -93,10 +106,11 @@ function isRecord(value: unknown): value is Record { return typeof value === "object" && value !== null && !Array.isArray(value); } +/** Validates that an optional configuration value is a non-empty string. */ function validateString( configPath: string, config: Record, - key: "$schema" | "frecencyDbPath" | "historyDbPath", + key: "$schema" | "frecencyDbPath" | "historyDbPath" | "tgrepBinPath", ): void { const value = config[key]; if (value !== undefined && (typeof value !== "string" || value.length === 0)) { @@ -104,13 +118,26 @@ function validateString( } } +/** Validates that an optional configuration value is boolean. */ function validateBoolean( configPath: string, config: Record, - key: "enableFsRootScanning" | "enableHomeDirScanning", + key: "enableFsRootScanning" | "enableHomeDirScanning" | "enableTgrep", ): void { const value = config[key]; if (value !== undefined && typeof value !== "boolean") { throw invalidConfig(configPath, `"${key}" must be a boolean`); } } + +function validatePositiveInteger( + configPath: string, + config: Record, + key: "tgrepTimeBudgetMs", +): void { + const value = config[key]; + if (value === undefined) return; + if (typeof value !== "number" || !Number.isInteger(value) || value < 1) { + throw invalidConfig(configPath, `"${key}" must be a positive integer`); + } +} diff --git a/packages/pi-tools/src/index.ts b/packages/pi-tools/src/index.ts index feab8df..0da429d 100644 --- a/packages/pi-tools/src/index.ts +++ b/packages/pi-tools/src/index.ts @@ -23,6 +23,18 @@ import { type FffMode, loadConfig, VALID_MODES } from "./config"; import { FilePickerFactory } from "./file-picker"; import { isHomeDir, resolveDbPaths } from "./paths"; import { buildQuery } from "./query"; +import { + buildTgrepArgs, + hasTgrepIndex, + resolveSearchRoot, + resolveTgrepBinary, + runTgrep, + TGREP_BIN_ENV, + TGREP_CONTEXT_MAX, + TGREP_TIME_BUDGET_ENV, + TGREP_TIME_BUDGET_MS, + TGREP_TOOL_NAME, +} from "./tgrep"; export { SCAN_TIMEOUT_MS } from "./sdk"; @@ -259,6 +271,10 @@ function createFffMentionProvider( }; } +/** + * Registers FFF tools, commands, lifecycle hooks, and autocomplete with Pi. + * Also registers tgrep search when the binary and a `.tgrep` index are available. + */ export default function fffExtension(pi: ExtensionAPI) { let mainFinder: FileFinderApi | null = null; let finderCwd: string | null = null; @@ -267,6 +283,17 @@ export default function fffExtension(pi: ExtensionAPI) { let activeCwd = process.cwd(); const config = loadConfig(); + // Resolved at session start so cwd (index) and flags are known. + let tgrepBin: string | undefined; + let tgrepTimeBudgetMs = TGREP_TIME_BUDGET_MS; + + function resolveTgrepBin(): string | undefined { + if (config.enableTgrep === false) return undefined; + return resolveTgrepBinary( + process.env[TGREP_BIN_ENV]?.trim() || config.tgrepBinPath, + process.env.PATH, + ); + } // Resolve startup options with flag > environment > file > fallback. function getConfigValue( @@ -304,6 +331,15 @@ export default function fffExtension(pi: ExtensionAPI) { : undefined; } + function parsePositiveInt(value: unknown): number | undefined { + if (typeof value === "number" && Number.isInteger(value) && value >= 1) return value; + if (typeof value === "string" && value.trim() !== "") { + const parsed = Number(value); + if (Number.isInteger(parsed) && parsed >= 1) return parsed; + } + return undefined; + } + let currentMode: FffMode = "tools-and-ui"; let toolNames = resolveToolNames(currentMode); let resolvedDbPaths: ReturnType; @@ -350,6 +386,13 @@ export default function fffExtension(pi: ExtensionAPI) { true, parseBoolean, ); + tgrepTimeBudgetMs = getConfigValue( + "tgrep-time-budget-ms", + TGREP_TIME_BUDGET_ENV, + config.tgrepTimeBudgetMs, + TGREP_TIME_BUDGET_MS, + parsePositiveInt, + ); } function getMode(): FffMode { @@ -596,6 +639,108 @@ export default function fffExtension(pi: ExtensionAPI) { toolsRegistered = true; } + function queueTgrepTool(bin: string): void { + const tgrepSchema = Type.Object({ + pattern: Type.String({ + description: + "Search pattern. Literal text by default; set literal: false for regex.", + }), + path: Type.Optional( + Type.String({ + description: + "Directory or file to search, relative to the workspace (default: workspace root). Globs go in glob.", + }), + ), + glob: Type.Optional( + Type.Union([Type.String(), Type.Array(Type.String())], { + description: "Repeatable file glob filter, e.g. '*.{ts,tsx}'.", + }), + ), + fileType: Type.Optional( + Type.Union([Type.String(), Type.Array(Type.String())], { + description: "Repeatable file type filter, e.g. 'rust', 'py', 'js'.", + }), + ), + literal: Type.Optional( + Type.Boolean({ + description: + "Treat pattern as literal text (default true). Set false for regex.", + }), + ), + caseSensitive: Type.Optional( + Type.Boolean({ + description: "Force case-sensitive matching. Default is smart-case.", + }), + ), + wholeWord: Type.Optional(Type.Boolean({ description: "Match whole words only." })), + filesOnly: Type.Optional( + Type.Boolean({ description: "Print only filenames with matches." }), + ), + count: Type.Optional(Type.Boolean({ description: "Print match count per file." })), + context: Type.Optional( + Type.Number({ + description: `Context lines before+after each match (0-${TGREP_CONTEXT_MAX})`, + }), + ), + maxCount: Type.Optional(Type.Number({ description: "Limit matches per file." })), + }); + + queueTool(() => TGREP_TOOL_NAME, { + description: + "Trigram-indexed exact content search (tgrep). Literal by default, vimgrep output. Use for symbols and exact strings; prefer FFF grep for fuzzy or frecency-ranked results. Default limit is output-capped; narrow with fileType/glob.", + promptSnippet: "Trigram-indexed exact content search", + promptGuidelines: (names) => [ + `${names.grep}: prefer for fuzzy, typo-tolerant, frecency-ranked search.`, + `${TGREP_TOOL_NAME}: prefer for exact literal or symbol search against the tgrep index.`, + `${TGREP_TOOL_NAME}: keep literal: true (default) for symbols; set false for regex.`, + `${TGREP_TOOL_NAME}: narrow with fileType/glob before raising maxCount.`, + `${TGREP_TOOL_NAME}: use filesOnly: true first on broad queries, then search specific files.`, + `${TGREP_TOOL_NAME}: after your own edits, use ${names.grep}; the index lags watcher events.`, + `${TGREP_TOOL_NAME}: a '[tgrep: ...]' line reports index freshness, never drop it from summaries.`, + ], + parameters: tgrepSchema, + + /** Executes tgrep with validated workspace-relative arguments. */ + async execute(_toolCallId, params, signal) { + if (signal?.aborted) throw new Error("Operation aborted"); + const root = resolveSearchRoot(params.path, activeCwd); + const output = await runTgrep( + bin, + buildTgrepArgs({ + pattern: params.pattern, + root, + literal: params.literal, + caseSensitive: params.caseSensitive, + wholeWord: params.wholeWord, + fileType: params.fileType, + glob: params.glob, + filesOnly: params.filesOnly, + count: params.count, + context: params.context, + maxCount: params.maxCount, + }), + { cwd: activeCwd, signal, timeoutMs: tgrepTimeBudgetMs }, + ); + return { content: [{ type: "text", text: output }], details: {} }; + }, + + renderCall(args, theme, context) { + const text = (context.lastComponent as Text | undefined) ?? new Text("", 0, 0); + text.setText( + theme.fg("toolTitle", theme.bold(TGREP_TOOL_NAME)) + + " " + + theme.fg("accent", `/${args?.pattern ?? ""}/`) + + theme.fg("toolOutput", ` in ${args?.path ?? "."}`), + ); + return text; + }, + + renderResult(result, options, theme, context) { + return renderTextResult(result, options, theme, context, 15); + }, + }); + } + pi.registerFlag("fff-mode", { description: "FFF mode: tools-and-ui | tools-only | override", type: "string", @@ -658,6 +803,10 @@ export default function fffExtension(pi: ExtensionAPI) { } initializeFinderFactories(); + tgrepBin = resolveTgrepBin(); + if (tgrepBin !== undefined && hasTgrepIndex(activeCwd)) { + queueTgrepTool(tgrepBin); + } registerPendingTools(); } @@ -1262,4 +1411,36 @@ export default function fffExtension(pi: ExtensionAPI) { ctx.ui.notify("FFF rescan triggered", "info"); }, }); + + pi.registerCommand("tgrep-status", { + description: "Show tgrep index and server status for the workspace", + handler: async (_args, ctx) => { + if (!toolsRegistered) { + try { + prepareSession(ctx); + } catch (error: unknown) { + reportInitFailure(ctx, error); + return; + } + } + + const bin = tgrepBin; + if (!bin) { + ctx.ui.notify("tgrep binary not found (TGREP_BIN or PATH)", "warning"); + return; + } + try { + const output = await runTgrep(bin, ["status", activeCwd], { + cwd: activeCwd, + timeoutMs: tgrepTimeBudgetMs, + }); + ctx.ui.notify(output || "tgrep status: no output", "info"); + } catch (error: unknown) { + ctx.ui.notify( + `tgrep status failed: ${error instanceof Error ? error.message : String(error)}`, + "error", + ); + } + }, + }); } diff --git a/packages/pi-tools/src/tgrep.ts b/packages/pi-tools/src/tgrep.ts new file mode 100644 index 0000000..e40003c --- /dev/null +++ b/packages/pi-tools/src/tgrep.ts @@ -0,0 +1,228 @@ +import { execFile } from "node:child_process"; +import fs from "node:fs"; +import path from "node:path"; +import { promisify } from "node:util"; + +export const TGREP_BIN_ENV = "TGREP_BIN"; +export const TGREP_TIME_BUDGET_ENV = "TGREP_TIME_BUDGET_MS"; +export const TGREP_TOOL_NAME = "tgrep"; +export const TGREP_INDEX_DIR = ".tgrep"; +export const TGREP_TIME_BUDGET_MS = 30_000; +export const TGREP_OUTPUT_MAX_BYTES = 200_000; +export const TGREP_CONTEXT_MAX = 20; + +export interface TgrepSearchOptions { + pattern: string; + root: string; + literal?: boolean; + caseSensitive?: boolean; + wholeWord?: boolean; + fileType?: string | string[]; + glob?: string | string[]; + filesOnly?: boolean; + count?: boolean; + context?: number; + maxCount?: number; +} + +export interface TgrepResult { + exit: number; + stdout: string; + stderr: string; +} + +export interface TgrepExecOptions { + cwd: string; + signal?: AbortSignal; + timeoutMs?: number; +} + +type ExecFn = ( + bin: string, + args: string[], + opts: TgrepExecOptions, +) => Promise; + +/** Normalizes an optional scalar or list into an iterable array. */ +function repeatAll(values: string | string[] | undefined): string[] { + if (!values) return []; + return Array.isArray(values) ? values : [values]; +} + +/** + * Builds tgrep arguments for vimgrep output with literal, smart-case matching by default. + * Clamps context and per-file limits and separates the pattern and root from options. + */ +export function buildTgrepArgs(options: TgrepSearchOptions): string[] { + const args = ["--vimgrep"]; + args.push(options.caseSensitive ? "--case-sensitive" : "--smart-case"); + if (options.literal !== false) args.push("--fixed-strings"); + if (options.wholeWord) args.push("--word-regexp"); + for (const t of repeatAll(options.fileType)) args.push("--type", t); + for (const g of repeatAll(options.glob)) args.push("--glob", g); + if (options.filesOnly) args.push("--files-with-matches"); + if (options.count) args.push("--count"); + const context = clampContext(options.context); + if (context > 0) args.push("-A", String(context), "-B", String(context)); + if (options.maxCount !== undefined) + args.push("--max-count", String(Math.max(1, Math.floor(options.maxCount)))); + // Separator keeps patterns like "serve" or "-x" from parsing as subcommands. + args.push("--", options.pattern, options.root); + return args; +} + +/** + * Resolves a file or directory to a normalized path relative to the workspace. + * Throws when the input contains glob syntax or resolves outside the workspace. + */ +export function resolveSearchRoot(pathParam: string | undefined, cwd: string): string { + const raw = (pathParam ?? ".").trim() || "."; + if (/[*?[{]/.test(raw)) + throw new Error(`tgrep path must be a directory or file, not a glob: ${raw}`); + const resolved = path.resolve(cwd, raw); + const relative = path.relative(cwd, resolved); + if (relative.startsWith("..") || path.isAbsolute(relative)) + throw new Error(`tgrep path must stay inside the workspace: ${raw}`); + return relative === "" ? "." : relative.split(path.sep).join("/"); +} + +/** Checks whether a path points to an executable regular file. */ +function isExecutable(file: string): boolean { + try { + fs.accessSync(file, fs.constants.X_OK); + return fs.statSync(file).isFile(); + } catch { + return false; + } +} + +/** + * Returns an explicit executable or the first executable named tgrep on the search path. + * A nonempty but unusable explicit path returns undefined without searching the path. + */ +export function resolveTgrepBinary(explicit?: string, pathEnv = ""): string | undefined { + const trimmed = explicit?.trim(); + if (trimmed) return isExecutable(trimmed) ? trimmed : undefined; + for (const dir of pathEnv.split(path.delimiter)) { + if (!dir) continue; + const candidate = path.join(dir, "tgrep"); + if (isExecutable(candidate)) return candidate; + } + return undefined; +} + +/** True when the session cwd has a `.tgrep` directory (default tgrep index location). */ +export function hasTgrepIndex(cwd: string): boolean { + try { + return fs.statSync(path.join(cwd, TGREP_INDEX_DIR)).isDirectory(); + } catch { + return false; + } +} + +const execFileAsync = promisify(execFile); + +/** Runs tgrep with bounded time and output, preserving output from normal process exits. */ +async function defaultExec( + bin: string, + args: string[], + opts: TgrepExecOptions, +): Promise { + try { + const { stdout, stderr } = await execFileAsync(bin, args, { + cwd: opts.cwd, + signal: opts.signal, + timeout: opts.timeoutMs ?? TGREP_TIME_BUDGET_MS, + maxBuffer: TGREP_OUTPUT_MAX_BYTES * 2, + }); + return { exit: 0, stdout, stderr }; + } catch (error: unknown) { + if (isAbortError(error) || opts.signal?.aborted) throw new Error("Operation aborted"); + const execError = error as { + code?: number | string; + stdout?: string; + stderr?: string; + message?: string; + }; + if (execError.code === "ERR_CHILD_PROCESS_STDIO_MAXBUFFER") + return { exit: 0, stdout: execError.stdout ?? "", stderr: execError.stderr ?? "" }; + if (typeof execError.code === "number") + return { + exit: execError.code, + stdout: execError.stdout ?? "", + stderr: execError.stderr ?? "", + }; + throw new Error(`tgrep failed to run: ${execError.message ?? String(error)}`); + } +} + +/** + * Executes tgrep and formats its output for a tool response. + * Rejects an already-aborted call; execution and formatting errors propagate. + */ +export async function runTgrep( + bin: string, + args: string[], + opts: TgrepExecOptions, + exec: ExecFn = defaultExec, +): Promise { + if (opts.signal?.aborted) throw new Error("Operation aborted"); + const result = await runTgrepRaw(bin, args, opts, exec); + return formatTgrepResult(result); +} + +/** + * Executes tgrep and returns its unformatted exit code and output streams. + * The optional executor replaces the default child-process invocation. + */ +export async function runTgrepRaw( + bin: string, + args: string[], + opts: TgrepExecOptions, + exec: ExecFn = defaultExec, +): Promise { + return exec(bin, args, opts); +} + +/** + * Formats tgrep output, preserving the first stderr line and truncating oversized results. + * Treats exit 1 or empty stdout as no matches and throws for exit 2. + */ +export function formatTgrepResult(result: TgrepResult): string { + const warning = result.stderr.trim().split("\n")[0]?.trim(); + const notice = warning ? `[tgrep: ${warning}]\n` : ""; + if (result.exit === 2) + throw new Error(`tgrep search failed: ${warning || "unknown error"}`); + const budget = Math.max(0, TGREP_OUTPUT_MAX_BYTES - Buffer.byteLength(notice)); + const body = truncateBytes(result.stdout.trim(), budget); + if (result.exit === 1 || body === "") return `${notice}No matches found`; + return `${notice}${body}`; +} + +/** Truncates oversized output on a UTF-8 character boundary. */ +function truncateBytes(text: string, maxBytes = TGREP_OUTPUT_MAX_BYTES): string { + const buf = Buffer.from(text); + if (buf.length <= maxBytes) return text; + const hint = (omitted: number) => + `\n… [truncated ${omitted} bytes: narrow with fileType/glob]`; + const reserved = Buffer.byteLength(hint(buf.length)); + if (reserved >= maxBytes) { + let end = maxBytes; + while (end > 0 && (buf[end] & 0xc0) === 0x80) end--; + return buf.subarray(0, end).toString(); + } + let end = maxBytes - reserved; + while (end > 0 && (buf[end] & 0xc0) === 0x80) end--; + return `${buf.subarray(0, end).toString()}${hint(buf.length - end)}`; +} + +/** Normalizes context to the supported non-negative integer range. */ +function clampContext(context: number | undefined): number { + if (!context || context < 0) return 0; + return Math.min(Math.floor(context), TGREP_CONTEXT_MAX); +} + +function isAbortError(error: unknown): boolean { + if (error instanceof Error && error.name === "AbortError") return true; + return (error as { code?: unknown }).code === "ABORT_ERR"; +} diff --git a/packages/pi-tools/test/config.test.ts b/packages/pi-tools/test/config.test.ts index 6231e9c..afbcbfb 100644 --- a/packages/pi-tools/test/config.test.ts +++ b/packages/pi-tools/test/config.test.ts @@ -31,6 +31,9 @@ describe("loadConfig", () => { historyDbPath: "/data/history", enableFsRootScanning: true, enableHomeDirScanning: false, + enableTgrep: false, + tgrepBinPath: "/usr/local/bin/tgrep", + tgrepTimeBudgetMs: 45000, }; writeConfig(config); @@ -66,6 +69,11 @@ describe("loadConfig", () => { [{ historyDbPath: false }, '"historyDbPath" must be a non-empty string'], [{ enableFsRootScanning: 1 }, '"enableFsRootScanning" must be a boolean'], [{ enableHomeDirScanning: "false" }, '"enableHomeDirScanning" must be a boolean'], + [{ enableTgrep: "yes" }, '"enableTgrep" must be a boolean'], + [{ tgrepBinPath: "" }, '"tgrepBinPath" must be a non-empty string'], + [{ tgrepTimeBudgetMs: 0 }, '"tgrepTimeBudgetMs" must be a positive integer'], + [{ tgrepTimeBudgetMs: 1.5 }, '"tgrepTimeBudgetMs" must be a positive integer'], + [{ tgrepTimeBudgetMs: "30000" }, '"tgrepTimeBudgetMs" must be a positive integer'], ]; for (const [config, message] of cases) { diff --git a/packages/pi-tools/test/extension.test.ts b/packages/pi-tools/test/extension.test.ts index 6b59138..17309bd 100644 --- a/packages/pi-tools/test/extension.test.ts +++ b/packages/pi-tools/test/extension.test.ts @@ -187,6 +187,8 @@ const CONFIG_ENV_KEYS = [ "FFF_HISTORY_DB", "FFF_ENABLE_ROOT_SCAN", "FFF_ENABLE_HOME_SCAN", + "TGREP_BIN", + "TGREP_TIME_BUDGET_MS", ] as const; const savedEnv: Record = {}; @@ -705,3 +707,401 @@ describe("pi-tools autocomplete registration", () => { expect(current.shouldTriggerFileCompletion).toHaveBeenCalledTimes(1); }); }); + +describe("pi-tools tgrep integration", () => { + let binDir = ""; + + /** Creates an executable shell stub for tgrep integration tests. */ + function writeFakeBin(name: string, body: string): string { + binDir = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-")); + const bin = path.join(binDir, name); + fs.writeFileSync(bin, `#!/bin/sh\n${body}\n`); + fs.chmodSync(bin, 0o755); + return bin; + } + + function writeIndexedCwd(): string { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + fs.mkdirSync(path.join(cwd, ".tgrep")); + return cwd; + } + + /** Returns the tool names registered through the mocked Pi API. */ + function toolNames(setup: { pi: { registerTool: any } }): string[] { + return setup.pi.registerTool.mock.calls.map(([tool]: [any]) => tool.name); + } + + test("registers tgrep and tgrep-status when the binary and index are found", async () => { + process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + const cwd = writeIndexedCwd(); + const setup = await start(undefined, cwd); + try { + expect(toolNames(setup)).toContain("tgrep"); + expect(setup.commands.has("tgrep-status")).toBe(true); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("registers the complete tgrep parameter schema", async () => { + process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + const cwd = writeIndexedCwd(); + const setup = await start(undefined, cwd); + try { + const tool = setup.pi.registerTool.mock.calls + .map(([registered]: [any]) => registered) + .find((registered: any) => registered.name === "tgrep"); + const properties = tool.parameters.properties; + + expect(Object.keys(properties).sort()).toEqual( + [ + "caseSensitive", + "context", + "count", + "filesOnly", + "fileType", + "glob", + "literal", + "maxCount", + "path", + "pattern", + "wholeWord", + ].sort(), + ); + expect(properties.pattern).toMatchObject({ type: "string" }); + expect(properties.pattern.optional).toBeUndefined(); + expect(properties.path).toMatchObject({ type: "string", optional: true }); + expect(properties.glob).toMatchObject({ type: "union", optional: true }); + expect(properties.fileType).toMatchObject({ type: "union", optional: true }); + expect(properties.context.options.description).toContain("0-20"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("discovers tgrep from PATH when no explicit path is configured", async () => { + writeFakeBin("tgrep", "exit 1"); + const cwd = writeIndexedCwd(); + const savedPath = process.env.PATH; + process.env.PATH = binDir; + try { + const setup = await start(undefined, cwd); + try { + expect(toolNames(setup)).toContain("tgrep"); + } finally { + await shutdown(setup); + } + } finally { + if (savedPath === undefined) delete process.env.PATH; + else process.env.PATH = savedPath; + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("skips tgrep when the binary is missing", async () => { + process.env.TGREP_BIN = "/nonexistent/tgrep"; + const setup = await start(); + try { + expect(toolNames(setup)).not.toContain("tgrep"); + expect(setup.commands.has("tgrep-status")).toBe(true); + } finally { + await shutdown(setup); + } + }); + + test("skips tgrep when enableTgrep is false", async () => { + process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + writeConfig({ enableTgrep: false }); + const cwd = writeIndexedCwd(); + const setup = await start(undefined, cwd); + try { + expect(toolNames(setup)).not.toContain("tgrep"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("skips tgrep when the binary is found but no index exists", async () => { + process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + const setup = await start(undefined, cwd); + try { + expect(toolNames(setup)).not.toContain("tgrep"); + expect(setup.commands.has("tgrep-status")).toBe(true); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("registers tgrep from the configured binary path", async () => { + const bin = writeFakeBin("configured-tgrep", "exit 1"); + writeConfig({ tgrepBinPath: bin }); + const cwd = writeIndexedCwd(); + const setup = await start(undefined, cwd); + try { + expect(toolNames(setup)).toContain("tgrep"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("does not fall back to config when TGREP_BIN is explicitly missing", async () => { + const bin = writeFakeBin("configured-tgrep", "exit 1"); + writeConfig({ tgrepBinPath: bin }); + process.env.TGREP_BIN = path.join(binDir, "missing"); + const setup = await start(); + try { + expect(toolNames(setup)).not.toContain("tgrep"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + } + }); + + test("registers tgrep independently of the FFF mode", async () => { + process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + const cwd = writeIndexedCwd(); + try { + for (const mode of ["tools-only", "override"]) { + const setup = await start(mode, cwd); + expect(toolNames(setup)).toContain("tgrep"); + await shutdown(setup); + } + } finally { + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("steers tgrep by capability rather than repository size", async () => { + process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + const cwd = writeIndexedCwd(); + const setup = await start(undefined, cwd); + try { + const tool = setup.pi.registerTool.mock.calls + .map(([t]: [any]) => t) + .find((t: any) => t.name === "tgrep"); + expect(tool.description).toContain("exact content search"); + expect(tool.description).not.toContain("large repo"); + expect(tool.promptGuidelines[0]).toContain("fuzzy, typo-tolerant, frecency-ranked"); + expect(tool.promptGuidelines[1]).toContain("exact literal or symbol search"); + expect(tool.promptGuidelines.join("\n")).not.toContain("large repo"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("executes searches through the binary and reports no-match", async () => { + const cwd = writeIndexedCwd(); + process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + const setup = await start(undefined, cwd); + try { + const tool = setup.pi.registerTool.mock.calls + .map(([t]: [any]) => t) + .find((t: any) => t.name === "tgrep"); + expect(tool).toBeDefined(); + const result = await tool.execute("call-1", { pattern: "hello" }, undefined); + expect(result.content[0].text).toBe("No matches found"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("returns vimgrep rows from the binary", async () => { + const cwd = writeIndexedCwd(); + process.env.TGREP_BIN = writeFakeBin("tgrep", "printf 'src/a.ts:3:7:hello\\n'"); + const setup = await start(undefined, cwd); + try { + const tool = setup.pi.registerTool.mock.calls + .map(([t]: [any]) => t) + .find((t: any) => t.name === "tgrep"); + const result = await tool.execute( + "call-1", + { pattern: "hello", path: "src/" }, + undefined, + ); + expect(result.content[0].text).toBe("src/a.ts:3:7:hello"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("forwards every search option with a normalized workspace path", async () => { + const cwd = writeIndexedCwd(); + process.env.TGREP_BIN = writeFakeBin("tgrep", "printf '%s\\n' \"$@\""); + const setup = await start(undefined, cwd); + try { + const tool = setup.pi.registerTool.mock.calls + .map(([t]: [any]) => t) + .find((t: any) => t.name === "tgrep"); + const result = await tool.execute( + "call-1", + { + pattern: "fn main", + path: "./src/../test", + glob: ["*.ts", "!*.test.ts"], + fileType: ["js", "ts"], + literal: false, + caseSensitive: true, + wholeWord: true, + filesOnly: true, + count: true, + context: 2, + maxCount: 5, + }, + new AbortController().signal, + ); + expect(result.content[0].text.split("\n")).toEqual([ + "--vimgrep", + "--case-sensitive", + "--word-regexp", + "--type", + "js", + "--type", + "ts", + "--glob", + "*.ts", + "--glob", + "!*.test.ts", + "--files-with-matches", + "--count", + "-A", + "2", + "-B", + "2", + "--max-count", + "5", + "--", + "fn main", + "test", + ]); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("rejects an already-aborted search without invoking the binary", async () => { + const cwd = writeIndexedCwd(); + const marker = path.join(cwd, "invoked"); + process.env.TGREP_BIN = writeFakeBin("tgrep", `touch '${marker}'`); + const setup = await start(undefined, cwd); + try { + const tool = setup.pi.registerTool.mock.calls + .map(([t]: [any]) => t) + .find((t: any) => t.name === "tgrep"); + const controller = new AbortController(); + controller.abort(); + + await expect( + tool.execute("call-1", { pattern: "hello" }, controller.signal), + ).rejects.toThrow("Operation aborted"); + expect(fs.existsSync(marker)).toBe(false); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("rejects paths outside the workspace before invoking the binary", async () => { + const cwd = writeIndexedCwd(); + const marker = path.join(cwd, "invoked"); + process.env.TGREP_BIN = writeFakeBin("tgrep", `touch '${marker}'`); + const setup = await start(undefined, cwd); + try { + const tool = setup.pi.registerTool.mock.calls + .map(([registered]: [any]) => registered) + .find((registered: any) => registered.name === "tgrep"); + + await expect( + tool.execute("call-1", { pattern: "hello", path: "../outside" }, undefined), + ).rejects.toThrow("tgrep path must stay inside the workspace"); + expect(fs.existsSync(marker)).toBe(false); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("/tgrep-status reports the server status", async () => { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + process.env.TGREP_BIN = writeFakeBin("tgrep", "echo 'Indexing: complete'"); + const setup = await start(undefined, cwd); + try { + await setup.commands.get("tgrep-status").handler("", setup.ctx); + expect(setup.ctx.ui.notify).toHaveBeenCalledWith("Indexing: complete", "info"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("/tgrep-status passes the active workspace to the binary", async () => { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + process.env.TGREP_BIN = writeFakeBin("tgrep", "printf '%s\\n' \"$@\""); + const setup = await start(undefined, cwd); + try { + await setup.commands.get("tgrep-status").handler("ignored", setup.ctx); + expect(setup.ctx.ui.notify).toHaveBeenCalledWith(`status\n${cwd}`, "info"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("/tgrep-status warns without a binary", async () => { + process.env.TGREP_BIN = "/nonexistent/tgrep"; + const setup = await start(); + try { + await setup.commands.get("tgrep-status").handler("", setup.ctx); + expect(setup.ctx.ui.notify).toHaveBeenCalledWith( + "tgrep binary not found (TGREP_BIN or PATH)", + "warning", + ); + } finally { + await shutdown(setup); + } + }); + + test("/tgrep-status reports binary failures", async () => { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + process.env.TGREP_BIN = writeFakeBin( + "tgrep", + "echo 'invalid index metadata' >&2; exit 2", + ); + const setup = await start(undefined, cwd); + try { + await setup.commands.get("tgrep-status").handler("", setup.ctx); + expect(setup.ctx.ui.notify).toHaveBeenCalledWith( + "tgrep status failed: tgrep search failed: invalid index metadata", + "error", + ); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); +}); diff --git a/packages/pi-tools/test/tgrep.test.ts b/packages/pi-tools/test/tgrep.test.ts new file mode 100644 index 0000000..2f18162 --- /dev/null +++ b/packages/pi-tools/test/tgrep.test.ts @@ -0,0 +1,430 @@ +import { describe, expect, mock, test } from "bun:test"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +import { + buildTgrepArgs, + formatTgrepResult, + hasTgrepIndex, + resolveSearchRoot, + resolveTgrepBinary, + runTgrep, + TGREP_CONTEXT_MAX, + TGREP_INDEX_DIR, + TGREP_OUTPUT_MAX_BYTES, +} from "../src/tgrep"; + +describe("buildTgrepArgs", () => { + test("defaults to vimgrep, smart-case and literal with separator", () => { + expect(buildTgrepArgs({ pattern: "foo", root: "." })).toEqual([ + "--vimgrep", + "--smart-case", + "--fixed-strings", + "--", + "foo", + ".", + ]); + }); + + test("keeps subcommand-like patterns from parsing as subcommands", () => { + const args = buildTgrepArgs({ pattern: "serve", root: "src/" }); + expect(args.slice(-3)).toEqual(["--", "serve", "src/"]); + }); + + test("maps search options to the allowlisted flags", () => { + expect( + buildTgrepArgs({ + pattern: "fn main", + root: ".", + literal: false, + caseSensitive: true, + wholeWord: true, + fileType: ["rust", "py"], + glob: "src/**", + filesOnly: true, + }), + ).toEqual([ + "--vimgrep", + "--case-sensitive", + "--word-regexp", + "--type", + "rust", + "--type", + "py", + "--glob", + "src/**", + "--files-with-matches", + "--", + "fn main", + ".", + ]); + }); + + test("repeats glob filters in the supplied order", () => { + const args = buildTgrepArgs({ + pattern: "x", + root: ".", + glob: ["*.ts", "!*.test.ts"], + }); + expect(args.slice(3, -3)).toEqual(["--glob", "*.ts", "--glob", "!*.test.ts"]); + }); + + test("emits count and per-file cap", () => { + const args = buildTgrepArgs({ pattern: "x", root: ".", count: true, maxCount: 0 }); + expect(args).toContain("--count"); + expect(args).toContain("--max-count"); + expect(args[args.indexOf("--max-count") + 1]).toBe("1"); + }); + + test("splits context into before/after and clamps to the max", () => { + const args = buildTgrepArgs({ + pattern: "x", + root: ".", + context: TGREP_CONTEXT_MAX + 100, + }); + expect(args).toContain("-A"); + expect(args).toContain("-B"); + expect(args[args.indexOf("-A") + 1]).toBe(String(TGREP_CONTEXT_MAX)); + }); + + test("floors fractional limits and omits non-positive context", () => { + const fractional = buildTgrepArgs({ + pattern: "x", + root: ".", + context: 3.9, + maxCount: 4.9, + }); + expect(fractional.slice(fractional.indexOf("-A"), fractional.indexOf("-B"))).toEqual([ + "-A", + "3", + ]); + expect(fractional[fractional.indexOf("--max-count") + 1]).toBe("4"); + + for (const context of [0, -1, Number.NaN]) { + const args = buildTgrepArgs({ pattern: "x", root: ".", context }); + expect(args).not.toContain("-A"); + expect(args).not.toContain("-B"); + } + }); + + test("never emits full-scan forcing flags", () => { + const args = buildTgrepArgs({ pattern: "x", root: "." }).join(" "); + for (const banned of [ + "--hidden", + "--no-ignore", + "--text", + "--encoding", + "-u", + "--no-index", + ]) { + expect(args.includes(banned)).toBe(false); + } + }); +}); + +describe("resolveSearchRoot", () => { + const cwd = path.resolve("/tmp/workspace"); + + test("defaults to the workspace root", () => { + expect(resolveSearchRoot(undefined, cwd)).toBe("."); + }); + + test("keeps directories and files inside the workspace", () => { + expect(resolveSearchRoot("src/", cwd)).toBe("src"); + expect(resolveSearchRoot("src/main.ts", cwd)).toBe("src/main.ts"); + }); + + test("trims and normalizes workspace-relative paths", () => { + expect(resolveSearchRoot(" ", cwd)).toBe("."); + expect(resolveSearchRoot(" ./src/../test/tgrep.test.ts ", cwd)).toBe( + "test/tgrep.test.ts", + ); + }); + + test("rejects globs and escapes", () => { + expect(() => resolveSearchRoot("src/**/*.ts", cwd)).toThrow("not a glob"); + expect(() => resolveSearchRoot("../outside", cwd)).toThrow("inside the workspace"); + expect(() => resolveSearchRoot("/etc/passwd", cwd)).toThrow("inside the workspace"); + expect(() => resolveSearchRoot(`${cwd}-sibling`, cwd)).toThrow( + "inside the workspace", + ); + }); +}); + +describe("resolveTgrepBinary", () => { + test("prefers an explicit executable path", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-bin-")); + try { + const bin = path.join(dir, "tgrep"); + fs.writeFileSync(bin, "#!/bin/sh\necho hi\n"); + fs.chmodSync(bin, 0o755); + expect(resolveTgrepBinary(bin, "")).toBe(bin); + expect(resolveTgrepBinary(path.join(dir, "missing"), "")).toBeUndefined(); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); + + test("finds tgrep on PATH and returns undefined when absent", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-path-")); + try { + const bin = path.join(dir, "tgrep"); + fs.writeFileSync(bin, "#!/bin/sh\n"); + fs.chmodSync(bin, 0o755); + expect(resolveTgrepBinary(undefined, dir)).toBe(bin); + expect(resolveTgrepBinary(undefined, path.join(dir, "empty"))).toBeUndefined(); + expect(resolveTgrepBinary(undefined, "")).toBeUndefined(); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); + + test("does not fall back to PATH when an explicit binary is missing", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-precedence-")); + try { + const pathBin = path.join(dir, "tgrep"); + fs.writeFileSync(pathBin, "#!/bin/sh\n"); + fs.chmodSync(pathBin, 0o755); + + expect(resolveTgrepBinary(path.join(dir, "missing"), dir)).toBeUndefined(); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); + + test("skips non-executable PATH entries and continues searching", () => { + const first = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-path-first-")); + const second = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-path-second-")); + try { + fs.writeFileSync(path.join(first, "tgrep"), "#!/bin/sh\n"); + const executable = path.join(second, "tgrep"); + fs.writeFileSync(executable, "#!/bin/sh\n"); + fs.chmodSync(executable, 0o755); + + expect(resolveTgrepBinary(undefined, [first, second].join(path.delimiter))).toBe( + executable, + ); + } finally { + fs.rmSync(first, { recursive: true, force: true }); + fs.rmSync(second, { recursive: true, force: true }); + } + }); + + test("skips executable directories named tgrep", () => { + const first = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-path-directory-")); + const second = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-path-file-")); + try { + fs.mkdirSync(path.join(first, "tgrep")); + fs.chmodSync(path.join(first, "tgrep"), 0o755); + const executable = path.join(second, "tgrep"); + fs.writeFileSync(executable, "#!/bin/sh\n"); + fs.chmodSync(executable, 0o755); + + expect(resolveTgrepBinary(undefined, [first, second].join(path.delimiter))).toBe( + executable, + ); + } finally { + fs.rmSync(first, { recursive: true, force: true }); + fs.rmSync(second, { recursive: true, force: true }); + } + }); +}); + +describe("hasTgrepIndex", () => { + test("is true only for a .tgrep directory in the given cwd", () => { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-index-")); + const nested = path.join(cwd, "pkg"); + try { + expect(hasTgrepIndex(cwd)).toBe(false); + + fs.writeFileSync(path.join(cwd, TGREP_INDEX_DIR), "not a dir"); + expect(hasTgrepIndex(cwd)).toBe(false); + fs.rmSync(path.join(cwd, TGREP_INDEX_DIR)); + + fs.mkdirSync(path.join(cwd, TGREP_INDEX_DIR)); + fs.mkdirSync(nested); + expect(hasTgrepIndex(cwd)).toBe(true); + expect(hasTgrepIndex(nested)).toBe(false); + } finally { + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); +}); + +describe("formatTgrepResult", () => { + test("returns match output verbatim", () => { + expect(formatTgrepResult({ exit: 0, stdout: "a.ts:1:5:foo\n", stderr: "" })).toBe( + "a.ts:1:5:foo", + ); + }); + + test("treats exit 1 as no matches, not failure", () => { + expect(formatTgrepResult({ exit: 1, stdout: "", stderr: "" })).toBe( + "No matches found", + ); + }); + + test("throws on exit 2 with the stderr cause", () => { + expect(() => + formatTgrepResult({ exit: 2, stdout: "", stderr: "bad regex\n" }), + ).toThrow("bad regex"); + }); + + test("surfaces the stderr freshness warning above results", () => { + const out = formatTgrepResult({ + exit: 0, + stdout: "a.ts:1:5:foo\n", + stderr: "warning: no index - scanning every file\n", + }); + expect(out.startsWith("[tgrep: warning: no index - scanning every file]")).toBe(true); + expect(out).toContain("a.ts:1:5:foo"); + }); + + test("keeps the first stderr warning when there are no matches", () => { + expect( + formatTgrepResult({ + exit: 1, + stdout: "", + stderr: "warning: stale index\nadditional diagnostic\n", + }), + ).toBe("[tgrep: warning: stale index]\nNo matches found"); + }); + + test("uses a stable fallback when an exit 2 error has no stderr", () => { + expect(() => formatTgrepResult({ exit: 2, stdout: "", stderr: "" })).toThrow( + "tgrep search failed: unknown error", + ); + }); + + test("does not truncate output at the exact byte limit", () => { + const exact = "x".repeat(TGREP_OUTPUT_MAX_BYTES); + expect(formatTgrepResult({ exit: 0, stdout: exact, stderr: "" })).toBe(exact); + }); + + test("truncates oversized output with a narrowing hint", () => { + const big = `${"x".repeat(TGREP_OUTPUT_MAX_BYTES + 10)}\n`; + const out = formatTgrepResult({ exit: 0, stdout: big, stderr: "" }); + expect(out).toContain("truncated"); + expect(out).toContain("fileType/glob"); + expect(Buffer.byteLength(out)).toBeLessThanOrEqual(TGREP_OUTPUT_MAX_BYTES); + }); + + test("keeps a stderr notice plus body within the output byte cap", () => { + const warning = "warning: stale index"; + const body = "x".repeat(TGREP_OUTPUT_MAX_BYTES); + const out = formatTgrepResult({ exit: 0, stdout: body, stderr: warning }); + expect(out.startsWith(`[tgrep: ${warning}]\n`)).toBe(true); + expect(out).toContain("truncated"); + expect(Buffer.byteLength(out)).toBeLessThanOrEqual(TGREP_OUTPUT_MAX_BYTES); + }); + + test("measures truncation in bytes for multibyte output and preserves warnings", () => { + const stdout = "€".repeat(Math.ceil((TGREP_OUTPUT_MAX_BYTES + 4) / 3)); + const out = formatTgrepResult({ + exit: 0, + stdout, + stderr: "warning: stale index\nignored diagnostic\n", + }); + + expect(out.startsWith("[tgrep: warning: stale index]\n")).toBe(true); + expect(out).not.toContain("\uFFFD"); + expect(out).not.toContain("ignored diagnostic"); + expect(Buffer.byteLength(out)).toBeLessThanOrEqual(TGREP_OUTPUT_MAX_BYTES); + const hint = out.match(/\[truncated (\d+) bytes: narrow with fileType\/glob\]/); + expect(hint).not.toBeNull(); + expect(Number(hint![1])).toBeGreaterThanOrEqual( + Buffer.byteLength(stdout) - TGREP_OUTPUT_MAX_BYTES, + ); + const printed = out.slice( + "[tgrep: warning: stale index]\n".length, + out.indexOf("\n… [truncated"), + ); + expect(printed.endsWith("€")).toBe(true); + }); +}); + +describe("runTgrep", () => { + test("truncates partial stdout after exceeding the child-process buffer", async () => { + const out = await runTgrep( + process.execPath, + ["-e", `process.stdout.write("x".repeat(${TGREP_OUTPUT_MAX_BYTES * 2 + 1}))`], + { cwd: "/tmp" }, + ); + expect(out).toContain("truncated"); + expect(out).toContain("fileType/glob"); + }); + + test("rejects aborted calls before spawning", async () => { + const controller = new AbortController(); + controller.abort(); + const exec = mock(async () => ({ exit: 0, stdout: "", stderr: "" })); + await expect( + runTgrep( + "/bin/false", + ["--", "x", "."], + { cwd: "/tmp", signal: controller.signal }, + exec, + ), + ).rejects.toThrow("Operation aborted"); + expect(exec).not.toHaveBeenCalled(); + }); + + test("maps a mid-run abort to Operation aborted", async () => { + const controller = new AbortController(); + const running = runTgrep(process.execPath, ["-e", "setTimeout(() => {}, 30000)"], { + cwd: "/tmp", + signal: controller.signal, + }); + await new Promise((resolve) => setImmediate(resolve)); + controller.abort(); + await expect(running).rejects.toThrow("Operation aborted"); + }); + + test("forwards a custom time budget to the executor", async () => { + let seen: number | undefined; + await runTgrep( + "/fake/tgrep", + ["--", "x", "."], + { cwd: "/tmp", timeoutMs: 12_000 }, + async (_bin, _args, opts) => { + seen = opts.timeoutMs; + return { exit: 1, stdout: "", stderr: "" }; + }, + ); + expect(seen).toBe(12_000); + }); + + test("maps the real process no-match exit and stderr warning", async () => { + const out = await runTgrep( + process.execPath, + ["-e", 'process.stderr.write("warning: stale index\\n"); process.exit(1)'], + { cwd: "/tmp" }, + ); + expect(out).toBe("[tgrep: warning: stale index]\nNo matches found"); + }); + + test("wraps child-process launch failures with tgrep context", async () => { + await expect( + runTgrep("/definitely/missing/tgrep", ["--", "x", "."], { cwd: "/tmp" }), + ).rejects.toThrow("tgrep failed to run:"); + }); + + test("delegates to the injected executor and formats", async () => { + const seen: unknown[] = []; + const controller = new AbortController(); + const out = await runTgrep( + "/fake/tgrep", + ["--", "x", "."], + { cwd: "/tmp", signal: controller.signal }, + async (bin, args, opts) => { + seen.push([bin, args, opts]); + return { exit: 0, stdout: "a.ts:2:1:x\n", stderr: "" }; + }, + ); + expect(out).toBe("a.ts:2:1:x"); + expect(seen).toEqual([ + ["/fake/tgrep", ["--", "x", "."], { cwd: "/tmp", signal: controller.signal }], + ]); + }); +});