From 9e2da51020fb705497705730b83162d757ff4801 Mon Sep 17 00:00:00 2001 From: chefadmin-netizen Date: Wed, 9 Sep 2026 06:37:59 +0200 Subject: [PATCH 01/11] feat(pi-tools): add gated tgrep trigram-index search tool Registers a mode-independent tgrep tool only when the binary resolves (TGREP_BIN, tgrepBinPath, or PATH) and enableTgrep is not false. Literal-by-default, vimgrep rows, exit 1 as no-match, stderr freshness warning surfaced, index-unsafe flags excluded. Adds /tgrep-status, schema keys, compat/README/CHANGELOG docs and 26 tests. --- docs/compatibility.md | 21 ++- packages/pi-tools/CHANGELOG.md | 2 + packages/pi-tools/README.md | 26 +++ packages/pi-tools/pi-tools.schema.json | 10 ++ packages/pi-tools/src/config.ts | 10 +- packages/pi-tools/src/index.ts | 150 +++++++++++++++++ packages/pi-tools/src/tgrep.ts | 173 +++++++++++++++++++ packages/pi-tools/test/config.test.ts | 4 + packages/pi-tools/test/extension.test.ts | 119 +++++++++++++ packages/pi-tools/test/tgrep.test.ts | 204 +++++++++++++++++++++++ 10 files changed, 711 insertions(+), 8 deletions(-) create mode 100644 packages/pi-tools/src/tgrep.ts create mode 100644 packages/pi-tools/test/tgrep.test.ts diff --git a/docs/compatibility.md b/docs/compatibility.md index 0e31bc6..2024d20 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -8,21 +8,22 @@ ## Tool surface -The extension registers two custom tools and one completion surface (names resolved from -mode; see below): +The extension registers two FFF tools, an optional `tgrep` tool, and one completion surface +(FFF names resolved from mode; see below): | Tool | Purpose | |---|---| | `fffind` | Typo-resistant file discovery + frecency-ranked access | | `ffgrep` | SIMD content search | +| `tgrep` | Trigram-indexed content search via an external binary (registered only when found) | -Source of truth: `packages/pi-tools/src/index.ts:52-53` (`grep: "ffgrep", find: "fffind"`), +Source of truth: `packages/pi-tools/src/index.ts:60` (`FFF_TOOL_NAMES`: `grep: "ffgrep", find: "fffind"`), `tgrep` via `queueTool(() => TGREP_TOOL_NAME, …)` gated on `resolveTgrepBinary` (`src/tgrep.ts`), registered via `queueTool(() => toolNames.grep, …)` / `queueTool(() => toolNames.find, …)`. ### Parameters Parameter shapes, defaults and allowed values are defined in the tool schemas in -`packages/pi-tools/src/index.ts` (`grepSchema`, `findSchema`, `multiGrepSchema`). +`packages/pi-tools/src/index.ts` (`grepSchema`, `findSchema`, `multiGrepSchema`, `tgrepSchema`). `pi-tools.schema.json` documents **config keys only** (mode, DB paths, scan toggles), not tool parameters. Compatibility surface = the parameter **names**, **types**, **enums**, and **defaults** for both tools, plus their pagination behavior: @@ -38,12 +39,18 @@ tool parameters. Compatibility surface = the parameter **names**, **types**, **e The file picker UI exists only in `tools-and-ui`; the `@`-mention completion surface is disabled only in `tools-only` (`shouldEnableMentions`: `currentMode !== 'tools-only'`). A tool disappearing when mode changes is expected; a mode value being removed is breaking. +- **Exit codes (`tgrep`):** exit `0` returns `file:line:col:text` rows, exit `1` reports + `No matches found` (not a failure), exit `2` throws with the stderr cause. A leading + `[tgrep: ...]` line always carries the binary's stderr, including the "no index" + freshness warning. Output truncates at `TGREP_OUTPUT_MAX_BYTES` with a narrowing hint. + `tgrep` is mode-independent: it keeps its name in every mode and is gated only on + binary availability plus `enableTgrep`. ### Config precedence Config values are resolved in this priority order -(`packages/pi-tools/src/index.ts:272` `getConfigValue`, read at startup -`resolveStartupConfig`): +(`packages/pi-tools/src/index.ts:291` `getConfigValue`, read at startup +`resolveStartupConfig`; `tgrep` resolves once at extension load): ``` flag (--fff-mode / --fff-frecency-db / --fff-history-db / …) @@ -61,6 +68,8 @@ Concrete defaults: | history DB | `--fff-history-db` | `FFF_HISTORY_DB` | `config.historyDbPath` | (platform db path) | | root scan | `--fff-enable-root-scan` | `FFF_ENABLE_ROOT_SCAN` | `config.enableFsRootScanning` | `false` | | home scan | `--fff-enable-home-scan` | `FFF_ENABLE_HOME_SCAN` | `config.enableHomeDirScanning` | `true` | +| tgrep binary | — (no CLI flag by design) | `TGREP_BIN` | `config.tgrepBinPath` | `PATH` lookup | +| tgrep toggle | — (config file only) | — | `config.enableTgrep` | `true` | Mode valid values (`packages/pi-tools/src/config.ts:8` `VALID_MODES`): `tools-and-ui`, `tools-only`, `override`. diff --git a/packages/pi-tools/CHANGELOG.md b/packages/pi-tools/CHANGELOG.md index f134ee8..901f783 100644 --- a/packages/pi-tools/CHANGELOG.md +++ b/packages/pi-tools/CHANGELOG.md @@ -4,6 +4,8 @@ ### Added +- Added the `tgrep` tool: trigram-indexed content search through an external [tgrep](https://github.com/microsoft/tgrep) binary. Literal by default, `file:line:col:text` output, exit 1 reported as no-match. Registered only when the binary resolves (`TGREP_BIN`, `tgrepBinPath`, or `PATH`) and `enableTgrep` is not `false`; mode-independent. +- Added the `/tgrep-status` command showing tgrep index and server status for the workspace. - Added `ffgrep.maxMatchesPerFile` to keep a single generated or vendored file from dominating a result page. - Added `ffgrep.compact` for deterministic `path:line:match` output without context blocks. diff --git a/packages/pi-tools/README.md b/packages/pi-tools/README.md index 38dbb82..85a0db7 100644 --- a/packages/pi-tools/README.md +++ b/packages/pi-tools/README.md @@ -84,6 +84,27 @@ Use `fffind` for **paths**. Use `ffgrep` when you know text that should occur in Use a concrete substring, identifier, or expression. A wildcard-only expression such as `.*` is rejected because it is not an efficient way to read an entire file. Keep the default grouped output when context matters; use `compact: true` when the next action only needs stable path-and-line references. Set `maxMatchesPerFile` when a generated or vendored file could otherwise dominate the page. Both options are additive and leave existing defaults unchanged. +### `tgrep` + +`tgrep` searches file content through an external [tgrep](https://github.com/microsoft/tgrep) binary: trigram-indexed search with a client/server architecture, fastest on large repositories with a built index. The tool is registered only when the binary is found and `enableTgrep` is not disabled; it keeps the name `tgrep` in every mode. Prefer `ffgrep` for fuzzy, frecency-ranked search on small and medium repositories. + +| Parameter | Type | Description | +| --- | --- | --- | +| `pattern` | string | Literal text by default; set `literal: false` for a regular expression. Patterns like `serve` cannot parse as subcommands. | +| `path` | string, optional | Directory or file to search, relative to the workspace; default is the workspace root. Globs go in `glob`. | +| `glob` | string or string array, optional | Repeatable file glob filter such as `*.{ts,tsx}`. | +| `fileType` | string or string array, optional | Repeatable file type filter such as `rust`, `py`, or `js`. | +| `literal` | boolean, optional | Treats the pattern as literal text; default is `true`. | +| `caseSensitive` | boolean, optional | Forces case-sensitive matching; default is smart-case. | +| `wholeWord` | boolean, optional | Matches whole words only. | +| `filesOnly` | boolean, optional | Prints only filenames with matches. | +| `count` | boolean, optional | Prints the match count per file. | +| `context` | number, optional | Context lines before and after a match; range 0–20. | +| `maxCount` | number, optional | Limits matches per file. | +| `noIndex` | boolean, optional | Reads files from disk instead of the index. Use after your own edits when the latest content must be visible. | + +Output is `file:line:col:text` rows. Exit code 1 (no match) is reported as `No matches found`, not as a failure. A leading `[tgrep: ...]` line carries the binary's stderr, including the "no index" warning: without an index (`tgrep index .` or `tgrep serve .`) the search scans every file like grep and may be slow. Only index-safe flags are forwarded; full-scan forcers (`--hidden`, `--no-ignore`, `-u`, `-a`, `--encoding`) are excluded by design. + ### Optional multi-pattern search Set `PI_FFF_MULTIGREP=1` before starting Pi to enable the experimental `fff-multi-grep` tool. It searches for **any** of several literal patterns in one request and is useful when an agent must check known naming variants together. @@ -105,6 +126,7 @@ The tool is opt-in while its interaction pattern is evaluated. Do not depend on | `/fff-mode [tools-and-ui \| tools-only \| override]` | Shows the current mode or records a mode for the current session. | | `/fff-health` | Displays the engine version, mode, Git integration, index status, persistence status, and active scan progress. | | `/fff-rescan` | Requests a new file scan for the active workspace. | +| `/tgrep-status` | Shows tgrep index and server status for the workspace. | ## Persistent configuration @@ -127,9 +149,13 @@ Create `pi-tools.json` in Pi’s agent directory. The default location is `~/.pi | `historyDbPath` | string | Auto-resolved | Location for query-selection history. | | `enableFsRootScanning` | boolean | `false` | Explicitly allows scans started from `/`. | | `enableHomeDirScanning` | boolean | `true` | Allows scanning when Pi starts in the home directory. | +| `enableTgrep` | boolean | `true` | Registers the `tgrep` tool when the binary is found. | +| `tgrepBinPath` | string | PATH lookup | Explicit path to the `tgrep` binary. | Malformed configuration, unknown fields, and invalid values prevent the extension from loading and identify the configuration path in the error. `/fff-mode` changes session state only; it does not edit this file. +The `tgrep` binary resolves as `TGREP_BIN` environment variable, then `tgrepBinPath`, then a `tgrep` executable on `PATH`. An explicit path that is set but not executable disables the tool instead of falling back, so a typo surfaces instead of silently changing the search backend. + ## Database resolution Frecency and history paths resolve independently in the following order: diff --git a/packages/pi-tools/pi-tools.schema.json b/packages/pi-tools/pi-tools.schema.json index ea6b63c..449a564 100644 --- a/packages/pi-tools/pi-tools.schema.json +++ b/packages/pi-tools/pi-tools.schema.json @@ -36,6 +36,16 @@ "type": "boolean", "default": true, "description": "Allows indexing when pi is launched from the home directory." + }, + "enableTgrep": { + "type": "boolean", + "default": true, + "description": "Registers the tgrep trigram-index search tool when the tgrep binary is found." + }, + "tgrepBinPath": { + "type": "string", + "minLength": 1, + "description": "Explicit path to the tgrep binary (overrides TGREP_BIN and PATH lookup)." } } } diff --git a/packages/pi-tools/src/config.ts b/packages/pi-tools/src/config.ts index f0d2569..d838e15 100644 --- a/packages/pi-tools/src/config.ts +++ b/packages/pi-tools/src/config.ts @@ -16,6 +16,8 @@ export interface FffConfig { historyDbPath?: string; enableFsRootScanning?: boolean; enableHomeDirScanning?: boolean; + enableTgrep?: boolean; + tgrepBinPath?: string; } const CONFIG_KEYS = new Set([ @@ -25,6 +27,8 @@ const CONFIG_KEYS = new Set([ "historyDbPath", "enableFsRootScanning", "enableHomeDirScanning", + "enableTgrep", + "tgrepBinPath", ]); export function loadConfig(agentDir = piDataDir()): FffConfig { @@ -75,8 +79,10 @@ export function loadConfig(agentDir = piDataDir()): FffConfig { validateString(configPath, parsed, "$schema"); validateString(configPath, parsed, "frecencyDbPath"); validateString(configPath, parsed, "historyDbPath"); + validateString(configPath, parsed, "tgrepBinPath"); validateBoolean(configPath, parsed, "enableFsRootScanning"); validateBoolean(configPath, parsed, "enableHomeDirScanning"); + validateBoolean(configPath, parsed, "enableTgrep"); return parsed as FffConfig; } @@ -96,7 +102,7 @@ function isRecord(value: unknown): value is Record { function validateString( configPath: string, config: Record, - key: "$schema" | "frecencyDbPath" | "historyDbPath", + key: "$schema" | "frecencyDbPath" | "historyDbPath" | "tgrepBinPath", ): void { const value = config[key]; if (value !== undefined && (typeof value !== "string" || value.length === 0)) { @@ -107,7 +113,7 @@ function validateString( function validateBoolean( configPath: string, config: Record, - key: "enableFsRootScanning" | "enableHomeDirScanning", + key: "enableFsRootScanning" | "enableHomeDirScanning" | "enableTgrep", ): void { const value = config[key]; if (value !== undefined && typeof value !== "boolean") { diff --git a/packages/pi-tools/src/index.ts b/packages/pi-tools/src/index.ts index feab8df..1a98a81 100644 --- a/packages/pi-tools/src/index.ts +++ b/packages/pi-tools/src/index.ts @@ -23,6 +23,15 @@ import { type FffMode, loadConfig, VALID_MODES } from "./config"; import { FilePickerFactory } from "./file-picker"; import { isHomeDir, resolveDbPaths } from "./paths"; import { buildQuery } from "./query"; +import { + buildTgrepArgs, + resolveSearchRoot, + resolveTgrepBinary, + runTgrep, + TGREP_BIN_ENV, + TGREP_CONTEXT_MAX, + TGREP_TOOL_NAME, +} from "./tgrep"; export { SCAN_TIMEOUT_MS } from "./sdk"; @@ -268,6 +277,16 @@ export default function fffExtension(pi: ExtensionAPI) { const config = loadConfig(); + // tgrep availability is session-independent: resolve once at load so the + // tool surface is fixed before registration. Explicit path wins over PATH. + const tgrepBin: string | undefined = + config.enableTgrep === false + ? undefined + : resolveTgrepBinary( + process.env[TGREP_BIN_ENV]?.trim() || config.tgrepBinPath, + process.env.PATH, + ); + // Resolve startup options with flag > environment > file > fallback. function getConfigValue( flagName: string, @@ -1168,6 +1187,115 @@ export default function fffExtension(pi: ExtensionAPI) { }); } + // tgrep is mode-independent and binary-gated: no binary, no tool. + if (tgrepBin !== undefined) { + const bin = tgrepBin; + const tgrepSchema = Type.Object({ + pattern: Type.String({ + description: + "Search pattern. Literal text by default; set literal: false for regex.", + }), + path: Type.Optional( + Type.String({ + description: + "Directory or file to search, relative to the workspace (default: workspace root). Globs go in glob.", + }), + ), + glob: Type.Optional( + Type.Union([Type.String(), Type.Array(Type.String())], { + description: "Repeatable file glob filter, e.g. '*.{ts,tsx}'.", + }), + ), + fileType: Type.Optional( + Type.Union([Type.String(), Type.Array(Type.String())], { + description: "Repeatable file type filter, e.g. 'rust', 'py', 'js'.", + }), + ), + literal: Type.Optional( + Type.Boolean({ + description: + "Treat pattern as literal text (default true). Set false for regex.", + }), + ), + caseSensitive: Type.Optional( + Type.Boolean({ + description: "Force case-sensitive matching. Default is smart-case.", + }), + ), + wholeWord: Type.Optional(Type.Boolean({ description: "Match whole words only." })), + filesOnly: Type.Optional( + Type.Boolean({ description: "Print only filenames with matches." }), + ), + count: Type.Optional(Type.Boolean({ description: "Print match count per file." })), + context: Type.Optional( + Type.Number({ + description: `Context lines before+after each match (0-${TGREP_CONTEXT_MAX})`, + }), + ), + maxCount: Type.Optional(Type.Number({ description: "Limit matches per file." })), + noIndex: Type.Optional( + Type.Boolean({ + description: + "Read files from disk instead of the index. Use after your own edits when the latest content must be visible.", + }), + ), + }); + + queueTool(() => TGREP_TOOL_NAME, { + description: `Trigram-indexed content search for large repos (tgrep). Literal by default, vimgrep output. Falls back to a full scan with a stderr warning when no index exists. Default limit is output-capped; narrow with fileType/glob.`, + promptSnippet: "Trigram-indexed content search", + promptGuidelines: (names) => [ + `${names.grep}: prefer for fuzzy/frecency-ranked search on small and medium repos.`, + `${TGREP_TOOL_NAME}: prefer on large repos with a built tgrep index; fastest path.`, + `${TGREP_TOOL_NAME}: keep literal: true (default) for symbols; set false for regex.`, + `${TGREP_TOOL_NAME}: narrow with fileType/glob before raising maxCount.`, + `${TGREP_TOOL_NAME}: use filesOnly: true first on broad queries, then search specific files.`, + `${TGREP_TOOL_NAME}: after your own edits, pass noIndex: true or use ${names.grep}; the index lags watcher events.`, + `${TGREP_TOOL_NAME}: a '[tgrep: ...]' line reports index freshness, never drop it from summaries.`, + ], + parameters: tgrepSchema, + + async execute(_toolCallId, params, signal) { + if (signal?.aborted) throw new Error("Operation aborted"); + const root = resolveSearchRoot(params.path, activeCwd); + const output = await runTgrep( + bin, + buildTgrepArgs({ + pattern: params.pattern, + root, + literal: params.literal, + caseSensitive: params.caseSensitive, + wholeWord: params.wholeWord, + fileType: params.fileType, + glob: params.glob, + filesOnly: params.filesOnly, + count: params.count, + context: params.context, + maxCount: params.maxCount, + noIndex: params.noIndex, + }), + { cwd: activeCwd, signal }, + ); + return { content: [{ type: "text", text: output }], details: {} }; + }, + + renderCall(args, theme, context) { + const text = (context.lastComponent as Text | undefined) ?? new Text("", 0, 0); + text.setText( + theme.fg("toolTitle", theme.bold(TGREP_TOOL_NAME)) + + " " + + theme.fg("accent", `/${args?.pattern ?? ""}/`) + + theme.fg("toolOutput", ` in ${args?.path ?? "."}`), + ); + return text; + }, + + renderResult(result, options, theme, context) { + return renderTextResult(result, options, theme, context, 15); + }, + }); + } + pi.registerCommand("fff-mode", { description: "Show or set FFF mode: /fff-mode [tools-and-ui | tools-only | override]", handler: async (args, ctx) => { @@ -1262,4 +1390,26 @@ export default function fffExtension(pi: ExtensionAPI) { ctx.ui.notify("FFF rescan triggered", "info"); }, }); + + pi.registerCommand("tgrep-status", { + description: "Show tgrep index and server status for the workspace", + handler: async (_args, ctx) => { + const bin = tgrepBin; + if (!bin) { + ctx.ui.notify("tgrep binary not found (TGREP_BIN or PATH)", "warning"); + return; + } + try { + const output = await runTgrep(bin, ["status", activeCwd], { + cwd: activeCwd, + }); + ctx.ui.notify(output || "tgrep status: no output", "info"); + } catch (error: unknown) { + ctx.ui.notify( + `tgrep status failed: ${error instanceof Error ? error.message : String(error)}`, + "error", + ); + } + }, + }); } diff --git a/packages/pi-tools/src/tgrep.ts b/packages/pi-tools/src/tgrep.ts new file mode 100644 index 0000000..a8b7300 --- /dev/null +++ b/packages/pi-tools/src/tgrep.ts @@ -0,0 +1,173 @@ +import { execFile } from "node:child_process"; +import fs from "node:fs"; +import path from "node:path"; +import { promisify } from "node:util"; + +export const TGREP_BIN_ENV = "TGREP_BIN"; +export const TGREP_TOOL_NAME = "tgrep"; +export const TGREP_TIME_BUDGET_MS = 30_000; +export const TGREP_OUTPUT_MAX_BYTES = 200_000; +export const TGREP_CONTEXT_MAX = 20; + +export interface TgrepSearchOptions { + pattern: string; + root: string; + literal?: boolean; + caseSensitive?: boolean; + wholeWord?: boolean; + fileType?: string | string[]; + glob?: string | string[]; + filesOnly?: boolean; + count?: boolean; + context?: number; + maxCount?: number; + noIndex?: boolean; +} + +export interface TgrepResult { + exit: number; + stdout: string; + stderr: string; +} + +type ExecFn = ( + bin: string, + args: string[], + opts: { cwd: string; signal?: AbortSignal }, +) => Promise; + +function repeatAll(values: string | string[] | undefined): string[] { + if (!values) return []; + return Array.isArray(values) ? values : [values]; +} + +// Allowlist of index-friendly flags. Full-scan forcers (--hidden, --no-ignore, +// -u, -a/--text, -E/--encoding) are excluded: they silently bypass the index. +export function buildTgrepArgs(options: TgrepSearchOptions): string[] { + const args = ["--vimgrep"]; + args.push(options.caseSensitive ? "--case-sensitive" : "--smart-case"); + if (options.literal !== false) args.push("--fixed-strings"); + if (options.wholeWord) args.push("--word-regexp"); + for (const t of repeatAll(options.fileType)) args.push("--type", t); + for (const g of repeatAll(options.glob)) args.push("--glob", g); + if (options.filesOnly) args.push("--files-with-matches"); + if (options.count) args.push("--count"); + const context = clampContext(options.context); + if (context > 0) args.push("-A", String(context), "-B", String(context)); + if (options.maxCount !== undefined) + args.push("--max-count", String(Math.max(1, Math.floor(options.maxCount)))); + if (options.noIndex) args.push("--no-index"); + // Separator keeps patterns like "serve" or "-x" from parsing as subcommands. + args.push("--", options.pattern, options.root); + return args; +} + +// Resolve the search root under cwd. Anything escaping the workspace is rejected. +export function resolveSearchRoot(pathParam: string | undefined, cwd: string): string { + const raw = (pathParam ?? ".").trim() || "."; + if (/[*?[{]/.test(raw)) + throw new Error(`tgrep path must be a directory or file, not a glob: ${raw}`); + const resolved = path.resolve(cwd, raw); + const relative = path.relative(cwd, resolved); + if (relative.startsWith("..") || path.isAbsolute(relative)) + throw new Error(`tgrep path must stay inside the workspace: ${raw}`); + return relative === "" ? "." : relative.split(path.sep).join("/"); +} + +function isExecutable(file: string): boolean { + try { + fs.accessSync(file, fs.constants.X_OK); + return fs.statSync(file).isFile(); + } catch { + return false; + } +} + +// Explicit path wins; a set-but-missing explicit path disables rather than +// falling back to PATH so a typo surfaces instead of silently changing tools. +export function resolveTgrepBinary(explicit?: string, pathEnv = ""): string | undefined { + const trimmed = explicit?.trim(); + if (trimmed) return isExecutable(trimmed) ? trimmed : undefined; + for (const dir of pathEnv.split(path.delimiter)) { + if (!dir) continue; + const candidate = path.join(dir, "tgrep"); + if (isExecutable(candidate)) return candidate; + } + return undefined; +} + +const execFileAsync = promisify(execFile); + +async function defaultExec( + bin: string, + args: string[], + opts: { cwd: string; signal?: AbortSignal }, +): Promise { + try { + const { stdout, stderr } = await execFileAsync(bin, args, { + cwd: opts.cwd, + signal: opts.signal, + timeout: TGREP_TIME_BUDGET_MS, + maxBuffer: TGREP_OUTPUT_MAX_BYTES * 2, + }); + return { exit: 0, stdout, stderr }; + } catch (error: unknown) { + const execError = error as { + code?: number; + stdout?: string; + stderr?: string; + message?: string; + }; + if (typeof execError.code === "number") + return { + exit: execError.code, + stdout: execError.stdout ?? "", + stderr: execError.stderr ?? "", + }; + throw new Error(`tgrep failed to run: ${execError.message ?? String(error)}`); + } +} + +export async function runTgrep( + bin: string, + args: string[], + opts: { cwd: string; signal?: AbortSignal }, + exec: ExecFn = defaultExec, +): Promise { + if (opts.signal?.aborted) throw new Error("Operation aborted"); + const result = await runTgrepRaw(bin, args, opts, exec); + return formatTgrepResult(result); +} + +export async function runTgrepRaw( + bin: string, + args: string[], + opts: { cwd: string; signal?: AbortSignal }, + exec: ExecFn = defaultExec, +): Promise { + return exec(bin, args, opts); +} + +// Exit 1 is "no match", not a failure. stderr always surfaces: it carries the +// "no index" warning that decides whether the result reflects current files. +export function formatTgrepResult(result: TgrepResult): string { + const warning = result.stderr.trim().split("\n")[0]?.trim(); + const notice = warning ? `[tgrep: ${warning}]\n` : ""; + if (result.exit === 2) + throw new Error(`tgrep search failed: ${warning || "unknown error"}`); + const body = truncateBytes(result.stdout.trim()); + if (result.exit === 1 || body === "") return `${notice}No matches found`; + return `${notice}${body}`; +} + +function truncateBytes(text: string): string { + const buf = Buffer.from(text); + if (buf.length <= TGREP_OUTPUT_MAX_BYTES) return text; + const head = buf.subarray(0, TGREP_OUTPUT_MAX_BYTES).toString(); + return `${head}\n… [truncated ${buf.length - TGREP_OUTPUT_MAX_BYTES} bytes: narrow with fileType/glob]`; +} + +function clampContext(context: number | undefined): number { + if (!context || context < 0) return 0; + return Math.min(Math.floor(context), TGREP_CONTEXT_MAX); +} diff --git a/packages/pi-tools/test/config.test.ts b/packages/pi-tools/test/config.test.ts index 6231e9c..df0aead 100644 --- a/packages/pi-tools/test/config.test.ts +++ b/packages/pi-tools/test/config.test.ts @@ -31,6 +31,8 @@ describe("loadConfig", () => { historyDbPath: "/data/history", enableFsRootScanning: true, enableHomeDirScanning: false, + enableTgrep: false, + tgrepBinPath: "/usr/local/bin/tgrep", }; writeConfig(config); @@ -66,6 +68,8 @@ describe("loadConfig", () => { [{ historyDbPath: false }, '"historyDbPath" must be a non-empty string'], [{ enableFsRootScanning: 1 }, '"enableFsRootScanning" must be a boolean'], [{ enableHomeDirScanning: "false" }, '"enableHomeDirScanning" must be a boolean'], + [{ enableTgrep: "yes" }, '"enableTgrep" must be a boolean'], + [{ tgrepBinPath: "" }, '"tgrepBinPath" must be a non-empty string'], ]; for (const [config, message] of cases) { diff --git a/packages/pi-tools/test/extension.test.ts b/packages/pi-tools/test/extension.test.ts index 6b59138..77fbf27 100644 --- a/packages/pi-tools/test/extension.test.ts +++ b/packages/pi-tools/test/extension.test.ts @@ -187,6 +187,7 @@ const CONFIG_ENV_KEYS = [ "FFF_HISTORY_DB", "FFF_ENABLE_ROOT_SCAN", "FFF_ENABLE_HOME_SCAN", + "TGREP_BIN", ] as const; const savedEnv: Record = {}; @@ -705,3 +706,121 @@ describe("pi-tools autocomplete registration", () => { expect(current.shouldTriggerFileCompletion).toHaveBeenCalledTimes(1); }); }); + +describe("pi-tools tgrep integration", () => { + let binDir = ""; + + function writeFakeBin(name: string, body: string): string { + binDir = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-")); + const bin = path.join(binDir, name); + fs.writeFileSync(bin, `#!/bin/sh\n${body}\n`); + fs.chmodSync(bin, 0o755); + return bin; + } + + function toolNames(setup: { pi: { registerTool: any } }): string[] { + return setup.pi.registerTool.mock.calls.map(([tool]: [any]) => tool.name); + } + + test("registers tgrep and tgrep-status when the binary is found", async () => { + process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + const setup = await start(); + try { + expect(toolNames(setup)).toContain("tgrep"); + expect(setup.commands.has("tgrep-status")).toBe(true); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + } + }); + + test("skips tgrep when the binary is missing", async () => { + process.env.TGREP_BIN = "/nonexistent/tgrep"; + const setup = await start(); + try { + expect(toolNames(setup)).not.toContain("tgrep"); + expect(setup.commands.has("tgrep-status")).toBe(true); + } finally { + await shutdown(setup); + } + }); + + test("skips tgrep when enableTgrep is false", async () => { + process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + writeConfig({ enableTgrep: false }); + const setup = await start(); + try { + expect(toolNames(setup)).not.toContain("tgrep"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + } + }); + + test("executes searches through the binary and reports no-match", async () => { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + const setup = await start(undefined, cwd); + try { + const tool = setup.pi.registerTool.mock.calls + .map(([t]: [any]) => t) + .find((t: any) => t.name === "tgrep"); + expect(tool).toBeDefined(); + const result = await tool.execute("call-1", { pattern: "hello" }, undefined); + expect(result.content[0].text).toBe("No matches found"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("returns vimgrep rows from the binary", async () => { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + process.env.TGREP_BIN = writeFakeBin("tgrep", "printf 'src/a.ts:3:7:hello\\n'"); + const setup = await start(undefined, cwd); + try { + const tool = setup.pi.registerTool.mock.calls + .map(([t]: [any]) => t) + .find((t: any) => t.name === "tgrep"); + const result = await tool.execute( + "call-1", + { pattern: "hello", path: "src/" }, + undefined, + ); + expect(result.content[0].text).toBe("src/a.ts:3:7:hello"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("/tgrep-status reports the server status", async () => { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + process.env.TGREP_BIN = writeFakeBin("tgrep", "echo 'Indexing: complete'"); + const setup = await start(undefined, cwd); + try { + await setup.commands.get("tgrep-status").handler("", setup.ctx); + expect(setup.ctx.ui.notify).toHaveBeenCalledWith("Indexing: complete", "info"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("/tgrep-status warns without a binary", async () => { + process.env.TGREP_BIN = "/nonexistent/tgrep"; + const setup = await start(); + try { + await setup.commands.get("tgrep-status").handler("", setup.ctx); + expect(setup.ctx.ui.notify).toHaveBeenCalledWith( + "tgrep binary not found (TGREP_BIN or PATH)", + "warning", + ); + } finally { + await shutdown(setup); + } + }); +}); diff --git a/packages/pi-tools/test/tgrep.test.ts b/packages/pi-tools/test/tgrep.test.ts new file mode 100644 index 0000000..993208e --- /dev/null +++ b/packages/pi-tools/test/tgrep.test.ts @@ -0,0 +1,204 @@ +import { describe, expect, test } from "bun:test"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +import { + buildTgrepArgs, + formatTgrepResult, + resolveSearchRoot, + resolveTgrepBinary, + runTgrep, + TGREP_CONTEXT_MAX, + TGREP_OUTPUT_MAX_BYTES, +} from "../src/tgrep"; + +describe("buildTgrepArgs", () => { + test("defaults to vimgrep, smart-case and literal with separator", () => { + expect(buildTgrepArgs({ pattern: "foo", root: "." })).toEqual([ + "--vimgrep", + "--smart-case", + "--fixed-strings", + "--", + "foo", + ".", + ]); + }); + + test("keeps subcommand-like patterns from parsing as subcommands", () => { + const args = buildTgrepArgs({ pattern: "serve", root: "src/" }); + expect(args.slice(-3)).toEqual(["--", "serve", "src/"]); + }); + + test("maps search options to the allowlisted flags", () => { + expect( + buildTgrepArgs({ + pattern: "fn main", + root: ".", + literal: false, + caseSensitive: true, + wholeWord: true, + fileType: ["rust", "py"], + glob: "src/**", + filesOnly: true, + noIndex: true, + }), + ).toEqual([ + "--vimgrep", + "--case-sensitive", + "--word-regexp", + "--type", + "rust", + "--type", + "py", + "--glob", + "src/**", + "--files-with-matches", + "--no-index", + "--", + "fn main", + ".", + ]); + }); + + test("emits count and per-file cap", () => { + const args = buildTgrepArgs({ pattern: "x", root: ".", count: true, maxCount: 0 }); + expect(args).toContain("--count"); + expect(args).toContain("--max-count"); + expect(args[args.indexOf("--max-count") + 1]).toBe("1"); + }); + + test("splits context into before/after and clamps to the max", () => { + const args = buildTgrepArgs({ + pattern: "x", + root: ".", + context: TGREP_CONTEXT_MAX + 100, + }); + expect(args).toContain("-A"); + expect(args).toContain("-B"); + expect(args[args.indexOf("-A") + 1]).toBe(String(TGREP_CONTEXT_MAX)); + }); + + test("never emits full-scan forcing flags", () => { + const args = buildTgrepArgs({ pattern: "x", root: "." }).join(" "); + for (const banned of ["--hidden", "--no-ignore", "--text", "--encoding", "-u"]) { + expect(args.includes(banned)).toBe(false); + } + }); +}); + +describe("resolveSearchRoot", () => { + const cwd = path.resolve("/tmp/workspace"); + + test("defaults to the workspace root", () => { + expect(resolveSearchRoot(undefined, cwd)).toBe("."); + }); + + test("keeps directories and files inside the workspace", () => { + expect(resolveSearchRoot("src/", cwd)).toBe("src"); + expect(resolveSearchRoot("src/main.ts", cwd)).toBe("src/main.ts"); + }); + + test("rejects globs and escapes", () => { + expect(() => resolveSearchRoot("src/**/*.ts", cwd)).toThrow("not a glob"); + expect(() => resolveSearchRoot("../outside", cwd)).toThrow("inside the workspace"); + expect(() => resolveSearchRoot("/etc/passwd", cwd)).toThrow("inside the workspace"); + }); +}); + +describe("resolveTgrepBinary", () => { + test("prefers an explicit executable path", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-bin-")); + try { + const bin = path.join(dir, "tgrep"); + fs.writeFileSync(bin, "#!/bin/sh\necho hi\n"); + fs.chmodSync(bin, 0o755); + expect(resolveTgrepBinary(bin, "")).toBe(bin); + expect(resolveTgrepBinary(path.join(dir, "missing"), "")).toBeUndefined(); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); + + test("finds tgrep on PATH and returns undefined when absent", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-path-")); + try { + const bin = path.join(dir, "tgrep"); + fs.writeFileSync(bin, "#!/bin/sh\n"); + fs.chmodSync(bin, 0o755); + expect(resolveTgrepBinary(undefined, dir)).toBe(bin); + expect(resolveTgrepBinary(undefined, path.join(dir, "empty"))).toBeUndefined(); + expect(resolveTgrepBinary(undefined, "")).toBeUndefined(); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); +}); + +describe("formatTgrepResult", () => { + test("returns match output verbatim", () => { + expect(formatTgrepResult({ exit: 0, stdout: "a.ts:1:5:foo\n", stderr: "" })).toBe( + "a.ts:1:5:foo", + ); + }); + + test("treats exit 1 as no matches, not failure", () => { + expect(formatTgrepResult({ exit: 1, stdout: "", stderr: "" })).toBe( + "No matches found", + ); + }); + + test("throws on exit 2 with the stderr cause", () => { + expect(() => + formatTgrepResult({ exit: 2, stdout: "", stderr: "bad regex\n" }), + ).toThrow("bad regex"); + }); + + test("surfaces the stderr freshness warning above results", () => { + const out = formatTgrepResult({ + exit: 0, + stdout: "a.ts:1:5:foo\n", + stderr: "warning: no index - scanning every file\n", + }); + expect(out.startsWith("[tgrep: warning: no index - scanning every file]")).toBe(true); + expect(out).toContain("a.ts:1:5:foo"); + }); + + test("truncates oversized output with a narrowing hint", () => { + const big = `${"x".repeat(TGREP_OUTPUT_MAX_BYTES + 10)}\n`; + const out = formatTgrepResult({ exit: 0, stdout: big, stderr: "" }); + expect(out).toContain("truncated"); + expect(out).toContain("fileType/glob"); + expect(Buffer.byteLength(out)).toBeLessThan(Buffer.byteLength(big) + 200); + }); +}); + +describe("runTgrep", () => { + test("rejects aborted calls before spawning", async () => { + const controller = new AbortController(); + controller.abort(); + await expect( + runTgrep( + "/bin/false", + ["--", "x", "."], + { cwd: "/tmp", signal: controller.signal }, + async () => ({ exit: 0, stdout: "", stderr: "" }), + ), + ).rejects.toThrow("Operation aborted"); + }); + + test("delegates to the injected executor and formats", async () => { + const seen: unknown[] = []; + const out = await runTgrep( + "/fake/tgrep", + ["--", "x", "."], + { cwd: "/tmp" }, + async (bin, args, opts) => { + seen.push([bin, args, opts.cwd]); + return { exit: 0, stdout: "a.ts:2:1:x\n", stderr: "" }; + }, + ); + expect(out).toBe("a.ts:2:1:x"); + expect(seen).toEqual([["/fake/tgrep", ["--", "x", "."], "/tmp"]]); + }); +}); From a32351c7f78e3f3cb08f2e1f3096633998bf78a5 Mon Sep 17 00:00:00 2001 From: chefadmin-netizen Date: Wed, 9 Sep 2026 07:46:05 +0200 Subject: [PATCH 02/11] chore: ignore tgrep index directories --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index b3bd507..d892411 100644 --- a/.gitignore +++ b/.gitignore @@ -96,3 +96,6 @@ Pods/ .env .env.local .env.*.local + +# tgrep trigram indexes (per-repo, rebuilt via `tgrep index .`) +.tgrep/ From c544c35ec0f21c761d38e691049c649e61dd1a38 Mon Sep 17 00:00:00 2001 From: "coderabbitai[bot]" <136622811+coderabbitai[bot]@users.noreply.github.com> Date: Sat, 12 Sep 2026 13:15:27 +0000 Subject: [PATCH 03/11] fix(pi-tools): handle tgrep output buffer overflow --- packages/pi-tools/pi-tools.schema.json | 2 +- packages/pi-tools/src/tgrep.ts | 4 +++- packages/pi-tools/test/tgrep.test.ts | 10 ++++++++++ 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/packages/pi-tools/pi-tools.schema.json b/packages/pi-tools/pi-tools.schema.json index 449a564..92e298a 100644 --- a/packages/pi-tools/pi-tools.schema.json +++ b/packages/pi-tools/pi-tools.schema.json @@ -45,7 +45,7 @@ "tgrepBinPath": { "type": "string", "minLength": 1, - "description": "Explicit path to the tgrep binary (overrides TGREP_BIN and PATH lookup)." + "description": "Explicit path to the tgrep binary; the TGREP_BIN environment variable takes precedence over config.tgrepBinPath, then PATH is used as a fallback." } } } diff --git a/packages/pi-tools/src/tgrep.ts b/packages/pi-tools/src/tgrep.ts index a8b7300..dfc755c 100644 --- a/packages/pi-tools/src/tgrep.ts +++ b/packages/pi-tools/src/tgrep.ts @@ -113,11 +113,13 @@ async function defaultExec( return { exit: 0, stdout, stderr }; } catch (error: unknown) { const execError = error as { - code?: number; + code?: number | string; stdout?: string; stderr?: string; message?: string; }; + if (execError.code === "ERR_CHILD_PROCESS_STDIO_MAXBUFFER") + return { exit: 0, stdout: execError.stdout ?? "", stderr: execError.stderr ?? "" }; if (typeof execError.code === "number") return { exit: execError.code, diff --git a/packages/pi-tools/test/tgrep.test.ts b/packages/pi-tools/test/tgrep.test.ts index 993208e..356e18d 100644 --- a/packages/pi-tools/test/tgrep.test.ts +++ b/packages/pi-tools/test/tgrep.test.ts @@ -174,6 +174,16 @@ describe("formatTgrepResult", () => { }); describe("runTgrep", () => { + test("truncates partial stdout after exceeding the child-process buffer", async () => { + const out = await runTgrep( + process.execPath, + ["-e", `process.stdout.write("x".repeat(${TGREP_OUTPUT_MAX_BYTES * 2 + 1}))`], + { cwd: "/tmp" }, + ); + expect(out).toContain("truncated"); + expect(out).toContain("fileType/glob"); + }); + test("rejects aborted calls before spawning", async () => { const controller = new AbortController(); controller.abort(); From 69d3681e15bfa57a3b0a5ca30e943ed4da1b16b0 Mon Sep 17 00:00:00 2001 From: "coderabbitai[bot]" <136622811+coderabbitai[bot]@users.noreply.github.com> Date: Sat, 12 Sep 2026 13:17:47 +0000 Subject: [PATCH 04/11] docs(pi-tools): document public extension APIs --- packages/pi-tools/src/config.ts | 4 ++++ packages/pi-tools/src/index.ts | 4 ++++ packages/pi-tools/src/tgrep.ts | 31 ++++++++++++++++++++++++------- 3 files changed, 32 insertions(+), 7 deletions(-) diff --git a/packages/pi-tools/src/config.ts b/packages/pi-tools/src/config.ts index d838e15..4b6c1e3 100644 --- a/packages/pi-tools/src/config.ts +++ b/packages/pi-tools/src/config.ts @@ -31,6 +31,10 @@ const CONFIG_KEYS = new Set([ "tgrepBinPath", ]); +/** + * Loads and validates pi-tools.json, falling back to the legacy filename. + * Returns an empty config when neither file exists and throws for unreadable or invalid files. + */ export function loadConfig(agentDir = piDataDir()): FffConfig { let configPath = join(agentDir, CONFIG_FILE_NAME); const legacyConfigPath = join(agentDir, LEGACY_CONFIG_FILE_NAME); diff --git a/packages/pi-tools/src/index.ts b/packages/pi-tools/src/index.ts index 1a98a81..fa3febe 100644 --- a/packages/pi-tools/src/index.ts +++ b/packages/pi-tools/src/index.ts @@ -268,6 +268,10 @@ function createFffMentionProvider( }; } +/** + * Registers FFF tools, commands, lifecycle hooks, and autocomplete with Pi. + * Also registers tgrep search when it is enabled and its executable is available. + */ export default function fffExtension(pi: ExtensionAPI) { let mainFinder: FileFinderApi | null = null; let finderCwd: string | null = null; diff --git a/packages/pi-tools/src/tgrep.ts b/packages/pi-tools/src/tgrep.ts index dfc755c..8bb22fa 100644 --- a/packages/pi-tools/src/tgrep.ts +++ b/packages/pi-tools/src/tgrep.ts @@ -41,8 +41,10 @@ function repeatAll(values: string | string[] | undefined): string[] { return Array.isArray(values) ? values : [values]; } -// Allowlist of index-friendly flags. Full-scan forcers (--hidden, --no-ignore, -// -u, -a/--text, -E/--encoding) are excluded: they silently bypass the index. +/** + * Builds tgrep arguments for vimgrep output with literal, smart-case matching by default. + * Clamps context and per-file limits and separates the pattern and root from options. + */ export function buildTgrepArgs(options: TgrepSearchOptions): string[] { const args = ["--vimgrep"]; args.push(options.caseSensitive ? "--case-sensitive" : "--smart-case"); @@ -62,7 +64,10 @@ export function buildTgrepArgs(options: TgrepSearchOptions): string[] { return args; } -// Resolve the search root under cwd. Anything escaping the workspace is rejected. +/** + * Resolves a file or directory to a normalized path relative to the workspace. + * Throws when the input contains glob syntax or resolves outside the workspace. + */ export function resolveSearchRoot(pathParam: string | undefined, cwd: string): string { const raw = (pathParam ?? ".").trim() || "."; if (/[*?[{]/.test(raw)) @@ -83,8 +88,10 @@ function isExecutable(file: string): boolean { } } -// Explicit path wins; a set-but-missing explicit path disables rather than -// falling back to PATH so a typo surfaces instead of silently changing tools. +/** + * Returns an explicit executable or the first executable named tgrep on the search path. + * A nonempty but unusable explicit path returns undefined without searching the path. + */ export function resolveTgrepBinary(explicit?: string, pathEnv = ""): string | undefined { const trimmed = explicit?.trim(); if (trimmed) return isExecutable(trimmed) ? trimmed : undefined; @@ -130,6 +137,10 @@ async function defaultExec( } } +/** + * Executes tgrep and formats its output for a tool response. + * Rejects an already-aborted call; execution and formatting errors propagate. + */ export async function runTgrep( bin: string, args: string[], @@ -141,6 +152,10 @@ export async function runTgrep( return formatTgrepResult(result); } +/** + * Executes tgrep and returns its unformatted exit code and output streams. + * The optional executor replaces the default child-process invocation. + */ export async function runTgrepRaw( bin: string, args: string[], @@ -150,8 +165,10 @@ export async function runTgrepRaw( return exec(bin, args, opts); } -// Exit 1 is "no match", not a failure. stderr always surfaces: it carries the -// "no index" warning that decides whether the result reflects current files. +/** + * Formats tgrep output, preserving the first stderr line and truncating oversized results. + * Treats exit 1 or empty stdout as no matches and throws for exit 2. + */ export function formatTgrepResult(result: TgrepResult): string { const warning = result.stderr.trim().split("\n")[0]?.trim(); const notice = warning ? `[tgrep: ${warning}]\n` : ""; From 5b0f71360593933bbc4e678d280d57b56e79c045 Mon Sep 17 00:00:00 2001 From: "coderabbitai[bot]" <136622811+coderabbitai[bot]@users.noreply.github.com> Date: Sat, 12 Sep 2026 13:19:54 +0000 Subject: [PATCH 05/11] test(pi-tools): expand tgrep integration and edge-case coverage --- packages/pi-tools/test/extension.test.ts | 139 +++++++++++++++++++++++ packages/pi-tools/test/tgrep.test.ts | 112 +++++++++++++++++- 2 files changed, 249 insertions(+), 2 deletions(-) diff --git a/packages/pi-tools/test/extension.test.ts b/packages/pi-tools/test/extension.test.ts index 77fbf27..ada41f8 100644 --- a/packages/pi-tools/test/extension.test.ts +++ b/packages/pi-tools/test/extension.test.ts @@ -757,6 +757,44 @@ describe("pi-tools tgrep integration", () => { } }); + test("registers tgrep from the configured binary path", async () => { + const bin = writeFakeBin("configured-tgrep", "exit 1"); + writeConfig({ tgrepBinPath: bin }); + const setup = await start(); + try { + expect(toolNames(setup)).toContain("tgrep"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + } + }); + + test("does not fall back to config when TGREP_BIN is explicitly missing", async () => { + const bin = writeFakeBin("configured-tgrep", "exit 1"); + writeConfig({ tgrepBinPath: bin }); + process.env.TGREP_BIN = path.join(binDir, "missing"); + const setup = await start(); + try { + expect(toolNames(setup)).not.toContain("tgrep"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + } + }); + + test("registers tgrep independently of the FFF mode", async () => { + process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + try { + for (const mode of ["tools-only", "override"]) { + const setup = await start(mode); + expect(toolNames(setup)).toContain("tgrep"); + await shutdown(setup); + } + } finally { + fs.rmSync(binDir, { recursive: true, force: true }); + } + }); + test("executes searches through the binary and reports no-match", async () => { const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); @@ -796,6 +834,87 @@ describe("pi-tools tgrep integration", () => { } }); + test("forwards every search option with a normalized workspace path", async () => { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + process.env.TGREP_BIN = writeFakeBin("tgrep", "printf '%s\\n' \"$@\""); + const setup = await start(undefined, cwd); + try { + const tool = setup.pi.registerTool.mock.calls + .map(([t]: [any]) => t) + .find((t: any) => t.name === "tgrep"); + const result = await tool.execute( + "call-1", + { + pattern: "fn main", + path: "./src/../test", + glob: ["*.ts", "!*.test.ts"], + fileType: ["js", "ts"], + literal: false, + caseSensitive: true, + wholeWord: true, + filesOnly: true, + count: true, + context: 2, + maxCount: 5, + noIndex: true, + }, + new AbortController().signal, + ); + expect(result.content[0].text.split("\n")).toEqual([ + "--vimgrep", + "--case-sensitive", + "--word-regexp", + "--type", + "js", + "--type", + "ts", + "--glob", + "*.ts", + "--glob", + "!*.test.ts", + "--files-with-matches", + "--count", + "-A", + "2", + "-B", + "2", + "--max-count", + "5", + "--no-index", + "--", + "fn main", + "test", + ]); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("rejects an already-aborted search without invoking the binary", async () => { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + const marker = path.join(cwd, "invoked"); + process.env.TGREP_BIN = writeFakeBin("tgrep", `touch '${marker}'`); + const setup = await start(undefined, cwd); + try { + const tool = setup.pi.registerTool.mock.calls + .map(([t]: [any]) => t) + .find((t: any) => t.name === "tgrep"); + const controller = new AbortController(); + controller.abort(); + + await expect( + tool.execute("call-1", { pattern: "hello" }, controller.signal), + ).rejects.toThrow("Operation aborted"); + expect(fs.existsSync(marker)).toBe(false); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + test("/tgrep-status reports the server status", async () => { const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); process.env.TGREP_BIN = writeFakeBin("tgrep", "echo 'Indexing: complete'"); @@ -823,4 +942,24 @@ describe("pi-tools tgrep integration", () => { await shutdown(setup); } }); + + test("/tgrep-status reports binary failures", async () => { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + process.env.TGREP_BIN = writeFakeBin( + "tgrep", + "echo 'invalid index metadata' >&2; exit 2", + ); + const setup = await start(undefined, cwd); + try { + await setup.commands.get("tgrep-status").handler("", setup.ctx); + expect(setup.ctx.ui.notify).toHaveBeenCalledWith( + "tgrep status failed: tgrep search failed: invalid index metadata", + "error", + ); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); }); diff --git a/packages/pi-tools/test/tgrep.test.ts b/packages/pi-tools/test/tgrep.test.ts index 356e18d..477fc15 100644 --- a/packages/pi-tools/test/tgrep.test.ts +++ b/packages/pi-tools/test/tgrep.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, test } from "bun:test"; +import { describe, expect, mock, test } from "bun:test"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; @@ -61,6 +61,15 @@ describe("buildTgrepArgs", () => { ]); }); + test("repeats glob filters in the supplied order", () => { + const args = buildTgrepArgs({ + pattern: "x", + root: ".", + glob: ["*.ts", "!*.test.ts"], + }); + expect(args.slice(3, -3)).toEqual(["--glob", "*.ts", "--glob", "!*.test.ts"]); + }); + test("emits count and per-file cap", () => { const args = buildTgrepArgs({ pattern: "x", root: ".", count: true, maxCount: 0 }); expect(args).toContain("--count"); @@ -79,6 +88,26 @@ describe("buildTgrepArgs", () => { expect(args[args.indexOf("-A") + 1]).toBe(String(TGREP_CONTEXT_MAX)); }); + test("floors fractional limits and omits non-positive context", () => { + const fractional = buildTgrepArgs({ + pattern: "x", + root: ".", + context: 3.9, + maxCount: 4.9, + }); + expect(fractional.slice(fractional.indexOf("-A"), fractional.indexOf("-B"))).toEqual([ + "-A", + "3", + ]); + expect(fractional[fractional.indexOf("--max-count") + 1]).toBe("4"); + + for (const context of [0, -1, Number.NaN]) { + const args = buildTgrepArgs({ pattern: "x", root: ".", context }); + expect(args).not.toContain("-A"); + expect(args).not.toContain("-B"); + } + }); + test("never emits full-scan forcing flags", () => { const args = buildTgrepArgs({ pattern: "x", root: "." }).join(" "); for (const banned of ["--hidden", "--no-ignore", "--text", "--encoding", "-u"]) { @@ -99,10 +128,20 @@ describe("resolveSearchRoot", () => { expect(resolveSearchRoot("src/main.ts", cwd)).toBe("src/main.ts"); }); + test("trims and normalizes workspace-relative paths", () => { + expect(resolveSearchRoot(" ", cwd)).toBe("."); + expect(resolveSearchRoot(" ./src/../test/tgrep.test.ts ", cwd)).toBe( + "test/tgrep.test.ts", + ); + }); + test("rejects globs and escapes", () => { expect(() => resolveSearchRoot("src/**/*.ts", cwd)).toThrow("not a glob"); expect(() => resolveSearchRoot("../outside", cwd)).toThrow("inside the workspace"); expect(() => resolveSearchRoot("/etc/passwd", cwd)).toThrow("inside the workspace"); + expect(() => resolveSearchRoot(`${cwd}-sibling`, cwd)).toThrow( + "inside the workspace", + ); }); }); @@ -133,6 +172,37 @@ describe("resolveTgrepBinary", () => { fs.rmSync(dir, { recursive: true, force: true }); } }); + + test("does not fall back to PATH when an explicit binary is missing", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-precedence-")); + try { + const pathBin = path.join(dir, "tgrep"); + fs.writeFileSync(pathBin, "#!/bin/sh\n"); + fs.chmodSync(pathBin, 0o755); + + expect(resolveTgrepBinary(path.join(dir, "missing"), dir)).toBeUndefined(); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); + + test("skips non-executable PATH entries and continues searching", () => { + const first = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-path-first-")); + const second = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-path-second-")); + try { + fs.writeFileSync(path.join(first, "tgrep"), "#!/bin/sh\n"); + const executable = path.join(second, "tgrep"); + fs.writeFileSync(executable, "#!/bin/sh\n"); + fs.chmodSync(executable, 0o755); + + expect(resolveTgrepBinary(undefined, [first, second].join(path.delimiter))).toBe( + executable, + ); + } finally { + fs.rmSync(first, { recursive: true, force: true }); + fs.rmSync(second, { recursive: true, force: true }); + } + }); }); describe("formatTgrepResult", () => { @@ -164,6 +234,27 @@ describe("formatTgrepResult", () => { expect(out).toContain("a.ts:1:5:foo"); }); + test("keeps the first stderr warning when there are no matches", () => { + expect( + formatTgrepResult({ + exit: 1, + stdout: "", + stderr: "warning: stale index\nadditional diagnostic\n", + }), + ).toBe("[tgrep: warning: stale index]\nNo matches found"); + }); + + test("uses a stable fallback when an exit 2 error has no stderr", () => { + expect(() => formatTgrepResult({ exit: 2, stdout: "", stderr: "" })).toThrow( + "tgrep search failed: unknown error", + ); + }); + + test("does not truncate output at the exact byte limit", () => { + const exact = "x".repeat(TGREP_OUTPUT_MAX_BYTES); + expect(formatTgrepResult({ exit: 0, stdout: exact, stderr: "" })).toBe(exact); + }); + test("truncates oversized output with a narrowing hint", () => { const big = `${"x".repeat(TGREP_OUTPUT_MAX_BYTES + 10)}\n`; const out = formatTgrepResult({ exit: 0, stdout: big, stderr: "" }); @@ -187,14 +278,31 @@ describe("runTgrep", () => { test("rejects aborted calls before spawning", async () => { const controller = new AbortController(); controller.abort(); + const exec = mock(async () => ({ exit: 0, stdout: "", stderr: "" })); await expect( runTgrep( "/bin/false", ["--", "x", "."], { cwd: "/tmp", signal: controller.signal }, - async () => ({ exit: 0, stdout: "", stderr: "" }), + exec, ), ).rejects.toThrow("Operation aborted"); + expect(exec).not.toHaveBeenCalled(); + }); + + test("maps the real process no-match exit and stderr warning", async () => { + const out = await runTgrep( + process.execPath, + ["-e", 'process.stderr.write("warning: stale index\\n"); process.exit(1)'], + { cwd: "/tmp" }, + ); + expect(out).toBe("[tgrep: warning: stale index]\nNo matches found"); + }); + + test("wraps child-process launch failures with tgrep context", async () => { + await expect( + runTgrep("/definitely/missing/tgrep", ["--", "x", "."], { cwd: "/tmp" }), + ).rejects.toThrow("tgrep failed to run:"); }); test("delegates to the injected executor and formats", async () => { From b4e3414303284af34b24e0cc2371319787d314e2 Mon Sep 17 00:00:00 2001 From: "coderabbitai[bot]" <136622811+coderabbitai[bot]@users.noreply.github.com> Date: Sat, 12 Sep 2026 13:30:19 +0000 Subject: [PATCH 06/11] docs(pi-tools): document internal extension helpers --- packages/pi-tools/src/config.ts | 2 ++ packages/pi-tools/src/index.ts | 1 + packages/pi-tools/src/tgrep.ts | 5 +++++ packages/pi-tools/test/extension.test.ts | 2 ++ 4 files changed, 10 insertions(+) diff --git a/packages/pi-tools/src/config.ts b/packages/pi-tools/src/config.ts index 4b6c1e3..fdbce23 100644 --- a/packages/pi-tools/src/config.ts +++ b/packages/pi-tools/src/config.ts @@ -103,6 +103,7 @@ function isRecord(value: unknown): value is Record { return typeof value === "object" && value !== null && !Array.isArray(value); } +/** Validates that an optional configuration value is a non-empty string. */ function validateString( configPath: string, config: Record, @@ -114,6 +115,7 @@ function validateString( } } +/** Validates that an optional configuration value is boolean. */ function validateBoolean( configPath: string, config: Record, diff --git a/packages/pi-tools/src/index.ts b/packages/pi-tools/src/index.ts index fa3febe..aad66d4 100644 --- a/packages/pi-tools/src/index.ts +++ b/packages/pi-tools/src/index.ts @@ -1259,6 +1259,7 @@ export default function fffExtension(pi: ExtensionAPI) { ], parameters: tgrepSchema, + /** Executes tgrep with validated workspace-relative arguments. */ async execute(_toolCallId, params, signal) { if (signal?.aborted) throw new Error("Operation aborted"); const root = resolveSearchRoot(params.path, activeCwd); diff --git a/packages/pi-tools/src/tgrep.ts b/packages/pi-tools/src/tgrep.ts index 8bb22fa..f341ff8 100644 --- a/packages/pi-tools/src/tgrep.ts +++ b/packages/pi-tools/src/tgrep.ts @@ -36,6 +36,7 @@ type ExecFn = ( opts: { cwd: string; signal?: AbortSignal }, ) => Promise; +/** Normalizes an optional scalar or list into an iterable array. */ function repeatAll(values: string | string[] | undefined): string[] { if (!values) return []; return Array.isArray(values) ? values : [values]; @@ -79,6 +80,7 @@ export function resolveSearchRoot(pathParam: string | undefined, cwd: string): s return relative === "" ? "." : relative.split(path.sep).join("/"); } +/** Checks whether a path points to an executable regular file. */ function isExecutable(file: string): boolean { try { fs.accessSync(file, fs.constants.X_OK); @@ -105,6 +107,7 @@ export function resolveTgrepBinary(explicit?: string, pathEnv = ""): string | un const execFileAsync = promisify(execFile); +/** Runs tgrep with bounded time and output, preserving output from normal process exits. */ async function defaultExec( bin: string, args: string[], @@ -179,6 +182,7 @@ export function formatTgrepResult(result: TgrepResult): string { return `${notice}${body}`; } +/** Truncates oversized output and appends a hint for narrowing the search. */ function truncateBytes(text: string): string { const buf = Buffer.from(text); if (buf.length <= TGREP_OUTPUT_MAX_BYTES) return text; @@ -186,6 +190,7 @@ function truncateBytes(text: string): string { return `${head}\n… [truncated ${buf.length - TGREP_OUTPUT_MAX_BYTES} bytes: narrow with fileType/glob]`; } +/** Normalizes context to the supported non-negative integer range. */ function clampContext(context: number | undefined): number { if (!context || context < 0) return 0; return Math.min(Math.floor(context), TGREP_CONTEXT_MAX); diff --git a/packages/pi-tools/test/extension.test.ts b/packages/pi-tools/test/extension.test.ts index ada41f8..7f6684d 100644 --- a/packages/pi-tools/test/extension.test.ts +++ b/packages/pi-tools/test/extension.test.ts @@ -710,6 +710,7 @@ describe("pi-tools autocomplete registration", () => { describe("pi-tools tgrep integration", () => { let binDir = ""; + /** Creates an executable shell stub for tgrep integration tests. */ function writeFakeBin(name: string, body: string): string { binDir = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-")); const bin = path.join(binDir, name); @@ -718,6 +719,7 @@ describe("pi-tools tgrep integration", () => { return bin; } + /** Returns the tool names registered through the mocked Pi API. */ function toolNames(setup: { pi: { registerTool: any } }): string[] { return setup.pi.registerTool.mock.calls.map(([tool]: [any]) => tool.name); } From a811354d0a8baabf37f9a7c417cc9e3ae81cb663 Mon Sep 17 00:00:00 2001 From: "coderabbitai[bot]" <136622811+coderabbitai[bot]@users.noreply.github.com> Date: Sat, 12 Sep 2026 13:33:44 +0000 Subject: [PATCH 07/11] test(pi-tools): cover tgrep schema, path safety, and output handling --- packages/pi-tools/test/extension.test.ts | 90 ++++++++++++++++++++++++ packages/pi-tools/test/tgrep.test.ts | 42 ++++++++++- 2 files changed, 129 insertions(+), 3 deletions(-) diff --git a/packages/pi-tools/test/extension.test.ts b/packages/pi-tools/test/extension.test.ts index 7f6684d..13054ee 100644 --- a/packages/pi-tools/test/extension.test.ts +++ b/packages/pi-tools/test/extension.test.ts @@ -736,6 +736,61 @@ describe("pi-tools tgrep integration", () => { } }); + test("registers the complete tgrep parameter schema", async () => { + process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + const setup = await start(); + try { + const tool = setup.pi.registerTool.mock.calls + .map(([registered]: [any]) => registered) + .find((registered: any) => registered.name === "tgrep"); + const properties = tool.parameters.properties; + + expect(Object.keys(properties).sort()).toEqual( + [ + "caseSensitive", + "context", + "count", + "filesOnly", + "fileType", + "glob", + "literal", + "maxCount", + "noIndex", + "path", + "pattern", + "wholeWord", + ].sort(), + ); + expect(properties.pattern).toMatchObject({ type: "string" }); + expect(properties.pattern.optional).toBeUndefined(); + expect(properties.path).toMatchObject({ type: "string", optional: true }); + expect(properties.glob).toMatchObject({ type: "union", optional: true }); + expect(properties.fileType).toMatchObject({ type: "union", optional: true }); + expect(properties.context.options.description).toContain("0-20"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + } + }); + + test("discovers tgrep from PATH when no explicit path is configured", async () => { + writeFakeBin("tgrep", "exit 1"); + const savedPath = process.env.PATH; + process.env.PATH = binDir; + try { + const setup = await start(); + try { + expect(toolNames(setup)).toContain("tgrep"); + } finally { + await shutdown(setup); + } + } finally { + if (savedPath === undefined) delete process.env.PATH; + else process.env.PATH = savedPath; + fs.rmSync(binDir, { recursive: true, force: true }); + } + }); + test("skips tgrep when the binary is missing", async () => { process.env.TGREP_BIN = "/nonexistent/tgrep"; const setup = await start(); @@ -917,6 +972,27 @@ describe("pi-tools tgrep integration", () => { } }); + test("rejects paths outside the workspace before invoking the binary", async () => { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + const marker = path.join(cwd, "invoked"); + process.env.TGREP_BIN = writeFakeBin("tgrep", `touch '${marker}'`); + const setup = await start(undefined, cwd); + try { + const tool = setup.pi.registerTool.mock.calls + .map(([registered]: [any]) => registered) + .find((registered: any) => registered.name === "tgrep"); + + await expect( + tool.execute("call-1", { pattern: "hello", path: "../outside" }, undefined), + ).rejects.toThrow("tgrep path must stay inside the workspace"); + expect(fs.existsSync(marker)).toBe(false); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + test("/tgrep-status reports the server status", async () => { const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); process.env.TGREP_BIN = writeFakeBin("tgrep", "echo 'Indexing: complete'"); @@ -931,6 +1007,20 @@ describe("pi-tools tgrep integration", () => { } }); + test("/tgrep-status passes the active workspace to the binary", async () => { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + process.env.TGREP_BIN = writeFakeBin("tgrep", "printf '%s\\n' \"$@\""); + const setup = await start(undefined, cwd); + try { + await setup.commands.get("tgrep-status").handler("ignored", setup.ctx); + expect(setup.ctx.ui.notify).toHaveBeenCalledWith(`status\n${cwd}`, "info"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + test("/tgrep-status warns without a binary", async () => { process.env.TGREP_BIN = "/nonexistent/tgrep"; const setup = await start(); diff --git a/packages/pi-tools/test/tgrep.test.ts b/packages/pi-tools/test/tgrep.test.ts index 477fc15..8c66b68 100644 --- a/packages/pi-tools/test/tgrep.test.ts +++ b/packages/pi-tools/test/tgrep.test.ts @@ -203,6 +203,25 @@ describe("resolveTgrepBinary", () => { fs.rmSync(second, { recursive: true, force: true }); } }); + + test("skips executable directories named tgrep", () => { + const first = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-path-directory-")); + const second = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-path-file-")); + try { + fs.mkdirSync(path.join(first, "tgrep")); + fs.chmodSync(path.join(first, "tgrep"), 0o755); + const executable = path.join(second, "tgrep"); + fs.writeFileSync(executable, "#!/bin/sh\n"); + fs.chmodSync(executable, 0o755); + + expect(resolveTgrepBinary(undefined, [first, second].join(path.delimiter))).toBe( + executable, + ); + } finally { + fs.rmSync(first, { recursive: true, force: true }); + fs.rmSync(second, { recursive: true, force: true }); + } + }); }); describe("formatTgrepResult", () => { @@ -262,6 +281,20 @@ describe("formatTgrepResult", () => { expect(out).toContain("fileType/glob"); expect(Buffer.byteLength(out)).toBeLessThan(Buffer.byteLength(big) + 200); }); + + test("measures truncation in bytes for multibyte output and preserves warnings", () => { + const stdout = "€".repeat(Math.ceil((TGREP_OUTPUT_MAX_BYTES + 4) / 3)); + const omittedBytes = Buffer.byteLength(stdout) - TGREP_OUTPUT_MAX_BYTES; + const out = formatTgrepResult({ + exit: 0, + stdout, + stderr: "warning: stale index\nignored diagnostic\n", + }); + + expect(out.startsWith("[tgrep: warning: stale index]\n")).toBe(true); + expect(out).toContain(`[truncated ${omittedBytes} bytes: narrow with fileType/glob]`); + expect(out).not.toContain("ignored diagnostic"); + }); }); describe("runTgrep", () => { @@ -307,16 +340,19 @@ describe("runTgrep", () => { test("delegates to the injected executor and formats", async () => { const seen: unknown[] = []; + const controller = new AbortController(); const out = await runTgrep( "/fake/tgrep", ["--", "x", "."], - { cwd: "/tmp" }, + { cwd: "/tmp", signal: controller.signal }, async (bin, args, opts) => { - seen.push([bin, args, opts.cwd]); + seen.push([bin, args, opts]); return { exit: 0, stdout: "a.ts:2:1:x\n", stderr: "" }; }, ); expect(out).toBe("a.ts:2:1:x"); - expect(seen).toEqual([["/fake/tgrep", ["--", "x", "."], "/tmp"]]); + expect(seen).toEqual([ + ["/fake/tgrep", ["--", "x", "."], { cwd: "/tmp", signal: controller.signal }], + ]); }); }); From e88c9a8c8f358942eee6370e87f5bfbdb7854cba Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Sat, 12 Sep 2026 13:35:51 +0000 Subject: [PATCH 08/11] fix(pi-tools): register tgrep only when a workspace index exists Gate the tool at session start on binary + cwd .tgrep/, resolve the binary after cwd is known, and steer the model on exact vs fuzzy search instead of repository size. Co-authored-by: OnlineChef --- docs/compatibility.md | 16 +- packages/pi-tools/CHANGELOG.md | 2 +- packages/pi-tools/README.md | 8 +- packages/pi-tools/pi-tools.schema.json | 2 +- packages/pi-tools/src/index.ts | 255 ++++++++++++----------- packages/pi-tools/src/tgrep.ts | 10 + packages/pi-tools/test/extension.test.ts | 76 +++++-- packages/pi-tools/test/tgrep.test.ts | 23 ++ 8 files changed, 245 insertions(+), 147 deletions(-) diff --git a/docs/compatibility.md b/docs/compatibility.md index 2024d20..05970fe 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -15,9 +15,9 @@ The extension registers two FFF tools, an optional `tgrep` tool, and one complet |---|---| | `fffind` | Typo-resistant file discovery + frecency-ranked access | | `ffgrep` | SIMD content search | -| `tgrep` | Trigram-indexed content search via an external binary (registered only when found) | +| `tgrep` | Trigram-indexed exact content search via an external binary (registered when the binary and a workspace `.tgrep/` index are found) | -Source of truth: `packages/pi-tools/src/index.ts:60` (`FFF_TOOL_NAMES`: `grep: "ffgrep", find: "fffind"`), `tgrep` via `queueTool(() => TGREP_TOOL_NAME, …)` gated on `resolveTgrepBinary` (`src/tgrep.ts`), +Source of truth: `packages/pi-tools/src/index.ts:60` (`FFF_TOOL_NAMES`: `grep: "ffgrep", find: "fffind"`), `tgrep` via `queueTgrepTool` / `queueTool(() => TGREP_TOOL_NAME, …)` gated on `resolveTgrepBinary` plus `hasTgrepIndex` (`src/tgrep.ts`) at session start, registered via `queueTool(() => toolNames.grep, …)` / `queueTool(() => toolNames.find, …)`. ### Parameters @@ -41,16 +41,16 @@ tool parameters. Compatibility surface = the parameter **names**, **types**, **e A tool disappearing when mode changes is expected; a mode value being removed is breaking. - **Exit codes (`tgrep`):** exit `0` returns `file:line:col:text` rows, exit `1` reports `No matches found` (not a failure), exit `2` throws with the stderr cause. A leading - `[tgrep: ...]` line always carries the binary's stderr, including the "no index" - freshness warning. Output truncates at `TGREP_OUTPUT_MAX_BYTES` with a narrowing hint. - `tgrep` is mode-independent: it keeps its name in every mode and is gated only on - binary availability plus `enableTgrep`. + `[tgrep: ...]` line always carries the binary's stderr freshness warning. Output + truncates at `TGREP_OUTPUT_MAX_BYTES` with a narrowing hint. + `tgrep` is mode-independent: it keeps its name in every mode and is gated on + binary availability, a workspace `.tgrep/` directory, and `enableTgrep`. ### Config precedence Config values are resolved in this priority order -(`packages/pi-tools/src/index.ts:291` `getConfigValue`, read at startup -`resolveStartupConfig`; `tgrep` resolves once at extension load): +(`packages/pi-tools/src/index.ts` `getConfigValue`, read at startup +`resolveStartupConfig`; `tgrep` binary and index are resolved at session start): ``` flag (--fff-mode / --fff-frecency-db / --fff-history-db / …) diff --git a/packages/pi-tools/CHANGELOG.md b/packages/pi-tools/CHANGELOG.md index 901f783..eeeb0ca 100644 --- a/packages/pi-tools/CHANGELOG.md +++ b/packages/pi-tools/CHANGELOG.md @@ -4,7 +4,7 @@ ### Added -- Added the `tgrep` tool: trigram-indexed content search through an external [tgrep](https://github.com/microsoft/tgrep) binary. Literal by default, `file:line:col:text` output, exit 1 reported as no-match. Registered only when the binary resolves (`TGREP_BIN`, `tgrepBinPath`, or `PATH`) and `enableTgrep` is not `false`; mode-independent. +- Added the `tgrep` tool: trigram-indexed exact content search through an external [tgrep](https://github.com/microsoft/tgrep) binary. Literal by default, `file:line:col:text` output, exit 1 reported as no-match. Registered at session start only when the binary resolves (`TGREP_BIN`, `tgrepBinPath`, or `PATH`), the workspace has a `.tgrep/` index, and `enableTgrep` is not `false`; mode-independent. Prompt guidelines steer on capability (exact/literal vs fuzzy/frecency), not repository size. - Added the `/tgrep-status` command showing tgrep index and server status for the workspace. - Added `ffgrep.maxMatchesPerFile` to keep a single generated or vendored file from dominating a result page. - Added `ffgrep.compact` for deterministic `path:line:match` output without context blocks. diff --git a/packages/pi-tools/README.md b/packages/pi-tools/README.md index 85a0db7..d0bd13e 100644 --- a/packages/pi-tools/README.md +++ b/packages/pi-tools/README.md @@ -86,7 +86,7 @@ Use a concrete substring, identifier, or expression. A wildcard-only expression ### `tgrep` -`tgrep` searches file content through an external [tgrep](https://github.com/microsoft/tgrep) binary: trigram-indexed search with a client/server architecture, fastest on large repositories with a built index. The tool is registered only when the binary is found and `enableTgrep` is not disabled; it keeps the name `tgrep` in every mode. Prefer `ffgrep` for fuzzy, frecency-ranked search on small and medium repositories. +`tgrep` searches file content through an external [tgrep](https://github.com/microsoft/tgrep) binary: trigram-indexed exact search with a client/server architecture. The tool is registered only when the binary is found, the session cwd has a `.tgrep/` index, and `enableTgrep` is not disabled; it keeps the name `tgrep` in every mode. Prefer `tgrep` for exact literals and symbols; prefer `ffgrep` for fuzzy, typo-tolerant, frecency-ranked search. After `tgrep index .` or `tgrep serve .`, reload the session to expose the tool. | Parameter | Type | Description | | --- | --- | --- | @@ -103,7 +103,7 @@ Use a concrete substring, identifier, or expression. A wildcard-only expression | `maxCount` | number, optional | Limits matches per file. | | `noIndex` | boolean, optional | Reads files from disk instead of the index. Use after your own edits when the latest content must be visible. | -Output is `file:line:col:text` rows. Exit code 1 (no match) is reported as `No matches found`, not as a failure. A leading `[tgrep: ...]` line carries the binary's stderr, including the "no index" warning: without an index (`tgrep index .` or `tgrep serve .`) the search scans every file like grep and may be slow. Only index-safe flags are forwarded; full-scan forcers (`--hidden`, `--no-ignore`, `-u`, `-a`, `--encoding`) are excluded by design. +Output is `file:line:col:text` rows. Exit code 1 (no match) is reported as `No matches found`, not as a failure. A leading `[tgrep: ...]` line carries the binary's stderr freshness warning. Only index-safe flags are forwarded; full-scan forcers (`--hidden`, `--no-ignore`, `-u`, `-a`, `--encoding`) are excluded by design. The tool is not registered without a workspace `.tgrep/` directory, so a no-index full scan is not offered as a search tool. ### Optional multi-pattern search @@ -149,12 +149,12 @@ Create `pi-tools.json` in Pi’s agent directory. The default location is `~/.pi | `historyDbPath` | string | Auto-resolved | Location for query-selection history. | | `enableFsRootScanning` | boolean | `false` | Explicitly allows scans started from `/`. | | `enableHomeDirScanning` | boolean | `true` | Allows scanning when Pi starts in the home directory. | -| `enableTgrep` | boolean | `true` | Registers the `tgrep` tool when the binary is found. | +| `enableTgrep` | boolean | `true` | Registers the `tgrep` tool when the binary and a `.tgrep/` index are found. | | `tgrepBinPath` | string | PATH lookup | Explicit path to the `tgrep` binary. | Malformed configuration, unknown fields, and invalid values prevent the extension from loading and identify the configuration path in the error. `/fff-mode` changes session state only; it does not edit this file. -The `tgrep` binary resolves as `TGREP_BIN` environment variable, then `tgrepBinPath`, then a `tgrep` executable on `PATH`. An explicit path that is set but not executable disables the tool instead of falling back, so a typo surfaces instead of silently changing the search backend. +The `tgrep` binary resolves as `TGREP_BIN` environment variable, then `tgrepBinPath`, then a `tgrep` executable on `PATH`. An explicit path that is set but not executable disables the tool instead of falling back, so a typo surfaces instead of silently changing the search backend. Binary and `.tgrep/` index are resolved at session start, when the workspace cwd is known. ## Database resolution diff --git a/packages/pi-tools/pi-tools.schema.json b/packages/pi-tools/pi-tools.schema.json index 92e298a..bdb884b 100644 --- a/packages/pi-tools/pi-tools.schema.json +++ b/packages/pi-tools/pi-tools.schema.json @@ -40,7 +40,7 @@ "enableTgrep": { "type": "boolean", "default": true, - "description": "Registers the tgrep trigram-index search tool when the tgrep binary is found." + "description": "Registers the tgrep trigram-index search tool when the tgrep binary is found and the workspace has a .tgrep index." }, "tgrepBinPath": { "type": "string", diff --git a/packages/pi-tools/src/index.ts b/packages/pi-tools/src/index.ts index aad66d4..1456265 100644 --- a/packages/pi-tools/src/index.ts +++ b/packages/pi-tools/src/index.ts @@ -25,6 +25,7 @@ import { isHomeDir, resolveDbPaths } from "./paths"; import { buildQuery } from "./query"; import { buildTgrepArgs, + hasTgrepIndex, resolveSearchRoot, resolveTgrepBinary, runTgrep, @@ -270,7 +271,7 @@ function createFffMentionProvider( /** * Registers FFF tools, commands, lifecycle hooks, and autocomplete with Pi. - * Also registers tgrep search when it is enabled and its executable is available. + * Also registers tgrep search when the binary and a `.tgrep` index are available. */ export default function fffExtension(pi: ExtensionAPI) { let mainFinder: FileFinderApi | null = null; @@ -280,16 +281,16 @@ export default function fffExtension(pi: ExtensionAPI) { let activeCwd = process.cwd(); const config = loadConfig(); - - // tgrep availability is session-independent: resolve once at load so the - // tool surface is fixed before registration. Explicit path wins over PATH. - const tgrepBin: string | undefined = - config.enableTgrep === false - ? undefined - : resolveTgrepBinary( - process.env[TGREP_BIN_ENV]?.trim() || config.tgrepBinPath, - process.env.PATH, - ); + // Resolved at session start so cwd (index) and flags are known. + let tgrepBin: string | undefined; + + function resolveTgrepBin(): string | undefined { + if (config.enableTgrep === false) return undefined; + return resolveTgrepBinary( + process.env[TGREP_BIN_ENV]?.trim() || config.tgrepBinPath, + process.env.PATH, + ); + } // Resolve startup options with flag > environment > file > fallback. function getConfigValue( @@ -619,6 +620,115 @@ export default function fffExtension(pi: ExtensionAPI) { toolsRegistered = true; } + function queueTgrepTool(bin: string): void { + const tgrepSchema = Type.Object({ + pattern: Type.String({ + description: + "Search pattern. Literal text by default; set literal: false for regex.", + }), + path: Type.Optional( + Type.String({ + description: + "Directory or file to search, relative to the workspace (default: workspace root). Globs go in glob.", + }), + ), + glob: Type.Optional( + Type.Union([Type.String(), Type.Array(Type.String())], { + description: "Repeatable file glob filter, e.g. '*.{ts,tsx}'.", + }), + ), + fileType: Type.Optional( + Type.Union([Type.String(), Type.Array(Type.String())], { + description: "Repeatable file type filter, e.g. 'rust', 'py', 'js'.", + }), + ), + literal: Type.Optional( + Type.Boolean({ + description: + "Treat pattern as literal text (default true). Set false for regex.", + }), + ), + caseSensitive: Type.Optional( + Type.Boolean({ + description: "Force case-sensitive matching. Default is smart-case.", + }), + ), + wholeWord: Type.Optional(Type.Boolean({ description: "Match whole words only." })), + filesOnly: Type.Optional( + Type.Boolean({ description: "Print only filenames with matches." }), + ), + count: Type.Optional(Type.Boolean({ description: "Print match count per file." })), + context: Type.Optional( + Type.Number({ + description: `Context lines before+after each match (0-${TGREP_CONTEXT_MAX})`, + }), + ), + maxCount: Type.Optional(Type.Number({ description: "Limit matches per file." })), + noIndex: Type.Optional( + Type.Boolean({ + description: + "Read files from disk instead of the index. Use after your own edits when the latest content must be visible.", + }), + ), + }); + + queueTool(() => TGREP_TOOL_NAME, { + description: + "Trigram-indexed exact content search (tgrep). Literal by default, vimgrep output. Use for symbols and exact strings; prefer FFF grep for fuzzy or frecency-ranked results. Default limit is output-capped; narrow with fileType/glob.", + promptSnippet: "Trigram-indexed exact content search", + promptGuidelines: (names) => [ + `${names.grep}: prefer for fuzzy, typo-tolerant, frecency-ranked search.`, + `${TGREP_TOOL_NAME}: prefer for exact literal or symbol search against the tgrep index.`, + `${TGREP_TOOL_NAME}: keep literal: true (default) for symbols; set false for regex.`, + `${TGREP_TOOL_NAME}: narrow with fileType/glob before raising maxCount.`, + `${TGREP_TOOL_NAME}: use filesOnly: true first on broad queries, then search specific files.`, + `${TGREP_TOOL_NAME}: after your own edits, pass noIndex: true or use ${names.grep}; the index lags watcher events.`, + `${TGREP_TOOL_NAME}: a '[tgrep: ...]' line reports index freshness, never drop it from summaries.`, + ], + parameters: tgrepSchema, + + /** Executes tgrep with validated workspace-relative arguments. */ + async execute(_toolCallId, params, signal) { + if (signal?.aborted) throw new Error("Operation aborted"); + const root = resolveSearchRoot(params.path, activeCwd); + const output = await runTgrep( + bin, + buildTgrepArgs({ + pattern: params.pattern, + root, + literal: params.literal, + caseSensitive: params.caseSensitive, + wholeWord: params.wholeWord, + fileType: params.fileType, + glob: params.glob, + filesOnly: params.filesOnly, + count: params.count, + context: params.context, + maxCount: params.maxCount, + noIndex: params.noIndex, + }), + { cwd: activeCwd, signal }, + ); + return { content: [{ type: "text", text: output }], details: {} }; + }, + + renderCall(args, theme, context) { + const text = (context.lastComponent as Text | undefined) ?? new Text("", 0, 0); + text.setText( + theme.fg("toolTitle", theme.bold(TGREP_TOOL_NAME)) + + " " + + theme.fg("accent", `/${args?.pattern ?? ""}/`) + + theme.fg("toolOutput", ` in ${args?.path ?? "."}`), + ); + return text; + }, + + renderResult(result, options, theme, context) { + return renderTextResult(result, options, theme, context, 15); + }, + }); + } + pi.registerFlag("fff-mode", { description: "FFF mode: tools-and-ui | tools-only | override", type: "string", @@ -681,6 +791,10 @@ export default function fffExtension(pi: ExtensionAPI) { } initializeFinderFactories(); + tgrepBin = resolveTgrepBin(); + if (tgrepBin !== undefined && hasTgrepIndex(activeCwd)) { + queueTgrepTool(tgrepBin); + } registerPendingTools(); } @@ -1191,116 +1305,6 @@ export default function fffExtension(pi: ExtensionAPI) { }); } - // tgrep is mode-independent and binary-gated: no binary, no tool. - if (tgrepBin !== undefined) { - const bin = tgrepBin; - const tgrepSchema = Type.Object({ - pattern: Type.String({ - description: - "Search pattern. Literal text by default; set literal: false for regex.", - }), - path: Type.Optional( - Type.String({ - description: - "Directory or file to search, relative to the workspace (default: workspace root). Globs go in glob.", - }), - ), - glob: Type.Optional( - Type.Union([Type.String(), Type.Array(Type.String())], { - description: "Repeatable file glob filter, e.g. '*.{ts,tsx}'.", - }), - ), - fileType: Type.Optional( - Type.Union([Type.String(), Type.Array(Type.String())], { - description: "Repeatable file type filter, e.g. 'rust', 'py', 'js'.", - }), - ), - literal: Type.Optional( - Type.Boolean({ - description: - "Treat pattern as literal text (default true). Set false for regex.", - }), - ), - caseSensitive: Type.Optional( - Type.Boolean({ - description: "Force case-sensitive matching. Default is smart-case.", - }), - ), - wholeWord: Type.Optional(Type.Boolean({ description: "Match whole words only." })), - filesOnly: Type.Optional( - Type.Boolean({ description: "Print only filenames with matches." }), - ), - count: Type.Optional(Type.Boolean({ description: "Print match count per file." })), - context: Type.Optional( - Type.Number({ - description: `Context lines before+after each match (0-${TGREP_CONTEXT_MAX})`, - }), - ), - maxCount: Type.Optional(Type.Number({ description: "Limit matches per file." })), - noIndex: Type.Optional( - Type.Boolean({ - description: - "Read files from disk instead of the index. Use after your own edits when the latest content must be visible.", - }), - ), - }); - - queueTool(() => TGREP_TOOL_NAME, { - description: `Trigram-indexed content search for large repos (tgrep). Literal by default, vimgrep output. Falls back to a full scan with a stderr warning when no index exists. Default limit is output-capped; narrow with fileType/glob.`, - promptSnippet: "Trigram-indexed content search", - promptGuidelines: (names) => [ - `${names.grep}: prefer for fuzzy/frecency-ranked search on small and medium repos.`, - `${TGREP_TOOL_NAME}: prefer on large repos with a built tgrep index; fastest path.`, - `${TGREP_TOOL_NAME}: keep literal: true (default) for symbols; set false for regex.`, - `${TGREP_TOOL_NAME}: narrow with fileType/glob before raising maxCount.`, - `${TGREP_TOOL_NAME}: use filesOnly: true first on broad queries, then search specific files.`, - `${TGREP_TOOL_NAME}: after your own edits, pass noIndex: true or use ${names.grep}; the index lags watcher events.`, - `${TGREP_TOOL_NAME}: a '[tgrep: ...]' line reports index freshness, never drop it from summaries.`, - ], - parameters: tgrepSchema, - - /** Executes tgrep with validated workspace-relative arguments. */ - async execute(_toolCallId, params, signal) { - if (signal?.aborted) throw new Error("Operation aborted"); - const root = resolveSearchRoot(params.path, activeCwd); - const output = await runTgrep( - bin, - buildTgrepArgs({ - pattern: params.pattern, - root, - literal: params.literal, - caseSensitive: params.caseSensitive, - wholeWord: params.wholeWord, - fileType: params.fileType, - glob: params.glob, - filesOnly: params.filesOnly, - count: params.count, - context: params.context, - maxCount: params.maxCount, - noIndex: params.noIndex, - }), - { cwd: activeCwd, signal }, - ); - return { content: [{ type: "text", text: output }], details: {} }; - }, - - renderCall(args, theme, context) { - const text = (context.lastComponent as Text | undefined) ?? new Text("", 0, 0); - text.setText( - theme.fg("toolTitle", theme.bold(TGREP_TOOL_NAME)) + - " " + - theme.fg("accent", `/${args?.pattern ?? ""}/`) + - theme.fg("toolOutput", ` in ${args?.path ?? "."}`), - ); - return text; - }, - - renderResult(result, options, theme, context) { - return renderTextResult(result, options, theme, context, 15); - }, - }); - } - pi.registerCommand("fff-mode", { description: "Show or set FFF mode: /fff-mode [tools-and-ui | tools-only | override]", handler: async (args, ctx) => { @@ -1399,6 +1403,15 @@ export default function fffExtension(pi: ExtensionAPI) { pi.registerCommand("tgrep-status", { description: "Show tgrep index and server status for the workspace", handler: async (_args, ctx) => { + if (!toolsRegistered) { + try { + prepareSession(ctx); + } catch (error: unknown) { + reportInitFailure(ctx, error); + return; + } + } + const bin = tgrepBin; if (!bin) { ctx.ui.notify("tgrep binary not found (TGREP_BIN or PATH)", "warning"); diff --git a/packages/pi-tools/src/tgrep.ts b/packages/pi-tools/src/tgrep.ts index f341ff8..dddff28 100644 --- a/packages/pi-tools/src/tgrep.ts +++ b/packages/pi-tools/src/tgrep.ts @@ -5,6 +5,7 @@ import { promisify } from "node:util"; export const TGREP_BIN_ENV = "TGREP_BIN"; export const TGREP_TOOL_NAME = "tgrep"; +export const TGREP_INDEX_DIR = ".tgrep"; export const TGREP_TIME_BUDGET_MS = 30_000; export const TGREP_OUTPUT_MAX_BYTES = 200_000; export const TGREP_CONTEXT_MAX = 20; @@ -105,6 +106,15 @@ export function resolveTgrepBinary(explicit?: string, pathEnv = ""): string | un return undefined; } +/** True when the session cwd has a `.tgrep` directory (default tgrep index location). */ +export function hasTgrepIndex(cwd: string): boolean { + try { + return fs.statSync(path.join(cwd, TGREP_INDEX_DIR)).isDirectory(); + } catch { + return false; + } +} + const execFileAsync = promisify(execFile); /** Runs tgrep with bounded time and output, preserving output from normal process exits. */ diff --git a/packages/pi-tools/test/extension.test.ts b/packages/pi-tools/test/extension.test.ts index 13054ee..6814ddf 100644 --- a/packages/pi-tools/test/extension.test.ts +++ b/packages/pi-tools/test/extension.test.ts @@ -719,26 +719,35 @@ describe("pi-tools tgrep integration", () => { return bin; } + function writeIndexedCwd(): string { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + fs.mkdirSync(path.join(cwd, ".tgrep")); + return cwd; + } + /** Returns the tool names registered through the mocked Pi API. */ function toolNames(setup: { pi: { registerTool: any } }): string[] { return setup.pi.registerTool.mock.calls.map(([tool]: [any]) => tool.name); } - test("registers tgrep and tgrep-status when the binary is found", async () => { + test("registers tgrep and tgrep-status when the binary and index are found", async () => { process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); - const setup = await start(); + const cwd = writeIndexedCwd(); + const setup = await start(undefined, cwd); try { expect(toolNames(setup)).toContain("tgrep"); expect(setup.commands.has("tgrep-status")).toBe(true); } finally { await shutdown(setup); fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); } }); test("registers the complete tgrep parameter schema", async () => { process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); - const setup = await start(); + const cwd = writeIndexedCwd(); + const setup = await start(undefined, cwd); try { const tool = setup.pi.registerTool.mock.calls .map(([registered]: [any]) => registered) @@ -770,15 +779,17 @@ describe("pi-tools tgrep integration", () => { } finally { await shutdown(setup); fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); } }); test("discovers tgrep from PATH when no explicit path is configured", async () => { writeFakeBin("tgrep", "exit 1"); + const cwd = writeIndexedCwd(); const savedPath = process.env.PATH; process.env.PATH = binDir; try { - const setup = await start(); + const setup = await start(undefined, cwd); try { expect(toolNames(setup)).toContain("tgrep"); } finally { @@ -788,6 +799,7 @@ describe("pi-tools tgrep integration", () => { if (savedPath === undefined) delete process.env.PATH; else process.env.PATH = savedPath; fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); } }); @@ -805,24 +817,42 @@ describe("pi-tools tgrep integration", () => { test("skips tgrep when enableTgrep is false", async () => { process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); writeConfig({ enableTgrep: false }); - const setup = await start(); + const cwd = writeIndexedCwd(); + const setup = await start(undefined, cwd); + try { + expect(toolNames(setup)).not.toContain("tgrep"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("skips tgrep when the binary is found but no index exists", async () => { + process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + const setup = await start(undefined, cwd); try { expect(toolNames(setup)).not.toContain("tgrep"); + expect(setup.commands.has("tgrep-status")).toBe(true); } finally { await shutdown(setup); fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); } }); test("registers tgrep from the configured binary path", async () => { const bin = writeFakeBin("configured-tgrep", "exit 1"); writeConfig({ tgrepBinPath: bin }); - const setup = await start(); + const cwd = writeIndexedCwd(); + const setup = await start(undefined, cwd); try { expect(toolNames(setup)).toContain("tgrep"); } finally { await shutdown(setup); fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); } }); @@ -841,19 +871,41 @@ describe("pi-tools tgrep integration", () => { test("registers tgrep independently of the FFF mode", async () => { process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + const cwd = writeIndexedCwd(); try { for (const mode of ["tools-only", "override"]) { - const setup = await start(mode); + const setup = await start(mode, cwd); expect(toolNames(setup)).toContain("tgrep"); await shutdown(setup); } } finally { fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); + + test("steers tgrep by capability rather than repository size", async () => { + process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); + const cwd = writeIndexedCwd(); + const setup = await start(undefined, cwd); + try { + const tool = setup.pi.registerTool.mock.calls + .map(([t]: [any]) => t) + .find((t: any) => t.name === "tgrep"); + expect(tool.description).toContain("exact content search"); + expect(tool.description).not.toContain("large repo"); + expect(tool.promptGuidelines[0]).toContain("fuzzy, typo-tolerant, frecency-ranked"); + expect(tool.promptGuidelines[1]).toContain("exact literal or symbol search"); + expect(tool.promptGuidelines.join("\n")).not.toContain("large repo"); + } finally { + await shutdown(setup); + fs.rmSync(binDir, { recursive: true, force: true }); + fs.rmSync(cwd, { recursive: true, force: true }); } }); test("executes searches through the binary and reports no-match", async () => { - const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + const cwd = writeIndexedCwd(); process.env.TGREP_BIN = writeFakeBin("tgrep", "exit 1"); const setup = await start(undefined, cwd); try { @@ -871,7 +923,7 @@ describe("pi-tools tgrep integration", () => { }); test("returns vimgrep rows from the binary", async () => { - const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + const cwd = writeIndexedCwd(); process.env.TGREP_BIN = writeFakeBin("tgrep", "printf 'src/a.ts:3:7:hello\\n'"); const setup = await start(undefined, cwd); try { @@ -892,7 +944,7 @@ describe("pi-tools tgrep integration", () => { }); test("forwards every search option with a normalized workspace path", async () => { - const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + const cwd = writeIndexedCwd(); process.env.TGREP_BIN = writeFakeBin("tgrep", "printf '%s\\n' \"$@\""); const setup = await start(undefined, cwd); try { @@ -950,7 +1002,7 @@ describe("pi-tools tgrep integration", () => { }); test("rejects an already-aborted search without invoking the binary", async () => { - const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + const cwd = writeIndexedCwd(); const marker = path.join(cwd, "invoked"); process.env.TGREP_BIN = writeFakeBin("tgrep", `touch '${marker}'`); const setup = await start(undefined, cwd); @@ -973,7 +1025,7 @@ describe("pi-tools tgrep integration", () => { }); test("rejects paths outside the workspace before invoking the binary", async () => { - const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-tgrep-cwd-")); + const cwd = writeIndexedCwd(); const marker = path.join(cwd, "invoked"); process.env.TGREP_BIN = writeFakeBin("tgrep", `touch '${marker}'`); const setup = await start(undefined, cwd); diff --git a/packages/pi-tools/test/tgrep.test.ts b/packages/pi-tools/test/tgrep.test.ts index 8c66b68..7ea30e9 100644 --- a/packages/pi-tools/test/tgrep.test.ts +++ b/packages/pi-tools/test/tgrep.test.ts @@ -6,10 +6,12 @@ import path from "node:path"; import { buildTgrepArgs, formatTgrepResult, + hasTgrepIndex, resolveSearchRoot, resolveTgrepBinary, runTgrep, TGREP_CONTEXT_MAX, + TGREP_INDEX_DIR, TGREP_OUTPUT_MAX_BYTES, } from "../src/tgrep"; @@ -224,6 +226,27 @@ describe("resolveTgrepBinary", () => { }); }); +describe("hasTgrepIndex", () => { + test("is true only for a .tgrep directory in the given cwd", () => { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "tgrep-index-")); + const nested = path.join(cwd, "pkg"); + try { + expect(hasTgrepIndex(cwd)).toBe(false); + + fs.writeFileSync(path.join(cwd, TGREP_INDEX_DIR), "not a dir"); + expect(hasTgrepIndex(cwd)).toBe(false); + fs.rmSync(path.join(cwd, TGREP_INDEX_DIR)); + + fs.mkdirSync(path.join(cwd, TGREP_INDEX_DIR)); + fs.mkdirSync(nested); + expect(hasTgrepIndex(cwd)).toBe(true); + expect(hasTgrepIndex(nested)).toBe(false); + } finally { + fs.rmSync(cwd, { recursive: true, force: true }); + } + }); +}); + describe("formatTgrepResult", () => { test("returns match output verbatim", () => { expect(formatTgrepResult({ exit: 0, stdout: "a.ts:1:5:foo\n", stderr: "" })).toBe( From fd3c9927e75d4721bd509f5f6f719529effd9ffc Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Sat, 12 Sep 2026 13:41:19 +0000 Subject: [PATCH 09/11] fix(pi-tools): harden tgrep truncation, timeout, and abort Keep UTF-8 character boundaries when capping output, make the 30s child-process budget configurable, and report mid-run cancels as Operation aborted instead of a generic launch failure. Co-authored-by: OnlineChef --- docs/compatibility.md | 1 + packages/pi-tools/CHANGELOG.md | 2 +- packages/pi-tools/README.md | 3 +- packages/pi-tools/pi-tools.schema.json | 6 ++++ packages/pi-tools/src/config.ts | 15 +++++++++ packages/pi-tools/src/index.ts | 22 ++++++++++++- packages/pi-tools/src/tgrep.ts | 36 ++++++++++++++-------- packages/pi-tools/test/config.test.ts | 4 +++ packages/pi-tools/test/extension.test.ts | 1 + packages/pi-tools/test/tgrep.test.ts | 39 ++++++++++++++++++++++-- 10 files changed, 112 insertions(+), 17 deletions(-) diff --git a/docs/compatibility.md b/docs/compatibility.md index 05970fe..710b8e7 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -70,6 +70,7 @@ Concrete defaults: | home scan | `--fff-enable-home-scan` | `FFF_ENABLE_HOME_SCAN` | `config.enableHomeDirScanning` | `true` | | tgrep binary | — (no CLI flag by design) | `TGREP_BIN` | `config.tgrepBinPath` | `PATH` lookup | | tgrep toggle | — (config file only) | — | `config.enableTgrep` | `true` | +| tgrep time budget | — (no CLI flag by design) | `TGREP_TIME_BUDGET_MS` | `config.tgrepTimeBudgetMs` | `30000` | Mode valid values (`packages/pi-tools/src/config.ts:8` `VALID_MODES`): `tools-and-ui`, `tools-only`, `override`. diff --git a/packages/pi-tools/CHANGELOG.md b/packages/pi-tools/CHANGELOG.md index eeeb0ca..37539e5 100644 --- a/packages/pi-tools/CHANGELOG.md +++ b/packages/pi-tools/CHANGELOG.md @@ -4,7 +4,7 @@ ### Added -- Added the `tgrep` tool: trigram-indexed exact content search through an external [tgrep](https://github.com/microsoft/tgrep) binary. Literal by default, `file:line:col:text` output, exit 1 reported as no-match. Registered at session start only when the binary resolves (`TGREP_BIN`, `tgrepBinPath`, or `PATH`), the workspace has a `.tgrep/` index, and `enableTgrep` is not `false`; mode-independent. Prompt guidelines steer on capability (exact/literal vs fuzzy/frecency), not repository size. +- Added the `tgrep` tool: trigram-indexed exact content search through an external [tgrep](https://github.com/microsoft/tgrep) binary. Literal by default, `file:line:col:text` output, exit 1 reported as no-match. Registered at session start only when the binary resolves (`TGREP_BIN`, `tgrepBinPath`, or `PATH`), the workspace has a `.tgrep/` index, and `enableTgrep` is not `false`; mode-independent. Prompt guidelines steer on capability (exact/literal vs fuzzy/frecency), not repository size. Optional `tgrepTimeBudgetMs` / `TGREP_TIME_BUDGET_MS` (default 30s). Output truncation stays on a UTF-8 character boundary; mid-run abort reports `Operation aborted`. - Added the `/tgrep-status` command showing tgrep index and server status for the workspace. - Added `ffgrep.maxMatchesPerFile` to keep a single generated or vendored file from dominating a result page. - Added `ffgrep.compact` for deterministic `path:line:match` output without context blocks. diff --git a/packages/pi-tools/README.md b/packages/pi-tools/README.md index d0bd13e..066d916 100644 --- a/packages/pi-tools/README.md +++ b/packages/pi-tools/README.md @@ -151,10 +151,11 @@ Create `pi-tools.json` in Pi’s agent directory. The default location is `~/.pi | `enableHomeDirScanning` | boolean | `true` | Allows scanning when Pi starts in the home directory. | | `enableTgrep` | boolean | `true` | Registers the `tgrep` tool when the binary and a `.tgrep/` index are found. | | `tgrepBinPath` | string | PATH lookup | Explicit path to the `tgrep` binary. | +| `tgrepTimeBudgetMs` | number | `30000` | Child-process time budget in milliseconds. `TGREP_TIME_BUDGET_MS` overrides this. | Malformed configuration, unknown fields, and invalid values prevent the extension from loading and identify the configuration path in the error. `/fff-mode` changes session state only; it does not edit this file. -The `tgrep` binary resolves as `TGREP_BIN` environment variable, then `tgrepBinPath`, then a `tgrep` executable on `PATH`. An explicit path that is set but not executable disables the tool instead of falling back, so a typo surfaces instead of silently changing the search backend. Binary and `.tgrep/` index are resolved at session start, when the workspace cwd is known. +The `tgrep` binary resolves as `TGREP_BIN` environment variable, then `tgrepBinPath`, then a `tgrep` executable on `PATH`. An explicit path that is set but not executable disables the tool instead of falling back, so a typo surfaces instead of silently changing the search backend. Binary and `.tgrep/` index are resolved at session start, when the workspace cwd is known. Installing tgrep or building an index mid-session requires `/reload` before the tool appears. ## Database resolution diff --git a/packages/pi-tools/pi-tools.schema.json b/packages/pi-tools/pi-tools.schema.json index bdb884b..92f54dc 100644 --- a/packages/pi-tools/pi-tools.schema.json +++ b/packages/pi-tools/pi-tools.schema.json @@ -46,6 +46,12 @@ "type": "string", "minLength": 1, "description": "Explicit path to the tgrep binary; the TGREP_BIN environment variable takes precedence over config.tgrepBinPath, then PATH is used as a fallback." + }, + "tgrepTimeBudgetMs": { + "type": "integer", + "minimum": 1, + "default": 30000, + "description": "Maximum milliseconds for a tgrep child process. TGREP_TIME_BUDGET_MS overrides this value." } } } diff --git a/packages/pi-tools/src/config.ts b/packages/pi-tools/src/config.ts index fdbce23..ed914f2 100644 --- a/packages/pi-tools/src/config.ts +++ b/packages/pi-tools/src/config.ts @@ -18,6 +18,7 @@ export interface FffConfig { enableHomeDirScanning?: boolean; enableTgrep?: boolean; tgrepBinPath?: string; + tgrepTimeBudgetMs?: number; } const CONFIG_KEYS = new Set([ @@ -29,6 +30,7 @@ const CONFIG_KEYS = new Set([ "enableHomeDirScanning", "enableTgrep", "tgrepBinPath", + "tgrepTimeBudgetMs", ]); /** @@ -87,6 +89,7 @@ export function loadConfig(agentDir = piDataDir()): FffConfig { validateBoolean(configPath, parsed, "enableFsRootScanning"); validateBoolean(configPath, parsed, "enableHomeDirScanning"); validateBoolean(configPath, parsed, "enableTgrep"); + validatePositiveInteger(configPath, parsed, "tgrepTimeBudgetMs"); return parsed as FffConfig; } @@ -126,3 +129,15 @@ function validateBoolean( throw invalidConfig(configPath, `"${key}" must be a boolean`); } } + +function validatePositiveInteger( + configPath: string, + config: Record, + key: "tgrepTimeBudgetMs", +): void { + const value = config[key]; + if (value === undefined) return; + if (typeof value !== "number" || !Number.isInteger(value) || value < 1) { + throw invalidConfig(configPath, `"${key}" must be a positive integer`); + } +} diff --git a/packages/pi-tools/src/index.ts b/packages/pi-tools/src/index.ts index 1456265..a20f2b8 100644 --- a/packages/pi-tools/src/index.ts +++ b/packages/pi-tools/src/index.ts @@ -31,6 +31,8 @@ import { runTgrep, TGREP_BIN_ENV, TGREP_CONTEXT_MAX, + TGREP_TIME_BUDGET_ENV, + TGREP_TIME_BUDGET_MS, TGREP_TOOL_NAME, } from "./tgrep"; @@ -283,6 +285,7 @@ export default function fffExtension(pi: ExtensionAPI) { const config = loadConfig(); // Resolved at session start so cwd (index) and flags are known. let tgrepBin: string | undefined; + let tgrepTimeBudgetMs = TGREP_TIME_BUDGET_MS; function resolveTgrepBin(): string | undefined { if (config.enableTgrep === false) return undefined; @@ -328,6 +331,15 @@ export default function fffExtension(pi: ExtensionAPI) { : undefined; } + function parsePositiveInt(value: unknown): number | undefined { + if (typeof value === "number" && Number.isInteger(value) && value >= 1) return value; + if (typeof value === "string" && value.trim() !== "") { + const parsed = Number(value); + if (Number.isInteger(parsed) && parsed >= 1) return parsed; + } + return undefined; + } + let currentMode: FffMode = "tools-and-ui"; let toolNames = resolveToolNames(currentMode); let resolvedDbPaths: ReturnType; @@ -374,6 +386,13 @@ export default function fffExtension(pi: ExtensionAPI) { true, parseBoolean, ); + tgrepTimeBudgetMs = getConfigValue( + "tgrep-time-budget-ms", + TGREP_TIME_BUDGET_ENV, + config.tgrepTimeBudgetMs, + TGREP_TIME_BUDGET_MS, + parsePositiveInt, + ); } function getMode(): FffMode { @@ -707,7 +726,7 @@ export default function fffExtension(pi: ExtensionAPI) { maxCount: params.maxCount, noIndex: params.noIndex, }), - { cwd: activeCwd, signal }, + { cwd: activeCwd, signal, timeoutMs: tgrepTimeBudgetMs }, ); return { content: [{ type: "text", text: output }], details: {} }; }, @@ -1420,6 +1439,7 @@ export default function fffExtension(pi: ExtensionAPI) { try { const output = await runTgrep(bin, ["status", activeCwd], { cwd: activeCwd, + timeoutMs: tgrepTimeBudgetMs, }); ctx.ui.notify(output || "tgrep status: no output", "info"); } catch (error: unknown) { diff --git a/packages/pi-tools/src/tgrep.ts b/packages/pi-tools/src/tgrep.ts index dddff28..04b2ba0 100644 --- a/packages/pi-tools/src/tgrep.ts +++ b/packages/pi-tools/src/tgrep.ts @@ -4,6 +4,7 @@ import path from "node:path"; import { promisify } from "node:util"; export const TGREP_BIN_ENV = "TGREP_BIN"; +export const TGREP_TIME_BUDGET_ENV = "TGREP_TIME_BUDGET_MS"; export const TGREP_TOOL_NAME = "tgrep"; export const TGREP_INDEX_DIR = ".tgrep"; export const TGREP_TIME_BUDGET_MS = 30_000; @@ -31,11 +32,13 @@ export interface TgrepResult { stderr: string; } -type ExecFn = ( - bin: string, - args: string[], - opts: { cwd: string; signal?: AbortSignal }, -) => Promise; +export interface TgrepExecOptions { + cwd: string; + signal?: AbortSignal; + timeoutMs?: number; +} + +type ExecFn = (bin: string, args: string[], opts: TgrepExecOptions) => Promise; /** Normalizes an optional scalar or list into an iterable array. */ function repeatAll(values: string | string[] | undefined): string[] { @@ -121,17 +124,19 @@ const execFileAsync = promisify(execFile); async function defaultExec( bin: string, args: string[], - opts: { cwd: string; signal?: AbortSignal }, + opts: TgrepExecOptions, ): Promise { try { const { stdout, stderr } = await execFileAsync(bin, args, { cwd: opts.cwd, signal: opts.signal, - timeout: TGREP_TIME_BUDGET_MS, + timeout: opts.timeoutMs ?? TGREP_TIME_BUDGET_MS, maxBuffer: TGREP_OUTPUT_MAX_BYTES * 2, }); return { exit: 0, stdout, stderr }; } catch (error: unknown) { + if (isAbortError(error) || opts.signal?.aborted) + throw new Error("Operation aborted"); const execError = error as { code?: number | string; stdout?: string; @@ -157,7 +162,7 @@ async function defaultExec( export async function runTgrep( bin: string, args: string[], - opts: { cwd: string; signal?: AbortSignal }, + opts: TgrepExecOptions, exec: ExecFn = defaultExec, ): Promise { if (opts.signal?.aborted) throw new Error("Operation aborted"); @@ -172,7 +177,7 @@ export async function runTgrep( export async function runTgrepRaw( bin: string, args: string[], - opts: { cwd: string; signal?: AbortSignal }, + opts: TgrepExecOptions, exec: ExecFn = defaultExec, ): Promise { return exec(bin, args, opts); @@ -192,12 +197,14 @@ export function formatTgrepResult(result: TgrepResult): string { return `${notice}${body}`; } -/** Truncates oversized output and appends a hint for narrowing the search. */ +/** Truncates oversized output on a UTF-8 character boundary. */ function truncateBytes(text: string): string { const buf = Buffer.from(text); if (buf.length <= TGREP_OUTPUT_MAX_BYTES) return text; - const head = buf.subarray(0, TGREP_OUTPUT_MAX_BYTES).toString(); - return `${head}\n… [truncated ${buf.length - TGREP_OUTPUT_MAX_BYTES} bytes: narrow with fileType/glob]`; + let end = TGREP_OUTPUT_MAX_BYTES; + while (end > 0 && (buf[end] & 0xc0) === 0x80) end--; + const head = buf.subarray(0, end).toString(); + return `${head}\n… [truncated ${buf.length - end} bytes: narrow with fileType/glob]`; } /** Normalizes context to the supported non-negative integer range. */ @@ -205,3 +212,8 @@ function clampContext(context: number | undefined): number { if (!context || context < 0) return 0; return Math.min(Math.floor(context), TGREP_CONTEXT_MAX); } + +function isAbortError(error: unknown): boolean { + if (error instanceof Error && error.name === "AbortError") return true; + return (error as { code?: unknown }).code === "ABORT_ERR"; +} diff --git a/packages/pi-tools/test/config.test.ts b/packages/pi-tools/test/config.test.ts index df0aead..afbcbfb 100644 --- a/packages/pi-tools/test/config.test.ts +++ b/packages/pi-tools/test/config.test.ts @@ -33,6 +33,7 @@ describe("loadConfig", () => { enableHomeDirScanning: false, enableTgrep: false, tgrepBinPath: "/usr/local/bin/tgrep", + tgrepTimeBudgetMs: 45000, }; writeConfig(config); @@ -70,6 +71,9 @@ describe("loadConfig", () => { [{ enableHomeDirScanning: "false" }, '"enableHomeDirScanning" must be a boolean'], [{ enableTgrep: "yes" }, '"enableTgrep" must be a boolean'], [{ tgrepBinPath: "" }, '"tgrepBinPath" must be a non-empty string'], + [{ tgrepTimeBudgetMs: 0 }, '"tgrepTimeBudgetMs" must be a positive integer'], + [{ tgrepTimeBudgetMs: 1.5 }, '"tgrepTimeBudgetMs" must be a positive integer'], + [{ tgrepTimeBudgetMs: "30000" }, '"tgrepTimeBudgetMs" must be a positive integer'], ]; for (const [config, message] of cases) { diff --git a/packages/pi-tools/test/extension.test.ts b/packages/pi-tools/test/extension.test.ts index 6814ddf..ff2b43e 100644 --- a/packages/pi-tools/test/extension.test.ts +++ b/packages/pi-tools/test/extension.test.ts @@ -188,6 +188,7 @@ const CONFIG_ENV_KEYS = [ "FFF_ENABLE_ROOT_SCAN", "FFF_ENABLE_HOME_SCAN", "TGREP_BIN", + "TGREP_TIME_BUDGET_MS", ] as const; const savedEnv: Record = {}; diff --git a/packages/pi-tools/test/tgrep.test.ts b/packages/pi-tools/test/tgrep.test.ts index 7ea30e9..22cd7ef 100644 --- a/packages/pi-tools/test/tgrep.test.ts +++ b/packages/pi-tools/test/tgrep.test.ts @@ -307,7 +307,6 @@ describe("formatTgrepResult", () => { test("measures truncation in bytes for multibyte output and preserves warnings", () => { const stdout = "€".repeat(Math.ceil((TGREP_OUTPUT_MAX_BYTES + 4) / 3)); - const omittedBytes = Buffer.byteLength(stdout) - TGREP_OUTPUT_MAX_BYTES; const out = formatTgrepResult({ exit: 0, stdout, @@ -315,8 +314,18 @@ describe("formatTgrepResult", () => { }); expect(out.startsWith("[tgrep: warning: stale index]\n")).toBe(true); - expect(out).toContain(`[truncated ${omittedBytes} bytes: narrow with fileType/glob]`); + expect(out).not.toContain("\uFFFD"); expect(out).not.toContain("ignored diagnostic"); + const hint = out.match(/\[truncated (\d+) bytes: narrow with fileType\/glob\]/); + expect(hint).not.toBeNull(); + expect(Number(hint![1])).toBeGreaterThanOrEqual( + Buffer.byteLength(stdout) - TGREP_OUTPUT_MAX_BYTES, + ); + const printed = out.slice( + "[tgrep: warning: stale index]\n".length, + out.indexOf("\n… [truncated"), + ); + expect(printed.endsWith("€")).toBe(true); }); }); @@ -346,6 +355,32 @@ describe("runTgrep", () => { expect(exec).not.toHaveBeenCalled(); }); + test("maps a mid-run abort to Operation aborted", async () => { + const controller = new AbortController(); + const running = runTgrep( + process.execPath, + ["-e", "setTimeout(() => {}, 30000)"], + { cwd: "/tmp", signal: controller.signal }, + ); + await new Promise((resolve) => setImmediate(resolve)); + controller.abort(); + await expect(running).rejects.toThrow("Operation aborted"); + }); + + test("forwards a custom time budget to the executor", async () => { + let seen: number | undefined; + await runTgrep( + "/fake/tgrep", + ["--", "x", "."], + { cwd: "/tmp", timeoutMs: 12_000 }, + async (_bin, _args, opts) => { + seen = opts.timeoutMs; + return { exit: 1, stdout: "", stderr: "" }; + }, + ); + expect(seen).toBe(12_000); + }); + test("maps the real process no-match exit and stderr warning", async () => { const out = await runTgrep( process.execPath, From 709f7d6f14687004cc27926832e7e932c5c434d5 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Sat, 12 Sep 2026 15:10:36 +0000 Subject: [PATCH 10/11] fix(pi-tools): drop tgrep noIndex and satisfy oxfmt Keep the tool index-only: --no-index is no longer a public parameter, after edits the guidelines point at ffgrep, and oxfmt is clean on the tgrep sources. Co-authored-by: OnlineChef --- README.md | 7 +++++-- packages/pi-tools/CHANGELOG.md | 2 +- packages/pi-tools/README.md | 3 +-- packages/pi-tools/src/index.ts | 9 +-------- packages/pi-tools/src/tgrep.ts | 11 ++++++----- packages/pi-tools/test/extension.test.ts | 3 --- packages/pi-tools/test/tgrep.test.ts | 20 ++++++++++++-------- 7 files changed, 26 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index 60402e6..ba06b7c 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,7 @@ pi install -l npm:@groeponline/pi-tools | `find` (spawns `fd`) | `fffind` (FFF `fileSearch`) | Fuzzy matching, frecency ranking, git-aware, pre-indexed | | `grep` (spawns `rg`) | `ffgrep` (FFF `grep`) | SIMD-accelerated, frecency-ordered, mmap-cached, no subprocess | | *(none)* | `fff-multi-grep` (FFF `multiGrep`, opt-in) | OR-logic multi-pattern search via Aho-Corasick | +| *(none)* | `tgrep` (external [tgrep](https://github.com/microsoft/tgrep) binary) | Trigram-indexed exact search when a workspace `.tgrep/` index exists | | `@` file autocomplete (fd-backed) | `@` file autocomplete (FFF-backed, default) | Fuzzy ranking from the FFF index and frecency | ### Modes @@ -54,20 +55,22 @@ Three operating modes, switchable at runtime with `/fff-mode`: | `tools-only` | Only tool injection. Keeps pi's native editor autocomplete. | | `override` | Replaces pi's built-in `grep` and `find` with FFF implementations. With `PI_FFF_MULTIGREP=1`, also registers `multi_grep`. | -Set `PI_FFF_MULTIGREP=1` to opt in to `fff-multi-grep` (or `multi_grep` in `override` mode). Without it, only `ffgrep` and `fffind` are registered. +Set `PI_FFF_MULTIGREP=1` to opt in to `fff-multi-grep` (or `multi_grep` in `override` mode). Without it, only `ffgrep` and `fffind` are registered. `tgrep` is registered independently of mode when the binary and a `.tgrep/` index are found. -Env vars: `PI_FFF_MODE`, `FFF_FRECENCY_DB`, `FFF_HISTORY_DB`. Flags: `--fff-mode`, `--fff-frecency-db`, `--fff-history-db`. The databases default to your existing fff.nvim ones when present, otherwise `~/.pi/agent/fff/`. +Env vars: `PI_FFF_MODE`, `FFF_FRECENCY_DB`, `FFF_HISTORY_DB`, `TGREP_BIN`, `TGREP_TIME_BUDGET_MS`. Flags: `--fff-mode`, `--fff-frecency-db`, `--fff-history-db`. Config (`~/.pi/agent/pi-tools.json`): `enableTgrep` (default true), `tgrepBinPath`, `tgrepTimeBudgetMs` (default 30000). The databases default to your existing fff.nvim ones when present, otherwise `~/.pi/agent/fff/`. ### Agent-facing tools - `ffgrep`. Content search. Accepts `path`, `exclude` (comma, space, or array; leading `!` optional), `caseSensitive`, `context`, and cursor pagination. Auto-detects regex, falls back to fuzzy on zero exact matches, rejects `.*`-style wildcard-only patterns up front. - `fffind`. Path and filename search. Matches the whole repo-relative path, not just the filename. Frecency-aware. The weak-match detector flags scattered fuzzy noise before it floods the agent's context. +- `tgrep`. Exact literal/symbol search through an external tgrep binary. Registered only when the binary resolves (`TGREP_BIN` → `tgrepBinPath` → `PATH`) and the workspace has a `.tgrep/` index. After your own edits, use `ffgrep`. ### Commands - `/fff-mode [tools-and-ui | tools-only | override]`. Show or switch the mode. - `/fff-health`. Picker, frecency, and git integration status. - `/fff-rescan`. Force a rescan. +- `/tgrep-status`. tgrep index and server status for the workspace. Source: [`packages/pi-tools/`](./packages/pi-tools/). Full documentation: [`packages/pi-tools/README.md`](./packages/pi-tools/README.md). diff --git a/packages/pi-tools/CHANGELOG.md b/packages/pi-tools/CHANGELOG.md index 37539e5..1dd8633 100644 --- a/packages/pi-tools/CHANGELOG.md +++ b/packages/pi-tools/CHANGELOG.md @@ -4,7 +4,7 @@ ### Added -- Added the `tgrep` tool: trigram-indexed exact content search through an external [tgrep](https://github.com/microsoft/tgrep) binary. Literal by default, `file:line:col:text` output, exit 1 reported as no-match. Registered at session start only when the binary resolves (`TGREP_BIN`, `tgrepBinPath`, or `PATH`), the workspace has a `.tgrep/` index, and `enableTgrep` is not `false`; mode-independent. Prompt guidelines steer on capability (exact/literal vs fuzzy/frecency), not repository size. Optional `tgrepTimeBudgetMs` / `TGREP_TIME_BUDGET_MS` (default 30s). Output truncation stays on a UTF-8 character boundary; mid-run abort reports `Operation aborted`. +- Added the `tgrep` tool: trigram-indexed exact content search through an external [tgrep](https://github.com/microsoft/tgrep) binary. Literal by default, `file:line:col:text` output, exit 1 reported as no-match. Registered at session start only when the binary resolves (`TGREP_BIN`, `tgrepBinPath`, or `PATH`), the workspace has a `.tgrep/` index, and `enableTgrep` is not `false`; mode-independent. Prompt guidelines steer on capability (exact/literal vs fuzzy/frecency), not repository size. Optional `tgrepTimeBudgetMs` / `TGREP_TIME_BUDGET_MS` (default 30s). Output truncation stays on a UTF-8 character boundary; mid-run abort reports `Operation aborted`. `--no-index` is not exposed; after edits use `ffgrep`. - Added the `/tgrep-status` command showing tgrep index and server status for the workspace. - Added `ffgrep.maxMatchesPerFile` to keep a single generated or vendored file from dominating a result page. - Added `ffgrep.compact` for deterministic `path:line:match` output without context blocks. diff --git a/packages/pi-tools/README.md b/packages/pi-tools/README.md index 066d916..c7807df 100644 --- a/packages/pi-tools/README.md +++ b/packages/pi-tools/README.md @@ -101,9 +101,8 @@ Use a concrete substring, identifier, or expression. A wildcard-only expression | `count` | boolean, optional | Prints the match count per file. | | `context` | number, optional | Context lines before and after a match; range 0–20. | | `maxCount` | number, optional | Limits matches per file. | -| `noIndex` | boolean, optional | Reads files from disk instead of the index. Use after your own edits when the latest content must be visible. | -Output is `file:line:col:text` rows. Exit code 1 (no match) is reported as `No matches found`, not as a failure. A leading `[tgrep: ...]` line carries the binary's stderr freshness warning. Only index-safe flags are forwarded; full-scan forcers (`--hidden`, `--no-ignore`, `-u`, `-a`, `--encoding`) are excluded by design. The tool is not registered without a workspace `.tgrep/` directory, so a no-index full scan is not offered as a search tool. +Output is `file:line:col:text` rows. Exit code 1 (no match) is reported as `No matches found`, not as a failure. A leading `[tgrep: ...]` line carries the binary's stderr freshness warning. Only index-safe flags are forwarded; full-scan forcers (`--hidden`, `--no-ignore`, `-u`, `-a`, `--encoding`, `--no-index`) are excluded by design. After your own edits, use `ffgrep` because the index lags watcher events. ### Optional multi-pattern search diff --git a/packages/pi-tools/src/index.ts b/packages/pi-tools/src/index.ts index a20f2b8..0da429d 100644 --- a/packages/pi-tools/src/index.ts +++ b/packages/pi-tools/src/index.ts @@ -683,12 +683,6 @@ export default function fffExtension(pi: ExtensionAPI) { }), ), maxCount: Type.Optional(Type.Number({ description: "Limit matches per file." })), - noIndex: Type.Optional( - Type.Boolean({ - description: - "Read files from disk instead of the index. Use after your own edits when the latest content must be visible.", - }), - ), }); queueTool(() => TGREP_TOOL_NAME, { @@ -701,7 +695,7 @@ export default function fffExtension(pi: ExtensionAPI) { `${TGREP_TOOL_NAME}: keep literal: true (default) for symbols; set false for regex.`, `${TGREP_TOOL_NAME}: narrow with fileType/glob before raising maxCount.`, `${TGREP_TOOL_NAME}: use filesOnly: true first on broad queries, then search specific files.`, - `${TGREP_TOOL_NAME}: after your own edits, pass noIndex: true or use ${names.grep}; the index lags watcher events.`, + `${TGREP_TOOL_NAME}: after your own edits, use ${names.grep}; the index lags watcher events.`, `${TGREP_TOOL_NAME}: a '[tgrep: ...]' line reports index freshness, never drop it from summaries.`, ], parameters: tgrepSchema, @@ -724,7 +718,6 @@ export default function fffExtension(pi: ExtensionAPI) { count: params.count, context: params.context, maxCount: params.maxCount, - noIndex: params.noIndex, }), { cwd: activeCwd, signal, timeoutMs: tgrepTimeBudgetMs }, ); diff --git a/packages/pi-tools/src/tgrep.ts b/packages/pi-tools/src/tgrep.ts index 04b2ba0..21908f4 100644 --- a/packages/pi-tools/src/tgrep.ts +++ b/packages/pi-tools/src/tgrep.ts @@ -23,7 +23,6 @@ export interface TgrepSearchOptions { count?: boolean; context?: number; maxCount?: number; - noIndex?: boolean; } export interface TgrepResult { @@ -38,7 +37,11 @@ export interface TgrepExecOptions { timeoutMs?: number; } -type ExecFn = (bin: string, args: string[], opts: TgrepExecOptions) => Promise; +type ExecFn = ( + bin: string, + args: string[], + opts: TgrepExecOptions, +) => Promise; /** Normalizes an optional scalar or list into an iterable array. */ function repeatAll(values: string | string[] | undefined): string[] { @@ -63,7 +66,6 @@ export function buildTgrepArgs(options: TgrepSearchOptions): string[] { if (context > 0) args.push("-A", String(context), "-B", String(context)); if (options.maxCount !== undefined) args.push("--max-count", String(Math.max(1, Math.floor(options.maxCount)))); - if (options.noIndex) args.push("--no-index"); // Separator keeps patterns like "serve" or "-x" from parsing as subcommands. args.push("--", options.pattern, options.root); return args; @@ -135,8 +137,7 @@ async function defaultExec( }); return { exit: 0, stdout, stderr }; } catch (error: unknown) { - if (isAbortError(error) || opts.signal?.aborted) - throw new Error("Operation aborted"); + if (isAbortError(error) || opts.signal?.aborted) throw new Error("Operation aborted"); const execError = error as { code?: number | string; stdout?: string; diff --git a/packages/pi-tools/test/extension.test.ts b/packages/pi-tools/test/extension.test.ts index ff2b43e..17309bd 100644 --- a/packages/pi-tools/test/extension.test.ts +++ b/packages/pi-tools/test/extension.test.ts @@ -765,7 +765,6 @@ describe("pi-tools tgrep integration", () => { "glob", "literal", "maxCount", - "noIndex", "path", "pattern", "wholeWord", @@ -966,7 +965,6 @@ describe("pi-tools tgrep integration", () => { count: true, context: 2, maxCount: 5, - noIndex: true, }, new AbortController().signal, ); @@ -990,7 +988,6 @@ describe("pi-tools tgrep integration", () => { "2", "--max-count", "5", - "--no-index", "--", "fn main", "test", diff --git a/packages/pi-tools/test/tgrep.test.ts b/packages/pi-tools/test/tgrep.test.ts index 22cd7ef..7edf497 100644 --- a/packages/pi-tools/test/tgrep.test.ts +++ b/packages/pi-tools/test/tgrep.test.ts @@ -43,7 +43,6 @@ describe("buildTgrepArgs", () => { fileType: ["rust", "py"], glob: "src/**", filesOnly: true, - noIndex: true, }), ).toEqual([ "--vimgrep", @@ -56,7 +55,6 @@ describe("buildTgrepArgs", () => { "--glob", "src/**", "--files-with-matches", - "--no-index", "--", "fn main", ".", @@ -112,7 +110,14 @@ describe("buildTgrepArgs", () => { test("never emits full-scan forcing flags", () => { const args = buildTgrepArgs({ pattern: "x", root: "." }).join(" "); - for (const banned of ["--hidden", "--no-ignore", "--text", "--encoding", "-u"]) { + for (const banned of [ + "--hidden", + "--no-ignore", + "--text", + "--encoding", + "-u", + "--no-index", + ]) { expect(args.includes(banned)).toBe(false); } }); @@ -357,11 +362,10 @@ describe("runTgrep", () => { test("maps a mid-run abort to Operation aborted", async () => { const controller = new AbortController(); - const running = runTgrep( - process.execPath, - ["-e", "setTimeout(() => {}, 30000)"], - { cwd: "/tmp", signal: controller.signal }, - ); + const running = runTgrep(process.execPath, ["-e", "setTimeout(() => {}, 30000)"], { + cwd: "/tmp", + signal: controller.signal, + }); await new Promise((resolve) => setImmediate(resolve)); controller.abort(); await expect(running).rejects.toThrow("Operation aborted"); From 59b799fc86956ba2c7634c85b6b48edbc2959617 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Sat, 12 Sep 2026 15:14:34 +0000 Subject: [PATCH 11/11] fix(pi-tools): keep tgrep output within the 200KB cap Reserve space for the truncation hint and any stderr notice so the complete tool response stays within TGREP_OUTPUT_MAX_BYTES. Co-authored-by: OnlineChef --- packages/pi-tools/src/tgrep.ts | 20 ++++++++++++++------ packages/pi-tools/test/tgrep.test.ts | 12 +++++++++++- 2 files changed, 25 insertions(+), 7 deletions(-) diff --git a/packages/pi-tools/src/tgrep.ts b/packages/pi-tools/src/tgrep.ts index 21908f4..e40003c 100644 --- a/packages/pi-tools/src/tgrep.ts +++ b/packages/pi-tools/src/tgrep.ts @@ -193,19 +193,27 @@ export function formatTgrepResult(result: TgrepResult): string { const notice = warning ? `[tgrep: ${warning}]\n` : ""; if (result.exit === 2) throw new Error(`tgrep search failed: ${warning || "unknown error"}`); - const body = truncateBytes(result.stdout.trim()); + const budget = Math.max(0, TGREP_OUTPUT_MAX_BYTES - Buffer.byteLength(notice)); + const body = truncateBytes(result.stdout.trim(), budget); if (result.exit === 1 || body === "") return `${notice}No matches found`; return `${notice}${body}`; } /** Truncates oversized output on a UTF-8 character boundary. */ -function truncateBytes(text: string): string { +function truncateBytes(text: string, maxBytes = TGREP_OUTPUT_MAX_BYTES): string { const buf = Buffer.from(text); - if (buf.length <= TGREP_OUTPUT_MAX_BYTES) return text; - let end = TGREP_OUTPUT_MAX_BYTES; + if (buf.length <= maxBytes) return text; + const hint = (omitted: number) => + `\n… [truncated ${omitted} bytes: narrow with fileType/glob]`; + const reserved = Buffer.byteLength(hint(buf.length)); + if (reserved >= maxBytes) { + let end = maxBytes; + while (end > 0 && (buf[end] & 0xc0) === 0x80) end--; + return buf.subarray(0, end).toString(); + } + let end = maxBytes - reserved; while (end > 0 && (buf[end] & 0xc0) === 0x80) end--; - const head = buf.subarray(0, end).toString(); - return `${head}\n… [truncated ${buf.length - end} bytes: narrow with fileType/glob]`; + return `${buf.subarray(0, end).toString()}${hint(buf.length - end)}`; } /** Normalizes context to the supported non-negative integer range. */ diff --git a/packages/pi-tools/test/tgrep.test.ts b/packages/pi-tools/test/tgrep.test.ts index 7edf497..2f18162 100644 --- a/packages/pi-tools/test/tgrep.test.ts +++ b/packages/pi-tools/test/tgrep.test.ts @@ -307,7 +307,16 @@ describe("formatTgrepResult", () => { const out = formatTgrepResult({ exit: 0, stdout: big, stderr: "" }); expect(out).toContain("truncated"); expect(out).toContain("fileType/glob"); - expect(Buffer.byteLength(out)).toBeLessThan(Buffer.byteLength(big) + 200); + expect(Buffer.byteLength(out)).toBeLessThanOrEqual(TGREP_OUTPUT_MAX_BYTES); + }); + + test("keeps a stderr notice plus body within the output byte cap", () => { + const warning = "warning: stale index"; + const body = "x".repeat(TGREP_OUTPUT_MAX_BYTES); + const out = formatTgrepResult({ exit: 0, stdout: body, stderr: warning }); + expect(out.startsWith(`[tgrep: ${warning}]\n`)).toBe(true); + expect(out).toContain("truncated"); + expect(Buffer.byteLength(out)).toBeLessThanOrEqual(TGREP_OUTPUT_MAX_BYTES); }); test("measures truncation in bytes for multibyte output and preserves warnings", () => { @@ -321,6 +330,7 @@ describe("formatTgrepResult", () => { expect(out.startsWith("[tgrep: warning: stale index]\n")).toBe(true); expect(out).not.toContain("\uFFFD"); expect(out).not.toContain("ignored diagnostic"); + expect(Buffer.byteLength(out)).toBeLessThanOrEqual(TGREP_OUTPUT_MAX_BYTES); const hint = out.match(/\[truncated (\d+) bytes: narrow with fileType\/glob\]/); expect(hint).not.toBeNull(); expect(Number(hint![1])).toBeGreaterThanOrEqual(