Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -105,3 +105,6 @@ Pods/
.factory/
.jules/
.remember/

# tgrep trigram indexes (per-repo, rebuilt via `tgrep index .`)
.tgrep/
7 changes: 5 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ pi install -l npm:@groeponline/pi-tools
| `find` (spawns `fd`) | `fffind` (FFF `fileSearch`) | Fuzzy matching, frecency ranking, git-aware, pre-indexed |
| `grep` (spawns `rg`) | `ffgrep` (FFF `grep`) | SIMD-accelerated, frecency-ordered, mmap-cached, no subprocess |
| *(none)* | `fff-multi-grep` (FFF `multiGrep`, opt-in) | OR-logic multi-pattern search via Aho-Corasick |
| *(none)* | `tgrep` (external [tgrep](https://github.com/microsoft/tgrep) binary) | Trigram-indexed exact search when a workspace `.tgrep/` index exists |
| `@` file autocomplete (fd-backed) | `@` file autocomplete (FFF-backed, default) | Fuzzy ranking from the FFF index and frecency |

### Modes
Expand All @@ -54,20 +55,22 @@ Three operating modes, switchable at runtime with `/fff-mode`:
| `tools-only` | Only tool injection. Keeps pi's native editor autocomplete. |
| `override` | Replaces pi's built-in `grep` and `find` with FFF implementations. With `PI_FFF_MULTIGREP=1`, also registers `multi_grep`. |

Set `PI_FFF_MULTIGREP=1` to opt in to `fff-multi-grep` (or `multi_grep` in `override` mode). Without it, only `ffgrep` and `fffind` are registered.
Set `PI_FFF_MULTIGREP=1` to opt in to `fff-multi-grep` (or `multi_grep` in `override` mode). Without it, only `ffgrep` and `fffind` are registered. `tgrep` is registered independently of mode when the binary and a `.tgrep/` index are found.

Env vars: `PI_FFF_MODE`, `FFF_FRECENCY_DB`, `FFF_HISTORY_DB`. Flags: `--fff-mode`, `--fff-frecency-db`, `--fff-history-db`. The databases default to your existing fff.nvim ones when present, otherwise `~/.pi/agent/fff/`.
Env vars: `PI_FFF_MODE`, `FFF_FRECENCY_DB`, `FFF_HISTORY_DB`, `TGREP_BIN`, `TGREP_TIME_BUDGET_MS`. Flags: `--fff-mode`, `--fff-frecency-db`, `--fff-history-db`. Config (`~/.pi/agent/pi-tools.json`): `enableTgrep` (default true), `tgrepBinPath`, `tgrepTimeBudgetMs` (default 30000). The databases default to your existing fff.nvim ones when present, otherwise `~/.pi/agent/fff/`.

### Agent-facing tools

- `ffgrep`. Content search. Accepts `path`, `exclude` (comma, space, or array; leading `!` optional), `caseSensitive`, `context`, and cursor pagination. Auto-detects regex, falls back to fuzzy on zero exact matches, rejects `.*`-style wildcard-only patterns up front.
- `fffind`. Path and filename search. Matches the whole repo-relative path, not just the filename. Frecency-aware. The weak-match detector flags scattered fuzzy noise before it floods the agent's context.
- `tgrep`. Exact literal/symbol search through an external tgrep binary. Registered only when the binary resolves (`TGREP_BIN` → `tgrepBinPath` → `PATH`) and the workspace has a `.tgrep/` index. After your own edits, use `ffgrep`.

### Commands

- `/fff-mode [tools-and-ui | tools-only | override]`. Show or switch the mode.
- `/fff-health`. Picker, frecency, and git integration status.
- `/fff-rescan`. Force a rescan.
- `/tgrep-status`. tgrep index and server status for the workspace.

Source: [`packages/pi-tools/`](./packages/pi-tools/). Full documentation: [`packages/pi-tools/README.md`](./packages/pi-tools/README.md).

Expand Down
22 changes: 16 additions & 6 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,21 +8,22 @@

## Tool surface

The extension registers two custom tools and one completion surface (names resolved from
mode; see below):
The extension registers two FFF tools, an optional `tgrep` tool, and one completion surface
(FFF names resolved from mode; see below):

| Tool | Purpose |
|---|---|
| `fffind` | Typo-resistant file discovery + frecency-ranked access |
| `ffgrep` | SIMD content search |
| `tgrep` | Trigram-indexed exact content search via an external binary (registered when the binary and a workspace `.tgrep/` index are found) |

Source of truth: `packages/pi-tools/src/index.ts:52-53` (`grep: "ffgrep", find: "fffind"`),
Source of truth: `packages/pi-tools/src/index.ts:60` (`FFF_TOOL_NAMES`: `grep: "ffgrep", find: "fffind"`), `tgrep` via `queueTgrepTool` / `queueTool(() => TGREP_TOOL_NAME, …)` gated on `resolveTgrepBinary` plus `hasTgrepIndex` (`src/tgrep.ts`) at session start,
registered via `queueTool(() => toolNames.grep, …)` / `queueTool(() => toolNames.find, …)`.

### Parameters

Parameter shapes, defaults and allowed values are defined in the tool schemas in
`packages/pi-tools/src/index.ts` (`grepSchema`, `findSchema`, `multiGrepSchema`).
`packages/pi-tools/src/index.ts` (`grepSchema`, `findSchema`, `multiGrepSchema`, `tgrepSchema`).
`pi-tools.schema.json` documents **config keys only** (mode, DB paths, scan toggles), not
tool parameters. Compatibility surface = the parameter **names**, **types**, **enums**, and
**defaults** for both tools, plus their pagination behavior:
Expand All @@ -38,12 +39,18 @@ tool parameters. Compatibility surface = the parameter **names**, **types**, **e
The file picker UI exists only in `tools-and-ui`; the `@`-mention completion surface is
disabled only in `tools-only` (`shouldEnableMentions`: `currentMode !== 'tools-only'`).
A tool disappearing when mode changes is expected; a mode value being removed is breaking.
- **Exit codes (`tgrep`):** exit `0` returns `file:line:col:text` rows, exit `1` reports
`No matches found` (not a failure), exit `2` throws with the stderr cause. A leading
`[tgrep: ...]` line always carries the binary's stderr freshness warning. Output
truncates at `TGREP_OUTPUT_MAX_BYTES` with a narrowing hint.
`tgrep` is mode-independent: it keeps its name in every mode and is gated on
binary availability, a workspace `.tgrep/` directory, and `enableTgrep`.

### Config precedence

Config values are resolved in this priority order
(`packages/pi-tools/src/index.ts:272` `getConfigValue`, read at startup
`resolveStartupConfig`):
(`packages/pi-tools/src/index.ts` `getConfigValue`, read at startup
`resolveStartupConfig`; `tgrep` binary and index are resolved at session start):

```
flag (--fff-mode / --fff-frecency-db / --fff-history-db / …)
Expand All @@ -61,6 +68,9 @@ Concrete defaults:
| history DB | `--fff-history-db` | `FFF_HISTORY_DB` | `config.historyDbPath` | (platform db path) |
| root scan | `--fff-enable-root-scan` | `FFF_ENABLE_ROOT_SCAN` | `config.enableFsRootScanning` | `false` |
| home scan | `--fff-enable-home-scan` | `FFF_ENABLE_HOME_SCAN` | `config.enableHomeDirScanning` | `true` |
| tgrep binary | — (no CLI flag by design) | `TGREP_BIN` | `config.tgrepBinPath` | `PATH` lookup |
| tgrep toggle | — (config file only) | — | `config.enableTgrep` | `true` |
| tgrep time budget | — (no CLI flag by design) | `TGREP_TIME_BUDGET_MS` | `config.tgrepTimeBudgetMs` | `30000` |

Mode valid values (`packages/pi-tools/src/config.ts:8` `VALID_MODES`):
`tools-and-ui`, `tools-only`, `override`.
Expand Down
2 changes: 2 additions & 0 deletions packages/pi-tools/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

### Added

- Added the `tgrep` tool: trigram-indexed exact content search through an external [tgrep](https://github.com/microsoft/tgrep) binary. Literal by default, `file:line:col:text` output, exit 1 reported as no-match. Registered at session start only when the binary resolves (`TGREP_BIN`, `tgrepBinPath`, or `PATH`), the workspace has a `.tgrep/` index, and `enableTgrep` is not `false`; mode-independent. Prompt guidelines steer on capability (exact/literal vs fuzzy/frecency), not repository size. Optional `tgrepTimeBudgetMs` / `TGREP_TIME_BUDGET_MS` (default 30s). Output truncation stays on a UTF-8 character boundary; mid-run abort reports `Operation aborted`. `--no-index` is not exposed; after edits use `ffgrep`.
- Added the `/tgrep-status` command showing tgrep index and server status for the workspace.
- Added `ffgrep.maxMatchesPerFile` to keep a single generated or vendored file from dominating a result page.
- Added `ffgrep.compact` for deterministic `path:line:match` output without context blocks.

Expand Down
26 changes: 26 additions & 0 deletions packages/pi-tools/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,26 @@ Use `fffind` for **paths**. Use `ffgrep` when you know text that should occur in

Use a concrete substring, identifier, or expression. A wildcard-only expression such as `.*` is rejected because it is not an efficient way to read an entire file. Keep the default grouped output when context matters; use `compact: true` when the next action only needs stable path-and-line references. Set `maxMatchesPerFile` when a generated or vendored file could otherwise dominate the page. Both options are additive and leave existing defaults unchanged.

### `tgrep`

`tgrep` searches file content through an external [tgrep](https://github.com/microsoft/tgrep) binary: trigram-indexed exact search with a client/server architecture. The tool is registered only when the binary is found, the session cwd has a `.tgrep/` index, and `enableTgrep` is not disabled; it keeps the name `tgrep` in every mode. Prefer `tgrep` for exact literals and symbols; prefer `ffgrep` for fuzzy, typo-tolerant, frecency-ranked search. After `tgrep index .` or `tgrep serve .`, reload the session to expose the tool.

| Parameter | Type | Description |
| --- | --- | --- |
| `pattern` | string | Literal text by default; set `literal: false` for a regular expression. Patterns like `serve` cannot parse as subcommands. |
| `path` | string, optional | Directory or file to search, relative to the workspace; default is the workspace root. Globs go in `glob`. |
| `glob` | string or string array, optional | Repeatable file glob filter such as `*.{ts,tsx}`. |
| `fileType` | string or string array, optional | Repeatable file type filter such as `rust`, `py`, or `js`. |
| `literal` | boolean, optional | Treats the pattern as literal text; default is `true`. |
| `caseSensitive` | boolean, optional | Forces case-sensitive matching; default is smart-case. |
| `wholeWord` | boolean, optional | Matches whole words only. |
| `filesOnly` | boolean, optional | Prints only filenames with matches. |
| `count` | boolean, optional | Prints the match count per file. |
| `context` | number, optional | Context lines before and after a match; range 0–20. |
| `maxCount` | number, optional | Limits matches per file. |

Output is `file:line:col:text` rows. Exit code 1 (no match) is reported as `No matches found`, not as a failure. A leading `[tgrep: ...]` line carries the binary's stderr freshness warning. Only index-safe flags are forwarded; full-scan forcers (`--hidden`, `--no-ignore`, `-u`, `-a`, `--encoding`, `--no-index`) are excluded by design. After your own edits, use `ffgrep` because the index lags watcher events.

### Optional multi-pattern search

Set `PI_FFF_MULTIGREP=1` before starting Pi to enable the experimental `fff-multi-grep` tool. It searches for **any** of several literal patterns in one request and is useful when an agent must check known naming variants together.
Expand All @@ -105,6 +125,7 @@ The tool is opt-in while its interaction pattern is evaluated. Do not depend on
| `/fff-mode [tools-and-ui \| tools-only \| override]` | Shows the current mode or records a mode for the current session. |
| `/fff-health` | Displays the engine version, mode, Git integration, index status, persistence status, and active scan progress. |
| `/fff-rescan` | Requests a new file scan for the active workspace. |
| `/tgrep-status` | Shows tgrep index and server status for the workspace. |

## Persistent configuration

Expand All @@ -127,9 +148,14 @@ Create `pi-tools.json` in Pi’s agent directory. The default location is `~/.pi
| `historyDbPath` | string | Auto-resolved | Location for query-selection history. |
| `enableFsRootScanning` | boolean | `false` | Explicitly allows scans started from `/`. |
| `enableHomeDirScanning` | boolean | `true` | Allows scanning when Pi starts in the home directory. |
| `enableTgrep` | boolean | `true` | Registers the `tgrep` tool when the binary and a `.tgrep/` index are found. |
| `tgrepBinPath` | string | PATH lookup | Explicit path to the `tgrep` binary. |
| `tgrepTimeBudgetMs` | number | `30000` | Child-process time budget in milliseconds. `TGREP_TIME_BUDGET_MS` overrides this. |

Malformed configuration, unknown fields, and invalid values prevent the extension from loading and identify the configuration path in the error. `/fff-mode` changes session state only; it does not edit this file.

The `tgrep` binary resolves as `TGREP_BIN` environment variable, then `tgrepBinPath`, then a `tgrep` executable on `PATH`. An explicit path that is set but not executable disables the tool instead of falling back, so a typo surfaces instead of silently changing the search backend. Binary and `.tgrep/` index are resolved at session start, when the workspace cwd is known. Installing tgrep or building an index mid-session requires `/reload` before the tool appears.

## Database resolution

Frecency and history paths resolve independently in the following order:
Expand Down
16 changes: 16 additions & 0 deletions packages/pi-tools/pi-tools.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,22 @@
"type": "boolean",
"default": true,
"description": "Allows indexing when pi is launched from the home directory."
},
"enableTgrep": {
"type": "boolean",
"default": true,
"description": "Registers the tgrep trigram-index search tool when the tgrep binary is found and the workspace has a .tgrep index."
},
"tgrepBinPath": {
"type": "string",
"minLength": 1,
"description": "Explicit path to the tgrep binary; the TGREP_BIN environment variable takes precedence over config.tgrepBinPath, then PATH is used as a fallback."
},
"tgrepTimeBudgetMs": {
"type": "integer",
"minimum": 1,
"default": 30000,
"description": "Maximum milliseconds for a tgrep child process. TGREP_TIME_BUDGET_MS overrides this value."
}
}
}
31 changes: 29 additions & 2 deletions packages/pi-tools/src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ export interface FffConfig {
historyDbPath?: string;
enableFsRootScanning?: boolean;
enableHomeDirScanning?: boolean;
enableTgrep?: boolean;
tgrepBinPath?: string;
tgrepTimeBudgetMs?: number;
}

const CONFIG_KEYS = new Set<keyof FffConfig>([
Expand All @@ -25,8 +28,15 @@ const CONFIG_KEYS = new Set<keyof FffConfig>([
"historyDbPath",
"enableFsRootScanning",
"enableHomeDirScanning",
"enableTgrep",
"tgrepBinPath",
"tgrepTimeBudgetMs",
]);

/**
* Loads and validates pi-tools.json, falling back to the legacy filename.
* Returns an empty config when neither file exists and throws for unreadable or invalid files.
*/
export function loadConfig(agentDir = piDataDir()): FffConfig {
let configPath = join(agentDir, CONFIG_FILE_NAME);
const legacyConfigPath = join(agentDir, LEGACY_CONFIG_FILE_NAME);
Expand Down Expand Up @@ -75,8 +85,11 @@ export function loadConfig(agentDir = piDataDir()): FffConfig {
validateString(configPath, parsed, "$schema");
validateString(configPath, parsed, "frecencyDbPath");
validateString(configPath, parsed, "historyDbPath");
validateString(configPath, parsed, "tgrepBinPath");
validateBoolean(configPath, parsed, "enableFsRootScanning");
validateBoolean(configPath, parsed, "enableHomeDirScanning");
validateBoolean(configPath, parsed, "enableTgrep");
validatePositiveInteger(configPath, parsed, "tgrepTimeBudgetMs");

return parsed as FffConfig;
}
Expand All @@ -93,24 +106,38 @@ function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}

/** Validates that an optional configuration value is a non-empty string. */
function validateString(
configPath: string,
config: Record<string, unknown>,
key: "$schema" | "frecencyDbPath" | "historyDbPath",
key: "$schema" | "frecencyDbPath" | "historyDbPath" | "tgrepBinPath",
): void {
const value = config[key];
if (value !== undefined && (typeof value !== "string" || value.length === 0)) {
throw invalidConfig(configPath, `"${key}" must be a non-empty string`);
}
}

/** Validates that an optional configuration value is boolean. */
function validateBoolean(
configPath: string,
config: Record<string, unknown>,
key: "enableFsRootScanning" | "enableHomeDirScanning",
key: "enableFsRootScanning" | "enableHomeDirScanning" | "enableTgrep",
): void {
const value = config[key];
if (value !== undefined && typeof value !== "boolean") {
throw invalidConfig(configPath, `"${key}" must be a boolean`);
}
}

function validatePositiveInteger(
configPath: string,
config: Record<string, unknown>,
key: "tgrepTimeBudgetMs",
): void {
const value = config[key];
if (value === undefined) return;
if (typeof value !== "number" || !Number.isInteger(value) || value < 1) {
throw invalidConfig(configPath, `"${key}" must be a positive integer`);
}
}
Loading
Loading