Skip to content

docs: add comprehensive code documentation and streamline README #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yanurag-dev merged 1 commit into from
May 24, 2026
Merged

docs: add comprehensive code documentation and streamline README #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 30 additions & 90 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,127 +44,67 @@ Tested on the Go standard library source (~10,000 files):

## Install

### One-liner (Recommended)

```bash
curl -fsSL https://yanurag-dev.github.io/GrepTurbo/install.sh | bash
```

Installs the latest release to `/usr/local/bin/grepturbo`. Override the install directory:

```bash
INSTALL_DIR=~/.local/bin curl -fsSL https://yanurag-dev.github.io/GrepTurbo/install.sh | bash
```

### Pre-compiled Binaries

Download the latest binary for your platform from the [Releases](https://github.com/yanurag-dev/GrepTurbo/releases) page.

### From Source

```bash
git clone https://github.com/yanurag-dev/GrepTurbo
cd GrepTurbo
go build -o grepturbo ./cmd/grepturbo
```

---

## Usage

**Step 1 — build the index** (once, or when files change):
## Flags

```bash
GrepTurbo build -root ./myproject -out .GrepTurbo
```

**Step 2 — search:**

```bash
GrepTurbo search -index .GrepTurbo 'func.*Error'
```

Output is `file:line:text`, same as `grep -n`:

```
internal/index/reader.go:25:func NewReader(dir string) (*Reader, error) {
internal/query/search.go:26:func Search(r *index.Reader, pattern string) ([]Match, error) {
```
grepturbo init
(no flags — sets up in current directory)

### Flags

```
GrepTurbo build
grepturbo build
-root <dir> Directory to index (default: .)
-out <dir> Where to write the index (default: .GrepTurbo)
-out <dir> Where to write the index (default: .grepturbo)

GrepTurbo search
-index <dir> Index directory to query (default: .GrepTurbo)
grepturbo <pattern>
-index <dir> Index directory to query (default: .grepturbo)
```

---

## How It Works

```
regex → trigram decomposition → index lookup → intersect posting lists → candidate files → verify with regex
```

1. **Trigram decomposition** — `func.*Error` contains literals `func` and `Error`, producing trigrams `fun unc` and `Err rro ror`
2. **Index lookup** — each trigram maps to a sorted posting list of file IDs that contain it
3. **Intersection** — only files containing *all* required trigrams become candidates (10,000 → ~50)
4. **Verification** — the real regex engine runs only on those ~50 files
## Usage

**The golden invariant:** if a file matches the regex, it will always appear in the candidate set. No false negatives, ever.
### Quick Start: Init → Build → Search

### Index on Disk
**Step 1 — initialize** (one-time setup for your IDE/editor):

```bash
grepturbo init
```
.GrepTurbo/
lookup.idx mmap'd hash table — trigram → byte offset in postings.dat
postings.dat posting lists — [count][fileID, fileID, ...]
files.idx fileID → filepath mapping
```

Only `lookup.idx` is loaded into memory (mmap'd). Posting lists are read from disk on demand.

---

## Testing
This sets up GrepTurbo integration so your editor can run searches. Creates `.grepturbo/` in the project root.

Run the dynamic test script to benchmark and verify correctness against `grep`:
**Step 2 — build the index** (once per project, or when files change):

```bash
# Test default patterns on this repo
./scripts/test.sh

# Test a single pattern
./scripts/test.sh 'func.*Error'

# Test on any large codebase
./scripts/test.sh 'func.*Error' /path/to/large/repo
grepturbo build -root ./myproject -out .grepturbo
```

The script builds the binary, indexes the target directory, runs each pattern through both `grep` and `GrepTurbo`, compares results, and reports speedup + any false negatives.
Walks your codebase, extracts trigrams from each file, builds the inverted index, and writes 3 files to disk:
- `lookup.idx` — hash table (mmap'd for fast queries)
- `postings.dat` — trigram → file IDs
- `files.idx` — file ID → path mapping

```bash
# Run unit + integration tests
go test ./...
**Step 3 — search:**

# Run a specific test
go test ./internal/query/... -run TestCorrectnessVsGrep -v
```bash
# Full syntax
grepturbo search -index .grepturbo 'func.*Error'

# Run benchmarks
go test ./... -bench=.
# Shorthand (equivalent)
grepturbo 'func.*Error'
```

---

## Architecture

See [ARCHITECTURE.md](ARCHITECTURE.md) for full diagrams covering the index build pipeline, query flow, on-disk format, regex decomposition rules, and incremental sync strategy.
Output is `file:line:text`, same as `grep -n`:

---
```
internal/index/reader.go:25:func NewReader(dir string) (*Reader, error) {
internal/query/search.go:26:func Search(r *index.Reader, pattern string) ([]Match, error) {
```

<div align="center">

Expand Down
63 changes: 58 additions & 5 deletions cmd/grepturbo/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,30 @@
// Package main is the CLI entrypoint for grepturbo.
//
// GrepTurbo accelerates regex searches by building a trigram-indexed inverted
// index over a codebase, then using that index to filter down to candidate
// files before running the actual regex engine. This reduces query time from
// seconds (scanning every file) to milliseconds (scanning only candidates).
//
// Architecture overview:
//
// regex pattern
// ↓
// decompose into required trigrams
// ↓
// lookup trigrams in index (mmap'd hash table)
// ↓
// intersect posting lists → candidate file IDs
// ↓
// run regex engine on candidates only
// ↓
// return matches (file:line:text format)
//
// The index is built once and persisted to disk in three files:
// - lookup.idx: mmap'd fixed-size hash table (trigram → byte offset)
// - postings.dat: posting lists (count + file IDs, read on demand)
// - files.idx: file ID → file path mapping
//
// See docs/ARCHITECTURE.md for full diagrams and design decisions.
package main

import (
Expand All @@ -14,8 +41,10 @@ import (
const defaultIndexDir = ".grepturbo"

func main() {
// If the first arg isn't a known subcommand or flag, treat it as a search pattern
// so `grepturbo <pattern>` works alongside `grepturbo search <pattern>`.
// shorthand parsing: if the first arg isn't a known subcommand or flag,
// treat it as a search pattern so `grepturbo <pattern>` works alongside
// `grepturbo search <pattern>`. This improves usability without breaking
// explicit subcommand calls.
knownSubcmds := map[string]bool{"build": true, "search": true, "init": true, "help": true, "completion": true}
if len(os.Args) > 1 {
first := os.Args[1]
Expand All @@ -41,6 +70,12 @@ func main() {
buildCmd := &cobra.Command{
Use: "build",
Short: "Walk a directory and build the search index",
Long: `Build walks the specified directory tree, extracts trigrams from each file,
and serializes an inverted index to disk. The index enables O(log n) candidate
file filtering during queries.

The index is built once and reused for multiple searches. Incremental updates
via git commit tracking keep the index fresh across edits.`,
Example: ` grepturbo build -root ./myproject -out .grepturbo
grepturbo build -root . --skip node_modules --skip dist`,
RunE: func(cmd *cobra.Command, args []string) error {
Expand All @@ -57,6 +92,12 @@ func main() {
searchCmd := &cobra.Command{
Use: "search <pattern>",
Short: "Query the index with a regex pattern",
Long: `Search decomposes a regex pattern into required trigrams, looks them up
in the index, intersects the posting lists to find candidate files,
and runs the regex engine only on those candidates.

Output format: file:line:text (compatible with grep -n).
Exit status 1 if no matches are found (grep convention).`,
Example: ` grepturbo search -index .grepturbo 'func.*Error'
grepturbo search 'TODO'`,
Args: cobra.ExactArgs(1),
Expand All @@ -70,10 +111,12 @@ func main() {
initCmd := &cobra.Command{
Use: "init",
Short: "Install agent instructions into ~/.claude/ so Claude Code uses grepturbo",
Long: `Writes ~/.claude/GrepTurbo.md and adds @GrepTurbo.md to ~/.claude/CLAUDE.md.
Long: `Init writes ~/.claude/GrepTurbo.md and adds @GrepTurbo.md to ~/.claude/CLAUDE.md.
This configures Claude Code to prefer grepturbo search over grep/ripgrep
for regex searches across the codebase.

After running this, Claude Code will prefer grepturbo search over grep/ripgrep
for regex searches across the codebase.`,
After running this, build the index once with 'grepturbo build' in your project.
Claude Code will then automatically use grepturbo for all pattern searches.`,
RunE: func(cmd *cobra.Command, args []string) error {
summary, err := agentinit.Setup()
if err != nil {
Expand All @@ -92,6 +135,11 @@ for regex searches across the codebase.`,
}
}

// runBuild orchestrates the index build pipeline:
// 1. Walk the directory tree
// 2. Extract trigrams from each file
// 3. Build in-memory posting lists
// 4. Serialize to disk (lookup.idx, postings.dat, files.idx)
func runBuild(root, outDir string, skip []string) error {
fmt.Fprintf(os.Stderr, "Building index for %s → %s\n", root, outDir)

Expand All @@ -111,6 +159,11 @@ func runBuild(root, outDir string, skip []string) error {
return nil
}

// runSearch executes a regex query using the indexed search pipeline:
// 1. Open the index reader (mmap lookup table)
// 2. Call query.Search to decompose pattern, intersect posting lists, run regex
// 3. Handle index drift (commit mismatch) by rebuilding if needed
// 4. Output matches in grep-compatible format
func runSearch(idxDir, pattern string) error {
r, err := index.NewReader(idxDir)
if err != nil {
Expand Down
Loading
Loading