workspaceindex.ShouldSkipDir denylist misses common build-cache dirs (target/, __pycache__/, .venv/, ...) — scan budget burned on build artifacts, not .gitignore

[FabioLeitao]

What happens

internal/workspaceindex.ShouldSkipDir is a fixed, hardcoded denylist of directory
names to skip while scanning (used by repo-map, repo-info, and anything else built
on workspaceindex.Scan). It does not consult the scanned repo's own .gitignore
(Scan has no git dependency at all), and the denylist itself is missing several
directory names that are build output or cache in virtually every ecosystem:

// internal/workspaceindex/workspaceindex.go
func ShouldSkipDir(name string) bool {
	switch strings.ToLower(strings.TrimSpace(name)) {
	case ".cache", ".git", ".next", ".worktrees", ".zero", "build", "coverage", "dist", "node_modules", "vendor":
		return true
	default:
		return false
	}
}

Missing (non-exhaustive, found by reproducing against a real multi-language repo):
target (Cargo/Rust), __pycache__, .venv/venv, .pytest_cache, .terraform,
.mypy_cache, .ruff_cache.

Impact

Scan has a fixed MaxFiles budget (DefaultMaxFiles = 2000). Against a real repo
with several small Rust crates, rust/*/target/debug/.fingerprint/ alone produced
308 files (.d/.rlib/.rmeta/.timestamp) that counted toward that budget before
real source did. repo-map --json against that repo reported "truncated": true at
2000 files, and a repo-map --query ranking run returned 15 results, all of them
build-cache/fingerprint files, zero real source — the query terms matched path
fragments inside target/debug/.fingerprint/* ahead of the actual source file the
query was looking for.

This repo's own .gitignore already excludes target/; the scanner just never looks
at it.

Repro

$ git clone <any repo with a sizeable target/ or __pycache__/ dir>
$ zero repo-map --json | head -5
# fileCount near/at MaxFiles, "truncated": true, driven by target/debug/.fingerprint/*
$ zero repo-map --query "<term that matches a real source file>" --max-files 15
# ranked results dominated by target/**/*.{d,rlib,rmeta,timestamp}, score identical (1000) for all, no real source in the top 15

Proposed fix (narrow, ready as a draft PR on my fork)

Extend the ShouldSkipDir switch with the missing common build-cache directory names
(same pattern already used for node_modules/vendor/dist/build/coverage). I have
this implemented + a regression test (both a ShouldSkipDir unit case and an end-to-end
Scan case reproducing the target/debug/.fingerprint/* scenario) ready on
FabioLeitao/zero, branch to be pushed once I can rebuild/verify locally (hit an
unrelated full-disk condition on my own machine mid-verification, not a repo issue).

Separate, bigger question for maintainers (not proposing this now)

The narrow fix above is whack-a-mole against an ever-growing list of ecosystem-specific
build-cache names. The more durable fix would be for Scan to actually honor the
target repo's .gitignore (and .git/info/exclude) when the root is a git repo,
falling back to the current denylist otherwise. That's a bigger, dependency-adding
change per AGENTS.md §1 ("discuss ... major dependency changes ... before
implementation"), so I'm flagging it here for discussion rather than attempting it
unasked. Happy to take either direction once a maintainer weighs in — the narrow fix
is independently correct and small either way.

I'll follow CONTRIBUTING.md's community-PR process (parent issue + issue-approved
label) before opening a PR against this repo; the fix is staged on my fork in the
meantime.

[FabioLeitao]

Concrete, independent confirmation that the narrow fix (PR #6, now merged-ready) is
necessarily incomplete, found while re-validating it tonight against a real third repo
(not one of the fixtures used to write the fix):

$ cd data-boar && zero repo-map
Repo map
Root: /home/leitao/Projects/dev/data-boar
Files: 2000  Directories: 184  Max depth: 6
Truncated: true
Important files: .locust_env/lib/python3.13/site-packages/flask/sansio/README.md

data-boar has a Python virtualenv named .locust_env/ (created by the locust load-testing
tool — not .venv/venv, the names PR #6 added). The scan burns its entire 2000-file budget
inside that virtualenv's vendored packages (Flask, etc.) and never reaches the repo's own
README.md/pyproject.toml — Important files reports exactly one hit, a README vendored three
levels deep inside site-packages.

The repo's own .gitignore already has this exact directory excluded:

$ grep -n locust_env data-boar/.gitignore
330:.locust_env/

This is the same class of bug PR #6 fixes (target/, __pycache__/, etc.), just with a
directory name no fixed list will ever anticipate — any tool, framework, or per-project
convention can name its venv/build-cache directory anything. PR #6 is still a correct,
narrow, independently-useful fix for the handful of near-universal names it covers (and is
tested/verified as such) — but it cannot be the complete fix for this class of problem.

Restating the deeper option from the issue body for visibility: Scan honoring the target
repo's real .gitignore (and .git/info/exclude) when the root is a git repo, falling back
to the current denylist otherwise, would catch this and every future case like it without
another PR per convention. That's a bigger, dependency-adding change per AGENTS.md §1
("discuss ... major dependency changes ... before implementation"), so flagging it here for a
maintainer call rather than attempting it unasked in PR #6's diff.

[FabioLeitao]

Working proof of concept for the general fix, so the call is made against code rather than prose: FabioLeitao#7 (fork, not opened here — waiting on the issue-approved process).

• Scan asks git for the ignored set (git status --porcelain=v1 -z --ignored=matching .), so the repo's own .gitignore wins; outside a git work tree / no git / 5s timeout it behaves exactly as today. No new Go dependency.
• Same real repo as above: Important files goes from one vendored Flask README to the repo's own AGENTS.md, README.md, config/README.md, ….
• Worth knowing before choosing an approach: git ls-files --others --ignored --directory looks like the obvious primitive but stops descending at the first untracked directory, so it misses nested ignored dirs/files — status --ignored=matching does not.
• It layers on the narrow denylist PR (our fork M0: foundation — ToolBase, layered config, write_file, line ranges, edit uniqueness #6) rather than replacing it; either can land alone. Happy to split, squash or drop whichever you prefer.

[Vasanthdev2004]

Approving, with a narrower place for the fix than ShouldSkipDir. The scan budget problem is real, but ShouldSkipDir isn't only the scanner's: glob, grep and list_directory skip whatever it names (through shouldSkipDirectory in internal/tools/workspace.go), and so does the TUI's path autocomplete. Adding target, .venv or __pycache__ there would also hide those directories from the tools when someone asks about a dependency in .venv or generated sources under target, and a real package named target anywhere in a tree would drop out of search.

So keep ShouldSkipDir as it is, and give workspaceindex.Scan its own extra list of build and cache directories. That's what repo-map, the workspace seed and the MCP resources go through. The regression test you describe fits as is: a Scan case with the target/debug/.fingerprint files, plus one asserting ShouldSkipDir("target") stays false so the tools keep seeing it. Honoring .gitignore in Scan would be the fuller fix, but it's a bigger change than this needs.

[Vasanthdev2004]

I posted that without reading your two comments above it, sorry. .locust_env/ settles it: no list of names catches every virtualenv, so asking git for the ignored set is the right fix, and that's the version I'd approve here. The scoping still stands: put it in Scan and leave ShouldSkipDir alone, so glob, grep, list_directory and autocomplete keep seeing ignored directories when someone asks about them.

Two things for the PR. Scan also runs for the workspace seed, not just repo-map, so the cost of the git call and the 5s fallback matter on large trees. And AGENTS.md asks for the minimum Git version behind status --ignored=matching to be stated, with the no-git path covering anything older. The narrow list can still go in alongside it for checkouts that aren't git work trees.

[FabioLeitao]

@Vasanthdev2004 thanks. I reworked it into the shape you approved and opened #1117: git's ignored set in Scan, a fixed fallback list only outside git, and ShouldSkipDir unchanged.

On your two points, the details and measurements are in the PR description:

• Workspace seed cost: 10ms on Zero, 20ms warm / 120ms cold on a 2,411-file Python repo with a large virtualenv, under the 5s bound. The lookup also runs with GIT_OPTIONAL_LOCKS=0, so it never takes .git/index.lock.
• Git version floor: 2.16 (status --ignored=<mode>), stated on the function. Older git takes the fallback, and the git-backed tests skip below 2.16.

Each new test fails when its part of the fix is reverted; the table is in the PR. No hurry while #1106 is in review.