feat(semantic): [merge candidate build] FTS5 index and search tools, provider-aware typed embeddings, reranking, diagnostics, and eval harness

[Zireael]

Summary

Semantic search in AFT moves from a minimal embedding-and-cosine prototype to a provider-capability-aware retrieval subsystem with typed vectors, optional reranking, background lifecycle management, diagnostics, and evaluation tooling. This is a public preview — the feature is functional and tested (~93 new tests) but expects iteration based on real-world feedback.

FTS5 index will allow introducing more advanced symbol operations.

What changed

The upgrade touches the full semantic pipeline — config, indexing, retrieval, diagnostics, and observability — without breaking the default fastembed experience.

Typed vector representations

Vectors are no longer opaque f32 blobs. Every stored vector carries explicit type metadata (DenseF32, Int8SourceDecoded, BinaryPacked) and is paired with its source kind so the correct distance metric is selected automatically. Binary packed vectors use Hamming search (native bitwise XOR + popcount) instead of cosine, which is both faster and semantically correct for quantized embeddings. This unlocks Perplexity's base64_binary and base64_int8 output modes alongside standard dense providers.

Provider capability profiles

Each embedding backend (fastembed, OpenAI-compatible, Ollama, Perplexity) declares what it supports: output encoding, distance metric, dimension range, max batch size. The config layer validates combinations at configure time — you cannot accidentally request binary vectors through a cosine-only provider. Profiles also carry fingerprint fields so switching providers triggers a clean index rebuild rather than silent corruption.

Fingerprint-driven index lifecycle

A SemanticIndexFingerprint captures every dimension that affects index correctness: backend, model, base_url, dimension, chunking_version, output_encoding, storage_strategy, vector kinds, normalization, and prompt hashes. diff() classifies changes as Rebuild (structural — re-embed everything), ClearQueryCache (query prompts changed — invalidate cached results only), or None. This replaces the previous "delete and hope" invalidation with precise, explainable rebuild decisions.

Non-blocking cold start

Index builds run in a background thread with cooperative cancellation (SemanticCancellationToken via AtomicU64 generation counter). The build checks the generation before each embedding batch and exits early when a reconfigure arrives. Priority ordering ensures high-value files (recently edited, high PageRank) get embedded first. Exponential backoff handles transient provider failures without blocking the session.

Stale-vector pruning

When files are edited, deleted, moved, excluded, or re-included, the index tracks which vectors are stale and prunes them during the next refresh cycle. Every vector record carries file/chunk ownership metadata (file path, version, chunk hash, index fingerprint) so pruning is traceable and deterministic.

File policy and docs chunking

A configurable file policy controls which files enter the index (include globs, exclude globs, max file size, max chunk count). The docs chunker splits Markdown and documentation files into semantic sections before embedding, improving recall for documentation-shaped queries.

Reranking pipeline

Optional reranking via any OpenAI-compatible /v1/rerank or chat-completion endpoint. The pipeline sends initial retrieval candidates to a reranker, parses the response (supporting multiple JSON shapes), and reorders results with safe fallback — if the reranker fails, the original cosine-similarity order is returned unchanged. Config fields: rerank.enabled, rerank.model, rerank.base_url, rerank.api_key_env, rerank.max_candidates.

Search pipeline metrics and diagnostics

Every aft_search call records timing, cache hits/misses, result counts, and reranker fallback events. Metrics are exposed through the status command and through JSONL diagnostic logs for offline analysis. The DiagnosticsOutputMode config controls verbosity in tool output (compact | verbose | off).

Semantic doctor

semantic_doctor is a health-check command that reports config summary, index summary, metrics summary, provider summary, and actionable suggestions. Use it to verify that the index is healthy, the provider is reachable, and the configuration is consistent.

Semantic eval harness

semantic_eval runs a JSONL-defined evaluation suite against the semantic index. Each case specifies a query, expected paths, expected symbols, and top-k. The harness computes recall@k and MRR (Mean Reciprocal Rank) for quantifying retrieval quality across config changes.

Status integration

The status command now includes semantic health metrics: lifecycle state, entry count, dimension, total queries, cache hit ratio, average query time, and provider info. The OpenCode TUI sidebar surfaces these alongside the existing index state.

Config trust boundary

backend, base_url, and api_key_env are user-only fields — project-level aft.jsonc cannot inject these. A hostile repository cannot redirect embeddings at an attacker-controlled endpoint or exfiltrate API keys. The plugin logs a warning when it strips a project-level setting.

Contextualized document-chunk embedding (partial)

Initial support for Perplexity-style document/chunk grouped embedding — chunks from the same source document are batched together rather than flattened. Oversized document handling and retry logic are still in progress (see roadmap).

How to test

Default fastembed (zero-config)

# Enable semantic search in your AFT config
# ~/.config/opencode/aft.jsonc or ~/.pi/agent/aft.jsonc:
{ "semantic_search": true }

# Start a session — index builds in background
# Run aft_search with a concept query:
aft_search({ "query": "authentication middleware" })

Verify: results appear with source: semantic or source: hybrid tags. Status shows [index: ready] after build completes.

Provider switching

// Switch to OpenAI-compatible
{
  "semantic_search": true,
  "semantic": {
    "backend": "openai_compatible",
    "model": "text-embedding-3-small",
    "base_url": "https://api.openai.com/v1",
    "api_key_env": "OPENAI_API_KEY"
  }
}

Verify: index rebuilds automatically on next session start. Status shows new provider/model.

Reranking

{
  "semantic_search": true,
  "semantic": {
    "backend": "openai_compatible",
    "model": "text-embedding-3-small",
    "base_url": "https://api.openai.com/v1",
    "api_key_env": "OPENAI_API_KEY"
  },
  "rerank": {
    "enabled": true,
    "model": "rerank-english-v3.0",
    "base_url": "https://api.cohere.com",
    "api_key_env": "COHERE_API_KEY"
  }
}

Verify: search results show reranker-sorted order. Disable reranker — results fall back to cosine order.

Semantic doctor

aft_search({ "query": "test" })  # trigger index build if cold
# Then check health via status command or semantic_doctor

Verify: health report shows ConfigSummary, IndexSummary, MetricsSummary, ProviderSummary.

Eval harness

// Create eval-cases.jsonl:
{"query": "authentication handler", "expected_paths": ["src/auth/middleware.ts"], "expected_symbols": ["authMiddleware"], "top_k": 10}
{"query": "database connection", "expected_paths": ["src/db/pool.ts"], "expected_symbols": ["createPool"], "top_k": 10}

Verify: returns recall@k and MRR scores.

Test coverage

~93 tests across 8 test sub-tasks covering:

• Config parsing and backward compatibility
• Fingerprint diff matrix (all field combinations → Rebuild/ClearQueryCache/None)
• File policy, docs chunking, and manifest handling
• VectorStore trait with DenseF32 and BinaryPacked implementations
• Binary packed-vector storage and Hamming search
• Lifecycle states, snapshots, and stale-vector pruning
• Search pipeline metrics, diagnostics, and DiagnosticsOutputMode
• Concurrency, race conditions, and cancellation token behavior
• Security trust boundary enforcement (project config stripping)
• Semantic doctor health report
• Semantic eval harness (JSONL parsing, scoring, recall/MRR)
• Reranking pipeline (parse multiple JSON shapes, fallback on failure)

Roadmap

Still in progress or planned for follow-up:

• aft-t6p.23: Complete contextualized document-chunk embedding (oversized docs, retry logic) — partially implemented
• aft-t6p.2.2: Configurable snippet truncation in reranking (currently hardcoded at 200 chars)
• aft-t6p.18: End-to-end verification across all backends
• aft-t6p.5: Configuration and operations documentation
• Performance benchmarking suite
• Migration tooling for index format upgrades

Architecture notes

Key new modules:

• crates/aft/src/semantic_rerank.rs — reranking pipeline with safe fallback
• crates/aft/src/semantic_diagnostics.rs — JSONL diagnostic logging
• crates/aft/src/semantic_doctor.rs — health-check report generation
• crates/aft/src/semantic_eval.rs — evaluation harness (JSONL parser, scoring)
• crates/aft/src/vector_store.rs — VectorStore trait with DenseF32 and BinaryPacked implementations
• crates/aft/src/commands/semantic_doctor.rs — doctor command handler
• crates/aft/src/commands/semantic_eval.rs — eval command handler

Modified significantly:

• crates/aft/src/semantic_index.rs — lifecycle management, fingerprint-driven invalidation, non-blocking build, stale pruning, typed vectors
• crates/aft/src/config.rs — provider profiles, rerank config, trust boundary fields
• crates/aft/src/commands/status.rs — semantic health metrics
• crates/aft/src/commands/semantic_search.rs — reranking integration, diagnostics output mode

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Provider‑aware semantic search with typed vectors, cross‑encoder/chat reranking, rich diagnostics, doctor/eval tools, and a Semble benchmark suite. This alpha also ships model2vec auto‑download with health/version checks, overflow‑safe embedding chunking, contextualized embeddings, an optional semantic-fts5 baseline, prompt profiles, result capping, and updated builds/workflows.

• New Features

  • Reranking: chat or /v1/rerank via rerank_api_type, custom rerank_prompt_template, fence‑tolerant parsing, 2 MiB body cap, and overfetch‑then‑truncate (rerank_max_candidates, rerank_max_candidate_chars[_cross_encoder]).
  • Embeddings/indexing: typed vectors (f32/int8/binary→Hamming), overflow‑safe chunking (max_embed_tokens, chunk_overlap_tokens), contextualized document‑chunk embeddings, prompt profiles, effective document_prompt_template at index time, and per‑file result caps via max_results_per_file.
  • Providers: capability profiles plus model2vec backend with HF download/cache/validation, doctor integration, version checks, and local support behind semantic-model2vec.
  • Lifecycle: file policy + docs chunker; fingerprint‑driven rebuilds; background cold start with Partial status and cancellation; stale‑vector pruning; VectorStore abstraction; V8 snapshot with per‑entry chunk_hash and a file manifest.
  • Diagnostics/bench/builds: JSONL logging with rotation/retention, DiagnosticsOutputMode, warning dedup, cache‑hit metrics, semantic_doctor, semantic_eval; optional FTS5 lexical baseline behind semantic-fts5; default builds enable semantic-model2vec/semantic-fts5; manual multi‑platform build-aft workflow; Docker‑based Rust validation; Semble pilot/CI scripts and reports.

• Bug Fixes

  • Security: project configs cannot set rerank_prompt_template; SSRF validation for rerank_base_url; normalized, traversal‑safe JSONL log paths; bounded streaming for reranker bodies.
  • Reranker/search correctness: score‑sorted handling across results/data/scores shapes, duplicate‑index filtering, out‑of‑bounds index warnings, accurate more_available after post‑rerank truncation, prevent candidate starvation by overfetching before rerank, clamp cosine NaNs, enforce per‑file caps and fusion ordering.
  • UX/diagnostics: surface reranker failures in minimal mode, warning dedup, fix diagnostics gating; eval harness wired to the real search pipeline.

^{Written for commit baacf80. Summary will update on new commits.}

Greptile Summary

This alpha PR replaces the prototype semantic search subsystem with a production-oriented retrieval pipeline: typed vectors (DenseF32, Int8SourceDecoded, BinaryPacked), provider capability profiles, fingerprint-driven index invalidation, background build with cooperative cancellation, optional reranking, diagnostics/doctor/eval tooling, and a security trust boundary that prevents project configs from redirecting embeddings to attacker-controlled endpoints.

• Core pipeline overhaul (semantic_index.rs, vector_store.rs): typed vectors with correct distance metrics (Hamming for binary, cosine for f32), stale-vector pruning, priority-ordered cold-start build, and fingerprint-based rebuild decisions.
• Reranking pipeline (semantic_rerank.rs): supports chat-completions and cross-encoder /v1/rerank endpoints with multiple response-format parsers, body-size cap, and safe fallback to original cosine order on any failure.
• Diagnostics, doctor, and eval (semantic_diagnostics.rs, semantic_doctor.rs, semantic_eval.rs): JSONL diagnostic logging, health-check command, and JSONL-based eval harness with recall@k and MRR scoring.

Confidence Score: 4/5

Safe to merge with awareness that several known issues from the previous review cycle remain open (reranker failures silent in minimal mode, more_available undercount, data-format rerank ordering, non-ASCII eval file corruption), plus the new gap where project configs can supply an adversarial rerank prompt template.

The new rerank_prompt_template stripping gap is the only genuinely new finding in this update; all other open issues were already identified in earlier review rounds. The change is an alpha build with documented limitations and strong test coverage (~93 tests). The trust boundary is mostly correct, the TypeScript enum mismatches are now fixed, and the vector-store and Hamming-distance math are sound.

packages/opencode-plugin/src/config.ts — add rerank_prompt_template to the stripProjectSemanticFields function. crates/aft/src/semantic_diagnostics.rs — surface RerankerFailure in minimal output mode.

Security Review

• Prompt injection via rerank_prompt_template in project config (packages/opencode-plugin/src/config.ts:1481): query_prompt_template and document_prompt_template are correctly stripped from project-level configs; rerank_prompt_template is not. A hostile repository can supply an adversarial reranker prompt that manipulates search result ordering for the user. Data exfiltration is not possible (the reranker endpoint URL is user-only), but result integrity can be silently degraded.
• No new credential-leakage, injection, or authentication bypass issues introduced by this PR. The trust-boundary stripping of backend, base_url, api_key_env, rerank_base_url, and rerank_api_key_env from project configs is correctly implemented.

Important Files Changed

Filename	Overview
crates/aft/src/semantic_rerank.rs	New reranking pipeline with body-size cap, fence stripping, and multiple response-format parsers. The build_rerank_endpoint trailing-slash fix is present. The data/results formats in extract_indices_from_rerank_results return insertion order rather than score-sorted order (flagged in previous review cycle).
crates/aft/src/semantic_eval.rs	New eval harness with JSONL parsing, recall@k, and MRR scoring. strip_trailing_commas is implemented but corrupts non-ASCII bytes (previously flagged). The score_case docstring says hits beyond k don't affect first_hit_rank, but the code does set it, so MRR includes full-list ranks, not @k ranks.
crates/aft/src/semantic_diagnostics.rs	New diagnostic subsystem. format_warning_minimal returns None for RerankerFailure, silently suppressing reranker failures in the default Minimal output mode (flagged in previous review cycle).
crates/aft/src/commands/semantic_search.rs	Reranking, diagnostics, and fusion-limit integration. more_available is computed against fusion_limit rather than top_k, so candidates between top_k and fusion_limit are silently dropped after reranking without setting more_available = true (flagged in previous cycle). OOB reranker-index warning still gated on diagnostics_enabled.
packages/opencode-plugin/src/config.ts	TypeScript Zod schema updated with 16+ new fields including corrected enum values (base64_binary, binary_packed, dot_product, perplexity). Trust-boundary stripping function added. rerank_prompt_template is exposed in the schema but not stripped from project configs, unlike query_prompt_template and document_prompt_template.
crates/aft/src/vector_store.rs	New VectorStore trait with FlatF32 (cosine) and FlatBinaryHamming (Hamming distance) implementations. Score normalization, orphan pruning, and stale-vector pruning look correct. Hamming similarity computation is sound.
crates/aft/src/config.rs	Provider capability profiles, RerankApiType enum, trust-boundary doc comments, and expanded SemanticBackendConfig. RerankApiType uses snake_case serde, matching the TypeScript ["chat", "rerank"] values.
crates/aft/src/compress/trust.rs	Added security-focused tests: atomic writes, multi-project trust, idempotent untrust, and reload survival. No logic changes.

Comments Outside Diff (2)

1. packages/opencode-plugin/src/config.ts, line 37-54 (link)

   TypeScript enum values don't match the Rust serde strings — config will fail to deserialize

   Several new enum schemas use values that don't align with the Rust serde representation:

   • SemanticOutputEncodingEnum allows "binary", "ubinary", "int8", "uint8" but Rust OutputEncoding deserializes from "base64_binary" and "base64_int8".
   • SemanticStorageStrategyEnum allows "flat" and "binary_pack" but Rust StorageStrategy expects "native_f32" and "binary_packed".
   • SemanticInputModeEnum includes "chunk_extracts" and "contextualized" but Rust InputMode only has "flat_texts" and "document_chunks".
   • SemanticDistanceMetricEnum uses "dot" but Rust DistanceMetric expects "dot_product".
   • SemanticBackendEnum is missing the new "perplexity" variant added to Rust.

   A user who follows the TypeScript autocomplete and picks output_encoding: "int8" will pass TypeScript validation but receive a deserialization error (or silent fallback to default) from the Rust binary at runtime.

2. crates/aft/src/commands/semantic_search.rs, line 116-119 (link)

   more_available understates available results when reranking is active

   fused_more_available is now computed as results.len() > fusion_limit (e.g., > 20) rather than > top_k (e.g., > 10). After reranking, results.truncate(top_k) discards any candidates between positions top_k and fusion_limit, but more_available has already been set and stays false. Concretely: if the fused pool yields 15 candidates (top_k=10, rerank_max_candidates=20), fused_more_available = 15 > 20 = false, more_available = false, and the 5 reranked-but-discarded candidates are silently dropped with no "more results" hint surfaced to the agent.

   Capture the pool size before truncation and fold it into more_available after the rerank block, before results.truncate(top_k).

_{Reviews (29): Last reviewed commit: "feat(semantic): add model prompt profile..." | Re-trigger Greptile}