Cafi (Content Addressable File Indexer)

Cafi is a fast, content-addressable file indexing system. Clients scan local directories and stream file metadata to a central server over a bidirectional gRPC connection. Files are indexed by their BLAKE3 hash, enabling efficient deduplication tracking across multiple sources (a laptop, a NAS, …).

Status. The sync protocol and metadata index are working. A read API, content extraction, search, thumbnails and vector search are designed but not yet implemented — see docs/architecture.md for the phase map. This README documents the current CLI surface only.

Features

• BLAKE3 content hashing for deduplication across sources.
• Bidirectional gRPC streaming sync with per-event ACKs.
• Multi-source support — track files from several machines as independent sources.
• Stateful client — a local SQLite database tracks what has already been sent, so re-scans only transmit deltas.
• Gitignore-style blocklist for excluding paths.
• MIME type detection from magic bytes.
• Goose-managed schema migrations, runnable in-process or standalone.

Architecture

 cafi-client (scan + sync)  ──gRPC (bidi)──►  cafi-server  ──SQL──►  ParadeDB (Postgres)
                                                 :50051

• cmd/cafi-server — gRPC server managing the database, auth, and ingestion.
• cmd/cafi-client — CLI that scans local directories and syncs to the server.
• internal/ — database layer, auth interceptor, gRPC handlers, scanner.
• proto/ — Protocol Buffer definitions for the gRPC service.

The server speaks plaintext gRPC on :50051; TLS is expected to be terminated by a reverse proxy in front of it (see TLS).

Prerequisites

• Go 1.25.7+ (see go.mod)
• PostgreSQL 14+ with the pg_search extension, or the bundled ParadeDB image (see below)
• mise-en-place (optional, for task management)
• Docker & Compose (optional, for running the database and integration tests)
• A C toolchain is required for the client, which links go-sqlite3 (CGO_ENABLED=1).

Quick start

1. Start the database

The provided compose.yaml runs ParadeDB (Postgres + pg_search), pinned to an explicit version:

docker compose up -d

Any PostgreSQL 14+ instance works for the current sync features; ParadeDB is required only once Phase 1's full-text search lands.

2. Build

mise run build
# or manually:
CGO_ENABLED=1 go build -o bin/cafi-client ./cmd/cafi-client/
CGO_ENABLED=0 go build -o bin/cafi-server ./cmd/cafi-server/

3. Configure the server

The server reads its database URL and port from the environment and runs migrations automatically on startup:

export DATABASE_URL="postgres://cafi:cafi@localhost:5434/cafi"
./bin/cafi-server serve

Create a user, a source, and a token (in a separate shell):

./bin/cafi-server user add vincent
./bin/cafi-server source add vincent laptop --strategy=remote
./bin/cafi-server token add vincent laptop-token
# prints: Token: cafi_<id>_<43-char secret>
./bin/cafi-server token add-source vincent laptop-token laptop

4. Connect and scan a client

./bin/cafi-client connect localhost:50051 --token "cafi_..."
./bin/cafi-client source add laptop ~/Pictures
./bin/cafi-client scan

scan walks every configured source, hashes new/modified files, and streams the deltas to the server. Re-runs only send what changed.

Token format

Machine tokens use a parseable wire format:

cafi_<token_id_base36>_<43-char base64url secret>

The server looks up the token by its embedded id (one indexed row fetch) and performs a single bcrypt comparison, rather than iterating over every token. Issued tokens are cached for 60 s, so a token created while the server is running authenticates within a minute without a restart.

Breaking change (Phase 0). Tokens issued by earlier versions lack the id prefix and cannot be recovered from their bcrypt hashes. Re-issue every token with cafi-server token refresh <user> <name> (or token add).

Migrations

Schema migrations are managed with goose and embedded in the binary. They run automatically when the server starts, but can also be driven explicitly:

./bin/cafi-server migrate up        # apply all pending
./bin/cafi-server migrate down      # roll back the most recent
./bin/cafi-server migrate status    # show applied/pending
./bin/cafi-server migrate create add_foo   # scaffold a new SQL migration

Or via mise: mise run migrate-up, migrate-down, migrate-status, migrate-create -- <name>.

TLS

The server speaks plaintext; terminate TLS at a reverse proxy. The client opts into TLS for its own connection to the proxy in either of two ways:

./bin/cafi-client connect grpcs://cafi.example.com:443 --token "cafi_..."
./bin/cafi-client connect cafi.example.com:443 --tls --token "cafi_..."

The choice is persisted in the client config and reused by scan. System roots are used for verification.

Configuration

Server

Variable	Default	Meaning
DATABASE_URL	—	PostgreSQL connection string (required).
PORT	50051	gRPC listen port.

Client

The client stores its server address, token, and source registry in a local SQLite database (default ~/.local/share/cafi/cafi.db). Configure it with cafi-client connect, not environment variables.

Variable	Default	Meaning
CAFI_DB_URL	(XDG)	Override the path to the client's SQLite database.
CAFI_SKIP_VERIFY	0	Set to 1 to skip the connectivity/token check during connect.

A gitignore-style blocklist at ~/.config/cafi/ignore.txt (if present) is applied to every scan to exclude paths.

CLI reference

cafi-server

serve                              Run the gRPC server
user    add|remove|list <username> Manage users
source  add|remove|update|list     Manage sources (--strategy, --path)
token   add|remove|refresh|        Manage tokens and their source links
        add-source|remove-source|
        list
migrate up|down|status|create      Manage database migrations (goose)

cafi-client

connect <server> [--token T] [--tls]   Configure the server connection
source  add|remove|list|               Manage local sources and the stored token
        refresh-token|update-path
scan [--parallelism N] [--dry-run]     Scan all sources and sync to the server
search                                 (not yet implemented)

Run any command with --help for the full flag list.

Development

Common tasks are defined in mise.toml:

Task	Description
mise run proto	Regenerate protobuf code with buf.
mise run build	Build both binaries.
mise run vet	go vet ./...
mise run lint	golangci-lint run ./...
mise run test	Unit tests in short mode (no Docker required).
mise run test-integration	All tests, including Docker-backed integration tests.
mise run docker-build	Build the server Docker image.

mise run test skips the container-backed integration test so it stays green on a machine without Docker. CI (.github/workflows/ci.yml) runs the same short-mode suite on every push.

License

See LICENSE.