Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .mcp.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@
"PERSONAL_SUPERPOWERS_DIR",
"XDG_CONFIG_HOME",
"EPISODIC_MEMORY_DB_PATH",
"EPISODIC_MEMORY_MODEL_CACHE_DIR",
"EPISODIC_MEMORY_OFFLINE",
"CONVERSATION_SEARCH_EXCLUDE_PROJECTS"
]
}
Expand Down
24 changes: 22 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -301,12 +301,26 @@ Codex summarization requires `codex-cli 0.130.0` or newer. If Codex app-server s
| Component | Uses custom config? |
|-----------|---------------------|
| Summarization | Yes (up to 10 calls/sync) |
| Embeddings | No (local Transformers.js) |
| Embeddings | Optional cache and offline settings (local Transformers.js) |
| Search | No (local SQLite) |
| MCP tools | No |

Summaries are display-only: they decorate search results and are never embedded or searched, so `EPISODIC_MEMORY_SKIP_SUMMARIES=1` costs you that line of context and nothing else.

### Embedding model cache and offline mode

Embedding computation is local, but a fresh installation may download its model
on first use. Model files are cached under the durable Episodic Memory
configuration directory in `models/`, outside npm-owned `node_modules`, so an
upgrade or dependency reinstall does not discard them. Set
`EPISODIC_MEMORY_MODEL_CACHE_DIR` to an absolute path to use a different or
pre-seeded cache.

Set `EPISODIC_MEMORY_OFFLINE=1` to forbid remote model downloads. In that mode,
seed the model cache before indexing or semantic search; a missing model fails
with an explicit cache/offline error rather than silently fetching it. Other
values retain the normal first-install download behavior.

## Commands

### `episodic-memory sync`
Expand Down Expand Up @@ -411,7 +425,7 @@ open output.html

1. **Sync** - Copies conversation files from Claude Code and Codex transcript directories to archive; exports opencode sessions from SQLite into generated JSONL transcripts
2. **Parse** - Extracts user-agent exchanges from Claude Code JSONL, Codex rollout JSONL, or opencode transcript JSONL
3. **Embed** - Generates vector embeddings using Transformers.js (local, offline)
3. **Embed** - Generates vectors locally with Transformers.js after the model is available
4. **Index** - Stores in SQLite with sqlite-vec for fast similarity search
5. **Search** - Semantic search using vector similarity or exact text matching

Expand Down Expand Up @@ -519,6 +533,12 @@ npm test
npm run build
```

`npm test` prepares the embedding model once in the ignored
`tmp/test-model-cache` directory, then runs test workers without remote model
downloads. A fresh test run may download the public model; setting
`EPISODIC_MEMORY_OFFLINE=1` requires a pre-seeded test cache instead. This does
not change the production model cache or summarization routes.

## License

MIT
19 changes: 16 additions & 3 deletions dist/embeddings.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { getModelCacheDir } from './paths.js';
/**
* Embedding model configuration.
*
Expand Down Expand Up @@ -65,6 +66,8 @@ export class EmbeddingsUnavailableError extends Error {
export async function initEmbeddings() {
if (embeddingPipeline)
return;
const modelCacheDir = getModelCacheDir();
const offline = process.env.EPISODIC_MEMORY_OFFLINE === '1';
// Load @huggingface/transformers lazily. Its module graph eagerly requires
// `sharp`, so a static top-level import would crash *every* consumer of this
// file at import time on hosts where sharp's native binding can't load —
Expand All @@ -73,9 +76,11 @@ export async function initEmbeddings() {
let pipeline;
try {
const transformers = await import('@huggingface/transformers');
// Disable progress callbacks / remote cache to prevent stdout pollution in
// MCP context, where stdout is reserved for JSON-RPC communication.
// Keep model files in durable operator-owned state. Offline mode is an
// explicit choice; a fresh default install can still fetch its model.
transformers.env.cacheDir = modelCacheDir;
transformers.env.allowLocalModels = true;
transformers.env.allowRemoteModels = !offline;
transformers.env.useBrowserCache = false;
pipeline = transformers.pipeline;
}
Expand All @@ -97,7 +102,15 @@ export async function initEmbeddings() {
interOpNumThreads: 1,
};
}
embeddingPipeline = await pipeline('feature-extraction', MODEL_ID, options);
try {
embeddingPipeline = await pipeline('feature-extraction', MODEL_ID, options);
}
catch (error) {
const message = offline
? 'Embedding model unavailable in the local cache while EPISODIC_MEMORY_OFFLINE=1; pre-seed the model cache or disable offline mode.'
: 'Embedding model unavailable; check the model cache and remote model access.';
throw new EmbeddingsUnavailableError(message, { cause: error });
}
console.error('Embedding model loaded');
}
export async function generateEmbedding(text) {
Expand Down
Loading