feat(admission): enforce record exclusions across indexing and recall

README.md

@@ -2,6 +2,21 @@
 
 Semantic search for Claude Code, Codex, Cursor, opencode, and Oh My Pi (OMP) conversations. Remember past discussions, decisions, and patterns.
 
+## Recall admission
+
+Search rechecks the current record-exclusion policy and configured credential scanner,
+including rows indexed before the policy changed. Excluded hits do not consume the
+requested result limit. A stale whole-conversation summary is omitted if any of its
+source records or tool calls is excluded; safe neighboring exchanges remain searchable.
+
+Conversation opt-out markers are read from decoded user text, including JSON escapes.
+Assistant quotations and tool outputs do not opt out the conversation. Tool-result
+blocks inside a user envelope are still tool output. Malformed legacy text retains
+conservative marker handling, and unreadable or oversized records fail closed.
+Ranged reads retain physical line coordinates without materializing unrelated message bodies.
+Original transcripts are never rewritten. Source and policy changes during a search
+abort the result rather than returning a partially validated view.
+
 ## Testimonial
 
 From an AI coding assistant's perspective:
@@ -214,7 +229,7 @@ In Codex and opencode, the skill guides the agent to use the episodic-memory MCP
 
 ## API Configuration
 
-By default, episodic-memory uses your Claude Code authentication for Claude Code summarization. Codex-indexed sessions with a session ID are summarized through `codex app-server` by creating an ephemeral `thread/fork`, so the summary can use Codex session context and reasoning summaries without appending to the original rollout.
+By default, episodic-memory uses your Claude Code authentication for Claude Code summarization. Codex-indexed sessions with a session ID are summarized through `codex app-server` by creating an ephemeral `thread/fork`, so the summary can use Codex session context and reasoning summaries without appending to the original rollout. A Codex transcript without a session ID is summarized from its admitted text in an isolated ephemeral `thread/start`; it is never sent to Claude.
 
 To route summarization through a custom Anthropic-compatible endpoint or override the model:
 
@@ -294,7 +309,7 @@ These pass through unchanged to episodic-memory's summarizer subprocess. Because
 
 These settings only affect episodic-memory's summarization calls, not your interactive Claude Code or Codex sessions.
 
-Codex summarization requires `codex-cli 0.130.0` or newer. If Codex app-server summarization is unavailable, sync logs the reason and falls back to transcript-text summarization.
+Codex summarization requires `codex-cli 0.130.0` or newer; isolated transcript-only summaries require `0.154.0` or newer. If Codex app-server summarization is unavailable, sync records a retryable summary failure rather than switching providers.
 
 ### What's Affected
 
@@ -417,14 +432,14 @@ open output.html
 
 ## Excluding Conversations
 
-Conversations containing this marker anywhere in their content will be archived but not indexed:
+Conversations with this marker in a decoded user instruction are excluded before new archive copying, parsing, and indexing:
 
 ```
 <INSTRUCTIONS-TO-EPISODIC-MEMORY>DO NOT INDEX THIS CHAT</INSTRUCTIONS-TO-EPISODIC-MEMORY>
 ```
 
 **Automatic exclusions:**
-- Conversations where Claude generates summaries (marker in system prompt)
+- Summarizer-generated conversations carrying the marker in their prompt
 - Meta-conversations about conversation processing
 
 **Use cases:**
@@ -433,7 +448,117 @@ Conversations containing this marker anywhere in their content will be archived
 - Test or experimental sessions
 - Any conversation you don't want searchable
 
-The marker can appear in any message (user or assistant) and excludes the entire conversation from the search index.
+The marker in user instructions prevents new derived copies and indexing; an assistant quotation or tool result does not opt out the conversation. It does not delete an existing archive or index entry.
+
+### Persistent record exclusions
+
+`record-exclusions.json` in the existing configuration directory defines exact
+records that must not enter new derived archives, parser output, summaries, or
+database insertions. It is independent of `CONVERSATION_SEARCH_EXCLUDE_PROJECTS`;
+that environment variable cannot replace or disable record exclusions.
+
+```json
+{
+  "version": 1,
+  "exchanges": [
+    {
+      "session_id": "00000000-1111-4222-8333-444444444444",
+      "transcript": "rollout-00000000-1111-4222-8333-444444444444.jsonl",
+      "line_start": 4,
+      "line_end": 5
+    }
+  ],
+  "tool_calls": [
+    {
+      "session_id": "00000000-1111-4222-8333-444444444444",
+      "transcript": "rollout-00000000-1111-4222-8333-444444444444.jsonl",
+      "call_id": "call-example"
+    }
+  ]
+}
+```
+
+Line ranges are inclusive physical JSONL coordinates in the original transcript.
+Excluded records become empty lines in a derived copy, preserving later line
+numbers. Session identity, exact transcript basename and native tool-call identity, not
+a machine-specific absolute path, bind the policy. Sibling subagent transcripts
+sharing a parent session are separate subjects. Safe text blocks remain when only a tool block
+is excluded. Original transcripts are never changed.
+
+Malformed, unreadable, symlinked or oversized policy files reject admission.
+When a policy is present, unchanged source timestamps do not bypass a fresh
+admission decision. Source/archive aliasing and changing input are rejected.
+The marker scan and record stream use the same descriptor; source identity is
+checked during and after streaming. Direct database insertion checks both the
+transcript identity and historical session metadata, which may name an ancestor.
+The foreground sync CLI exits nonzero for per-file admission errors and does not
+claim successful synchronization or start embedding migration after those errors.
+Full, session and incremental indexing, repair, MCP reads and the display CLI use
+the same admission boundary. Display pagination retains physical line numbers.
+An absent policy retains ordinary behavior; deleting the policy intentionally
+removes its protection, so maintain it as private operator configuration.
+
+Summarization with a record policy uses admitted transcript text, never a resumed
+source session. Codex retains its selected provider through an ephemeral fresh
+context and fails rather than silently falling back to another provider. Claude
+receives the admitted text without session resume. Policy-enabled Codex summaries
+require Codex 0.154.0 or newer. The isolated summary path disables inherited
+tools, plugins, hooks, and skills, and refuses to proceed if the configured
+MCP or skill boundary cannot be established. It uses a temporary empty working
+directory; provider error payloads are suppressed and the directory is removed
+after process termination.
+
+### Optional local secret screening
+
+Add an optional `screening` object to the same policy to reject newly detected
+content before derived archive writes, reads, summaries and database insertion:
+
+```json
+"screening": {
+  "engine": "gitleaks",
+  "executable": "/absolute/resolved/path/to/gitleaks",
+  "sha256": "<64 lowercase hex characters from the trusted executable>",
+  "timeout_ms": 2000,
+  "max_record_bytes": 1048576
+}
+```
+
+Keep the existing `version`, `exchanges` and `tool_calls` fields. Install Gitleaks
+through a trusted package source, resolve its executable symlink and verify its provenance before pinning its
+SHA-256. On migration or upgrade, explicitly replace the host-local path and
+digest after verifying the new executable. The scanner must be a regular,
+executable file owned by the current user or root and not group/world-writable.
+Do not copy credential stores or an old host's executable-path assumption.
+
+Exact exclusions run first. Each remaining physical record is screened locally
+with the pinned Gitleaks built-in rules. Nested JSON strings are decoded while
+retaining key/value assignment context; separate exchange messages are screened
+separately. A finding rejects the operation rather than silently dropping a new
+message and breaking conversation pairing. Missing/changed executables, byte or
+nesting limits, timeouts and malformed or contradictory reports fail closed.
+Verification reports scanner rejections and scanner unavailability separately
+from transcript corruption, even when a summary is missing. A broad repair
+refuses to start while either screening failure remains.
+No findings, input, scanner diagnostics or secret fingerprints appear in errors.
+The scanner receives stdin in a private empty working directory with no inherited
+environment/configuration and with inline allow comments disabled. Its temporary
+directory is removed after execution. Original transcripts and previously
+published archives remain unchanged when admission fails.
+
+Screening launches a bounded subprocess per record; it is not an unbounded bulk
+collection service. Measure a finite representative batch before large imports.
+No automatic sync or background job is enabled by this setting. Before enabling
+it, replace all policy consumers with this version through their native lifecycle:
+older strict-schema consumers reject the additional field. Installing a plugin
+does not replace already-loaded code, so do not enable the field while protected
+old consumers still require the previous schema.
+
+Without `screening`, this policy remains an exact exclusion mechanism. Even with
+screening, pattern matching does not prove universal unknown-secret absence,
+purge preexisting derived data, rewrite original
+sessions, or retroactively filter already-loaded code. Old randomly generated
+tool-row IDs need verified native source-call mapping before policy migration;
+do not broaden those entries to whole sessions or claim a complete migration.
 
 ## MCP Server
 

dist/db.js

@@ -1,4 +1,5 @@
 import Database from 'better-sqlite3';
+import { admitExchange } from './record-admission.js';
 import path from 'path';
 import fs from 'fs';
 import * as sqliteVec from 'sqlite-vec';
@@ -186,6 +187,10 @@ export function initDatabase() {
     return db;
 }
 export function insertExchange(db, exchange, embedding, toolNames) {
+    const admitted = admitExchange(exchange);
+    if (!admitted)
+        return;
+    exchange = admitted;
     const now = Date.now();
     const stmt = db.prepare(`
     INSERT OR REPLACE INTO exchanges

dist/index-cli.js

@@ -65,13 +65,16 @@ async function main() {
                 console.log(`Missing summaries: ${issues.missing.length}`);
                 console.log(`Orphaned entries: ${issues.orphaned.length}`);
                 console.log(`Outdated files: ${issues.outdated.length}`);
+                console.log(`Screening-rejected archives: ${issues.screeningRejected.length}`);
+                console.log(`Screening-unavailable archives: ${issues.screeningUnavailable.length}`);
                 console.log(`Corrupted files: ${issues.corrupted.length}`);
                 if (issues.missing.length > 0) {
                     console.log('\nMissing summaries:');
                     issues.missing.forEach(m => console.log(`  ${m.path}`));
                 }
-                if (issues.missing.length + issues.orphaned.length + issues.outdated.length + issues.corrupted.length > 0) {
-                    console.log('\nRun with --repair to fix these issues.');
+                if (issues.missing.length + issues.orphaned.length + issues.outdated.length +
+                    issues.screeningRejected.length + issues.screeningUnavailable.length + issues.corrupted.length > 0) {
+                    console.log('\nReview the reported categories before repair; --repair may regenerate embeddings and AI summaries.');
                     process.exit(1);
                 }
                 else {
@@ -81,6 +84,11 @@ async function main() {
             case 'repair':
                 console.log('Verifying conversation index...');
                 const repairIssues = await verifyIndex();
+                if (repairIssues.screeningRejected.length + repairIssues.screeningUnavailable.length > 0) {
+                    console.error('Record screening failures require review; repair was not started.');
+                    process.exitCode = 1;
+                    break;
+                }
                 if (repairIssues.missing.length + repairIssues.orphaned.length + repairIssues.outdated.length > 0) {
                     await repairIndex(repairIssues);
                 }

dist/indexer.js

@@ -1,4 +1,5 @@
 import fs from 'fs';
+import { archiveAdmittedConversation, readRecordExclusions } from './record-admission.js';
 import path from 'path';
 import { initDatabase, insertExchange } from './db.js';
 import { parseConversation } from './parser.js';
@@ -26,6 +27,7 @@ function sessionIdForSummary(exchanges) {
     return exchanges.find(exchange => exchange.sessionId)?.sessionId;
 }
 export async function indexConversations(limitToProject, maxConversations, concurrency = 1, noSummaries = false) {
+    readRecordExclusions();
     console.log('Initializing database...');
     const db = initDatabase();
     console.log('Loading embedding model...');
@@ -74,15 +76,16 @@ export async function indexConversations(limitToProject, maxConversations, concu
                 let exchanges;
                 try {
                     // Copy to archive (ensure parent dirs exist for subagent files)
-                    if (!fs.existsSync(archivePath)) {
-                        fs.mkdirSync(path.dirname(archivePath), { recursive: true });
-                        fs.copyFileSync(sourcePath, archivePath);
+                    if (await archiveAdmittedConversation(sourcePath, archivePath))
                         console.log(`  Archived: ${file}`);
-                    }
                     // Parse conversation
                     exchanges = await parseConversation(sourcePath, project, archivePath);
                 }
                 catch (error) {
+                    if (error.code !== 'ENOENT') {
+                        db.close();
+                        throw error;
+                    }
                     console.log(`  Skipped ${file} (read failed: ${error instanceof Error ? error.message : error})`);
                     continue;
                 }
@@ -159,6 +162,7 @@ export async function indexConversations(limitToProject, maxConversations, concu
     console.log(`\n✅ Indexing complete! Conversations: ${conversationsProcessed}, Exchanges: ${totalExchanges}`);
 }
 export async function indexSession(sessionId, concurrency = 1, noSummaries = false) {
+    readRecordExclusions();
     console.log(`Indexing session: ${sessionId}`);
     // Find the conversation file for this session
     const sourceDirs = getConversationSourceDirs();
@@ -187,15 +191,14 @@ export async function indexSession(sessionId, concurrency = 1, noSummaries = fal
                 // Archive + parse — source may vanish mid-run (Claude Code cleanup).
                 let exchanges;
                 try {
-                    if (!fs.existsSync(archivePath)) {
-                        fs.mkdirSync(path.dirname(archivePath), { recursive: true });
-                        fs.copyFileSync(sourcePath, archivePath);
-                    }
+                    await archiveAdmittedConversation(sourcePath, archivePath);
                     exchanges = await parseConversation(sourcePath, project, archivePath);
                 }
                 catch (error) {
-                    console.log(`Skipped ${file} (read failed: ${error instanceof Error ? error.message : error})`);
                     db.close();
+                    if (error.code !== 'ENOENT')
+                        throw error;
+                    console.log(`Skipped ${file} (read failed: ${error instanceof Error ? error.message : error})`);
                     break;
                 }
                 if (exchanges.length > 0) {
@@ -249,6 +252,7 @@ export async function indexSession(sessionId, concurrency = 1, noSummaries = fal
     }
 }
 export async function indexUnprocessed(concurrency = 1, noSummaries = false) {
+    readRecordExclusions();
     console.log('Finding unprocessed conversations...');
     if (concurrency > 1)
         console.log(`Concurrency: ${concurrency}`);
@@ -284,9 +288,7 @@ export async function indexUnprocessed(concurrency = 1, noSummaries = false) {
                 try {
                     fs.mkdirSync(path.dirname(archivePath), { recursive: true });
                     // Refresh the archive when the source may have grown beyond what we've seen.
-                    if (!fs.existsSync(archivePath) || maxIndexedLine > 0) {
-                        fs.copyFileSync(sourcePath, archivePath);
-                    }
+                    await archiveAdmittedConversation(sourcePath, archivePath);
                     // Parse and filter to exchanges past the high-water mark
                     const exchanges = await parseConversation(sourcePath, project, archivePath);
                     const newExchanges = maxIndexedLine > 0
@@ -297,6 +299,10 @@ export async function indexUnprocessed(concurrency = 1, noSummaries = false) {
                     unprocessed.push({ project, file, sourcePath, archivePath, summaryPath, exchanges: newExchanges });
                 }
                 catch (error) {
+                    if (error.code !== 'ENOENT') {
+                        db.close();
+                        throw error;
+                    }
                     console.log(`  Skipped ${file} (read failed: ${error instanceof Error ? error.message : error})`);
                     continue;
                 }