Redact secrets before they reach the archive, index, embeddings and summaries by darthmolen · Pull Request #187 · obra/episodic-memory

darthmolen · 2026-10-05T14:52:26Z

Summary

Pasted secrets currently end up in the conversation archive, the SQLite index, the embeddings, and the prompts sent to the summarizer. This PR replaces them with typed tokens ([REDACTED:<ruleId>]) as transcripts are copied into the archive. Every downstream sink reads the archive, so that one copy is the only place redaction has to happen.

docs/REDACTION.md covers where redaction happens, how secrets are recognized, the guarantees and the limits.

What changes

Engine and rules (src/redaction.ts, src/redaction-rules.ts): rules derived from gitleaks for provider keys, JWTs, private keys, and Azure connection strings, SAS tokens and storage keys. Key-context rules cover secrets with no recognizable shape, such as appsettings.json, web.config, publish profiles, az CLI output and C# assignments. Parsed JSON is also redacted by field name. Git SHAs and GUIDs are allowlisted so they stay searchable. Redaction is idempotent.
Single choke point:
- Sync's archive copy goes through the redactor.
- index now parses the archive instead of the source transcript.
- opencode and legacy Cursor staging exports are redacted when written.
- Summarizer resume (Claude) and fork (Codex) are turned off while redaction is on, because both would let the model read the unredacted source.
Fails closed: in strict mode (the default), sync, index and index repair abort if the rules fail to load. Redaction is on by default, and EPISODIC_MEMORY_REDACTION=off restores the old byte-for-byte copy.
Backfill: episodic-memory redact --rewrite redacts an existing archive and index in place. It re-embeds only the rows that changed and deletes summaries built from unredacted text, so the next sync regenerates them.
- --dry-run writes nothing. It opens the index read-only.
- --dry-run --report lists each hit: location, rule, the value's shape (length, character classes, entropy) and the redacted text around it. The value itself is never printed.
Custom rules go in ~/.config/superpowers/redaction-rules.json and extend the defaults. episodic-memory redact --stdin tries rules on any text.

Behavior changes worth a look

Redaction is on by default.
With redaction on, summaries always come from the redacted text, so Codex-only setups summarize through the Claude Agent SDK. The docs explain this and point to EPISODIC_MEMORY_SKIP_SUMMARIES=1.
index now forces a refresh of the archived copy for transcripts it has already indexed, as it did before. It parses that copy rather than the source.

dist/mcp-server.js is unchanged because the MCP server doesn't import the redaction code. Rebuilding it here would only pick up dependency drift, since the repo has no lockfile.

Test plan

Unit tests for each rule: positive cases, allowlist and false-positive negatives, idempotency, and JSONL structure preserved. All fake secrets are generated at runtime, so no literal in the repo matches a rule.
End-to-end pipeline test: a seeded secret appears in none of the archive, SQLite, embedding input, summarizer input or logs, for both sync and index.
Tests for: fail-closed behavior on a corrupt rules file (including index repair), archive copies keeping the source file's mode, a read-only dry run on an older-schema index, and --report pairing each hit with its token.
Dogfooded on a real archive of about 430 MB on Windows 11: --rewrite --dry-run --report flagged 175 values across 4 rules, then --rewrite applied them. Running the plugin from this branch, a live sync redacted 146 values in 39 newly copied conversations. Reviewing the hits turned up one false-positive pattern, Markdown such as `Password=`, now fixed and tested.
npm test on Windows: 435 passed. The 4 failures are Windows-only and are fixed separately in Make the test suite pass on Windows #186.
CI.

🤖 Generated with Claude Code

Survey of the sync/index/summarize pipeline for the inline secret redaction fork. Picks the archive write as the choke point (option b) and records the three supporting changes it needs: indexer parses the archive, summarizer resume/fork disabled under redaction, staging exports redacted at write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0141gGbp343kbYP6imaJaji2

…bed/summarize Adds an inline, rules-driven redaction stage at the one point every harness passes through: the copy into the conversation archive (Phase 0 option b). Secrets become typed tokens ([REDACTED:<ruleId>]); every downstream sink reads the archive, so SQLite text, tool_calls, embeddings, summaries and show/read only see redacted text. - src/redaction.ts: engine, rules loading/validation, JSONL-preserving line redaction, streaming copyFileRedacted. Findings are rule IDs and counts only. Idempotent: tokens never re-match. - src/redaction-rules.ts: bundled defaults ported from gitleaks for this stack (Azure client secret / storage key / SAS / connection strings, private keys, JWT, provider keys, url creds, key=value assignments); git SHA + GUID allowlist; entropy fallback off by default. - sync.ts / indexer.ts: archive copy goes through the redactor; indexer now parses the archive instead of the source. - summarizer: allowResume option; callers disable Claude resume and Codex fork under redaction, since both read the unredacted source transcript. - opencode / legacy Cursor staging exports are redacted at write time. - Fail closed: strict mode (default) aborts sync/index/import when rules fail to load. Env: EPISODIC_MEMORY_REDACTION, _RULES, _STRICT. - `episodic-memory redact --rewrite [--dry-run]` backfills the existing archive, staging dirs and index in place (re-embeds changed rows, deletes summaries built from unredacted text); also --stdin and --print-default-rules. - Tests: per-rule positives, allowlist negatives, idempotency, config, JSONL structure, end-to-end pipeline (archive/SQLite/embedding input/ summarizer input/logs), fail-closed, rewrite. Fake secrets are built at runtime; no literal in the repo matches a rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0141gGbp343kbYP6imaJaji2

… are caught The first cut only redacted values whose own shape matched a rule, plus an unquoted key=value rule that required a digit. Common .NET/Azure leaks all slipped through: appsettings.json fields, web.config <add key value/>, publish-profile userPWD, `az webapp config appsettings list` and `az keyvault secret show` output, C# `ClientSecret = "..."`, and structured MCP results (parsed JSON lost field names; keys were never checked). Text rules (key context, no digit requirement, placeholder-aware): - azure-keyvault-secret, name-value-secret (either order), xml-appsettings-secret (either order), xml-secret-element, xml-secret-attribute, quoted-secret-assignment. - These opt out of the SHA/GUID allowlist: a GUID in a password slot (legacy create-for-rbac) is a secret. Structured JSON (secretFields, on by default, configurable keyPattern): - a string whose field name ends in a secret keyword is redacted whole; - {name, value} pairs with a secret-looking name and Key Vault bundles have their value redacted; - keys that are themselves secrets are renamed to their token. Harness fields (thinking signature, usage, token_count, apiKeySource) are untouched. Engine: secretGroup may list several groups (first that matched); per-rule useAllowlist. connection-string-secret no longer allows spaces around '=', fixing a C# false positive (options.Password = configuration[...]). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0141gGbp343kbYP6imaJaji2

repairIndex disables summarizer resume when redaction is enabled, while the indexer disables it only when a redactor loaded. They differ only when non-strict rules fail to load, and there verify is the stricter one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

copyFileRedacted opened its output with the 0666 & umask default, so a 0600 transcript became world-readable in the archive, and redact --rewrite widened existing archive files the same way. copyFileSync, used before redaction, kept the source mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

indexUnprocessed used to re-copy every indexed transcript and parse the source. Parsing the redacted archive instead made it depend on copyIfNewer's mtime check, which skips an append that lands within the mtime granularity, so those exchanges were never indexed. copyIfNewer takes a force flag, and indexUnprocessed sets it once a file is indexed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The rule's trailing lookahead excluded '.', which is also in the secret alphabet, so a 40-char secret followed by a period could match neither with nor without it and was left in clear text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ADO.NET keywords are case-insensitive, but the rule only matched the listed capitalization, so `password=` and `PASSWORD=` values were kept. A letters-only value isn't caught by secret-assignment either, which needs a digit. A path after `Pwd=` is still left alone: that's the shell PWD variable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

applyRule skipped any match that overlapped a token, so in partly redacted text (an older rule set, before redact --rewrite) `Password=<secret>[REDACTED:jwt]` kept the secret. The existing tokens are now kept and each stretch between them with a letter or digit is replaced. A stretch already fully covered yields no change, so a second pass is still a no-op. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The field-name rule skipped any value containing a token, so in {"password": "<word> <JWT>"} the JWT rule's token kept the field from being replaced and the word stayed. Only values made up entirely of tokens are skipped now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

repairIndex checked only whether redaction was enabled, so with a corrupt rules file in strict mode it still deleted, re-indexed and summarized, while sync and index abort. It now loads the redactor first, which throws in strict mode, and derives allowResume from the result as the indexer does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

`redact --rewrite --dry-run` opened the index with initDatabase(), which runs schema migrations and switches to WAL, so a dry run changed the database it promised not to touch (and created one if missing). A dry run now opens the existing index read-only, or skips the index if there is none. `--report` (with --dry-run) lists each value the rewrite would redact: file and line or index row, the rule, the value's shape (length, character classes, Shannon entropy) and the redacted text around it. The value itself is never printed. It uses an optional onMatch callback on RedactionContext, so it runs the real redaction path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The report matched values to tokens by rule and order, skipping as many tokens as the original line already had. A new hit before an existing token of the same rule took that token's context. A secret field redacted whole after a text rule hit part of it lost the text rule's hit, and its shape described the partly redacted value. onMatch may now return the token to write. In report mode each hit gets a numbered one ([REDACTED:<rule>--hit<n>]), so it is found exactly; unnumbered tokens were already there. A field value is restored from the numbered tokens inside it, and hits it swallowed are reported at its token. rewriteArchive also refuses report without dryRun: after a real rewrite the files would have nothing left to report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Markdown like `Password=`, `Pwd=` matched with "`," as the value, so docs and chat about connection strings were mangled and inflated the counts (found dogfooding on a real archive). A real value always has a letter or digit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/REDACTION.md now covers where redaction happens (with links to the choke points), the three detection layers, the guarantees, the backfill, configuration and limits, including how this differs from NER. It picks up behavior added since the first draft: index repair failing closed, partial tokens, whole-field replacement, kept file modes, and the connection-string exclusions. PHASE0-FINDINGS.md described the build process rather than the result, so its one lasting point (why the archive write is the choke point) moved into the guide and source comments now point there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

Copilot review overview

🟡 Changes recommended

Dry-run currently writes temporary files, interrupted backfills can retain stale summaries, and unsafe CLI/configuration edge cases remain.

Review effort: Balanced
Findings: 2 · 2

Open (4)

Unknown and conflicting options are silently accepted · New
Stale summaries persist when embedding fails · New
Dry-run creates temporary files and requires writable archives · New
Custom-only configuration still enables bundled secret detection · New

What changed in this PR

Adds default-on secret redaction before conversations reach archives, indexes, embeddings, or summaries, plus tooling to clean existing data.

Changes:

Introduces configurable redaction rules and archive/staging integration.
Adds redact --rewrite, dry-run/reporting, and selective re-embedding.
Adds pipeline, backfill, and fail-closed tests and documentation.

File	Description
`test/verify.test.ts`	Tests fail-closed repair behavior.
`test/redaction-pipeline.test.ts`	Tests end-to-end secret containment.
`test/redact-rewrite.test.ts`	Tests archive/index backfill behavior.
`test/fake-secrets.ts`	Generates deterministic test credentials.
`src/verify.ts`	Applies redaction safeguards during repair.
`src/sync.ts`	Redacts archive copies before downstream processing.
`src/sync-cli.ts`	Loads rules and reports redaction results.
`src/summarizer.ts`	Supports disabling source-session resume.
`src/redaction.ts`	Implements configuration and redaction engine.
`src/redaction-rules.ts`	Defines bundled secret-detection rules.
`src/redact-rewrite.ts`	Implements archive/index backfill.
`src/redact-cli.ts`	Adds redaction CLI modes.
`src/opencode-sync.ts`	Redacts opencode staging exports.
`src/indexer.ts`	Indexes redacted archive copies.
`src/db.ts`	Adds read-only database opening.
`src/cursor-legacy.ts`	Redacts legacy Cursor exports.
`README.md`	Documents redaction usage and settings.
`docs/REDACTION.md`	Provides detailed redaction documentation.
`dist/verify.js`	Generated repair implementation.
`dist/sync.js`	Generated sync implementation.
`dist/sync.d.ts`	Generated sync declarations.
`dist/sync-cli.js`	Generated sync CLI implementation.
`dist/summarizer.js`	Generated summarizer implementation.
`dist/summarizer.d.ts`	Generated summarizer declarations.
`dist/redaction.js`	Generated redaction engine.
`dist/redaction.d.ts`	Generated redaction declarations.
`dist/redaction-rules.js`	Generated bundled rules.
`dist/redaction-rules.d.ts`	Generated rules declarations.
`dist/redact-rewrite.js`	Generated backfill implementation.
`dist/redact-rewrite.d.ts`	Generated backfill declarations.
`dist/redact-cli.js`	Generated redaction CLI.
`dist/redact-cli.d.ts`	Generated CLI declarations.
`dist/opencode-sync.js`	Generated opencode integration.
`dist/opencode-sync.d.ts`	Generated opencode declarations.
`dist/indexer.js`	Generated indexer implementation.
`dist/db.js`	Generated database implementation.
`dist/db.d.ts`	Generated database declarations.
`dist/cursor-legacy.js`	Generated Cursor integration.
`dist/cursor-legacy.d.ts`	Generated Cursor declarations.
`cli/episodic-memory.js`	Registers the redaction command.
`CHANGELOG.md`	Describes the security feature.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+  if (args.includes('--rewrite')) {
+    await runRewrite(args.includes('--dry-run'), args.includes('--report'));
+    return;


+    if (rewriteFileInPlace(file, redactor, dryRun, tally) > 0) {
+      result.filesRewritten++;
+      staleSummaries.add(summaryPathFor(file));
+      if (options.report) reportFile(file, path.relative(archiveDir, file), redactor, options.report);


+    copyFileRedacted(file, temp, redactor, { source: 'rewrite', path: file }, fileTally);
+    if (fileTally.total > 0 && !dryRun) {


+  const config: RedactionConfig = {
+    rules: replaceById(baseRules, file.rules ?? []).filter(rule => !disabled.has(rule.id)),
+    allowlist: replaceById(baseAllowlist, file.allowlist ?? []),
+    entropy: { ...DEFAULT_REDACTION_CONFIG.entropy, ...(file.entropy ?? {}) },
+    secretFields: { ...DEFAULT_REDACTION_CONFIG.secretFields, ...(file.secretFields ?? {}) },
+  };


claude and others added 15 commits October 5, 2026 09:16

Copilot AI balanced review requested due to automatic review settings October 5, 2026 14:52

Copilot started reviewing on behalf of darthmolen October 5, 2026 14:53 View session

Copilot AI reviewed Oct 5, 2026

View reviewed changes

Navigation Menu

Repository navigation

Redact secrets before they reach the archive, index, embeddings and summaries - #187

Redact secrets before they reach the archive, index, embeddings and summaries#187
darthmolen wants to merge 15 commits into
obra:mainfrom
darthmolen:secret-redaction

darthmolen commented Oct 5, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		copyFileRedacted(file, temp, redactor, { source: 'rewrite', path: file }, fileTally);
		if (fileTally.total > 0 && !dryRun) {

Conversation

darthmolen commented Oct 5, 2026

Summary

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants