feat: add StateCore as a memory method - #24
Conversation
StateCore (github.com/yul761/StateCore) is an auditable memory engine: a corrected fact supersedes its predecessor on a recorded chain, retirement and discards are logged rather than silent. Wired the same way as the existing memory agents so its row sits beside the published ones on equal terms. DRIVEN THROUGH ITS PUBLISHED PACKAGE. The wrapper spawns `npx -y statecore-mcp@0.5.0 --data <fresh temp dir>` and speaks MCP over stdio, so nothing is installed into this venv and no service needs starting by hand (Node >= 20 on PATH is the only requirement; the version is pinned for reproducibility, STATECORE_MCP_SPEC overrides). Every run gets its own data directory, so runs cannot see each other. ZERO MODEL CALLS ON THE MEMORY SIDE, stated up front so nobody reads the comparison as like-for-like on spend: extraction and supersession are deterministic (a write that reads as a revision of an active fact replaces it on a recorded chain) and retrieval is lexical. The only LLM in the loop is the shared reader every method uses. Whatever this row scores is the floor of the engine's LLM-assisted mode at zero memory-side token cost. NORMALIZED INPUT AND READER are identical to the knowl/agentmemory rows: parse_fact_lines is reused from methods.agentmemory, and the reader path reproduces _handle_bm25_rag through the shared knowl helpers. Configs are copied from Simple_rag_bm25, not chosen: retrieve_num 10, temperature 0.7, input_length_limit 10000000, buffer_length 200. Smoke-tested against the published package: a numbered context with a conflicting update writes 3 facts with 1 supersession at write time, and recall returns the latest version plus the unrelated fact, never the superseded one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NvB2hir4oGnomEDWwVHiCz
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8170a63e6e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| spec = os.environ.get("STATECORE_MCP_SPEC", DEFAULT_SPEC) | ||
| agent.statecore = StateCoreClient(spec) |
There was a problem hiding this comment.
When this method is run on a multi-sample split, main.py creates a new AgentWrapper for every context, so this allocation launches a new npx/Node server each time. Nothing invokes StateCoreMcpClient.close() or removes data_dir; because the child remains running with its stdio pipes open, subprocess.Popen retains it for later reaping rather than terminating it. Splits containing 100–500 contexts can therefore accumulate the same number of Node processes and SQLite directories until interpreter shutdown, potentially exhausting memory or process limits. Add lifecycle cleanup when each context finishes.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in bce6d4e. The harness has no end-of-context hook, so the cleanup is handoff-shaped: contexts run sequentially, so the wrapper keeps a module-level reference to the active client — initializing context N closes context N-1's server and deletes its tempdir store, and an atexit hook releases the last one. At most one Node process and one data dir are alive at any point in a run now. Verified end-to-end against the published statecore-mcp@0.6.0 package: process reaped, directory removed, release idempotent.
Second config, like the knowl pair: statecore_digest picks the arm, so a run
is reproducible from the config alone (STATECORE_DIGEST overrides for
one-off ablations).
The deterministic arm's known blind spot is entity substitution ('prefers X'
vs 'prefers Y'): low lexical overlap, so both survive as active facts. The
digest arm stores writes as events and lets the engine's own distillation
run -- LLM extraction into supersession-tracked facts with semantic conflict
resolution. The spawned process gets FEATURE_LLM=true and inherits
OPENAI_API_KEY (the key the harness already requires); the distillation
model is gpt-5-mini, the engine's recommended model (its runtime sends
reasoning_effort, which the gpt-4o family rejects). The reader stays
gpt-4o-mini like every other row.
Distillation runs at a pending-events threshold during ingestion; flush then
restarts the process once (the startup catch-up pass digests the tail) and
polls the facts tool until the distilled state is stable, with that time
charged to memory_construction_time.
Smoke-tested against the published package with an entity-substitution
conflict ('favorite player is Ronaldo' then 'is Messi now'): the distilled
fact registry holds only the Messi version plus the unrelated fact; the
superseded entity survives only in the raw event history, which trails the
facts in reader order and rarely makes the top-k cut at benchmark scale.
The delta between the two StateCore rows is the measured value of the
engine's distillation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NvB2hir4oGnomEDWwVHiCz
|
Pushed a second commit adding the digest arm of the pair, mirroring how the knowl PR shipped two configs:
Rationale: the deterministic arm's known blind spot is entity substitution ("prefers X" → "prefers Y" — low lexical overlap, so both survive), which FactConsolidation is likely to probe. Smoke-tested exactly that case against the published package: the distilled fact registry holds only the latest version; the superseded entity survives only in raw event history. The delta between the two rows is the measured value of the engine's distillation. |
A startup catch-up pass digests one batch per scope, so with a large pending backlog one restart is not the whole backlog -- the previous settle could observe a stable facts snapshot while undigested events remained. Restart until a fresh catch-up pass changes nothing: that is the fixpoint where another pass has nothing left to digest. Bounded by max_restarts and the same overall deadline, and still charged to memory_construction_time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NvB2hir4oGnomEDWwVHiCz
Three fixes found by running factconsolidation_sh_6k end to end (455 facts, 100 questions) rather than smoke-scale: - The spawned process now gets MODEL_TIMEOUT_MS=120000: the engine's 20s default LLM timeout, tuned for interactive calls, aborted every large-backlog distillation chunk on gpt-5-mini and the digest arm silently ran undistilled. (Upstream default raised in statecore 0.5.1 as well; the explicit env keeps the pinned 0.5.0 correct.) - Reader contents interleave the fact and event layers instead of concatenating facts first. Both layers are relevance-ranked by the engine, but distillation is selective: on a partially distilled store facts-first let a handful of facts crowd every event out of the top-k. Measured: facts-first scored 4.0 where interleaving scores 39.0. - Settle is driven by the computed number of passes the backlog needs (a pass consumes ~40 events), not the facts-snapshot fixpoint alone: on template-heavy corpora a fresh pass's output can be entirely deduped away, leaving the snapshot unchanged while a backlog remains. Budget knob: STATECORE_SETTLE_SECONDS (default 3600). Self-run numbers on factconsolidation_sh_6k (official metric substring_exact_match, n=100, single run, configs as checked in): note arm 15.0, digest arm 39.0. Paper baselines for context: full-context gpt-4o 60.0, HippoRAG-v2 54.0, BM25 48.0, Mem0 18.0, Zep 7.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NvB2hir4oGnomEDWwVHiCz
|
Ran Self-run numbers, official metric (
The 24-point spread between the two arms is the measured value of the engine's distillation, which is what shipping the pair was for. Happy to run other splits or re-run under any settings you'd prefer for the comparison. |
499ebdc to
8e504dd
Compare
0.6.0 ships what this harness surfaced: IDF-weighted relevance scoring from the token index (digest arm on factconsolidation_sh_6k: 39.0 -> 48.0-49.0 across two runs, drawing level with the BM25 baseline), a digest-path LLM timeout default that survives reasoning models, and the documented gpt-5-mini digest default. Confirmed against the published npm package: 48.0. Also adds a .gitignore for the local venv/outputs/env this harness generates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NvB2hir4oGnomEDWwVHiCz
8e504dd to
1cd02e0
Compare
|
Final update: the wrapper now pins Updated numbers on
The IDF change alone moved the digest arm 39.0 → 48–49, drawing level with BM25 — a nice illustration of the loop this PR enables: the benchmark exposed a scoring weakness, the engine fixed it, and the fixed engine reproduces from the pinned public package. The branch history also gained a .gitignore for the venv/outputs this harness generates (an earlier push briefly included a venv; force-pushed clean). |
main.py builds a fresh AgentWrapper per context with no end-of-context hook, so every context leaked its Node process (held via Popen) and its tempdir SQLite store for the life of the run -- on a 100-500 context split that accumulates to real memory/process pressure. Contexts run sequentially, so a module-level slot suffices: initializing context N closes context N-1's server and deletes its store, and an atexit hook releases the last. At most one server and one data dir are now alive at any time. Addresses the Codex P2 review comment on the PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WwCmmaEyf74nqGaEu3USRT
1 participant
Adds StateCore as a memory method, wired the same way as the existing memory agents so its row sits beside the published ones on equal terms — following the pattern of #23.
What StateCore is. An auditable memory engine: a corrected fact supersedes its predecessor on a recorded chain (
supersededBy), and retirement/discards are logged rather than silent. FactConsolidation isolates exactly this write-time behaviour, which is why the row belongs here.How it is driven. Through its published MCP front end: the wrapper spawns
npx -y statecore-mcp@0.5.0 --data <fresh temp dir>and speaks JSON-RPC over stdio. Nothing is installed into the venv and no service needs starting by hand — Node ≥ 20 on PATH is the only requirement. The version is pinned so a run is reproducible from the file alone (STATECORE_MCP_SPECoverrides). Every run gets its own data directory, so runs cannot see each other.Zero model calls on the memory side — stated up front so nobody reads the comparison as like-for-like on spend. Extraction and supersession are deterministic (a write that reads as a revision of an active fact replaces it in place), and retrieval is lexical (ASCII + CJK bigrams). The only LLM in the loop is the shared reader every method uses; whatever this row scores is the floor of the engine's LLM-assisted mode at zero memory-side token cost.
Normalized input and reader are identical to the knowl/agentmemory rows:
parse_fact_linesis reused frommethods.agentmemory, and the reader path reproduces_handle_bm25_ragthrough the shared knowl helpers. Configs are copied fromSimple_rag_bm25, not chosen:retrieve_num10,temperature0.7,input_length_limit10000000,buffer_length200.Smoke-tested against the published npm package: a numbered context with a conflicting update writes 3 facts with 1 supersession at write time; recall returns the latest version plus the unrelated fact, never the superseded one.
Happy to adjust anything to match harness conventions, and to run whatever splits you'd like numbers for.
🤖 Generated with Claude Code
https://claude.ai/code/session_01NvB2hir4oGnomEDWwVHiCz