Issues

• Cat 5: --edge-tables narrows edges but not nodes, so isolate/connectivity rates count structurally-impossible isolates (69% of the reported figure on a real graph)

  M0nkeyFl0wer/multipass-structural-memory-eval#56 · Filter by author M0nkeyFl0wer opened on Aug 21, 2026

• Benchmarking campaign 2026-05-29: retrieval near-ceiling, the reader is the bottleneck — results page

  M0nkeyFl0wer/multipass-structural-memory-eval#52 · Filter by author jphein opened on May 29, 2026

• Judge-human agreement calibration on SME corpora (N=50)

  M0nkeyFl0wer/multipass-structural-memory-eval#46 · Filter by author M0nkeyFl0wer opened on May 26, 2026
  1

• Ontology-sensitivity reading: how much do Cat 4 / Cat 5 readings move under different ontology choices?

  M0nkeyFl0wer/multipass-structural-memory-eval#45 · Filter by author M0nkeyFl0wer opened on May 26, 2026
  1

• Diagnostic actionability: every finding ships with 'fix this and re-run' guidance + public case-study catalog

  M0nkeyFl0wer/multipass-structural-memory-eval#44 · Filter by author M0nkeyFl0wer opened on May 26, 2026

• Integrate standard corpora (LoCoMo, HotpotQA, MINE) for per-category validation evidence

  M0nkeyFl0wer/multipass-structural-memory-eval#43 · Filter by author M0nkeyFl0wer opened on May 26, 2026

• Tier-1 standards integration: ship the five audited items

  M0nkeyFl0wer/multipass-structural-memory-eval#41 · Filter by author M0nkeyFl0wer opened on May 26, 2026

• corpus-doctor: synthetic defect injection for stress-testing adapters (YCSB + TPC-H + CheckList framings)

  M0nkeyFl0wer/multipass-structural-memory-eval#27 · Filter by author M0nkeyFl0wer opened on May 19, 2026

• Cat 10 — The Echo: read-after-write consistency for agentic memory (JEPSEN-style)

  M0nkeyFl0wer/multipass-structural-memory-eval#24 · Filter by author M0nkeyFl0wer opened on May 19, 2026

• Judge variance: characterize intra-rater LLM-judge stochasticity, K-replicate readings

  M0nkeyFl0wer/multipass-structural-memory-eval#22 · Filter by author M0nkeyFl0wer opened on May 19, 2026

• Test coverage gaps for measurement-critical modules (multi_hop, ontology_coherence, topology, adapters)

  M0nkeyFl0wer/multipass-structural-memory-eval#18 · Filter by author jphein opened on May 16, 2026
  1

• Ingestigation roadmap: integrate SHACL / PROV-O / ProVe / Splink / OpenLineage where SME currently rolls its own

  M0nkeyFl0wer/multipass-structural-memory-eval#12 · Filter by author M0nkeyFl0wer opened on May 1, 2026
  1

• Cross-validate SME categories against LongMemEval / LoCoMo / MemoryBench

  M0nkeyFl0wer/multipass-structural-memory-eval#9 · Filter by author M0nkeyFl0wer opened on May 1, 2026
  14

• Proposed: phantom-edge category — graph edges asserted with no support in source files

  M0nkeyFl0wer/multipass-structural-memory-eval#4 · Filter by author jphein opened on Apr 30, 2026
  4

• Cat 9a (invocation rate) — proposed measurement protocol from two RLM-orchestrator runs

  M0nkeyFl0wer/multipass-structural-memory-eval#3 · Filter by author jphein opened on Apr 30, 2026
  11