Skip to content

A role word speaks while its chunk is live, so a re-push that changes a value reads back only the new one - #1450

Open
gedaiu wants to merge 2 commits into
deeplethe:devfrom
re-cinq:feat/role-words-speak-while-live
Open

gedaiu wants to merge 2 commits into
deeplethe:devfrom
re-cinq:feat/role-words-speak-while-live

Conversation

@gedaiu

@gedaiu gedaiu commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Part of #1267, PR 2 of 3, stacked on #1428 (only the last commit is new).

This builds decisions 2 and 3 of ADR 0070. Role words are now read through one rule in record_axis, next to chunk_live_at. A row speaks when its chunk and that chunk's document are live at the read's as_of. When a fact has no live evidence at all, the rows on the evidence chunks that were superseded last speak instead, as of as_of. A deleted document's chunks are dated by deleted_at, and chunks superseded at the same moment all count. With no as_of, the rule uses the single-branch forms that the module header asks for. The rule has two parts: role_word_fact_ends works out, once per fact, whether any chunk is live and when the latest one ended, and role_word_speaks_given tests one row against that. role_word_speaks_at puts the two together for a single row. The main reader, graph::statement_qualifiers_for, computes the per-fact part once in a CTE and now takes as_of, and edges_among and entity_detail pass theirs, so the entity panel, graph edges and MCP entity_facts all follow it. As agreed, live rows are not ordered against each other, so two live chunks that disagree both speak.

The three mood checks in materialize (retiring a source, finding due rows, and phrase rules) now go through role_word_speaks_at("q", None) instead of accepting any row. A re-push whose new chunk drops mood lets the typed row be made, and one that adds mood retires it on the next run. The ADR 0070 status line says so.

The read shape grows by one field. Each statement_qualifiers entry keeps role, value, entity_id and entity_name and adds chunk_ids, the chunks that say that value. Two live values come back as two entries, each with its own chunk. StatementQualifier, web/src/api.ts and the MCP docs describe the new field. Typed rows read through the same rule. Since #1428 a typed row's role word sits only on the chunk of the statement that said it, so the typed row speaks when that chunk does. Until PR 3 re-syncs materialize, a typed row whose source was re-pushed still holds the old chunk. It shows the old value while it has no live evidence, and the role drops if its only live evidence comes from a source that never said it.

Tests: eight small store tests in an_open_statement_keeps_the_documents_words.rs. They cover a superseded chunk going quiet once a live chunk gives a new value, two live chunks that disagree, a role the live chunk drops, a stale statement showing its last-superseded chunk, a deleted document dated by deleted_at, a restored document whose live chunk speaks alone again, an as_of read between two pushes through entity_detail, and an as_of read of a stale statement from before its latest chunk existed. a_bound_statement_becomes_a_typed_fact now ends with a re-push that drops mood, after which the typed row is made. There is also an end-to-end test in sources_statements_tests.rs for the ADR's own example: a re-push that moves lines from 40-52 to 58-70 reads back only 58-70. The read tests from #1428 only gain chunk_ids. Each new test fails when I remove the part of the rule it covers. I ran the store, server, core and cli suites against Postgres with UTOPIA_TEST_REQUIRE_DB=1, the migrations twice, the web checks, and fmt and clippy with -D warnings.

Cost of the fallback: I ran EXPLAIN ANALYZE on statement_qualifiers_for for one statement pushed 200 times. That's 200 evidence chunks, 3 role words on each (600 rows), in a base with 5,000 other facts, 25,000 chunks and 75,000 role-word rows. The first version ran both fallback subqueries once per role-word row, so the work grew with rows times evidence. The new version aggregates each fact's evidence once. Each number is the median of 10 runs after a warm-up, in ms.

Case Per row (before) Once per fact (after)
Latest push live, no as_of 72.1 3.0
Latest push live, as_of mid-history 92.8 3.6
Stale (all 200 superseded), no as_of 208.2 4.4
Stale, as_of mid-history 92.8 4.6

Machine: Lenovo laptop (Intel Core Ultra 7 258V, 8 cores, 32 GB RAM, NVMe SSD), Fedora, Postgres 16 in Docker (pgvector/pgvector:pg16, default settings), after VACUUM ANALYZE.

Thanks for taking a look!

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the fragment reads well and the fallback follows the agreed rule, including deleted_at and the as_of limit. Three things: (1) the three mood checks in materialize.rs still accept any row, so a re-push that drops mood shows no mood but still holds the typed row back. Please wire them through role_word_speaks_at("q", None) here, or note in the description and the 0070 status line that they move to PR 3. (2) Please add a restore test (clear deleted_at, the live chunk speaks again) and an as_of read of a stale statement where the latest chunk didn't exist yet. (3) The fallback subqueries run once per superseded row, so a statement re-pushed hundreds of times may get slow. Could you EXPLAIN ANALYZE one with about 200 pushes, and compute live evidence and the latest end once per fact if it shows? I'll merge this after #1428; it needs update-branch then.

@gedaiu

gedaiu commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Thanks so much for the thoughtful review, all three points made this PR better.

  1. The three mood checks in materialize.rs now go through role_word_speaks_at("q", None), in one small helper, no_mood_spoken, so they all use the same rule. a_bound_statement_becomes_a_typed_fact now ends with a re-push of the mood statement whose new chunk drops mood, and the typed row gets made. If any mood row still blocked, the test would fail with added 0.
  2. Two new store tests. a_restored_document_lets_its_live_chunk_speak_58_70_alone_again: while the document is deleted, both chunks reach their end at the same moment and both speak. Once deleted_at is cleared, only the live chunk speaks. a_stale_read_as_of_february_shows_40_52_before_the_latest_chunk_existed: at the as_of, the statement had no live chunk, and the later chunk didn't exist yet, so the earlier one speaks. Each one fails when I remove the part of the rule it covers.
  3. You were right that it got slow. With one statement pushed 200 times (600 role-word rows, on top of 5,000 other facts), the read took 72 to 208 ms, because both subqueries ran once per row. The rule is now split into a per-fact aggregate (role_word_fact_ends, which gives "any live chunk" and "latest end" once per fact) and a per-row test (role_word_speaks_given). statement_qualifiers_for computes the aggregate once in a CTE, so the same read takes 3 to 5 ms. The numbers are in the description. The single-row form that the mood checks use is built from the same two pieces.

I rebased onto the new #1428 head as well. Thanks again for your patience with the stack!

…can say something new

Signed-off-by: Bogdan Szabo <bogdan@re-cinq.com>
… a value reads back only the new one

Signed-off-by: Bogdan Szabo <bogdan@re-cinq.com>
@gedaiu
gedaiu force-pushed the feat/role-words-speak-while-live branch from ace51a9 to 92abb23 Compare October 30, 2026 22:24
@gedaiu

gedaiu commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto the new #1428 head (787a6a2), still one commit on top. Nothing in this one changed, and the mood, restore and as_of tests all pass. Thank you for your patience with the stack!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants