Repository navigation
Conversation
WaylandYang
left a comment
There was a problem hiding this comment.
Thanks, the fragment reads well and the fallback follows the agreed rule, including deleted_at and the as_of limit. Three things: (1) the three mood checks in materialize.rs still accept any row, so a re-push that drops mood shows no mood but still holds the typed row back. Please wire them through role_word_speaks_at("q", None) here, or note in the description and the 0070 status line that they move to PR 3. (2) Please add a restore test (clear deleted_at, the live chunk speaks again) and an as_of read of a stale statement where the latest chunk didn't exist yet. (3) The fallback subqueries run once per superseded row, so a statement re-pushed hundreds of times may get slow. Could you EXPLAIN ANALYZE one with about 200 pushes, and compute live evidence and the latest end once per fact if it shows? I'll merge this after #1428; it needs update-branch then.
ed433a4 to
ace51a9
Compare
|
Thanks so much for the thoughtful review, all three points made this PR better.
I rebased onto the new #1428 head as well. Thanks again for your patience with the stack! |
…can say something new Signed-off-by: Bogdan Szabo <bogdan@re-cinq.com>
… a value reads back only the new one Signed-off-by: Bogdan Szabo <bogdan@re-cinq.com>
ace51a9 to
92abb23
Compare
Part of #1267, PR 2 of 3, stacked on #1428 (only the last commit is new).
This builds decisions 2 and 3 of ADR 0070. Role words are now read through one rule in
record_axis, next tochunk_live_at. A row speaks when its chunk and that chunk's document are live at the read'sas_of. When a fact has no live evidence at all, the rows on the evidence chunks that were superseded last speak instead, as ofas_of. A deleted document's chunks are dated bydeleted_at, and chunks superseded at the same moment all count. With noas_of, the rule uses the single-branch forms that the module header asks for. The rule has two parts:role_word_fact_endsworks out, once per fact, whether any chunk is live and when the latest one ended, androle_word_speaks_giventests one row against that.role_word_speaks_atputs the two together for a single row. The main reader,graph::statement_qualifiers_for, computes the per-fact part once in a CTE and now takesas_of, andedges_amongandentity_detailpass theirs, so the entity panel, graph edges and MCPentity_factsall follow it. As agreed, live rows are not ordered against each other, so two live chunks that disagree both speak.The three
moodchecks inmaterialize(retiring a source, finding due rows, and phrase rules) now go throughrole_word_speaks_at("q", None)instead of accepting any row. A re-push whose new chunk dropsmoodlets the typed row be made, and one that addsmoodretires it on the next run. The ADR 0070 status line says so.The read shape grows by one field. Each
statement_qualifiersentry keepsrole,value,entity_idandentity_nameand addschunk_ids, the chunks that say that value. Two live values come back as two entries, each with its own chunk.StatementQualifier,web/src/api.tsand the MCP docs describe the new field. Typed rows read through the same rule. Since #1428 a typed row's role word sits only on the chunk of the statement that said it, so the typed row speaks when that chunk does. Until PR 3 re-syncsmaterialize, a typed row whose source was re-pushed still holds the old chunk. It shows the old value while it has no live evidence, and the role drops if its only live evidence comes from a source that never said it.Tests: eight small store tests in
an_open_statement_keeps_the_documents_words.rs. They cover a superseded chunk going quiet once a live chunk gives a new value, two live chunks that disagree, a role the live chunk drops, a stale statement showing its last-superseded chunk, a deleted document dated bydeleted_at, a restored document whose live chunk speaks alone again, anas_ofread between two pushes throughentity_detail, and anas_ofread of a stale statement from before its latest chunk existed.a_bound_statement_becomes_a_typed_factnow ends with a re-push that dropsmood, after which the typed row is made. There is also an end-to-end test insources_statements_tests.rsfor the ADR's own example: a re-push that moveslinesfrom40-52to58-70reads back only58-70. The read tests from #1428 only gainchunk_ids. Each new test fails when I remove the part of the rule it covers. I ran the store, server, core and cli suites against Postgres withUTOPIA_TEST_REQUIRE_DB=1, the migrations twice, the web checks, and fmt and clippy with-D warnings.Cost of the fallback: I ran
EXPLAIN ANALYZEonstatement_qualifiers_forfor one statement pushed 200 times. That's 200 evidence chunks, 3 role words on each (600 rows), in a base with 5,000 other facts, 25,000 chunks and 75,000 role-word rows. The first version ran both fallback subqueries once per role-word row, so the work grew with rows times evidence. The new version aggregates each fact's evidence once. Each number is the median of 10 runs after a warm-up, in ms.as_ofas_ofmid-historyas_ofas_ofmid-historyMachine: Lenovo laptop (Intel Core Ultra 7 258V, 8 cores, 32 GB RAM, NVMe SSD), Fedora, Postgres 16 in Docker (
pgvector/pgvector:pg16, default settings), afterVACUUM ANALYZE.Thanks for taking a look!