You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Store node indexes so engines skip the scan at start-up #117
With persist_csr=True (#109), an engine loads each relationship type's CSR from a stored file instead of building it. Node indexes still get built at every start. A node index (NodeRows, reported by node_index_stats()) maps each node key of a label to its Lance row id, so a hop can fetch target rows with take_rows. It's built from a scan of every node table, even when the CSR itself is loaded.
On GDC SF1 read lazily, a start that loads every stored CSR takes 0.46 s, against 1.71 s for one that builds. Nobody has measured how much of that 0.46 s goes to node indexes, so measure that first. If it's small, this issue may not be worth doing.
Idea
Store each label's node index the same way #109 stores the CSR: an immutable file per dataset version in the node table's own _lance_csr/ folder, holding keys and row ids. It would be validated against the dataset version and fragments, and brought up to date with a delta from added fragments only. Writes to a node table already invalidate only that label's node index, never the CSR.
Open question: when the node table already has a Lance B-tree scalar index on its ID column, can we use that instead of our own file?
Smaller follow-ups
If loading many relationship types still adds up, an engine could load each type's CSR file on the first query that uses it, not at construction. That moves the cost rather than removing it.
A future Lance release could garbage-collect files it doesn't recognise, or claim the _lance_csr folder name. Validation would turn either into a rebuild rather than a wrong answer, but it may be worth asking the Lance maintainers to reserve the name.
Activity