Skip to content

Draft multiply indexed filesystem link graph - #10

Draft
isomorphisms wants to merge 34 commits into
mainfrom
draft/fragment-link-index
Draft

isomorphisms wants to merge 34 commits into
mainfrom
draft/fragment-link-index

Conversation

@isomorphisms

@isomorphisms isomorphisms commented Sep 9, 2026

Copy link
Copy Markdown
Owner

Purpose

Exercise the fragment/link/index storage idea entirely on the Grease side, with ordinary files and symlinks and no Idriç dependency, against the access shape described by IB PR #52.

Semantic boundary

This draft is testing four roles:

  • durable source/link records with stable fragment identities;
  • rebuildable source- and destination-oriented query projections;
  • batched graph input rather than one process/table scan per edge;
  • a materialized ordered strand that the reader can consume without graph hops.

The current names root-cellar/, fragments/, cauldron/, pensive/strands/, from/, and to/ are prototype spellings, not an architecture or compatibility contract. A later representation may rename, collapse, or compile them while preserving the access semantics.

Current prototype

  • links.tsv is the current durable logical graph: source fragment, relation, destination fragment.
  • every graph edge gets rebuildable forward and reverse symlink projections;
  • from and to enumerate those generated endpoint projections instead of rescanning the complete link table;
  • links ROOT accepts a batch of tab-separated edges and rebuilds once;
  • indexes.tsv describes additional caller-defined projections;
  • strand reads the logical link table once, follows one relation in memory, and writes the selected ordered fragment ids to stdout;
  • the caller chooses where a materialized strand is stored;
  • rebuild destroys and reconstructs generated projections from the durable tables.

The replacement path also avoids plain mv temporary existing-symlink-to-directory, which GNU mv interprets as a directory destination rather than replacing the projection.

IB-shaped exercise

tools/linkfs/ib-shape-acceptance.sh uses:

  • 4,096 document-order fragments, matching the current IB prepaint block ceiling;
  • 64 sparse citation edges and 15 annotation edges, for 4,174 logical edges total;
  • source and reverse-destination queries through the generated projections;
  • one full materialized document-order strand;
  • the 64-fragment window used by the IB fragment-link design note;
  • deletion of the graph/projection state after strand construction, followed by another successful strand/window read.

For the current literal projection layout the receipt observes 8,348 symlinks and 16,514 directories. It reports filesystem KiB and strand bytes without making runner-specific storage costs a pass/fail threshold.

Finding

The prototype now supports the important semantic split in IB #52:

  • filesystem projections work as rebuildable shell/query indexes in both directions;
  • strand construction avoids a graph lookup per hop;
  • a materialized strand remains independently sequentially consumable.

It does not establish that literal directory/symlink projections belong on the Android reader hot path. The inode/path amplification is already large at the 4,096-fragment shape, and the current newline strand does not yet prove a final mmap format or constant-time positional range lookup. Compact source/destination adjacency and an indexed/range-readable strand representation remain separate implementation work.

Keep this PR draft while those boundaries are being measured; do not promote the present directory vocabulary into permanent IB architecture.

Acceptance

The workflow runs two receipts:

  • tools/linkfs/acceptance.sh for small correctness, rebuild, replacement, and non-destructive initialization;
  • tools/linkfs/ib-shape-acceptance.sh for the IB-sized topology and strand hot-path boundary.

Both pass on the current PR head. The workflow does not bootstrap or checkout Idriç.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant