Skip to content

Document fragment link table and indexed strands - #52

Open
isomorphisms wants to merge 3 commits into
mainfrom
notes/fragment-link-table
Open

isomorphisms wants to merge 3 commits into
mainfrom
notes/fragment-link-table

Conversation

@isomorphisms

@isomorphisms isomorphisms commented Sep 9, 2026

Copy link
Copy Markdown
Owner

Adds architecture notes for device-dependent fragment delivery, Pensive exposure observations, a persistent link table, multiply indexed adjacency, materialized strands, Android-friendly mmap/range reads, and composable shell queries. Adds a separate filesystem-index note for ~/from/<fragment>/ / ~/to/<fragment>/ projections, symlink-based edges, directory/filename indexing layers, and the rule that hot readers may compile those projections into compact mmap indexes. Also links the fragment design from the persistent storage model.

Copy link
Copy Markdown
Owner Author

User-story / architecture checkpoint; this is deliberately not an implementation request yet.

Two different use cases need to stay visible while the fragment/link/strand design evolves.

1. Original browser-size / render-latency story

The original motivation was roughly: why is the browser so large, and why does it take so long to render anything?

The intended shape was to keep only a very small moving frontier warm — approximately the current material plus 3–5 likely successors — and leave the rest durable/on disk. The present fragment note has the conceptual stages (painted, prepainted, laid out, fetched/decoded, not resident), but that should eventually become an acceptance story rather than only a diagram.

The fragment architecture should be judged partly by whether it permits:

  • a bounded resident set independent of corpus size;
  • only a few warm leads ahead of the current view;
  • cheap eviction of distant parsed/layout/render state;
  • cheap cold-to-warm promotion from durable fragment/strand storage;
  • no requirement to reconstruct a known strand by random graph lookup per fragment;
  • useful first paint / continued scrolling without keeping the whole document, much less the whole corpus, live.

The upcoming Grease native-call work (mmap, munmap, mprotect, msync, openat, linkat, etc.) may make a larger practical difference to this original story than another layer of abstract graph design. In particular, mapped/range-oriented compact indexes plus explicit unmapping give us a plausible way to keep the hot frontier small. Do not make that an architectural dependency prematurely, but revisit this acceptance story once the native-call substrate exists.

A useful scale test remains the large-corpus story: ingest on the order of 100,000 documents/tabs/fragments and show that the reader's resident working set is still governed by the small warm frontier rather than corpus size.

2. Reading should be able to grow into a cross-document trail and then into work

Concrete example from today:

  1. encounter Bob's Hella Cheap / ActivityPub note about revisiting microkernels;
  2. follow its linked explanatory article;
  3. follow from there into a Wikipedia/reference article;
  4. potentially decide: "this sounds useful; find related implementations or code on GitHub and investigate whether any of it should affect one of my projects."

The important question for IB is not whether that particular investigation should run automatically. It should not: at this point there is no job scope, acceptance boundary, cost/token budget, or estimate of how large the investigation would become.

What IB should preserve is enough structure that this progression is cheap and natural:

  • the viewed fragment knows its source document;
  • outgoing links can point into other source documents/fragments, not only within one document;
  • the path actually followed can be retained with provenance;
  • a followed cross-document path can be materialized as a strand without erasing the underlying graph;
  • the reader/view observations can attach to the fragments actually exposed along that path;
  • a later assistant/task action can receive the relevant fragment(s), followed-link trail, and source provenance as context;
  • an unscoped idea can remain a candidate task/todo rather than silently becoming an autonomous research/code job.

This is a useful stress case for the distinction:

graph = all available relationships

strand = one selected reading/work traversal

A strand must therefore be able to cross source-document boundaries. Document order remains one cheap special strand, not the definition of a strand.

Questions to keep open

  • Does the current stable fragment identity scheme work cleanly when a link crosses documents/sources?
  • Can source_fragment -> link -> destination_fragment represent an ordinary Web link before the destination has been ingested? We may need a durable unresolved destination/source-address form that can later resolve to a fragment/source identity.
  • When a user follows note -> article -> Wikipedia, do we preserve only the graph edges plus view history, or also materialize the followed trail as a strand/history object?
  • Can the hot reader fetch the next likely 3–5 leads in batches while leaving other outgoing graph possibilities cold?
  • What measurements prove that fragmentization actually reduces resident memory and cold/render latency instead of merely adding indexing machinery?
  • How should a viewed fragment/link trail become the seed of a model-assisted task while keeping task scope/budget/acceptance explicit and separate from reading?

Treat these as acceptance/user-story constraints on later design work, not as a call to implement all of them in PR #52.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant