diff --git a/docs/filesystem-link-indexes.md b/docs/filesystem-link-indexes.md new file mode 100644 index 0000000..d1addcd --- /dev/null +++ b/docs/filesystem-link-indexes.md @@ -0,0 +1,127 @@ +# Filesystem link indexes + +The fragment graph should be usable through ordinary files and directories as well as compact binary indexes. The filesystem representation is not merely a debug dump: directory names, filenames, and symlinks provide another set of indexes that shell programs can traverse directly. + +A basic bidirectional projection is: + +```text +~/from/231/... +~/to/231/... +``` + +`from/231` answers "what does fragment 231 point to?" and `to/231` answers "what points to fragment 231?" without scanning the complete link table. + +## Prefer symlinks for the filesystem projection + +An empty file whose name is another fragment id can represent an edge, but a symlink carries useful extra structure: the directory entry names the relationship while the target identifies the canonical fragment object. + +For example: + +```text +~/fragment/231 +~/fragment/804 + +~/from/231/804 -> ../../fragment/804 +~/to/804/231 -> ../../fragment/231 +``` + +The forward and reverse directories deliberately duplicate one logical edge. They are indexes optimized for opposite queries. + +Relative symlinks are preferable when practical because moving the containing corpus does not require rewriting every absolute path. + +The durable graph must still have stable fragment identities. A symlink is one materialized projection of that identity, not a process pointer. + +## Directories and filenames are another indexing layer + +Do not reserve the directory hierarchy merely for `from/` and `to/`. Directory structure can classify the same edges several ways. + +For example: + +```text +~/from/231/ + next/ + 232 -> ../../../fragment/232 + citation/ + 804 -> ../../../fragment/804 + annotation/ + 991 -> ../../../fragment/991 + +~/to/804/ + citation/ + 231 -> ../../../fragment/231 +``` + +That permits ordinary filesystem queries for both endpoint and link type. + +Additional rebuildable projections might include: + +```text +~/by-link-kind/citation/from/231/804 +~/by-link-kind/citation/to/804/231 + +~/by-source/book-17/paragraph/0042/231 +~/by-document-order/book-17/0000042/231 + +~/by-strand/27/0000500/231 +~/by-fragment/231/strand/27/0000500 + +~/by-tag/conditioning/231 +~/by-task/42/231 +``` + +The exact hierarchy is intentionally not fixed. One purpose of the semantic operating-system experiment is to permit the same object to appear in many useful directory indexes without pretending that there is one privileged tree. + +Filenames can carry sortable fields where that is useful. A zero-padded ordinal such as `0000500` makes ordinary directory order represent strand order. A filename may also encode a compact edge id, rank, source coordinate, or other value when doing so makes a common shell query simpler. + +Avoid stuffing all metadata into filenames merely because it is possible. Use names for fields that are useful as lookup or ordering keys; keep richer metadata in the link record or a sidecar record. + +## Logical graph versus projections + +The system can therefore have several simultaneous representations of the same relationship: + +```text +canonical / append-friendly link record + | + +-> from/ filesystem index + +-> to/ filesystem index + +-> link-kind filesystem indexes + +-> strand directories + +-> compact source adjacency index + +-> compact destination adjacency index + `-> mmap/range-read indexes used by the reader +``` + +This duplication is deliberate. Indexes should be cheap to rebuild and disposable independently of the durable relationship record. + +The filesystem projections serve shell traversal, inspection, composition, and semantic organization. The compact binary indexes serve hot paths where resolving thousands of pathname components or performing one filesystem operation per edge would be unnecessarily expensive. + +## Strand construction + +Filesystem indexes can help construct a strand without requiring one random disk lookup for every graph hop. A shell program can enumerate a `from///` directory or another precomputed projection in batches, choose edges, and emit a strand file or strand directory. + +Once selected, the hot reading form should still be materialized contiguously when appropriate: + +```text +strand 27 -> [231, 804, 991, ...] +``` + +The filesystem graph is therefore a rich query surface; a strand is a compiled traversal optimized for sequential consumption. + +## Android first + +The first implementation should use whatever ordinary filesystem and symlink support is available in the IB application's accessible storage on the current Android phone. Do not require a new filesystem. + +Measure both costs: + +- how many inode/directory entries the projections consume; +- how much latency comes from pathname and symlink resolution versus compact mmap indexes. + +If a particular projection becomes too expensive as literal directories, retain its semantics and materialize it as a compact index instead. The important design commitment is multiple query-oriented indexes, not that every index must literally be a directory. + +## Semantic operating-system direction + +Multiply indexing the same fragments and links is a core semantic-system idea. Filesystems already provide several useful primitives—names, directories, links, ordering by names, permissions, and ordinary stream tools. IB should exploit those first. + +If the same patterns recur everywhere, they provide concrete evidence for later OS/filesystem primitives: typed links, reverse-link queries, indexed directory projections, stable fragment identities, or fast materialization of a graph walk into a sequential strand. + +See also `fragment-link-table.md` and `storage-model.md`. diff --git a/docs/fragment-link-table.md b/docs/fragment-link-table.md new file mode 100644 index 0000000..2cf71bd --- /dev/null +++ b/docs/fragment-link-table.md @@ -0,0 +1,230 @@ +# Fragment links, indexed strands, and reading observations + +IB should be able to present a document as a bounded stream of fragments while Pensive records what the view actually exposed. The phone is one view adapter, not the definition of reading. The first implementation should work with ordinary files on Android; filesystem or OS changes can come later if the file representation demonstrates a useful primitive. + +The main storage rule is deliberate redundancy: + +> Index the same durable relationships several ways when doing so turns an important query into a RAM lookup, contiguous range read, or sequential walk. + +Indexes are not canonical truth. They are rebuildable projections of durable fragment, link, strand, and observation records. Duplication is acceptable when it removes repeated random I/O from a hot path. + +## Fragments are delivered by the view + +Do not require an in-band sentinel character. A paragraph sign or other marker may be shown in a workbench to make chunk boundaries visible, but it is instrumentation, not document content. + +A view consumes a moving window of fragments. A phone may prepare them in stages: + +```text +current painted +next prepainted +next + 1 laid out +next + 2 fetched/decoded +farther not resident +``` + +The window size is adapter- and memory-dependent. A laptop may keep much more resident; another view may not scroll at all. + +A fragment may line up with a paragraph, viewport-sized chunk, source block, or another cut chosen by the adapter/chunker. Reading is device-dependent, so IB should not pretend that a phone screen and a laptop pane expose identical units. + +Stable content coordinates still matter. A view-specific fragment should be able to identify the underlying source range it covers so Pensive can relate observations made by different views. + +## View observations for Pensive + +The view reports observations, not conclusions such as `read` or `understood`. + +Useful observations include: + +```text +fragment entered view +fragment left view +fragment passed +fragment returned +fragment activated or selected +visible duration +visible fraction, when available +view adapter / device class +``` + +Scroll position, scroll velocity, pointer position, and viewport geometry are optional adapter-specific evidence. A non-scrolling view must still be able to report useful exposure and navigation events. + +Repeated returns are particularly useful evidence. Pensive may later infer that a repeatedly revisited range deserves a stronger index entry, prefetch priority, landmark, or workbench shortcut. That inference remains separate from the raw observation log. + +## Fragment records + +Persistent relationships must not depend on process virtual addresses. Raw C pointers die across restart or remapping. + +A compact fragment record may contain hot fixed fields such as: + +```text +fragment_id +source_id +source_start +source_end +previous_fragment +next_fragment +edge_start +edge_count +``` + +`previous_fragment` and `next_fragment` may be duplicated here even when the same relationship also appears in the general link table. They are hot enough to justify duplication. + +Inside a particular mmap file, a derived index may use relative offsets for cheap pointer-like access. Durable records should retain stable fragment identities or otherwise survive remapping and rebuilding. + +## Link table + +The logical graph is a link table. One possible logical row is: + +```text +source_fragment +link_kind +destination_fragment +rank +flags +metadata_reference +``` + +Examples of `link_kind` include document order, parent/child structure, citation, annotation, source, semantic association, task membership, and Pensive-created relationships. + +The logical table does not imply that every query scans or probes a generic database table. Physical representations should be specialized for the queries IB actually performs. + +## Multiply index the graph + +At minimum maintain source-oriented and destination-oriented indexes. + +A source-oriented adjacency index can look like compressed sparse row storage: + +```text +fragment_directory + fragment 184 -> edges[8301..8307] + fragment 185 -> edges[8308..8310] + +edges + 8301 -> next 185 + 8302 -> parent 173 + 8303 -> citation 9271 +``` + +The small `fragment -> (start,count)` directory can remain resident in RAM. All outgoing links for one fragment are contiguous, so `links from 184` becomes one small lookup followed by one range access. + +Maintain the reverse projection as well so `what points to 184?` does not scan the graph. + +Likely useful indexes include: + +```text +source_fragment -> contiguous outgoing edges +destination_fragment -> contiguous incoming edges +(source_fragment, link_kind) -> matching outgoing edge range +(destination_fragment, link_kind) -> matching incoming edge range +link_kind -> edge range or posting list +source coordinate -> fragment(s) +fragment -> strand memberships +strand -> ordered fragment ids +``` + +More indexes are acceptable when measurements show a recurring query. The semantic-system direction is to make important relationships cheap from several directions rather than treating one normalized representation as sacred. + +The durable relationship set can remain simple and append-friendly while indexes are generated, compacted, replaced, and rebuilt independently. + +## Strands + +A strand is one chosen ordered journey through fragments. It is distinct from the full graph. + +```text +graph = all available relationships +strand = an ordered sequence selected for reading or work +``` + +A known document-order strand should not be reconstructed by repeatedly looking up `next` and seeking again. + +Store a materialized strand as a contiguous sequence: + +```text +strand 27 + 184 + 185 + 186 + 191 + 205 + 206 +``` + +Then a view can request a window such as positions `500..563`, receive many fragment ids in one range read, and queue their text/layout/prepaint work without one graph lookup per hop. + +An arbitrary Pensive graph walk may also be materialized into a temporary or durable strand when the user chooses to read it. Materialization is an optimization and a record of a chosen traversal; it does not erase the graph that produced it. + +## Ordinary-file first layout + +A first Android implementation can use ordinary files and mmap/range reads. A possible conceptual layout is: + +```text +state/fragments/ + fragments.idx + text.dat + links.dat + observations.log + indexes/ + links-by-source.idx + links-by-destination.idx + links-by-source-kind.idx + source-ranges.idx + fragment-strands.idx + strands/ + .idx +``` + +This grammar is not frozen. The important properties are: + +- small directories/index roots can remain resident; +- neighboring records needed together are stored contiguously; +- distant rendered/layout state can be evicted; +- indexes can be rebuilt from durable records; +- a reader can fetch a useful window with a few range operations rather than many random seeks; +- the same storage can be queried without running a heavyweight database server. + +Large text bodies are not necessarily the main RAM cost. Parsed structures, layout objects, decoded images, glyph runs, and rendered surfaces can dominate. Keep those bounded separately from durable fragment metadata. + +## Shell query surface + +The graph and strand representation should have small composable commands. Exact names remain open, but the operations should include the equivalent of: + +```text +ib-fragment text 184 +ib-link from 184 +ib-link to 184 +ib-link from 184 --kind citation +ib-strand window 27 500 64 +ib-strand containing 184 +``` + +Commands should support batch/range input and ordinary stdin/stdout composition so a program can traverse hundreds of relationships without launching one query per edge. + +For example, a strand window command should be able to emit a block of fragment ids that another command consumes as one batch, rather than forcing: + +```text +next(184) +next(185) +next(186) +... +``` + +The binary/index formats may be optimized, but inspection and debugging should remain possible through these tools. + +## Relation to prefetch and prepaint + +The strand gives the view a cheap ordered lookahead source. Preparation remains staged: + +```text +fetch -> decode/parse -> layout -> prepaint -> display +``` + +The view can walk the strand ahead of the current position and decide independently how far to advance each stage. Android may keep only a narrow painted/prepainted window while retaining a larger cheap fragment-id window. + +Pensive observations flow the other direction: the view records which fragments/ranges were actually exposed and for how long. Prefetch policy may use those observations later, but exposure telemetry is not itself a claim that the material was read. + +## Filesystem and OS direction + +This representation may eventually suggest filesystem or operating-system primitives for typed links, fragment addressing, graph indexes, or strand traversal. Do not require those changes for the first implementation. + +First make the ordinary-file version fast on the current phone. If the implementation repeatedly recreates the same indexing, adjacency, range-fetch, or typed-link mechanisms, those repeated mechanisms become concrete candidates for lower-level support. + +See also `storage-model.md`, `prefetch-and-reading.md`, `resource-constrained-rendering.md`, and `prepaint-display-contract.md`. diff --git a/docs/storage-model.md b/docs/storage-model.md index abf5623..18cf909 100644 --- a/docs/storage-model.md +++ b/docs/storage-model.md @@ -81,6 +81,8 @@ A task persists the user's intention and unfinished frontier even when every ass Fetched bodies, extracted views, rankings, and summaries are not the task itself. See `docs/personal-workbench.md` and `docs/prefetch-and-reading.md`. +For fragment-level reading graphs, multiply indexed link tables, materialized strands, and the ordinary-file Android path, see `docs/fragment-link-table.md`. The durable graph may remain simple while rebuildable source-, destination-, type-, and strand-oriented indexes duplicate relationships deliberately to make hot traversals sequential or RAM-addressable. + ## Snapshots and representations Large content belongs separately from small metadata. A resource or event can refer to zero or more stored representations: