Begin IB 0.2 as a shell-first Pensieve

.github/workflows/ib-0.2-shell.yml

@@ -0,0 +1,18 @@
+name: ib 0.2 shell smoke
+
+on:
+  pull_request:
+    branches: [main]
+  push:
+    branches:
+      - '0.2/**'
+
+jobs:
+  shell-smoke:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Exercise Cauldron and Pensieve shell boundaries
+        run: sh tests/test_0_2_shell.grease
+      - name: Exercise deterministic arXiv round trip
+        run: sh tests/test_0_2_arxiv_roundtrip.grease

README.md

@@ -1,36 +1,73 @@
-# ib
+# ib 0.2
 
-An experimental personal browser and task-workbench substrate built around durable browsing state rather than renderer-owned tabs.
+> browsers don't have bookmarks.
 
-IB's immediate target is one person's real browsing corpus and workflows, not general-purpose web compatibility. It optimizes the task behind navigation: learning a documentation set, recovering a fact, finding and sharing an image, comparing delivered prices, or resuming an investigation after the live browser processes are gone.
+The 0.2 line starts again from the persistent information underneath a browser rather than from a renderer, Android application, or large typed browser core.
 
-The substrate supports multiple frontends over the same browser-owned state. A conventional page surface, a small phone frontend, a text-and-action workbench, and developer inspectors may coexist. Renderers, acquisition adapters, extractors, and models remain replaceable; none owns tabs, history, tasks, or accepted organization.
+The first implementation is deliberately shell-first and arXiv-only.
 
-The browser core owns resource, tab, event, and task identity; sleeping and waking; snapshots; organization; indexes; inference acceptance; and renderer selection. Only roughly 3–10 renderer working sets should normally be resident even when the known corpus reaches 10,000 resources.
+## Cauldron and Pensieve
 
-## Design notes
+A fetched thing first enters the **Cauldron**.  The Cauldron is intake: original HTML, PDF bytes, fetched figures, source URLs, and acquisition time.  It may be messy, but it should retain enough source material to reinterpret later.
 
-- `docs/architecture.md` — ownership and replaceable-service boundaries
-- `docs/personal-workbench.md` — personal scope, task frontend, user stories, and latency targets
-- `docs/prefetch-and-reading.md` — durable investigation frontiers, disposable fetches, and `~/reading`
-- `docs/tab-categorization.md` — overlapping personal categories and adaptive refinement
-- `docs/inference-and-learning.md` — local-model proposals, validation, ensembles, and correction events
-- `docs/storage-model.md` — identity levels and canonical, proposed, and derived state
-- `docs/developer-workbench.md` — fixture and memory-pressure harness
+The **Pensieve** is the next layer.  Material distilled from the Cauldron becomes locally searchable text and small metadata/relationship files there.  Indexes are derived from the Pensieve and must be rebuildable.
 
-## Implementation languages
+```text
+arXiv
+  |
+  v
+ICU + shell
+  |
+  v
+Cauldron
+  |  original HTML / PDF / figures / provenance
+  v
+distill
+  |
+  v
+Pensieve
+  |  searchable text / title / links / figure manifest
+  v
+indexes and later model hooks
+```
 
-IB is implemented in **Idriç**. Browser-owned state, policy, and invariants belong in `.idric` source under `src/`.
+A tab or renderer is a temporary view onto this persistent state.  Neither is part of the 0.2 acceptance boundary.
 
-**Grease** is the shell and operating-system language for orchestration. HTTP fetching, temporary directories, file movement, invoking compilers or parsers, cache maintenance, and low-priority model batch passes belong in `.grease` programs rather than being reimplemented as Idriç application logic.
+Three future reader-support contracts are recorded separately:
 
-Python and Ithon are not IB implementation layers. A disposable comparison may exist outside the runtime, but the browser core, storage and index policy, inspector model, and phone-facing application logic must not depend on them.
+- [`docs/reading-feedback.md`](docs/reading-feedback.md): opening is not reading, passive interaction is evidence rather than proof of comprehension, and completed items may deliberately remain in a rereading/reminder rotation.
+- [`docs/reading-assistance.md`](docs/reading-assistance.md): the Pensieve should support interrupted and question-driven reading, fiction recall without spoilers, corpus triage for papers/repositories, and replaceable source-backed model-generated guides/fragments/strands rather than optimizing for finishing long reads.
+- [`docs/reader-support-system.md`](docs/reader-support-system.md): physical bookshelf photographs may enter as source evidence; recognized books and inferred interests remain derived; long-book prefetch can prepare spoiler-aware character/appearance/relationship support; and the division of work among Pensieve, deterministic indexes, and language-model hooks remains deliberately open.
 
-Android or other native code is a narrow platform boundary for facilities Idriç and Grease cannot yet reach directly: NativeActivity, EGL, renderers, kernel-enforced filesystem operations, clipboard and share handoff, and similar FFI edges. Those adapters do not own browser state.
+## Commands
 
-`android-prepaint/` is a deliberately small phone-visible harness for the Idriç
-information prepaint. It uses native Android views rather than `WebView`, applies
-a fixed dark presentation to extracted text, preserves fetched image colors, and
-replaces partial projections with later complete revisions.
+```sh
+bin/ib fetch https://arxiv.org/abs/2203.11355
+bin/ib distill 2203.11355
+bin/ib add https://arxiv.org/abs/1901.09021 https://arxiv.org/abs/2305.00241
+bin/ib search 'Bergman kernel'
+bin/ib reindex
+bin/ib paths
+```
 
-The current Idriç core covers ordered history values, rebuildable indexes, storage classification and read policy, and the renderer-independent inspector model. Scientific-media work adds HTML-first arXiv harvesting, ordered image downloads, caption and alternate-text naming, PDF fallback, and a low-priority second naming pass. The task, category, and generic inference records documented above remain design boundaries rather than claims of completed implementation.
+`add` is just `fetch` followed by `distill`, then an exact-text index rebuild.
+
+By default persistent data lives under `${XDG_DATA_HOME:-$HOME/.local/share}/ib`.  Set `IB_HOME` to put the whole experiment somewhere else.  Set `IB_ICU` to the ICU executable when it is not on `PATH` as `icu`.
+
+The first arXiv corpus is in `tests/fixtures/arxiv-0.2.urls`.
+
+## Host tools
+
+The acquisition path requires ICU.  HTML extraction prefers `xmlstarlet` and has a deliberately crude shell fallback.  PDF-to-text extraction uses `pdftotext` when available.  Missing `pdftotext` does not prevent the HTML-backed Pensieve entry from being created.
+
+This is not a claim of general Web compatibility.  The only site-specific adapter in 0.2 is arXiv.
+
+## Derived indexes and hooks
+
+`bin/ib reindex` currently builds the simplest possible exact-text index: a sorted file list over Pensieve text representations.  `bin/ib search` searches those local files; it does not return to the Web.
+
+After an item is distilled, executable files under `$IB_HOME/hooks/after-distill.d/` are called with the Pensieve item path and its Cauldron source path.  This is intentionally a small process boundary for later vector spaces, hyperplanes, other indexing methods, model-context adapters, and language-model-generated reading guides.  Derived systems may append replaceable, provenance-linked artifacts; they do not own or silently rewrite the Pensieve, source evidence, human annotations, or reading observations.
+
+## Earlier work
+
+The older Idriç browser core, Android prepaint, scientific-media, and workbench material remain on this branch as reference while 0.2 is established.  The active 0.2 path does not depend on them.  They can be converged or removed after this smaller shell architecture proves itself.

bin/ib

@@ -0,0 +1,68 @@
+#!/bin/sh
+set -eu
+
+repository_root=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
+. "$repository_root/lib/ib/pensieve.grease"
+
+usage() {
+  cat >&2 <<'EOF'
+usage:
+  ib fetch ARXIV-ID-OR-URL...
+  ib distill ARXIV-ID-OR-URL...
+  ib add ARXIV-ID-OR-URL...
+  ib search TEXT
+  ib reindex
+  ib paths
+EOF
+  exit 2
+}
+
+test "$#" -gt 0 || usage
+command=$1
+shift
+
+case "$command" in
+  fetch)
+    test "$#" -gt 0 || usage
+    for input in "$@"; do
+      ib_fetch_arxiv "$input"
+    done
+    ;;
+
+  distill)
+    test "$#" -gt 0 || usage
+    for input in "$@"; do
+      ib_distill_arxiv "$input"
+    done
+    ;;
+
+  add)
+    test "$#" -gt 0 || usage
+    for input in "$@"; do
+      ib_fetch_arxiv "$input" >/dev/null
+      ib_distill_arxiv "$input"
+    done
+    ib_reindex_exact_text >/dev/null
+    ;;
+
+  search)
+    test "$#" -eq 1 || usage
+    ib_search_exact_text "$1"
+    ;;
+
+  reindex)
+    test "$#" -eq 0 || usage
+    ib_reindex_exact_text
+    ;;
+
+  paths)
+    test "$#" -eq 0 || usage
+    printf 'home\t%s\n' "$(ib_home)"
+    printf 'cauldron\t%s\n' "$(ib_cauldron)"
+    printf 'pensieve\t%s\n' "$(ib_pensieve)"
+    ;;
+
+  *)
+    usage
+    ;;
+esac

docs/reader-support-system.md

@@ -0,0 +1,159 @@
+# Reader-support system: intake, prefetch, and derived support structures
+
+This note deliberately records goals before IB has decided which operations belong to the Pensieve itself, which belong to deterministic indexes, and which belong to replaceable language-model hooks.
+
+The architecture should preserve that uncertainty rather than prematurely assign every useful behavior to one component.
+
+## Reader support is larger than a document viewer
+
+The intended system should gradually become useful around a person's actual reading life, not only around URLs opened in a browser.
+
+A person may have:
+
+- physical books on shelves;
+- ebooks or PDFs;
+- novels currently being read;
+- scientific papers queued for later;
+- GitHub repositories and documentation sets under investigation;
+- poems or other short works deliberately kept in rotation;
+- prior notes, questions, and remembered connections.
+
+IB should be able to use these as a reader-support environment while keeping source material, human facts, and machine-derived guesses distinct.
+
+## Bookshelf photographs as intake
+
+A photograph of a bookshelf is a legitimate source object.
+
+The original photograph belongs with source/provenance material. It should be retained unchanged if the user chooses to keep it.
+
+OCR, vision, or language-model processing may then propose:
+
+- candidate titles;
+- authors;
+- editions when visible;
+- likely duplicates or volumes in a series;
+- links to already-known Pensieve material;
+- possible topics or relationships among the books.
+
+Those recognitions are derived artifacts. A blurry spine or model guess must not silently become the canonical fact that the person owns or has read a particular book.
+
+A useful model may also form provisional higher-level observations such as `this collection contains a great deal of numerical analysis and geometry`. Those may help later ranking or conversation, but they remain model-derived interpretations with provenance and should be replaceable or discardable.
+
+The aim is to let the system get to know the reader through material the reader has deliberately supplied, without confusing inference with fact.
+
+## Do not freeze Pensieve versus model ownership yet
+
+Some support structures can be built deterministically. Others probably benefit from a language model. Some may eventually have both implementations.
+
+Examples:
+
+- exact occurrence index: deterministic;
+- chapter/section boundaries: often deterministic;
+- names and candidate entities: deterministic, model-assisted, or hybrid;
+- aliases and character identity: often model-assisted;
+- relationship descriptions: likely derived/model-assisted;
+- source-backed summaries: derived;
+- vector or hyperplane indexes: derived and rebuildable;
+- reading observations: durable human interaction evidence, not model output;
+- explicit user annotations or corrections: durable human facts.
+
+The architectural invariant is more important than the initial implementation choice: every support structure must say what source it came from, whether it is observed or inferred, and whether it can be rebuilt.
+
+## Prefetch means preparing useful support, not merely fetching bytes
+
+For a long work, prefetch can extend beyond acquiring the next network resource.
+
+If a person is reading *War and Peace*, background preparation may build reader-support structures before the person explicitly asks for them. Useful work may include:
+
+- chapter and section map;
+- exact character-name occurrence index;
+- candidate aliases, titles, patronymics, and alternate forms;
+- first and previous appearances relative to the reading frontier;
+- places and recurring events;
+- source-backed candidate relationships among characters;
+- local passages likely to answer `who is this person again?`;
+- compact support fragments generated for recurring entities.
+
+This is the same architectural idea as prefetching figures for an arXiv paper or walking a bounded GitHub dependency/documentation graph: spend bounded background work now so the likely next question is cheap later.
+
+The work must remain bounded by storage, CPU, model, and power budgets. Prefetch should improve responsiveness without requiring a permanently live model or renderer.
+
+## Spoiler-aware book support
+
+A complete book may be locally available while the reader is only halfway through it.
+
+Support structures can be built over the whole source when useful, but reader-facing answers should default to the known reading frontier. A query such as `who is this guy?` should prefer evidence from earlier appearances and avoid revealing a later relationship or event merely because the index already knows it.
+
+This suggests separating:
+
+- what the background system is allowed to index;
+- what an answer is allowed to reveal at the current reading frontier.
+
+A model hook that has seen the whole book must obey the same frontier rule as a deterministic index.
+
+## Fragments and support structures are plural
+
+There probably will not be one universal `Fragment` type.
+
+Different generated/support objects may have different purposes, for example:
+
+- character card;
+- previous-appearance fragment;
+- relationship fragment;
+- chapter orientation;
+- paper summary;
+- prerequisite note;
+- repository map;
+- question-answer fragment;
+- corpus-level strand connecting several sources;
+- reminder or rereading cue.
+
+The final vocabulary and type hierarchy remain open. The important requirement is that these objects remain source-linked, inspectable, and distinguishable from original material and human annotations.
+
+## Display is deliberately unresolved
+
+The semantic capability should not depend on a final UI decision.
+
+`Who is this guy again?` might eventually appear as:
+
+- a transient inline card;
+- a side panel;
+- a small overlay;
+- a text-only task response;
+- a search result;
+- a generated strand inserted near the current reading position;
+- some interface not yet designed.
+
+The first contract should therefore be renderer-neutral: given the current item, reading frontier, and a question or selected entity, return source-backed support plus provenance. Presentation can evolve separately.
+
+## Language-model hooks
+
+The existing hook architecture should allow a model to participate in bookshelf intake, book indexing, summaries, relationship extraction, question answering, and corpus synthesis.
+
+The model is not required to own any of these concepts permanently. A later deterministic or specialized implementation may replace a model-generated structure without changing the source corpus or human reading history.
+
+Useful model output should retain:
+
+- source item(s);
+- source revision/hash;
+- source locations when available;
+- model/generator identity;
+- prompt or policy revision when applicable;
+- creation time;
+- staleness;
+- confidence or uncertainty when the operation is genuinely uncertain.
+
+Human correction should outrank a later regenerated guess unless the user explicitly revises that correction.
+
+## Minimum deterministic acceptance cases
+
+1. **Bookshelf source survives.** A synthetic shelf image remains unchanged while recognition artifacts can be deleted and rebuilt.
+2. **Recognition is not ownership fact.** A deliberately ambiguous spine may yield two candidate titles; neither becomes a confirmed owned/read book without human confirmation or stronger evidence.
+3. **Model interpretation stays derived.** A generated observation about the reader's interests can be removed without modifying the source collection or human annotations.
+4. **Book prefetch prepares the next question.** A long synthetic novel is indexed in bounded background work before a `who is this character?` query occurs.
+5. **Previous appearances are cheap.** After prefetch, a returning-character query can retrieve earlier source-backed occurrences without scanning the entire book synchronously.
+6. **No spoiler leakage.** Whole-book indexes may exist, but reader-facing answers at chapter N reveal no fact supported only after chapter N unless unrestricted scope is explicitly selected.
+7. **Different support structures coexist.** Character, chapter, relationship, and question-answer artifacts can refer to the same source without being collapsed into one generic truth record.
+8. **Model replacement is safe.** Rebuilding character relationships with another model changes only derived artifacts, not source text, reading observations, or confirmed human corrections.
+9. **UI independence.** The same support response can be consumed by two different Views without changing its source/provenance semantics.
+10. **Bounded background work.** Increasing book size or corpus size does not imply an unbounded number of live renderers or permanently resident model sessions.