Clarify inclusion learning and browsing context

.github/workflows/category-hyperplane-real-embedding.yml

@@ -0,0 +1,180 @@
+name: IB real embedding probe
+
+on:
+  pull_request:
+    paths:
+      - 'experiments/category-hyperplanes/**'
+      - 'tests/fixtures/real_world_urls.txt'
+      - '.github/workflows/category-hyperplane-real-embedding.yml'
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  mxbai-reading-probe:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+
+      - name: Install pinned experiment dependencies
+        run: |
+          python -m pip install \
+            numpy==1.26.4 \
+            scikit-learn==1.5.2 \
+            onnxruntime==1.19.2 \
+            transformers==4.45.2 \
+            huggingface-hub==0.25.2
+
+      - name: Validate recovered assertions and build canonical inputs
+        run: |
+          python experiments/category-hyperplanes/validate_recovered_labels.py
+          python experiments/category-hyperplanes/build_url_inputs.py \
+            --source tests/fixtures/real_world_urls.txt \
+            --output /tmp/ib-recovered-inputs.tsv
+
+      - name: Embed the pinned fixture with real mxbai INT8 ONNX weights
+        run: |
+          python experiments/category-hyperplanes/embed_mxbai_onnx.py \
+            --input-texts /tmp/ib-recovered-inputs.tsv \
+            --vectors /tmp/ib-mxbai-vectors.npz \
+            --provenance /tmp/ib-mxbai-provenance.json \
+            --max-input-tokens 256
+
+      - name: Fit the reading positive-unlabeled probe
+        run: |
+          MODEL_SHA="$(python -c 'import json; print(json.load(open("/tmp/ib-mxbai-provenance.json"))["model_file_sha256"])')"
+          TOKENIZER_SHA="$(python -c 'import json; print(json.load(open("/tmp/ib-mxbai-provenance.json"))["tokenizer_sha256"])')"
+          BACKEND_VERSION="$(python -c 'import json; print(json.load(open("/tmp/ib-mxbai-provenance.json"))["backend_version"])')"
+          python experiments/category-hyperplanes/probe.py \
+            --vectors /tmp/ib-mxbai-vectors.npz \
+            --input-texts /tmp/ib-recovered-inputs.tsv \
+            --labels experiments/category-hyperplanes/recovered-reading-fit.tsv \
+            --evaluation-labels experiments/category-hyperplanes/recovered-reading-evaluation.tsv \
+            --category reading \
+            --model-id mixedbread-ai/mxbai-embed-xsmall-v1 \
+            --model-revision b0561d9a97e6b298da39f0ef3e7d3cf153b1b29a \
+            --model-file-sha256 "onnx/model_quantized.onnx=${MODEL_SHA}" \
+            --backend onnxruntime \
+            --backend-version "${BACKEND_VERSION}" \
+            --weight-precision int8 \
+            --tokenizer-revision b0561d9a97e6b298da39f0ef3e7d3cf153b1b29a \
+            --tokenizer-sha256 "${TOKENIZER_SHA}" \
+            --pooling mean \
+            --input-prefix '' \
+            --input-grammar 'url: <url>' \
+            --input-builder-revision d6b67a1c18bbfac56ab61ab3c57f721f8c95f556 \
+            --max-input-tokens 256 \
+            --truncation-side right \
+            --truncation-strategy longest_first \
+            --truncation-dimension 384 \
+            --output /tmp/ib-mxbai-reading-probe.json
+
+      - name: Print diagnostic summary
+        run: |
+          python - <<'PY' > /tmp/ib-mxbai-reading-summary.txt
+          import csv
+          import json
+          from collections import Counter, defaultdict
+
+          import numpy as np
+
+          with open('/tmp/ib-mxbai-reading-probe.json', encoding='utf-8') as stream:
+              report = json.load(stream)
+          with open('/tmp/ib-recovered-inputs.tsv', encoding='utf-8', newline='') as stream:
+              texts = {row['id']: row['text'] for row in csv.DictReader(stream, delimiter='\t')}
+
+          result = report['categories'][0]
+          ranking = {row['id']: row for row in result['ranking']}
+          rank = {row['id']: offset for offset, row in enumerate(result['ranking'], start=1)}
+          support_count = Counter()
+          sample_count = Counter()
+          max_slack = defaultdict(float)
+          slack_count = Counter()
+          normals = []
+          for plane in result['planes']:
+              support_count.update(plane['support_ids'])
+              sample_count.update(plane['provisional_unlabeled_ids'])
+              normals.append(np.asarray(plane['w'], dtype=np.float64) / plane['w_norm'])
+              for violation in plane['largest_margin_violations']:
+                  row_id = violation['id']
+                  slack_count[row_id] += 1
+                  max_slack[row_id] = max(max_slack[row_id], violation['slack'])
+
+          normals = np.vstack(normals)
+          cosine = normals @ normals.T
+          upper = cosine[np.triu_indices(len(normals), 1)]
+          support_sizes = [len(plane['support_ids']) for plane in result['planes']]
+          proposal_count = sum(row['model_proposed'] for row in result['ranking'])
+
+          print(f"retained_unique_plane_count={result['retained_unique_plane_count']}")
+          print(f"provisional_unlabeled_per_plane={result['provisional_unlabeled_per_plane']}")
+          print(f"support_vectors_per_plane={min(support_sizes)}..{max(support_sizes)}")
+          print(f"plane_normal_cosine_min={upper.min():.6f}")
+          print(f"plane_normal_cosine_median={np.median(upper):.6f}")
+          print(f"plane_normal_cosine_max={upper.max():.6f}")
+          print(f"inclusion_threshold={result['inclusion_threshold']:.9f}")
+          print(f"model_proposal_count={proposal_count}")
+          print(f"held_out_positive_recall={result['held_out_positive_recall']}")
+          print()
+          print('held-out positives:')
+          for row in result['ranking']:
+              if row['evaluation_positive']:
+                  print(
+                      f"  rank={rank[row['id']]} {row['id']} score={row['score']:.9f} "
+                      f"zero_vote={row['zero_surface_vote_fraction']:.3f} "
+                      f"proposed={row['model_proposed']} {texts[row['id']]}"
+                  )
+
+          print()
+          print('fit positives:')
+          for row in result['ranking']:
+              if row['fit_positive']:
+                  row_id = row['id']
+                  print(
+                      f"  rank={rank[row_id]} {row_id} score={row['score']:.9f} "
+                      f"zero_vote={row['zero_surface_vote_fraction']:.3f} "
+                      f"support={support_count[row_id]}/{len(result['planes'])} "
+                      f"slack_nonzero={slack_count[row_id]}/{len(result['planes'])} "
+                      f"largest_reported_slack={max_slack[row_id]:.9f} "
+                      f"{texts[row_id]}"
+                  )
+
+          print()
+          print('most frequently sampled unlabeled support examples:')
+          fit_positive_ids = {
+              row['id'] for row in result['ranking'] if row['fit_positive']
+          }
+          for row_id, sampled in sample_count.most_common():
+              if row_id in fit_positive_ids:
+                  continue
+              row = ranking[row_id]
+              print(
+                  f"  {row_id} support={support_count[row_id]}/{sampled}_sampled "
+                  f"score={row['score']:.9f} proposed={row['model_proposed']} "
+                  f"slack_nonzero={slack_count[row_id]}/{sampled}_sampled "
+                  f"largest_reported_slack={max_slack[row_id]:.9f} {texts[row_id]}"
+              )
+
+          print()
+          print('nearest model-policy boundary:')
+          for row in result['nearest_model_policy_boundary'][:10]:
+              print(f"  rank={rank[row['id']]} {row['id']} score={row['score']:.9f} {texts[row['id']]}")
+          PY
+          cat /tmp/ib-mxbai-reading-summary.txt
+
+      - name: Preserve real-model evidence
+        uses: actions/upload-artifact@v4
+        with:
+          name: ib-mxbai-reading-probe
+          path: |
+            /tmp/ib-recovered-inputs.tsv
+            /tmp/ib-mxbai-vectors.npz
+            /tmp/ib-mxbai-provenance.json
+            /tmp/ib-mxbai-reading-probe.json
+            /tmp/ib-mxbai-reading-summary.txt
+          if-no-files-found: error

README.md

@@ -4,7 +4,7 @@ An experimental personal browser and task-workbench substrate built around durab
 
 IB's immediate target is one person's real browsing corpus and workflows, not general-purpose web compatibility. It optimizes the task behind navigation: learning a documentation set, recovering a fact, finding and sharing an image, comparing delivered prices, or resuming an investigation after the live browser processes are gone.
 
-The substrate supports multiple frontends over the same browser-owned state. A conventional page surface, a small phone frontend, a text-and-action workbench, and developer inspectors may coexist. Renderers, acquisition adapters, extractors, and models remain replaceable; none owns tabs, history, tasks, or accepted organization.
+The first two user frontends share the same browser-owned state: a visual frontend that immediately pre-paints the cheapest useful source-backed view, and a ChatGPT-like text-only-by-default task frontend that answers questions and offers actions over the browsing corpus. Additional page surfaces and developer inspectors may coexist later. Renderers, acquisition adapters, extractors, and models remain replaceable; none owns tabs, history, tasks, or accepted organization.
 
 The browser core owns resource, tab, event, and task identity; sleeping and waking; snapshots; organization; indexes; inference acceptance; and renderer selection. Only roughly 3–10 renderer working sets should normally be resident even when the known corpus reaches 10,000 resources.
 
@@ -14,9 +14,10 @@ The browser core owns resource, tab, event, and task identity; sleeping and waki
 - `docs/personal-workbench.md` — personal scope, task frontend, user stories, and latency targets
 - `docs/prefetch-and-reading.md` — durable investigation frontiers, disposable fetches, and `~/reading`
 - `docs/tab-categorization.md` — overlapping personal categories and adaptive refinement
-- `docs/inference-and-learning.md` — local-model proposals, validation, ensembles, and correction events
+- `docs/inference-and-learning.md` — configured-model proposals, explicit hyperplanes, ensembles, and human supervision
 - `docs/storage-model.md` — identity levels and canonical, proposed, and derived state
 - `docs/developer-workbench.md` — fixture and memory-pressure harness
+- `experiments/category-hyperplanes/README.md` — disposable embedding and explicit affine-separator probe
 
 ## Implementation languages
 

bin/ci_browser_foundation.grease

@@ -60,6 +60,14 @@ exercise_workbench() {
   cd "$repository_root"
   sh -n tests/test_real_world_url_fixture.grease
   sh tests/test_real_world_url_fixture.grease
+  python3 -m py_compile \
+    experiments/category-hyperplanes/build_url_inputs.py \
+    experiments/category-hyperplanes/validate_recovered_labels.py
+  python3 experiments/category-hyperplanes/validate_recovered_labels.py
+  python3 experiments/category-hyperplanes/build_url_inputs.py \
+    --source tests/fixtures/real_world_urls.txt \
+    --output /tmp/ib-recovered-inputs.tsv
+  test "$(wc -l < /tmp/ib-recovered-inputs.tsv)" = 220
 
   cd "$repository_root/src"
   "$idric_prefix/bin/idris2" Workbench.idric -o ib-workbench 2>&1 | tee /tmp/idric-workbench-compile.txt

docs/architecture.md

@@ -6,7 +6,7 @@
 
 A tab is not a renderer process. It is a persistent navigation thread that may currently have a renderer attached. A task may span several tabs, resources, searches, and actions.
 
-IB's present product target is a personal browser/workbench, not universal web compatibility. The substrate still supports multiple first-class frontends and renderer adapters so a broader browser can be built over it without owning or changing the stored model. See `docs/personal-workbench.md`.
+IB's present product target is a personal browser/workbench, not universal web compatibility. Its first two first-class frontends are a progressively augmented visual pre-paint and a ChatGPT-like text-only-by-default task workbench. The substrate still permits additional frontends and renderer adapters so a broader browser can be built over it without owning or changing the stored model. See `docs/personal-workbench.md`.
 
 ## Ownership
 
@@ -31,16 +31,16 @@ A frontend projects browser and task state and issues commands. It does not beco
 ## Main layers
 
 ```text
-page frontend       task workbench       inspector/commands
-       \                  |                    /
-        +--------- browser and task core -----+
+visual pre-paint     text task frontend      inspector/commands
+       \                    |                      /
+        +----------- browser and task core -------+
                   /           |          \
        persistent store   acquisition   renderer adapters
                            / extraction    |     |     |
                            HTTP, parsers  Servo WebView text/etc.
 ```
 
-The page frontend, text-first task workbench, inspector, information extractor, and text renderer are distinct roles. In particular, a text-oriented renderer is not the ChatGPT-like workbench frontend.
+The visual pre-paint frontend, text-first task frontend, inspector, information extractor, and text renderer are distinct roles. In particular, a text-oriented renderer is not the ChatGPT-like workbench frontend.
 
 The persistent store remains intelligible and useful without a rendering engine or language model installed.
 
@@ -139,7 +139,7 @@ IB is implemented in Idriç, with Grease for operating-system and process orches
 The current work does not promise:
 
 - universal web, MIME, renderer, or malformed-input compatibility;
-- one mandatory frontend;
+- one singular frontend that owns browser state;
 - faithful reproduction of interfaces irrelevant to the user's task;
 - preserving a JavaScript heap across renderer changes;
 - automatic understanding of every private application protocol;

docs/developer-workbench.md

@@ -100,6 +100,8 @@ Other required distinctions include:
 - a malformed or unavailable model cannot mutate canonical history or block browsing;
 - adding a category membership does not remove another membership;
 - removing a category from `_active` creates no negative training event.
+- a focus-priority hint may reorder safe prefetch work but creates no speculative tab or renderer;
+- the configured assistant receives only the explicitly scoped, inspectable task-context bundle.
 
 If RAM grows approximately with known-resource count, or rebuilding a derived view loses a human correction, the architecture has coupled state classes that must remain separate.
 
@@ -113,3 +115,6 @@ If RAM grows approximately with known-resource count, or rebuilding a derived vi
 6. Add operative-document-link and shared-child documentation fixtures.
 7. Add proposal, validation, correction, and reversible materialization fixtures.
 8. Continue live or recorded scientific-media fixtures through Grease.
+9. Add a GitLab-shaped seventeen-link fixture: changing visual focus reprioritizes safe links, creates zero speculative tabs or renderers, and supports a cited text answer.
+10. Add a multi-paper arXiv fixture: early per-paper summaries and one cross-paper answer require no renderer per paper.
+11. Add a mock video fixture: captions and playback position enter an authorized assistant context bundle without fetching video bytes or exposing secrets.