Observer gate: compile-time elision of observer bookkeeping (#915)

.github/workflows/ci.yml

@@ -418,7 +418,14 @@ jobs:
     name: asan + ubsan (full suite)
     needs: [dev-image, scope]
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    # 45, not 30: the green run before #915's [99u] section landed took 19m57,
+    # and that section legitimately adds minutes under ASan (it launches the
+    # sanitized binary ~60 times; sanitizer process startup dominates) — a slow
+    # runner then hit the old ceiling and the job rendered as CANCELLED at
+    # 30:02 with the suite mid-section. A timeout is not a verdict: the same
+    # tree's local ASan suite was 4117/4117. Headroom target ~2x the observed
+    # green duration, per the suite-runtime-baseline rule.
+    timeout-minutes: 45
     permissions:
       contents: read
       packages: read

CHANGELOG.md

@@ -6,6 +6,27 @@ All notable changes to EigenScript are documented here.
 
 ### Added
 
+- **The observer gate ships: programs that provably never read observer state
+  skip entropy bookkeeping entirely (#915).** Decided per compiled unit at
+  compile time (opcode scan + observer-builtin name scan + eager compilation of
+  string-literal `load_file` targets, so a clean module tree still gates), per
+  `EigsState` at run time. **8.51x on EigenMiniSat's 4x4 Tseitin workload**
+  (n=5 interleaved, one binary, solver counters identical across arms) — the
+  ungated observer walk was 88% of that workload's runtime. Byte-identical on a
+  416-program corpus against a pre-gate build. Conservative everywhere the
+  scan cannot be sure: computed load paths, aliasing, `eval`, `import`,
+  multithreaded compiles, and anything over the 1 MiB speculative-read budget
+  keep full observation; a module rewritten between scan and load raises
+  loudly rather than answering from a gap. `EIGS_OBS_FORCE=1` restores
+  pre-gate behaviour exactly; `EIGS_OBS_GATE_STATS=1` prints per-unit
+  verdicts. Verified by a nineteen-round adversarial loop plus the full CI
+  matrix; suite section [99u] pins 44 checks over the gate's mechanisms.
+  Residuals are filed, not hidden: #1031 (literal modules compile twice;
+  budget-bounded), #1027 (descriptor pre-call history), #1032/#1033 (minor).
+  The observer arming flags and trace-history flags are now relaxed-atomic —
+  fixing a data race reachable from worker threads (also present, unfixed,
+  in three pre-existing sites now ledgered on #1035/#1036).
+
 - **Every opcode now carries a recorded observer classification, and a gate
   fails on any that does not (#972).** The optimisation this unblocks — stop
   emitting observer bookkeeping when a program provably never reads observer

docs/OBSERVER.md

@@ -429,11 +429,16 @@ interrogation.
 `unobserved:` is the only opt-out, and it is a real one: it skips the emission,
 so a hot region inside it pays nothing.
 
-### The classification that a future opt-out rests on (#972)
+### The automatic opt-out — the observer gate (#915/#972)
 
-The obvious improvement is to skip the emission automatically when a program
-provably never reads observer state. That optimisation is not shipped, and the
-reason is worth stating: its failure mode is silent and total. If any opcode
+That improvement IS shipped: the runtime skips the emission automatically when a
+program provably never reads observer state. On a consumer that uses no observer
+features it is worth **8.5x** (EigenMiniSat 4x4 Tseitin, 293 s -> 34 s, n=5 per
+arm interleaved with the solver's counters identical). See "Using the gate"
+below for the controls.
+
+The reason the rest of this section is written so carefully is that the failure
+mode is silent and total. If any opcode
 that reads observer state is missing from the scan that decides "is anything
 reading?", a program gates its own bookkeeping off and then reads slots nobody
 updated — every binding answers `equilibrium` forever, with no crash and no
@@ -455,7 +460,7 @@ marker instead:
 
 | marker | meaning |
 |---|---|
-| `obs:READS` | answers **from** recorded observer/temporal state — the set the future liveness scan consumes |
+| `obs:READS` | answers **from** recorded observer/temporal state — the set the liveness scan consumes |
 | `obs:WRITES` | records, updates, resets or stamps that state |
 | `obs:DIAG` | reaches it only through the SIGUSR1 diagnostic dump, never through program-visible semantics |
 | `obs:NONE` | none of the above |
@@ -473,3 +478,81 @@ observes at `OP_OBSERVE_NAME_POST` after the SET), and the bare `OP_INTERROGATE`
 reads **no** observer state at all — `when` / `where` / `why` / `how` on a value
 operand return constants, because observer state is binding-keyed and a bare
 value has no binding.
+
+## Using the gate
+
+The gate is automatic and needs no source change. A program that never reads
+observer state pays nothing for it; a program that does is unaffected.
+
+| control | effect |
+|---|---|
+| `EIGS_OBS_FORCE=1` | force observer recording ON, whatever the scan decided. The escape hatch, and the baseline arm for any measurement — one byte-identical binary serves both arms. |
+| `EIGS_OBS_GATE_STATS=1` | print one `obs-gate: observed\|unobserved <unit>` line per compiled unit on stderr. |
+
+Both follow the tree's flag convention: any non-empty value that does not
+start with `0` turns the control on, so `=0` and `=` leave it off.
+
+### When the gate refuses instead of answering
+
+The gate decides at COMPILE time, and a few constructs can make that decision
+stale at RUN time. Where the runtime can prove the decision was wrong, it raises
+rather than answering — the recorded history of bindings already assigned cannot
+be reconstructed, so a late discovery is not recoverable and a quiet rest value
+would be a wrong answer with nothing to fail on.
+
+```
+load_file: 'x.eigs' reads observer state, but the observer gate was closed when
+this program's earlier assignments ran — they have no recorded history...
+```
+
+You will see this if a program **rewrites a module between the compile and the
+load**, or creates a file that **shadows** the one the compile-time scan
+resolved (resolution tries the cwd before the script directory), or `chdir`s so
+the same literal path resolves elsewhere. All three are the same shape: the file
+the gate inspected is not the file that ran.
+
+Re-run with `EIGS_OBS_FORCE=1` to disable the gate for that program. That is
+always safe — it restores the pre-gate behaviour exactly.
+
+### When the gate declines to look
+
+To decide before the program runs, the gate compiles literally-loaded modules
+itself — including ones reached only from a function that is never called, since
+a `load_file` inside an uncalled function still contributes to the answer. That
+work is speculative, so it is bounded: a per-thread cumulative ceiling on how
+many bytes the pass may read on the program's behalf, plus a rejection of
+anything that is not a regular file (a FIFO target once hung the compiler
+indefinitely, before `vm_execute`, with nothing printed).
+
+When the ceiling is spent the pass stops looking and the gate stays **open** —
+the conservative answer. Nothing is silently wrong; the program simply pays for
+observer bookkeeping it may not need. `EIGS_OBS_GATE_STATS=1` shows this as
+`observed` lines on a program you expected to gate closed.
+
+The ceiling is picked against the real population rather than chosen round: the
+largest transitive module tree in `lib/` is `ui.eigs` at 287 KiB across 19
+units, the next largest is 69 KiB, and the budget clears the largest by 3.5x.
+A suite check pins that — if a stdlib tree grows past the budget, the check
+fails and the number gets re-picked deliberately instead of the win quietly
+disappearing.
+
+Shared modules are charged once, not once per reference, and identity is the
+file itself rather than the path spelling — the same module reached through a
+relative path, an absolute one and a symlink is one charge, not three.
+
+### Known residual
+
+A chunk run through `vm_run_bytecode` or `sandbox_run` that reads observer state
+about a binding the HOST assigned before the call gets a rest value rather than
+the truth, silently. The descriptor's own work is recorded (both sites arm the
+observer before running, the twin of `chunk_arm_temporal`); only reads of state
+that predates the call are affected. Tracked separately with reproducers and two
+candidate fixes; `EIGS_OBS_FORCE=1` avoids it.
+
+Separately, every literally-loaded module is compiled **twice** — once by the
+gate to learn one bit, once for real by `load_file`, which has no module cache
+by design. Measured on `lib/ui.eigs`: 0.12-0.15s for the literal spelling that
+gates closed against 0.05-0.07s for a computed spelling that skips the pass, so
+a program that loads a large tree and does little work can pay more than it
+saves. The fix is to hand the eagerly-compiled chunk to `load_file` instead of
+discarding it; the budget above bounds the cost meanwhile.

src/arena.c

@@ -33,6 +33,10 @@
 #endif
 
 static void x_oom(size_t size) {
+    /* #915: the observer gate's eager pass may have stderr muted; a fatal
+     * message must not be discarded because a module was loaded by one spelling
+     * rather than another. No-op when nothing is muted. */
+    eigs_obs_unmute_for_fatal();
     fprintf(stderr, "eigenscript: out of memory (requested %zu bytes)\n", size);
     abort();
 }

src/builtins.c

@@ -3182,12 +3182,51 @@ static EigsChunk *vm_build_chunk_desc(Value *desc, int off, int sandbox_mode) {
  * output runs through — reusing the bytecode VM and its JIT. The caller is
  * responsible for a well-formed chunk ending in OP_RETURN, stamped with the
  * bytecode ABI revision it was built against (#704). */
+/* #915: a descriptor chunk never passes through compile_ast, so the observer
+ * gate's compile-time scan never saw it. Two things follow, and only one of
+ * them is solved here.
+ *
+ * SOLVED — the descriptor's OWN work. eigs_obs_enable() arms recording before
+ * vm_execute, the observer twin of chunk_arm_temporal below (#831: "a
+ * descriptor must turn recording ON itself"). It also records the history gap,
+ * so a mid-run arming cannot disarm the load_file guard — a benign descriptor
+ * call used to do exactly that, silently, for the rest of the process.
+ *
+ * NOT SOLVED, and filed rather than half-guarded — see the issue referenced in
+ * docs: a descriptor that READS observer state about a host binding assigned
+ * before the call gets a rest value, because that history was never recorded.
+ * Three static guards were tried and each traded one wrong answer for another:
+ * "reads at all" broke 57 bridge assertions; "reads a NAME operand" missed the
+ * slot form; "reads a NAME or an in-range assigned slot, or the thread alias"
+ * fires on tests/test_vm_run_bytecode.eigs's own #737 fixture, whose operand
+ * bytes are load-bearing (1,1 == two OP_NULLs, chosen so a drifted operand walk
+ * stays synced) and whose reader is JUMPED OVER. Distinguishing that from a
+ * real host read needs reachability analysis over caller-supplied bytecode,
+ * which is its own change with its own review. Shipping a guard that breaks a
+ * legitimate fixture, or a fourth variant tuned until the suite passes, would
+ * both be worse than a stated residual. */
+
 Value* builtin_vm_run_bytecode(Value *arg) {
     char abibuf[256];
     const char *abi_err = vm_desc_abi_error(arg, abibuf, sizeof abibuf);
     if (abi_err) { rt_error(EK_VALUE, 0, "%s", abi_err); return make_null(); }
     EigsChunk *chunk = vm_build_chunk_desc(arg, 1, 0);
     if (!chunk) return make_null();
+    /* #915: ARM the observer for what this descriptor itself does, exactly as
+     * chunk_arm_temporal two lines below arms the temporal channel (#831: "a
+     * descriptor must turn recording ON itself" — nothing scanned this chunk).
+     *
+     * The guard above and this line answer DIFFERENT questions and an earlier
+     * revision wrongly swapped one for the other: the guard covers host
+     * bindings assigned BEFORE the call, whose history is unrecoverable; this
+     * covers everything the descriptor writes and reads AFTER it. Deleting this
+     * made a descriptor that writes a geometric series into its own frame slot
+     * and reads it back answer `equilibrium` — a regression a blind critic
+     * bisected to the commit that removed it. Both are needed.
+     *
+     * Through eigs_obs_enable, not a bare assignment: this flip happens mid-
+     * execution, so it must also record that earlier bindings have no history. */
+    eigs_obs_enable();
     /* #831: the compiler's temporal scan is what turns history recording on,
      * and it never saw this chunk — arm from the verified bytecode instead,
      * or the chunk's own `prev of` / `at` reads answer null whenever the
@@ -3350,6 +3389,11 @@ static int sandbox_value_has_callable(Value *v, int depth, long *budget,
  * are caught (not propagated). Returns {"ok": 1/0, "result": value} — the graded
  * "does it run?" rung for a self-hosted compiler validating generated code. */
 Value* builtin_sandbox_run(Value *arg) {
+    /* #915: same descriptor hazard as vm_run_bytecode. Unexploitable TODAY only
+     * because the sandbox env is a sealed root (parent == NULL), so a descriptor
+     * cannot reach a host binding's slot — that is the sandbox's defence, not
+     * the gate's, and it evaporates the day sealing is relaxed. The guard runs
+     * below, once the chunk exists. */
     Value *desc = (arg && arg->type == VAL_LIST && arg->data.list.count >= 1)
                   ? arg->data.list.items[0] : arg;
     int max_iter = 1000000;
@@ -3379,6 +3423,7 @@ Value* builtin_sandbox_run(Value *arg) {
     char abibuf[256];
     const char *abi_err = vm_desc_abi_error(desc, abibuf, sizeof abibuf);
     EigsChunk *chunk = abi_err ? NULL : vm_build_chunk_desc(desc, 1, 1);
+    if (chunk) eigs_obs_enable();     /* #915: see vm_run_bytecode */
     Value *out = make_dict(2);
     if (!chunk) {
         /* Descriptor verification may already have interned constants before
@@ -3636,7 +3681,7 @@ Value* builtin_record_history(Value *arg) {
     int on = (arg->data.num != 0.0) ? 1 : 0;
     /* #827: no name to narrow on — a self-hosted compiler calling this is
      * standing in for the whole-program arming, so it gets the wildcard. */
-    if (on) { trace_arm_history_all(); g_trace_obs_hist = 1; }
+    if (on) { trace_arm_history_all(); trace_flag_store(g_trace_obs_hist_storage, 1); }
     else    trace_history_disable();
     return make_num((double)prev);
 }

src/builtins_host.c

@@ -1034,8 +1034,57 @@ Value* builtin_load_file(Value *arg) {
     Env *target = g_load_env ? g_load_env : g_global_env;
     int saved_boundary = g_compile_module_boundary;
     g_compile_module_boundary = 1;                       /* #373 */
+    /* #915: the observer gate may have CLOSED on evidence gathered when this
+     * unit's parent was compiled — the eager pre-pass resolved this literal
+     * target and compiled it then. The file it read and the file being compiled
+     * now are two separate reads with the whole program running in between, so
+     * they can differ: the program can rewrite the module (`write_text` then
+     * `load_file`), or create a file in the cwd that SHADOWS the one the
+     * pre-pass resolved (resolve_eigenscript_file tries cwd before the script
+     * dir). Both were executed and both produced a silently wrong answer —
+     * `report of x` read `equilibrium` under the gate and `moving` without it.
+     *
+     * ASK THE ACTUAL QUESTION. A first draft compared the observer bit before
+     * and after the module's compile and raised on a 0 -> 1 transition. Two
+     * blind-critic repros killed it:
+     *
+     *   - ONE-SHOT. The bit is monotonic, so that first transition leaves it at
+     *     1 and every later load saw "already open" and skipped the check. The
+     *     error is catchable, so a single `try:` around the first load disarmed
+     *     the guard for the rest of the run and restored the exact silent-wrong
+     *     answer the guard was written to stop.
+     *   - OVER-BROAD. The bit flips for any of the eager pass's SIX conservative
+     *     bail-outs, not just for staleness. One of them (`L.count > 0 &&
+     *     g_vm_multithreaded`) is reachable at run time but not at the parent's
+     *     compile time, so `spawn` + a module that itself loads a module became
+     *     a hard error with every clause of the message false. That shape is
+     *     shipped: lib/io.eigs does `load_file of "lib/string.eigs"`.
+     *
+     * So the predicate is the module's OWN verdict — does this chunk read
+     * observer state? — and the precondition is "this program has bindings with
+     * no recorded history", which is sticky rather than derived from a bit that
+     * the detection itself changes. */
+    /* ACQUIRE: this is the one read that pairs with eigs_obs_enable's
+     * store ORDER (gap then needed) — see obs_flag_store in eigenscript.h. */
+    int obs_before_module = obs_flag_load_acquire(obs_needed);
     EigsChunk *lf_chunk = compile_ast(ast, target, source);
     g_compile_module_boundary = saved_boundary;
+    if (lf_chunk && chunk_reads_observer(lf_chunk) &&
+        (!obs_before_module || g_obs_history_gap)) {
+        obs_flag_store(obs_history_gap, 1);
+        g_parse_errors = saved_errors;
+        chunk_free(lf_chunk);
+        free_ast(ast);
+        free_tokenlist(&tl);
+        free(source);
+        rt_error(EK_IO, 0,
+            "load_file: '%s' reads observer state, but the observer gate was "
+            "closed when this program's earlier assignments ran — they have no "
+            "recorded history, so an observer query about them would answer a "
+            "rest value rather than the truth. Re-run with EIGS_OBS_FORCE=1.",
+            arg->data.str);
+        return make_null();
+    }
     if (g_parse_errors > 0) {
         g_parse_errors = saved_errors;
         chunk_free(lf_chunk);