dln_c_loader_interpose: all-series patches + the spec-22 acceptance harness — DRAFT, chain-gated

ci/include/tebako/fs/c_api.h

@@ -85,6 +85,7 @@ int tebako_get_errno(void);
 
 char* tebako_fs_dlmap2file(const char* path);
 char* tebako_fs_mounts(void);
+char* tebako_fs_mount_of(const char* path);
 
 #ifdef __cplusplus
 }

ci/spec22/README.md

@@ -0,0 +1,96 @@
+# ci/spec22 — spec 22 phase 1 acceptance harness (loader interposition, POSIX)
+
+This directory is the **local, reproducible** acceptance proof for the
+`dln_c_loader_interpose` patch family (`patches/*/dln_c_loader_interpose.patch`):
+spec 22 class L — the process's `dlopen`/`dlerror` are interposed inside the
+runtime binary, so any native-library load of a VFS-resident path is
+materialized (with its dependency closure) to the exec cache and loaded for
+real, from any caller in the process. No per-gem Ruby adapter is present in
+the runtime under test.
+
+It is deliberately **not** wired into CI: it needs the three repos on one
+machine plus a compiled runtime, which the patch-level gates (lint,
+compile-smoke) intentionally do not build. Run it by hand; it is idempotent
+and fully deterministic from pinned inputs.
+
+## What it proves (fixtures/probe.rb, jailed)
+
+| check | assertion |
+|---|---|
+| `fiddle` | `Fiddle.dlopen("/probe/lib/libvfsprobe.<ext>")` — a VFS path — loads and its function returns 42 (dependency `libvfsdep` resolved through the closure walk) |
+| `cext-self-dlopen` | a hand-rolled C extension calls its **own** `dlopen` on the VFS path from `Init` (bypassing `dln_load`) and returns 42 |
+| `named-error` | `Fiddle.dlopen("/probe/lib")` (a directory) raises `Fiddle::DLError` whose message is the tebako verdict line: `cannot materialize VFS-resident library '/probe/lib' (mount '/'): …` |
+| `jail-deny` | `File.read("/etc/passwd")` is denied under `TEBAKO_JAIL=deny;<scratch>:<scratch>:rw` — interposition did not punch through the jail |
+
+The probe exits 0 only when all four print `ok`.
+
+## Inputs and pins
+
+- **This repo** (any checkout of tamatebako/ruby with the patch wired into
+  `patches/*/patch-*.yaml`) — supplies the patched source tree via
+  `tools/apply` + a sha256-pinned mirror, never the release channel.
+- **tamatebako/tebako** at a commit carrying `tebako_fs_mount_of`
+  (`crates/tfs`), staged with `tools/stage_link_unit` (env per
+  AGENTS.md §13: `DWARFS_RS_VCPKG_ROOT`/`VCPKG_ROOT`/`SQFS_SYS_VCPKG_ROOT`,
+  `CARGO_NET_GIT_FETCH_WITH_CLI=true`). Default: `../../tebako-wt-spec22-mountof`
+  (override `TEBAKO_REPO`).
+- **tamatebako/tebako-runtime-ruby** (the runtime factory), run from a
+  detached git worktree at `origin/main` — its checkout is never mutated
+  and its source pin is never bumped: `tools/build_runtime
+  --src-mirror file://… --src-release spec22-local-…` consumes the local
+  mirror.
+- **The adapter-less tebako-runtime gem** in a local gem repo (default
+  `/tmp/tebako-gem-repo`, override `GEM_REPO_DIR`). Build it from the
+  spec-22 deletion branch of tamatebako/tebako-runtime:
+
+  ```sh
+  git worktree add ../tebako-runtime-wt-spec22-drop-ff -b feat/drop-class-l-adapters origin/main
+  cd ../tebako-runtime-wt-spec22-drop-ff
+  # ffi/fiddle entries removed from POST_REQUIRE_MAP + adapters deleted (the branch)
+  gem build tebako-runtime.gemspec   # with VERSION temporarily "0.8.2.local"
+  mkdir -p /tmp/tebako-gem-repo/gems && cp tebako-runtime-0.8.2.local.gem /tmp/tebako-gem-repo/gems/
+  gem generate_index --directory /tmp/tebako-gem-repo
+  ```
+
+  The harness points `GEMRC` at that repo, so the factory's
+  `gem install tebako-runtime` resolves the adapter-less build — without
+  publishing anything and without touching `~/.gemrc`.
+
+## Run
+
+```sh
+export DWARFS_RS_VCPKG_ROOT=/path/to/vcpkg VCPKG_ROOT=/path/to/vcpkg \
+       SQFS_SYS_VCPKG_ROOT=/path/to/vcpkg CARGO_NET_GIT_FETCH_WITH_CLI=true
+ci/spec22/run.sh 4.0.6        # any version in versions.yml
+```
+
+Verdict: `SPEC22-ACCEPTANCE-OK <version>` on success; a named `FAIL
+spec22 (…)` line otherwise. Everything transient is under
+`/tmp/spec22-scratch-<version>` (`SCRATCH` to override); delete it to
+rebuild from scratch, or delete a single artifact to rebuild just that
+stage (the link unit is reused across runs by design).
+
+Platform notes: on macOS dyld honors `__DATA,__interpose` tuples only
+from dylib images (never from the main executable — verified
+empirically), so the macOS interposition is the DRIVER's self-insertion
+at boot (crates/tebako-driver's embedded interpose dylib +
+`DYLD_INSERT_LIBRARIES` + a once-only re-exec); ELF (linux-gnu/musl)
+uses the main binary's own `dlopen`/`dlerror` definitions from this
+patch, which preempt process-wide (verify on ELF with
+`nm -D <runtime-exe> | grep ' T dlopen'`). The probe is POSIX-only;
+windows is out of phase-1 scope.
+
+## The ELF leg (linux/amd64, containerized)
+
+`elf/` runs the same four assertions on linux/amd64 (qemu-user on Apple
+Silicon, or any Docker host) — one container, one command:
+
+```sh
+ci/spec22/elf/run-elf-leg.sh
+```
+
+plus the ELF-only gates (`nm -D` export of `dlopen`/`dlerror`; zero
+GNU_UNIQUE in the staged `libtfs.a`). See `elf/README.md` for the layout,
+the container-local deviations (the libjemalloc neuter, the reseal), and
+the reseal procedure. Proven green on ruby 4.0.6 (linux-gnu x86_64):
+`SPEC22-ACCEPTANCE-OK 4.0.6`.

ci/spec22/elf/README.md

@@ -0,0 +1,96 @@
+# ci/spec22/elf — the ELF (linux/amd64) acceptance leg, containerized
+
+The linux/amd64 twin of `../run.sh` (macOS): same four jailed assertions
+(`fiddle`, `cext-self-dlopen`, `named-error`, `jail-deny`) plus two ELF-only
+gates:
+
+- **GNU_UNIQUE gate** — the staged `libtfs.a` carries zero `STB_GNU_UNIQUE`
+  definitions (gcc's libstdc++ emits inline-template statics as GNU_UNIQUE
+  inside the vendored rnp closure; they break the static-archive model and
+  mis-link under qemu-user). See *Reseal* below when the gate fires.
+- **nm -D export gate** — the runtime exe dynamically exports `dlopen` and
+  `dlerror`. ruby compiles `dln.c` with `-fvisibility=hidden`; the patch's
+  definitions carry `__attribute__((visibility("default")))` so they can
+  preempt process-wide (`-Wl,-export-dynamic` cannot export a hidden
+  symbol — the pre-fix builds failed here with exit 66).
+
+Everything runs inside one container (`linux/amd64` under Rosetta/qemu-user
+on Apple Silicon); the host only runs `run-elf-leg.sh`.
+
+## Layout (populate once)
+
+```
+/tmp/spec22-linux-scratch/            # $SCRATCH, bind-mounted
+  ruby/                               # THIS repo checkout — the harness rides it
+  factory/                            # tamatebako/tebako-runtime-ruby
+                                      #   @feat/boot-smoke-loader-interpose
+  tebako-runtime/                     # tamatebako/tebako-runtime
+                                      #   @feat/drop-class-l-adapters (gem 0.8.2)
+  ws/tebako-rs/                       # tamatebako/tebako @feat/tfs-mount-of
+  ws/dwarfs-rs/                       # tamatebako/dwarfs-rs (+ dwarfs-t submodule)
+  ws/limnifs/limnifs/                 # limnifs/limnifs (tebako-rs contract-tests
+                                      #   sibling path dep; cargo metadata fails
+                                      #   without it)
+/tmp/spec22-link-unit-linux/          # $LINK_UNIT, bind-mounted (empty dir)
+```
+
+Paths are overridable (`SCRATCH`, `LINK_UNIT`, `NAME`, `IMAGE` in the env);
+the defaults match this document. The ruby checkout MUST live at
+`$SCRATCH/ruby` — the driver refuses to run from anywhere else (the scripts
+have to be visible inside the container through the bind mount).
+
+## Run
+
+```sh
+ci/spec22/elf/run-elf-leg.sh
+```
+
+Steps: container up → `setup-toolchain.sh` (once; cmake 3.31, clang-19,
+gcc-11, rustup, vcpkg, the libjemalloc neuter below) → `roll-source.sh`
+(rolls `tfs-ruby-4.0.6-src.tar.gz` from this checkout into
+`$SCRATCH/mirror`, guarding the interpose block landed) →
+`stage-gem-repo.sh` (builds the adapter-less gem into a file:// repo +
+`gemrc`) → `build-link-unit.sh` (cargo + `tools/stage_link_unit`, nm
+evidence, preload-deps gate, GNU_UNIQUE gate) → `build-runtime.sh`
+(factory `tools/build_runtime` with `--src-mirror` + `GEMRC`, then the
+nm -D export gate) → `probe.sh` (builds the probe natives from
+`../fixtures/`, packs the payload with the staged `tfs` CLI, runs the exe
+jailed: `TEBAKO_JAIL="deny;$SCRATCH:$SCRATCH:rw"`).
+
+Success prints `SPEC22-ACCEPTANCE-OK 4.0.6` and `RUN-ELF-LEG-OK`.
+
+## Container-local deviations (documented, not portable)
+
+1. **libjemalloc neuter** (`setup-toolchain.sh`, idempotent): the ruby
+   build's `-ljemalloc` probe resolves the distro's static
+   `/lib/x86_64-linux-gnu/libjemalloc.a` via the `-l:` fallback and links
+   it into miniruby — under qemu-user that combination hangs the build.
+   The setup replaces the distro archive with an empty one
+   (`libjemalloc.a.real` rides alongside as the backup). Container fs
+   only; never run this on a host.
+
+2. **Reseal** (manual, gated): when the GNU_UNIQUE gate fires, the flagged
+   members (rnp/sexp objects carrying `std::__detail::__to_chars_10_impl::
+   __digits` / `std::_Sp_make_shared_tag::_S_ti()::__tag` as `u` in `nm`)
+   must be rebuilt with clang-19 (which emits them hidden, not GNU_UNIQUE)
+   and resealed:
+
+   ```sh
+   # inside the container: rebuild the flagged TUs with clang-19, then
+   nm /tmp/spec22-link-unit-linux/libtfs.a | grep -B1 ' u '   # the members
+   cp /tmp/spec22-link-unit-linux/libtfs.a{,.pre-reseal-backup}
+   python3 reseal.py /tmp/spec22-link-unit-linux/libtfs.a{,.resealed} <fresh .o ...>
+   mv /tmp/spec22-link-unit-linux/libtfs.a{.resealed,}
+   ```
+
+   `reseal.py` = `swap-members.py` (prefix-rename exactly as tebako-arscope
+   would emit) + `ar-rebuild.py` (positional extract/rebuild — GNU ar 2.34
+   mangles this writer's archive on in-place rewrite; llvm-ar rebuilds
+   cleanly). The durable fix is upstream (arscope must neutralize
+   GNU_UNIQUE, or the rnp closure builds with clang); the gate keeps the
+   manual step honest meanwhile.
+
+3. **qemu-user clock**: the container's clock can lag the host by hours;
+   artifact mtimes (cache keys, `make` decisions) follow the CONTAINER
+   clock. Nothing in the harness compares mtimes across the boundary —
+   keep it that way.

ci/spec22/elf/ar-rebuild.py

@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+"""ar-rebuild.py — positional extract/rebuild for an ar archive that GNU
+ar's rewriter mangles. Extracts every member to a directory with
+position-unique names, then rebuilds a fresh GNU archive in the ORIGINAL
+member order. Members named in <replace-dir> (by original member name)
+are swapped for the replacement file contents.
+
+Usage: ar-rebuild.py <in.a> <out.a> [replace-dir]"""
+
+import os
+import shutil
+import struct
+import subprocess
+import sys
+
+
+def parse(data):
+    assert data[:8] == b"!<arch>\n", "not an ar archive"
+    members = []  # (name, bytes)
+    strtab = b""
+    off = 8
+    while off < len(data):
+        raw_name = data[off:off + 16].decode("ascii", "replace")
+        size = int(data[off + 48:off + 58].decode("ascii").strip())
+        body_off = off + 60
+        body = data[body_off:body_off + size]
+        off = body_off + size + (size & 1)
+        field = raw_name.strip()
+        if field == "/":
+            continue  # symbol table — regenerated by ranlib
+        if field == "//":
+            strtab = body
+            continue
+        if field.startswith("/") and field[1:].isdigit():
+            start = int(field[1:])
+            end = strtab.index(b"\n", start)
+            name = strtab[start:end].decode("ascii", "replace").rstrip("/")
+        else:
+            name = field.rstrip("/").strip()
+        members.append((name, body))
+    return members
+
+
+def main():
+    src, dst = sys.argv[1], sys.argv[2]
+    replace_dir = sys.argv[3] if len(sys.argv) > 3 else None
+    with open(src, "rb") as f:
+        members = parse(f.read())
+
+    tmp = "/tmp/ar-rebuild-work"
+    shutil.rmtree(tmp, ignore_errors=True)
+    os.makedirs(tmp)
+    seen = {}
+    files = []
+    replaced = []
+    for i, (name, body) in enumerate(members):
+        seen[name] = seen.get(name, 0) + 1
+        fname = f"{i:05d}-{name.replace('/', '_')}"
+        path = os.path.join(tmp, fname)
+        if replace_dir and os.path.exists(os.path.join(replace_dir, name)):
+            with open(os.path.join(replace_dir, name), "rb") as f:
+                body = f.read()
+            replaced.append(name)
+        with open(path, "wb") as f:
+            f.write(body)
+        files.append(path)
+
+    if os.path.exists(dst):
+        os.remove(dst)
+    # GNU ar keys members by basename; the position-prefixed names would
+    # leak into the member table, so archive through basename-preserved
+    # hardlinks in per-member dirs is overkill — GNU ar stores the name
+    # as given on the command line (basename). Use a manifest pass.
+    subprocess.run(["llvm-ar-18", "rcs", dst] + files, check=True)
+    return members, replaced, dst
+
+
+if __name__ == "__main__":
+    members, replaced, dst = main()
+    print(f"members: {len(members)}, replaced: {replaced}")
+    out = subprocess.run(["llvm-ar-18", "t", dst], capture_output=True, text=True).stdout.splitlines()
+    print(f"out members: {len(out)}")

ci/spec22/elf/build-link-unit.sh

@@ -0,0 +1,71 @@
+#!/bin/bash
+# build-link-unit.sh — the linux-gnu v2 link unit + tfs CLI, inside the
+# toolchain container. Mirrors ci/gnu-floor-build.sh's env and ordering:
+# serialized sqfs pre-install first (sqfs-sys's build.rs would otherwise
+# race dwarfs-t-sys's vcpkg run on the root lock), then the cargo build,
+# then tools/stage_link_unit --skip-build into $LINK_UNIT.
+set -euo pipefail
+. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/env.sh"
+cd "$SCRATCH/ws/tebako-rs"
+
+echo "== vcpkg pre-install squashfs-tools-ng ($TRIPLET) =="
+if [ ! -f "$SQFS_SYS_VCPKG_INSTALLED_DIR/lib/libsquashfs.a" ]; then
+  "$VCPKG_ROOT/vcpkg" install \
+    --vcpkg-root "$VCPKG_ROOT" \
+    --x-wait-for-lock \
+    --x-manifest-root crates/sqfs-sys \
+    --x-install-root /sqfs-installed \
+    --triplet "$TRIPLET" \
+    --overlay-triplets crates/sqfs-sys/vcpkg_triplets \
+    --overlay-ports crates/sqfs-sys/vcpkg_ports
+fi
+
+echo "== cargo build --release ($TARGET): tfs, tebako-driver, libtfs-preload, tfs-cli =="
+cargo build --release --target "$TARGET" -p tfs -p tebako-driver -p libtfs-preload -p tfs-cli
+
+echo "== bridge the pre-installed sqfs tree into the sqfs-sys out dir =="
+# (link-unit-stage.sh's bridge: the harvest globs <sqfs-out>/vcpkg_installed/
+# <triplet>/lib/libsquashfs.a, which a pre-install leaves empty)
+sqfs_out=$(ls -dt "/cargo-target/$TARGET/release/build"/sqfs-sys-*/out | head -1)
+trip=$(basename "$SQFS_SYS_VCPKG_INSTALLED_DIR")
+if [ ! -e "$sqfs_out/vcpkg_installed/$trip/lib" ]; then
+  mkdir -p "$sqfs_out/vcpkg_installed"
+  ln -sfn "$SQFS_SYS_VCPKG_INSTALLED_DIR" "$sqfs_out/vcpkg_installed/$trip"
+  echo "bridged $SQFS_SYS_VCPKG_INSTALLED_DIR -> $sqfs_out/vcpkg_installed/$trip"
+fi
+
+echo "== stage the link unit =="
+ruby tools/stage_link_unit $LINK_UNIT --target "$TARGET" --skip-build
+
+echo "== nm evidence =="
+for sym in tebako_fs_mount_of tebako_fs_dlmap2file tebako_path_is_embedded; do
+  hit_tfs=$(nm $LINK_UNIT/libtfs.a 2>/dev/null | grep -c " T $sym\$" || true)
+  hit_drv=$(nm $LINK_UNIT/libtebako_driver.a 2>/dev/null | grep -c " T $sym\$" || true)
+  echo "  $sym: libtfs.a T-defs=$hit_tfs libtebako_driver.a T-defs=$hit_drv"
+done
+echo "== preload cdylib dynamic deps (must be glibc-only) =="
+ls -la $LINK_UNIT/
+NEEDED=$(readelf -d $LINK_UNIT/libtfs_preload.so | grep NEEDED || true)
+echo "$NEEDED"
+echo "$NEEDED" | grep -E 'libstdc\+\+|libgcc_s' && { echo "FAIL: preload NEEDs the C++ runtime chain"; exit 65; } || true
+
+echo "== GNU_UNIQUE gate (libtfs.a must carry none) =="
+# gcc's libstdc++ emits inline-template statics (std::__detail::__to_chars_
+# _10_impl::__digits, _Sp_make_shared_tag::__tag) as STB_GNU_UNIQUE inside
+# the vendored rnp closure — process-singleton semantics that break the
+# static-archive model (and mis-link under qemu-user). tebako-arscope's
+# prefixing does not rewrite them away (upstream fix owed); the acceptance
+# gate is that the STAGED archive carries zero. clang-built members do not
+# emit them, so the container-local remedy is to rebuild the flagged
+# members with clang-19 and reseal (ci/spec22/elf/reseal.py — see the
+# README's reseal section). Backup the pre-reseal archive first.
+UNIQUE=$(nm $LINK_UNIT/libtfs.a 2>/dev/null | grep -cE ' u [a-zA-Z_]' || true)
+echo "  GNU_UNIQUE defs in libtfs.a: $UNIQUE"
+[ "$UNIQUE" -eq 0 ] || {
+  echo "GATE-FAIL: libtfs.a carries $UNIQUE GNU_UNIQUE definition(s) —"
+  echo "rebuild the flagged members with clang-19 and reseal per"
+  echo "ci/spec22/elf/README.md (reseal section), then re-run."
+  exit 65
+}
+
+echo "BUILD-LINK-UNIT-OK"