From ab8445818154fb9e7cf932d6deaac7eb27450408 Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Sun, 25 Oct 2026 01:05:54 +0200 Subject: [PATCH 1/2] docs(research): nsl-reproducible-baselines hypothesis and falsification gates --- .../07-nsl-parity-benchmark-plan.md | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 research/2026-10-25/07-nsl-parity-benchmark-plan.md diff --git a/research/2026-10-25/07-nsl-parity-benchmark-plan.md b/research/2026-10-25/07-nsl-parity-benchmark-plan.md new file mode 100644 index 0000000..5cb4bec --- /dev/null +++ b/research/2026-10-25/07-nsl-parity-benchmark-plan.md @@ -0,0 +1,31 @@ +# NSL cross-substrate benchmark: preregistered, independent comparison + +Status: RESEARCH PROPOSAL, unimplemented. + +## Why +A shared substrate claim must earn evidence for accuracy, speed, memory and interpretability separately. Avoid cherry-picking comparisons, implicit reliance on an unfrozen reference, or CI passes mistaken for clinical validation. + +## Reference tasks +1. NSL natural graph tutorial (Cora): graph-regularized classifier with explicit neighbor features. +2. NSL adversarial tutorial (MNIST): adversarial perturbation during training. +3. Synthetic perturbation/sphere toy: geometry and exact analytic oracle. +4. Tiny exported neural model: frozen ONNX/TFLite outputs, with per-op dtype and quantization constraints. +5. Duplicate-source provenance toy: expected refusal under repeated evidence. +Use public/synthetic data only; do not import patients or protected clinical examples. + +## Reproducibility contract +- Freeze repository revisions, Python/TensorFlow/NSL versions, test vectors and seeds; record hash manifests and training hyperparameters. +- Distinguish training accuracy and loss from inference byte/ULP parity. Floating training is not presumed deterministic across GPU/compiler variants. +- Pin CPU ISA (AVX2/AVX-512/NEON/WASM), Rust toolchain, memory limits, target board and thermal conditions; publish raw sample distributions. +- Pre-register baselines: unmodified NSL/TF, ONNX Runtime, TFLite Micro when applicable, CSR/COO graph path, dense or sparse DR. +- Do not include non-equivalent pipeline work in speed denominators without labeling it. +- Run failing negative controls: stale revision, invalid ordinal, duplicate causal evidence, unsupported ONNX op, adversarial vs causal confusion, incorrect rounding. +- Require an independent source/oracle for numerical parity (e.g. actual TFLite C++), not a copy of the implementation. + +## Metrics +Task accuracy/F1 or loss with uncertainty; output bit parity or max ULP; graph-loss deviation; expected calibration error; angular drift; wall-clock latency percentiles; peak allocated RAM; Flash/ROM footprint; energy if available; trace bytes; cold/warm focus and refusal rates. Include confidence intervals and multiple seeds. + +## Admission +Publish passing AND failing trials, environment manifest and exact commands. A failed candidate still counts as useful evidence; do not promote to 'supported' until gates are met. No clinical/AGI claim inferred from a synthetic benchmark. + +References: https://github.com/tensorflow/neural-structured-learning ; https://github.com/AdaWorldAPI/ndarray/pull/374 ; https://github.com/AdaWorldAPI/ndarray/pull/375 From 94edc3cbe982082b537b140948b2dc131617c1b6 Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Sun, 25 Oct 2026 01:18:33 +0200 Subject: [PATCH 2/2] docs(research): detailed experiment contract for Concrete oracle manifest and negative controls --- .../07a-oracle-manifest-and-measurements.md | 103 ++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 research/2026-10-25/07a-oracle-manifest-and-measurements.md diff --git a/research/2026-10-25/07a-oracle-manifest-and-measurements.md b/research/2026-10-25/07a-oracle-manifest-and-measurements.md new file mode 100644 index 0000000..5f32a1b --- /dev/null +++ b/research/2026-10-25/07a-oracle-manifest-and-measurements.md @@ -0,0 +1,103 @@ +# H7 / Reproducible comparison harness: oracle manifest v0 + +**Status:** draft preregistration; no tests run in this proposal. Companion to [PR #7](https://github.com/AdaWorldAPI/neural-structured-learning/pull/7). + +## Experimental truth hierarchy + +1. *Semantic oracle:* unmodified NSL behavior or independently defined mathematical result. +2. *Numerical oracle:* reference implementation / golden test vector generated without reusing our implementation code, e.g. TFLM C++ for #375. +3. *Performance baseline:* well-implemented alternative under the same input and hardware assumptions. +4. *Ablation:* intentionally disabled condition that **must fail** the corresponding check. +5. *Claim:* only the conjunction justified by evidence. Accuracy alone never demonstrates causal reasoning; CI green alone never demonstrates latency advantage. + +## Manifest, one record per experiment configuration + +~~~yaml +experiment_id: NSL-DR-001 +pr_id: 1 +hypothesis_revision: sha256:... +dataset_name: synthetic-directed-graph +dataset_sha256: ... +source_repos: + nsl: + lance_graph: + ndarray: + tensorflow: +references: + onnxruntime: + tflite_micro: +codegen: + rustc: + target_triple: + compile_flags: + simd_backend: + rounding_policy: +environment: + cpu_model: + cpu_flags: + os_kernel: + memory_limit_bytes: + board_name: + flash_limit_bytes: + ram_limit_bytes: +randomness: + seeds: [11, 23, 37, 41, 53] +measurements: + warmup_iterations: + samples_per_seed: + raw_csv_sha256: ... +oracle: + script_sha256: ... + output_sha256: ... + tolerance_policy: exact_bits | ulp_bound | bounded_error | statistical + confidence_interval_method: +negative_controls: + required: [] +~~~ + +The example fields are **placeholders**, not actual recorded commits, CPU or test measurements. Any test runner must reject unfilled required fields instead of publishing a success badge. + +## Dataset/task table + +| Task | Input | Primary comparison | Secondary | +| --- | --- | --- | --- | +| B01 | NSL Cora graph tutorial | final accuracy and graph-loss distribution over pinned seeds | neighbor retrieval loss, runtime | +| B02 | NSL MNIST adversarial tutorial | loss/robustness under *same configured* attack | gradient, projection, perturbation norms | +| B03 | deterministic 12-edge directed graph | packed NL_* features vs focused references exactly | retained bytes, gather latency | +| B04 | tiny f32 ONNX MatMul+Add+Tanh | op-by-op output and max ULP | graph import and inference latency | +| B05 | TFLM int8 requant fixtures | exact bytes with pinned rounding regime | cycles and code size | +| B06 | 256-codebook sphere relation | certified LUT code, angular error | flash/RAM/energy | +| B07 | DR³ 0.01/1/50/100% occupancy | relational query results vs triple-set oracle | memory and access p95 | +| B08 | duplicate-source provenance fixture | correct refusal / shared-source lineage | trace size and replay latency | +| B09 | analytic diffusion/spring perturbation | deviation vs exact/controlled numerical reference | conservation and long-horizon error | + +Cora/MNIST training accuracy depends on TensorFlow version, hardware, seed, dataset processing and nondeterministic kernels. Use distributional statistics and document non-determinism; do not require bit-parity for independent training runs. + +## Timing / memory protocol + +- All primary latency comparisons on the same physical device and power/thermal envelope. Predeclare whether allocation, import, serialization, data load and transfer overhead are included. +- Record *cold* and *warm* scenarios separately. No warmed focus cache on one arm and cold graph rebuild on another. +- Use repeated paired measurements with controlled random execution order, warmup, bootstrap intervals and raw samples. Report uncertainty and negative outcomes. +- Distinguish flash/ROM and RAM on MCU. Measure table residency and paging, not merely array bytes. +- If only simulation of a device is possible, label it "emulated," never claim hardware energy or hardware cycles. + +## Mandatory negative controls + +- Collapse duplicate directed edges by minimum, rather than maximum: graph oracle must fail. +- Reverse neighbor tie-break order: packed-graph oracle must fail. +- Bypass SourceId/index_seal validation: stale map must be refused by baseline check, and mutated version must be caught. +- Replace fused multiply-add by separate multiply+add on a known midpoint: bit parity gate must fail. +- Switch TFLM requantizer default double rounding to single rounding on a differentiating witness: parity gate must fail. +- Promote an adversarial correlation edge into a causal intervention: epistemic/causal test must refuse. +- Inject unsupported ONNX op and an unverified LUT rounding tie: compilation/load must refuse rather than fallback. +- Reuse same source evidence family twice: independent-evidence counter must stay one. + +## Result artifacts (future, if/when experiments actually run) + +- immutable manifest.yaml, commands.sh, golden vectors and input SHA list; +- tests.junit.xml or equivalent failure report; +- raw per-sample latencies and resident/flash byte counts; +- summary with confidence intervals, hardware fingerprint, known limitations; +- explicit *supported / bounded / refused / rejected* state per operator/experiment. + +A research conclusion may be "baseline wins." That's a useful result, not a failure of recordkeeping.