diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index ca5d6e2..15cea45 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,4 +1,4 @@ -# SAPIENS Phase-0 Architecture +# SAPIENS Architecture (Phases 0–4) ## Design goals @@ -13,7 +13,18 @@ ```text src/sapiens/ models.py immutable candidate/evidence/manifest models - adapter.py DomainAdapter protocol and Phase-0 adapter validation + adapter.py DomainAdapter protocol; validation routes to the registry + registry.py trust-tiered adapter registry (Phase 1) + permissions.py owner-permission/licence manifest for third-party code (Phase 1) + isolation.py subprocess + rlimit execution for UNTRUSTED adapters (Phase 1) + checkpoints.py signed ledger checkpoints + external anchor export (Phase 1) + validation.py L1/L2 automated gates + holdout protocols + leakage controls (Phase 2) + fixtures.py seeded-bias fixture suite with labelled outcomes (Phase 2) + calibration.py gate-performance calibration reports (Phase 2) + confidence.py calibration-gated confidence aggregation (Phase 2) + review.py L3 panel protocol: roles, objections, rounds, gates (Phase 3) + reviewers.py deterministic reference reviewers, four roles (Phase 3) + catchrate.py panel catch-rate scoring over seeded fixtures (Phase 3) ledger.py JSONL hash-chain ledger and L0→L4 transition guard kernel.py domain-neutral candidate registration and next-gate validation bridge.py cross-domain structure transfer with mandatory L0 reset @@ -21,7 +32,9 @@ src/sapiens/ queue.py bounded SQLite job queue with leases/idempotency daemon.py bounded background worker skeleton cli.py synthetic-only demo - adapters/ two deterministic synthetic adapters + adapters/ synthetic adapters + Phase-4 real-data Kepler adapter + (published-signal re-derivation; validators sandboxed + behind the adapter contract) ``` Dependency rule: `sapiens` core modules do not import `sapiens.adapters`; tests enforce this. Adapters import core contracts. @@ -32,7 +45,9 @@ Adapters may propose candidates and produce evidence, but they cannot set eviden Required methods: -- `manifest`: domain name/version/vocabulary and `synthetic_only=True` for Phase 0. +- `manifest`: domain name/version/vocabulary plus provenance facts + (`synthetic_only`, `code_origin`, `data_sources`, `third_party_source`). + The Phase-1 registry derives a trust tier from these facts. - `propose(seed, limit)`: deterministic candidates. - `validate(candidate, stage, seed, context)`: bounded evidence for `internal`, `replication`, or `review`. - `import_structure(structure, candidate_id)`: target-domain candidate from cross-domain structure. @@ -41,7 +56,19 @@ Required methods: The ledger is newline-delimited canonical JSON. Each event stores the previous event hash and its own hash. Replay validates sequence, hashes, candidate creation, evidence scope, one-step promotion, required evidence kinds, demotion reasons, and the L4 human gate. -Hash chaining detects tampering but does **not** prove authorship, signatures, scientific truth, or external timestamping. +Hash chaining detects tampering but does **not** prove authorship, scientific truth, or external timestamping. Phase-1 `checkpoint` events summarise the chain (event count + head hash) and may carry an HMAC-SHA256 signature (environment-held key, never stored); `sapiens.checkpoints` also exports/verifies external anchor files. HMAC is symmetric: it proves key possession, not third-party authorship. + +## Validation gates (Phase 2) + +`DiscoveryKernel(validation=ValidationGates(...))` opts into automated L1/L2 gates. L1 runs statistical sanity checks over a candidate's internal evidence (determinism across identical reruns, degenerate constant scores, score presence). L2 requires a declared `HoldoutProtocol` for the domain and enforces holdout discipline: replication evidence must come from declared holdout datasets, dataset collisions and (dataset, seed) reuse across the boundary are leakage and reject the gate, and a minimum pass fraction applies. Gate verdicts are appended to `kernel.gate_log` (inspectable, recomputable) — the kernel never fabricates gate outcomes as ledger evidence. Gates are pure functions in `sapiens.validation`; `sapiens.fixtures` ships a labelled seeded-bias suite; `sapiens.calibration` scores gates against it; `sapiens.confidence` refuses to aggregate confidence without the resulting report. + +## Real domain adapters (Phase 4) + +`KeplerPhotometryAdapter` is the first non-synthetic adapter: first-party clean-room code over a bundled, checksum-pinned public NASA/MAST Kepler Q1 light curve (CORE trust tier, no permission entry needed, in-process). Its validators are staged behind the contract: internal = fold SNR on the full curve; replication = independent time-halves must each re-detect at the same period; review = adversarial odd/even, secondary-eclipse and harmonic checks. All transit arithmetic lives in `adapters/_transit.py`; the kernel sees only evidence. The adapter re-derives a published signal as validation — it claims no discovery — and ships negative controls (flat, tampered, shifted-period curves). A declared `kepler_holdout_protocol()` wires it into the Phase-2 gates, and the reference panel reviews it in Phase-3 integration tests. + +## L3 review panels (Phase 3) + +`DiscoveryKernel(panel=ReviewPanel(...))` gates L3 promotion on a structured panel. Reviewers are pure deterministic functions in four roles (statistician, domain theorist, methodologist, devil's advocate). The panel convenes bounded rounds; objections carry severity (MINOR/MAJOR/BLOCKING) and a tracked lifecycle (raised/sustained/withdrawn); reference reviewers escalate re-affirmed MAJOR findings to BLOCKING. Approval requires no sustained MAJOR/BLOCKING objection; MINOR caveats are recorded but non-fatal. The panel's verdict is recorded in the ledger as review evidence (`panel-transcript` dataset) — approval adds it to the promotion refs, rejection leaves the candidate at L2 with the rejection on record. `sapiens.catchrate` scores panels against the seeded fixture suite. ## Cross-domain bridge @@ -51,4 +78,4 @@ Hash chaining detects tampering but does **not** prove authorship, signatures, s `WorkQueue` gives bounded jobs, serialized payload-size limits, idempotency keys, leases, stale-lease rejection, and retry/dead states. `DiscoveryDaemon.run_bounded` executes only explicitly registered handlers under time/step budgets; it does not dynamically import or shell out from queue payloads. -Phase 0 uses cooperative in-process preemption. Subprocess/cgroup isolation is a roadmap item before untrusted adapters. +Synthetic and CORE (first-party, real-data) adapters run in-process with cooperative preemption. UNTRUSTED (third-party) adapters run only via `sapiens.isolation`: a child process applies POSIX rlimits (CPU, address space, open files) to itself, the parent enforces a wall-clock timeout, and every failure mode is contained fail-closed (no evidence on failure). Third-party adapters additionally require a recorded owner permission (`permissions.json`, empty by default) before the registry will validate them at all. rlimits bound resources; they are not a full security sandbox. diff --git a/PROVENANCE.md b/PROVENANCE.md index 1ecdd46..60aac67 100644 --- a/PROVENANCE.md +++ b/PROVENANCE.md @@ -33,3 +33,16 @@ No third-party source code is included. Therefore no third-party notices are emb ## Known licence gate ASTRA-dev/ASTRA/GEODISC/BIODISC lack explicit licences at inspected refs. Any future kernel extraction from those repositories requires a signed/committed compatible licence or separate written permission. Until then: architecture-only references or clean-room reimplementation only. + +## Phase 1 — permission manifest mechanism + +Phase 1 operationalises the gate: [`permissions.json`](permissions.json) is the +machine-readable owner-permission manifest consumed by `sapiens.permissions` +and enforced by `sapiens.registry`. Any adapter declaring +`code_origin="third-party"` is UNTRUSTED-tier: it cannot validate without a +matching active permission entry (`adapter:` scope for its declared +`third_party_source`), and it executes only inside the resource-limited +subprocess (`sapiens.isolation`). **The shipped manifest is empty** — zero +ASTRA-family permissions — so the clean-room boundary above is enforced by +code, not by convention. Entries may be added only with explicit owner +sign-off (recorded grantor, licence, evidence reference, validity window). diff --git a/README.md b/README.md index ae08089..29147e6 100644 --- a/README.md +++ b/README.md @@ -8,8 +8,8 @@ SAPIENS is an **experimental platform for traceable cross-domain scientific-discovery workflows** — a `DomainAdapter` boundary, a hash-chained -L0→L4 evidence ledger, and kernel-gated promotions. **Phase 0: synthetic -adapters only, no discoveries claimed.** +L0→L4 evidence ledger, and kernel-gated promotions. **Phases 0–1 shipped: +synthetic adapters only in practice, no discoveries claimed.** It provides the plumbing a discovery system needs before it can be trusted: a domain-neutral adapter boundary, an append-only hash-chained evidence @@ -20,11 +20,13 @@ budgets. > **Read this first — despite the acronym:** SAPIENS is **not** AGI, ASI, or > superintelligence, and does not claim to be. It is an experimental research -> platform. **No scientific discoveries are claimed.** Phase 0 ships with -> deterministic **synthetic adapters only**; the included examples discover +> platform. **No scientific discoveries are claimed.** The shipped adapters +> remain deterministic and **synthetic only**; the included examples discover > nothing about nature. The CLI reports `"scientific_discoveries_claimed": 0` > by construction, and the test suite enforces the honesty and boundary -> invariants described below. +> invariants described below. Phase 1 added the *machinery* for real-domain +> work (trust tiers, isolation, permissions) — but no real-data adapter +> ships yet. ## Why @@ -39,7 +41,18 @@ inherits traceability and bounded confidence instead of retrofitting them. ```text src/sapiens/ models.py immutable candidate / evidence / manifest models - adapter.py DomainAdapter protocol + Phase-0 adapter validation + adapter.py DomainAdapter protocol; validation routes to the registry + registry.py trust-tiered adapter registry (SYNTHETIC / CORE / UNTRUSTED) + permissions.py owner-permission/licence manifest for third-party code + isolation.py subprocess + rlimit execution for UNTRUSTED adapters + checkpoints.py HMAC-signed ledger checkpoints + external anchor export + validation.py L1/L2 automated gates: sanity checks, holdout + leakage + fixtures.py seeded-bias fixture suite (labelled ground truth) + calibration.py gate-performance calibration reports + confidence.py calibration-gated confidence aggregation (refuses blindly) + review.py L3 panel protocol: roles, objections, multi-round gate + reviewers.py deterministic reference reviewers (four roles) + catchrate.py panel catch-rate scoring over the seeded fixtures ledger.py JSONL hash-chained evidence ledger, L0→L4 transition guard kernel.py domain-neutral DiscoveryKernel; owns all promotions bridge.py cross-domain structure transfer — ALWAYS resets target to L0 @@ -51,7 +64,9 @@ src/sapiens/ human-gated cli.py synthetic-only demo entrypoint adapters/ three deterministic synthetic adapters (linear, threshold, - periodic-signal photometry) + periodic-signal photometry) plus the Phase-4 real-data + Kepler photometry adapter (re-derivation of a published + signal; no discovery claimed) ``` Key design rules (enforced by tests in [`tests/`](tests/)): @@ -68,7 +83,7 @@ Key design rules (enforced by tests in [`tests/`](tests/)): - **L2 Replication** — passed held-out / reproducibility checks. - **L3 Review** — passed bounded structured review / adversarial checks. - **L4 External-ready** — requires an explicit **human gate**; autonomous - promotion to L4 is disabled in Phase 0. + promotion to L4 is disabled. - **Kernel-owned promotions** — adapters propose, only the `DiscoveryKernel` promotes, and only through the ledger's transition guard. - **Cross-domain bridge resets to L0** — transfer moves *structure and @@ -156,33 +171,71 @@ astrophysical result**. ## Status & roadmap -**Phase 0 — shipped** (current package version `0.1.0`): clean-room foundation, +**Phase 0 — shipped** (package version `0.1.0`): clean-room foundation, three deterministic synthetic adapters, synthetic-only orchestration, hash-chained ledger, kernel gates, bridge, bounded queue/daemon, and CI on -Python 3.10/3.11/3.12. The test suite currently includes positive and negative -period-detection fixtures, L0 reset/provenance checks, ledger tamper checks, -promotion guards, and bounded queue/daemon behavior. - -Next, in order (see [`ROADMAP.md`](ROADMAP.md)): - -1. **Phase 1** — legal/licence gate and adapter hardening (trust-tiered - adapter registry, sandboxing, signed ledger checkpoints). -2. **Phase 2** — validation framework v1 (statistical gates, holdout - protocols, leakage controls, seeded-bias fixtures, calibration). -3. **Phase 3** — structured L3 review panels (role-specialized reviewers, - multi-round objection tracking, catch-rate scoring). -4. **Phase 4** — **real domain adapters** (ASTRA / GEODISC / BIODISC / SLATE) - — only after licence and owner review. +Python 3.10/3.11/3.12. + +**Phase 1 — shipped** (package version `0.2.0`): the synthetic-only +gate is replaced by a **trust-tiered adapter registry** (SYNTHETIC / CORE / +UNTRUSTED), an **owner-permission/licence manifest** +([`permissions.json`](permissions.json) — empty by default: no third-party +code may power an adapter without a recorded entry), **subprocess isolation +with OS-level resource limits** for UNTRUSTED adapters (rlimit CPU / +address-space / open-files plus wall-clock timeout, fail-closed), and +**HMAC-signed ledger checkpoints** with external anchor export +(key from the environment only, never stored). No real-data adapter ships in +Phase 1; tiers are exercised by synthetic adapters and test doubles. + +**Phase 2 — shipped** (package version `0.3.0`): automated L0→L2 +**validation gates** — L1 statistical sanity checks (determinism, +degenerate-score rejection) and L2 declared holdout protocols with explicit +leakage controls (dataset collision, (dataset, seed) reuse) — plus a +labelled **seeded-bias fixture suite** (known-good / overfit / leakage / +degenerate), **calibration reports** (catch rate and false-reject rate with +sample counts), and **calibration-gated confidence aggregation** that +refuses to emit a number without sufficient calibration data. + +**Phase 3 — shipped** (package version `0.4.0`): structured **L3 +review panels** — four role-specialized deterministic reviewers +(statistician, domain theorist, methodologist, devil's advocate), a bounded +multi-round protocol with objection lifecycle tracking (raised / sustained +/ withdrawn) and disagreement gates (sustained MAJOR/BLOCKING objections +reject; MINOR caveats are recorded but non-fatal), panel verdicts recorded +in the ledger as review evidence, and **catch-rate scoring** over the +seeded fixture suite (panel catches 3/3 known-bad, 0 false rejects — exact +for this suite, not an estimate). + +**Phase 4 — shipped** (current package version `0.5.0`): the first +**real-data adapter** — `KeplerPhotometryAdapter` (CORE trust tier) +re-derives the **already published** Kepler-10 b transit signal from a +bundled, checksum-pinned public NASA/MAST Q1 light curve and climbs it +L0→L3 through the full stack (trust registry → validation gates → review +panel), with L4 still human-gated. Negative controls included: flat curves +propose nothing, tampered curves refuse to load, shifted-period curves fail +replication honestly. **This is a validation of a published result, not a +discovery.** Clean-room: zero ASTRA-family code; the permission manifest +stays empty, so ASTRA/GEODISC/BIODISC/SLATE adapters remain gated. + +Next (see [`ROADMAP.md`](ROADMAP.md)): + +1. ~~**Phase 1** — legal/licence gate and adapter hardening~~ **shipped**. +2. ~~**Phase 2** — validation framework v1~~ **shipped**. +3. ~~**Phase 3** — structured L3 review panels~~ **shipped**. +4. ~~**Phase 4** — real domain adapters (clean-room Kepler photometry)~~ + **shipped**. ASTRA / GEODISC / BIODISC / SLATE adapters remain gated on + licence and owner review. 5. **Phase 5** — external-review workflows with human L4 gates and reproduction bundles. ## Provenance & legal boundary -SAPIENS Phase 0 is a **clean-room implementation**: no source code from +SAPIENS is a **clean-room implementation**: no source code from ASTRA-dev, ASTRA, GEODISC, BIODISC, or SLATE was copied into this repository -(those codebases carry unresolved licensing; reuse is explicitly gated to -Phase 1+ with owner permission). See [`PROVENANCE.md`](PROVENANCE.md) for the -documented boundary. +(those codebases carry unresolved licensing; reuse is gated on recorded owner +permission — the Phase-1 permission manifest ships empty, so every +third-party adapter is refused today). See [`PROVENANCE.md`](PROVENANCE.md) +for the documented boundary. ## Credits diff --git a/ROADMAP.md b/ROADMAP.md index 47c5d8a..e3debfb 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -7,30 +7,95 @@ - Synthetic integrated orchestration. - Tests and Python matrix CI. -## Phase 1 — legal/licence gate and adapter hardening +## Phase 1 — shipped (package version 0.2.0) -- Obtain explicit licences/permissions for any ASTRA-family code reuse before extraction. -- Replace synthetic-only adapter gate with a trust-tiered adapter registry. -- Add subprocess isolation and OS-level resource limits for untrusted adapters. -- Add signed ledger checkpoints or external anchoring. +Legal/licence gate and adapter hardening: -## Phase 2 — validation framework v1 +- ~~Obtain explicit licences/permissions for any ASTRA-family code reuse + before extraction.~~ Mechanism shipped: `permissions.json` + + `sapiens.permissions` record owner grants; the manifest is **empty** — no + ASTRA-family (or any third-party) permissions exist, so every third-party + adapter is refused until an owner records one. +- ~~Replace synthetic-only adapter gate with a trust-tiered adapter + registry.~~ Shipped: `sapiens.registry` (SYNTHETIC / CORE / UNTRUSTED). +- ~~Add subprocess isolation and OS-level resource limits for untrusted + adapters.~~ Shipped: `sapiens.isolation` — rlimit CPU / address-space / + open-files plus wall-clock timeout, fail-closed; the kernel runs + UNTRUSTED-tier adapters only through it. +- ~~Add signed ledger checkpoints or external anchoring.~~ Shipped: + `sapiens.checkpoints` — HMAC-SHA256 checkpoint events (key from + environment only) plus external anchor export/verify. -- Expand L0→L2 automated gates with statistical sanity checks, holdout protocols, and explicit leakage controls. -- Add seeded-bias fixtures and calibration reports. -- Add confidence aggregation only after calibration data exists; do not invent precision. +Honest limits: rlimits bound resource use but are not a security sandbox; +HMAC proves local key possession, not third-party authorship. No real-data +adapter ships in Phase 1. -## Phase 3 — structured L3 review panels +## Phase 2 — shipped (package version 0.3.0) -- Role-specialized reviewer schemas: statistician, domain theorist, methodologist, devil's advocate. -- Multi-round reports, objection tracking, disagreement gates. -- Catch-rate scoring on seeded known-bad and known-good candidates. +Validation framework v1: -## Phase 4 — real domain adapters +- ~~Expand L0→L2 automated gates~~ Shipped: `sapiens.validation` — L1 + internal-consistency gate (score presence/range, determinism across reruns, + degenerate constant-score rejection) and L2 holdout-replication gate + (declared `HoldoutProtocol` per domain; explicit leakage controls on + dataset collision and (dataset, seed) reuse; pass-fraction threshold). + Opt-in kernel wiring via `DiscoveryKernel(validation=...)`; gate verdicts + are logged, never fabricated as evidence; a configured domain without a + declared protocol fails closed. +- ~~Add seeded-bias fixtures and calibration reports.~~ Shipped: + `sapiens.fixtures` (known-good / overfit / leakage / degenerate, labelled + with expected outcomes) and `sapiens.calibration` (`CalibrationReport`: + catch rate, false-reject rate, sample counts; report ids are content + hashes). +- ~~Add confidence aggregation only after calibration data exists~~ Shipped: + `sapiens.confidence.aggregate_confidence` raises `UncalibratedError` + without a sufficiently sampled calibration report; with one, it emits a + documented heuristic (raw pass fraction × demonstrated catch rate) with + full provenance. No invented precision. +## Phase 3 — shipped (package version 0.4.0) + +Structured L3 review panels: + +- ~~Role-specialized reviewer schemas~~ Shipped: `sapiens.review` — + statistician, domain theorist, methodologist, devil's advocate; typed + approve/object/abstain verdicts with severity-graded objections. +- ~~Multi-round reports, objection tracking, disagreement gates~~ Shipped: + bounded deterministic multi-round protocol; objection lifecycle + (raised/sustained/withdrawn) tracked in every `PanelReport`; sustained + MAJOR/BLOCKING objections reject, MINOR caveats are recorded but + non-fatal (documented gate semantics). Deterministic reference reviewers + in `sapiens.reviewers` re-run the Phase-2 gates independently and hunt + seeded-bias signatures. +- ~~Catch-rate scoring~~ Shipped: `sapiens.catchrate.score_panel` over the + seeded fixture suite — per-role and panel-level catch rates plus + false-reject rate, with an explicit small-sample caveat. +- Kernel integration: with a panel configured, L3 promotion requires panel + approval, and the verdict is recorded in the ledger as review evidence + (no side channel). Without a panel, Phase-2 behaviour is unchanged. + +## Phase 4 — shipped (package version 0.5.0) + +Real domain adapters: + +- ~~First: one clean-room Kepler photometry adapter~~ Shipped: + `sapiens.adapters.kepler.KeplerPhotometryAdapter` (CORE trust tier) + re-derives the **published** Kepler-10 b signal from a bundled public + NASA/MAST Q1 light curve (checksum-pinned, provenance in + `src/sapiens/adapters/data/README.md`). Framed everywhere as validation + of a published result, never a discovery. The pipeline is this + repository's own Apache-2.0 demo code path (`adapters/_transit.py`); + **zero ASTRA-family code** (the permission manifest remains empty). - ASTRA/GEODISC/BIODISC/SLATE adapters only after licence and owner review. -- Domain-specific validators remain sandboxed behind adapters. -- Cross-domain method transfer enters target domain at L0 every time. + (Unchanged; still gated.) +- ~~Domain-specific validators remain sandboxed behind adapters~~ Shipped: + BLS search, detrending, fold measurement and adversarial checks live + inside the adapter; the kernel sees only contract evidence. +- ~~Cross-domain method transfer enters target domain at L0 every time~~ + Re-verified against the real adapter (bridge test). +- Negative controls: flat curves propose nothing; tampered curves refuse to + load (pinned checksum); shifted-period curves fail replication with the + disagreement shown in evidence details. ## Phase 5 — external-review workflows diff --git a/VALIDATION.md b/VALIDATION.md index 14477d4..8925106 100644 --- a/VALIDATION.md +++ b/VALIDATION.md @@ -23,7 +23,7 @@ python -m sapiens.cli - Queue capacity, idempotency, stale-lease rejection, oversized-payload rejection. - Daemon executes only registered handlers under bounded context. - Boundary test: core modules do not import synthetic adapters. -- Phase-0 rejects non-synthetic adapters. +- Phase-0 rejects non-synthetic adapters (superseded in Phase 1 by trust-tier tests). ## CI @@ -44,10 +44,30 @@ GitHub Actions runs Python 3.10, 3.11, and 3.12 with: 6. Ledger uses canonical JSON and rejects hash/sequence/transition violations. 7. Cross-domain bridge discards source confidence and starts target at L0. +## Phase 1–2 additions + +- Trust-tier registry: third-party adapters refused without permission; tier derivation from manifest facts. +- Isolation: CPU-hog / memory-hog / sleeper children killed and contained; protocol corruption impossible from child stdout. +- Checkpoints: continuity verified; HMAC sign/verify; wrong-key detection; anchor export/verify. +- L1 gate: determinism, degenerate-score, missing-score rejection. +- L2 gate: holdout discipline, dataset-collision and (dataset, seed) leakage rejection, pass-fraction threshold. +- Calibration: fixture labels verified against gate behaviour; report rates match ground truth. +- Confidence: refuses without calibration / with thin calibration / without evidence. +- Panels: schema coherence (objections only on OBJECT), multi-round lifecycle (raise/sustain/withdraw/escalate), disagreement gates, budget bounds. +- Reference reviewers: each catches its designed fixture class; clean per-role attribution. +- Catch-rate harness: panel 3/3 known-bad, 0 false rejects on the seeded suite (regression thresholds). +- Kernel panel gate: approval path recorded end-to-end in the ledger; rejection blocks L3 with verdict on record; repeated attempts do not collide. +- Kepler adapter: published-period re-derivation within tolerance; staged validators pass on real data; flat/shifted/tampered negative controls; CORE tier enforcement; L0 transfer reset re-verified; L4 refusal verified on the real candidate. + ## Known limits - Hash-chain integrity is not cryptographic authorship or external timestamping. -- Phase-0 daemon is cooperative and in-process; hostile adapters need subprocess/cgroup isolation later. +- rlimits bound resource use but are not a full security sandbox; third-party code additionally requires a recorded owner permission. +- HMAC checkpoint signatures are symmetric: key possession, not third-party authorship. +- Confidence values are documented heuristics (raw pass fraction × demonstrated catch rate), not probability estimates; fixture suites are small and rates are exact only for the fixtures included. +- Reference reviewers are deterministic heuristics, not domain experts; panel approval is procedural evidence, not peer review by scientists. Catch rates are exact for the seeded suite only. +- The Kepler adapter re-derives one known signal from one public quarter of data. An L3 climb here validates pipeline plumbing against an answer key; it is not evidence of discovery capability, and no discovery is claimed. +- The daemon is cooperative and in-process for SYNTHETIC/CORE tiers; UNTRUSTED adapters run via subprocess isolation. - Synthetic adapters are toy harnesses, not scientific models. - L3 is represented as a bounded `review` evidence gate; full multi-agent review panels are roadmap. - L4 is only a human-gated transition rule, not automated. diff --git a/permissions.json b/permissions.json new file mode 100644 index 0000000..823128d --- /dev/null +++ b/permissions.json @@ -0,0 +1,5 @@ +{ + "version": 1, + "comment": "Owner permissions for third-party code reuse (Phase 1). EMPTY BY DEFAULT: no ASTRA-family (or any other third-party) permissions are recorded, so every third-party adapter is refused. Add entries only with explicit owner sign-off; see PROVENANCE.md.", + "entries": [] +} diff --git a/pyproject.toml b/pyproject.toml index 56feaa4..579d273 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "sapiens-discovery" -version = "0.1.0" +version = "0.5.0" description = "Experimental foundation for traceable cross-domain scientific-discovery workflows" readme = "README.md" requires-python = ">=3.10" @@ -29,6 +29,9 @@ sapiens = "sapiens.cli:main" [tool.setuptools.packages.find] where = ["src"] +[tool.setuptools.package-data] +sapiens = ["adapters/data/*.csv", "adapters/data/*.md"] + [tool.pytest.ini_options] addopts = "-q" testpaths = ["tests", "demos/ledger-grok/tests", "mcp/tests"] diff --git a/src/sapiens/__init__.py b/src/sapiens/__init__.py index 0d6a2fc..5381af4 100644 --- a/src/sapiens/__init__.py +++ b/src/sapiens/__init__.py @@ -2,19 +2,50 @@ from .adapter import DomainAdapter from .bridge import TransferEnvelope, transfer +from .calibration import CalibrationReport, run_calibration +from .catchrate import CatchRateReport, score_panel +from .confidence import CalibratedConfidence, UncalibratedError, aggregate_confidence +from .fixtures import FixtureKind, SeededFixture, fixture_suite from .kernel import DiscoveryKernel from .ledger import EvidenceLedger from .models import AdapterManifest, Candidate, Evidence, EvidenceLevel +from .permissions import PermissionEntry, PermissionManifest +from .registry import AdapterRegistry, TrustTier +from .review import PanelOutcome, PanelReport, ReviewerRole, ReviewPanel +from .reviewers import reference_panel +from .validation import GateVerdict, HoldoutProtocol, ValidationGates __all__ = [ "AdapterManifest", + "AdapterRegistry", + "CalibratedConfidence", + "CalibrationReport", "Candidate", + "CatchRateReport", "DiscoveryKernel", "DomainAdapter", "Evidence", "EvidenceLedger", "EvidenceLevel", + "FixtureKind", + "GateVerdict", + "HoldoutProtocol", + "PanelOutcome", + "PanelReport", + "PermissionEntry", + "PermissionManifest", + "ReviewerRole", + "ReviewPanel", + "SeededFixture", "TransferEnvelope", + "TrustTier", + "UncalibratedError", + "ValidationGates", + "aggregate_confidence", + "fixture_suite", + "reference_panel", + "run_calibration", + "score_panel", "transfer", ] -__version__ = "0.1.0" +__version__ = "0.5.0" diff --git a/src/sapiens/adapter.py b/src/sapiens/adapter.py index 9829e4b..6627cb6 100644 --- a/src/sapiens/adapter.py +++ b/src/sapiens/adapter.py @@ -25,8 +25,15 @@ def import_structure(self, structure: dict[str, object], *, candidate_id: str) - def validate_adapter(adapter: DomainAdapter) -> None: - if not isinstance(adapter, DomainAdapter): - raise TypeError("adapter does not implement DomainAdapter") - manifest = adapter.manifest - if not manifest.synthetic_only: - raise ValueError("Phase 0 refuses non-synthetic adapters") + """Phase-1 gate: auto-tiered registry validation. + + Kept for backward compatibility with Phase-0 callers. Synthetic adapters + pass exactly as before; first-party clean-room real-data adapters pass at + CORE tier; third-party adapters require a recorded permission and raise + ``MissingPermissionError`` without one. Callers that need the tier (e.g. + the kernel, to decide on isolation) should use ``AdapterRegistry`` + directly. + """ + from .registry import AdapterRegistry + + AdapterRegistry().validate_adapter(adapter) diff --git a/src/sapiens/adapters/__init__.py b/src/sapiens/adapters/__init__.py index efa227b..a168cd3 100644 --- a/src/sapiens/adapters/__init__.py +++ b/src/sapiens/adapters/__init__.py @@ -1,11 +1,19 @@ -"""Synthetic-only demonstration adapters; not scientific models.""" +"""Demonstration adapters; not scientific models. +The synthetic adapters exercise orchestration on deterministic fake data. +The Kepler adapter re-derives a **published** signal from public data as a +validation of the pipeline — it claims no discovery. +""" + +from .kepler import KeplerPhotometryAdapter, kepler_holdout_protocol from .linear import SyntheticLinearAdapter from .photometry import SyntheticPhotometryAdapter from .threshold import SyntheticThresholdAdapter __all__ = [ + "KeplerPhotometryAdapter", "SyntheticLinearAdapter", "SyntheticPhotometryAdapter", "SyntheticThresholdAdapter", + "kepler_holdout_protocol", ] diff --git a/src/sapiens/adapters/_transit.py b/src/sapiens/adapters/_transit.py new file mode 100644 index 0000000..7db8aeb --- /dev/null +++ b/src/sapiens/adapters/_transit.py @@ -0,0 +1,268 @@ +"""Stdlib transit-search primitives for the Kepler photometry adapter. + +Ported from this repository's own Apache-2.0 `demos/ledger-grok` pipeline +(same methods, tuned for bounded CI runtime): running-median detrend, an +unweighted box-least-squares-style matched filter over a uniform frequency +grid with a refine pass, phase-fold measurement (depth / SNR / epoch / +odd-even / secondary), and harmonic power checks. Pure standard library; +no network; no astrophysics claims beyond arithmetic on the input curve. + +The Kepler-10 C transit mask (BKJD 138.54–138.86, a ~450 ppm event from a +different, also already-known planet) is applied by the caller and +disclosed in every result, exactly as the demo does. +""" + +from __future__ import annotations + +import csv +import math +from pathlib import Path + +DEFAULT_DURATIONS = (0.03, 0.05, 0.07, 0.09, 0.11) + +# Known Kepler-10 C transit inside the Q1 segment; masked and disclosed. +MASK_EVENTS = ({"t_start_bkjd": 138.54, "t_end_bkjd": 138.86, "label": "Kepler-10 C"},) + + +class DataIntegrityError(RuntimeError): + """Bundled data failed its pinned checksum, or a curve is unusable.""" + + +def load_csv(path: str | Path) -> tuple[list[float], list[float]]: + """Load a two-column CSV (time BKJD, flux); '#' lines are comments.""" + times: list[float] = [] + fluxes: list[float] = [] + with Path(path).open(newline="", encoding="utf-8") as fh: + for row in csv.reader(fh): + if not row or row[0].lstrip().startswith("#"): + continue + if row[0].strip().lower() in ("time_bkjd", "time"): + continue + t, f = float(row[0]), float(row[1]) + if math.isfinite(t) and math.isfinite(f): + times.append(t) + fluxes.append(f) + if len(times) < 10: + raise DataIntegrityError(f"not enough finite rows in {path}") + return times, fluxes + + +def apply_mask( + times: list[float], fluxes: list[float], mask_events=MASK_EVENTS +) -> tuple[list[float], list[float], int]: + """Drop rows inside known contaminating events; return (t, f, n_masked).""" + keep_t: list[float] = [] + keep_f: list[float] = [] + for t, f in zip(times, fluxes, strict=True): + if any(event["t_start_bkjd"] <= t <= event["t_end_bkjd"] for event in mask_events): + continue + keep_t.append(t) + keep_f.append(f) + return keep_t, keep_f, len(times) - len(keep_t) + + +def _median(values: list[float]) -> float: + s = sorted(values) + n = len(s) + mid = n // 2 + if n % 2: + return s[mid] + return 0.5 * (s[mid - 1] + s[mid]) + + +def detrend(times: list[float], fluxes: list[float], window_days: float = 1.5) -> list[float]: + """Median-normalise, then divide by a running-median baseline.""" + n = len(times) + base = _median(fluxes) + if base == 0: + raise DataIntegrityError("median flux is zero; cannot normalise") + y = [f / base for f in fluxes] + dt = _median([times[i + 1] - times[i] for i in range(n - 1)]) + w = max(5, int(round(window_days / dt))) + if w % 2 == 0: + w += 1 + half = w // 2 + out = [] + for i in range(n): + lo = max(0, i - half) + hi = min(n, i + half + 1) + m = _median(y[lo:hi]) + out.append(y[i] / m if m else 1.0) + return out + + +def _eval_period(times: list[float], resid: list[float], period: float, ms: list[int]) -> dict: + """Evaluate the best box (max summed residual) at one trial period.""" + n = len(times) + order = sorted(range(n), key=lambda i: math.fmod(times[i], period)) + ys = [resid[i] for i in order] + ph = [math.fmod(times[i], period) / period for i in order] + mmax = max(ms) + ext = ys + ys[:mmax] + pre = [0.0] + acc = 0.0 + for v in ext: + acc += v + pre.append(acc) + best = None + for m in ms: + boxes = [pre[i + m] - pre[i] for i in range(n)] + i0 = max(range(n), key=boxes.__getitem__) + s = boxes[i0] + power = s * s * n / (m * (n - m)) + if best is None or power > best[0]: + best = (power, s, m, i0) + power, s, m, i0 = best + phi_start = ph[i0] + wsum = 0.0 + csum = 0.0 + for j in range(i0, i0 + m): + idx = j % n + w = max(ys[idx], 0.0) + rel = (ph[idx] - phi_start) % 1.0 + wsum += w + csum += w * rel + phi = (phi_start + (csum / wsum if wsum else 0.0)) % 1.0 + return {"power": power, "box_sum": s, "m": m, "q": m / n, "phase_center": phi} + + +def bls( + times: list[float], + y: list[float], + pmin: float = 0.5, + pmax: float = 10.0, + nfreq: int = 1500, + durations: tuple[float, ...] = DEFAULT_DURATIONS, + refine: bool = True, +) -> dict: + """Box-least-squares grid search, uniform in frequency, plus a refine pass.""" + n = len(times) + if n < 20: + raise ValueError("need at least 20 points for a meaningful search") + resid = [1.0 - v for v in y] + ms = sorted({max(2, int(round(q * n))) for q in durations}) + fmax, fmin = 1.0 / pmin, 1.0 / pmax + fstep = (fmax - fmin) / (nfreq - 1) + best_p, best_r = None, None + for k in range(nfreq): + p = 1.0 / (fmax - fstep * k) + r = _eval_period(times, resid, p, ms) + if best_r is None or r["power"] > best_r["power"]: + best_p, best_r = p, r + if refine: + f_best = 1.0 / best_p + p_lo = 1.0 / (f_best + 2.5 * fstep) + p_hi = 1.0 / (f_best - 2.5 * fstep) + for k in range(801): + p = p_lo + (p_hi - p_lo) * k / 800 + r = _eval_period(times, resid, p, ms) + if r["power"] > best_r["power"]: + best_p, best_r = p, r + return { + "period_days": best_p, + "power": best_r["power"], + "q": best_r["q"], + "phase_center": best_r["phase_center"], + "grid": { + "pmin": pmin, + "pmax": pmax, + "nfreq": nfreq, + "durations": list(durations), + "refined": bool(refine), + }, + } + + +def _circ_dist(a: float, b: float) -> float: + d = abs(a - b) % 1.0 + return min(d, 1.0 - d) + + +def fold_measure( + times: list[float], y: list[float], period: float, phase_center: float, q: float +) -> dict: + """Measure depth, SNR, epoch, odd/even consistency and a secondary-dip check.""" + half = q / 2.0 + phases = [math.fmod(t, period) / period for t in times] + ins: list[float] = [] + outs: list[float] = [] + for i, ph in enumerate(phases): + d = _circ_dist(ph, phase_center) + if d <= half: + ins.append(y[i]) + elif d > half * 1.6: + outs.append(y[i]) + if len(ins) < 2 or len(outs) < 10: + raise ValueError("degenerate fold: too few in/out points") + mean_out = sum(outs) / len(outs) + mean_in = sum(ins) / len(ins) + std_out = math.sqrt(sum((v - mean_out) ** 2 for v in outs) / (len(outs) - 1)) + depth = mean_out - mean_in + snr = depth / (std_out / math.sqrt(len(ins))) + t0 = min(times) + # phase_center is an ABSOLUTE fold phase (fmod origin t=0): transit + # centres are (m + phase_center) * period; pick the one nearest t0. + m0 = round(t0 / period - phase_center) + epoch = (m0 + phase_center) * period + n_transits = max(1, int(round((max(times) - t0) / period))) + + def depth_for(parity: int) -> tuple[float, int]: + sel = [ + y[i] + for i, t in enumerate(times) + if _circ_dist(phases[i], phase_center) <= half + and int(round((t - epoch) / period)) % 2 == parity + ] + if not sel: + return float("nan"), 0 + return mean_out - sum(sel) / len(sel), len(sel) + + d_even, n_even = depth_for(0) + d_odd, n_odd = depth_for(1) + denom = std_out * math.sqrt(1.0 / max(n_even, 1) + 1.0 / max(n_odd, 1)) + delta_sigma = abs(d_even - d_odd) / denom if denom else float("nan") + + center2 = (phase_center + 0.5) % 1.0 + sec = [y[i] for i, ph in enumerate(phases) if _circ_dist(ph, center2) <= half] + d_sec = (mean_out - sum(sec) / len(sec)) if sec else float("nan") + sec_sigma = d_sec / (std_out / math.sqrt(len(sec))) if sec else float("nan") + + return { + "depth": depth, + "depth_ppm": depth * 1e6, + "snr": snr, + "n_in": len(ins), + "n_out": len(outs), + "std_out_ppm": std_out * 1e6, + "epoch_bkjd": epoch, + "n_transits": n_transits, + "odd_even": { + "depth_even_ppm": d_even * 1e6, + "depth_odd_ppm": d_odd * 1e6, + "n_even": n_even, + "n_odd": n_odd, + "delta_sigma": delta_sigma, + }, + "secondary": { + "depth_ppm": d_sec * 1e6, + "sigma": sec_sigma, + "n": len(sec), + }, + } + + +def harmonic_powers( + times: list[float], + y: list[float], + period: float, + durations: tuple[float, ...] = DEFAULT_DURATIONS, +) -> dict: + """Box power at P/2, P and 2P (odd/even + secondary are the discriminators).""" + n = len(times) + resid = [1.0 - v for v in y] + ms = sorted({max(2, int(round(q * n))) for q in durations}) + out = {} + for label, mult in (("half", 0.5), ("fundamental", 1.0), ("double", 2.0)): + r = _eval_period(times, resid, period * mult, ms) + out[label] = {"period_days": period * mult, "power": r["power"]} + return out diff --git a/src/sapiens/adapters/data/README.md b/src/sapiens/adapters/data/README.md new file mode 100644 index 0000000..b755f31 --- /dev/null +++ b/src/sapiens/adapters/data/README.md @@ -0,0 +1,29 @@ +# Adapter data provenance — Kepler-10 (KIC 11904151) Quarter 1 light curve + +`kepler10_kic11904151_q1_lc.csv` (32 KB, 1,432 rows) is a small excerpt of +**public NASA Kepler data**, bundled so the Phase-4 adapter can re-derive an +already-published signal offline. + +| Field | Value | +|---|---| +| Target | Kepler-10 (KIC 11904151), Kepler magnitude 10.96 | +| Quarter | Q1 (33.5 days; 2009-05-13 → 2009-06-15) | +| Product | PDCSAP flux (pre-search data conditioning simple aperture photometry) | +| Source | NASA/MAST public archive — https://mast.stsci.edu | +| Retrieved | 2026-07-19 via the MAST Kepler archive (public, no authentication) | +| File | `kplr011904151-2009166043257_llc.fits` → two-column CSV (time BKJD, PDCSAP flux) | +| Licence | NASA Kepler data are public-domain works of the US federal government | +| SHA-256 | `1d82ef8ced447bb44ac80c7389a8ef3dde93bf1983d619d56fe2f6cda3a56ded` (pinned; verified at load) | + +This is the same reference dataset used by `demos/ledger-grok` (identical +bytes, same checksum). **Data is not code**: this file carries no program +logic, and no ASTRA-family material of any kind is included. + +One known Kepler-10 C transit inside this segment (BKJD 138.54–138.86, +~450 ppm) is masked at load time and disclosed in adapter evidence details, +exactly as in the demo — it belongs to a different, also already-known +planet and would otherwise tilt the period search. + +The bundled curve is used strictly to **re-derive the published Kepler-10 b +signal as a validation of the pipeline**. It is not used to claim any +discovery. diff --git a/src/sapiens/adapters/data/kepler10_kic11904151_q1_lc.csv b/src/sapiens/adapters/data/kepler10_kic11904151_q1_lc.csv new file mode 100644 index 0000000..f029f28 --- /dev/null +++ b/src/sapiens/adapters/data/kepler10_kic11904151_q1_lc.csv @@ -0,0 +1,1440 @@ +# Kepler-10 (KIC 11904151) long-cadence light curve +# source_uri: mast:Kepler/url/missions/kepler/lightcurves/0119/011904151/kplr011904151-2009166043257_llc.fits +# source_url: https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:Kepler/url/missions/kepler/lightcurves/0119/011904151/kplr011904151-2009166043257_llc.fits +# retrieved_utc: 2026-07-19T12:44:04Z +# fits_sha256: a47542bd31fa92111f280a29885f6fbe860f0956012b442f780ca6d2cc3a19c6 +# fits_bytes: 192960 +# time: BKJD (BJD - 2454833); flux: PDCSAP_FLUX (e-/s), SAP_QUALITY==0 rows only +time_bkjd,flux_e_per_s +131.512315,544142.062 +131.532749,544175.250 +131.553183,544114.312 +131.573617,544097.562 +131.594052,544053.062 +131.614486,544143.438 +131.634920,544165.188 +131.655354,544154.062 +131.675789,544167.000 +131.696223,544178.625 +131.716657,544115.500 +131.737091,544181.125 +131.757525,544131.188 +131.777959,544188.750 +131.798394,544141.562 +131.818828,544189.938 +131.839262,544155.500 +131.859696,544188.688 +131.880130,544174.000 +131.900565,544113.875 +131.920999,544152.188 +131.941433,544163.438 +131.961867,544145.500 +131.982301,544162.688 +132.002736,544201.500 +132.023170,544183.688 +132.043604,544172.188 +132.064038,544136.062 +132.084472,544138.750 +132.104906,544168.062 +132.125341,544146.188 +132.145775,544165.750 +132.166209,544134.500 +132.186643,544164.625 +132.207077,544182.750 +132.227512,544180.688 +132.247946,544125.375 +132.268380,544148.375 +132.288814,544181.250 +132.309248,544157.750 +132.329683,544120.375 +132.350117,544169.000 +132.370551,544113.312 +132.390985,544083.875 +132.411419,544047.812 +132.431853,544054.812 +132.452288,544167.812 +132.472722,544148.938 +132.493156,544131.938 +132.513590,544132.188 +132.534024,544109.375 +132.554459,544184.312 +132.574893,544154.812 +132.595327,544130.438 +132.615761,544145.938 +132.636195,544108.875 +132.656630,544151.312 +132.677064,544137.188 +132.697498,544159.750 +132.717932,544189.062 +132.738366,544146.875 +132.758801,544191.625 +132.779235,544154.500 +132.799669,544220.938 +132.820103,544164.938 +132.840537,544150.562 +132.860971,544179.188 +132.881406,544180.188 +132.901840,544147.062 +132.922274,544228.500 +132.942708,544161.500 +132.963142,544165.562 +132.983577,544154.438 +133.004011,544131.938 +133.024445,544163.500 +133.065313,544165.625 +133.085747,544175.938 +133.106182,544140.688 +133.126616,544090.562 +133.147050,544186.250 +133.167484,544181.062 +133.187918,544135.875 +133.208353,544097.062 +133.228787,544003.812 +133.249221,544038.500 +133.269655,544026.750 +133.290089,544132.562 +133.330958,544074.000 +133.351392,544146.812 +133.371826,544169.062 +133.392260,544097.938 +133.412695,544095.000 +133.433129,544176.188 +133.453563,544152.938 +133.473997,544146.000 +133.494431,544101.062 +133.514865,544159.062 +133.535300,544112.812 +133.555734,544109.812 +133.576168,544115.875 +133.596602,544115.938 +133.617036,544145.000 +133.637471,544077.875 +133.657905,544121.438 +133.678339,544153.125 +133.698773,544141.438 +133.719207,544135.688 +133.739641,544137.062 +133.760076,544152.062 +133.780510,544098.625 +133.800944,544152.688 +133.821378,544145.125 +133.841812,544108.750 +133.862247,544139.938 +133.882681,544104.625 +133.903115,544169.438 +133.923549,544100.562 +133.964417,544098.500 +133.984852,544145.562 +134.005286,544129.312 +134.025720,544142.375 +134.046154,544055.812 +134.066588,544060.562 +134.107457,544072.812 +134.127891,544119.500 +134.148325,544100.438 +134.168759,544080.375 +134.209628,544102.000 +134.230062,544124.312 +134.270930,544167.062 +134.291364,544143.812 +134.311799,544139.875 +134.332233,544167.688 +134.352667,544169.812 +134.373101,544169.562 +134.413969,544112.562 +134.434404,544234.750 +134.454838,544157.875 +134.475272,544171.375 +134.495706,544167.438 +134.516140,544168.750 +134.536575,544226.000 +134.557009,544242.812 +134.577443,544203.688 +134.597877,544153.438 +134.618311,544193.688 +134.638745,544205.562 +134.659180,544231.750 +134.679614,544210.062 +134.700048,544194.125 +134.720482,544190.688 +134.740916,544212.125 +134.761351,544228.875 +134.781785,544216.250 +134.802219,544199.938 +134.822653,544179.750 +134.843087,544220.375 +134.863522,544187.750 +134.883956,544189.062 +134.904390,544114.312 +134.924824,544101.250 +134.945258,544124.688 +134.965692,544204.250 +134.986127,544237.000 +135.006561,544200.688 +135.026995,544225.688 +135.047429,544214.750 +135.067863,544251.750 +135.088298,544234.938 +135.108732,544242.500 +135.129166,544196.750 +135.149600,544204.812 +135.170034,544230.125 +135.190468,544215.500 +135.210903,544198.125 +135.231337,544218.500 +135.251771,544243.125 +135.272205,544183.250 +135.292639,544220.812 +135.313073,544197.938 +135.333508,544160.250 +135.353942,544156.438 +135.374376,544226.125 +135.394810,544211.562 +135.415244,544221.875 +135.435678,544217.750 +135.456113,544221.250 +135.476547,544195.062 +135.496981,544201.500 +135.517415,544159.625 +135.537849,544194.750 +135.558284,544235.938 +135.578718,544162.375 +135.599152,544206.750 +135.619586,544200.812 +135.640020,544198.938 +135.660455,544205.375 +135.680889,544210.312 +135.701323,544231.688 +135.721757,544152.688 +135.742191,544060.250 +135.762625,544120.688 +135.783060,544076.312 +135.803494,544161.750 +135.823928,544176.812 +135.844362,544227.500 +135.864796,544236.312 +135.885230,544142.250 +135.905665,544204.562 +135.926099,544157.000 +135.946533,544108.938 +135.966967,544185.812 +135.987401,544183.062 +136.007835,544155.062 +136.028270,544226.312 +136.048704,544190.375 +136.069138,544182.312 +136.089572,544239.125 +136.110006,544209.625 +136.130441,544168.062 +136.171309,544192.312 +136.191743,544208.375 +136.212177,544171.938 +136.232611,544188.938 +136.253046,544185.750 +136.273480,544156.062 +136.293914,544205.875 +136.314348,544128.562 +136.334782,544164.812 +136.355216,544207.812 +136.375651,544164.812 +136.396085,544212.062 +136.416519,544149.375 +136.436953,544175.562 +136.457387,544129.875 +136.477822,544191.562 +136.498256,544139.625 +136.518690,544138.875 +136.539124,544130.812 +136.559558,544162.312 +136.579992,544087.375 +136.600427,544070.000 +136.620861,544043.562 +136.641295,544223.438 +136.661729,544184.312 +136.682163,544163.125 +136.702597,544113.875 +136.723032,544192.250 +136.743466,544124.312 +136.763900,544165.438 +136.784334,544167.625 +136.804768,544190.500 +136.825203,544138.312 +136.845637,544177.500 +136.866071,544221.938 +136.886505,544121.625 +136.906939,544153.875 +136.927373,544163.062 +136.947807,544142.125 +136.988676,544116.000 +137.009110,544145.625 +137.029544,544125.688 +137.049978,544118.250 +137.070413,544143.062 +137.090847,544131.938 +137.111281,544157.938 +137.131715,544180.312 +137.152149,544174.125 +137.172583,544150.750 +137.193018,544162.250 +137.213452,544133.688 +137.233886,544187.312 +137.254320,544097.062 +137.274754,544106.562 +137.295188,544194.625 +137.315623,544141.438 +137.336057,544138.000 +137.356491,544105.312 +137.397359,544128.188 +137.417793,544078.938 +137.438228,544011.938 +137.458662,544065.438 +137.479096,544139.750 +137.499530,544173.438 +137.519964,544144.375 +137.540398,544158.125 +137.560833,544152.375 +137.581267,544185.375 +137.601701,544166.000 +137.622135,544151.000 +137.642569,544146.812 +137.663003,544150.688 +137.683438,544186.438 +137.703872,544137.875 +137.724306,544175.875 +137.744740,544169.812 +137.765174,544142.938 +137.785609,544177.562 +137.806043,544150.188 +137.826477,544161.000 +137.846911,544164.375 +137.867345,544178.875 +137.887779,544196.562 +137.908213,544201.875 +137.949082,544202.938 +137.969516,544221.875 +137.989950,544201.375 +138.010384,544170.750 +138.030818,544239.188 +138.051253,544150.062 +138.071687,544170.312 +138.092121,544204.938 +138.112555,544162.062 +138.132989,544167.312 +138.153424,544112.688 +138.173858,544191.812 +138.214726,544142.188 +138.235160,544080.562 +138.255594,544074.812 +138.276029,544010.188 +138.296463,544045.438 +138.316897,544103.000 +138.337331,544135.062 +138.357765,544175.062 +138.378199,544194.875 +138.419068,544187.000 +138.439502,544140.375 +138.459936,544151.375 +138.480370,544148.875 +138.500804,544193.312 +138.521239,544206.688 +138.541673,544046.750 +138.562107,544009.000 +138.582541,543968.875 +138.602975,543943.312 +138.623409,543909.312 +138.643844,543962.000 +138.664278,543939.188 +138.684712,543888.500 +138.705146,543914.125 +138.725580,543902.500 +138.746014,543914.688 +138.766448,543938.875 +138.786883,543943.562 +138.807317,544002.750 +138.827751,544166.125 +138.848185,544210.188 +138.868619,544144.188 +138.889054,544192.188 +138.909488,544171.750 +138.929922,544117.500 +138.950356,544207.062 +138.970790,544188.125 +138.991224,544177.812 +139.011659,544199.875 +139.032093,544175.062 +139.052527,544197.750 +139.072961,544141.625 +139.093395,544114.688 +139.113829,544058.750 +139.134263,544087.188 +139.154698,544135.562 +139.175132,544154.000 +139.195566,544151.000 +139.216000,544147.312 +139.236434,544138.250 +139.256868,544182.312 +139.277303,544148.688 +139.318171,544170.250 +139.338605,544149.062 +139.359039,544188.500 +139.379473,544134.062 +139.399908,544159.000 +139.420342,544159.188 +139.440776,544152.375 +139.461210,544138.812 +139.481644,544177.312 +139.502078,544194.750 +139.522513,544213.500 +139.542947,544178.812 +139.563381,544201.062 +139.583815,544212.625 +139.604249,544220.312 +139.624683,544187.062 +139.645118,544215.312 +139.665552,544160.188 +139.685986,544164.562 +139.706420,544188.062 +139.726854,544146.125 +139.767723,544182.750 +139.788157,544135.250 +139.808591,544097.312 +139.829025,544106.188 +139.849459,544197.688 +139.869893,544112.562 +139.890327,544168.812 +139.910762,544092.562 +139.931196,544043.438 +139.951630,544029.125 +139.972064,543992.438 +139.992498,544085.125 +140.012932,544108.250 +140.033367,544136.812 +140.053801,544080.000 +140.074235,544129.750 +140.094669,544123.000 +140.115103,544119.062 +140.135537,544101.938 +140.155972,544077.875 +140.176406,544134.062 +140.196840,544133.250 +140.217274,544091.500 +140.237708,544098.250 +140.258142,544119.875 +140.278576,544167.625 +140.299011,544127.688 +140.319445,544192.750 +140.339879,544127.875 +140.380747,544172.500 +140.401182,544114.688 +140.421616,544123.625 +140.442050,544141.938 +140.462484,544140.250 +140.482918,544153.500 +140.503352,544153.875 +140.523786,544123.312 +140.544221,544119.688 +140.564655,544126.750 +140.585089,544163.625 +140.605523,544119.312 +140.625957,544132.688 +140.646391,544176.188 +140.666825,544139.875 +140.687260,544151.562 +140.707694,544123.562 +140.728128,544142.375 +140.748562,544089.625 +140.768996,544055.750 +140.789430,544041.062 +140.809865,544011.938 +140.830299,544129.375 +140.850733,544143.250 +140.891601,544139.625 +140.912035,544132.875 +140.932470,544167.500 +140.952904,544181.188 +140.973338,544158.625 +140.993772,544203.750 +141.014206,544194.438 +141.034640,544188.562 +141.055074,544168.125 +141.075509,544164.812 +141.095943,544137.250 +141.116377,544171.250 +141.136811,544167.812 +141.157245,544136.875 +141.177679,544184.875 +141.198114,544183.500 +141.218548,544157.562 +141.238982,544174.250 +141.259416,544214.250 +141.279850,544156.312 +141.300284,544217.562 +141.320718,544149.750 +141.341153,544159.438 +141.361587,544172.938 +141.382021,544189.688 +141.402455,544213.875 +141.422889,544194.562 +141.463758,544171.875 +141.484192,544189.125 +141.504626,544279.500 +141.525060,544218.500 +141.545494,544215.750 +141.565928,544205.938 +141.586363,544136.188 +141.606797,544101.625 +141.627231,544048.188 +141.647665,544064.938 +141.668099,544187.188 +141.708967,544185.688 +141.729402,544192.125 +141.770270,544184.500 +141.790704,544230.500 +141.811138,544244.375 +141.831572,544218.500 +141.852006,544178.438 +141.872441,544185.250 +141.892875,544157.188 +141.913309,544250.750 +141.933743,544185.375 +141.954177,544218.000 +141.974611,544205.688 +141.995046,544217.500 +142.015480,544158.562 +142.035914,544147.062 +142.056348,544236.438 +142.076782,544163.500 +142.097216,544197.000 +142.117650,544173.062 +142.138085,544201.125 +142.158519,544178.875 +142.178953,544174.125 +142.199387,544212.938 +142.240255,544207.062 +142.260689,544167.750 +142.281124,544184.625 +142.301558,544164.250 +142.321992,544212.188 +142.342426,544187.375 +142.362860,544166.688 +142.383294,544180.375 +142.403729,544180.188 +142.424163,544184.062 +142.444597,544073.875 +142.465031,544070.688 +142.485465,544047.562 +142.505899,544107.375 +142.526333,544186.312 +142.546768,544186.062 +142.567202,544161.250 +142.587636,544216.625 +142.608070,544173.938 +142.628504,544166.438 +142.648938,544177.500 +142.669372,544191.000 +142.689807,544174.688 +142.710241,544163.688 +142.730675,544164.438 +142.751109,544212.750 +142.771543,544131.875 +142.791977,544174.938 +142.812411,544179.188 +142.832846,544209.438 +142.853280,544201.625 +142.873714,544189.188 +142.894148,544188.062 +142.914582,544208.562 +142.935016,544233.812 +142.955450,544202.625 +142.975885,544165.875 +142.996319,544231.312 +143.016753,544219.812 +143.037187,544214.750 +143.057621,544237.875 +143.078055,544234.562 +143.098490,544242.875 +143.118924,544207.812 +143.139358,544203.688 +143.159792,544212.625 +143.180226,544202.438 +143.200660,544215.812 +143.221094,544148.000 +143.241529,544192.812 +143.261963,544138.562 +143.282397,544063.125 +143.302831,544102.062 +143.323265,544135.375 +143.364133,544214.438 +143.384568,544207.688 +143.405002,544180.562 +143.425436,544187.000 +143.445870,544163.000 +143.466304,544200.188 +143.486738,544189.375 +143.507172,544167.750 +143.527607,544211.688 +143.568475,544262.500 +143.588909,544232.000 +143.609343,544197.062 +143.629777,544182.375 +143.650211,544202.312 +143.670646,544255.438 +143.691080,544210.500 +143.711514,544174.188 +143.731948,544177.688 +143.752382,544135.688 +143.772816,544172.812 +143.793250,544242.875 +143.813685,544176.625 +143.834119,544160.000 +143.854553,544182.750 +143.874987,544141.125 +143.895421,544190.000 +143.915855,544179.750 +143.936289,544182.938 +143.956724,544207.062 +143.977158,544193.688 +143.997592,544167.938 +144.018026,544172.688 +144.038460,544203.312 +144.058894,544177.750 +144.099763,544227.500 +144.120197,544041.375 +144.140631,544078.000 +144.161065,544061.000 +144.181499,544174.688 +144.201933,544213.938 +144.222367,544192.375 +144.242802,544162.188 +144.263236,544157.938 +144.283670,544222.375 +144.324538,544163.000 +144.344972,544173.750 +144.365406,544187.875 +144.385840,544164.500 +144.406275,544146.062 +144.426709,544208.125 +144.447143,544173.188 +144.467577,544140.438 +144.488011,544158.375 +144.508445,544210.500 +144.528879,544208.188 +144.549314,544176.562 +144.569748,544189.562 +144.590182,544178.562 +144.631050,544242.125 +144.651484,544191.500 +144.671918,544197.062 +144.692353,544199.062 +144.712787,544209.812 +144.733221,544156.312 +144.753655,544160.062 +144.774089,544159.062 +144.794523,544122.250 +144.814957,544178.812 +144.835392,544149.188 +144.855826,544182.312 +144.876260,544208.125 +144.896694,544185.938 +144.937562,544162.375 +144.957996,544073.250 +144.978431,544073.000 +144.998865,544034.688 +145.019299,544108.000 +145.039733,544170.312 +145.060167,544162.312 +145.080601,544104.688 +145.101035,544148.000 +145.121470,544145.125 +145.141904,544161.750 +145.162338,544176.812 +145.182772,544162.500 +145.203206,544221.812 +145.223640,544137.938 +145.244074,544153.938 +145.264508,544148.562 +145.284943,544123.312 +145.305377,544137.812 +145.325811,544155.250 +145.346245,544130.875 +145.366679,544197.438 +145.387113,544128.625 +145.407547,544148.812 +145.427981,544125.250 +145.448416,544165.938 +145.468850,544122.062 +145.489284,544146.500 +145.509718,544151.938 +145.530152,544139.062 +145.550586,544176.062 +145.571020,544193.688 +145.591454,544129.812 +145.611889,544161.875 +145.632323,544167.250 +145.652757,544169.000 +145.673191,544166.938 +145.693625,544126.875 +145.714059,544205.500 +145.734493,544161.562 +145.754928,544166.625 +145.775362,544071.625 +145.795796,544105.812 +145.816230,544043.875 +145.836664,544063.125 +145.857098,544226.938 +145.877532,544134.312 +145.897966,544172.000 +145.918401,544180.125 +145.938835,544135.625 +145.959269,544188.000 +145.979703,544170.750 +146.000137,544172.375 +146.020571,544098.688 +146.041005,544135.125 +146.061440,544132.188 +146.081874,544163.125 +146.102308,544181.062 +146.122742,544145.625 +146.143176,544158.750 +146.163610,544165.125 +146.184044,544143.250 +146.204478,544143.000 +146.224913,544143.625 +146.265781,544097.938 +146.286215,544203.312 +146.306649,544148.125 +146.347517,544152.062 +146.367951,544175.500 +146.388386,544141.312 +146.408820,544151.500 +146.429254,544148.438 +146.449688,544130.125 +146.470122,544151.625 +146.510990,544171.812 +146.551859,544121.438 +146.572293,544088.375 +146.592727,544120.562 +146.613161,544111.125 +146.633595,544028.062 +146.654029,544082.688 +146.674463,544125.875 +146.694897,544142.938 +146.715332,544109.000 +146.735766,544108.688 +146.756200,544171.625 +146.776634,544148.812 +146.817502,544135.500 +146.837936,544101.188 +146.858371,544100.750 +146.878805,544129.625 +146.899239,544141.500 +146.919673,544107.125 +146.940107,544103.875 +146.960541,544143.812 +146.980975,544135.188 +147.001409,544150.562 +147.021844,544124.375 +147.042278,544151.250 +147.062712,544108.500 +147.083146,544099.438 +147.103580,544145.875 +147.124014,544189.625 +147.144448,544095.938 +147.164882,544122.312 +147.185317,544085.188 +147.205751,544093.938 +147.226185,544101.000 +147.267053,544135.562 +147.307921,544122.375 +147.328355,544144.812 +147.348789,544180.125 +147.369224,544105.688 +147.389658,544128.000 +147.430526,544134.125 +147.450960,544087.750 +147.471394,544113.938 +147.491828,544036.188 +147.532697,544103.000 +147.553131,544118.750 +147.573565,544148.750 +147.593999,544181.000 +147.614433,544096.188 +147.634867,544054.438 +147.655301,544184.062 +147.675736,544158.688 +147.696170,544197.750 +147.716604,544178.875 +147.737038,544138.500 +147.757472,544136.438 +147.798340,544141.500 +147.818774,544143.062 +147.839209,544131.125 +147.859643,544169.000 +147.880077,544118.125 +147.920945,544155.750 +147.941379,544108.000 +147.961813,544156.938 +147.982247,544146.312 +148.023116,544171.312 +148.043550,544174.188 +148.063984,544187.125 +148.084418,544174.938 +148.104852,544205.875 +148.125286,544201.875 +148.145720,544151.750 +148.166154,544137.250 +148.186589,544150.688 +148.207023,544064.500 +148.227457,544143.250 +148.268325,544128.500 +148.309193,544103.188 +148.329627,544012.438 +148.350061,544133.438 +148.370496,544219.312 +148.390930,544163.000 +148.411364,544187.688 +148.431798,544216.875 +148.452232,544158.312 +148.472666,544144.688 +148.493100,544108.375 +148.513534,544164.438 +148.533968,544144.812 +148.554403,544126.250 +148.574837,544143.875 +148.595271,544161.875 +148.615705,544149.062 +148.656573,544144.812 +148.677007,544148.938 +148.697442,544146.812 +148.717876,544154.312 +148.758744,544169.562 +148.779178,544202.125 +148.799612,544199.125 +148.820046,544119.375 +148.840480,544149.188 +148.860914,544219.562 +148.881348,544164.250 +148.901783,544162.312 +148.922217,544139.125 +148.942651,544178.125 +148.963085,544184.000 +148.983519,544179.625 +149.003953,544167.750 +149.024387,544182.250 +149.044821,544131.188 +149.065256,544164.312 +149.106124,544107.562 +149.126558,544096.062 +149.146992,544104.938 +149.167426,544068.625 +149.187860,544045.750 +149.208294,544144.938 +149.228728,544129.812 +149.249163,544124.688 +149.269597,544116.312 +149.290031,544148.875 +149.330899,544173.812 +149.351333,544146.250 +149.371767,544159.188 +149.412636,544168.062 +149.453504,544147.625 +149.473938,544151.750 +149.494372,544156.688 +149.514806,544158.250 +149.535240,544118.750 +149.555674,544138.250 +149.576108,544160.875 +149.596542,544160.125 +149.616976,544141.062 +149.637411,544162.875 +149.657845,544130.250 +149.678279,544159.000 +149.698713,544179.188 +149.719147,544148.562 +149.739581,544103.250 +149.760015,544175.812 +149.780449,544190.500 +149.821318,544180.375 +149.862186,544138.000 +149.882620,544158.062 +149.903054,544153.250 +149.923488,544128.000 +149.943922,544215.875 +149.964356,544102.062 +150.005225,544069.250 +150.025659,544075.750 +150.046093,544145.500 +150.066527,544145.875 +150.086961,544165.438 +150.107395,544139.750 +150.127829,544164.812 +150.148264,544182.188 +150.168698,544147.562 +150.189132,544189.250 +150.209566,544158.125 +150.230000,544173.000 +150.250434,544206.188 +150.270868,544183.312 +150.291302,544203.688 +150.311736,544172.375 +150.332170,544161.000 +150.352604,544146.750 +150.373039,544246.812 +150.393473,544198.125 +150.413907,544171.625 +150.454775,544226.562 +150.475209,544198.000 +150.495643,544241.500 +150.516077,544163.750 +150.556946,544160.750 +150.577380,544181.312 +150.597814,544196.625 +150.618248,544187.188 +150.638682,544220.062 +150.659116,544212.562 +150.679550,544176.875 +150.699984,544230.562 +150.720418,544240.000 +150.740853,544194.500 +150.761287,544178.000 +150.781721,544191.312 +150.822589,544096.125 +150.843023,544066.312 +150.863457,544132.250 +150.883891,544150.188 +150.904325,544146.625 +150.945194,544226.938 +150.965628,544266.875 +150.986062,544182.000 +151.006496,544118.250 +151.026930,544188.875 +151.047364,544199.625 +151.067798,544222.188 +151.088232,544201.438 +151.108666,544199.125 +151.129100,544171.438 +151.149535,544221.938 +151.169969,544197.625 +151.190403,544182.188 +151.210837,544178.938 +151.231271,544178.562 +151.251705,544185.062 +151.272139,544209.625 +151.292573,544226.250 +151.313007,544191.875 +151.333441,544155.875 +151.353876,544175.875 +151.374310,544210.062 +151.394744,544186.312 +151.415178,544237.062 +151.435612,544249.188 +151.456046,544241.250 +151.476480,544195.250 +151.496914,544245.938 +151.517348,544236.562 +151.537782,544237.000 +151.558217,544283.812 +151.578651,544238.875 +151.599085,544207.062 +151.619519,544188.250 +151.639953,544136.062 +151.680821,544081.062 +151.701255,544085.188 +151.721689,544221.562 +151.742124,544187.938 +151.803426,544204.312 +151.823860,544147.188 +151.844294,544188.125 +151.864728,544187.500 +151.885162,544213.062 +151.905596,544199.062 +151.926030,544226.938 +151.946464,544190.125 +151.966899,544199.188 +151.987333,544221.875 +152.007767,544157.688 +152.028201,544188.750 +152.048635,544171.250 +152.069069,544190.688 +152.089503,544162.938 +152.109937,544234.938 +152.130371,544202.625 +152.150805,544114.625 +152.171240,544187.062 +152.191674,544153.125 +152.212108,544227.938 +152.232542,544143.188 +152.252976,544230.312 +152.273410,544182.750 +152.314278,544186.562 +152.355146,544170.500 +152.375580,544166.000 +152.396015,544135.688 +152.416449,544170.000 +152.436883,544200.062 +152.457317,544160.188 +152.477751,544116.812 +152.498185,544073.312 +152.518619,544077.250 +152.539053,544059.875 +152.559487,544147.688 +152.579921,544161.500 +152.600355,544187.312 +152.620790,544169.188 +152.641224,544180.938 +152.682092,544174.938 +152.702526,544162.250 +152.865999,544207.188 +152.886433,544189.875 +152.906867,544159.312 +153.131642,544204.438 +153.152076,544184.500 +153.172510,544207.812 +153.213378,544181.125 +153.233812,544167.812 +153.254246,544208.750 +153.274681,544243.375 +153.295115,544199.625 +153.315549,544141.062 +153.356417,544059.562 +153.376851,544098.500 +153.397285,544168.688 +153.417719,544163.938 +153.438153,544171.375 +153.458587,544123.688 +153.499456,544163.938 +153.540324,544172.938 +153.560758,544141.625 +153.581192,544198.250 +153.601626,544159.688 +153.622060,544135.250 +153.642494,544135.062 +153.662928,544151.188 +153.683362,544160.688 +153.724230,544145.688 +153.744665,544210.062 +153.765099,544177.000 +153.785533,544141.812 +153.805967,544146.812 +153.826401,544142.750 +153.846835,544175.438 +153.867269,544196.875 +153.887703,544203.875 +153.908137,544179.625 +153.928571,544129.812 +153.949005,544189.625 +153.969439,544191.500 +153.989874,544171.688 +154.010308,544131.625 +154.030742,544107.562 +154.051176,544139.062 +154.071610,544162.938 +154.092044,544138.125 +154.132912,544169.312 +154.153346,544060.188 +154.173780,544052.688 +154.194214,544095.250 +154.235083,544127.125 +154.255517,544174.500 +154.275951,544140.812 +154.296385,544167.375 +154.316819,544162.188 +154.337253,544169.000 +154.357687,544169.188 +154.378121,544144.062 +154.398555,544142.062 +154.418989,544160.562 +154.439423,544117.000 +154.459858,544133.125 +154.480292,544094.188 +154.500726,544141.438 +154.521160,544174.688 +154.541594,544215.438 +154.562028,544135.250 +154.582462,544190.438 +154.602896,544148.750 +154.623330,544146.812 +154.643764,544157.938 +154.664198,544130.562 +154.684632,544136.562 +154.705066,544110.250 +154.725501,544155.938 +154.745935,544156.438 +154.766369,544130.938 +154.786803,544124.812 +154.807237,544133.375 +154.827671,544150.250 +154.868539,544183.812 +154.888973,544161.625 +154.909407,544176.438 +154.929841,544184.188 +154.950275,544151.188 +155.032012,544067.062 +155.052446,544060.438 +155.072880,544169.750 +155.093314,544131.250 +155.134182,544097.500 +155.154616,544121.062 +155.175050,544137.188 +155.195484,544111.438 +155.215918,544124.812 +155.236352,544157.500 +155.256787,544105.438 +155.297655,544095.875 +155.318089,544126.188 +155.338523,544129.875 +155.358957,544089.375 +155.379391,544151.875 +155.399825,544116.500 +155.420259,544153.375 +155.440693,544157.000 +155.461127,544132.562 +155.481561,544171.500 +155.522429,544149.188 +155.542864,544086.062 +155.583732,544102.000 +155.604166,544062.188 +155.624600,544089.188 +155.645034,544123.875 +155.665468,544137.938 +155.685902,544167.938 +155.706336,544162.688 +155.726770,544195.938 +155.747204,544116.625 +155.767638,544135.875 +155.788072,544131.125 +155.808507,544098.625 +155.828941,544072.812 +155.849375,544073.812 +155.869809,544090.750 +155.890243,544076.625 +155.910677,544108.312 +155.931111,544115.438 +155.951545,544181.562 +155.971979,544152.000 +155.992413,544120.438 +156.012847,544174.812 +156.033281,544070.750 +156.053715,544142.438 +156.074150,544144.000 +156.094584,544114.625 +156.115018,544131.500 +156.135452,544116.188 +156.155886,544139.375 +156.176320,544158.688 +156.196754,544106.750 +156.217188,544124.188 +156.237622,544092.562 +156.258056,544098.312 +156.278490,544105.062 +156.298924,544158.938 +156.319358,544119.312 +156.339792,544125.688 +156.360226,544205.188 +156.380661,544164.438 +156.401095,544145.250 +156.421529,544126.875 +156.441963,544130.250 +156.462397,544125.375 +156.482831,544193.125 +156.503265,544160.188 +156.523699,544141.750 +156.544133,544097.625 +156.564567,544143.750 +156.585001,544147.438 +156.605435,544137.188 +156.625869,544155.500 +156.646303,544145.375 +156.666738,544120.000 +156.687172,544091.062 +156.707606,544006.688 +156.728040,544080.812 +156.748474,544112.250 +156.768908,544125.750 +156.789342,544138.562 +156.809776,544131.000 +156.830210,544168.688 +156.850644,544143.750 +156.871078,544091.750 +156.891512,544126.812 +156.911946,544136.625 +156.932380,544106.812 +156.952815,544153.688 +156.973248,544124.062 +156.993683,544126.000 +157.014117,544147.250 +157.034551,544110.000 +157.054985,544127.375 +157.075419,544122.688 +157.095853,544094.500 +157.116287,544012.500 +157.136721,544059.188 +157.157155,544118.000 +157.177589,544093.438 +157.198023,544125.375 +157.218457,544150.500 +157.238891,544142.562 +157.259325,544114.562 +157.300194,544143.000 +157.320628,544117.250 +157.341062,544120.875 +157.361496,544148.188 +157.381930,544149.000 +157.402364,544191.500 +157.422798,544170.062 +157.463666,544141.188 +157.484100,544100.125 +157.504534,544097.500 +157.524968,544028.688 +157.545402,544081.500 +157.565836,544052.250 +157.586270,544155.250 +157.606704,544136.562 +157.627138,544133.312 +157.647573,544116.750 +157.688441,544218.125 +157.708875,544124.500 +157.729309,544134.125 +157.749743,544177.812 +157.770177,544148.625 +157.790611,544158.375 +157.811045,544087.625 +157.831479,544144.312 +157.851913,544210.688 +157.872347,544180.125 +157.892781,544136.812 +157.913215,544140.688 +157.954084,544169.500 +157.974518,544157.812 +157.994952,544182.750 +158.015386,544193.562 +158.035820,544207.938 +158.056254,544206.250 +158.076688,544175.875 +158.097122,544164.500 +158.117556,544128.812 +158.137990,544174.500 +158.158424,544183.250 +158.178858,544164.188 +158.199292,544176.750 +158.219726,544237.062 +158.240160,544152.438 +158.301463,544222.312 +158.321897,544187.625 +158.342331,544121.688 +158.362765,544054.188 +158.383199,544073.312 +158.403633,544095.750 +158.424067,544158.375 +158.464935,544186.500 +158.485369,544163.375 +158.505803,544168.750 +158.526237,544182.312 +158.546671,544191.125 +158.567105,544118.250 +158.587539,544168.938 +158.607973,544187.688 +158.628407,544207.500 +158.648841,544171.625 +158.669276,544176.875 +158.689710,544213.250 +158.710144,544150.062 +158.730578,544173.062 +158.751012,544157.000 +158.771446,544261.562 +158.791880,544135.062 +158.812314,544171.500 +158.832748,544199.875 +158.853182,544185.188 +158.873616,544170.812 +158.894050,544162.312 +158.914484,544222.000 +158.934918,544170.375 +158.955352,544165.562 +158.975786,544189.312 +158.996220,544194.500 +159.016654,544191.625 +159.037088,544224.438 +159.057522,544198.625 +159.077957,544154.688 +159.098391,544227.250 +159.118825,544233.062 +159.139259,544186.375 +159.159693,544135.000 +159.180127,544129.750 +159.200561,544072.562 +159.220995,544118.875 +159.241429,544100.062 +159.261863,544217.812 +159.302731,544163.875 +159.323165,544186.125 +159.343599,544174.062 +159.364033,544189.625 +159.384467,544226.438 +159.404901,544160.250 +159.425335,544220.812 +159.445769,544213.125 +159.466204,544190.938 +159.507072,544199.875 +159.527506,544167.875 +159.547940,544203.625 +159.568374,544199.562 +159.588808,544165.250 +159.609242,544207.250 +159.629676,544188.000 +159.670544,544153.250 +159.690978,544126.250 +159.711412,544192.312 +159.731846,544158.688 +159.752280,544270.125 +159.772714,544169.125 +159.793148,544166.188 +159.813582,544169.125 +159.834016,544151.688 +159.854450,544193.250 +159.874884,544260.875 +159.895318,544217.438 +159.915753,544215.312 +159.936187,544223.500 +159.956621,544212.938 +159.977055,544220.688 +159.997489,544194.250 +160.017923,544186.938 +160.038357,544147.312 +160.058791,544126.625 +160.079225,544121.875 +160.099659,544241.438 +160.120093,544233.562 +160.140527,544138.750 +160.160961,544207.938 +160.181395,544173.062 +160.201829,544159.000 +160.222263,544167.188 +160.242697,544205.750 +160.263131,544216.812 +160.283565,544186.438 +160.303999,544201.375 +160.324433,544237.125 +160.365302,544231.688 +160.385735,544210.562 +160.406170,544204.250 +160.426604,544226.438 +160.447038,544183.125 +160.467472,544157.812 +160.487906,544226.812 +160.508340,544192.312 +160.528774,544184.875 +160.549208,544220.625 +160.569642,544219.000 +160.590076,544186.625 +160.610510,544188.250 +160.630944,544224.562 +160.651378,544233.500 +160.671812,544204.562 +160.692246,544219.312 +160.712680,544195.562 +160.733114,544187.500 +160.753548,544196.188 +160.773982,544101.438 +160.794416,544196.750 +160.814850,544194.312 +160.835284,544219.125 +160.855718,544153.000 +160.876153,544127.250 +160.896586,544110.375 +160.917021,544146.750 +160.937455,544170.938 +160.957889,544212.312 +160.978323,544187.375 +160.998757,544178.812 +161.019191,544201.375 +161.039625,544170.562 +161.060059,544188.500 +161.080493,544188.438 +161.100927,544153.938 +161.121361,544167.375 +161.162229,544169.625 +161.182663,544126.250 +161.203097,544187.500 +161.223531,544165.812 +161.264399,544150.188 +161.284833,544192.312 +161.305267,544216.625 +161.325701,544144.125 +161.346135,544168.812 +161.366569,544174.188 +161.387003,544148.000 +161.407437,544193.438 +161.448306,544207.938 +161.468739,544180.188 +161.489174,544172.375 +161.509608,544182.500 +161.530042,544181.688 +161.550476,544180.500 +161.591344,544204.062 +161.611778,544146.812 +161.938722,544151.688 +161.979590,544161.812 +162.000024,544171.250 +162.020458,544161.750 +162.040892,544180.250 +162.061327,544126.812 +162.081760,544163.875 +162.102195,544160.375 +162.122629,544151.562 +162.143063,544181.625 +162.163497,544171.125 +162.183931,544213.500 +162.204365,544172.875 +162.224799,544117.562 +162.245233,544137.438 +162.265667,544153.750 +162.286101,544168.125 +162.306535,544229.125 +162.326969,544187.188 +162.347403,544166.875 +162.367837,544168.438 +162.388271,544154.812 +162.408705,544169.812 +162.429139,544165.062 +162.449573,544191.125 +162.470007,544191.125 +162.490441,544129.000 +162.510875,544204.375 +162.531309,544043.875 +162.551743,544093.062 +162.572177,544113.750 +162.592611,544066.812 +162.613045,544192.375 +164.247766,544074.000 +164.268200,544064.250 +164.309068,544133.188 +164.329502,544144.188 +164.349936,544128.000 +164.370370,544138.875 +164.390804,544121.938 +164.411238,544135.312 +164.431672,544127.688 +164.452106,544083.625 +164.472540,544130.938 +164.533842,544178.125 +164.554276,544185.062 +164.574710,544163.000 +164.595144,544153.875 +164.615578,544173.062 +164.636012,544186.125 +164.656447,544187.938 +164.697315,544143.000 +164.717749,544143.562 +164.738183,544130.125 +164.779051,544141.938 +164.799485,544120.812 +164.819919,544172.625 +164.840353,544174.500 +164.860787,544177.750 +164.881221,544128.062 +164.901655,544174.125 +164.922089,544175.625 +164.942523,544193.750 +164.962957,544180.250 +164.983391,544187.188 diff --git a/src/sapiens/adapters/kepler.py b/src/sapiens/adapters/kepler.py new file mode 100644 index 0000000..2265dce --- /dev/null +++ b/src/sapiens/adapters/kepler.py @@ -0,0 +1,309 @@ +"""Real-data Kepler photometry adapter (Phase 4). + +The first non-synthetic SAPIENS adapter: it re-derives the **already known, +published** Kepler-10 b transit signal (Batalha et al. 2011) from a small +public NASA/MAST Quarter-1 light curve bundled as package data. This is a +validation of a published result against an answer key — **not a discovery, +and nothing here claims one**. The candidate claim is deliberately a +measurement statement ("periodic transit-like dimming"), never a planet +claim. + +Clean-room boundary: the transit arithmetic is ported from this +repository's own Apache-2.0 `demos/ledger-grok` pipeline. Zero code from +ASTRA / GEODISC / BIODISC / SLATE. The adapter registers at CORE trust tier +(first-party clean-room code, real public data). All domain validators live +behind this adapter; the kernel sees only contract evidence. + +Determinism: the analysis has no random component; the contract ``seed`` is +accepted and recorded but does not alter results. +""" + +from __future__ import annotations + +import hashlib +from pathlib import Path + +from sapiens.budget import ExecutionContext +from sapiens.models import AdapterManifest, Candidate, Evidence +from sapiens.validation import HoldoutProtocol + +from . import _transit + +DATA_DIR = Path(__file__).resolve().parent / "data" +DEFAULT_DATA = DATA_DIR / "kepler10_kic11904151_q1_lc.csv" +# Pinned integrity for the bundled public curve (sha256 of the file). +EXPECTED_SHA256 = "1d82ef8ced447bb44ac80c7389a8ef3dde93bf1983d619d56fe2f6cda3a56ded" + +MAST_SOURCE = ( + "https://mast.stsci.edu (NASA Kepler public archive): KIC 11904151 Q1 PDCSAP" +) +PUBLISHED_PERIOD_DAYS = 0.8374912 # NASA Exoplanet Archive, Kepler-10 b + +# Gate thresholds (documented, deterministic). +PROPOSE_SNR_FLOOR = 6.0 +INTERNAL_SNR_THRESHOLD = 10.0 +REPLICATION_SNR_THRESHOLD = 5.0 +REPLICATION_PERIOD_TOLERANCE = 0.02 +REVIEW_ODD_EVEN_SIGMA = 3.0 +REVIEW_SECONDARY_SIGMA = 4.0 + +DATASET_FULL = "kepler-q1-full" +DATASET_HALVES = "kepler-q1-holdout-halves" +DATASET_REVIEW = "kepler-q1-review-adversarial" + + +def kepler_holdout_protocol() -> HoldoutProtocol: + """Declared train/holdout split for this domain (Phase-2 wiring).""" + return HoldoutProtocol( + "kepler-q1-v1", + train_datasets=(DATASET_FULL,), + holdout_datasets=(DATASET_HALVES, DATASET_REVIEW), + ) + + +def _identifier(*parts: object) -> str: + return hashlib.sha256("|".join(map(str, parts)).encode()).hexdigest()[:20] + + +def verify_data_integrity(path: Path) -> None: + digest = hashlib.sha256(path.read_bytes()).hexdigest() + if digest != EXPECTED_SHA256: + raise _transit.DataIntegrityError( + f"bundled light curve checksum mismatch: {digest} != {EXPECTED_SHA256}" + ) + + +class KeplerPhotometryAdapter: + """Re-derivation adapter over the bundled public Kepler Q1 light curve.""" + + manifest = AdapterManifest( + name="kepler-photometry", + version="1.0", + domain="kepler-photometry", + vocabulary=("time", "flux", "period", "transit", "depth", "epoch", "snr"), + synthetic_only=False, + code_origin="first-party-clean-room", + data_sources=(MAST_SOURCE,), + ) + + def __init__( + self, + data_path: str | Path | None = None, + *, + nfreq: int = 1500, + verify_checksum: bool = True, + ) -> None: + self._path = Path(data_path) if data_path is not None else DEFAULT_DATA + if verify_checksum and self._path == DEFAULT_DATA: + verify_data_integrity(self._path) + self._nfreq = nfreq + self._cache: dict[str, object] = {} + + # -- data handling ----------------------------------------------------- + + def _curve(self) -> tuple[list[float], list[float], list[float], int]: + if "curve" not in self._cache: + times, fluxes = _transit.load_csv(self._path) + times, fluxes, masked = _transit.apply_mask(times, fluxes) + detrended = _transit.detrend(times, fluxes) + self._cache["curve"] = (times, fluxes, detrended, masked) + return self._cache["curve"] # type: ignore[return-value] + + def _search(self) -> dict: + if "search" not in self._cache: + times, _, y, _ = self._curve() + self._cache["search"] = _transit.bls(times, y, nfreq=self._nfreq) + return self._cache["search"] # type: ignore[return-value] + + # -- DomainAdapter contract --------------------------------------------- + + def propose(self, *, seed: int, limit: int) -> tuple[Candidate, ...]: + """Bounded BLS search; propose the strongest periodic transit candidate. + + Returns an empty tuple when the curve shows no signal above the SNR + floor — proposing nothing is the honest negative result. + """ + if limit <= 0: + return () + times, _, y, _ = self._curve() + search = self._search() + measured = _transit.fold_measure( + times, y, search["period_days"], search["phase_center"], search["q"] + ) + if measured["snr"] < PROPOSE_SNR_FLOOR: + return () + period = search["period_days"] + candidate = Candidate( + _identifier(self.manifest.name, "kic11904151-q1", f"{period:.7f}"), + self.manifest.domain, + "KIC 11904151 Q1 photometry shows periodic transit-like dimming " + f"at period {period:.6f} d (re-derivation of a published signal; " + "not a discovery)", + { + "relation": "periodic-transit", + "arity": 1, + "period": period, + "q": search["q"], + "phase_center": search["phase_center"], + "depth_ppm": measured["depth_ppm"], + "snr": measured["snr"], + "epoch_bkjd": measured["epoch_bkjd"], + }, + source_adapter=self.manifest.name, + ) + return (candidate,)[:limit] + + def validate( + self, candidate: Candidate, *, stage: str, seed: int, context: ExecutionContext + ) -> tuple[Evidence, ...]: + context.checkpoint() + if stage == "internal": + return self._validate_internal(candidate, seed) + if stage == "replication": + return self._validate_replication(candidate, seed, context) + if stage == "review": + return self._validate_review(candidate, seed, context) + raise ValueError(f"unknown stage {stage!r}") + + def import_structure(self, structure: dict[str, object], *, candidate_id: str) -> Candidate: + period = structure.get("period") + return Candidate( + candidate_id, + self.manifest.domain, + "test a transferred periodic-transit structure against the bundled " + "Kepler Q1 light curve (validation, not a discovery)", + { + "relation": "periodic-transit", + "arity": structure.get("arity", 1), + "period": float(period) if period is not None else PUBLISHED_PERIOD_DAYS, + }, + parent_id=str(structure.get("_source_candidate_id", "")) or None, + source_adapter=self.manifest.name, + ) + + # -- staged validators (sandboxed behind the adapter) -------------------- + + def _validate_internal(self, candidate: Candidate, seed: int) -> tuple[Evidence, ...]: + times, _, y, masked = self._curve() + measured = _transit.fold_measure( + times, + y, + float(candidate.parameters["period"]), + float(candidate.parameters["phase_center"]), + float(candidate.parameters["q"]), + ) + snr = float(measured["snr"]) + return ( + Evidence( + _identifier(candidate.candidate_id, "internal", seed), + candidate.candidate_id, + "internal", + snr >= INTERNAL_SNR_THRESHOLD, + "kepler-bls-fold-v1", + DATASET_FULL, + seed, + min(1.0, snr / 50.0), + { + "snr": snr, + "depth_ppm": measured["depth_ppm"], + "n_transits": measured["n_transits"], + "masked_rows": masked, + "mask_events": list(_transit.MASK_EVENTS), + "deterministic": True, + }, + ), + ) + + def _validate_replication( + self, candidate: Candidate, seed: int, context: ExecutionContext + ) -> tuple[Evidence, ...]: + """Independent halves must each re-detect the signal at the same period.""" + times, _, y, masked = self._curve() + period = float(candidate.parameters["period"]) + mid = len(times) // 2 + halves = {"half-a": (times[:mid], y[:mid]), "half-b": (times[mid:], y[mid:])} + details: dict[str, object] = {"masked_rows": masked} + snrs: list[float] = [] + ok = True + for label, (t_half, y_half) in halves.items(): + context.checkpoint() + search = _transit.bls( + t_half, + y_half, + pmin=period * (1 - 0.1), + pmax=period * (1 + 0.1), + nfreq=400, + refine=False, + ) + measured = _transit.fold_measure( + t_half, y_half, search["period_days"], search["phase_center"], search["q"] + ) + period_match = ( + abs(search["period_days"] - period) / period <= REPLICATION_PERIOD_TOLERANCE + ) + half_ok = bool(measured["snr"] >= REPLICATION_SNR_THRESHOLD and period_match) + ok = ok and half_ok + snrs.append(float(measured["snr"])) + details[label] = { + "period_days": search["period_days"], + "snr": measured["snr"], + "period_match": period_match, + "passed": half_ok, + } + score = min(1.0, min(snrs) / 20.0) if snrs else 0.0 + return ( + Evidence( + _identifier(candidate.candidate_id, "replication", seed), + candidate.candidate_id, + "replication", + ok, + "kepler-half-replication-v1", + DATASET_HALVES, + seed, + score, + details, + ), + ) + + def _validate_review( + self, candidate: Candidate, seed: int, context: ExecutionContext + ) -> tuple[Evidence, ...]: + """Adversarial checks: odd/even depth, secondary eclipse, harmonics.""" + times, _, y, _ = self._curve() + period = float(candidate.parameters["period"]) + measured = _transit.fold_measure( + times, + y, + period, + float(candidate.parameters["phase_center"]), + float(candidate.parameters["q"]), + ) + context.checkpoint() + harmonics = _transit.harmonic_powers(times, y, period) + odd_even_sigma = abs(float(measured["odd_even"]["delta_sigma"])) + secondary_sigma = float(measured["secondary"]["sigma"]) + odd_even_ok = odd_even_sigma < REVIEW_ODD_EVEN_SIGMA + secondary_ok = secondary_sigma < REVIEW_SECONDARY_SIGMA + harmonic_ok = harmonics["fundamental"]["power"] >= harmonics["half"]["power"] + passed = bool(odd_even_ok and secondary_ok and harmonic_ok) + return ( + Evidence( + _identifier(candidate.candidate_id, "review", seed), + candidate.candidate_id, + "review", + passed, + "kepler-adversarial-review-v1", + DATASET_REVIEW, + seed, + None, + { + "odd_even_delta_sigma": odd_even_sigma, + "odd_even_ok": odd_even_ok, + "secondary_sigma": secondary_sigma, + "secondary_ok": secondary_ok, + "harmonic_fundamental_power": harmonics["fundamental"]["power"], + "harmonic_half_power": harmonics["half"]["power"], + "harmonic_ok": harmonic_ok, + }, + ), + ) diff --git a/src/sapiens/calibration.py b/src/sapiens/calibration.py new file mode 100644 index 0000000..7ea8377 --- /dev/null +++ b/src/sapiens/calibration.py @@ -0,0 +1,127 @@ +"""Calibration reports (Phase 2): gate performance against ground truth. + +A calibration report runs the validation gates over the seeded-bias fixture +suite and records, per fixture, whether each gate behaved as the fixture's +label demands. The resulting rates — known-bad catch rate, known-good false +-reject rate — are the *only* legitimate basis for confidence aggregation +(see ``sapiens.confidence``). A report also carries its sample counts so a +thin report cannot masquerade as a strong one. +""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import dataclass + +from .fixtures import FixtureKind, SeededFixture +from .validation import check_internal_consistency, check_replication + +KNOWN_BAD_KINDS = (FixtureKind.OVERFIT, FixtureKind.LEAKAGE, FixtureKind.DEGENERATE) + + +@dataclass(frozen=True) +class FixtureOutcome: + fixture_id: str + kind: str + l1_passed: bool + l2_passed: bool + l1_reasons: tuple[str, ...] + l2_reasons: tuple[str, ...] + + +@dataclass(frozen=True) +class CalibrationReport: + """Immutable gate-performance record over a labelled fixture suite.""" + + report_id: str + outcomes: tuple[FixtureOutcome, ...] + known_good_total: int + known_good_accepted: int + known_bad_total: int + known_bad_caught: int + + @property + def catch_rate(self) -> float: + """Fraction of known-bad fixtures at least one gate correctly rejected.""" + return self.known_bad_caught / self.known_bad_total if self.known_bad_total else 0.0 + + @property + def false_reject_rate(self) -> float: + """Fraction of known-good fixtures some gate wrongly rejected.""" + if not self.known_good_total: + return 1.0 # no evidence of reliability: assume the worst, honestly + return 1.0 - self.known_good_accepted / self.known_good_total + + def meets_minimum(self, *, min_known_bad: int, min_known_good: int) -> bool: + return self.known_bad_total >= min_known_bad and self.known_good_total >= min_known_good + + def to_dict(self) -> dict[str, object]: + return { + "report_id": self.report_id, + "known_good_total": self.known_good_total, + "known_good_accepted": self.known_good_accepted, + "known_bad_total": self.known_bad_total, + "known_bad_caught": self.known_bad_caught, + "catch_rate": self.catch_rate, + "false_reject_rate": self.false_reject_rate, + "outcomes": [ + { + "fixture_id": o.fixture_id, + "kind": o.kind, + "l1_passed": o.l1_passed, + "l2_passed": o.l2_passed, + } + for o in self.outcomes + ], + } + + +def _report_id(outcomes: tuple[FixtureOutcome, ...]) -> str: + canonical = json.dumps( + [(o.fixture_id, o.l1_passed, o.l2_passed) for o in outcomes], + separators=(",", ":"), + ) + return hashlib.sha256(canonical.encode()).hexdigest()[:16] + + +def run_calibration(fixtures: tuple[SeededFixture, ...]) -> CalibrationReport: + """Evaluate both gates on every fixture and tally against labels.""" + outcomes: list[FixtureOutcome] = [] + good_total = good_accepted = bad_total = bad_caught = 0 + for fixture in fixtures: + l1 = check_internal_consistency(fixture.internal) + l2 = check_replication(fixture.internal, fixture.replication, fixture.protocol) + if l1.passed != fixture.expect_l1_pass or l2.passed != fixture.expect_l2_pass: + raise AssertionError( + f"fixture {fixture.fixture_id!r} label disagrees with gates: " + f"expected L1={fixture.expect_l1_pass} L2={fixture.expect_l2_pass}, " + f"got L1={l1.passed} L2={l2.passed} — fix the fixture or the gate" + ) + outcomes.append( + FixtureOutcome( + fixture.fixture_id, + fixture.kind.value, + l1.passed, + l2.passed, + l1.reasons, + l2.reasons, + ) + ) + if fixture.kind == FixtureKind.KNOWN_GOOD: + good_total += 1 + if l1.passed and l2.passed: + good_accepted += 1 + elif fixture.kind in KNOWN_BAD_KINDS: + bad_total += 1 + if not (l1.passed and l2.passed): + bad_caught += 1 + result = tuple(outcomes) + return CalibrationReport( + _report_id(result), + result, + good_total, + good_accepted, + bad_total, + bad_caught, + ) diff --git a/src/sapiens/catchrate.py b/src/sapiens/catchrate.py new file mode 100644 index 0000000..20af3e0 --- /dev/null +++ b/src/sapiens/catchrate.py @@ -0,0 +1,114 @@ +"""Catch-rate scoring for review panels (Phase 3). + +Runs a panel over the labelled seeded-bias fixture suite and reports, per +role and for the panel as a whole, how often known-bad candidates draw +substantive (MAJOR/BLOCKING) objections and known-good candidates are +approved. Rates are exact for the fixtures included — they are *not* +estimates of performance on unseeded candidates, and the report says so. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from .calibration import KNOWN_BAD_KINDS +from .fixtures import FixtureKind, SeededFixture +from .review import PanelOutcome, ReviewPanel, Severity + +CAVEAT = ( + "Catch rates are exact for the seeded fixture suite only; they do not " + "estimate performance on unseeded candidates." +) + + +@dataclass(frozen=True) +class RoleCatchStats: + role: str + bad_fixtures_objected: int + bad_fixtures_total: int + + @property + def rate(self) -> float: + return ( + self.bad_fixtures_objected / self.bad_fixtures_total + if self.bad_fixtures_total + else 0.0 + ) + + +@dataclass(frozen=True) +class CatchRateReport: + per_role: tuple[RoleCatchStats, ...] + panel_catches: int + known_bad_total: int + known_good_total: int + known_good_approved: int + caveat: str = CAVEAT + + @property + def panel_catch_rate(self) -> float: + return self.panel_catches / self.known_bad_total if self.known_bad_total else 0.0 + + @property + def false_reject_rate(self) -> float: + if not self.known_good_total: + return 1.0 + return 1.0 - self.known_good_approved / self.known_good_total + + def to_dict(self) -> dict[str, object]: + return { + "per_role": { + stats.role: { + "bad_fixtures_objected": stats.bad_fixtures_objected, + "bad_fixtures_total": stats.bad_fixtures_total, + "rate": stats.rate, + } + for stats in self.per_role + }, + "panel_catches": self.panel_catches, + "known_bad_total": self.known_bad_total, + "panel_catch_rate": self.panel_catch_rate, + "known_good_total": self.known_good_total, + "known_good_approved": self.known_good_approved, + "false_reject_rate": self.false_reject_rate, + "caveat": self.caveat, + } + + +def score_panel( + panel: ReviewPanel, fixtures: tuple[SeededFixture, ...], *, seed: int = 0 +) -> CatchRateReport: + """Convene the panel over every fixture and tally against labels.""" + roles = [reviewer.role.value for reviewer in panel.reviewers] + role_catches = {role: 0 for role in roles} + bad_total = catches = good_total = good_approved = 0 + for fixture in fixtures: + evidence = fixture.internal + fixture.replication + report = panel.convene(fixture.candidate, evidence, seed=seed) + if fixture.kind == FixtureKind.KNOWN_GOOD: + good_total += 1 + if report.outcome == PanelOutcome.APPROVED: + good_approved += 1 + elif fixture.kind in KNOWN_BAD_KINDS: + bad_total += 1 + if report.outcome == PanelOutcome.REJECTED: + catches += 1 + substantive_roles = { + objection.role.value + for objection in ( + obj for rnd in report.rounds for obj in rnd.objections() + ) + if objection.severity in (Severity.MAJOR, Severity.BLOCKING) + } + for role in substantive_roles: + if role in role_catches: + role_catches[role] += 1 + return CatchRateReport( + tuple( + RoleCatchStats(role, role_catches[role], bad_total) for role in sorted(role_catches) + ), + catches, + bad_total, + good_total, + good_approved, + ) diff --git a/src/sapiens/checkpoints.py b/src/sapiens/checkpoints.py new file mode 100644 index 0000000..d944b38 --- /dev/null +++ b/src/sapiens/checkpoints.py @@ -0,0 +1,163 @@ +"""Signed ledger checkpoints and external anchor export (Phase 1). + +A checkpoint is a ledger event that summarises the chain so far: the event +count and the current head hash. With an HMAC key (environment-only, never +stored) the checkpoint is *signed*; without one it is still a structural +marker the verifier checks for continuity. + +Honest scope, matching the ledger's own warnings: + +- HMAC is symmetric. A signed checkpoint proves the writer held the local + key; it is **not** a public signature and proves nothing to a third party + who does not hold the key. +- ``export_anchor`` writes the head hash to a separate file so it can be + published or archived elsewhere. Comparing an anchor against a ledger + detects whole-file rewrites by anyone who could not also update the + anchor. Anchoring to a truly external system (timestamping service, + public chain) is a Phase-5 workflow; this module provides the hook. + +The key is read from the environment (``SAPIENS_CHECKPOINT_KEY``) at call +time and is never written to the ledger, the anchor file, logs, or any +artifact. +""" + +from __future__ import annotations + +import hashlib +import hmac +import json +import os +from collections.abc import Mapping +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from .ledger import GENESIS, EvidenceLedger, LedgerEvent + +CHECKPOINT_KEY_ENV = "SAPIENS_CHECKPOINT_KEY" +SIGNATURE_SCHEME = "hmac-sha256" + + +def _canonical(value: Any) -> bytes: + return json.dumps( + value, sort_keys=True, separators=(",", ":"), allow_nan=False, ensure_ascii=False + ).encode("utf-8") + + +def key_from_env(env: Mapping[str, str] | None = None) -> bytes | None: + """Read the HMAC key from the environment. Never log or persist the result.""" + source = env if env is not None else os.environ + raw = source.get(CHECKPOINT_KEY_ENV) + if not raw: + return None + return raw.encode("utf-8") + + +def signature_payload(*, event_count: int, head_hash: str) -> bytes: + return _canonical({"event_count": event_count, "head_hash": head_hash}) + + +def sign(*, event_count: int, head_hash: str, key: bytes) -> str: + if not key: + raise ValueError("signing key must be non-empty") + return hmac.new( + key, signature_payload(event_count=event_count, head_hash=head_hash), hashlib.sha256 + ).hexdigest() + + +@dataclass(frozen=True) +class CheckpointVerification: + """Outcome of checkpoint signature verification. Data, not vibes.""" + + checkpoints: int + signed: int + signatures_verified: int + signatures_unverifiable: tuple[int, ...] # seq numbers: signed, but no key available + signature_failures: tuple[int, ...] # seq numbers: signature does not match + + +def verify_checkpoints( + events: tuple[LedgerEvent, ...], *, key: bytes | None +) -> CheckpointVerification: + """Check every checkpoint event's signature (when verifiable). + + Structural continuity (event count + head hash) is already enforced by + ``EvidenceLedger.verify``; this layer handles only signatures. + """ + checkpoints = signed = verified = 0 + unverifiable: list[int] = [] + failures: list[int] = [] + for event in events: + if event.kind != "checkpoint": + continue + checkpoints += 1 + signature = event.payload.get("signature") + if not signature: + continue + signed += 1 + if key is None: + unverifiable.append(event.seq) + continue + expected = sign( + event_count=int(event.payload["event_count"]), + head_hash=str(event.payload["head_hash"]), + key=key, + ) + if hmac.compare_digest(expected, str(signature)): + verified += 1 + else: + failures.append(event.seq) + return CheckpointVerification( + checkpoints=checkpoints, + signed=signed, + signatures_verified=verified, + signatures_unverifiable=tuple(unverifiable), + signature_failures=tuple(failures), + ) + + +def record_checkpoint(ledger: EvidenceLedger, *, key: bytes | None = None) -> LedgerEvent: + """Append a checkpoint over the current chain. Key from env when omitted.""" + if key is None: + key = key_from_env() + events = ledger.events() + head = events[-1].event_hash if events else GENESIS + payload: dict[str, Any] = { + "event_count": len(events), + "head_hash": head, + "signed": key is not None, + "signature": None, + "scheme": SIGNATURE_SCHEME if key is not None else None, + } + if key is not None: + payload["signature"] = sign(event_count=len(events), head_hash=head, key=key) + return ledger.append("checkpoint", "__ledger__", payload) + + +def export_anchor(ledger: EvidenceLedger, path: str | Path) -> dict[str, Any]: + """Write the current head hash to a separate anchor file (JSON).""" + events = ledger.events() + head = events[-1].event_hash if events else GENESIS + anchor = { + "kind": "sapiens-ledger-anchor", + "version": 1, + "event_count": len(events), + "head_hash": head, + } + target = Path(path) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(json.dumps(anchor, indent=2, sort_keys=True) + "\n", encoding="utf-8") + return anchor + + +def verify_anchor(ledger: EvidenceLedger, path: str | Path) -> bool: + """True iff the anchor file matches the ledger's current head.""" + try: + anchor = json.loads(Path(path).read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise ValueError(f"unreadable anchor file: {exc}") from exc + if anchor.get("kind") != "sapiens-ledger-anchor" or anchor.get("version") != 1: + raise ValueError("not a sapiens ledger anchor file") + events = ledger.events() + head = events[-1].event_hash if events else GENESIS + return anchor.get("head_hash") == head and anchor.get("event_count") == len(events) diff --git a/src/sapiens/confidence.py b/src/sapiens/confidence.py new file mode 100644 index 0000000..853eedb --- /dev/null +++ b/src/sapiens/confidence.py @@ -0,0 +1,73 @@ +"""Confidence aggregation (Phase 2) — only on top of calibration data. + +There is no confidence without calibration. ``aggregate_confidence`` raises +:class:`UncalibratedError` unless handed a :class:`CalibrationReport` built +from enough labelled fixtures; the gates' *demonstrated* catch rate is the +only discount applied, and the formula is a documented heuristic, not a +probability estimate. We do not invent precision. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from .calibration import CalibrationReport +from .models import Evidence + +FORMULA_VERSION = "catch-rate-discount-v1" + + +class UncalibratedError(RuntimeError): + """No trustworthy calibration data exists; refuse to emit a number.""" + + +@dataclass(frozen=True) +class CalibratedConfidence: + """A bounded heuristic score with its full provenance attached.""" + + value: float # raw_pass_fraction * catch_rate, in [0, 1] + raw_pass_fraction: float + catch_rate: float + false_reject_rate: float + calibration_report_id: str + formula: str + caveat: str = ( + "Heuristic score, not a probability: raw pass fraction discounted by " + "the gates' demonstrated catch rate on seeded fixtures." + ) + + +def aggregate_confidence( + evidence: tuple[Evidence, ...], + calibration: CalibrationReport | None, + *, + min_known_bad: int = 2, + min_known_good: int = 1, +) -> CalibratedConfidence: + """Aggregate evidence into a calibrated score, or refuse. + + Refusal is the point: without a calibration report meeting minimum + fixture counts, any number would be invented precision. + """ + if calibration is None: + raise UncalibratedError("no calibration report supplied; confidence refused") + if not calibration.meets_minimum( + min_known_bad=min_known_bad, min_known_good=min_known_good + ): + raise UncalibratedError( + f"calibration report {calibration.report_id} is too thin " + f"(known_bad={calibration.known_bad_total}, " + f"known_good={calibration.known_good_total}); confidence refused" + ) + if not evidence: + raise UncalibratedError("no evidence supplied; confidence refused") + raw = sum(1 for item in evidence if item.passed) / len(evidence) + value = raw * calibration.catch_rate + return CalibratedConfidence( + value=value, + raw_pass_fraction=raw, + catch_rate=calibration.catch_rate, + false_reject_rate=calibration.false_reject_rate, + calibration_report_id=calibration.report_id, + formula=FORMULA_VERSION, + ) diff --git a/src/sapiens/fixtures.py b/src/sapiens/fixtures.py new file mode 100644 index 0000000..72b527d --- /dev/null +++ b/src/sapiens/fixtures.py @@ -0,0 +1,141 @@ +"""Seeded-bias fixture suite (Phase 2). + +Deterministic, labelled candidates with planted failure modes, used to +calibrate the validation gates (Phase 2) and to score review-panel catch +rates (Phase 3). Every fixture declares its expected gate outcome; the +calibration report is only meaningful because this ground truth exists. + +Fixtures are pure evidence data — no adapter behaviour is simulated — so the +suite is stable across Python versions and never touches the ledger. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum + +from .models import Candidate, Evidence +from .validation import HoldoutProtocol, synthetic_holdout_protocol + +PROTOCOL = synthetic_holdout_protocol() +DOMAIN = "synthetic-fixtures" + + +class FixtureKind(Enum): + KNOWN_GOOD = "known-good" # honest evidence; gates must accept + OVERFIT = "overfit" # passes train, fails holdout; L2 must reject + LEAKAGE = "leakage" # train/holdout boundary violated; L2 must reject + DEGENERATE = "degenerate" # constant scores across seeds; L1 must reject + + +@dataclass(frozen=True) +class SeededFixture: + fixture_id: str + kind: FixtureKind + candidate: Candidate + internal: tuple[Evidence, ...] + replication: tuple[Evidence, ...] + protocol: HoldoutProtocol + expect_l1_pass: bool + expect_l2_pass: bool + rationale: str + + +def _evidence( + fixture: str, stage: str, runs: tuple[tuple[int, str, float, bool], ...] +) -> tuple[Evidence, ...]: + return tuple( + Evidence( + f"{fixture}-{stage}-{index}", + f"cand-{fixture}", + stage, + passed, + f"fixture-{stage}-v1", + dataset, + seed, + score, + {"fixture": fixture}, + ) + for index, (seed, dataset, score, passed) in enumerate(runs) + ) + + +def _candidate(fixture: str, claim: str) -> Candidate: + return Candidate(f"cand-{fixture}", DOMAIN, claim, source_adapter="fixture-suite") + + +def fixture_suite() -> tuple[SeededFixture, ...]: + """The canonical seeded-bias suite. Deterministic; do not mutate.""" + good = SeededFixture( + "good-1", + FixtureKind.KNOWN_GOOD, + _candidate("good-1", "honest signal replicates on holdout"), + _evidence( + "good-1", + "internal", + ((101, "synthetic-train", 0.91, True), (102, "synthetic-train", 0.89, True)), + ), + _evidence( + "good-1", + "replication", + ((201, "synthetic-holdout", 0.88, True), (202, "synthetic-holdout", 0.9, True)), + ), + PROTOCOL, + expect_l1_pass=True, + expect_l2_pass=True, + rationale="varying scores, clean split, independent seeds, all runs pass", + ) + overfit = SeededFixture( + "overfit-1", + FixtureKind.OVERFIT, + _candidate("overfit-1", "signal that only exists in the training split"), + _evidence( + "overfit-1", + "internal", + ((101, "synthetic-train", 0.95, True), (102, "synthetic-train", 0.93, True)), + ), + _evidence( + "overfit-1", + "replication", + ((201, "synthetic-holdout", 0.31, False), (202, "synthetic-holdout", 0.28, False)), + ), + PROTOCOL, + expect_l1_pass=True, + expect_l2_pass=False, + rationale="strong on train, collapses on holdout — L2 must catch it", + ) + leakage = SeededFixture( + "leakage-1", + FixtureKind.LEAKAGE, + _candidate("leakage-1", "'replication' run on the training data itself"), + _evidence("leakage-1", "internal", ((101, "synthetic-train", 0.92, True),)), + _evidence("leakage-1", "replication", ((101, "synthetic-train", 0.92, True),)), + PROTOCOL, + expect_l1_pass=True, + expect_l2_pass=False, + rationale="same dataset and seed on both sides — leakage controls must fire", + ) + degenerate = SeededFixture( + "degenerate-1", + FixtureKind.DEGENERATE, + _candidate("degenerate-1", "constant score regardless of seed"), + _evidence( + "degenerate-1", + "internal", + ( + (101, "synthetic-train", 0.5, True), + (102, "synthetic-train", 0.5, True), + (103, "synthetic-train", 0.5, True), + ), + ), + _evidence( + "degenerate-1", + "replication", + ((201, "synthetic-holdout", 0.5, True),), + ), + PROTOCOL, + expect_l1_pass=False, + expect_l2_pass=True, + rationale="identical score across three independent seeds — no information", + ) + return (good, overfit, leakage, degenerate) diff --git a/src/sapiens/isolation.py b/src/sapiens/isolation.py new file mode 100644 index 0000000..520cf6f --- /dev/null +++ b/src/sapiens/isolation.py @@ -0,0 +1,229 @@ +"""Subprocess isolation with OS-level resource limits for untrusted adapters (Phase 1). + +UNTRUSTED-tier adapters never execute in the kernel's process. Instead the +kernel serialises the work order (adapter location, candidate, stage, seed, +budget) to a child Python process that: + +1. applies POSIX rlimits to *itself* before importing any adapter code + (CPU seconds, address space, open files — thread-safe, unlike preexec_fn), +2. loads the adapter class from its source file, constructs it with no + arguments, and runs ``validate`` under a fresh ``ExecutionContext``, +3. prints exactly one JSON result line on stdout (adapter stdout is + redirected to stderr so a noisy adapter cannot corrupt the protocol). + +The parent enforces a wall-clock timeout and maps every failure — crash, +limit kill, timeout, malformed output — to :class:`IsolationError`. Isolation +is fail-closed: a contained failure produces no evidence. + +Honest limits: rlimits bound resource use; they are not a security sandbox. +A malicious adapter could still attempt network or filesystem mischief within +those bounds — which is why third-party code additionally requires a recorded +owner permission before it may run at all. +""" + +from __future__ import annotations + +import importlib.util +import inspect +import json +import subprocess +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from .budget import ExecutionContext +from .models import Candidate, Evidence + +PROTOCOL_VERSION = 1 + + +class IsolationError(RuntimeError): + """An isolated run failed or its result could not be trusted.""" + + +@dataclass(frozen=True) +class ResourceLimits: + cpu_seconds: int = 30 + address_space_bytes: int = 1 << 30 # 1 GiB + max_open_files: int = 64 + + def __post_init__(self) -> None: + if self.cpu_seconds <= 0 or self.address_space_bytes <= 0 or self.max_open_files <= 0: + raise ValueError("resource limits must be positive") + + +DEFAULT_LIMITS = ResourceLimits() + + +def _apply_limits(limits: ResourceLimits) -> None: + import resource + + resource.setrlimit(resource.RLIMIT_CPU, (limits.cpu_seconds, limits.cpu_seconds)) + resource.setrlimit( + resource.RLIMIT_AS, (limits.address_space_bytes, limits.address_space_bytes) + ) + resource.setrlimit(resource.RLIMIT_NOFILE, (limits.max_open_files, limits.max_open_files)) + + +def _load_adapter(path: str, qualname: str) -> Any: + if "" in qualname: + raise IsolationError("isolated adapters must be module-level classes") + spec = importlib.util.spec_from_file_location("sapiens_isolated_adapter", path) + if spec is None or spec.loader is None: + raise IsolationError(f"cannot load adapter module from {path!r}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + obj: Any = module + for part in qualname.split("."): + obj = getattr(obj, part, None) + if obj is None: + raise IsolationError(f"adapter class {qualname!r} not found in {path!r}") + return obj() + + +def _serialize_evidence(item: Evidence) -> dict[str, Any]: + return { + "evidence_id": item.evidence_id, + "candidate_id": item.candidate_id, + "kind": item.kind, + "passed": item.passed, + "protocol": item.protocol, + "dataset": item.dataset, + "seed": item.seed, + "score": item.score, + "details": dict(item.details), + } + + +def _child_main() -> None: + """Entry point: ``python -m sapiens.isolation`` (reads the work order on stdin).""" + try: + spec = json.loads(sys.stdin.read()) + if spec.get("protocol") != PROTOCOL_VERSION: + raise IsolationError("isolation protocol version mismatch") + limits = ResourceLimits(**spec["limits"]) + _apply_limits(limits) + adapter = _load_adapter(spec["adapter_path"], spec["adapter_qualname"]) + raw = spec["candidate"] + candidate = Candidate( + raw["candidate_id"], + raw["domain"], + raw["claim"], + raw.get("parameters") or {}, + raw.get("parent_id"), + raw.get("source_adapter", ""), + ) + budget = spec["budget"] + context = ExecutionContext( + max_steps=int(budget["max_steps"]), max_seconds=float(budget["max_seconds"]) + ) + import contextlib + + with contextlib.redirect_stdout(sys.stderr): + evidence = adapter.validate( + candidate, stage=spec["stage"], seed=int(spec["seed"]), context=context + ) + payload = [_serialize_evidence(item) for item in evidence] + # Fail loudly inside the child if anything is not canonical-JSON safe. + result = json.dumps( + {"ok": True, "evidence": payload}, allow_nan=False, ensure_ascii=False + ) + except BaseException as exc: # contained: report, never traceback-spam the parent + result = json.dumps({"ok": False, "error": f"{type(exc).__name__}: {exc}"}) + sys.stdout.write(result + "\n") + sys.stdout.flush() + + +def adapter_location(adapter: object) -> tuple[str, str]: + """(source file, qualname) for an adapter instance; raises if not locatable.""" + cls = type(adapter) + path = inspect.getsourcefile(cls) + if path is None or not Path(path).is_file(): + raise IsolationError(f"cannot locate source file for adapter class {cls.__name__!r}") + qualname = cls.__qualname__ + if "" in qualname: + raise IsolationError("isolated adapters must be defined at module level") + return path, qualname + + +def run_validate_isolated( + adapter: object, + candidate: Candidate, + *, + stage: str, + seed: int, + context: ExecutionContext, + limits: ResourceLimits = DEFAULT_LIMITS, + timeout_seconds: float = 60.0, +) -> tuple[Evidence, ...]: + """Run ``adapter.validate`` in a resource-limited subprocess. Fail-closed.""" + if sys.platform == "win32": + raise IsolationError("subprocess isolation requires POSIX rlimits") + if timeout_seconds <= 0: + raise ValueError("timeout must be positive") + path, qualname = adapter_location(adapter) + spec = { + "protocol": PROTOCOL_VERSION, + "adapter_path": path, + "adapter_qualname": qualname, + "candidate": { + "candidate_id": candidate.candidate_id, + "domain": candidate.domain, + "claim": candidate.claim, + "parameters": dict(candidate.parameters), + "parent_id": candidate.parent_id, + "source_adapter": candidate.source_adapter, + }, + "stage": stage, + "seed": seed, + "budget": {"max_steps": context.max_steps, "max_seconds": context.max_seconds}, + "limits": { + "cpu_seconds": limits.cpu_seconds, + "address_space_bytes": limits.address_space_bytes, + "max_open_files": limits.max_open_files, + }, + } + try: + order = json.dumps(spec, allow_nan=False) + except (TypeError, ValueError) as exc: + raise IsolationError(f"work order is not JSON-serialisable: {exc}") from exc + try: + proc = subprocess.run( + [sys.executable, "-m", "sapiens.isolation"], + input=order, + capture_output=True, + text=True, + timeout=timeout_seconds, + check=False, + ) + except subprocess.TimeoutExpired as exc: + raise IsolationError( + f"isolated adapter exceeded the {timeout_seconds}s wall-clock timeout" + ) from exc + if proc.returncode != 0: + tail = (proc.stderr or "").strip().splitlines() + detail = tail[-1] if tail else "no stderr" + raise IsolationError( + f"isolated adapter died (exit {proc.returncode}; killed by rlimit or crash): {detail}" + ) + lines = [line for line in proc.stdout.splitlines() if line.strip()] + if len(lines) != 1: + raise IsolationError("isolated adapter produced malformed stdout") + try: + result = json.loads(lines[0]) + except json.JSONDecodeError as exc: + raise IsolationError("isolated adapter produced invalid JSON") from exc + if not result.get("ok"): + raise IsolationError(f"isolated adapter failed: {result.get('error', 'unknown')}") + evidence: list[Evidence] = [] + for raw in result["evidence"]: + try: + evidence.append(Evidence(**raw)) + except (TypeError, ValueError) as exc: + raise IsolationError(f"isolated adapter returned invalid evidence: {exc}") from exc + return tuple(evidence) + + +if __name__ == "__main__": + _child_main() diff --git a/src/sapiens/kernel.py b/src/sapiens/kernel.py index 370489a..40070e7 100644 --- a/src/sapiens/kernel.py +++ b/src/sapiens/kernel.py @@ -2,10 +2,14 @@ from __future__ import annotations -from .adapter import DomainAdapter, validate_adapter +from .adapter import DomainAdapter from .budget import ExecutionContext +from .isolation import run_validate_isolated from .ledger import EvidenceLedger -from .models import Candidate, EvidenceLevel +from .models import Candidate, Evidence, EvidenceLevel +from .registry import AdapterRegistry, TrustTier +from .review import PanelReport, ReviewPanel +from .validation import GateVerdict, ValidationGates _STAGE_BY_LEVEL = { EvidenceLevel.L1: "internal", @@ -15,8 +19,19 @@ class DiscoveryKernel: - def __init__(self, ledger: EvidenceLedger) -> None: + def __init__( + self, + ledger: EvidenceLedger, + registry: AdapterRegistry | None = None, + validation: ValidationGates | None = None, + panel: ReviewPanel | None = None, + ) -> None: self.ledger = ledger + self.registry = registry if registry is not None else AdapterRegistry() + self.validation = validation + self.panel = panel + self.gate_log: list[GateVerdict] = [] + self.panel_log: list[PanelReport] = [] def register(self, candidate: Candidate, *, transferred_from: str | None = None) -> None: self.ledger.record_candidate(candidate.candidate_id, transferred_from=transferred_from) @@ -29,15 +44,22 @@ def validate_next( seed: int, context: ExecutionContext, ) -> EvidenceLevel: - validate_adapter(adapter) + tier = self.registry.validate_adapter(adapter) if candidate.domain != adapter.manifest.domain: raise ValueError("candidate domain does not match adapter") current = self.ledger.state(candidate.candidate_id).level if current >= EvidenceLevel.L3: - raise ValueError("Phase 0 automated kernel cannot promote beyond L3") + raise ValueError("automated kernel cannot promote beyond L3; L4 stays human-gated") target = EvidenceLevel(current + 1) stage = _STAGE_BY_LEVEL[target] - evidence = adapter.validate(candidate, stage=stage, seed=seed, context=context) + if tier == TrustTier.UNTRUSTED: + # Third-party code never runs in this process. A contained + # isolation failure yields no evidence and no promotion. + evidence = run_validate_isolated( + adapter, candidate, stage=stage, seed=seed, context=context + ) + else: + evidence = adapter.validate(candidate, stage=stage, seed=seed, context=context) refs: list[str] = [] for item in evidence: if item.candidate_id != candidate.candidate_id or item.kind != stage: @@ -47,5 +69,77 @@ def validate_next( refs.append(item.evidence_id) if not refs: return current + if not self._passes_validation_gates(candidate, target): + return current + if target == EvidenceLevel.L3 and self.panel is not None: + panel_evidence_id = self._convene_panel(candidate, seed=seed) + if panel_evidence_id is None: + return current # panel rejected; verdict already on the ledger + refs.append(panel_evidence_id) self.ledger.promote(candidate.candidate_id, target, tuple(refs)) return target + + def _convene_panel(self, candidate: Candidate, *, seed: int) -> str | None: + """Convene the L3 panel and record its verdict as review evidence. + + Returns the panel evidence id on approval, None on rejection. The + verdict is ledger-recorded either way — no side channels. + """ + from .review import PanelOutcome + + recorded = self._evidence_for(candidate.candidate_id) + report = self.panel.convene(candidate, recorded, seed=seed) # type: ignore[union-attr] + self.panel_log.append(report) + approved = report.outcome == PanelOutcome.APPROVED + # Unique per attempt: an identical re-convened report must not collide + # with the previously recorded panel verdict (ledger forbids id reuse). + evidence_id = f"panel-{report.report_id}-e{len(self.ledger.events())}" + self.ledger.record_evidence( + Evidence( + evidence_id, + candidate.candidate_id, + "review", + approved, + "review-panel-v1", + "panel-transcript", + seed, + None, + {"report": report.to_dict()}, + ) + ) + return evidence_id if approved else None + + def _evidence_for(self, candidate_id: str) -> tuple[Evidence, ...]: + items: list[Evidence] = [] + for event in self.ledger.events(): + if event.kind == "evidence" and event.candidate_id == candidate_id: + items.append(Evidence(**event.payload)) + return tuple(items) + + def _passes_validation_gates(self, candidate: Candidate, target: EvidenceLevel) -> bool: + """Phase-2 opt-in gates. Verdicts are logged, never fabricated as evidence.""" + if self.validation is None or target not in (EvidenceLevel.L1, EvidenceLevel.L2): + return True + from .validation import check_internal_consistency, check_replication + + recorded = self._evidence_for(candidate.candidate_id) + if target == EvidenceLevel.L1: + verdict = check_internal_consistency(recorded) + self.gate_log.append(verdict) + return verdict.passed + protocol = self.validation.protocol_for(candidate.domain) + if protocol is None: + raise ValueError( + f"validation gates configured but no holdout protocol declared for " + f"domain {candidate.domain!r}; refusing L2 promotion" + ) + internal = tuple(item for item in recorded if item.kind == "internal") + replication = tuple(item for item in recorded if item.kind == "replication") + verdict = check_replication( + internal, + replication, + protocol, + min_pass_fraction=self.validation.min_replication_pass_fraction, + ) + self.gate_log.append(verdict) + return verdict.passed diff --git a/src/sapiens/ledger.py b/src/sapiens/ledger.py index 58c6ec2..a1ccca6 100644 --- a/src/sapiens/ledger.py +++ b/src/sapiens/ledger.py @@ -2,7 +2,9 @@ Hash chaining detects accidental or after-the-fact modification; it does not prove identity, authorship, scientific validity, or resistance to an actor who can rewrite -an entire file. Signatures and external anchoring are roadmap items. +an entire file. Phase 1 adds ``checkpoint`` events (see ``sapiens.checkpoints``): +HMAC-signed or unsigned markers that summarise the chain, plus external anchor +export. Signatures are symmetric and prove key possession, not authorship. """ from __future__ import annotations @@ -19,7 +21,8 @@ from .models import Evidence, EvidenceLevel GENESIS = "0" * 64 -_ALLOWED_KINDS = {"candidate", "evidence", "promotion", "demotion", "transfer"} +_ALLOWED_KINDS = {"candidate", "evidence", "promotion", "demotion", "transfer", "checkpoint"} +CHECKPOINT_ACTOR = "__ledger__" def _canonical(value: Any) -> bytes: @@ -105,6 +108,15 @@ def _verify(events: Iterable[LedgerEvent]) -> None: bool(event.payload.get("passed")), str(event.payload.get("kind", "")), ) + elif event.kind == "checkpoint": + if event.candidate_id != CHECKPOINT_ACTOR: + raise LedgerIntegrityError("checkpoint must be recorded by the ledger actor") + if int(event.payload.get("event_count", -1)) != event.seq - 1: + raise LedgerIntegrityError("checkpoint event count does not match the chain") + if event.payload.get("head_hash") != previous: + raise LedgerIntegrityError("checkpoint head hash does not match the chain") + if bool(event.payload.get("signed")) != bool(event.payload.get("signature")): + raise LedgerIntegrityError("checkpoint signed flag and signature disagree") else: if current is None: raise LedgerIntegrityError("transition references unknown candidate") @@ -155,8 +167,9 @@ def state(self, candidate_id: str) -> CandidateState: level = EvidenceLevel.L0 elif event.kind == "evidence": evidence_ids.add(str(event.payload["evidence_id"])) - else: + elif event.kind in {"promotion", "demotion"}: level = EvidenceLevel(int(event.payload["to_level"])) + # checkpoint events carry no per-candidate state if level is None: raise KeyError(candidate_id) return CandidateState(level, frozenset(evidence_ids)) diff --git a/src/sapiens/models.py b/src/sapiens/models.py index 333ccc5..8d953e2 100644 --- a/src/sapiens/models.py +++ b/src/sapiens/models.py @@ -58,19 +58,39 @@ def __post_init__(self) -> None: object.__setattr__(self, "details", frozen_mapping(self.details)) +CODE_ORIGINS = ("first-party-clean-room", "third-party") + + @dataclass(frozen=True) class AdapterManifest: + """What an adapter is made of. Trust *decisions* live in the registry (Phase 1). + + ``synthetic_only`` describes the data: True means the adapter touches only + deterministic synthetic data. ``code_origin`` describes the code: + first-party-clean-room or third-party. Third-party code additionally + requires ``third_party_source`` so the permission manifest can be checked. + """ + name: str version: str domain: str vocabulary: tuple[str, ...] synthetic_only: bool = True + code_origin: str = "first-party-clean-room" + data_sources: tuple[str, ...] = () + third_party_source: str | None = None def __post_init__(self) -> None: if not self.name or not self.version or not self.domain: raise ValueError("adapter manifest fields must be non-empty") - if not self.synthetic_only: - raise ValueError("Phase 0 accepts synthetic-only adapters") + if self.code_origin not in CODE_ORIGINS: + raise ValueError(f"code_origin must be one of {CODE_ORIGINS}") + if self.synthetic_only and self.data_sources: + raise ValueError("synthetic-only adapters must not declare real data sources") + if self.code_origin == "third-party" and not self.third_party_source: + raise ValueError("third-party adapters must declare third_party_source") + if self.code_origin == "first-party-clean-room" and self.third_party_source: + raise ValueError("clean-room adapters must not declare a third_party_source") @dataclass(frozen=True) diff --git a/src/sapiens/permissions.py b/src/sapiens/permissions.py new file mode 100644 index 0000000..50746eb --- /dev/null +++ b/src/sapiens/permissions.py @@ -0,0 +1,144 @@ +"""Owner-permission/licence manifest for third-party code reuse (Phase 1). + +No third-party code may power an adapter without a recorded permission entry. +The repository ships with an **empty** manifest: zero ASTRA-family permissions +exist, and the clean-room invariant holds until an owner records otherwise. + +The manifest is data, not code: a JSON document listing each grant with its +source, scope, licence, grantor, evidence reference, and validity window. +Entries are immutable once loaded; expiry is checked against an explicit date +(injected, never a hidden clock) so verification is deterministic. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from datetime import date +from pathlib import Path +from typing import Any + +SCHEMA_VERSION = 1 + + +class MissingPermissionError(PermissionError): + """Raised when third-party code is used without a recorded permission.""" + + +class ManifestFormatError(ValueError): + """Raised when a permission manifest is malformed. Fail loudly, never guess.""" + + +def _parse_date(value: Any, *, field: str) -> date | None: + if value is None: + return None + if not isinstance(value, str): + raise ManifestFormatError(f"{field} must be an ISO date string or null") + try: + return date.fromisoformat(value) + except ValueError as exc: + raise ManifestFormatError(f"{field} is not a valid ISO date: {value!r}") from exc + + +@dataclass(frozen=True) +class PermissionEntry: + """One recorded owner grant for third-party material.""" + + source: str # upstream project, e.g. "ASTRA-dev" + scope: str # what the grant covers, e.g. "adapter:astra-photometry" or "module:swarm/*" + licence: str # licence or permission basis, e.g. "MIT" or "written-permission" + granted_by: str # identity of the owner who granted it + reference: str # evidence: URL, document id, or message reference + granted_on: date + expires_on: date | None = None + + def __post_init__(self) -> None: + for name in ("source", "scope", "licence", "granted_by", "reference"): + if not getattr(self, name): + raise ManifestFormatError(f"permission entry field {name} must be non-empty") + if self.expires_on is not None and self.expires_on < self.granted_on: + raise ManifestFormatError("permission expires before it was granted") + + @classmethod + def from_dict(cls, raw: Any) -> PermissionEntry: + if not isinstance(raw, dict): + raise ManifestFormatError("permission entries must be objects") + known = { + "source", "scope", "licence", "granted_by", "reference", "granted_on", "expires_on", + } + unknown = set(raw) - known + if unknown: + raise ManifestFormatError(f"unknown permission entry fields: {sorted(unknown)}") + granted_on = _parse_date(raw.get("granted_on"), field="granted_on") + if granted_on is None: + raise ManifestFormatError("granted_on is required") + return cls( + source=str(raw.get("source", "")), + scope=str(raw.get("scope", "")), + licence=str(raw.get("licence", "")), + granted_by=str(raw.get("granted_by", "")), + reference=str(raw.get("reference", "")), + granted_on=granted_on, + expires_on=_parse_date(raw.get("expires_on"), field="expires_on"), + ) + + def active(self, *, on: date) -> bool: + return self.granted_on <= on and (self.expires_on is None or on <= self.expires_on) + + def covers(self, *, source: str, scope: str) -> bool: + if self.source != source: + return False + if self.scope == scope: + return True + # Prefix wildcard: "module:swarm/*" covers "module:swarm/pheromone_dynamics". + return self.scope.endswith("/*") and scope.startswith(self.scope[:-1]) + + +@dataclass(frozen=True) +class PermissionManifest: + """Immutable set of recorded owner permissions.""" + + entries: tuple[PermissionEntry, ...] = () + + @classmethod + def empty(cls) -> PermissionManifest: + return cls(()) + + @classmethod + def load(cls, path: str | Path) -> PermissionManifest: + try: + raw = json.loads(Path(path).read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + raise ManifestFormatError(f"permission manifest is not valid JSON: {exc}") from exc + if not isinstance(raw, dict): + raise ManifestFormatError("permission manifest must be a JSON object") + version = raw.get("version") + if version != SCHEMA_VERSION: + raise ManifestFormatError(f"unsupported manifest version: {version!r}") + entries_raw = raw.get("entries") + if not isinstance(entries_raw, list): + raise ManifestFormatError("manifest entries must be a list") + entries = tuple(PermissionEntry.from_dict(item) for item in entries_raw) + sources = [(e.source, e.scope) for e in entries] + if len(set(sources)) != len(sources): + raise ManifestFormatError("duplicate (source, scope) permission entries") + return cls(entries) + + def permits(self, *, source: str, scope: str, on: date) -> bool: + return any( + entry.covers(source=source, scope=scope) and entry.active(on=on) + for entry in self.entries + ) + + def require(self, *, source: str, scope: str, on: date) -> PermissionEntry: + for entry in self.entries: + if entry.covers(source=source, scope=scope): + if entry.active(on=on): + return entry + raise MissingPermissionError( + f"permission for {source!r} scope {scope!r} is not active on {on}" + ) + raise MissingPermissionError( + f"no recorded owner permission for {source!r} scope {scope!r}; " + "third-party code requires an explicit permission entry" + ) diff --git a/src/sapiens/registry.py b/src/sapiens/registry.py new file mode 100644 index 0000000..840a4fe --- /dev/null +++ b/src/sapiens/registry.py @@ -0,0 +1,98 @@ +"""Trust-tiered adapter registry (Phase 1). + +Replaces the Phase-0 synthetic-only gate. Three tiers: + +- ``SYNTHETIC``: deterministic synthetic data only. Runs in-process. +- ``CORE``: first-party clean-room code; real data allowed. Runs in-process. +- ``UNTRUSTED``: third-party code. Requires a recorded owner permission + (``PermissionManifest``) *and* subprocess isolation for every execution. + +An adapter's tier derives from its manifest, never from adapter-supplied +claims at call time: synthetic data + any first-party code ⇒ SYNTHETIC; +real data + first-party clean-room code ⇒ CORE; third-party code ⇒ UNTRUSTED. +Explicit registration pins the tier so a manifest edit cannot silently +downgrade a check that already happened. +""" + +from __future__ import annotations + +from datetime import date +from enum import IntEnum + +from .adapter import DomainAdapter +from .permissions import PermissionManifest + + +class TrustTier(IntEnum): + SYNTHETIC = 0 + CORE = 1 + UNTRUSTED = 2 + + +def derive_tier(adapter: DomainAdapter) -> TrustTier: + """Tier from manifest facts alone.""" + manifest = adapter.manifest + if manifest.code_origin == "third-party": + return TrustTier.UNTRUSTED + return TrustTier.SYNTHETIC if manifest.synthetic_only else TrustTier.CORE + + +class AdapterRegistry: + """Validates adapters against trust tiers and recorded permissions.""" + + def __init__( + self, + permissions: PermissionManifest | None = None, + *, + today: date | None = None, + ) -> None: + self._permissions = permissions if permissions is not None else PermissionManifest.empty() + self._today = today if today is not None else date.today() + self._tiers: dict[str, TrustTier] = {} + + @property + def permissions(self) -> PermissionManifest: + return self._permissions + + def _check(self, adapter: DomainAdapter, tier: TrustTier) -> None: + manifest = adapter.manifest + if tier == TrustTier.SYNTHETIC and not manifest.synthetic_only: + raise ValueError( + f"adapter {manifest.name!r} registered SYNTHETIC but declares real data sources" + ) + if tier == TrustTier.CORE and manifest.code_origin != "first-party-clean-room": + raise ValueError( + f"adapter {manifest.name!r} registered CORE but is not first-party clean-room code" + ) + if tier == TrustTier.UNTRUSTED: + self._permissions.require( + source=manifest.third_party_source or "", + scope=f"adapter:{manifest.name}", + on=self._today, + ) + + def register(self, adapter: DomainAdapter, tier: TrustTier) -> None: + """Pin an adapter to an explicit tier after validating manifest coherence.""" + if not isinstance(adapter, DomainAdapter): + raise TypeError("adapter does not implement DomainAdapter") + self._check(adapter, tier) + name = adapter.manifest.name + existing = self._tiers.get(name) + if existing is not None and existing != tier: + raise ValueError(f"adapter {name!r} already registered at tier {existing.name}") + self._tiers[name] = tier + + def tier_of(self, adapter: DomainAdapter) -> TrustTier: + registered = self._tiers.get(adapter.manifest.name) + return registered if registered is not None else derive_tier(adapter) + + def validate_adapter(self, adapter: DomainAdapter) -> TrustTier: + """Phase-1 gate: structural check + tier rules + permission enforcement.""" + if not isinstance(adapter, DomainAdapter): + raise TypeError("adapter does not implement DomainAdapter") + tier = self.tier_of(adapter) + self._check(adapter, tier) + return tier + + def requires_isolation(self, adapter: DomainAdapter) -> bool: + return self.validate_adapter(adapter) == TrustTier.UNTRUSTED diff --git a/src/sapiens/review.py b/src/sapiens/review.py new file mode 100644 index 0000000..7ccf048 --- /dev/null +++ b/src/sapiens/review.py @@ -0,0 +1,270 @@ +"""Structured L3 review panels (Phase 3). + +A panel is a bounded, deterministic, multi-round protocol over typed +reviewer verdicts. Four roles are defined — statistician, domain theorist, +methodologist, devil's advocate — each emitting approve / object / abstain +verdicts with severity-graded objections and rationales. + +Protocol (deliberately strict, documented here and nowhere else): + +1. **Round 1** — every reviewer verdicts independently over the candidate's + recorded evidence. +2. **Objection lifecycle** — an objection is RAISED in its round; in each + later round its raiser must re-affirm it (SUSTAINED) or drop it + (WITHDRAWN). Objections cannot silently vanish: the transcript records + every transition. +3. **Rebuttal rounds** — while any objection stands and the round budget + lasts, the panel convenes another round. Reference reviewers escalate a + re-affirmed MAJOR to BLOCKING; MINOR findings stay MINOR. +4. **Disagreement gate** — the panel approves only if no MAJOR or BLOCKING + objection is sustained in the final round. MINOR objections are recorded + as standing caveats in the transcript but do not block: the gate is + strict on substance, tolerant of caveats. L3 is the last automated rung + before the human L4 gate, so any substantive disagreement rejects. + +Panels produce :class:`PanelReport` data — the kernel may record it as +review evidence, but the report itself is the verdict, never a promotion. +""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import dataclass +from enum import Enum +from typing import Protocol, runtime_checkable + +from .models import Candidate, Evidence + + +class ReviewerRole(Enum): + STATISTICIAN = "statistician" + DOMAIN_THEORIST = "domain-theorist" + METHODOLOGIST = "methodologist" + DEVILS_ADVOCATE = "devils-advocate" + + +class VerdictKind(Enum): + APPROVE = "approve" + OBJECT = "object" + ABSTAIN = "abstain" + + +class Severity(Enum): + MINOR = "minor" + MAJOR = "major" + BLOCKING = "blocking" + + +class ObjectionStatus(Enum): + RAISED = "raised" + SUSTAINED = "sustained" + WITHDRAWN = "withdrawn" + + +class PanelOutcome(Enum): + APPROVED = "approved" + REJECTED = "rejected" + + +@dataclass(frozen=True) +class Objection: + objection_id: str + role: ReviewerRole + severity: Severity + text: str + raised_round: int + + def __post_init__(self) -> None: + if not self.objection_id or not self.text: + raise ValueError("objections require an id and text") + if self.raised_round < 1: + raise ValueError("rounds are 1-indexed") + + @property + def key(self) -> str: + """Identity across rounds: same reviewer, same concern.""" + return hashlib.sha256(f"{self.role.value}|{self.text}".encode()).hexdigest()[:16] + + +@dataclass(frozen=True) +class ReviewerVerdict: + role: ReviewerRole + verdict: VerdictKind + rationale: str + objections: tuple[Objection, ...] = () + + def __post_init__(self) -> None: + if self.verdict != VerdictKind.OBJECT and self.objections: + raise ValueError("only an objecting verdict carries objections") + if self.verdict == VerdictKind.OBJECT and not self.objections: + raise ValueError("an objecting verdict must raise at least one objection") + if not self.rationale: + raise ValueError("every verdict requires a rationale") + for objection in self.objections: + if objection.role != self.role: + raise ValueError("objections must belong to their verdict's role") + + +@dataclass(frozen=True) +class ReviewRound: + round_number: int + verdicts: tuple[ReviewerVerdict, ...] + + def objections(self) -> tuple[Objection, ...]: + return tuple(obj for verdict in self.verdicts for obj in verdict.objections) + + +@runtime_checkable +class Reviewer(Protocol): + """A role-specialized, deterministic reviewer. Pure function of its inputs.""" + + @property + def role(self) -> ReviewerRole: ... + + def review( + self, + candidate: Candidate, + evidence: tuple[Evidence, ...], + *, + round_number: int, + prior_objections: tuple[Objection, ...], + seed: int, + ) -> ReviewerVerdict: ... + + +@dataclass(frozen=True) +class PanelReport: + candidate_id: str + outcome: PanelOutcome + rounds: tuple[ReviewRound, ...] + sustained_blocking: tuple[Objection, ...] + withdrawn: tuple[Objection, ...] + lifecycle: tuple[tuple[str, ObjectionStatus], ...] = () + + def to_dict(self) -> dict[str, object]: + return { + "candidate_id": self.candidate_id, + "outcome": self.outcome.value, + "rounds": [ + { + "round_number": r.round_number, + "verdicts": [ + { + "role": v.role.value, + "verdict": v.verdict.value, + "rationale": v.rationale, + "objections": [ + { + "objection_id": o.objection_id, + "severity": o.severity.value, + "text": o.text, + "raised_round": o.raised_round, + } + for o in v.objections + ], + } + for v in r.verdicts + ], + } + for r in self.rounds + ], + "sustained_blocking": [o.text for o in self.sustained_blocking], + "withdrawn": [o.text for o in self.withdrawn], + "lifecycle": {key: status.value for key, status in self.lifecycle}, + } + + @property + def report_id(self) -> str: + canonical = json.dumps(self.to_dict(), sort_keys=True, separators=(",", ":")) + return hashlib.sha256(canonical.encode()).hexdigest()[:16] + + +class ReviewPanel: + """Convenes reviewers over bounded multi-round protocols. Deterministic.""" + + def __init__(self, reviewers: tuple[Reviewer, ...], *, max_rounds: int = 3) -> None: + if not reviewers: + raise ValueError("a panel requires at least one reviewer") + roles = [reviewer.role for reviewer in reviewers] + if len(set(roles)) != len(roles): + raise ValueError("reviewer roles must be unique on a panel") + if max_rounds < 1: + raise ValueError("max_rounds must be positive") + self._reviewers = reviewers + self._max_rounds = max_rounds + + @property + def reviewers(self) -> tuple[Reviewer, ...]: + return self._reviewers + + def convene( + self, candidate: Candidate, evidence: tuple[Evidence, ...], *, seed: int + ) -> PanelReport: + rounds: list[ReviewRound] = [] + active: dict[str, Objection] = {} # key -> latest objection form + withdrawn: list[Objection] = [] + for round_number in range(1, self._max_rounds + 1): + prior = tuple(active.values()) + verdicts = tuple( + reviewer.review( + candidate, + evidence, + round_number=round_number, + prior_objections=prior, + seed=seed, + ) + for reviewer in self._reviewers + ) + rounds.append(ReviewRound(round_number, verdicts)) + current: dict[str, Objection] = {} + for verdict in verdicts: + for objection in verdict.objections: + key = objection.key + if key in active: + current[key] = Objection( + objection.objection_id, + objection.role, + objection.severity, + objection.text, + active[key].raised_round, + ) + else: + current[key] = objection + for key, previous in active.items(): + if key not in current: + withdrawn.append(previous) + active = current + # Continue convening while any objection stands: rebuttal rounds + # give raisers the chance to sustain, escalate, or withdraw. + # Termination is bounded by the round budget. + if not active: + break # clean consensus (or everything withdrawn) + final = rounds[-1] + sustained_blocking = tuple( + obj for obj in active.values() if obj.severity == Severity.BLOCKING + ) + sustained_substantive = tuple( + obj + for obj in active.values() + if obj.severity in (Severity.MAJOR, Severity.BLOCKING) + ) + approved = not sustained_substantive + lifecycle: list[tuple[str, ObjectionStatus]] = [] + for key, obj in active.items(): + status = ( + ObjectionStatus.RAISED + if obj.raised_round == final.round_number + else ObjectionStatus.SUSTAINED + ) + lifecycle.append((key, status)) + for obj in withdrawn: + lifecycle.append((obj.key, ObjectionStatus.WITHDRAWN)) + return PanelReport( + candidate.candidate_id, + PanelOutcome.APPROVED if approved else PanelOutcome.REJECTED, + tuple(rounds), + sustained_blocking, + tuple(withdrawn), + tuple(lifecycle), + ) diff --git a/src/sapiens/reviewers.py b/src/sapiens/reviewers.py new file mode 100644 index 0000000..49c88b5 --- /dev/null +++ b/src/sapiens/reviewers.py @@ -0,0 +1,347 @@ +"""Deterministic reference reviewers for the L3 panel (Phase 3). + +Four role-specialized reviewers, each a pure function of (candidate, +evidence, round, prior objections, seed). They overlap the Phase-2 gates +deliberately: the statistician re-runs them as an independent check, and +the devil's advocate hunts planted-bias signatures directly. Redundancy is +the point of a panel. + +Escalation policy (uniform): findings are recomputed every round; a MAJOR +finding re-affirmed in round 2 or later escalates to BLOCKING. MINOR stays +MINOR. The seed is accepted for interface uniformity; every check here is +deterministic without it. +""" + +from __future__ import annotations + +from .calibration import CalibrationReport +from .models import Candidate, Evidence +from .review import ( + Objection, + ReviewerRole, + ReviewerVerdict, + ReviewPanel, + Severity, + VerdictKind, +) +from .validation import ( + HoldoutProtocol, + check_internal_consistency, + check_replication, +) + + +def _objection( + role: ReviewerRole, severity: Severity, text: str, round_number: int, index: int +) -> Objection: + return Objection( + f"{role.value}-r{round_number}-{index}", + role, + severity, + text, + round_number, + ) + + +def _escalate(severity: Severity, round_number: int) -> Severity: + if severity == Severity.MAJOR and round_number >= 2: + return Severity.BLOCKING + return severity + + +class StatisticianReviewer: + """Re-runs the Phase-2 gates as an independent statistical check.""" + + role = ReviewerRole.STATISTICIAN + + def __init__( + self, + protocol: HoldoutProtocol | None = None, + calibration: CalibrationReport | None = None, + ) -> None: + self._protocol = protocol + self._calibration = calibration + + def review( + self, + candidate: Candidate, + evidence: tuple[Evidence, ...], + *, + round_number: int, + prior_objections: tuple[Objection, ...], + seed: int, + ) -> ReviewerVerdict: + objections: list[Objection] = [] + internal = tuple(item for item in evidence if item.kind == "internal") + replication = tuple(item for item in evidence if item.kind == "replication") + l1 = check_internal_consistency(evidence) + if not l1.passed: + for reason in l1.reasons: + objections.append( + _objection( + self.role, Severity.BLOCKING, f"L1 gate: {reason}", round_number, 0 + ) + ) + if self._protocol is not None: + l2 = check_replication(internal, replication, self._protocol) + if not l2.passed: + for reason in l2.reasons: + objections.append( + _objection( + self.role, Severity.BLOCKING, f"L2 gate: {reason}", round_number, 1 + ) + ) + if ( + self._calibration is not None + and self._calibration.known_bad_total > 0 + and self._calibration.catch_rate < 1.0 + ): + objections.append( + _objection( + self.role, + Severity.MINOR, + f"calibration report {self._calibration.report_id} shows gates catch " + f"only {self._calibration.catch_rate:.2f} of known-bad fixtures", + round_number, + 2, + ) + ) + if objections: + return ReviewerVerdict( + self.role, + VerdictKind.OBJECT, + f"statistical checks found {len(objections)} issue(s)", + tuple(objections), + ) + return ReviewerVerdict( + self.role, VerdictKind.APPROVE, "statistical gates satisfied" + ) + + +class MethodologistReviewer: + """Protocol and dataset hygiene: consistency within stages, no cross-stage reuse.""" + + role = ReviewerRole.METHODOLOGIST + + def review( + self, + candidate: Candidate, + evidence: tuple[Evidence, ...], + *, + round_number: int, + prior_objections: tuple[Objection, ...], + seed: int, + ) -> ReviewerVerdict: + objections: list[Objection] = [] + if not evidence: + objections.append( + _objection( + self.role, Severity.BLOCKING, "no evidence to review", round_number, 0 + ) + ) + by_stage: dict[str, list[Evidence]] = {} + for item in evidence: + by_stage.setdefault(item.kind, []).append(item) + index = 1 + for stage, items in sorted(by_stage.items()): + protocols = {item.protocol for item in items} + if len(protocols) > 1: + objections.append( + _objection( + self.role, + Severity.BLOCKING, + f"stage {stage!r} mixes protocols {sorted(protocols)}", + round_number, + index, + ) + ) + index += 1 + # Leakage signal: an internal-stage (train) dataset reappearing in a + # later stage. Holdout reuse across replication/review is legitimate. + internal_datasets = {item.dataset for item in by_stage.get("internal", [])} + later_datasets = { + item.dataset + for stage, items in by_stage.items() + if stage != "internal" + for item in items + } + reused = internal_datasets & later_datasets + if reused: + objections.append( + _objection( + self.role, + Severity.BLOCKING, + f"train dataset(s) {sorted(reused)} reappear beyond the internal " + "stage — leakage signal", + round_number, + index, + ) + ) + if objections: + return ReviewerVerdict( + self.role, + VerdictKind.OBJECT, + f"methodology checks found {len(objections)} issue(s)", + tuple(objections), + ) + return ReviewerVerdict( + self.role, VerdictKind.APPROVE, "protocol and dataset hygiene satisfied" + ) + + +class DomainTheoristReviewer: + """Claim coherence: the claim must be expressible in the domain vocabulary.""" + + role = ReviewerRole.DOMAIN_THEORIST + + def __init__(self, vocabulary: tuple[str, ...]) -> None: + if not vocabulary: + raise ValueError("domain theorist requires a vocabulary") + self._vocabulary = vocabulary + + def review( + self, + candidate: Candidate, + evidence: tuple[Evidence, ...], + *, + round_number: int, + prior_objections: tuple[Objection, ...], + seed: int, + ) -> ReviewerVerdict: + haystack = " ".join( + [candidate.claim, *(str(key) for key in candidate.parameters)] + ).lower() + hits = [term for term in self._vocabulary if term.lower() in haystack] + if not hits: + severity = _escalate(Severity.MAJOR, round_number) + return ReviewerVerdict( + self.role, + VerdictKind.OBJECT, + "claim is not expressible in the declared domain vocabulary", + ( + _objection( + self.role, + severity, + f"claim references none of {self._vocabulary}", + round_number, + 0, + ), + ), + ) + return ReviewerVerdict( + self.role, + VerdictKind.APPROVE, + f"claim coheres with domain vocabulary ({', '.join(hits)})", + ) + + +class DevilsAdvocateReviewer: + """Adversarial hunt for seeded-bias signatures. Never satisfied by default.""" + + role = ReviewerRole.DEVILS_ADVOCATE + + def review( + self, + candidate: Candidate, + evidence: tuple[Evidence, ...], + *, + round_number: int, + prior_objections: tuple[Objection, ...], + seed: int, + ) -> ReviewerVerdict: + objections: list[Objection] = [] + # Signature 1: constant scores across independent seeds (degenerate). + by_stage: dict[str, list[Evidence]] = {} + for item in evidence: + by_stage.setdefault(item.kind, []).append(item) + index = 0 + for stage, items in sorted(by_stage.items()): + seeds = {item.seed for item in items} + scores = [item.score for item in items if item.score is not None] + if len(seeds) >= 3 and len(scores) >= 3 and len(set(scores)) == 1: + objections.append( + _objection( + self.role, + Severity.BLOCKING, + f"constant score {scores[0]} across {len(seeds)} seeds in stage " + f"{stage!r} — degenerate signature", + round_number, + index, + ) + ) + index += 1 + # Signature 2: a train (internal-stage) dataset reappearing in a later + # stage (leakage). Holdout reuse across replication/review is fine. + internal_datasets = {item.dataset for item in by_stage.get("internal", [])} + for stage, items in sorted(by_stage.items()): + if stage == "internal": + continue + overlap = internal_datasets & {item.dataset for item in items} + if overlap: + objections.append( + _objection( + self.role, + Severity.BLOCKING, + f"train dataset(s) {sorted(overlap)} reappear in stage " + f"{stage!r} — leakage signature", + round_number, + index, + ) + ) + index += 1 + # Signature 3: everything perfect (all scores 1.0) — too clean to trust. + all_scores = [item.score for item in evidence if item.score is not None] + if all_scores and all(score == 1.0 for score in all_scores): + objections.append( + _objection( + self.role, + _escalate(Severity.MAJOR, round_number), + "every score is exactly 1.0 — implausibly perfect", + round_number, + index, + ) + ) + index += 1 + # Signature 4: thin stages (a single run decides a stage). Caveat only. + thin = [stage for stage, items in sorted(by_stage.items()) if len(items) == 1] + if thin: + objections.append( + _objection( + self.role, + Severity.MINOR, + f"stage(s) {thin} rest on a single run", + round_number, + index, + ) + ) + if objections: + return ReviewerVerdict( + self.role, + VerdictKind.OBJECT, + f"adversarial hunt found {len(objections)} signature(s)", + tuple(objections), + ) + return ReviewerVerdict( + self.role, + VerdictKind.APPROVE, + "adversarial hunt found no seeded-bias signature", + ) + + +def reference_panel( + vocabulary: tuple[str, ...], + protocol: HoldoutProtocol | None = None, + calibration: CalibrationReport | None = None, + *, + max_rounds: int = 3, +) -> ReviewPanel: + """A panel of the four deterministic reference reviewers.""" + return ReviewPanel( + ( + StatisticianReviewer(protocol, calibration), + DomainTheoristReviewer(vocabulary), + MethodologistReviewer(), + DevilsAdvocateReviewer(), + ), + max_rounds=max_rounds, + ) diff --git a/src/sapiens/validation.py b/src/sapiens/validation.py new file mode 100644 index 0000000..45449ae --- /dev/null +++ b/src/sapiens/validation.py @@ -0,0 +1,176 @@ +"""Validation framework v1 (Phase 2): automated L0→L2 gates. + +Two gate families, both pure functions over recorded evidence: + +- **Internal consistency (L1)**: statistical sanity checks — scores present + and in range, determinism (identical protocol/dataset/seed reruns must + agree), degenerate-distribution rejection (constant scores across + independent seeds carry no information). +- **Replication (L2)**: a declared :class:`HoldoutProtocol` per domain plus + explicit leakage controls — replication evidence must come from declared + holdout datasets, and any dataset collision or (dataset, seed) pair reuse + across the train/holdout boundary rejects the gate. + +Gates return :class:`GateVerdict` with explicit reasons; they never fabricate +or mutate evidence. Wiring them into promotion is the kernel's job +(``DiscoveryKernel(validation=...)``); here they stay inspectable and testable +in isolation. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass, field +from types import MappingProxyType + +from .models import Evidence + +INTERNAL_KIND = "internal" +REPLICATION_KIND = "replication" + + +@dataclass(frozen=True) +class GateVerdict: + gate: str + passed: bool + reasons: tuple[str, ...] = () + + def __post_init__(self) -> None: + if self.passed and self.reasons: + raise ValueError("a passing verdict carries no failure reasons") + if not self.passed and not self.reasons: + raise ValueError("a failing verdict must say why") + + +@dataclass(frozen=True) +class HoldoutProtocol: + """Declared train/holdout split for a domain. Leakage controls key off this.""" + + name: str + train_datasets: tuple[str, ...] + holdout_datasets: tuple[str, ...] + + def __post_init__(self) -> None: + if not self.name or not self.train_datasets or not self.holdout_datasets: + raise ValueError("holdout protocol requires a name and both dataset splits") + collision = set(self.train_datasets) & set(self.holdout_datasets) + if collision: + raise ValueError(f"holdout protocol is self-leaking: {sorted(collision)}") + + +def _scores(items: list[Evidence]) -> list[float]: + return [item.score for item in items if item.score is not None] # type: ignore[misc] + + +def check_internal_consistency(evidence: tuple[Evidence, ...]) -> GateVerdict: + """L1 gate: statistical sanity over internal evidence for one candidate.""" + gate = "L1-internal-consistency" + reasons: list[str] = [] + items = [item for item in evidence if item.kind == INTERNAL_KIND] + if not items: + return GateVerdict(gate, False, ("no internal evidence to validate",)) + for item in items: + if item.score is None: + reasons.append(f"evidence {item.evidence_id} carries no score") + elif not 0.0 <= item.score <= 1.0: + reasons.append(f"evidence {item.evidence_id} score out of range") + # Determinism: reruns of the identical protocol/dataset/seed must agree. + by_run: dict[tuple[str, str, int], set[tuple[float | None, bool]]] = {} + for item in items: + key = (item.protocol, item.dataset, item.seed) + by_run.setdefault(key, set()).add((item.score, item.passed)) + for (protocol, dataset, seed), outcomes in by_run.items(): + if len(outcomes) > 1: + reasons.append( + f"non-deterministic rerun: {protocol}/{dataset}/seed={seed} " + f"yielded {len(outcomes)} distinct outcomes" + ) + # Degenerate distribution: identical scores across >=3 independent seeds + # carry no information about the candidate. + seeds = {item.seed for item in items} + scores = _scores(items) + if len(seeds) >= 3 and len(scores) >= 3 and len(set(scores)) == 1: + reasons.append( + f"degenerate evidence: constant score {scores[0]} across {len(seeds)} seeds" + ) + if reasons: + return GateVerdict(gate, False, tuple(reasons)) + return GateVerdict(gate, True) + + +def check_replication( + internal: tuple[Evidence, ...], + replication: tuple[Evidence, ...], + protocol: HoldoutProtocol, + *, + min_pass_fraction: float = 1.0, +) -> GateVerdict: + """L2 gate: holdout discipline + leakage controls over replication evidence.""" + gate = "L2-holdout-replication" + if not 0.0 < min_pass_fraction <= 1.0: + raise ValueError("min_pass_fraction must be in (0, 1]") + reasons: list[str] = [] + train = [item for item in internal if item.kind == INTERNAL_KIND] + holdout = [item for item in replication if item.kind == REPLICATION_KIND] + if not holdout: + return GateVerdict(gate, False, ("no replication evidence to validate",)) + for item in train: + if item.dataset not in protocol.train_datasets: + reasons.append( + f"internal evidence {item.evidence_id} uses undeclared dataset " + f"{item.dataset!r} (protocol {protocol.name!r})" + ) + for item in holdout: + if item.dataset not in protocol.holdout_datasets: + reasons.append( + f"replication evidence {item.evidence_id} uses non-holdout dataset " + f"{item.dataset!r} (protocol {protocol.name!r})" + ) + # Leakage control 1: dataset identifiers must not cross the boundary. + train_ids = {item.dataset for item in train} + holdout_ids = {item.dataset for item in holdout} + collision = train_ids & holdout_ids + if collision: + reasons.append(f"dataset leakage across train/holdout boundary: {sorted(collision)}") + # Leakage control 2: an identical (dataset, seed) pair on both sides means + # the "independent" run literally reproduced the training draw. + train_draws = {(item.dataset, item.seed) for item in train} + holdout_draws = {(item.dataset, item.seed) for item in holdout} + reused = train_draws & holdout_draws + if reused: + reasons.append(f"(dataset, seed) reuse across boundary: {sorted(reused)}") + if holdout: + pass_fraction = sum(1 for item in holdout if item.passed) / len(holdout) + if pass_fraction < min_pass_fraction: + reasons.append( + f"replication pass fraction {pass_fraction:.3f} below " + f"required {min_pass_fraction:.3f}" + ) + if reasons: + return GateVerdict(gate, False, tuple(reasons)) + return GateVerdict(gate, True) + + +@dataclass(frozen=True) +class ValidationGates: + """Opt-in kernel wiring: per-domain holdout protocols + replication threshold.""" + + protocols: Mapping[str, HoldoutProtocol] = field( + default_factory=lambda: MappingProxyType({}) + ) + min_replication_pass_fraction: float = 1.0 + + def __post_init__(self) -> None: + object.__setattr__(self, "protocols", MappingProxyType(dict(self.protocols))) + + def protocol_for(self, domain: str) -> HoldoutProtocol | None: + return self.protocols.get(domain) + + +def synthetic_holdout_protocol() -> HoldoutProtocol: + """The split every shipped synthetic adapter already honours.""" + return HoldoutProtocol( + "synthetic-v1", + train_datasets=("synthetic-train",), + holdout_datasets=("synthetic-holdout",), + ) diff --git a/tests/isolation_doubles.py b/tests/isolation_doubles.py new file mode 100644 index 0000000..4ca6632 --- /dev/null +++ b/tests/isolation_doubles.py @@ -0,0 +1,87 @@ +"""Module-level adapter doubles for isolation tests. + +Isolation loads adapters by source-file path in a child process, so these +must live at module level in an importable file (not inside test functions). +Not a pytest module (no test_ functions); ruff still applies. +""" + +from __future__ import annotations + +import time + +from sapiens.models import AdapterManifest, Candidate, Evidence + + +class WellBehavedAdapter: + manifest = AdapterManifest( + "well-behaved", + "1", + "iso-domain", + ("x",), + synthetic_only=False, + code_origin="third-party", + third_party_source="doubles", + data_sources=("https://example.org/public",), + ) + + def propose(self, *, seed: int, limit: int): + return () + + def validate(self, candidate, *, stage: str, seed: int, context): + context.checkpoint() + return ( + Evidence( + f"ev-{candidate.candidate_id}-{stage}", + candidate.candidate_id, + stage, + True, + "double-protocol", + f"double-{stage}", + seed, + 0.75, + {"isolated": True}, + ), + ) + + def import_structure(self, structure, *, candidate_id: str): + return Candidate(candidate_id, "iso-domain", "claim") + + +class NoisyAdapter(WellBehavedAdapter): + def validate(self, candidate, *, stage: str, seed: int, context): + print("NOISE THAT MUST NOT CORRUPT THE PROTOCOL") # noqa: T201 + return super().validate(candidate, stage=stage, seed=seed, context=context) + + +class CpuHogAdapter(WellBehavedAdapter): + def validate(self, candidate, *, stage: str, seed: int, context): + while True: + pass + + +class MemoryHogAdapter(WellBehavedAdapter): + def validate(self, candidate, *, stage: str, seed: int, context): + blob = bytearray(1 << 31) # 2 GiB, exceeds any test rlimit + return blob # unreachable under limits; keeps linters quiet about unused work + + +class SleeperAdapter(WellBehavedAdapter): + def validate(self, candidate, *, stage: str, seed: int, context): + time.sleep(30) + return () + + +class BadEvidenceAdapter(WellBehavedAdapter): + def validate(self, candidate, *, stage: str, seed: int, context): + return ( + Evidence( + "ev-bad", + candidate.candidate_id, + stage, + True, + "double-protocol", + "double-data", + seed, + 1.5, # out of range: model must reject on the parent side + ), + ) diff --git a/tests/test_boundaries.py b/tests/test_boundaries.py index 269318a..298f513 100644 --- a/tests/test_boundaries.py +++ b/tests/test_boundaries.py @@ -5,6 +5,8 @@ from sapiens.adapter import validate_adapter from sapiens.models import AdapterManifest, Candidate, Evidence +from sapiens.permissions import MissingPermissionError +from sapiens.registry import TrustTier ROOT = Path(__file__).resolve().parents[1] @@ -21,23 +23,69 @@ def test_core_does_not_import_adapters(): ) -def test_phase0_rejects_non_synthetic_adapter(): - class RealAdapter: - @property - def manifest(self): # type: ignore[override] - return AdapterManifest("real", "0", "unsafe", ("x",), synthetic_only=False) +class _RealDataAdapter: + @property + def manifest(self): # type: ignore[override] + return AdapterManifest( + "real", + "0", + "unsafe", + ("x",), + synthetic_only=False, + data_sources=("https://example.org/public-data",), + ) - def propose(self, *, seed: int, limit: int): - return () + def propose(self, *, seed: int, limit: int): + return () - def validate(self, candidate, *, stage: str, seed: int, context): - return () + def validate(self, candidate, *, stage: str, seed: int, context): + return () - def import_structure(self, structure, *, candidate_id: str): - return Candidate(candidate_id, "unsafe", "claim") + def import_structure(self, structure, *, candidate_id: str): + return Candidate(candidate_id, "unsafe", "claim") + +class _ThirdPartyAdapter(_RealDataAdapter): + @property + def manifest(self): # type: ignore[override] + return AdapterManifest( + "third-party-real", + "0", + "unsafe", + ("x",), + synthetic_only=False, + code_origin="third-party", + third_party_source="ASTRA-dev", + data_sources=("https://example.org/public-data",), + ) + + +def test_phase1_accepts_first_party_real_data_adapter_at_core_tier(): + # Phase 1 replaced the synthetic-only gate with trust tiers: first-party + # clean-room code on real data is CORE and runs in-process. + validate_adapter(_RealDataAdapter()) + + +def test_phase1_rejects_third_party_adapter_without_permission(): + with pytest.raises(MissingPermissionError): + validate_adapter(_ThirdPartyAdapter()) + + +def test_phase1_registry_reports_untrusted_tier_for_third_party(): + from sapiens.registry import AdapterRegistry + + registry = AdapterRegistry() + assert registry.tier_of(_ThirdPartyAdapter()) == TrustTier.UNTRUSTED + assert registry.tier_of(_RealDataAdapter()) == TrustTier.CORE + + +def test_manifest_rejects_incoherent_provenance(): + with pytest.raises(ValueError): + AdapterManifest("m", "1", "d", ("x",), code_origin="third-party") # no source + with pytest.raises(ValueError): + AdapterManifest("m", "1", "d", ("x",), code_origin="not-a-real-origin") with pytest.raises(ValueError): - validate_adapter(RealAdapter()) + AdapterManifest("m", "1", "d", ("x",), data_sources=("real-data",)) # synthetic lie def test_evidence_rejects_invalid_confidence_score(): diff --git a/tests/test_calibration_confidence.py b/tests/test_calibration_confidence.py new file mode 100644 index 0000000..8ea754b --- /dev/null +++ b/tests/test_calibration_confidence.py @@ -0,0 +1,96 @@ +import json + +import pytest + +from sapiens.calibration import CalibrationReport, run_calibration +from sapiens.confidence import UncalibratedError, aggregate_confidence +from sapiens.fixtures import FixtureKind, fixture_suite +from sapiens.models import Evidence + + +def test_fixture_suite_labels_match_gate_behaviour(): + # run_calibration raises if any fixture's label disagrees with the gates. + report = run_calibration(fixture_suite()) + assert len(report.outcomes) == 4 + + +def test_calibration_report_rates_match_ground_truth(): + report = run_calibration(fixture_suite()) + assert report.known_good_total == 1 + assert report.known_good_accepted == 1 + assert report.known_bad_total == 3 + assert report.known_bad_caught == 3 + assert report.catch_rate == 1.0 + assert report.false_reject_rate == 0.0 + + +def test_fixture_kinds_cover_documented_failure_modes(): + kinds = {fixture.kind for fixture in fixture_suite()} + assert FixtureKind.KNOWN_GOOD in kinds + assert FixtureKind.OVERFIT in kinds + assert FixtureKind.LEAKAGE in kinds + assert FixtureKind.DEGENERATE in kinds + for fixture in fixture_suite(): + assert fixture.rationale # every fixture explains itself + + +def test_report_is_json_serialisable_and_identified(): + report = run_calibration(fixture_suite()) + blob = json.dumps(report.to_dict()) + assert report.report_id in blob + # Same fixtures ⇒ same report id (deterministic). + assert run_calibration(fixture_suite()).report_id == report.report_id + + +def test_empty_suite_report_is_honest_about_thinness(): + report = run_calibration(()) + assert report.known_bad_total == 0 + assert report.catch_rate == 0.0 + assert report.false_reject_rate == 1.0 # no evidence: assume the worst + assert not report.meets_minimum(min_known_bad=2, min_known_good=1) + + +def ev(eid, passed): + return Evidence(eid, "cand-x", "internal", passed, "p", "synthetic-train", 1, 0.9) + + +def test_confidence_refused_without_calibration(): + with pytest.raises(UncalibratedError, match="no calibration"): + aggregate_confidence((ev("a", True),), None) + + +def test_confidence_refused_with_thin_calibration(): + good_only = tuple(f for f in fixture_suite() if f.kind == FixtureKind.KNOWN_GOOD) + thin = run_calibration(good_only) + with pytest.raises(UncalibratedError, match="too thin"): + aggregate_confidence((ev("a", True),), thin) + + +def test_confidence_refused_without_evidence(): + report = run_calibration(fixture_suite()) + with pytest.raises(UncalibratedError, match="no evidence"): + aggregate_confidence((), report) + + +def test_calibrated_confidence_value_and_provenance(): + report = run_calibration(fixture_suite()) + result = aggregate_confidence((ev("a", True), ev("b", False)), report) + assert result.raw_pass_fraction == 0.5 + assert result.catch_rate == 1.0 + assert result.value == 0.5 # raw * catch_rate + assert result.calibration_report_id == report.report_id + assert "not a probability" in result.caveat + + +def test_confidence_discounted_by_weak_calibration(): + weak = CalibrationReport( + "weak-report", + (), + known_good_total=2, + known_good_accepted=2, + known_bad_total=4, + known_bad_caught=2, + ) + result = aggregate_confidence((ev("a", True),), weak) + assert result.catch_rate == 0.5 + assert result.value == 0.5 # perfect evidence, halved by demonstrated weakness diff --git a/tests/test_catchrate.py b/tests/test_catchrate.py new file mode 100644 index 0000000..a3a594b --- /dev/null +++ b/tests/test_catchrate.py @@ -0,0 +1,53 @@ +"""Catch-rate scoring harness (Phase 3) with regression thresholds.""" + +import json + +from sapiens.catchrate import score_panel +from sapiens.fixtures import fixture_suite +from sapiens.reviewers import reference_panel +from sapiens.validation import synthetic_holdout_protocol + +VOCAB = ("signal", "score", "data", "training", "replication", "holdout") + + +def make_report(): + panel = reference_panel(VOCAB, synthetic_holdout_protocol()) + return score_panel(panel, fixture_suite(), seed=0) + + +def test_panel_catches_every_seeded_bad_fixture(): + # Regression threshold on this suite: exact, not estimated (see caveat). + report = make_report() + assert report.known_bad_total == 3 + assert report.panel_catches == 3 + assert report.panel_catch_rate == 1.0 + + +def test_panel_does_not_false_reject_known_good(): + report = make_report() + assert report.known_good_total == 1 + assert report.known_good_approved == 1 + assert report.false_reject_rate == 0.0 + + +def test_per_role_attribution_is_honest(): + report = make_report() + stats = {entry.role: entry for entry in report.per_role} + assert set(stats) == { + "statistician", + "domain-theorist", + "methodologist", + "devils-advocate", + } + # The theorist's job is coherence, not bias: zero catches is the honest + # result, and the report must show it rather than inflate roles. + assert stats["domain-theorist"].bad_fixtures_objected == 0 + assert stats["statistician"].bad_fixtures_objected == 3 + assert stats["devils-advocate"].bad_fixtures_objected >= 2 + + +def test_report_serialisable_with_caveat(): + report = make_report() + blob = json.dumps(report.to_dict()) + assert "do not estimate" in report.caveat + assert "statistician" in blob diff --git a/tests/test_checkpoints.py b/tests/test_checkpoints.py new file mode 100644 index 0000000..b28211e --- /dev/null +++ b/tests/test_checkpoints.py @@ -0,0 +1,128 @@ +import json + +import pytest + +from sapiens.checkpoints import ( + export_anchor, + key_from_env, + record_checkpoint, + sign, + verify_anchor, + verify_checkpoints, +) +from sapiens.ledger import EvidenceLedger, LedgerIntegrityError +from sapiens.models import Evidence + +KEY = b"test-key-not-a-secret" +OTHER_KEY = b"different-key" + + +def filled_ledger(tmp_path): + ledger = EvidenceLedger(tmp_path / "evidence.jsonl") + ledger.record_candidate("cand-1") + ledger.record_evidence( + Evidence("ev-1", "cand-1", "internal", True, "proto", "data", 7, 0.9) + ) + ledger.promote("cand-1", 1, ("ev-1",)) + return ledger + + +def test_unsigned_checkpoint_roundtrip(tmp_path): + ledger = filled_ledger(tmp_path) + event = record_checkpoint(ledger, key=None) + assert event.kind == "checkpoint" + assert event.payload["event_count"] == 3 + assert event.payload["signed"] is False + assert event.payload["signature"] is None + assert ledger.verify() + report = verify_checkpoints(ledger.events(), key=None) + assert report.checkpoints == 1 and report.signed == 0 + # More events may follow a checkpoint; the chain stays valid. + ledger.record_candidate("cand-2") + assert ledger.verify() + + +def test_signed_checkpoint_verifies_with_key(tmp_path): + ledger = filled_ledger(tmp_path) + record_checkpoint(ledger, key=KEY) + assert ledger.verify() + events = ledger.events() + report = verify_checkpoints(events, key=KEY) + assert report.signed == 1 and report.signatures_verified == 1 + assert not report.signature_failures + + +def test_signed_checkpoint_reports_unverifiable_without_key(tmp_path): + ledger = filled_ledger(tmp_path) + record_checkpoint(ledger, key=KEY) + report = verify_checkpoints(ledger.events(), key=None) + assert report.signed == 1 + assert report.signatures_verified == 0 + assert report.signatures_unverifiable == (4,) + + +def test_wrong_key_detected(tmp_path): + ledger = filled_ledger(tmp_path) + record_checkpoint(ledger, key=KEY) + report = verify_checkpoints(ledger.events(), key=OTHER_KEY) + assert report.signature_failures == (4,) + + +def test_tampered_history_breaks_checkpoint_continuity(tmp_path): + ledger = filled_ledger(tmp_path) + record_checkpoint(ledger, key=KEY) + path = tmp_path / "evidence.jsonl" + lines = path.read_text().splitlines() + # Rewrite a historical payload without fixing the chain. + first = json.loads(lines[0]) + first["payload"]["level"] = 99 + lines[0] = json.dumps(first) + path.write_text("\n".join(lines) + "\n") + with pytest.raises(LedgerIntegrityError): + ledger.verify() + + +def test_forged_checkpoint_event_rejected(tmp_path): + ledger = filled_ledger(tmp_path) + with pytest.raises(LedgerIntegrityError): + # Wrong head hash: verifier catches it at append time. + ledger.append( + "checkpoint", + "__ledger__", + {"event_count": 3, "head_hash": "0" * 64, "signed": False, "signature": None}, + ) + + +def test_anchor_export_and_verify(tmp_path): + ledger = filled_ledger(tmp_path) + anchor_path = tmp_path / "anchor.json" + anchor = export_anchor(ledger, anchor_path) + assert anchor["event_count"] == 3 + assert verify_anchor(ledger, anchor_path) + # Ledger moves on: the anchor no longer matches the head. + ledger.record_candidate("cand-2") + assert not verify_anchor(ledger, anchor_path) + + +def test_anchor_rejects_foreign_file(tmp_path): + ledger = filled_ledger(tmp_path) + bogus = tmp_path / "bogus.json" + bogus.write_text(json.dumps({"kind": "something-else"})) + with pytest.raises(ValueError): + verify_anchor(ledger, bogus) + + +def test_key_from_env_never_returns_empty(monkeypatch): + monkeypatch.delenv("SAPIENS_CHECKPOINT_KEY", raising=False) + assert key_from_env() is None + monkeypatch.setenv("SAPIENS_CHECKPOINT_KEY", "env-key") + assert key_from_env() == b"env-key" + monkeypatch.setenv("SAPIENS_CHECKPOINT_KEY", "") + assert key_from_env() is None + + +def test_sign_is_deterministic_and_key_dependent(): + a = sign(event_count=3, head_hash="ab" * 32, key=KEY) + assert a == sign(event_count=3, head_hash="ab" * 32, key=KEY) + assert a != sign(event_count=3, head_hash="ab" * 32, key=OTHER_KEY) + assert a != sign(event_count=4, head_hash="ab" * 32, key=KEY) diff --git a/tests/test_isolation.py b/tests/test_isolation.py new file mode 100644 index 0000000..fa50e26 --- /dev/null +++ b/tests/test_isolation.py @@ -0,0 +1,104 @@ +import pytest +from isolation_doubles import ( + BadEvidenceAdapter, + CpuHogAdapter, + MemoryHogAdapter, + NoisyAdapter, + SleeperAdapter, + WellBehavedAdapter, +) + +from sapiens.budget import ExecutionContext +from sapiens.isolation import ( + IsolationError, + ResourceLimits, + run_validate_isolated, +) +from sapiens.models import Candidate + +CANDIDATE = Candidate("cand-1", "iso-domain", "isolated validation works", {"k": 1}) +CONTEXT = ExecutionContext(max_steps=10, max_seconds=5.0) +FAST_LIMITS = ResourceLimits(cpu_seconds=2, address_space_bytes=256 << 20, max_open_files=32) + + +def test_well_behaved_adapter_round_trips_evidence(): + evidence = run_validate_isolated( + WellBehavedAdapter(), CANDIDATE, stage="internal", seed=7, context=CONTEXT + ) + assert len(evidence) == 1 + item = evidence[0] + assert item.candidate_id == "cand-1" + assert item.kind == "internal" + assert item.passed is True + assert item.score == 0.75 + assert item.details["isolated"] is True + + +def test_noisy_adapter_stdout_does_not_corrupt_protocol(): + evidence = run_validate_isolated( + NoisyAdapter(), CANDIDATE, stage="internal", seed=7, context=CONTEXT + ) + assert len(evidence) == 1 and evidence[0].passed + + +def test_cpu_hog_killed_by_rlimit(): + with pytest.raises(IsolationError, match="died"): + run_validate_isolated( + CpuHogAdapter(), + CANDIDATE, + stage="internal", + seed=7, + context=CONTEXT, + limits=FAST_LIMITS, + timeout_seconds=20.0, + ) + + +def test_memory_hog_contained_by_rlimit(): + with pytest.raises(IsolationError): + run_validate_isolated( + MemoryHogAdapter(), + CANDIDATE, + stage="internal", + seed=7, + context=CONTEXT, + limits=FAST_LIMITS, + timeout_seconds=20.0, + ) + + +def test_sleeper_killed_by_wall_clock_timeout(): + with pytest.raises(IsolationError, match="wall-clock"): + run_validate_isolated( + SleeperAdapter(), + CANDIDATE, + stage="internal", + seed=7, + context=CONTEXT, + timeout_seconds=2.0, + ) + + +def test_invalid_adapter_evidence_is_contained(): + with pytest.raises(IsolationError): + run_validate_isolated( + BadEvidenceAdapter(), CANDIDATE, stage="internal", seed=7, context=CONTEXT + ) + + +def test_non_serialisable_candidate_parameters_rejected_before_spawn(): + candidate = Candidate("cand-2", "iso-domain", "bad params", {"fn": object()}) + with pytest.raises(IsolationError, match="JSON"): + run_validate_isolated( + WellBehavedAdapter(), candidate, stage="internal", seed=7, context=CONTEXT + ) + + +def test_locally_defined_adapter_cannot_be_isolated(): + class Local(WellBehavedAdapter): + pass + + with pytest.raises(IsolationError, match="module level"): + run_validate_isolated( + Local(), CANDIDATE, stage="internal", seed=7, context=CONTEXT + ) diff --git a/tests/test_kepler_adapter.py b/tests/test_kepler_adapter.py new file mode 100644 index 0000000..da50045 --- /dev/null +++ b/tests/test_kepler_adapter.py @@ -0,0 +1,157 @@ +"""Phase-4 Kepler adapter tests on the bundled public light curve. + +Everything here validates a re-derivation of the published Kepler-10 b +signal. No discovery is claimed anywhere in this file. +""" + +import math +import random + +import pytest + +from sapiens.adapters._transit import DataIntegrityError +from sapiens.adapters.kepler import ( + PUBLISHED_PERIOD_DAYS, + KeplerPhotometryAdapter, + verify_data_integrity, +) +from sapiens.budget import ExecutionContext +from sapiens.models import Candidate + + +@pytest.fixture(scope="module") +def adapter(): + return KeplerPhotometryAdapter() + + +@pytest.fixture(scope="module") +def candidate(adapter): + proposed = adapter.propose(seed=0, limit=1) + assert len(proposed) == 1 + return proposed[0] + + +def ctx(): + return ExecutionContext(max_steps=20, max_seconds=60.0) + + +def test_manifest_declares_core_tier_facts(adapter): + manifest = adapter.manifest + assert manifest.synthetic_only is False + assert manifest.code_origin == "first-party-clean-room" + assert manifest.third_party_source is None + assert any("mast.stsci.edu" in source for source in manifest.data_sources) + + +def test_propose_rederives_published_period(candidate): + period = float(candidate.parameters["period"]) + assert abs(period - PUBLISHED_PERIOD_DAYS) / PUBLISHED_PERIOD_DAYS < 0.01 + assert candidate.parameters["snr"] > 20 + # Honesty: the claim is a measurement statement, never a discovery claim. + assert "not a discovery" in candidate.claim + assert "transit-like dimming" in candidate.claim + + +def test_internal_stage_passes_on_real_data(adapter, candidate): + evidence = adapter.validate(candidate, stage="internal", seed=40, context=ctx()) + assert len(evidence) == 1 + item = evidence[0] + assert item.kind == "internal" + assert item.dataset == "kepler-q1-full" + assert item.passed is True + assert 0.0 <= item.score <= 1.0 + assert item.details["snr"] > 20 + assert item.details["masked_rows"] > 0 # Kepler-10 C mask disclosed + + +def test_replication_stage_redetects_in_independent_halves(adapter, candidate): + evidence = adapter.validate(candidate, stage="replication", seed=41, context=ctx()) + assert len(evidence) == 1 + item = evidence[0] + assert item.dataset == "kepler-q1-holdout-halves" + assert item.passed is True + assert item.details["half-a"]["passed"] is True + assert item.details["half-b"]["passed"] is True + + +def test_review_stage_adversarial_checks_pass(adapter, candidate): + evidence = adapter.validate(candidate, stage="review", seed=42, context=ctx()) + item = evidence[0] + assert item.dataset == "kepler-q1-review-adversarial" + assert item.passed is True + assert item.details["odd_even_ok"] is True + assert item.details["secondary_ok"] is True + assert item.details["harmonic_ok"] is True + + +def test_bundled_curve_checksum_is_pinned(): + from sapiens.adapters.kepler import DEFAULT_DATA + + verify_data_integrity(DEFAULT_DATA) # must not raise + + +def test_tampered_curve_refused(tmp_path): + from sapiens.adapters.kepler import DEFAULT_DATA + + blob = bytearray(DEFAULT_DATA.read_bytes()) + blob[-10] ^= 0xFF + forged = tmp_path / "forged.csv" + forged.write_bytes(bytes(blob)) + with pytest.raises(DataIntegrityError): + verify_data_integrity(forged) + + +def _write_curve(path, times, fluxes): + lines = ["time_bkjd,flux"] + [f"{t:.6f},{f:.8f}" for t, f in zip(times, fluxes, strict=True)] + path.write_text("\n".join(lines) + "\n") + + +def test_flat_curve_proposes_nothing(tmp_path): + rng = random.Random(7) + times = [200.0 + i * 0.02 for i in range(1500)] + fluxes = [1.0 + rng.gauss(0, 0.0002) for _ in times] + curve = tmp_path / "flat.csv" + _write_curve(curve, times, fluxes) + adapter = KeplerPhotometryAdapter(curve) + assert adapter.propose(seed=0, limit=1) == () + + +def _shifted_curve(path): + """Box dips at P=1.000 d in the first half, P=1.060 d in the second.""" + rng = random.Random(11) + times: list[float] = [] + fluxes: list[float] = [] + for i in range(2000): + t = 200.0 + i * 0.02 + period = 1.0 if t < 220.0 else 1.06 + in_dip = math.fmod(t, period) / period < 0.05 + flux = (0.99 if in_dip else 1.0) + rng.gauss(0, 0.0002) + times.append(t) + fluxes.append(flux) + _write_curve(path, times, fluxes) + + +def test_shifted_period_curve_fails_replication_honestly(tmp_path): + curve = tmp_path / "shifted.csv" + _shifted_curve(curve) + adapter = KeplerPhotometryAdapter(curve) + candidate = Candidate( + "cand-shifted", + adapter.manifest.domain, + "synthetic shifted-period control (not real data, not a discovery)", + { + "relation": "periodic-transit", + "arity": 1, + "period": 1.0, + "q": 0.05, + "phase_center": 0.025, + }, + source_adapter="test", + ) + evidence = adapter.validate(candidate, stage="replication", seed=41, context=ctx()) + item = evidence[0] + assert item.passed is False + per_half = {k: v["period_days"] for k, v in item.details.items() if k.startswith("half")} + assert len(per_half) == 2 + # The halves genuinely disagree — the failure is in the data, not the gate. + assert abs(per_half["half-a"] - per_half["half-b"]) / 1.0 > 0.02 diff --git a/tests/test_kepler_integration.py b/tests/test_kepler_integration.py new file mode 100644 index 0000000..152d160 --- /dev/null +++ b/tests/test_kepler_integration.py @@ -0,0 +1,86 @@ +"""Phase-4 integration: registry tier, full kernel climb, bridge L0 reset.""" + +import pytest + +from sapiens.adapters import ( + KeplerPhotometryAdapter, + SyntheticThresholdAdapter, + kepler_holdout_protocol, +) +from sapiens.bridge import transfer +from sapiens.budget import ExecutionContext +from sapiens.kernel import DiscoveryKernel +from sapiens.ledger import EvidenceLedger +from sapiens.models import EvidenceLevel +from sapiens.registry import AdapterRegistry, TrustTier +from sapiens.reviewers import reference_panel +from sapiens.validation import ValidationGates + + +@pytest.fixture(scope="module") +def adapter(): + return KeplerPhotometryAdapter() + + +@pytest.fixture(scope="module") +def candidate(adapter): + return adapter.propose(seed=0, limit=1)[0] + + +def ctx(): + return ExecutionContext(max_steps=30, max_seconds=90.0) + + +class TestRegistryTier: + def test_auto_tiers_core_from_manifest_facts(self, adapter): + registry = AdapterRegistry() + assert registry.validate_adapter(adapter) == TrustTier.CORE + assert not registry.requires_isolation(adapter) + + def test_explicit_core_registration(self, adapter): + registry = AdapterRegistry() + registry.register(adapter, TrustTier.CORE) + assert registry.tier_of(adapter) == TrustTier.CORE + + def test_synthetic_registration_rejected_for_real_data(self, adapter): + registry = AdapterRegistry() + with pytest.raises(ValueError, match="real data sources"): + registry.register(adapter, TrustTier.SYNTHETIC) + + +def test_full_climb_to_l3_with_gates_and_panel(tmp_path, adapter, candidate): + """The capstone: real public data through every shipped gate.""" + gates = ValidationGates({adapter.manifest.domain: kepler_holdout_protocol()}) + panel = reference_panel(adapter.manifest.vocabulary, kepler_holdout_protocol()) + kernel = DiscoveryKernel( + EvidenceLedger(tmp_path / "events.jsonl"), validation=gates, panel=panel + ) + kernel.register(candidate) + assert kernel.validate_next(adapter, candidate, seed=40, context=ctx()) == EvidenceLevel.L1 + assert kernel.validate_next(adapter, candidate, seed=41, context=ctx()) == EvidenceLevel.L2 + assert kernel.validate_next(adapter, candidate, seed=42, context=ctx()) == EvidenceLevel.L3 + # L4 stays human-gated: the automated kernel refuses to go further. + with pytest.raises(ValueError, match="human-gated"): + kernel.validate_next(adapter, candidate, seed=43, context=ctx()) + assert all(verdict.passed for verdict in kernel.gate_log) + assert kernel.panel_log[-1].outcome.value == "approved" + assert kernel.ledger.verify() + + +def test_climb_without_gates_or_panel_also_promotes(tmp_path, adapter, candidate): + kernel = DiscoveryKernel(EvidenceLedger(tmp_path / "events.jsonl")) + kernel.register(candidate) + for seed in (40, 41, 42): + reached = kernel.validate_next(adapter, candidate, seed=seed, context=ctx()) + assert reached == EvidenceLevel.L3 + + +def test_transfer_into_synthetic_domain_resets_to_l0(adapter, candidate): + target = SyntheticThresholdAdapter() + imported, level, envelope = transfer( + candidate, EvidenceLevel.L3, target, candidate_id="kepler-transfer-1" + ) + assert level == EvidenceLevel.L0 # the iron rule, re-verified on real data + assert imported.domain == target.manifest.domain + assert imported.parent_id == candidate.candidate_id + assert envelope.source_level_discarded == EvidenceLevel.L3 diff --git a/tests/test_kernel_gates.py b/tests/test_kernel_gates.py new file mode 100644 index 0000000..5da55d8 --- /dev/null +++ b/tests/test_kernel_gates.py @@ -0,0 +1,133 @@ +"""Kernel + Phase-2 validation gates integration.""" + +import pytest + +from sapiens.adapters import SyntheticLinearAdapter +from sapiens.budget import ExecutionContext +from sapiens.kernel import DiscoveryKernel +from sapiens.ledger import EvidenceLedger +from sapiens.models import AdapterManifest, Candidate, Evidence, EvidenceLevel +from sapiens.validation import ValidationGates, synthetic_holdout_protocol + + +def ctx(): + return ExecutionContext(max_steps=10, max_seconds=10.0) + + +def gated_kernel(tmp_path, adapter_domain, protocol=None): + gates = ValidationGates({adapter_domain: protocol or synthetic_holdout_protocol()}) + return DiscoveryKernel(EvidenceLedger(tmp_path / "events.jsonl"), validation=gates) + + +def test_synthetic_adapter_promotes_through_gates(tmp_path): + adapter = SyntheticLinearAdapter() + kernel = gated_kernel(tmp_path, adapter.manifest.domain) + candidate = adapter.propose(seed=5, limit=1)[0] + kernel.register(candidate) + assert kernel.validate_next(adapter, candidate, seed=40, context=ctx()) == EvidenceLevel.L1 + assert kernel.validate_next(adapter, candidate, seed=41, context=ctx()) == EvidenceLevel.L2 + assert all(verdict.passed for verdict in kernel.gate_log) + + +class LeakyReplicationAdapter: + """Replication 'evidence' produced on the training dataset itself.""" + + manifest = AdapterManifest("leaky", "1", "leaky-domain", ("x",)) + + def propose(self, *, seed: int, limit: int): + return (Candidate("cand-leaky", "leaky-domain", "leaks holdout"),) + + def validate(self, candidate, *, stage: str, seed: int, context): + context.checkpoint() + dataset = "synthetic-train" # same dataset for every stage: leakage + return ( + Evidence( + f"ev-{stage}", candidate.candidate_id, stage, True, "leaky-v1", dataset, seed, 0.9 + ), + ) + + def import_structure(self, structure, *, candidate_id: str): + return Candidate(candidate_id, "leaky-domain", "claim") + + +def test_kernel_blocks_l2_on_dataset_leakage(tmp_path): + adapter = LeakyReplicationAdapter() + kernel = gated_kernel(tmp_path, "leaky-domain") + candidate = adapter.propose(seed=1, limit=1)[0] + kernel.register(candidate) + assert kernel.validate_next(adapter, candidate, seed=40, context=ctx()) == EvidenceLevel.L1 + assert kernel.validate_next(adapter, candidate, seed=41, context=ctx()) == EvidenceLevel.L1 + assert kernel.ledger.state("cand-leaky").level == EvidenceLevel.L1 + failing = [v for v in kernel.gate_log if not v.passed] + assert failing and any("leakage" in r for r in failing[-1].reasons) + + +class NonDeterministicAdapter: + """Same seed, different outcomes across calls: fails the determinism check.""" + + manifest = AdapterManifest("flaky", "1", "flaky-domain", ("x",)) + + def __init__(self): + self.calls = 0 + + def propose(self, *, seed: int, limit: int): + return (Candidate("cand-flaky", "flaky-domain", "unstable signal"),) + + def validate(self, candidate, *, stage: str, seed: int, context): + context.checkpoint() + self.calls += 1 + score = 0.1 if self.calls == 1 else 0.9 # first attempt fails, then "recovers" + return ( + Evidence( + f"ev-{self.calls}", + candidate.candidate_id, + stage, + score > 0.5, + "flaky-v1", + "synthetic-train", + seed, + score, + ), + ) + + def import_structure(self, structure, *, candidate_id: str): + return Candidate(candidate_id, "flaky-domain", "claim") + + +def test_kernel_blocks_l1_on_nondeterminism(tmp_path): + adapter = NonDeterministicAdapter() + kernel = gated_kernel(tmp_path, "flaky-domain") + candidate = adapter.propose(seed=1, limit=1)[0] + kernel.register(candidate) + # Attempt 1 records failing evidence → stays L0. Attempt 2 reruns the same + # (protocol, dataset, seed) but reports a contradictory passing outcome, so + # the determinism check must block promotion even though evidence passed. + assert kernel.validate_next(adapter, candidate, seed=42, context=ctx()) == EvidenceLevel.L0 + assert kernel.validate_next(adapter, candidate, seed=42, context=ctx()) == EvidenceLevel.L0 + assert kernel.ledger.state("cand-flaky").level == EvidenceLevel.L0 + failing = [v for v in kernel.gate_log if not v.passed] + assert failing + assert any("non-deterministic" in reason for reason in failing[-1].reasons) + + +def test_kernel_refuses_l2_without_declared_protocol(tmp_path): + adapter = SyntheticLinearAdapter() + gates = ValidationGates({}) # configured but declares nothing + kernel = DiscoveryKernel( + EvidenceLedger(tmp_path / "events.jsonl"), validation=gates + ) + candidate = adapter.propose(seed=5, limit=1)[0] + kernel.register(candidate) + assert kernel.validate_next(adapter, candidate, seed=40, context=ctx()) == EvidenceLevel.L1 + with pytest.raises(ValueError, match="no holdout protocol"): + kernel.validate_next(adapter, candidate, seed=41, context=ctx()) + + +def test_kernel_without_gates_keeps_phase1_behaviour(tmp_path): + adapter = LeakyReplicationAdapter() # would fail gates; no gates configured + kernel = DiscoveryKernel(EvidenceLedger(tmp_path / "events.jsonl")) + candidate = adapter.propose(seed=1, limit=1)[0] + kernel.register(candidate) + assert kernel.validate_next(adapter, candidate, seed=40, context=ctx()) == EvidenceLevel.L1 + assert kernel.validate_next(adapter, candidate, seed=41, context=ctx()) == EvidenceLevel.L2 + assert kernel.gate_log == [] diff --git a/tests/test_kernel_isolation.py b/tests/test_kernel_isolation.py new file mode 100644 index 0000000..1d08247 --- /dev/null +++ b/tests/test_kernel_isolation.py @@ -0,0 +1,66 @@ +import json +from datetime import date + +import pytest +from isolation_doubles import WellBehavedAdapter + +from sapiens.budget import ExecutionContext +from sapiens.kernel import DiscoveryKernel +from sapiens.ledger import EvidenceLedger +from sapiens.models import Candidate, EvidenceLevel +from sapiens.permissions import MissingPermissionError, PermissionManifest +from sapiens.registry import AdapterRegistry + +PERMISSION = { + "source": "doubles", + "scope": "adapter:well-behaved", + "licence": "MIT", + "granted_by": "owner@example.org", + "reference": "https://example.org/permission/1", + "granted_on": "2026-07-01", + "expires_on": None, +} + + +def registry(tmp_path, entries): + path = tmp_path / "permissions.json" + path.write_text(json.dumps({"version": 1, "entries": entries})) + return AdapterRegistry(PermissionManifest.load(path), today=date(2026, 7, 20)) + + +def candidate(): + return Candidate("iso-cand", "iso-domain", "claim under isolation") + + +def test_kernel_runs_untrusted_adapter_isolated(tmp_path): + kernel = DiscoveryKernel( + EvidenceLedger(tmp_path / "evidence.jsonl"), registry(tmp_path, [PERMISSION]) + ) + cand = candidate() + kernel.register(cand) + reached = kernel.validate_next( + WellBehavedAdapter(), + cand, + seed=7, + context=ExecutionContext(max_steps=10, max_seconds=5.0), + ) + assert reached == EvidenceLevel.L1 + state = kernel.ledger.state("iso-cand") + assert state.level == EvidenceLevel.L1 + + +def test_kernel_refuses_third_party_without_permission(tmp_path): + kernel = DiscoveryKernel( + EvidenceLedger(tmp_path / "evidence.jsonl"), registry(tmp_path, []) + ) + cand = candidate() + kernel.register(cand) + with pytest.raises(MissingPermissionError): + kernel.validate_next( + WellBehavedAdapter(), + cand, + seed=7, + context=ExecutionContext(max_steps=10, max_seconds=5.0), + ) + # No promotion, no evidence: the ledger shows the candidate still at L0. + assert kernel.ledger.state("iso-cand").level == EvidenceLevel.L0 diff --git a/tests/test_kernel_panel.py b/tests/test_kernel_panel.py new file mode 100644 index 0000000..6446ae9 --- /dev/null +++ b/tests/test_kernel_panel.py @@ -0,0 +1,132 @@ +"""Kernel + L3 review panel integration (Phase 3).""" + +import json + +from sapiens.adapters import SyntheticPhotometryAdapter +from sapiens.budget import ExecutionContext +from sapiens.kernel import DiscoveryKernel +from sapiens.ledger import EvidenceLedger +from sapiens.models import AdapterManifest, Candidate, Evidence, EvidenceLevel +from sapiens.review import PanelOutcome +from sapiens.reviewers import reference_panel +from sapiens.validation import synthetic_holdout_protocol + + +def ctx(): + return ExecutionContext(max_steps=20, max_seconds=20.0) + + +def climb(kernel, adapter, candidate, seeds=(40, 41, 42)): + reached = EvidenceLevel.L0 + for seed in seeds: + reached = kernel.validate_next(adapter, candidate, seed=seed, context=ctx()) + return reached + + +def test_panel_approval_path_recorded_end_to_end(tmp_path): + adapter = SyntheticPhotometryAdapter() + panel = reference_panel(adapter.manifest.vocabulary, synthetic_holdout_protocol()) + kernel = DiscoveryKernel(EvidenceLedger(tmp_path / "events.jsonl"), panel=panel) + candidate = adapter.propose(seed=5, limit=1)[0] + kernel.register(candidate) + assert climb(kernel, adapter, candidate) == EvidenceLevel.L3 + assert kernel.panel_log[-1].outcome == PanelOutcome.APPROVED + # The verdict lives in the ledger as review evidence — no side channel. + panel_records = [ + event + for event in kernel.ledger.events() + if event.kind == "evidence" and event.payload["dataset"] == "panel-transcript" + ] + assert len(panel_records) == 1 + assert panel_records[0].payload["passed"] is True + assert panel_records[0].payload["details"]["report"]["outcome"] == "approved" + assert kernel.ledger.verify() + + +class PerfectScoreAdapter: + """Everything exactly 1.0: passes naive gates, trips the devil's advocate.""" + + manifest = AdapterManifest("perfect", "1", "perfect-domain", ("signal",)) + + def propose(self, *, seed: int, limit: int): + return (Candidate("cand-perfect", "perfect-domain", "a signal claim"),) + + def validate(self, candidate, *, stage: str, seed: int, context): + context.checkpoint() + dataset = "synthetic-train" if stage == "internal" else "synthetic-holdout" + return ( + Evidence( + f"ev-{stage}-{seed}", + candidate.candidate_id, + stage, + True, + f"perfect-{stage}-v1", + dataset, + seed, + 1.0, + ), + ) + + def import_structure(self, structure, *, candidate_id: str): + return Candidate(candidate_id, "perfect-domain", "claim") + + +def test_panel_rejection_blocks_l3_and_records_verdict(tmp_path): + adapter = PerfectScoreAdapter() + panel = reference_panel(adapter.manifest.vocabulary, synthetic_holdout_protocol()) + kernel = DiscoveryKernel(EvidenceLedger(tmp_path / "events.jsonl"), panel=panel) + candidate = adapter.propose(seed=1, limit=1)[0] + kernel.register(candidate) + assert climb(kernel, adapter, candidate) == EvidenceLevel.L2 + assert kernel.ledger.state("cand-perfect").level == EvidenceLevel.L2 + report = kernel.panel_log[-1] + assert report.outcome == PanelOutcome.REJECTED + assert len(report.rounds) == 3 # escalated objection held through the budget + assert any("implausibly perfect" in o.text for o in report.sustained_blocking) + panel_records = [ + event + for event in kernel.ledger.events() + if event.kind == "evidence" and event.payload["dataset"] == "panel-transcript" + ] + assert len(panel_records) == 1 + assert panel_records[0].payload["passed"] is False + assert kernel.ledger.verify() + + +def test_repeated_rejection_attempts_do_not_collide(tmp_path): + adapter = PerfectScoreAdapter() + panel = reference_panel(adapter.manifest.vocabulary, synthetic_holdout_protocol()) + kernel = DiscoveryKernel(EvidenceLedger(tmp_path / "events.jsonl"), panel=panel) + candidate = adapter.propose(seed=1, limit=1)[0] + kernel.register(candidate) + assert kernel.validate_next(adapter, candidate, seed=40, context=ctx()) == EvidenceLevel.L1 + assert kernel.validate_next(adapter, candidate, seed=41, context=ctx()) == EvidenceLevel.L2 + assert kernel.validate_next(adapter, candidate, seed=42, context=ctx()) == EvidenceLevel.L2 + # A second L3 attempt with a fresh seed must not hit duplicate evidence ids. + assert kernel.validate_next(adapter, candidate, seed=43, context=ctx()) == EvidenceLevel.L2 + assert kernel.ledger.verify() + reports = [ + event + for event in kernel.ledger.events() + if event.kind == "evidence" and event.payload["dataset"] == "panel-transcript" + ] + assert len(reports) == 2 + + +def test_kernel_without_panel_unchanged(tmp_path): + adapter = SyntheticPhotometryAdapter() + kernel = DiscoveryKernel(EvidenceLedger(tmp_path / "events.jsonl")) + candidate = adapter.propose(seed=5, limit=1)[0] + kernel.register(candidate) + assert climb(kernel, adapter, candidate) == EvidenceLevel.L3 + assert kernel.panel_log == [] + + +def test_panel_report_blob_is_json_safe(tmp_path): + adapter = PerfectScoreAdapter() + panel = reference_panel(adapter.manifest.vocabulary, synthetic_holdout_protocol()) + kernel = DiscoveryKernel(EvidenceLedger(tmp_path / "events.jsonl"), panel=panel) + candidate = adapter.propose(seed=1, limit=1)[0] + kernel.register(candidate) + climb(kernel, adapter, candidate) + json.dumps(kernel.panel_log[-1].to_dict()) diff --git a/tests/test_permissions.py b/tests/test_permissions.py new file mode 100644 index 0000000..8863570 --- /dev/null +++ b/tests/test_permissions.py @@ -0,0 +1,115 @@ +import json +from datetime import date + +import pytest + +from sapiens.permissions import ( + ManifestFormatError, + MissingPermissionError, + PermissionEntry, + PermissionManifest, +) + +TODAY = date(2026, 7, 20) + +ENTRY = { + "source": "ASTRA-dev", + "scope": "adapter:astra-photometry", + "licence": "MIT", + "granted_by": "owner@example.org", + "reference": "https://example.org/permission/1", + "granted_on": "2026-07-01", + "expires_on": "2026-12-31", +} + + +def make_manifest(tmp_path, entries): + path = tmp_path / "permissions.json" + path.write_text(json.dumps({"version": 1, "entries": entries})) + return PermissionManifest.load(path) + + +def test_repo_manifest_is_empty_and_valid(): + from pathlib import Path + + root = Path(__file__).resolve().parents[1] + manifest = PermissionManifest.load(root / "permissions.json") + assert manifest.entries == () + # The clean-room invariant: zero ASTRA-family permissions recorded. + assert not any("ASTRA" in e.source or "astra" in e.source for e in manifest.entries) + + +def test_empty_manifest_refuses_everything(): + manifest = PermissionManifest.empty() + assert not manifest.permits(source="ASTRA-dev", scope="adapter:x", on=TODAY) + with pytest.raises(MissingPermissionError): + manifest.require(source="ASTRA-dev", scope="adapter:x", on=TODAY) + + +def test_load_roundtrip_and_permit(tmp_path): + manifest = make_manifest(tmp_path, [ENTRY]) + assert manifest.permits( + source="ASTRA-dev", scope="adapter:astra-photometry", on=date(2026, 8, 1) + ) + entry = manifest.require( + source="ASTRA-dev", scope="adapter:astra-photometry", on=date(2026, 8, 1) + ) + assert entry.licence == "MIT" + + +def test_expired_permission_refused(tmp_path): + manifest = make_manifest(tmp_path, [ENTRY]) + assert not manifest.permits( + source="ASTRA-dev", scope="adapter:astra-photometry", on=date(2027, 1, 1) + ) + with pytest.raises(MissingPermissionError, match="not active"): + manifest.require( + source="ASTRA-dev", scope="adapter:astra-photometry", on=date(2027, 1, 1) + ) + + +def test_not_yet_granted_permission_refused(tmp_path): + manifest = make_manifest(tmp_path, [ENTRY]) + assert not manifest.permits( + source="ASTRA-dev", scope="adapter:astra-photometry", on=date(2026, 6, 1) + ) + + +def test_scope_wildcard(tmp_path): + wildcard = {**ENTRY, "scope": "module:swarm/*"} + manifest = make_manifest(tmp_path, [wildcard]) + assert manifest.permits( + source="ASTRA-dev", scope="module:swarm/pheromone_dynamics", on=date(2026, 8, 1) + ) + assert not manifest.permits(source="ASTRA-dev", scope="module:other/x", on=date(2026, 8, 1)) + + +def test_wrong_source_refused(tmp_path): + manifest = make_manifest(tmp_path, [ENTRY]) + assert not manifest.permits( + source="SLATE", scope="adapter:astra-photometry", on=date(2026, 8, 1) + ) + + +@pytest.mark.parametrize( + "raw", + [ + {"version": 2, "entries": []}, # unsupported version + {"version": 1}, # missing entries list + {"version": 1, "entries": [{"source": "x"}]}, # incomplete entry + {"version": 1, "entries": [{**ENTRY, "granted_on": "not-a-date"}]}, + {"version": 1, "entries": [{**ENTRY, "expires_on": "2026-01-01"}]}, # before grant + {"version": 1, "entries": [{**ENTRY, "surprise_field": 1}]}, # unknown field + {"version": 1, "entries": [ENTRY, ENTRY]}, # duplicates + ], +) +def test_malformed_manifests_fail_loudly(tmp_path, raw): + path = tmp_path / "bad.json" + path.write_text(json.dumps(raw)) + with pytest.raises(ManifestFormatError): + PermissionManifest.load(path) + + +def test_entry_requires_nonempty_fields(): + with pytest.raises(ManifestFormatError): + PermissionEntry.from_dict({**ENTRY, "licence": ""}) diff --git a/tests/test_photometry_adapter.py b/tests/test_photometry_adapter.py index a57316f..7a669b7 100644 --- a/tests/test_photometry_adapter.py +++ b/tests/test_photometry_adapter.py @@ -4,7 +4,11 @@ from sapiens.adapters import SyntheticPhotometryAdapter, SyntheticThresholdAdapter from sapiens.budget import ExecutionContext -CTX = ExecutionContext(10, 2) + +def ctx() -> ExecutionContext: + # Fresh context per call: a module-level time-budgeted context starts its + # wall clock at import/collection time and goes stale as the suite grows. + return ExecutionContext(10, 10) def test_photometry_true_period_promotes_to_l3(tmp_path: Path): @@ -13,9 +17,9 @@ def test_photometry_true_period_promotes_to_l3(tmp_path: Path): adapter = SyntheticPhotometryAdapter() candidate = adapter.propose(seed=5, limit=1)[0] kernel.register(candidate) - assert kernel.validate_next(adapter, candidate, seed=40, context=CTX) == EvidenceLevel.L1 - assert kernel.validate_next(adapter, candidate, seed=41, context=CTX) == EvidenceLevel.L2 - assert kernel.validate_next(adapter, candidate, seed=42, context=CTX) == EvidenceLevel.L3 + assert kernel.validate_next(adapter, candidate, seed=40, context=ctx()) == EvidenceLevel.L1 + assert kernel.validate_next(adapter, candidate, seed=41, context=ctx()) == EvidenceLevel.L2 + assert kernel.validate_next(adapter, candidate, seed=42, context=ctx()) == EvidenceLevel.L3 assert ledger.verify() is True @@ -25,7 +29,7 @@ def test_photometry_wrong_period_does_not_promote(tmp_path: Path): adapter = SyntheticPhotometryAdapter() candidate = adapter.propose(seed=5, limit=2)[1] # the wrong-period candidate kernel.register(candidate) - assert kernel.validate_next(adapter, candidate, seed=40, context=CTX) == EvidenceLevel.L0 + assert kernel.validate_next(adapter, candidate, seed=40, context=ctx()) == EvidenceLevel.L0 def test_photometry_evidence_is_well_formed(): diff --git a/tests/test_registry.py b/tests/test_registry.py new file mode 100644 index 0000000..fe18151 --- /dev/null +++ b/tests/test_registry.py @@ -0,0 +1,115 @@ +import json +from datetime import date + +import pytest + +from sapiens.models import AdapterManifest, Candidate +from sapiens.permissions import MissingPermissionError, PermissionManifest +from sapiens.registry import AdapterRegistry, TrustTier, derive_tier + + +class SyntheticDouble: + manifest = AdapterManifest("synth", "1", "synth-domain", ("x",)) + + def propose(self, *, seed: int, limit: int): + return () + + def validate(self, candidate, *, stage: str, seed: int, context): + return () + + def import_structure(self, structure, *, candidate_id: str): + return Candidate(candidate_id, "synth-domain", "claim") + + +class CoreDouble(SyntheticDouble): + manifest = AdapterManifest( + "core-real", + "1", + "core-domain", + ("x",), + synthetic_only=False, + data_sources=("https://example.org/public",), + ) + + +class ThirdPartyDouble(SyntheticDouble): + manifest = AdapterManifest( + "tp", + "1", + "tp-domain", + ("x",), + synthetic_only=False, + code_origin="third-party", + third_party_source="ASTRA-dev", + data_sources=("https://example.org/public",), + ) + + +PERMISSION = { + "source": "ASTRA-dev", + "scope": "adapter:tp", + "licence": "MIT", + "granted_by": "owner@example.org", + "reference": "https://example.org/permission/1", + "granted_on": "2026-07-01", + "expires_on": None, +} + + +def manifest_with(entries, tmp_path): + path = tmp_path / "permissions.json" + path.write_text(json.dumps({"version": 1, "entries": entries})) + return PermissionManifest.load(path) + + +def test_derive_tier_from_manifest_facts(): + assert derive_tier(SyntheticDouble()) == TrustTier.SYNTHETIC + assert derive_tier(CoreDouble()) == TrustTier.CORE + assert derive_tier(ThirdPartyDouble()) == TrustTier.UNTRUSTED + + +def test_unregistered_adapters_auto_tier_on_validate(): + registry = AdapterRegistry(today=date(2026, 7, 20)) + assert registry.validate_adapter(SyntheticDouble()) == TrustTier.SYNTHETIC + assert registry.validate_adapter(CoreDouble()) == TrustTier.CORE + + +def test_third_party_requires_permission_entry(tmp_path): + registry = AdapterRegistry(today=date(2026, 7, 20)) + with pytest.raises(MissingPermissionError): + registry.validate_adapter(ThirdPartyDouble()) + permitted = AdapterRegistry( + manifest_with([PERMISSION], tmp_path), today=date(2026, 7, 20) + ) + assert permitted.validate_adapter(ThirdPartyDouble()) == TrustTier.UNTRUSTED + assert permitted.requires_isolation(ThirdPartyDouble()) + + +def test_expired_permission_blocks_validation(tmp_path): + expired = {**PERMISSION, "expires_on": "2026-07-19"} + registry = AdapterRegistry(manifest_with([expired], tmp_path), today=date(2026, 7, 20)) + with pytest.raises(MissingPermissionError): + registry.validate_adapter(ThirdPartyDouble()) + + +def test_registration_pins_tier_and_rejects_incoherent(tmp_path): + registry = AdapterRegistry(today=date(2026, 7, 20)) + registry.register(SyntheticDouble(), TrustTier.SYNTHETIC) + with pytest.raises(ValueError, match="already registered"): + registry.register(SyntheticDouble(), TrustTier.CORE) + with pytest.raises(ValueError, match="real data sources"): + registry.register(CoreDouble(), TrustTier.SYNTHETIC) + with pytest.raises(ValueError, match="clean-room"): + registry.register(ThirdPartyDouble(), TrustTier.CORE) + + +def test_register_third_party_untrusted_with_permission(tmp_path): + registry = AdapterRegistry(manifest_with([PERMISSION], tmp_path), today=date(2026, 7, 20)) + registry.register(ThirdPartyDouble(), TrustTier.UNTRUSTED) + assert registry.tier_of(ThirdPartyDouble()) == TrustTier.UNTRUSTED + + +def test_non_adapter_rejected(): + registry = AdapterRegistry() + with pytest.raises(TypeError): + registry.validate_adapter(object()) diff --git a/tests/test_review.py b/tests/test_review.py new file mode 100644 index 0000000..369e87f --- /dev/null +++ b/tests/test_review.py @@ -0,0 +1,209 @@ +"""Panel protocol + schema tests (scripted reviewer doubles).""" + +import json + +import pytest + +from sapiens.models import Candidate, Evidence +from sapiens.review import ( + Objection, + ObjectionStatus, + PanelOutcome, + ReviewerRole, + ReviewerVerdict, + ReviewPanel, + Severity, + VerdictKind, +) + +CANDIDATE = Candidate("cand-p", "dom", "a claim about signal") +EVIDENCE = (Evidence("e1", "cand-p", "internal", True, "p", "d", 1, 0.9),) + + +class Scripted: + """Reviewer double following a per-round script of (verdict, [(severity, text)]).""" + + def __init__(self, role, script): + self.role = role + self._script = script + + def review(self, candidate, evidence, *, round_number, prior_objections, seed): + kind, objections = self._script.get(round_number, (VerdictKind.APPROVE, [])) + return ReviewerVerdict( + self.role, + kind, + f"scripted round {round_number}", + tuple( + Objection( + f"{self.role.value}-r{round_number}-{i}", + self.role, + severity, + text, + round_number, + ) + for i, (severity, text) in enumerate(objections) + ), + ) + + +def approve_all(role): + return Scripted(role, {}) + + +class TestSchema: + def test_approve_carries_no_objections(self): + with pytest.raises(ValueError): + ReviewerVerdict( + ReviewerRole.STATISTICIAN, + VerdictKind.APPROVE, + "rationale", + (Objection("o1", ReviewerRole.STATISTICIAN, Severity.MINOR, "x", 1),), + ) + + def test_object_requires_objections(self): + with pytest.raises(ValueError): + ReviewerVerdict(ReviewerRole.STATISTICIAN, VerdictKind.OBJECT, "rationale") + + def test_abstain_carries_no_objections(self): + with pytest.raises(ValueError): + ReviewerVerdict( + ReviewerRole.STATISTICIAN, + VerdictKind.ABSTAIN, + "rationale", + (Objection("o1", ReviewerRole.STATISTICIAN, Severity.MINOR, "x", 1),), + ) + + def test_rationale_required(self): + with pytest.raises(ValueError): + ReviewerVerdict(ReviewerRole.STATISTICIAN, VerdictKind.APPROVE, "") + + def test_objection_role_must_match(self): + with pytest.raises(ValueError): + ReviewerVerdict( + ReviewerRole.STATISTICIAN, + VerdictKind.OBJECT, + "rationale", + (Objection("o1", ReviewerRole.METHODOLOGIST, Severity.MINOR, "x", 1),), + ) + + def test_objection_needs_text_and_valid_round(self): + with pytest.raises(ValueError): + Objection("o1", ReviewerRole.STATISTICIAN, Severity.MINOR, "", 1) + with pytest.raises(ValueError): + Objection("o1", ReviewerRole.STATISTICIAN, Severity.MINOR, "x", 0) + + def test_panel_requires_unique_roles_and_budget(self): + with pytest.raises(ValueError): + ReviewPanel(()) + with pytest.raises(ValueError): + ReviewPanel((approve_all(ReviewerRole.STATISTICIAN),) * 2) + with pytest.raises(ValueError): + ReviewPanel((approve_all(ReviewerRole.STATISTICIAN),), max_rounds=0) + + +class TestProtocol: + def panel(self, *reviewers, max_rounds=3): + return ReviewPanel(reviewers, max_rounds=max_rounds) + + def test_clean_consensus_approves_in_one_round(self): + panel = self.panel( + approve_all(ReviewerRole.STATISTICIAN), approve_all(ReviewerRole.METHODOLOGIST) + ) + report = panel.convene(CANDIDATE, EVIDENCE, seed=1) + assert report.outcome == PanelOutcome.APPROVED + assert len(report.rounds) == 1 + + def test_minor_raised_then_withdrawn_approves(self): + wavering = Scripted( + ReviewerRole.STATISTICIAN, + {1: (VerdictKind.OBJECT, [(Severity.MINOR, "caveat")])}, + ) + panel = self.panel(wavering, approve_all(ReviewerRole.METHODOLOGIST)) + report = panel.convene(CANDIDATE, EVIDENCE, seed=1) + assert report.outcome == PanelOutcome.APPROVED + assert len(report.rounds) == 2 + assert [status for _, status in report.lifecycle] == [ObjectionStatus.WITHDRAWN] + assert report.withdrawn[0].text == "caveat" + + def test_major_escalates_and_rejects(self): + # The raiser holds the objection through the whole budget: MAJOR in + # round 1, escalated to BLOCKING in the rebuttals, never withdrawn. + escalating = Scripted( + ReviewerRole.STATISTICIAN, + { + 1: (VerdictKind.OBJECT, [(Severity.MAJOR, "suspicious")]), + 2: (VerdictKind.OBJECT, [(Severity.BLOCKING, "suspicious")]), + 3: (VerdictKind.OBJECT, [(Severity.BLOCKING, "suspicious")]), + }, + ) + panel = self.panel(escalating, approve_all(ReviewerRole.METHODOLOGIST)) + report = panel.convene(CANDIDATE, EVIDENCE, seed=1) + assert report.outcome == PanelOutcome.REJECTED + assert len(report.rounds) == 3 + assert report.sustained_blocking[0].text == "suspicious" + statuses = dict(report.lifecycle) + assert list(statuses.values()) == [ObjectionStatus.SUSTAINED] + + def test_escalated_objection_late_withdrawal_approves(self): + # Escalated to BLOCKING in round 2, then withdrawn in round 3: the + # lifecycle must record the withdrawal and the panel approves. + relenting = Scripted( + ReviewerRole.STATISTICIAN, + { + 1: (VerdictKind.OBJECT, [(Severity.MAJOR, "suspicious")]), + 2: (VerdictKind.OBJECT, [(Severity.BLOCKING, "suspicious")]), + }, + ) + panel = self.panel(relenting, approve_all(ReviewerRole.METHODOLOGIST)) + report = panel.convene(CANDIDATE, EVIDENCE, seed=1) + assert report.outcome == PanelOutcome.APPROVED + assert len(report.rounds) == 3 + assert report.withdrawn[0].text == "suspicious" + assert report.withdrawn[0].severity == Severity.BLOCKING + + def test_blocking_withdrawn_in_rebuttal_approves(self): + relenting = Scripted( + ReviewerRole.DEVILS_ADVOCATE, + {1: (VerdictKind.OBJECT, [(Severity.BLOCKING, "fatal flaw")])}, + ) + panel = self.panel(relenting, approve_all(ReviewerRole.METHODOLOGIST)) + report = panel.convene(CANDIDATE, EVIDENCE, seed=1) + assert report.outcome == PanelOutcome.APPROVED + assert len(report.rounds) == 2 + assert report.withdrawn[0].text == "fatal flaw" + + def test_residual_minor_is_caveat_not_fatal(self): + stubborn_minor = Scripted( + ReviewerRole.METHODOLOGIST, + { + 1: (VerdictKind.OBJECT, [(Severity.MINOR, "nit")]), + 2: (VerdictKind.OBJECT, [(Severity.MINOR, "nit")]), + 3: (VerdictKind.OBJECT, [(Severity.MINOR, "nit")]), + }, + ) + panel = self.panel(stubborn_minor, approve_all(ReviewerRole.STATISTICIAN)) + report = panel.convene(CANDIDATE, EVIDENCE, seed=1) + assert report.outcome == PanelOutcome.APPROVED + assert len(report.rounds) == 3 # budget exhausted on the caveat + + def test_sustained_major_at_budget_rejects(self): + stubborn_major = Scripted( + ReviewerRole.METHODOLOGIST, + { + 1: (VerdictKind.OBJECT, [(Severity.MAJOR, "concern")]), + 2: (VerdictKind.OBJECT, [(Severity.MAJOR, "concern")]), + 3: (VerdictKind.OBJECT, [(Severity.MAJOR, "concern")]), + }, + ) + panel = self.panel(stubborn_major, approve_all(ReviewerRole.STATISTICIAN)) + report = panel.convene(CANDIDATE, EVIDENCE, seed=1) + assert report.outcome == PanelOutcome.REJECTED + assert len(report.rounds) == 3 + + def test_report_serialisable_and_deterministic_id(self): + panel = self.panel(approve_all(ReviewerRole.STATISTICIAN)) + first = panel.convene(CANDIDATE, EVIDENCE, seed=1) + second = panel.convene(CANDIDATE, EVIDENCE, seed=1) + assert first.report_id == second.report_id + blob = json.dumps(first.to_dict()) + assert "cand-p" in blob and "approved" in blob diff --git a/tests/test_reviewers.py b/tests/test_reviewers.py new file mode 100644 index 0000000..5e5985d --- /dev/null +++ b/tests/test_reviewers.py @@ -0,0 +1,161 @@ +"""Reference reviewers against the seeded fixture suite.""" + +from sapiens.fixtures import FixtureKind, fixture_suite +from sapiens.models import Candidate, Evidence +from sapiens.review import Severity, VerdictKind +from sapiens.reviewers import ( + DevilsAdvocateReviewer, + DomainTheoristReviewer, + MethodologistReviewer, + StatisticianReviewer, + reference_panel, +) +from sapiens.validation import synthetic_holdout_protocol + +FIXTURES = {f.kind: f for f in fixture_suite()} +PROTOCOL = synthetic_holdout_protocol() +VOCAB = ("signal", "score", "data", "training", "replication", "holdout") + + +def review(reviewer, fixture, round_number=1): + return reviewer.review( + fixture.candidate, + fixture.internal + fixture.replication, + round_number=round_number, + prior_objections=(), + seed=0, + ) + + +class TestStatistician: + reviewer = StatisticianReviewer(PROTOCOL) + + def test_approves_known_good(self): + assert review(self.reviewer, FIXTURES[FixtureKind.KNOWN_GOOD]).verdict == ( + VerdictKind.APPROVE + ) + + def test_catches_overfit_at_l2(self): + verdict = review(self.reviewer, FIXTURES[FixtureKind.OVERFIT]) + assert verdict.verdict == VerdictKind.OBJECT + assert any( + o.severity == Severity.BLOCKING and "L2 gate" in o.text for o in verdict.objections + ) + + def test_catches_leakage_at_l2(self): + verdict = review(self.reviewer, FIXTURES[FixtureKind.LEAKAGE]) + assert verdict.verdict == VerdictKind.OBJECT + assert any("leakage" in o.text for o in verdict.objections) + + def test_catches_degenerate_at_l1(self): + verdict = review(self.reviewer, FIXTURES[FixtureKind.DEGENERATE]) + assert verdict.verdict == VerdictKind.OBJECT + assert any("L1 gate" in o.text for o in verdict.objections) + + +class TestMethodologist: + reviewer = MethodologistReviewer() + + def test_approves_known_good(self): + assert review(self.reviewer, FIXTURES[FixtureKind.KNOWN_GOOD]).verdict == ( + VerdictKind.APPROVE + ) + + def test_catches_cross_stage_dataset_reuse(self): + verdict = review(self.reviewer, FIXTURES[FixtureKind.LEAKAGE]) + assert verdict.verdict == VerdictKind.OBJECT + assert any( + o.severity == Severity.BLOCKING and "leakage signal" in o.text + for o in verdict.objections + ) + + def test_blocks_on_empty_evidence(self): + verdict = self.reviewer.review( + Candidate("c", "d", "claim about signal"), + (), + round_number=1, + prior_objections=(), + seed=0, + ) + assert verdict.verdict == VerdictKind.OBJECT + assert verdict.objections[0].severity == Severity.BLOCKING + + +class TestDomainTheorist: + reviewer = DomainTheoristReviewer(VOCAB) + + def test_approves_all_fixture_claims(self): + # Fixture claims are deliberately vocabulary-coherent so per-role + # attribution stays clean: the theorist's job is coherence, not bias. + for fixture in FIXTURES.values(): + assert review(self.reviewer, fixture).verdict == VerdictKind.APPROVE + + def test_incoherent_claim_escalates_across_rounds(self): + candidate = Candidate("c", "d", "zzz qqq nothing coherent") + evidence = (Evidence("e", "c", "internal", True, "p", "d", 1, 0.9),) + first = self.reviewer.review( + candidate, evidence, round_number=1, prior_objections=(), seed=0 + ) + second = self.reviewer.review( + candidate, evidence, round_number=2, prior_objections=(), seed=0 + ) + assert first.objections[0].severity == Severity.MAJOR + assert second.objections[0].severity == Severity.BLOCKING + + +class TestDevilsAdvocate: + reviewer = DevilsAdvocateReviewer() + + def test_approves_known_good(self): + assert review(self.reviewer, FIXTURES[FixtureKind.KNOWN_GOOD]).verdict == ( + VerdictKind.APPROVE + ) + + def test_catches_degenerate_signature(self): + verdict = review(self.reviewer, FIXTURES[FixtureKind.DEGENERATE]) + assert any( + o.severity == Severity.BLOCKING and "degenerate" in o.text + for o in verdict.objections + ) + + def test_catches_leakage_signature(self): + verdict = review(self.reviewer, FIXTURES[FixtureKind.LEAKAGE]) + assert any( + o.severity == Severity.BLOCKING and "leakage" in o.text + for o in verdict.objections + ) + + def test_overfit_alone_not_advocate_territory(self): + # Overfit passes the advocate's signature checks; the statistician + # owns that catch. Attribution honesty matters for per-role rates. + verdict = review(self.reviewer, FIXTURES[FixtureKind.OVERFIT]) + assert all( + o.severity != Severity.BLOCKING or "single run" in o.text + for o in verdict.objections + ) + + def test_perfect_scores_escalate(self): + candidate = Candidate("c", "d", "signal claim") + evidence = ( + Evidence("e1", "c", "internal", True, "p", "train", 1, 1.0), + Evidence("e2", "c", "replication", True, "p", "holdout", 2, 1.0), + ) + first = self.reviewer.review( + candidate, evidence, round_number=1, prior_objections=(), seed=0 + ) + second = self.reviewer.review( + candidate, evidence, round_number=2, prior_objections=(), seed=0 + ) + assert any(o.severity == Severity.MAJOR for o in first.objections) + assert any(o.severity == Severity.BLOCKING for o in second.objections) + + def test_thin_stage_is_minor_caveat(self): + verdict = review(self.reviewer, FIXTURES[FixtureKind.LEAKAGE]) + # leakage fixture has one run per stage: minor caveat rides alongside + assert any(o.severity == Severity.MINOR for o in verdict.objections) + + +def test_reference_panel_has_four_unique_roles(): + panel = reference_panel(VOCAB, PROTOCOL) + assert len(panel.reviewers) == 4 + assert len({reviewer.role for reviewer in panel.reviewers}) == 4 diff --git a/tests/test_validation.py b/tests/test_validation.py new file mode 100644 index 0000000..83aca79 --- /dev/null +++ b/tests/test_validation.py @@ -0,0 +1,170 @@ +import pytest + +from sapiens.models import Evidence +from sapiens.validation import ( + GateVerdict, + HoldoutProtocol, + ValidationGates, + check_internal_consistency, + check_replication, + synthetic_holdout_protocol, +) + +PROTOCOL = synthetic_holdout_protocol() + + +def ev(eid, kind, dataset, seed, score, passed=True, protocol="proto-v1"): + return Evidence(eid, "cand-x", kind, passed, protocol, dataset, seed, score) + + +class TestGateVerdictCoherence: + def test_passing_verdict_has_no_reasons(self): + with pytest.raises(ValueError): + GateVerdict("g", True, ("reason",)) + + def test_failing_verdict_must_explain(self): + with pytest.raises(ValueError): + GateVerdict("g", False, ()) + + +class TestHoldoutProtocolCoherence: + def test_self_leaking_protocol_rejected(self): + with pytest.raises(ValueError, match="self-leaking"): + HoldoutProtocol("bad", ("train",), ("train",)) + + def test_empty_splits_rejected(self): + with pytest.raises(ValueError): + HoldoutProtocol("bad", (), ("holdout",)) + + +class TestInternalConsistency: + def test_honest_evidence_passes(self): + evidence = ( + ev("a", "internal", "synthetic-train", 1, 0.9), + ev("b", "internal", "synthetic-train", 2, 0.85), + ) + assert check_internal_consistency(evidence).passed + + def test_no_evidence_fails(self): + verdict = check_internal_consistency(()) + assert not verdict.passed + assert "no internal evidence" in verdict.reasons[0] + + def test_missing_score_fails(self): + verdict = check_internal_consistency( + (ev("a", "internal", "synthetic-train", 1, None),) + ) + assert not verdict.passed + assert "no score" in verdict.reasons[0] + + def test_nondeterministic_rerun_fails(self): + evidence = ( + ev("a", "internal", "synthetic-train", 7, 0.9, passed=True), + ev("b", "internal", "synthetic-train", 7, 0.2, passed=False), + ) + verdict = check_internal_consistency(evidence) + assert not verdict.passed + assert any("non-deterministic" in reason for reason in verdict.reasons) + + def test_deterministic_rerun_agrees_passes(self): + evidence = ( + ev("a", "internal", "synthetic-train", 7, 0.9), + ev("b", "internal", "synthetic-train", 7, 0.9), + ) + assert check_internal_consistency(evidence).passed + + def test_degenerate_constant_scores_fail(self): + evidence = tuple( + ev(f"e{seed}", "internal", "synthetic-train", seed, 0.5) for seed in (1, 2, 3) + ) + verdict = check_internal_consistency(evidence) + assert not verdict.passed + assert any("degenerate" in reason for reason in verdict.reasons) + + def test_two_seeds_not_enough_for_degenerate_check(self): + evidence = tuple( + ev(f"e{seed}", "internal", "synthetic-train", seed, 0.5) for seed in (1, 2) + ) + assert check_internal_consistency(evidence).passed + + def test_other_kinds_ignored(self): + evidence = (ev("a", "replication", "synthetic-holdout", 1, 0.9),) + verdict = check_internal_consistency(evidence) + assert not verdict.passed # no internal items at all + + +class TestReplicationGate: + def honest(self): + internal = (ev("i1", "internal", "synthetic-train", 1, 0.9),) + replication = ( + ev("r1", "replication", "synthetic-holdout", 2, 0.88), + ev("r2", "replication", "synthetic-holdout", 3, 0.91), + ) + return internal, replication + + def test_honest_holdout_passes(self): + internal, replication = self.honest() + assert check_replication(internal, replication, PROTOCOL).passed + + def test_no_replication_evidence_fails(self): + internal, _ = self.honest() + verdict = check_replication(internal, (), PROTOCOL) + assert not verdict.passed + + def test_undeclared_train_dataset_fails(self): + internal = (ev("i1", "internal", "secret-train", 1, 0.9),) + _, replication = self.honest() + verdict = check_replication(internal, replication, PROTOCOL) + assert not verdict.passed + assert any("undeclared dataset" in reason for reason in verdict.reasons) + + def test_non_holdout_replication_dataset_fails(self): + internal = (ev("i1", "internal", "synthetic-train", 1, 0.9),) + replication = (ev("r1", "replication", "synthetic-train", 2, 0.9),) + verdict = check_replication(internal, replication, PROTOCOL) + assert not verdict.passed + assert any("non-holdout dataset" in reason for reason in verdict.reasons) + + def test_dataset_collision_is_leakage(self): + internal = (ev("i1", "internal", "synthetic-train", 1, 0.9),) + replication = (ev("r1", "replication", "synthetic-train", 2, 0.9),) + verdict = check_replication(internal, replication, PROTOCOL) + assert any("leakage" in reason for reason in verdict.reasons) + + def test_dataset_seed_pair_reuse_is_leakage(self): + internal = (ev("i1", "internal", "synthetic-train", 7, 0.9),) + replication = (ev("r1", "replication", "synthetic-train", 7, 0.9),) + verdict = check_replication(internal, replication, PROTOCOL) + assert any("reuse" in reason for reason in verdict.reasons) + + def test_same_seed_different_dataset_is_not_leakage(self): + # Numeric seed overlap across disjoint datasets is how the shipped + # synthetic adapters derive independent holdout noise; it must pass. + internal = (ev("i1", "internal", "synthetic-train", 7, 0.9),) + replication = (ev("r1", "replication", "synthetic-holdout", 7, 0.88),) + assert check_replication(internal, replication, PROTOCOL).passed + + def test_pass_fraction_enforced(self): + internal = (ev("i1", "internal", "synthetic-train", 1, 0.9),) + replication = ( + ev("r1", "replication", "synthetic-holdout", 2, 0.9, passed=True), + ev("r2", "replication", "synthetic-holdout", 3, 0.2, passed=False), + ) + verdict = check_replication(internal, replication, PROTOCOL) + assert not verdict.passed + assert any("pass fraction" in reason for reason in verdict.reasons) + relaxed = check_replication( + internal, replication, PROTOCOL, min_pass_fraction=0.5 + ) + assert relaxed.passed + + def test_invalid_threshold_rejected(self): + internal, replication = self.honest() + with pytest.raises(ValueError): + check_replication(internal, replication, PROTOCOL, min_pass_fraction=0.0) + + +def test_validation_gates_protocol_lookup(): + gates = ValidationGates({"synthetic-x": PROTOCOL}) + assert gates.protocol_for("synthetic-x") is PROTOCOL + assert gates.protocol_for("unknown") is None