Skip to content

Research: requirements-to-code conformance checking (BDD + reviewed formal contracts + OwnIR evidence) #448

Open

Description

@PhysShell

Classification

Hypothesis

Teams already document desired behavior in issue descriptions, acceptance criteria, business rules, ADRs, and sometimes regulatory documents, but a passing BDD scenario or unit test covers only the selected examples.

Could Own.NET help check generalizable, explicitly approved properties against an implementation, and flag previously unanticipated executable paths or version/state combinations?

Desired product-level outcome:

Given requirement R (with source/version), an approved formal interpretation P, and code revision C, report VIOLATED with a reproducible source-linked witness, PROVEN only within an explicitly declared complete model, or UNKNOWN / NOT COVERED when the system cannot justify either.

The natural-language → property step is not trusted. LLM output is a candidate for human review, never an independent proof that the requirement was interpreted correctly.

Motivating example (illustrative; not a request to embed customs semantics in Own.NET)

Human description: "A declaration may be submitted only after validation succeeds."

Ambiguities to surface before analysis:

  • Must validation apply to the same version of the declaration that is submitted?
  • Does editing fields invalidate prior validation?
  • What counts as a blocking error vs warning?
  • Are there explicit, source-backed exceptions?

Candidate reviewed property:

Submit(d, version=v) ⇒ a successful Validate(d, version=v) happened since the last relevant mutation and no blocking errors remain

A branch that bypasses validation, or a mutation between validation and submit, may violate this property. The verifier must map events to real C# calls/effects and object identity/version; simply checking that Validate() is somewhere earlier in the method is inadequate.

What makes this different from BDD

  • BDD/Reqnroll: curated executable examples through step definitions. Very useful, but does not exhaust unenumerated paths.
  • Property-based testing: many generated inputs under a test oracle; strong testing, not a proof of all behaviors.
  • Own.NET-backed static verification: explores represented branches/states/transfers within the declared model, returning a valid source witness or stating limits.
  • Runtime/systematic testing (optional): can validate a candidate counterexample or explore scheduling-dependent paths; explicitly label tested vs proved.

Do not market one green BDD suite, a bounded search, or an incomplete call graph as a universal conformance proof.

Gate 0: inventory and product constraints

  1. Review Research: FeaVer/Ultimate-style property-guided model extraction and source-level witnesses (bounded spike) #447 and the existing Roslyn→OwnIR→CFG/MOS/obligations/progress pipeline. Identify reusable event facts, call resolution, source locations, analysis completion status, and diagnostic codeFlows.
  2. Check existing required/available BDD and property-based testing paths in the repo. Do not introduce Reqnroll, FRET, LTL, Promela, SMT, or a new DSL simply because prior-art tools exist.
  3. Define a traceability record: requirement ID + original text/span/URL + source document version/effective date + approved interpretation/reviewer + property schema version + mapped code symbols/commit + verifier result/coverage limits. Do not assert current legal correctness from an outdated or unverified regulation.
  4. Determine whether this is better developed as an external opt-in layer/product consuming Own.NET evidence, rather than widening ownership diagnostics by default.

Gate 1: choose a narrow property language and baseline

Start with a small, auditable set of business/protocol patterns:

  • Never / forbidden-before: operation B forbidden while an obligation is open or before event A.
  • Only-after / version-matched: B requires a successful A for the same identified entity/version with no intervening invalidation.
  • Must-close-before-exit: obligation opened must be discharged before return/throw (when modeled).
  • Conditional permission: operation permitted only if a specified predicate has been established; unknown predicate ≠ false/true.

Avoid arbitrary prose, unrestricted first-order logic, temporal liveness, distributed-system correctness and whole-program equivalence in the first vertical.

For each candidate:

Gate 2: bounded pilot, not product rollout

Use 10 real, permission-appropriate requirements from a single C# domain with 2–3 implementable property shapes. Freeze the acceptance oracle before looking at tool output. Prepare independently reviewed fixtures per requirement:

  • correct implementation,
  • seeded/real violating implementation (including a branch bypass),
  • near-neighbor negative to test false positives,
  • one UNKNOWN case with deliberately unresolved call/effect or incomplete model.

At least one case should involve validation → mutation/version change → use, to defeat naive "check A was called sometime" heuristics, provided the required identity/version facts exist; otherwise mark it as a verified representational blocker.

Optional system shape:

human requirement + provenance
         ↓
LLM candidate property + ambiguities
         ↓
human-reviewed property / scope / assumptions
         ↓
mapped source operations + OwnIR facts
         ↓
existing checker / small test-only monitor
         ↓
VIOLATED + path | PROVEN (scoped) | UNKNOWN + reason
         ↓
requirement↔code↔commit evidence for PR/CI

Evaluate two distinct questions:

  1. Specification fidelity: how often did LLM propose the correct meaning, miss a condition, invent a constraint, or require clarification? Do not silently repair meaning after seeing model output.
  2. Implementation analysis: with an approved fixed property, how often were real violations found, false alarms raised, and unknowns honestly reported? Keep errors in spec translation separate from errors in code analysis.

Baselines: existing BDD/unit tests and current Own.NET rules, using the same reviewed corpus. Report number of new confirmed violations not already caught by those baselines, not just raw findings.

Success / STOP criteria

  • Short state-of-the-art / donor map (FRET vs FeaVer vs Reqnroll/Coyote vs existing Own.NET).
  • Concrete traceability record and a human-approval step for every property.
  • Frozen corpus: 10 reviewed requirements, corresponding implementation/control fixtures, and explicit assumptions.
  • Distinct spec-translation outcomes and code-verification outcomes (TP/FP/FN/UNKNOWN, by case and reason); no invented accuracy percentage.
  • At least one adjudicated new violation missed by the existing baseline, or compelling quantified reduction in review/diagnostic effort; otherwise STOP-NO-INCREMENTAL-VALUE.
  • A sample report linking requirement text/version → approved property → relevant C# symbols → source-level witness → code commit, with clear coverage/completeness limits.
  • Decide GO-EXTERNAL-PILOT, DEFER-MISSING-FACTS, or STOP-NO-UTILITY; any production changes become separately scoped issues, with owner approval and Python/Rust parity requirements where applicable.

Hard guardrails


Lexpedite / Blawx Rules as Code donor (2026-11-14)

Lexpedite / Blawx Rules as Code donor: review Blawx-powered agentic coding and bmcpdemo for human-approved rule model → inverse/forward scenarios → Hypothesis properties → implementation tests, and validation roadmap for approval/revision/gated revalidation. Full notes. Fit only as an optional external contract oracle/test-generation consumer of current OwnIR facts and source-linked witnesses; do not expand default ownership engine. A passing finite PBT suite is TESTED, never PROVEN; semantic legal/requirements interpretation needs human review. Test positive/negative/missing/exception/date boundaries, deliberately corrupted implementations, evidence gaps and explicit UNKNOWN/NOT_COVERED. Compare against plain contracts + existing testing; STOP if no measurable coverage or witness gain.


Logical English 2 / InterProlog donor audit (2026-11-14). Research page · LE2 source · docs · InterProlog. LE2 offers controlled-language typed rule templates, scenario/expected/unknown tests, explanations, Prolog and s(CASP) execution, inverse queries, local SWI-Prolog + REST/MCP. Core is Apache-2.0; translators/InsurLE extensions licensed separately; no native Russian support stated. InterProlog is JVM↔Prolog infrastructure, not C# bridge. Bounded research only; compare current pure rules/decision-table baseline before any runtime/DSL dependency. Distinguish evidence/test coverage from semantic/legal correctness.

Use LE2 only as optional approved rule → executable scenarios → implementation witness oracle. Compare default Prolog and s(CASP), negative mutations, UNKNOWN/NOT_COVERED; do not infer whole-code proof or introduce Prolog into ownership analyzer. InterProlog's Java bridge has no direct advantage for C#/Go.


Potassco ASP: clasp / clingo / Anthem / asplain — bounded donor (2026-11-14)

Полная страница исследования · clingo · clasp · Anthem · asplain. clasp — CDNL solver для grounded ASP; clingo совмещает gringo+clasp и должен быть первым кандидатом на эксперименты. Anthem через Vampire исследует ограниченные equivalence/specification свойства формальных ASP-программ; asplain — контрастивные объяснения при изменении фактов/правил, не доказательство юридической правильности. Brave = хотя бы одна допустимая модель, cautious = все допустимые модели при существовании моделей. UNSAT подтверждает невыполнимость формальной постановки, а не правовую невозможность. ASP not — default negation; UNKNOWN из отсутствующего факта не превращать в FALSE. Исследование / KILL-FIRST, не разрешение на интеграцию или production изменения.

Requirements-to-code: на human-reviewed formalized contracts проверить гипотезу поиска несовместимых свойств (policy constraints/required vs forbidden states) средствами clingo по маленькому явному abstraction model. Это contract-level model SAT, не анализ произвольного C#/OwnIR и не доказательство реальной реализации. Если в модели несколько допустимых путей, brave/cautious различают «существует» и «во всех», но не исправляют incomplete interprocedural facts. Anthem + Vampire могут проверить ограниченную equivalence формальных ASP правил; сравнить с обычным typed assertion/SMT baseline. Negative control: исключение пропущено, UNKNOWN переосмыслен в false, недоказанная ветвь выдана за доказанную. Без интеграции в ownership engine.

2026-11-15 — Future Conditions / verification evidence ladder

New prior art: Yahui Song's future-conditions study, AutoRocq, CTL-guided repair, incorrectness of OO code; full analysis.

Extend approved requirement taxonomy to preconditions, postconditions, future safety and conditional liveness with explicit fairness. Verification/evidence levels: TRACE-VIOLATION, TRACE-CONFORMS-BOUNDED, MODEL-CHECKED-SCOPED, CODE-PROVED-SCOPED, UNKNOWN. A passing finite trace does not establish liveness or full program correctness; a model proof does not prove implementation-model conformance. Proposed kill-first evaluation: 1 small authored model (3–6 states), 2 safety + 1 liveness properties, 2 happy / 3 mutated / 2 incomplete traces, independent oracle and comparison against current existing assertions/PBT. If no incremental bug/evidence value, KILL. No production DSL, new solver, CI gate, or LLM-authored property approved.

2026-11-15 — NTP4VC donor: VC generation and proof checking are distinct layers

Source: NTP4VC (ICLR 2026), analysis. Why3/Frama-C generate source-derived verification conditions; separate rule-based translation maps them to Lean/Rocq/Isabelle. The contribution is a benchmark and extraction/translation pipeline, not a C# or OwnIR verification backend. Do not claim OwnIR currently emits sound VCs or that Frama-C accepts arbitrary C#.

Potential requirement-to-proof evidence structure: reviewed R and version + source hashes → explicit code/CFG model scope → trusted VC generator and mapped source locations → proof language/axioms/libraries/translation assumptions → deterministic ATP baseline → optional LLM-generated proof candidate → trusted checker verdict + replay. Any unmodeled alias/dynamic dispatch/reflection/missing identity stays UNKNOWN; proof acceptance establishes only the formalized obligation under assumptions.

A specific future contract such as after refresh-token rotation, old token cannot be accepted first needs a precise transition abstraction and source mapping. A proof of one VC does not establish liveness, distributed consistency, or full program correctness. Start with #272 finite obligations and #453 auth conformance baseline; only research new VC machinery if baseline leaves a verified, consequential gap.

Kill-first: compare existing OwnIR diagnostics and a tiny state model against 1 meaningful example before attempting Why3 extraction from C#; no expensive 2400-rule translation port or custom proof generator by default. NTP4VC has a deliberately difficulty-selected corpus, mixed licensing, Frama-C 18 and Rocq 8.20.1 reproduction constraints; paper pass rates are not a predictor of own analyzer quality.

2026-11-15 — Structured proof trees and countermodel evidence

Two source-backed additions: Qiyuan Xu AoA/Minilang as a proof-workflow reference (AST JSON edits, proof-state tree, proof-store replay, deterministic Sledgehammer-first); Simon Cruanes Zipperposition/SMBC as distinct automated theorem prover vs bounded model finder. The targeted requirement → approved property → actual source events/identities → test/checker/oracle evidence chain remains unchanged. CHECKER-ACCEPTED, VALIDATED-COUNTERMODEL, TEST-FAIL, UNKNOWN/INCONCLUSIVE must never collapse into generic PROVED; source-code semantics, unmodeled branches, and legal/business specification fidelity still require independent justification. No new proof language, CAS engine, or solver dependency before a relevant concrete property.

Activity

  1. PhysShell commented on Nov 14, 2026

    @PhysShell
    OwnerAuthor

    Addendum (2026-11-14): Act + Gambit + Echidna as method donors for the existing #448 pilot

    Relevant reviewed upstreams:

    • Act: human-readable EVM state-transition specification with bytecode conformance/proof backends. Not a ready C#/OwnIR DSL or a generic proof of natural-language requirements.
    • Certora Gambit / mutation verifier: deliberately mutate implementation to see whether a test/verification specification detects an injected defect. Gambit itself is Solidity-specific; reuse the experimental method, not the binary.
    • Echidna: stateful invariant/property fuzzing with a counterexample sequence; again an EVM test tool, not a C# dependency.
    • Stryker.NET: existing .NET mutator may be a baseline for a small pure C# seam after TFM/build compatibility is verified.

    Concrete refinement to Gate 2, not an additional framework: for 2–3 already-reviewed requirements, freeze the rule and independent acceptance oracle first; seed narrowly chosen implementation mutants:

    1. bypass validation on an alternate branch;
    2. wrong entity/version binding after an intervening mutation;
    3. invert an explicit exception/condition or mis-handle an unknown predicate.

    Ask: which mutants are killed by existing tests/BDD, which only by an additional executable property, which by supported OwnIR analysis, and which remain UNKNOWN due to missing call/effect/version facts? Include correct near-neighbor cases to avoid vacuous false positives.

    A real extra signal is one reproducible specification-blind-spot / defect caught beyond the baseline, not a high mutation score from lots of irrelevant mutants. Report mutant type, source witness, model coverage and human adjudication. Keep the #448 GO/STOP budget unchanged; don't auto-upgrade TESTED to PROVEN.

    Separate application-level vertical: AI-Assistent-skeleton #22 examines business-rule applicability in generated answers (dates, exceptions, conflicting revisions, claim-level uncertainty). It should reuse a narrow test-oracle/mutation technique without turning Own.NET's checker into a RAG system. That issue is not a request to analyze internal company code or documents here.

  2. PhysShell commented on Nov 14, 2026

    @PhysShell
    OwnerAuthor

    Research addendum (2026-11-14): ACL2 / Kestrel APT / Syntheto / Axe as external verification donors

    Scope: supplemental methods for #448 requirements→code conformance; not an OwnIR rewrite, ACL2 dependency, or green-light for universal source equivalence. Relevant researcher: Alessandro Coglio.

    Upstream, checked in code

    • ACL2 has active commits on 2026-11-14; Kestrel APT implements mechanized transformations. Concrete simplify docs show the generated f1-is-f2 theorem. isodata docs describe proving behavior preservation across isomorphic data representations, subject to applicability/guard restrictions. finite-difference implements a proposed incrementalization path but explicitly says IN-PROGRESS.
    • Syntheto: typed surface DSL and frontend↔ACL2/APT backend with feedback on success/failure. isomorphism.synth shows subtype, ensures, transform. Its example status README explicitly identifies a non-working frontend isomorphism example; frontend repo has no published commits after Dec 2021. Not an implementation-ready UI.
    • Axe supports symbolic rewriting/proofs, including architecture/bytecode-specific tooling and dependence on STP for many use cases. Not a verified drop-in Roslyn/C# or Go semantics model.
    • Specware: older, separate approach to modular specifications/morphisms and refinement obligations. Useful architecture prior art, not a trusted direct importer of C# implementations.
    • ProvableHQ/workshop is a Leo/Aleo ZK application tutorial, not program-conformance tooling. Auction assumptions explicitly assume an honest auctioneer; Battleship demonstrates hidden state and anti-double-move invariants. ZK correctness of a circuit ≠ correctness of its business rule or implementation-to-requirement mapping. Do not integrate.

    Recommended #448 use: a deliberately tiny equivalence/refinement case

    1. Fix one reviewer-approved, pure, deterministic function with an independently defined property; e.g., model a single payment calculation (exclude mutable WPF/EF/SQL context, clocks, rounding ambiguity until explicitly modeled).
    2. Model two algorithms/representations in ACL2 only if a manual formulation is small enough. Attempt a scoped equivalence/property proof or APT transformation. Record exact assumptions, input domain, overflow/rounding and partiality; PROVED is limited to the mathematical model.
    3. Independently compare original vs transformed real C# on a frozen set of valid/invalid boundaries using existing unit/BDD/property tests. Identify the unproven model↔code translation boundary; any source property not mapped to trustworthy OwnIR facts stays UNKNOWN, not PROVEN.
    4. Seed (a) changed branch condition, (b) wrong state/version binding, and (c) changed rounding/overflow boundary. Track which violation is caught by baseline tests, OwnIR checker, separate formal model, or neither. Do not count an equivalent mutant as a kill.
    5. STOP if writing an ACL2 representation dominates effort or provides no new confirmed violation/proof over the existing Research: requirements-to-code conformance checking (BDD + reviewed formal contracts + OwnIR evidence) #448 baseline. Keep production analyzer unchanged.

    This is a narrow research lane for provable pure-function transformations, not a promise of general old-vs-new implementation equivalence. For customs payment rules, prove equivalence only in a specifically delimited unchanged-law subdomain; intentional changes in legislation must be tested against new approved normative contracts, not called regressions.

    Connected notes: Assurance-as-Code · Semantic Requirements Workbench · Demand Radar #4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions