Initial Foundational Semantics Spec

[willpugh]

Summary

Related Issues

This is MD version of the foundational semantics spec. It started in Google Docs: Proposed Foundational Semantics.

This follows from the discussions in the expression language slack group, and mail

NOTE: There were two comments in the original doc around join direction behaviour (to not lose fact table rows) and null safe joins. These have been addressed in this version, by being explicit about when null safe joins are required and being explicit about joining from the fact tables out.

Checklist

Specification

• Spec changes are included in core-spec/ and follow the existing structure
• Spec changes have been discussed on the mailing list or in a linked issue
• Breaking changes to the spec are clearly called out in the summary

Ontology

• Ontology changes in ontology/ are consistent with spec changes
• New or modified terms are defined and documented

Converters

• Converter logic in converters/ is updated to reflect spec or ontology changes
• New converters include tests under the converter's test directory

Validation

• Validation rules in validation/ are updated if the spec changed
• New validation cases are covered by tests

Documentation

• docs/ is updated to reflect any user-facing changes
• New features or behaviors are documented with examples where appropriate
• CONTRIBUTING.md is updated if the contribution process changed

Examples

• examples/ are added or updated for any new spec constructs or converter support

Tests

• All existing tests pass (pytest / CI green)
• New functionality is covered by tests

Compliance

• ASF license headers are present on all new source files
• No third-party dependencies are added without PMC/IPMC approval

[marioapiphani]

Will — read the full document. Strong support, both for the content and for the foundational-first approach: As discussed in #46, full support from an SAP perspective.

The five user-visible semantics — especially Semantic 5 (typed error, never a plausible-but-wrong number) and Semantic 1's NULL-bucketing of unmatched fact rows — address the query-time family of what we saw in the field. The NULL determinism work (§5.1 ordering defaults, §6.11 empty/NULL aggregate behaviour, NULL-safe stitch merges) pins down the places where observed engines quietly disagree. Adding composite keys as first-class citizens in relationships (from_columns/to_columns, unique_keys) covers the compound-identity shape that SAP-class models live on — our four-part keys map onto §6.4's inference very cleanly.

The worst losses we measured were not query-time — they were transport-time: an observed ontology platform (Fabric) accepted parts of our semantic model with a success response and silently discarded them. We only caught it by re-reading what was stored. The Foundation guarantees no silently wrong answer; nothing yet guarantees no silently dropped model construct. A conformance rule in the same spirit — an implementation MUST either represent a construct, reject it with a typed error, or explicitly report it as unsupported (never accept-and-drop) — would extend Semantic 5's philosophy to interchange itself, and would have caught every silent loss in our audit. The working draft's reference to a conformance suite (§11.1) looks like this rule's natural home. If useful, I'm happy to share the anonymized loss table as test material.

In SAP, there are 2 validations

1. Sentinel values vs. NULL. In SAP-sourced data, "missing" is usually not NULL — it's an initial value ('', '0000000000', '00000000'). Semantic 1's NULL bucket won't catch those; a broken reference surfaces as a legitimate-looking key instead. Not a Foundation blocker, but worth a future hook (declared sentinel values on a dataset, folding them into the NULL bucket) — otherwise real-world ERP data quietly defeats the guarantee.
2. Declared cardinality, when it lands. §6.4's inference is the right conservative default, but closed-world sources like SAP always know their cardinality — associations declare it. When the deferred declared-cardinality/RI proposal comes, two requests from the emitter side: keep declaration cheap (we declare, we never infer), and make declared-vs-inferred disagreement a typed error and not a silenced accept on any side.

Two editorial catches: (1) the body references §9, §10, §11, §12 (SQL-interface proposal, deferred-features registry, extension flags + conformance suite, vendor comparison incl. "§12.A divergence note") — none of which exist in the committed file or, as far as I can see, in the working draft either. As committed, readers hit dangling anchors; either trimming the references or landing stub sections would tighten it. (2) The working draft's §4.6.1 explicitly pins quoted-identifier case-sensitivity ("id" ≠ id, "ID" = id) and notes the conformance suite asserts it — the committed §4.6.1 dropped that pin. If the position now lives in the expression-language proposal, a pointer would help; if it was dropped accidentally, it's worth restoring — it's exactly the kind of cross-engine divergence this document exists to kill.

One clarifying question: the ontology layer's requires/derived_by fields also carry ANSI-SQL expressions. Is the intent that Ossie_SQL_2026 plus these evaluation semantics govern those expressions too, or does the ontology layer get its own semantics pass? Asking because we're building a CSN (SAP model language) → Ossie converter that emits closed-world constraints into requires, and we'd like to target whichever contract will govern them.

Happy to turn any of the above into concrete test fixtures — that's the form feedback is most useful in.

[MarioDeFelipe]

A short follow-up to the transport-time point above (Im the same Mario), because its failure mode just surfaced inside this repo — worth pinning while it's fresh.

#311 (filed and fixed by @mattfaltyn) caught the dbt converter silently corrupting composite keys: primary_key: [tenant_id, order_id] goes in, the key group is flattened into independent columns, and the round-trip comes back as primary_key: [order_id] — accepted, no error, no warning.
That means its wrong Downstream, wrong cardinality and wrong aggregations that look right: the plausible-but-wrong family Semantic 5 exists to forbid at query time, produced one layer down at interchange time. And composite keys are not an edge cases, they are common norm for tenant-scoped and line-item data (§6.4's territory; every SAP table I've ever shipped starts with a tenant-scoped compound key).

The fix in #312 lands, in the author's own words, on "an explicit ValueError" replacing "plausible but incorrect" output, that is precisely the second arm of the conformance rule proposed above: represent, reject with a typed error, or report unsupported; never accept-and-drop.

Happy to contribute a composite-key fixture pair as a first transport-conformance test, so the next case is caught by CI rather than by whoever happens to be looking.

[christianeu-db]

To follow up from the 8/12 semantics working group, the core concern we raised in this thread is how this interface will be consumed by BI tools. Many BI tools implement a high-level interface for taking a multi-table model and compiling SQL. The challenge these tools will face is layering their query semantics on top of a similar high-level query interface through SQL.

We've prepared some example Tableau queries in this linked section. We'll also create a PR containing some of these patterns and solicit other vendors for examples. Even when querying a simple single-table model, Tableau can issue multiple subqueries, self joins and unaggregated references to measures / metrics.

The other core challenge is around correctness when querying these objects through SQL, which may look like a table but act quite differently.

Concretely, tables have the property that adding grouping columns or aggregate expressions does not change which underlying rows contribute possible dimension values. For example,

SELECT d1, d2, d3 FROM T
SELECT d1 FROM T GROUP BY ALL
SELECT d1, d2 FROM T GROUP BY ALL
SELECT d1, SUM(m1) FROM T GROUP BY ALL
SELECT d1, SUM(m2), MIN(m3) FROM T GROUP BY ALL

all return the same distinct values for d1.

As shown in this Top N example, Tableau depends on this property to optimize queries. Specifically, it can combine two subqueries together that compute two different measures by the same dimension because it relies on the fact that these dimension domains will be the same.

That is not the case for the proposed foundational semantics. Because each measure anchors on its own fact and join path, the dimension values in a result depend on which measures are present: a measure contributes only the values its fact reaches (§6.6, LEFT-from-fact) plus a null bucket for its unmatched rows (Semantic 1), and combining measures from different facts unions their group sets (Semantic 3). So two queries differing only in their measures can return different domains for the same dimension.

In the 8/12 meeting, Will described adding a fixed grain for an Ossie model. However, that idea should be fleshed out formally. As we understood this proposal, the idea is that all joins would join out from a specific anchor table. That preserves domain consistency but also requires careful integration with the rest of the proposal. For example,

1. Let's start with the model of Actors-Appearances-Movies. The current semantics say no row input for a measure should be dropped. Fixing an anchor of actors and left-joining outward means a measure on movies is computed only over movies that have a matching appearance - so movies with no matching appearances are dropped. This behavior directly violates the spec's Semantic 1 (no row input for a measure should be dropped).
2. The Ossie model allows for edges which could introduce path ambiguity if some combinations of dimensions / measures are used together. If grain/domain is fixed, path ambiguity can't be resolved lazily per query (some queries would just fail); it has to be resolved at model-definition time.

Of course, we are advocating for general SQL interoperability by extending tables, so we are happy to collaborate on exploring this direction.

Some of the decision points we’d like to see clarity on are:

1. Are Ossie models designed to be safely consumable by BI tools that generate their own SQL? How can we prove this correctness?
2. From the perspective of SQL consumers, does an Ossie model expose the semantics of a single relational object with a stable domain, or of a multi-table model whose effective joins / domains depend on the query - which likely contradicts general SQL integration?

[willpugh]

Yes. The idea of the Foundational Semantics is that they are something we get out that describes the current Ossie mode, that we can build on later. Many of the things proposed are things we all want, e.g. a SQL interface for querying, metrics on tables, etc. They were just prioritized behind adding semantics for the model we currently have. The big stickiness is how we want to deal with the "wide table"-ness. This is the root of the semantic troubles. Can we query an Ossie model as a single table (with heuristic joins under the cover) or does it act more like a schema and each tool joins on its own? Justin put out a doc a while ago that resonated with me (although, I can't find a public version). It proposes 3 layers: * Layer 3: Wide table Interface * Layer 2: SQL Measures * Layer 1: Common expression language Some of the core of this discussion is that in my mind, we want to build out all 3. However, the original Ossie spec started with a description of Layer 3. This current discussion, I believe is about what a proper Layer 2 would look like. My current approach was to get semantics that define what Layer 3 looks like, so we at least have defined behaviour. As you mentioned, I believe we can add better behaviour through the concept of a root grain.

…

On Wed, Aug 12, 2026 at 2:58 PM Chris Eubank ***@***.***> wrote: *christianeu-db* left a comment (apache/ossie#246) <#246 (comment)> To follow up from the 8/12 semantics working group, the core concern we raised in this thread <https://lists.apache.org/thread/mqlbbb4ndhb0qo9sf2yn60fb9tlzv59t> is how this interface will be consumed by BI tools. Many BI tools implement a high-level interface for taking a multi-table model and compiling SQL. The challenge these tools will face is layering their query semantics on top of a similar high-level query interface through SQL. We've prepared some example Tableau queries in this linked section <https://docs.google.com/document/d/1uKc3x9NlvV9O9UPdvlnCTDxsc-XWsQrxb0ylG45H-IQ/edit?tab=t.22qc6753kxb3#heading=h.ousekbpu23pw>. We'll also create a PR containing some of these patterns and solicit other vendors for examples. Even when querying a simple single-table model, Tableau can issue multiple subqueries, self joins and unaggregated references to measures / metrics. The other core challenge is around correctness when querying these objects through SQL, which may look like a table but act quite differently. Concretely, tables have the property that adding grouping columns or aggregate expressions does not change which underlying rows contribute possible dimension values. For example, SELECT d1, d2, d3 FROM T SELECT d1 FROM T GROUP BY ALL SELECT d1, d2 FROM T GROUP BY ALL SELECT d1, SUM(m1) FROM T GROUP BY ALL SELECT d1, SUM(m2), MIN(m3) FROM T GROUP BY ALL all return the same distinct values for d1. As shown in this Top N example <https://docs.google.com/document/d/1uKc3x9NlvV9O9UPdvlnCTDxsc-XWsQrxb0ylG45H-IQ/edit?tab=t.22qc6753kxb3#heading=h.ousekbpu23pw>, Tableau depends on this property to optimize queries. Specifically, it can combine two subqueries together that compute two different measures by the same dimension because it relies on the fact that these dimension domains will be the same. That is not the case for the proposed foundational semantics. Because each measure anchors on its own fact and join path, the dimension values in a result depend on which measures are present: a measure contributes only the values its fact reaches (§6.6, LEFT-from-fact) plus a null bucket for its unmatched rows (Semantic 1), and combining measures from different facts unions their group sets (Semantic 3). So two queries differing only in their measures can return different domains for the same dimension. In the 8/12 meeting, Will described adding a fixed grain for an Ossie model. However, that idea should be fleshed out formally. As we understood this proposal, the idea is that all joins would join out from a specific anchor table. That preserves domain consistency but also requires careful integration with the rest of the proposal. For example, 1. Let's start with the model of Actors-Appearances-Movies. The current semantics say no row input for a measure should be dropped. Fixing an anchor of actors and left-joining outward means a measure on movies is computed only over movies that have a matching appearance - so movies with no matching appearances are dropped. This behavior directly violates the spec's Semantic 1 (no row input for a measure should be dropped). 2. The Ossie model allows for edges which could introduce path ambiguity if some combinations of dimensions / measures are used together. If grain/domain is fixed, path ambiguity can't be resolved lazily per query (some queries would just fail); it has to be resolved at model-definition time. Of course, we are advocating for general SQL interoperability by extending tables <https://docs.google.com/document/d/1uKc3x9NlvV9O9UPdvlnCTDxsc-XWsQrxb0ylG45H-IQ/edit?tab=t.0#heading=h.5vtm2vjqu4mp>, so we are happy to collaborate on exploring this direction. Some of the decision points we’d like to see clarity on are: 1. Are Ossie models designed to be safely consumable by BI tools that generate their own SQL? How can we prove this correctness? 2. From the perspective of SQL consumers, does an Ossie model expose the semantics of a single relational object with a stable domain, or of a multi-table model whose effective joins / domains depend on the query - which likely contradicts general SQL integration? — Reply to this email directly, view it on GitHub <#246?email_source=notifications&email_token=AAQXKZHTDYNWDPBGNWUSU535JTSARA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRXGMZDKOBRGI42M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5273258129>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQXKZBI2QNEWOCKSXVKPGT5JTSARAVCNFSNUABGKJSXA33TNF2G64TZHMYTAOJZGM4DEMJTG45US43TOVSTWNBZGQ4TKOJRHE2THILWAI> . You are receiving this because you authored the thread.Message ID: ***@***.***>