Add semantic SQL discovery and definition lifecycle

.github/workflows/ci.yml

@@ -783,7 +783,9 @@ jobs:
         env:
           SIDEMANTIC_DUCKDB_BINARY: sidemantic-duckdb/build/release/duckdb
           SIDEMANTIC_DUCKDB_EXTENSION: sidemantic-duckdb/build/release/extension/sidemantic/sidemantic.duckdb_extension
-        run: uv run sidemantic-duckdb/test/test_semantic_input_host.py
+        run: |
+          uv run sidemantic-duckdb/test/test_semantic_input_host.py
+          uv run sidemantic-duckdb/test/test_catalog_host.py
 
       - name: Save completed build caches
         if: ${{ !cancelled() }}

sidemantic-duckdb/CMakeLists.txt

@@ -145,7 +145,7 @@ add_custom_target(sidemantic_rust_staticlib DEPENDS "${SIDEMANTIC_LIB}")
 # Include Rust library headers
 include_directories(${SIDEMANTIC_INCLUDE})
 
-set(EXTENSION_SOURCES src/sidemantic_extension.cpp src/sidemantic_parser.cpp src/sidemantic_grammar.cpp src/sidemantic_catalog.cpp)
+set(EXTENSION_SOURCES src/sidemantic_extension.cpp src/sidemantic_parser.cpp src/sidemantic_grammar.cpp src/sidemantic_catalog.cpp src/sidemantic_api.cpp)
 
 build_static_extension(${TARGET_NAME} ${EXTENSION_SOURCES})
 build_loadable_extension(${TARGET_NAME} " " ${EXTENSION_SOURCES})

sidemantic-duckdb/README.md

@@ -10,6 +10,7 @@ A DuckDB extension that adds a SQL-first semantic layer. Define metrics and dime
 - **Fan-out Protection**: Uses model keys to protect supported aggregations against duplicated rows from joins
 - **Definition Files**: Load native YAML, Cube.js YAML, and native SQL definition files
 - **Transactional Definitions**: Commit and roll back model changes with DuckDB transactions
+- **Catalog Discovery**: Inspect definitions, metadata, relationships and compatible dimensions with SQL
 - **SQL Containers**: Use semantic queries in views, `CREATE TABLE AS`, and `INSERT ... SELECT`
 - **Native PEG Grammar**: Composable grammar on the pinned Cyanoptera development build, with a DuckDB 1.5.6 compatibility frontend
 
@@ -42,11 +43,8 @@ LOAD 'build/release/extension/sidemantic/sidemantic.duckdb_extension';
 
 For embedded clients, set DuckDB's `allow_unsigned_extensions` database configuration before opening the connection. Community extension installation is planned, but this repository does not yet publish the signed multi-platform artifacts required for `INSTALL sidemantic FROM community`.
 
-In DuckDB 1.5.6 shells that preload `autocomplete` (including Homebrew), use
-`SEMANTIC SELECT` for semantic queries. That extension's PEG override consumes
-ordinary SQL before Sidemantic receives it, and 1.5.6 has no parser-priority setting.
-Clients without that competing override support automatic routing. Explicit
-`SEMANTIC SELECT` works in both configurations.
+Semantic queries use ordinary `SELECT`, including in shells that preload
+`autocomplete`. The older `SEMANTIC SELECT` spelling remains supported.
 
 ## Quick Start (Pure SQL)
 
@@ -76,7 +74,7 @@ METRIC avg_order_value AS AVG(amount);
 DIMENSION (name status, type categorical);
 
 -- 5. Query using semantic layer
-SEMANTIC SELECT orders_model.status, orders_model.revenue FROM orders_model;
+SELECT orders_model.status, orders_model.revenue FROM orders_model;
 -- Automatically rewrites to:
 -- SELECT status, SUM(amount) FROM orders GROUP BY 1
 
@@ -89,6 +87,60 @@ SEMANTIC SELECT orders_model.status, orders_model.revenue FROM orders_model;
 -- └───────────┴────────────────────┘
 ```
 
+## Discover and manage definitions
+
+```sql
+SHOW MODELS;
+SHOW METRICS FROM orders_model;
+SHOW DIMENSIONS FROM orders_model;
+SHOW RELATIONSHIPS FROM orders_model;
+SHOW SEGMENTS FROM orders_model;
+DESCRIBE MODEL orders_model;
+
+-- Dimensions that can be combined with this metric
+SHOW DIMENSIONS FOR orders_model.revenue;
+
+-- Validate the complete selection, predicates and physical database bindings
+EXPLAIN SELECT orders_model.status, orders_model.revenue
+FROM orders_model WHERE orders_model.status = 'completed';
+```
+
+Discovery returns the definition kind, model and qualified name, label,
+description, semantic and declared data types, SQL, aggregation, relationship
+target and cardinality, granularity, visibility, and full JSON definition.
+Absent metadata is SQL `NULL`. `DESCRIBE MODEL` includes the model and all its
+fields. `SHOW DIMENSIONS FOR` checks each dimension with the query compiler;
+use `EXPLAIN SELECT` to validate a complete combination of metrics and dimensions.
+`SHOW SEMANTIC METRICS`, and the corresponding forms for other kinds, are also
+accepted.
+
+Qualified definitions work independently of the connection's active model:
+
+```sql
+CREATE OR REPLACE METRIC orders_model.revenue AS SUM(amount);
+CREATE OR REPLACE DIMENSION orders_model.status AS status;
+CREATE SEGMENT orders_model.completed AS status = 'completed';
+DROP SEGMENT orders_model.completed;
+DROP METRIC IF EXISTS orders_model.old_metric;
+DROP MODEL IF EXISTS retired_model;
+```
+
+Removal uses dependency restrictions: a referenced model or field cannot be
+dropped until the dependent definitions are removed or changed. There is no
+implicit cascade. Definitions, removal and import participate in transactions;
+`PREPARE` and `EXPLAIN` do not apply a mutation.
+
+```sql
+EXPORT SEMANTIC CATALOG;
+-- Returns one versioned JSON snapshot in the definition column.
+IMPORT SEMANTIC CATALOG '<snapshot returned by EXPORT>';
+```
+
+Export includes models, global metrics, parameters, table calculations and
+metadata. Import validates the complete candidate and merges it atomically into
+the selected database. A definition with the same name is replaced. Export and
+discovery work on read-only databases and observe the current transaction.
+
 ## Versioned SemanticInput
 
 The stateless SQL functions `sidemantic_compile_semantic_input(input_json, query_json)`
@@ -236,13 +288,13 @@ files raise errors when the semantic catalog is accessed.
 ## Native PEG frontend
 
 DuckDB 1.5.6 uses the compatibility frontend. The pinned `v2.0-cyanoptera` build
-registers a `sidemantic` grammar extension for model/item declarations and
-`SEMANTIC SELECT`, including nested `PREPARE` and `EXPLAIN`. Definition properties
+registers a `sidemantic` grammar extension for definitions, discovery, removal,
+import/export and explicit semantic queries, including nested `PREPARE` and `EXPLAIN`. Definition properties
 still use the shared Rust configuration parser; semantic query compilation still
 uses the Rust SQL AST.
 
-Loading Sidemantic enables its parser override for automatic routing, subject to
-the 1.5.6 autocomplete limitation above. To explicitly compose the native
+Loading Sidemantic enables automatic routing, including a planning hook for
+hosts with another parser override. To explicitly compose the native
 grammar with other installed grammars, include it in `active_grammar_extensions`:
 
 ```sql
@@ -479,10 +531,10 @@ rows, including equal-valued metrics on distinct customers.
 ## Current support boundaries
 
 - The query frontend handles `SELECT`, `CREATE VIEW`, `CREATE TABLE AS`, and
-  `INSERT ... SELECT`, including `PREPARE` and `EXPLAIN` wrappers. Use the
-  `SEMANTIC` prefix when another extension takes priority for ordinary SQL.
-- Semantic definitions support creation, replacement, loading, and active-model
-  selection. There is no `DROP MODEL`, `ALTER MODEL`, or item-removal SQL interface.
+  `INSERT ... SELECT`, including `PREPARE` and `EXPLAIN` wrappers.
+- Semantic definitions support creation, replacement, loading, discovery,
+  export/import, active-model selection and dependency-restricted removal.
+  Use `CREATE OR REPLACE` to update definitions; there is no `ALTER MODEL` syntax.
 - Semantic queries support a subset of DuckDB SQL. Correlated semantic subqueries
   and grouping modifiers such as `ROLLUP` are rejected; explicit `GROUP BY` must
   repeat the selected semantic dimensions.

sidemantic-duckdb/src/include/sidemantic_api.hpp

@@ -0,0 +1,7 @@
+#pragma once
+
+#include "duckdb/parser/parser_extension.hpp"
+
+namespace duckdb {
+ParserExtensionPlanResult PlanSidemanticCatalog(const string &kind, const string &model, const string &metric);
+}

sidemantic-duckdb/src/include/sidemantic_parser.hpp

@@ -33,6 +33,7 @@ ParserExtensionParseResult sidemantic_parse(ParserExtensionInfo *, const string
 ParserExtensionPlanResult sidemantic_plan(ParserExtensionInfo *, ClientContext &,
                                          unique_ptr<ParserExtensionParseData>);
 BoundStatement sidemantic_bind(ClientContext &, Binder &, OperatorExtensionInfo *, SQLStatement &);
+void RegisterSidemanticRouting(DatabaseInstance &db);
 
 struct SidemanticParserExtension : ParserExtension {
     SidemanticParserExtension() {

sidemantic-duckdb/src/sidemantic_api.cpp

@@ -0,0 +1,109 @@
+#include "sidemantic_api.hpp"
+#include "sidemantic_catalog.hpp"
+#include "sidemantic_compat.hpp"
+#include "sidemantic.h"
+
+#include "duckdb/function/table_function.hpp"
+#include "duckdb/planner/binder.hpp"
+
+namespace duckdb {
+namespace {
+
+struct CatalogBindData : TableFunctionData {
+    string kind;
+    string model;
+    string metric;
+};
+
+struct CatalogState : GlobalTableFunctionState {
+    vector<vector<Value>> rows;
+    idx_t offset = 0;
+};
+
+static unique_ptr<FunctionData> CatalogBind(ClientContext &context, TableFunctionBindInput &input,
+                                          vector<LogicalType> &types, vector<SidemanticIdentifier> &names) {
+    if (input.binder) RegisterSidemanticCatalogRead(context, input.binder->GetStatementProperties());
+    auto data = make_uniq<CatalogBindData>();
+    data->kind = input.inputs[0].GetValue<string>();
+    data->model = input.inputs[1].GetValue<string>();
+    data->metric = input.inputs[2].GetValue<string>();
+    if (data->kind == "export") {
+        names.emplace_back("definition");
+        types.emplace_back(LogicalType::VARCHAR);
+        return std::move(data);
+    }
+    for (auto name : {"kind", "model_name", "name", "qualified_name", "label", "description", "type", "data_type",
+                      "sql", "aggregation", "target_model", "relationship_type", "granularity"}) {
+        names.emplace_back(name);
+        types.emplace_back(LogicalType::VARCHAR);
+    }
+    names.emplace_back("is_public");
+    types.emplace_back(LogicalType::BOOLEAN);
+    names.emplace_back("definition");
+    types.emplace_back(LogicalType::VARCHAR);
+    return std::move(data);
+}
+
+static unique_ptr<GlobalTableFunctionState> CatalogInit(ClientContext &context, TableFunctionInitInput &input) {
+    auto &data = input.bind_data->Cast<CatalogBindData>();
+    auto snapshot = ReadSidemanticCatalog(context);
+    auto state = make_uniq<CatalogState>();
+    if (data.kind == "export") {
+        auto result = sidemantic_snapshot_export(snapshot.payload.c_str());
+        if (result.error) {
+            string error(result.error);
+            sidemantic_free_result(result);
+            throw InvalidInputException("Sidemantic: %s", error);
+        }
+        string definition(result.sql);
+        sidemantic_free_result(result);
+        state->rows.push_back({Value(definition)});
+        return std::move(state);
+    }
+    struct CatalogResult {
+        SidemanticCatalogEntries result;
+        ~CatalogResult() { sidemantic_free_catalog_entries(result); }
+    } result {sidemantic_snapshot_catalog(snapshot.payload.c_str(), data.kind.c_str(), data.model.c_str(),
+                                          data.metric.empty() ? nullptr : data.metric.c_str())};
+    if (result.result.error) throw InvalidInputException("Sidemantic: %s", result.result.error);
+    for (idx_t i = 0; i < result.result.count; ++i) {
+        auto &entry = result.result.entries[i];
+        vector<Value> row;
+        for (auto value : {entry.kind, entry.model_name, entry.name, entry.qualified_name, entry.label,
+                          entry.description, entry.semantic_type, entry.data_type, entry.sql, entry.aggregation,
+                          entry.target_model, entry.relationship_type, entry.granularity}) {
+            row.push_back(value ? Value(value) : Value(LogicalType::VARCHAR));
+        }
+        row.push_back(Value::BOOLEAN(entry.is_public));
+        row.push_back(Value(entry.definition));
+        state->rows.push_back(std::move(row));
+    }
+    return std::move(state);
+}
+
+static void CatalogFunction(ClientContext &, TableFunctionInput &input, DataChunk &output) {
+    auto &state = input.global_state->Cast<CatalogState>();
+    auto count = MinValue<idx_t>(STANDARD_VECTOR_SIZE, state.rows.size() - state.offset);
+    for (idx_t row = 0; row < count; ++row) {
+        auto &values = state.rows[state.offset + row];
+        for (idx_t column = 0; column < values.size(); ++column) output.SetValue(column, row, values[column]);
+    }
+    state.offset += count;
+    output.SetCardinality(count);
+}
+
+} // namespace
+
+ParserExtensionPlanResult PlanSidemanticCatalog(const string &kind, const string &model, const string &metric) {
+    // Parser-owned lowering only: these functions are never registered in the
+    // SQL function catalog. Users inspect definitions with SHOW and DESCRIBE.
+    ParserExtensionPlanResult result;
+    result.function = TableFunction("sidemantic_catalog_scan",
+                                   {LogicalType::VARCHAR, LogicalType::VARCHAR, LogicalType::VARCHAR},
+                                   CatalogFunction, CatalogBind, CatalogInit);
+    result.parameters = {Value(kind), Value(model), Value(metric)};
+    result.return_type = StatementReturnType::QUERY_RESULT;
+    return result;
+}
+
+} // namespace duckdb

sidemantic-duckdb/src/sidemantic_extension.cpp

@@ -160,6 +160,7 @@ static void LoadInternal(ExtensionLoader &loader) {
     parser.parser_info = RegisterSidemanticGrammar(db);
     ParserExtension::Register(config, parser);
     OperatorExtension::Register(config, make_shared_ptr<SidemanticOperatorExtension>());
+    RegisterSidemanticRouting(db);
 
     loader.RegisterFunction(TableFunction("sidemantic_load", {LogicalType::VARCHAR}, LoadFunction, LoadBind<false>, LoadInit));
     loader.RegisterFunction(TableFunction("sidemantic_load_file", {LogicalType::VARCHAR}, LoadFunction, LoadBind<true>, LoadInit));

sidemantic-duckdb/src/sidemantic_grammar.cpp

@@ -123,11 +123,21 @@ class SidemanticGrammar final : public GrammarExtension {
             GrammarChange::AddRule("SidemanticModel <- 'MODEL' (SidemanticBlock / ColIdOrString ('FROM' (SidemanticSource / SidemanticBlock) SidemanticBlock / SidemanticBlock)?)"),
             GrammarChange::AddRule("SidemanticItem <- ('METRIC' / 'DIMENSION' / 'SEGMENT') (SidemanticBlock / SidemanticSource ('AS' Expression / SidemanticBlock))"),
             GrammarChange::AddRule("SidemanticDefinition <- 'SEMANTIC'? ('CREATE' ('OR' 'REPLACE')?)? (SidemanticModel / SidemanticItem)", StartDefinition),
+            GrammarChange::AddRule("SidemanticDrop <- 'SEMANTIC'? 'DROP' ('MODEL' / 'METRIC' / 'DIMENSION' / 'SEGMENT') ('IF' 'EXISTS')? SidemanticSource", StartDefinition),
+            GrammarChange::AddRule("SidemanticShow <- 'SHOW' 'SEMANTIC'? ('MODELS' / 'METRICS' / 'DIMENSIONS' / 'SEGMENTS' / 'RELATIONSHIPS') (('FROM' ColIdOrString) / ('FOR' SidemanticSource))?", StartDefinition),
+            GrammarChange::AddRule("SidemanticDescribe <- ('DESCRIBE' / 'DESC') 'MODEL' ColIdOrString", StartDefinition),
+            GrammarChange::AddRule("SidemanticExport <- 'EXPORT' 'SEMANTIC' 'CATALOG'", StartDefinition),
+            GrammarChange::AddRule("SidemanticImport <- 'IMPORT' 'SEMANTIC' 'CATALOG' ColIdOrString", StartDefinition),
             GrammarChange::AddRule("SidemanticQuery <- 'SEMANTIC' (SelectStatement / CreateStatement / InsertStatement)", StartSemanticQuery),
-            GrammarChange::AddRule("SidemanticPreparable <- SidemanticDefinition / SidemanticQuery"),
+            GrammarChange::AddRule("SidemanticPreparable <- SidemanticShow / SidemanticDescribe / SidemanticExport / SidemanticImport / SidemanticDrop / SidemanticDefinition / SidemanticQuery"),
             GrammarChange::AddRule("SidemanticPrepare <- 'PREPARE' ColIdOrString 'AS' SidemanticPreparable", StartPrepare),
             GrammarChange::AddRule("SidemanticExplain <- 'EXPLAIN' AnalyzeKeyword? ExplainOptionList? SidemanticPreparable", StartExplain),
             GrammarChange::PrependChoice("Statement", "SidemanticDefinition"),
+            GrammarChange::PrependChoice("Statement", "SidemanticDrop"),
+            GrammarChange::PrependChoice("Statement", "SidemanticShow"),
+            GrammarChange::PrependChoice("Statement", "SidemanticDescribe"),
+            GrammarChange::PrependChoice("Statement", "SidemanticExport"),
+            GrammarChange::PrependChoice("Statement", "SidemanticImport"),
             GrammarChange::PrependChoice("Statement", "SidemanticQuery"),
             GrammarChange::PrependChoice("Statement", "SidemanticPrepare"),
             GrammarChange::PrependChoice("Statement", "SidemanticExplain"),