From 18b7d8a37d997af7df68755e92eca54e9695d4dd Mon Sep 17 00:00:00 2001 From: boringdata Date: Tue, 27 Oct 2026 16:04:07 +0000 Subject: [PATCH] docs: render code blocks and multi-paragraph content inside notes (#312) The Note component only kept plain-string children, so anything after the first blank line inside a (code blocks, extra paragraphs) was dropped. On query-methods the blank line sat inside a code fence, which ended the raw HTML block mid-fence: the rest of the example leaked out as a heading and the stray fence swallowed the following section. - Note renders already-parsed element children alongside its markdown strings, with note-sized paragraph styles. - query-methods: blank line before the fence so it parses as markdown. - semantic-table: blank line before so a trailing paragraph doesn't absorb the closing tag. Co-Authored-By: Claude Opus 5.5 --- docs/md/doc/query-methods.md | 1 + docs/md/doc/semantic-table.md | 1 + docs/web/public/bsl-data/query-methods.json | 2 +- docs/web/public/bsl-data/semantic-table.json | 2 +- docs/web/src/components/Note.tsx | 38 +++++++++++--------- 5 files changed, 25 insertions(+), 19 deletions(-) diff --git a/docs/md/doc/query-methods.md b/docs/md/doc/query-methods.md index 23a50825..3cd831d3 100644 --- a/docs/md/doc/query-methods.md +++ b/docs/md/doc/query-methods.md @@ -129,6 +129,7 @@ The `aggregate()` method calculates measures after grouping. You can reference p **CRITICAL**: `aggregate()` takes **measure names as strings**, not expressions or lambdas directly. Use measure names from `get_model()` output. + ```python # ✅ CORRECT - measure names as strings model.group_by("category").aggregate("flight_count", "total_revenue") diff --git a/docs/md/doc/semantic-table.md b/docs/md/doc/semantic-table.md index cfeb1480..a256e132 100644 --- a/docs/md/doc/semantic-table.md +++ b/docs/md/doc/semantic-table.md @@ -241,6 +241,7 @@ dropoff_locs = to_semantic_table(locs_tbl.view(), "dropoff_locs") The distinct names prevent ambiguous semantic prefixes; `.view()` prevents Ibis from treating both roles as the same relation. + diff --git a/docs/web/public/bsl-data/query-methods.json b/docs/web/public/bsl-data/query-methods.json index bd4d9cbd..ec1ab6cf 100644 --- a/docs/web/public/bsl-data/query-methods.json +++ b/docs/web/public/bsl-data/query-methods.json @@ -1,5 +1,5 @@ { - "markdown": "# Query Methods\n\n## Overview\n\nBSL provides a simple and consistent query API for retrieving data from your semantic tables. Queries are built by chaining methods, then executed or inspected using output methods.\n\nStart with a semantic table and chain methods together. Here's the typical query flow:\n\n```setup_table\nimport ibis\nfrom ibis import _\nfrom boring_semantic_layer import to_semantic_table\n\n# Create Ibis table\nflights_tbl = ibis.memtable({\n \"origin\": [\"NYC\", \"LAX\", \"NYC\", \"SFO\", \"LAX\", \"NYC\", \"SFO\", \"LAX\", \"NYC\"],\n \"carrier\": [\"AA\", \"UA\", \"AA\", \"UA\", \"AA\", \"UA\", \"AA\", \"UA\", \"AA\"],\n \"distance\": [2789, 2789, 2902, 2902, 347, 2789, 347, 347, 2789],\n \"duration\": [330, 330, 360, 360, 65, 330, 65, 65, 330],\n})\n\n# Create semantic table\nflights_st = (\n to_semantic_table(flights_tbl, name=\"flights\")\n .with_dimensions(\n origin=lambda t: t.origin,\n carrier=lambda t: t.carrier,\n )\n .with_measures(\n flight_count=lambda t: t.count(),\n total_distance=lambda t: t.distance.sum(),\n avg_duration=lambda t: t.duration.mean(),\n )\n)\n```\n\n\n\n```python\nresult = (\n flights_st # Start with semantic table\n .filter(_.distance > 1000) # 1. Filter (optional)\n .group_by(\"origin\") # 2. Group by dimensions\n .aggregate(\"flight_count\", \"total_distance\") # 3. Aggregate measures\n .mutate(avg=lambda t: t.total_distance / t.flight_count) # 4. Transform (optional)\n .order_by(ibis.desc(\"flight_count\")) # 5. Sort (optional)\n .limit(10) # 6. Limit rows (optional)\n)\n```\n\nOnce you've built a query, you can inspect it or execute it:\n\n```simple_demo\n\n# Build a query\nresult = flights_st.group_by(\"origin\").aggregate(\"flight_count\")\n\n# Option 1: Execute and get data as pandas DataFrame\ndf = result.execute()\n\n# Option 2: View the generated SQL\nprint(result.sql())\n\n# Option 3: Generate a visualization (when applicable)\nchart = result.chart()\n\n# Option 4: See the semantic query plan\nprint(result)\n\nresult\n```\n\n\nThe output above includes a **Query Plan** tab showing how BSL translates this query into semantic operations. \n\nYou print the query object directly to see the plan:\n```python\nprint(result)\n```\n\nThis displays operations like `SemanticTableOp`, `SemanticGroupByOp`, and `SemanticAggregateOp`, useful for debugging and understanding query execution.\n\nLet's get now into the details of each query method.\n\n## group_by()\n\nThe `group_by()` method groups data by one or more dimensions.\n\n\n`group_by()` only accepts string dimension names that were previously defined in `with_dimensions()`. It does not support lambda functions or unbound `_` syntax.\n\n\n### Single Dimension\n\nGroup by a single dimension:\n\n```query_single_dimension\n# Group by one dimension\nresult = flights_st.group_by(\"origin\").aggregate(\"flight_count\")\n```\n\n\n\n### Multiple Dimensions\n\nGroup by multiple dimensions to create detailed breakdowns:\n\n```query_multiple_dimensions\n# Group by multiple dimensions\nresult = flights_st.group_by(\"origin\", \"carrier\").aggregate(\"flight_count\")\n```\n\n\n\n### No Grouping\n\nCalculate overall statistics across all rows using `group_by()` with no arguments:\n\n```query_no_grouping\n# Aggregate entire dataset without grouping\nresult = flights_st.group_by().aggregate(\"flight_count\", \"total_distance\", \"avg_duration\")\n```\n\n\n\n## aggregate()\n\nThe `aggregate()` method calculates measures after grouping. You can reference pre-defined measures or compute new ones on-the-fly.\n\n\n**CRITICAL**: `aggregate()` takes **measure names as strings**, not expressions or lambdas directly. Use measure names from `get_model()` output.\n```python\n# \u2705 CORRECT - measure names as strings\nmodel.group_by(\"category\").aggregate(\"flight_count\", \"total_revenue\")\n\n# \u274c WRONG - no standalone lambdas in aggregate\nmodel.aggregate(total=lambda t: t.sum()) # ERROR!\n```\n\n\n### Pre-defined Measures\n\nReference measures by their string names:\n\n```query_predefined_measures\n# Use measures defined in with_measures()\nresult = flights_st.group_by(\"origin\").aggregate(\"flight_count\", \"avg_duration\")\n```\n\n\n\n### On-the-Fly Transformations\n\nAdd computed measures directly in `aggregate()` without modifying the semantic table:\n\n```query_onthefly_measures\n# Mix predefined and computed measures\nresult = (\n flights_st\n .group_by(\"origin\")\n .aggregate(\n \"flight_count\", # Pre-defined measure\n \"avg_duration\", # Pre-defined measure\n total_miles=lambda t: t.distance.sum(), # Computed on-the-fly\n max_distance=lambda t: t.flight_count + 2 # You can reference other measures as well\n )\n)\n```\n\n\n\n\nOn-the-fly measures let you add context-specific calculations without modifying your semantic table definition. This keeps your base model clean while enabling flexible queries.\n\n\n### Referencing Table Columns\n\nYou can reference **any column from the underlying table** in `aggregate()`, not just pre-defined measures. This is useful when you need one-off calculations without cluttering your semantic table definition.\n\n```query_table_columns\n# Reference table columns directly in aggregate()\nresult = (\n flights_st\n .group_by(\"origin\")\n .aggregate(\n \"flight_count\", # Pre-defined measure\n total_distance=lambda t: t.distance.sum(), # Table column 'distance'\n avg_duration=lambda t: t.duration.mean(), # Table column 'duration'\n distance_in_km=lambda t: (t.distance * 1.60934).sum() # Transform then aggregate\n )\n)\n```\n\n\n\n**Key points:**\n- Table columns **must be aggregated** (e.g., `.sum()`, `.mean()`, `.max()`, `.count()`)\n- You can transform columns before aggregating (e.g., `(t.distance * 1.60934).sum()`)\n- This works for any column in the underlying table, even if not defined as a dimension or measure\n- Use this for ad-hoc calculations without modifying your semantic table\n\n\nTable columns cannot be used without an aggregation function. For example, `lambda t: t.distance` will fail. You must use `lambda t: t.distance.sum()` or another aggregation.\n\n\n## filter() / order_by() / limit() \n\nCombine `filter()`, `order_by()`, and `limit()` to refine your query results.\n\n```query_filter_order_limit\nfrom ibis import _\n\n# Filter data, sort, and limit results\nresult = (\n flights_st\n .filter(lambda t: t.origin.isin([\"NYC\", \"LAX\"])) # Filter origins\n .filter(_.distance > 500) # Filter distance using _ syntax\n .group_by(\"origin\")\n .aggregate(\"flight_count\", \"avg_duration\") # Aggregate both measures\n .order_by(ibis.desc(\"flight_count\")) # Sort by flight_count descending\n .limit(5) # Top 5 results\n)\n```\n\n\n\n**Key points:**\n- **`filter()`**: Use lambda or `_` syntax to apply conditions before aggregation\n- **`order_by()`**: Use `ibis.desc()` for descending order, or column name for ascending\n- **`limit()`**: Restrict the number of rows returned\n\n### Critical Filter Patterns\n\n**Multiple conditions** - use `ibis.and_()` or `ibis.or_()`:\n\n```python\n# Multiple conditions with AND\nmodel.filter(lambda t: ibis.and_(t.amount > 1000, t.year >= 2023))\n\n# Multiple conditions with OR\nmodel.filter(lambda t: ibis.or_(t.status == \"active\", t.status == \"pending\"))\n```\n\n**IN operator** - MUST use `.isin()` method:\n\n```python\n# \u2705 CORRECT - use .isin() method\nmodel.filter(lambda t: t.region.isin([\"US\", \"EU\", \"APAC\"]))\n\n# \u274c WRONG - Python's 'in' does NOT work!\nmodel.filter(lambda t: t.region in [\"US\", \"EU\"]) # ERROR: truth value of Ibis expression is not defined\n```\n\n**Lambda column names** - use column names directly, never prefix with model name:\n\n```python\n# \u2705 CORRECT - use column name directly\nmodel.filter(lambda t: t.carrier == \"AA\")\n\n# \u274c WRONG - do NOT prefix with model name\nmodel.filter(lambda t: t.model.carrier == \"AA\") # ERROR!\n```\n\n**Joined columns** - use exact prefixed name from `get_model()`:\n\n```python\n# If get_model() shows \"customers.country\", use it exactly:\nmodel.filter(lambda t: t.customers.country == \"US\")\n\n# \u274c WRONG - don't call methods on ID columns\nmodel.filter(lambda t: t.customer_id.country()) # ERROR: no such method!\n```\n\n## nest()\n\nThe `nest` parameter in `aggregate()` creates nested data structures (arrays of structs) in your query results. This is useful for API responses, hierarchical visualizations, and preserving relationships in aggregated data.\n\nUse `nest` to collect rows as structured arrays within each group:\n\n```query_basic_nest\nfrom ibis import _\n\n# Nest flight details within each origin\nresult = (\n flights_st\n .group_by(\"origin\")\n .aggregate(\n \"flight_count\",\n \"total_distance\",\n # Create nested array of flight details\n nest={\"flights\": lambda t: t.group_by([\"carrier\", \"distance\"])}\n )\n)\n```\n\n\n\n**How it works:**\n- The `nest` parameter accepts a dictionary: `{\"column_name\": lambda t: ...}`\n- A bare `.group_by()` in the lambda specifies which row fields to collect\n- Results in an array of structs column named `flights`\n\nTo nest selected row-level fields, use the bare `group_by(...)` form. In a\n`nest` lambda this selects the struct fields to collect; it does not collapse\nduplicate source rows:\n\n```query_nest_select\n# Nest specific row-level fields\nresult = (\n flights_st\n .group_by(\"carrier\")\n .aggregate(\n \"flight_count\",\n nest={\"routes\": lambda t: t.group_by(\"origin\", \"distance\", \"duration\")}\n )\n)\n```\n\n\n\nAfter nesting, you can re-group which automatically unnests, then access the nested fields.\n\n**Step 1: Create nested data**\n\nFirst, create the nested structure. Notice the `flights` column contains arrays of structs:\n\n```query_nest_step1\nfrom ibis import _\n\n# Create nested data structure\nresult = (\n flights_st\n .group_by(\"origin\")\n .aggregate(\n \"flight_count\",\n nest={\"flights\": lambda t: t.group_by([\"carrier\", \"distance\"])}\n )\n)\n```\n\n\n\n**Step 2: Re-group to unnest and access fields**\n\nNow re-group on the same dimension, which automatically unnests the array, allowing you to access the nested fields:\n\n```query_nest_step2\nfrom ibis import _\n\n# Re-grouping automatically unnests the 'flights' array\nresult = (\n result\n .group_by(\"origin\")\n .aggregate(\n total_flights=lambda t: t.flight_count.sum(),\n # Access unnested fields from the flights array\n unique_carriers=lambda t: t.flights.carrier.nunique(),\n avg_distance=lambda t: t.flights.distance.mean()\n )\n)\n```\n\n\n\n**Use cases for nesting:**\n- **API responses**: Create JSON-compatible hierarchical structures\n- **Hierarchical data**: Preserve parent-child relationships in results\n- **Data export**: Generate nested documents for external systems\n- **Drill-down analysis**: Keep detailed records available in aggregated views\n\n\nFor more complex nesting patterns and multi-level hierarchies, see [Nested Subtotals](/advanced/nested-subtotals).\n\n\n## mutate()\n\nThe `mutate()` method transforms aggregated results by adding new computed columns. This is different from on-the-fly measures in `aggregate()` \u2014 `mutate()` works on already-aggregated data.\n\n\n**Key difference:** `.aggregate()` computes from raw data, while `.mutate()` transforms already-aggregated results.\n\n\n```query_mutate\nfrom ibis import _\n\n# Add post-aggregation calculations\nresult = (\n flights_st\n .group_by(\"origin\")\n .aggregate(\"flight_count\", \"total_distance\")\n .mutate(\n avg_distance_per_flight=lambda t: t.total_distance / t.flight_count,\n flight_category=lambda t: xo.case()\n .when(t.flight_count >= 3, \"high\")\n .when(t.flight_count >= 2, \"medium\")\n .else_(\"low\")\n .end()\n )\n)\n```\n\n\n\n**Use cases for `mutate()`:**\n- Calculate ratios from aggregated measures (e.g., `total / count`)\n- Create categories based on aggregated values\n- Add labels or formatting to results\n- Transform aggregated columns using the full power of Ibis\n\nFor more transformations, see [Ibis Table API reference](https://ibis-project.org/reference/expression-tables.html#ibis.expr.types.relations.Table.mutate).\n\n## Window Functions with .over()\n\nWindow functions perform calculations across ordered rows, enabling operations like running totals, moving averages, and ranking. Unlike regular aggregations that reduce many rows to one, window functions preserve row count while adding computed values.\n\n\n**Important:** Window functions can only be applied **after aggregation**, typically within a `.mutate()` call. They cannot be defined directly in measures.\n\n\n**Common window functions:**\n- **`lag()` / `lead()`**: Access previous/next row values for period-over-period comparisons\n- **`cumsum()`**: Calculate running totals\n- **`.over(window)`**: Apply functions over sliding windows (e.g., moving averages)\n- **`rank()` / `row_number()`**: Assign ranks or sequential numbers to rows\n\nHere's a simple example:\n\n```query_window_example\nfrom ibis import _\n\n# First aggregate to origin level\ndaily_flights = (\n flights_st\n .group_by(\"origin\")\n .aggregate(\"flight_count\", \"total_distance\")\n)\n\n# Then apply window functions directly on the aggregate \u2014 the window\n# carries its own ordering, and the result stays a semantic query\nresult = daily_flights.mutate(\n cumulative_distance=lambda t: t.total_distance.sum().over(\n rows=(None, 0), order_by=\"origin\"\n ),\n flight_rank=lambda t: xo.rank().over(xo.window(order_by=xo.desc(t.flight_count)))\n).order_by(\"origin\").limit(10)\n```\n\n\n\n**Key points:**\n- Window functions are applied via `.mutate()` **directly on the aggregate** (before `.order_by()`/`.limit()` \u2014 after those the result is a plain table and `.mutate()` raises)\n- Give each window its own ordering via the keyword form of `.over()` (e.g. `rows=(None, 0), order_by=\"origin\"`)\n- For row math over a *filtered* result, drop to ibis explicitly with `.to_untagged().mutate(...)`\n\nFor comprehensive examples including lag/lead, moving averages, and ranking, see [Window Functions](/advanced/windowing).\n\n## as_table()\n\nAfter filtering or aggregating data, you may want to perform additional semantic operations. However, intermediate results don't always preserve the semantic table's dimensions and measures.\n\nThe Problem: Lost Semantic Information\n\nWhen you aggregate data, the result loses semantic metadata. The aggregated result is a `SemanticAggregate` expression, which doesn't have `.dimensions` or `.measures` attributes:\n\n```query_as_table_problem\nfrom ibis import _\n\n# Aggregate the data - this returns a SemanticAggregate\nagg_result = flights_st.group_by(\"origin\").aggregate(\"flight_count\", \"total_distance\")\n\n# Show the type/class of the result\nresult_type = type(agg_result).__name__\n\n# Try to access .dimensions - this will raise an AttributeError\ntry:\n dimensions = agg_result.dimensions\n result = f\"Type: {result_type}\\nDimensions: {dimensions}\"\nexcept AttributeError as e:\n result = f\"Type: {result_type}\\nError: {str(e)}\"\n\nresult\n```\n\n\n\n\nAfter aggregation, you can no longer access the original semantic table's dimensions and measures metadata.\n\nThe Solution: Use as_table()\n\nThe `as_table()` method converts results back into a `SemanticModel`. However, note that for aggregations, the metadata is intentionally cleared (since columns are now materialized):\n\n```query_as_table_after_aggregate\nfrom ibis import _\n\n# Aggregate the data\nagg_result = flights_st.group_by(\"origin\").aggregate(\"flight_count\", \"total_distance\")\n\n# Convert to SemanticModel using as_table()\nagg_table = agg_result.as_table()\n\n# Now .dimensions and .measures attributes exist, but they're empty (metadata was cleared)\nresult = f\"Type: {type(agg_table).__name__}\\nDimensions: {agg_table.dimensions}\\nMeasures: {agg_table.measures}\"\n```\n\n\n\nWhen are metadata preserved ?\n\nFor operations like `filter()`, `order_by()`, and `limit()`, `as_table()` **preserves** the original semantic metadata:\n\n```query_as_table_filter_preserved\nfrom ibis import _\n\n# Filter the data\nfiltered = flights_st.filter(_.distance > 2000)\n\n# Convert back to SemanticModel - metadata is preserved!\nfiltered_table = filtered.as_table()\n\n# Dimensions and measures are still available (preserved from original semantic table)\nresult = f\"Type: {type(filtered_table).__name__}\\nDimensions: {filtered_table.dimensions}\\nMeasures: {filtered_table.measures}\"\n```\n\n\n\nNotice how the dimensions and measures are preserved, unlike the aggregation case above where they were empty.\n\n**Key points:**\n- **Operations that preserve metadata**: `filter()`, `order_by()`, `limit()`, `unnest()` \u2014 calling `as_table()` restores full semantic capabilities with original dimensions/measures\n- **Operations that clear metadata**: `aggregate()`, `mutate()` \u2014 calling `as_table()` returns a `SemanticModel` with empty dimensions/measures (columns are materialized)\n- Use `as_table()` when you need to continue semantic operations on intermediate results\n\n## Next Steps\n\n- Learn about [Building Semantic Tables](/building/semantic-tables) to define dimensions and measures\n- Explore [Composing Models](/building/compose) for multi-table queries\n- Try [Advanced Patterns](/advanced/percentage-total) for complex analytics\n", + "markdown": "# Query Methods\n\n## Overview\n\nBSL provides a simple and consistent query API for retrieving data from your semantic tables. Queries are built by chaining methods, then executed or inspected using output methods.\n\nStart with a semantic table and chain methods together. Here's the typical query flow:\n\n```setup_table\nimport ibis\nfrom ibis import _\nfrom boring_semantic_layer import to_semantic_table\n\n# Create Ibis table\nflights_tbl = ibis.memtable({\n \"origin\": [\"NYC\", \"LAX\", \"NYC\", \"SFO\", \"LAX\", \"NYC\", \"SFO\", \"LAX\", \"NYC\"],\n \"carrier\": [\"AA\", \"UA\", \"AA\", \"UA\", \"AA\", \"UA\", \"AA\", \"UA\", \"AA\"],\n \"distance\": [2789, 2789, 2902, 2902, 347, 2789, 347, 347, 2789],\n \"duration\": [330, 330, 360, 360, 65, 330, 65, 65, 330],\n})\n\n# Create semantic table\nflights_st = (\n to_semantic_table(flights_tbl, name=\"flights\")\n .with_dimensions(\n origin=lambda t: t.origin,\n carrier=lambda t: t.carrier,\n )\n .with_measures(\n flight_count=lambda t: t.count(),\n total_distance=lambda t: t.distance.sum(),\n avg_duration=lambda t: t.duration.mean(),\n )\n)\n```\n\n\n\n```python\nresult = (\n flights_st # Start with semantic table\n .filter(_.distance > 1000) # 1. Filter (optional)\n .group_by(\"origin\") # 2. Group by dimensions\n .aggregate(\"flight_count\", \"total_distance\") # 3. Aggregate measures\n .mutate(avg=lambda t: t.total_distance / t.flight_count) # 4. Transform (optional)\n .order_by(ibis.desc(\"flight_count\")) # 5. Sort (optional)\n .limit(10) # 6. Limit rows (optional)\n)\n```\n\nOnce you've built a query, you can inspect it or execute it:\n\n```simple_demo\n\n# Build a query\nresult = flights_st.group_by(\"origin\").aggregate(\"flight_count\")\n\n# Option 1: Execute and get data as pandas DataFrame\ndf = result.execute()\n\n# Option 2: View the generated SQL\nprint(result.sql())\n\n# Option 3: Generate a visualization (when applicable)\nchart = result.chart()\n\n# Option 4: See the semantic query plan\nprint(result)\n\nresult\n```\n\n\nThe output above includes a **Query Plan** tab showing how BSL translates this query into semantic operations. \n\nYou print the query object directly to see the plan:\n```python\nprint(result)\n```\n\nThis displays operations like `SemanticTableOp`, `SemanticGroupByOp`, and `SemanticAggregateOp`, useful for debugging and understanding query execution.\n\nLet's get now into the details of each query method.\n\n## group_by()\n\nThe `group_by()` method groups data by one or more dimensions.\n\n\n`group_by()` only accepts string dimension names that were previously defined in `with_dimensions()`. It does not support lambda functions or unbound `_` syntax.\n\n\n### Single Dimension\n\nGroup by a single dimension:\n\n```query_single_dimension\n# Group by one dimension\nresult = flights_st.group_by(\"origin\").aggregate(\"flight_count\")\n```\n\n\n\n### Multiple Dimensions\n\nGroup by multiple dimensions to create detailed breakdowns:\n\n```query_multiple_dimensions\n# Group by multiple dimensions\nresult = flights_st.group_by(\"origin\", \"carrier\").aggregate(\"flight_count\")\n```\n\n\n\n### No Grouping\n\nCalculate overall statistics across all rows using `group_by()` with no arguments:\n\n```query_no_grouping\n# Aggregate entire dataset without grouping\nresult = flights_st.group_by().aggregate(\"flight_count\", \"total_distance\", \"avg_duration\")\n```\n\n\n\n## aggregate()\n\nThe `aggregate()` method calculates measures after grouping. You can reference pre-defined measures or compute new ones on-the-fly.\n\n\n**CRITICAL**: `aggregate()` takes **measure names as strings**, not expressions or lambdas directly. Use measure names from `get_model()` output.\n\n```python\n# \u2705 CORRECT - measure names as strings\nmodel.group_by(\"category\").aggregate(\"flight_count\", \"total_revenue\")\n\n# \u274c WRONG - no standalone lambdas in aggregate\nmodel.aggregate(total=lambda t: t.sum()) # ERROR!\n```\n\n\n### Pre-defined Measures\n\nReference measures by their string names:\n\n```query_predefined_measures\n# Use measures defined in with_measures()\nresult = flights_st.group_by(\"origin\").aggregate(\"flight_count\", \"avg_duration\")\n```\n\n\n\n### On-the-Fly Transformations\n\nAdd computed measures directly in `aggregate()` without modifying the semantic table:\n\n```query_onthefly_measures\n# Mix predefined and computed measures\nresult = (\n flights_st\n .group_by(\"origin\")\n .aggregate(\n \"flight_count\", # Pre-defined measure\n \"avg_duration\", # Pre-defined measure\n total_miles=lambda t: t.distance.sum(), # Computed on-the-fly\n max_distance=lambda t: t.flight_count + 2 # You can reference other measures as well\n )\n)\n```\n\n\n\n\nOn-the-fly measures let you add context-specific calculations without modifying your semantic table definition. This keeps your base model clean while enabling flexible queries.\n\n\n### Referencing Table Columns\n\nYou can reference **any column from the underlying table** in `aggregate()`, not just pre-defined measures. This is useful when you need one-off calculations without cluttering your semantic table definition.\n\n```query_table_columns\n# Reference table columns directly in aggregate()\nresult = (\n flights_st\n .group_by(\"origin\")\n .aggregate(\n \"flight_count\", # Pre-defined measure\n total_distance=lambda t: t.distance.sum(), # Table column 'distance'\n avg_duration=lambda t: t.duration.mean(), # Table column 'duration'\n distance_in_km=lambda t: (t.distance * 1.60934).sum() # Transform then aggregate\n )\n)\n```\n\n\n\n**Key points:**\n- Table columns **must be aggregated** (e.g., `.sum()`, `.mean()`, `.max()`, `.count()`)\n- You can transform columns before aggregating (e.g., `(t.distance * 1.60934).sum()`)\n- This works for any column in the underlying table, even if not defined as a dimension or measure\n- Use this for ad-hoc calculations without modifying your semantic table\n\n\nTable columns cannot be used without an aggregation function. For example, `lambda t: t.distance` will fail. You must use `lambda t: t.distance.sum()` or another aggregation.\n\n\n## filter() / order_by() / limit() \n\nCombine `filter()`, `order_by()`, and `limit()` to refine your query results.\n\n```query_filter_order_limit\nfrom ibis import _\n\n# Filter data, sort, and limit results\nresult = (\n flights_st\n .filter(lambda t: t.origin.isin([\"NYC\", \"LAX\"])) # Filter origins\n .filter(_.distance > 500) # Filter distance using _ syntax\n .group_by(\"origin\")\n .aggregate(\"flight_count\", \"avg_duration\") # Aggregate both measures\n .order_by(ibis.desc(\"flight_count\")) # Sort by flight_count descending\n .limit(5) # Top 5 results\n)\n```\n\n\n\n**Key points:**\n- **`filter()`**: Use lambda or `_` syntax to apply conditions before aggregation\n- **`order_by()`**: Use `ibis.desc()` for descending order, or column name for ascending\n- **`limit()`**: Restrict the number of rows returned\n\n### Critical Filter Patterns\n\n**Multiple conditions** - use `ibis.and_()` or `ibis.or_()`:\n\n```python\n# Multiple conditions with AND\nmodel.filter(lambda t: ibis.and_(t.amount > 1000, t.year >= 2023))\n\n# Multiple conditions with OR\nmodel.filter(lambda t: ibis.or_(t.status == \"active\", t.status == \"pending\"))\n```\n\n**IN operator** - MUST use `.isin()` method:\n\n```python\n# \u2705 CORRECT - use .isin() method\nmodel.filter(lambda t: t.region.isin([\"US\", \"EU\", \"APAC\"]))\n\n# \u274c WRONG - Python's 'in' does NOT work!\nmodel.filter(lambda t: t.region in [\"US\", \"EU\"]) # ERROR: truth value of Ibis expression is not defined\n```\n\n**Lambda column names** - use column names directly, never prefix with model name:\n\n```python\n# \u2705 CORRECT - use column name directly\nmodel.filter(lambda t: t.carrier == \"AA\")\n\n# \u274c WRONG - do NOT prefix with model name\nmodel.filter(lambda t: t.model.carrier == \"AA\") # ERROR!\n```\n\n**Joined columns** - use exact prefixed name from `get_model()`:\n\n```python\n# If get_model() shows \"customers.country\", use it exactly:\nmodel.filter(lambda t: t.customers.country == \"US\")\n\n# \u274c WRONG - don't call methods on ID columns\nmodel.filter(lambda t: t.customer_id.country()) # ERROR: no such method!\n```\n\n## nest()\n\nThe `nest` parameter in `aggregate()` creates nested data structures (arrays of structs) in your query results. This is useful for API responses, hierarchical visualizations, and preserving relationships in aggregated data.\n\nUse `nest` to collect rows as structured arrays within each group:\n\n```query_basic_nest\nfrom ibis import _\n\n# Nest flight details within each origin\nresult = (\n flights_st\n .group_by(\"origin\")\n .aggregate(\n \"flight_count\",\n \"total_distance\",\n # Create nested array of flight details\n nest={\"flights\": lambda t: t.group_by([\"carrier\", \"distance\"])}\n )\n)\n```\n\n\n\n**How it works:**\n- The `nest` parameter accepts a dictionary: `{\"column_name\": lambda t: ...}`\n- A bare `.group_by()` in the lambda specifies which row fields to collect\n- Results in an array of structs column named `flights`\n\nTo nest selected row-level fields, use the bare `group_by(...)` form. In a\n`nest` lambda this selects the struct fields to collect; it does not collapse\nduplicate source rows:\n\n```query_nest_select\n# Nest specific row-level fields\nresult = (\n flights_st\n .group_by(\"carrier\")\n .aggregate(\n \"flight_count\",\n nest={\"routes\": lambda t: t.group_by(\"origin\", \"distance\", \"duration\")}\n )\n)\n```\n\n\n\nAfter nesting, you can re-group which automatically unnests, then access the nested fields.\n\n**Step 1: Create nested data**\n\nFirst, create the nested structure. Notice the `flights` column contains arrays of structs:\n\n```query_nest_step1\nfrom ibis import _\n\n# Create nested data structure\nresult = (\n flights_st\n .group_by(\"origin\")\n .aggregate(\n \"flight_count\",\n nest={\"flights\": lambda t: t.group_by([\"carrier\", \"distance\"])}\n )\n)\n```\n\n\n\n**Step 2: Re-group to unnest and access fields**\n\nNow re-group on the same dimension, which automatically unnests the array, allowing you to access the nested fields:\n\n```query_nest_step2\nfrom ibis import _\n\n# Re-grouping automatically unnests the 'flights' array\nresult = (\n result\n .group_by(\"origin\")\n .aggregate(\n total_flights=lambda t: t.flight_count.sum(),\n # Access unnested fields from the flights array\n unique_carriers=lambda t: t.flights.carrier.nunique(),\n avg_distance=lambda t: t.flights.distance.mean()\n )\n)\n```\n\n\n\n**Use cases for nesting:**\n- **API responses**: Create JSON-compatible hierarchical structures\n- **Hierarchical data**: Preserve parent-child relationships in results\n- **Data export**: Generate nested documents for external systems\n- **Drill-down analysis**: Keep detailed records available in aggregated views\n\n\nFor more complex nesting patterns and multi-level hierarchies, see [Nested Subtotals](/advanced/nested-subtotals).\n\n\n## mutate()\n\nThe `mutate()` method transforms aggregated results by adding new computed columns. This is different from on-the-fly measures in `aggregate()` \u2014 `mutate()` works on already-aggregated data.\n\n\n**Key difference:** `.aggregate()` computes from raw data, while `.mutate()` transforms already-aggregated results.\n\n\n```query_mutate\nfrom ibis import _\n\n# Add post-aggregation calculations\nresult = (\n flights_st\n .group_by(\"origin\")\n .aggregate(\"flight_count\", \"total_distance\")\n .mutate(\n avg_distance_per_flight=lambda t: t.total_distance / t.flight_count,\n flight_category=lambda t: xo.case()\n .when(t.flight_count >= 3, \"high\")\n .when(t.flight_count >= 2, \"medium\")\n .else_(\"low\")\n .end()\n )\n)\n```\n\n\n\n**Use cases for `mutate()`:**\n- Calculate ratios from aggregated measures (e.g., `total / count`)\n- Create categories based on aggregated values\n- Add labels or formatting to results\n- Transform aggregated columns using the full power of Ibis\n\nFor more transformations, see [Ibis Table API reference](https://ibis-project.org/reference/expression-tables.html#ibis.expr.types.relations.Table.mutate).\n\n## Window Functions with .over()\n\nWindow functions perform calculations across ordered rows, enabling operations like running totals, moving averages, and ranking. Unlike regular aggregations that reduce many rows to one, window functions preserve row count while adding computed values.\n\n\n**Important:** Window functions can only be applied **after aggregation**, typically within a `.mutate()` call. They cannot be defined directly in measures.\n\n\n**Common window functions:**\n- **`lag()` / `lead()`**: Access previous/next row values for period-over-period comparisons\n- **`cumsum()`**: Calculate running totals\n- **`.over(window)`**: Apply functions over sliding windows (e.g., moving averages)\n- **`rank()` / `row_number()`**: Assign ranks or sequential numbers to rows\n\nHere's a simple example:\n\n```query_window_example\nfrom ibis import _\n\n# First aggregate to origin level\ndaily_flights = (\n flights_st\n .group_by(\"origin\")\n .aggregate(\"flight_count\", \"total_distance\")\n)\n\n# Then apply window functions directly on the aggregate \u2014 the window\n# carries its own ordering, and the result stays a semantic query\nresult = daily_flights.mutate(\n cumulative_distance=lambda t: t.total_distance.sum().over(\n rows=(None, 0), order_by=\"origin\"\n ),\n flight_rank=lambda t: xo.rank().over(xo.window(order_by=xo.desc(t.flight_count)))\n).order_by(\"origin\").limit(10)\n```\n\n\n\n**Key points:**\n- Window functions are applied via `.mutate()` **directly on the aggregate** (before `.order_by()`/`.limit()` \u2014 after those the result is a plain table and `.mutate()` raises)\n- Give each window its own ordering via the keyword form of `.over()` (e.g. `rows=(None, 0), order_by=\"origin\"`)\n- For row math over a *filtered* result, drop to ibis explicitly with `.to_untagged().mutate(...)`\n\nFor comprehensive examples including lag/lead, moving averages, and ranking, see [Window Functions](/advanced/windowing).\n\n## as_table()\n\nAfter filtering or aggregating data, you may want to perform additional semantic operations. However, intermediate results don't always preserve the semantic table's dimensions and measures.\n\nThe Problem: Lost Semantic Information\n\nWhen you aggregate data, the result loses semantic metadata. The aggregated result is a `SemanticAggregate` expression, which doesn't have `.dimensions` or `.measures` attributes:\n\n```query_as_table_problem\nfrom ibis import _\n\n# Aggregate the data - this returns a SemanticAggregate\nagg_result = flights_st.group_by(\"origin\").aggregate(\"flight_count\", \"total_distance\")\n\n# Show the type/class of the result\nresult_type = type(agg_result).__name__\n\n# Try to access .dimensions - this will raise an AttributeError\ntry:\n dimensions = agg_result.dimensions\n result = f\"Type: {result_type}\\nDimensions: {dimensions}\"\nexcept AttributeError as e:\n result = f\"Type: {result_type}\\nError: {str(e)}\"\n\nresult\n```\n\n\n\n\nAfter aggregation, you can no longer access the original semantic table's dimensions and measures metadata.\n\nThe Solution: Use as_table()\n\nThe `as_table()` method converts results back into a `SemanticModel`. However, note that for aggregations, the metadata is intentionally cleared (since columns are now materialized):\n\n```query_as_table_after_aggregate\nfrom ibis import _\n\n# Aggregate the data\nagg_result = flights_st.group_by(\"origin\").aggregate(\"flight_count\", \"total_distance\")\n\n# Convert to SemanticModel using as_table()\nagg_table = agg_result.as_table()\n\n# Now .dimensions and .measures attributes exist, but they're empty (metadata was cleared)\nresult = f\"Type: {type(agg_table).__name__}\\nDimensions: {agg_table.dimensions}\\nMeasures: {agg_table.measures}\"\n```\n\n\n\nWhen are metadata preserved ?\n\nFor operations like `filter()`, `order_by()`, and `limit()`, `as_table()` **preserves** the original semantic metadata:\n\n```query_as_table_filter_preserved\nfrom ibis import _\n\n# Filter the data\nfiltered = flights_st.filter(_.distance > 2000)\n\n# Convert back to SemanticModel - metadata is preserved!\nfiltered_table = filtered.as_table()\n\n# Dimensions and measures are still available (preserved from original semantic table)\nresult = f\"Type: {type(filtered_table).__name__}\\nDimensions: {filtered_table.dimensions}\\nMeasures: {filtered_table.measures}\"\n```\n\n\n\nNotice how the dimensions and measures are preserved, unlike the aggregation case above where they were empty.\n\n**Key points:**\n- **Operations that preserve metadata**: `filter()`, `order_by()`, `limit()`, `unnest()` \u2014 calling `as_table()` restores full semantic capabilities with original dimensions/measures\n- **Operations that clear metadata**: `aggregate()`, `mutate()` \u2014 calling `as_table()` returns a `SemanticModel` with empty dimensions/measures (columns are materialized)\n- Use `as_table()` when you need to continue semantic operations on intermediate results\n\n## Next Steps\n\n- Learn about [Building Semantic Tables](/building/semantic-tables) to define dimensions and measures\n- Explore [Composing Models](/building/compose) for multi-table queries\n- Try [Advanced Patterns](/advanced/percentage-total) for complex analytics\n", "queries": { "setup_table": { "semantic_table": true, diff --git a/docs/web/public/bsl-data/semantic-table.json b/docs/web/public/bsl-data/semantic-table.json index 65eea0f7..e2c4669e 100644 --- a/docs/web/public/bsl-data/semantic-table.json +++ b/docs/web/public/bsl-data/semantic-table.json @@ -1,5 +1,5 @@ { - "markdown": "# Building a Semantic Table\n\nDefine your data model with dimensions and measures using Ibis expressions.\n\n## Overview\n\nA Semantic Table is the core building block of BSL. It transforms a raw Ibis table into a reusable, self-documenting data model by defining:\n- **Dimensions**: Attributes to group by (e.g., origin, carrier, year)\n- **Measures**: Aggregations and calculations (e.g., flight count, total distance)\n\n## to_semantic_table()\n\n```setup_flights\nimport ibis\nfrom boring_semantic_layer import to_semantic_table\n\n# 1. Start with an Ibis table\ncon = ibis.duckdb.connect(\":memory:\")\nflights_data = ibis.memtable({\n \"origin\": [\"JFK\", \"LAX\", \"SFO\"],\n \"dest\": [\"LAX\", \"SFO\", \"JFK\"],\n \"carrier\": [\"AA\", \"UA\", \"DL\"],\n \"year\": [2023, 2023, 2024],\n \"distance\": [2475, 337, 382],\n \"dep_delay\": [10, 5, 0]\n})\nflights_tbl = con.create_table(\"flights\", flights_data)\n\n# 2. Convert to a Semantic Table\nflights_st = to_semantic_table(flights_tbl, name=\"flights\")\n```\n\n## with_dimensions()\n\nDimensions define the attributes you can group by in your queries. They represent the categorical or descriptive aspects of your data that you want to analyze.\n\nYou can define dimensions using lambda expressions, unbound syntax (`_.`), or the `Dimension` class with descriptions:\n\n```dimensions_demo\nfrom ibis import _\nfrom boring_semantic_layer import Dimension\n\nflights_st = flights_st.with_dimensions(\n # Lambda expressions - simple and explicit\n origin=lambda t: t.origin,\n\n # Unbound syntax - cleaner and more concise\n destination=_.dest,\n year=_.year,\n\n # Dimension - self-documenting and AI-friendly\n carrier=Dimension(\n expr=lambda t: t.carrier,\n description=\"Airline carrier code\"\n )\n)\n\nflights_st.dimensions\n```\n\n\n## with_measures()\n\nMeasures define the aggregations and calculations you can query. They represent the quantitative aspects of your data that you want to analyze (counts, sums, averages, etc.).\n\nYou can define measures using lambda expressions, reference other measures for composition, or use the `Measure` class with descriptions:\n\n```measures_demo\nfrom boring_semantic_layer import Measure\n\nflights_st = flights_st.with_measures(\n # Lambda expressions - simple and concise\n total_flights=lambda t: t.count(),\n total_distance=lambda t: t.distance.sum(),\n max_delay=lambda t: t.dep_delay.max(),\n\n # Reference other measures for composition\n avg_distance_per_flight=lambda t: t.total_distance / t.total_flights,\n\n # Measure - self-documenting and AI-friendly\n avg_distance=Measure(\n expr=lambda t: t.distance.mean(),\n description=\"Average flight distance in miles\"\n )\n)\n\nflights_st.measures\n```\n\n\n\n### all()\n\nThe `all()` function references the entire dataset within measure definitions, enabling percent-of-total and comparison calculations.\n\n**Example:** Calculate market share as a percentage\n\n```measure_all_demo\nflights_with_pct = flights_st.with_measures(\n flight_count=lambda t: t.count(),\n market_share=lambda t: t.flight_count / t.all(t.flight_count) * 100 # Percent of total\n )\n\n# Query by carrier\nresult = (\n flights_with_pct\n .group_by(\"carrier\")\n .aggregate(\"flight_count\", \"market_share\")\n)\n```\n\n\n\n\n`t.all(ref)` is available on the table parameter `t` in measure definitions. It\nevaluates the supplied measure or reduction over the entire dataset regardless\nof grouping, making it useful for percentages and comparisons with the total.\n\n\nFor more examples, see the [Percent of Total pattern](/advanced/percentage-total).\n\n## graph\n\nThe `graph` property provides a dependency graph showing how dimensions and measures relate to each other. This is useful for:\n- **Understanding dependencies**: See what columns or fields each dimension/measure depends on\n- **Impact analysis**: Find what breaks when changing a field\n- **Documentation**: Generate visual representations of your data model\n- **Validation**: Ensure your model doesn't have circular dependencies\n\n```graph_demo\n# Build a semantic table with dependencies\nflights_with_deps = flights_st.with_dimensions(\n origin=lambda t: t.origin,\n destination=lambda t: t.dest,\n).with_measures(\n flight_count=lambda t: t.count(),\n total_distance=lambda t: t.distance.sum(),\n avg_distance_per_flight=lambda t: t.total_distance / t.flight_count\n)\n\n# Access the dependency graph\ngraph = flights_with_deps.get_graph()\ngraph\n```\n\n\n### Understanding the Graph Structure\n\nThe graph is a dictionary where:\n- **Keys**: Dimension or measure names\n- **Values**: Metadata containing:\n - `deps`: Dependencies mapped to their types (`'column'`, `'dimension'`, or `'measure'`)\n - `type`: The field type (`'dimension'`, `'measure'`, or `'calc_measure'`)\n\n```graph_structure\n# Access the graph - it's a dict-like object\ngraph = flights_with_deps.get_graph()\ngraph\n```\n\n\n```python\n# Find what a specific field depends on\nflights_with_deps.get_graph()['avg_distance_per_flight']['deps']\n# Output: {'total_distance': 'measure', 'flight_count': 'measure'}\n```\n\n### Graph Traversal\n\nUse `graph_predecessors()` and `graph_successors()` to navigate dependencies:\n\n```graph_traversal\nfrom boring_semantic_layer.graph_utils import graph_predecessors, graph_successors\n\ngraph = flights_with_deps.get_graph()\n\n# What does this field depend on? (predecessors)\ngraph_predecessors(graph, 'avg_distance_per_flight')\n# {'total_distance', 'flight_count'}\n\n# What depends on this field? (successors)\ngraph_successors(graph, 'total_distance')\n# {'avg_distance_per_flight'}\n```\n\n\n### Working with the Dependency Graph\n\nThe dependency graph is a dict-like object where each key is a field name and the value is a dict with `\"type\"` (dimension/measure/calc_measure/column) and `\"deps\"` (dependencies with their types):\n\n```python\n# Access the graph directly as a dict\ngraph = flights_with_deps.get_graph()\n\n# Iterate over fields and their dependencies\nfor field, info in graph.items():\n print(f\"{field} ({info['type']}): depends on {info['deps']}\")\n```\n\n## join_one() / join_many() / join_cross()\n\nJoin semantic tables together to query across relationships. Joins allow you to combine data from multiple semantic tables and access dimensions and measures across all joined tables.\n\n**What Makes Semantic Joins Different?**\n\nSemantic joins explicitly capture the **relationship type** between tables, rather than just specifying SQL join mechanics:\n\n**SQL Joins:**\n```python\n# Specifies HOW to join, but not the analytical relationship\nflights_tbl.left_join(carriers_tbl, flights_tbl.carrier == carriers_tbl.code)\n```\n\n**Semantic Joins:**\n```python\n# One carrier row can match many flight rows\ncarriers.join_many(flights_st, lambda c, f: c.code == f.carrier)\n```\n\n**What You Get:**\n- **Explicit relationships**: `join_many()` documents that this is a one-to-many relationship\n- **Table hierarchy information**: The method name describes how tables relate to each other\n- **Richer metadata**: Makes the data model structure explicit for documentation and tooling\n\n\nAfter joining, dimensions and measures are prefixed with table names (e.g., `flights.origin`, `carriers.name`) to avoid naming conflicts.\n\n\n\n**Give every source in a composed model a unique name.** BSL uses model names as\nsource aliases for dimensions, measures, and grain metadata, and rejects a join\ntree containing duplicate names. If you join the same underlying table more than\nonce (for example, pickup and dropoff locations), create distinct table references\nand assign explicit aliases:\n\n```python\n# Create distinct references when joining same table twice\npickup_locs = to_semantic_table(locs_tbl.view(), \"pickup_locs\")\ndropoff_locs = to_semantic_table(locs_tbl.view(), \"dropoff_locs\")\n```\n\nThe distinct names prevent ambiguous semantic prefixes; `.view()` prevents Ibis\nfrom treating both roles as the same relation.\n\n\n\n**Source-aware aggregation requires equality-key joins.** When BSL aggregates\nmeasures at their source grain before joining, each non-cross join predicate must\nbe a direct field equality or a conjunction of direct field equalities. String,\nDeferred, and compound equality-key shorthands are supported. Predicates using\ninequality, `OR`, casts, or transformed expressions are rejected because reducing\nthem to join-key bridges could change the matched row set. Aggregate the models\nfirst or restate the relationship with plain equality keys; use `join_cross()` for\na Cartesian product.\n\n\nLet's get some additional data:\n\n```setup_carriers\nimport ibis\nfrom boring_semantic_layer import to_semantic_table\n\ncon = ibis.duckdb.connect(\":memory:\")\n\n# Create carriers data\ncarriers_data = ibis.memtable({\n \"code\": [\"AA\", \"UA\", \"DL\"],\n \"name\": [\"American Airlines\", \"United Airlines\", \"Delta Air Lines\"]\n})\ncarriers_tbl = con.create_table(\"carriers\", carriers_data)\n```\n\n\nAnd create a carriers semantic table:\n\n```carriers_st\ncarriers = (\n to_semantic_table(carriers_tbl, name=\"carriers\")\n .with_dimensions(\n code=lambda t: t.code,\n name=lambda t: t.name\n )\n .with_measures(\n carrier_count=lambda t: t.count()\n )\n)\n```\n\n### join_many() - One-to-Many Relationships\n\nUse `join_many()` when one row in the left table can match multiple rows in the right table (LEFT JOIN).\n\n```join_demo\n# One carrier row can match many flight rows\ncarriers_with_flights = carriers.join_many(\n flights_st,\n lambda c, f: c.code == f.carrier\n)\n\n# Inspect available dimensions and measures\ncarriers_with_flights.dimensions\n```\n\n\nAfter joining, all dimensions and measures from both tables are available. Each is prefixed with its table name to avoid conflicts:\n\n\n### join_one() - At-Most-One Right Match\n\nUse `join_one()` when each row on the left can match at most one row on the\nright. Like all non-cross semantic joins, it uses a LEFT JOIN so unmatched left\nrows remain visible to measures.\n\n```python\n# Many flights \u2192 one carrier (each flight matches at most one carrier)\nflights_with_carrier = flights_st.join_one(\n carriers,\n lambda f, c: f.carrier == c.code\n)\n```\n\n\n**Join predicates resolve physical columns.** A string or Deferred shorthand\nnames the same underlying column on both sides. If the columns have different\nnames, use a two-argument lambda instead.\n\n**Example:**\n```python\n# If users table has column 'id' but dimension 'customer_id':\nusers = to_semantic_table(users_tbl, \"users\").with_dimensions(\n customer_id=lambda t: t.id # Dimension renamed\n)\n\n# Compare the underlying columns explicitly:\norders.join_one(users, on=lambda order, user: order.customer_id == user.id)\n```\n\n\n### join_cross() - Cross Join\n\nUse `join_cross()` to create every possible combination of rows from both tables (CARTESIAN PRODUCT).\n\n```python\n# Every flight \u00d7 every carrier combination\nall_combinations = flights_st.join_cross(carriers)\n```\n\n### Requiring a Match\n\n`join_one()` and `join_many()` are always LEFT joins (there is no `how=`\nparameter). When a query should\nrequire a match, make the row removal explicit with a filter on a non-nullable\nfield from the right table:\n\n```python\nflights_matched = flights_st.join_one(\n carriers,\n lambda f, c: f.carrier == c.code,\n).filter(lambda t: t[\"carriers.name\"].notnull())\n```\n\nUse `join_cross()` for Cartesian products.\n\n## Next Steps\n\n- Learn about [Composing Models](/building/compose)\n- Explore [YAML Configuration](/building/yaml)\n- Start [Querying Semantic Tables](/querying/methods)\n", + "markdown": "# Building a Semantic Table\n\nDefine your data model with dimensions and measures using Ibis expressions.\n\n## Overview\n\nA Semantic Table is the core building block of BSL. It transforms a raw Ibis table into a reusable, self-documenting data model by defining:\n- **Dimensions**: Attributes to group by (e.g., origin, carrier, year)\n- **Measures**: Aggregations and calculations (e.g., flight count, total distance)\n\n## to_semantic_table()\n\n```setup_flights\nimport ibis\nfrom boring_semantic_layer import to_semantic_table\n\n# 1. Start with an Ibis table\ncon = ibis.duckdb.connect(\":memory:\")\nflights_data = ibis.memtable({\n \"origin\": [\"JFK\", \"LAX\", \"SFO\"],\n \"dest\": [\"LAX\", \"SFO\", \"JFK\"],\n \"carrier\": [\"AA\", \"UA\", \"DL\"],\n \"year\": [2023, 2023, 2024],\n \"distance\": [2475, 337, 382],\n \"dep_delay\": [10, 5, 0]\n})\nflights_tbl = con.create_table(\"flights\", flights_data)\n\n# 2. Convert to a Semantic Table\nflights_st = to_semantic_table(flights_tbl, name=\"flights\")\n```\n\n## with_dimensions()\n\nDimensions define the attributes you can group by in your queries. They represent the categorical or descriptive aspects of your data that you want to analyze.\n\nYou can define dimensions using lambda expressions, unbound syntax (`_.`), or the `Dimension` class with descriptions:\n\n```dimensions_demo\nfrom ibis import _\nfrom boring_semantic_layer import Dimension\n\nflights_st = flights_st.with_dimensions(\n # Lambda expressions - simple and explicit\n origin=lambda t: t.origin,\n\n # Unbound syntax - cleaner and more concise\n destination=_.dest,\n year=_.year,\n\n # Dimension - self-documenting and AI-friendly\n carrier=Dimension(\n expr=lambda t: t.carrier,\n description=\"Airline carrier code\"\n )\n)\n\nflights_st.dimensions\n```\n\n\n## with_measures()\n\nMeasures define the aggregations and calculations you can query. They represent the quantitative aspects of your data that you want to analyze (counts, sums, averages, etc.).\n\nYou can define measures using lambda expressions, reference other measures for composition, or use the `Measure` class with descriptions:\n\n```measures_demo\nfrom boring_semantic_layer import Measure\n\nflights_st = flights_st.with_measures(\n # Lambda expressions - simple and concise\n total_flights=lambda t: t.count(),\n total_distance=lambda t: t.distance.sum(),\n max_delay=lambda t: t.dep_delay.max(),\n\n # Reference other measures for composition\n avg_distance_per_flight=lambda t: t.total_distance / t.total_flights,\n\n # Measure - self-documenting and AI-friendly\n avg_distance=Measure(\n expr=lambda t: t.distance.mean(),\n description=\"Average flight distance in miles\"\n )\n)\n\nflights_st.measures\n```\n\n\n\n### all()\n\nThe `all()` function references the entire dataset within measure definitions, enabling percent-of-total and comparison calculations.\n\n**Example:** Calculate market share as a percentage\n\n```measure_all_demo\nflights_with_pct = flights_st.with_measures(\n flight_count=lambda t: t.count(),\n market_share=lambda t: t.flight_count / t.all(t.flight_count) * 100 # Percent of total\n )\n\n# Query by carrier\nresult = (\n flights_with_pct\n .group_by(\"carrier\")\n .aggregate(\"flight_count\", \"market_share\")\n)\n```\n\n\n\n\n`t.all(ref)` is available on the table parameter `t` in measure definitions. It\nevaluates the supplied measure or reduction over the entire dataset regardless\nof grouping, making it useful for percentages and comparisons with the total.\n\n\nFor more examples, see the [Percent of Total pattern](/advanced/percentage-total).\n\n## graph\n\nThe `graph` property provides a dependency graph showing how dimensions and measures relate to each other. This is useful for:\n- **Understanding dependencies**: See what columns or fields each dimension/measure depends on\n- **Impact analysis**: Find what breaks when changing a field\n- **Documentation**: Generate visual representations of your data model\n- **Validation**: Ensure your model doesn't have circular dependencies\n\n```graph_demo\n# Build a semantic table with dependencies\nflights_with_deps = flights_st.with_dimensions(\n origin=lambda t: t.origin,\n destination=lambda t: t.dest,\n).with_measures(\n flight_count=lambda t: t.count(),\n total_distance=lambda t: t.distance.sum(),\n avg_distance_per_flight=lambda t: t.total_distance / t.flight_count\n)\n\n# Access the dependency graph\ngraph = flights_with_deps.get_graph()\ngraph\n```\n\n\n### Understanding the Graph Structure\n\nThe graph is a dictionary where:\n- **Keys**: Dimension or measure names\n- **Values**: Metadata containing:\n - `deps`: Dependencies mapped to their types (`'column'`, `'dimension'`, or `'measure'`)\n - `type`: The field type (`'dimension'`, `'measure'`, or `'calc_measure'`)\n\n```graph_structure\n# Access the graph - it's a dict-like object\ngraph = flights_with_deps.get_graph()\ngraph\n```\n\n\n```python\n# Find what a specific field depends on\nflights_with_deps.get_graph()['avg_distance_per_flight']['deps']\n# Output: {'total_distance': 'measure', 'flight_count': 'measure'}\n```\n\n### Graph Traversal\n\nUse `graph_predecessors()` and `graph_successors()` to navigate dependencies:\n\n```graph_traversal\nfrom boring_semantic_layer.graph_utils import graph_predecessors, graph_successors\n\ngraph = flights_with_deps.get_graph()\n\n# What does this field depend on? (predecessors)\ngraph_predecessors(graph, 'avg_distance_per_flight')\n# {'total_distance', 'flight_count'}\n\n# What depends on this field? (successors)\ngraph_successors(graph, 'total_distance')\n# {'avg_distance_per_flight'}\n```\n\n\n### Working with the Dependency Graph\n\nThe dependency graph is a dict-like object where each key is a field name and the value is a dict with `\"type\"` (dimension/measure/calc_measure/column) and `\"deps\"` (dependencies with their types):\n\n```python\n# Access the graph directly as a dict\ngraph = flights_with_deps.get_graph()\n\n# Iterate over fields and their dependencies\nfor field, info in graph.items():\n print(f\"{field} ({info['type']}): depends on {info['deps']}\")\n```\n\n## join_one() / join_many() / join_cross()\n\nJoin semantic tables together to query across relationships. Joins allow you to combine data from multiple semantic tables and access dimensions and measures across all joined tables.\n\n**What Makes Semantic Joins Different?**\n\nSemantic joins explicitly capture the **relationship type** between tables, rather than just specifying SQL join mechanics:\n\n**SQL Joins:**\n```python\n# Specifies HOW to join, but not the analytical relationship\nflights_tbl.left_join(carriers_tbl, flights_tbl.carrier == carriers_tbl.code)\n```\n\n**Semantic Joins:**\n```python\n# One carrier row can match many flight rows\ncarriers.join_many(flights_st, lambda c, f: c.code == f.carrier)\n```\n\n**What You Get:**\n- **Explicit relationships**: `join_many()` documents that this is a one-to-many relationship\n- **Table hierarchy information**: The method name describes how tables relate to each other\n- **Richer metadata**: Makes the data model structure explicit for documentation and tooling\n\n\nAfter joining, dimensions and measures are prefixed with table names (e.g., `flights.origin`, `carriers.name`) to avoid naming conflicts.\n\n\n\n**Give every source in a composed model a unique name.** BSL uses model names as\nsource aliases for dimensions, measures, and grain metadata, and rejects a join\ntree containing duplicate names. If you join the same underlying table more than\nonce (for example, pickup and dropoff locations), create distinct table references\nand assign explicit aliases:\n\n```python\n# Create distinct references when joining same table twice\npickup_locs = to_semantic_table(locs_tbl.view(), \"pickup_locs\")\ndropoff_locs = to_semantic_table(locs_tbl.view(), \"dropoff_locs\")\n```\n\nThe distinct names prevent ambiguous semantic prefixes; `.view()` prevents Ibis\nfrom treating both roles as the same relation.\n\n\n\n\n**Source-aware aggregation requires equality-key joins.** When BSL aggregates\nmeasures at their source grain before joining, each non-cross join predicate must\nbe a direct field equality or a conjunction of direct field equalities. String,\nDeferred, and compound equality-key shorthands are supported. Predicates using\ninequality, `OR`, casts, or transformed expressions are rejected because reducing\nthem to join-key bridges could change the matched row set. Aggregate the models\nfirst or restate the relationship with plain equality keys; use `join_cross()` for\na Cartesian product.\n\n\nLet's get some additional data:\n\n```setup_carriers\nimport ibis\nfrom boring_semantic_layer import to_semantic_table\n\ncon = ibis.duckdb.connect(\":memory:\")\n\n# Create carriers data\ncarriers_data = ibis.memtable({\n \"code\": [\"AA\", \"UA\", \"DL\"],\n \"name\": [\"American Airlines\", \"United Airlines\", \"Delta Air Lines\"]\n})\ncarriers_tbl = con.create_table(\"carriers\", carriers_data)\n```\n\n\nAnd create a carriers semantic table:\n\n```carriers_st\ncarriers = (\n to_semantic_table(carriers_tbl, name=\"carriers\")\n .with_dimensions(\n code=lambda t: t.code,\n name=lambda t: t.name\n )\n .with_measures(\n carrier_count=lambda t: t.count()\n )\n)\n```\n\n### join_many() - One-to-Many Relationships\n\nUse `join_many()` when one row in the left table can match multiple rows in the right table (LEFT JOIN).\n\n```join_demo\n# One carrier row can match many flight rows\ncarriers_with_flights = carriers.join_many(\n flights_st,\n lambda c, f: c.code == f.carrier\n)\n\n# Inspect available dimensions and measures\ncarriers_with_flights.dimensions\n```\n\n\nAfter joining, all dimensions and measures from both tables are available. Each is prefixed with its table name to avoid conflicts:\n\n\n### join_one() - At-Most-One Right Match\n\nUse `join_one()` when each row on the left can match at most one row on the\nright. Like all non-cross semantic joins, it uses a LEFT JOIN so unmatched left\nrows remain visible to measures.\n\n```python\n# Many flights \u2192 one carrier (each flight matches at most one carrier)\nflights_with_carrier = flights_st.join_one(\n carriers,\n lambda f, c: f.carrier == c.code\n)\n```\n\n\n**Join predicates resolve physical columns.** A string or Deferred shorthand\nnames the same underlying column on both sides. If the columns have different\nnames, use a two-argument lambda instead.\n\n**Example:**\n```python\n# If users table has column 'id' but dimension 'customer_id':\nusers = to_semantic_table(users_tbl, \"users\").with_dimensions(\n customer_id=lambda t: t.id # Dimension renamed\n)\n\n# Compare the underlying columns explicitly:\norders.join_one(users, on=lambda order, user: order.customer_id == user.id)\n```\n\n\n### join_cross() - Cross Join\n\nUse `join_cross()` to create every possible combination of rows from both tables (CARTESIAN PRODUCT).\n\n```python\n# Every flight \u00d7 every carrier combination\nall_combinations = flights_st.join_cross(carriers)\n```\n\n### Requiring a Match\n\n`join_one()` and `join_many()` are always LEFT joins (there is no `how=`\nparameter). When a query should\nrequire a match, make the row removal explicit with a filter on a non-nullable\nfield from the right table:\n\n```python\nflights_matched = flights_st.join_one(\n carriers,\n lambda f, c: f.carrier == c.code,\n).filter(lambda t: t[\"carriers.name\"].notnull())\n```\n\nUse `join_cross()` for Cartesian products.\n\n## Next Steps\n\n- Learn about [Composing Models](/building/compose)\n- Explore [YAML Configuration](/building/yaml)\n- Start [Querying Semantic Tables](/querying/methods)\n", "queries": { "dimensions_demo": { "output": "('origin', 'destination', 'year', 'carrier')" diff --git a/docs/web/src/components/Note.tsx b/docs/web/src/components/Note.tsx index 36dcce3d..c655424a 100644 --- a/docs/web/src/components/Note.tsx +++ b/docs/web/src/components/Note.tsx @@ -28,28 +28,32 @@ export function Note({ children, type = 'info' }: NoteProps) { const style = styles[type] - // Extract text content if children is a string or can be converted - const content = typeof children === 'string' - ? children - : React.Children.toArray(children).map(child => - typeof child === 'string' ? child : '' - ).join('') + // Text up to the first blank line arrives as a raw string; markdown after + // it (code blocks, extra paragraphs) arrives already rendered by the page. + const parts = React.Children.toArray(children).map((child, i) => + typeof child === 'string' ? ( + <>{children}, + strong: ({ children }) => {children}, + code: ({ children }) => {children}, + a: ({ href, children }) => {children} + }} + > + {child} + + ) : ( + child + ) + ) return (
-
- <>{children}, - strong: ({ children }) => {children}, - code: ({ children }) => {children}, - a: ({ href, children }) => {children} - }} - > - {content} - +
+ {parts}