Add the catalog-driven Spark UDF generator (foundation)#27
Open
estebanzimanyi wants to merge 2 commits into
Open
Add the catalog-driven Spark UDF generator (foundation)#27estebanzimanyi wants to merge 2 commits into
estebanzimanyi wants to merge 2 commits into
Conversation
estebanzimanyi
force-pushed
the
feat/spark-udf-generator
branch
from
9bb3956 to
c5dd081
Compare
tools/codegen_spark_udfs.py emits MobilitySpark UDF-registration classes from the MEOS-API catalog (output/meos-idl.json), resolving each SQL name to its MEOS-C backing via the @sqlfn / @sqlop map (MEOS-API MobilityDB#18). Two modes: - SINGLE: one backing -> a 1:1 UDF (type-marshalling: each MEOS C type <-> its parse-from-String / serialize-to-String form). - DISPATCH: an overloaded SQL name / operator (overlaps via &&, stbox(geom,time), timeSpan) -> ONE UDF that classifies each arg by its MEOS type and routes to the catalog-determined backing. Classification is MEOS-driven and wire-format-safe: spans/stboxes/geometries travel as TEXT, only temporals as hex, so the leading token disambiguates ('['/'(' span, STBOX stbox, hex temporal, else geometry) and temporal_from_hexwkb is never fed a non-temporal. Emitted lambdas call only static GeneratedFunctions (no captured state -> Spark-serializable). Zero hand heuristics, zero new MEOS functions.
estebanzimanyi
force-pushed
the
feat/spark-udf-generator
branch
from
c5dd081 to
b939673
Compare
…roup Generalize the generator over the whole JMEOS public surface (was a 4-UDF POC): mirror JMEOS FunctionsGenerator's marshalling conventions — temporals / spans / sets / boxes / jsonb as hex-WKB or type text, TimestampTz as OffsetDateTime, DateADT as int, and bool f(.., result) out-params dropped with their value returned. Cross-check every emission against the JMEOS jar signatures (arity + return kind) so a collapsed catalog type can never miscompile. Organize the emitted UDFs into one class per doxygen @InGroup module — the reference-manual structure, so a function is found in the same place across tools — excluding meos_internal_*, and splitting oversized groups to stay under the JVM class limits. Emits ~2200 1:1 UDFs, compiling green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
North-Star step ([meos-api-codegen-regularity]): MobilitySpark's UDF layer (
*UDFs.java) is hand-written (~1334 registrations = debt). This addstools/codegen_spark_udfs.py, which generates that surface from the single MEOS-API catalog:@sqlfnmap (MEOS-API Resolve the MEOS surface through the unified GeneratedFunctions jar #18)spark.udf().register(...)Single-signature UDFs emit and compile (proven on
whenTrue); overloaded SQL names (eContains6 sigs,overlaps,stbox) are correctly skipped pending runtime type-dispatch, the next feature. Thencreate()registers the generated class and the bench hand-stubs are deleted → the bench runs on a fully generated surface. The regenerated.javaoutput is not committed (build artifact), matching how MEOS-API #18 commits the parser, not the catalog.