[SPARK-58366][SQL] Support JSON_TABLE table-valued function

[ganeshashree]

What changes were proposed in this pull request?

Add the ANSI SQL:2016 JSON_TABLE table-valued function, which shreds a JSON document into a relational table. A row path selects a sequence of JSON items and a COLUMNS clause projects a typed value out of each item into a column.

Syntax (flat, non-nested subset):

JSON_TABLE(json_expr, row_path
  COLUMNS (
    col1 FOR ORDINALITY,
    col2 <type> [PATH '<json_path>'],
    col3 <type> EXISTS [PATH '<json_path>']
  )
  [ { NULL | ERROR } ON ERROR ]
) [AS] alias

• A row path ending in [*] expands a JSON array into one row per element; a non-wildcard path yields a single row for the matched value.
• FOR ORDINALITY: a 1-based BIGINT row counter.
• Value columns are extracted and cast to the declared type; the path may be explicit (PATH '...') or implicit ('$.<columnName>').
• EXISTS columns are a presence test cast to the declared type; a present-but-null JSON value counts as existing, only an absent path is false.
• { NULL | ERROR } ON ERROR: NULL ON ERROR (default) yields no rows on null/malformed input; ERROR ON ERROR raises.
• Usable in a comma join and with LATERAL.

Implementation notes:

• A JsonTable Generator expression is wrapped by the existing Generate operator, so no new execution operator is introduced.
• A dedicated grammar production handles the COLUMNS(...) clause; AstBuilder.visitJsonTableRelation builds the plan. ERROR, ORDINALITY, and JSON_TABLE are added as non-reserved keywords.
• A token-aware navigator (JsonTableEvaluator) extracts values so that a missing path, a JSON null, and the literal string "null" are all distinguished (unlike get_json_object). Non-trailing wildcard paths are rejected during analysis via DATATYPE_MISMATCH.INVALID_JSON_TABLE_PATH.
• Only the flat subset is implemented; NESTED PATH, FORMAT JSON query columns, and DEFAULT ... ON EMPTY are left as follow-ups.

Why are the changes needed?

JSON_TABLE is the SQL-standard way to turn JSON into rows and columns, and is supported by Oracle, DB2, MySQL 8, PostgreSQL 17, Snowflake, and Trino. Spark previously required chaining from_json + explode/inline + get_json_object to achieve the same result. This folds that into one declarative, standard construct and eases migration from those systems.

Does this PR introduce any user-facing change?

Yes. It adds the JSON_TABLE SQL table-valued function and a SQL reference documentation page. ERROR, ORDINALITY, and JSON_TABLE are added as non-reserved keywords, so existing queries using them as identifiers continue to parse. There is no change to existing behavior.

How was this patch tested?

New end-to-end suite JsonTableSuite covering array expansion, ordinality, explicit/implicit paths, missing-vs-null semantics for value and EXISTS columns, NULL/ERROR ON ERROR, [*] over a non-array, mid-path wildcard rejection, non-explode string row items, arrays of scalars/strings, duplicate keys, structure-valued columns, LATERAL joins, nested-field extraction, and column aliases. Existing JsonExpressionsSuite, JsonFunctionsSuite, GeneratorFunctionSuite, PlanParserSuite, DDLParserSuite, SQLKeywordSuite, and SparkThrowableSuite continue to pass.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

[cloud-fan]

0 blocking, 1 non-blocking, 1 nit.
The semantics are coherent and well covered, but wide projections should avoid reparsing every row once per output column.

Suggestions (1)

• sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:548: JSON_TABLE reparses each row item once per projected value or EXISTS column, so wide projections multiply parsing work by the column count. -- see inline

Nits: 1 minor item (see inline comments).

Verification

I traced the new grammar through AstBuilder into JsonTable and the existing Generate execution path. I also checked path validation, missing-versus-null handling, ANSI cast-mode capture, lazy parser cleanup, lateral activation, and the corresponding focused tests.

[cloud-fan]

can we fix merge conflicts?

[ganeshashree]

can we fix merge conflicts?

Fixed merge conflicts.