From 323a57bd839a9039c55e55f0dea444292da6fef6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Manciot?= Date: Mon, 3 Aug 2026 07:34:19 +0200 Subject: [PATCH] =?UTF-8?q?fix:=20bump=20Hadoop=20to=203.4.3=20=E2=80=94?= =?UTF-8?q?=20COPY=20INTO=20Parquet/Delta/remote=20work=20on=20JDK=2023+?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hadoop 3.4.2's `UserGroupInformation.getCurrentUser()` called `javax.security.auth.Subject.getSubject(AccessControlContext)`, which JDK 23 re-specified to throw by default and JEP 486 (JDK 24) made throw unconditionally. That broke every Hadoop-bound `COPY INTO` source — local PARQUET, local DELTA_LAKE and all remote schemes — on JDK 23+. HADOOP-19212 fixed it upstream in 3.4.3 (released 2026-02-24): `UserGroupInformation` now goes through `SubjectUtil.current()`, which binds `Subject.current()` on JDK 18+ and falls back to the old API below that, so the JDK 8 floor is preserved. hadoop-common 3.4.3 is still Java 8 bytecode. One-line change: `Versions.hadoop` 3.4.2 -> 3.4.3. hadoop-client, hadoop-aws and hadoop-azure all read that constant. Verified: - FileSourceSpec forked onto Zulu 25.0.4: 26 passed / 0 failed (was 19/7; the 7 failures were exactly the Parquet + Delta tests, all now green) - core suite on the default JDK 11: 716 passed / 0 failed - `+ core/compile` clean (2.12 + 2.13) - JavaClientCopyIntoS3Spec (MinIO, Docker): 3 passed — 3.4.3's cloud-connector reorganisation did not break S3A wiring - Remote scheme on JDK 25: `UserGroupInformation.getCurrentUser()` OK and `FileSystem.get("s3a://...")` returns S3AFileSystem — no `getSubject` Docs: the COPY INTO JVM-compatibility matrix collapses to all-✔; the older-release guidance (incl. the DBeaver -vm workaround) is kept under a dedicated subsection, and the now-empty "Runtime / JVM limitations" section is removed from known_limitations.md. Story R1FIX.4 follow-up. Closes #183 Co-Authored-By: Claude Opus 5 (1M context) --- documentation/sql/dml_statements.md | 50 ++++++++++++++------------ documentation/sql/known_limitations.md | 10 ------ project/Versions.scala | 2 +- 3 files changed, 29 insertions(+), 33 deletions(-) diff --git a/documentation/sql/dml_statements.md b/documentation/sql/dml_statements.md index 989a2407..1163a214 100644 --- a/documentation/sql/dml_statements.md +++ b/documentation/sql/dml_statements.md @@ -397,42 +397,48 @@ This allows fine-grained property overrides without changing environment variabl (`/path/to/file.jsonl`, `file:///path/to/file.jsonl`) directly through `java.nio.file`, so they work on every supported JVM, **including JDK 23, 24 and 25**. -`PARQUET`, `DELTA_LAKE` and **every remote scheme** go through Apache Hadoop, whose -`UserGroupInformation.getCurrentUser()` calls -`javax.security.auth.Subject.getSubject(AccessControlContext)`. That method was re-specified in -**JDK 23** to throw whenever a Security Manager is not allowed (which is the default), and -[JEP 486](https://openjdk.org/jeps/486) made it throw *unconditionally* in **JDK 24** while removing -the escape hatch — a JDK 24+ launcher refuses to start if `java.security.manager` is set. Those -paths therefore fail with: +`PARQUET`, `DELTA_LAKE` and every remote scheme go through Apache Hadoop, and **they now work on +every supported JVM too** — including JDK 23, 24 and 25. + +| `COPY INTO` source | JDK 8 – 22 | JDK 23 | JDK 24+ | +|---|---|---|---| +| local `JSON` / `JSON_ARRAY`, schemeless or `file:` | ✔ | ✔ | ✔ | +| local file with auto-detected format | ✔ | ✔ | ✔ | +| local `PARQUET` | ✔ | ✔ | ✔ | +| local `DELTA_LAKE` | ✔ | ✔ | ✔ | +| any remote scheme (`s3a://`, `s3://`, `gs://`, `abfs*://`, `wasb*://`, `hdfs://`) | ✔ | ✔ | ✔ | + +##### If you are on an older release + +Releases up to and including **0.20.3** fail on **JDK 23 and newer** for `PARQUET`, `DELTA_LAKE` and +remote schemes, with: | JDK | Error text | `-Djava.security.manager=allow` | |---|---|---| | 23 | `getSubject is supported only if a security manager is allowed` | works around it | | 24 and newer | `getSubject is not supported` | JVM refuses to start | -| `COPY INTO` source | JDK 8 – 22 | JDK 23 | JDK 24+ | -|---|---|---|---| -| local `JSON` / `JSON_ARRAY`, schemeless or `file:` | ✔ | ✔ | ✔ | -| local file with auto-detected format | ✔ | ✔ | ✔ | -| local `PARQUET` | ✔ | ✖ * | ✖ | -| local `DELTA_LAKE` | ✔ | ✖ * | ✖ | -| any remote scheme (`s3a://`, `s3://`, `gs://`, `abfs*://`, `wasb*://`, `hdfs://`) | ✔ | ✖ * | ✖ | +The cause was Apache Hadoop: `UserGroupInformation.getCurrentUser()` called +`javax.security.auth.Subject.getSubject(AccessControlContext)`, which **JDK 23** re-specified to +throw whenever a Security Manager is not allowed (the default), and which +[JEP 486](https://openjdk.org/jeps/486) made throw *unconditionally* in **JDK 24** while removing the +escape hatch — a JDK 24+ launcher refuses to start if `java.security.manager` is set. -\* works on JDK 23 if the host process is started with `-Djava.security.manager=allow`. +This is fixed upstream in **Hadoop 3.4.3** ([HADOOP-19212](https://issues.apache.org/jira/browse/HADOOP-19212)), +which this client now bundles; `UserGroupInformation` uses `Subject.current()` instead. Local +`JSON` / `JSON_ARRAY` were additionally moved off Hadoop entirely in 0.20.3, which is why they work +on every release. -**Workaround for the unsupported combinations:** run the host process on **JDK 21** (or any -JDK ≤ 22). On JDK 23 you may instead add `-Djava.security.manager=allow`; on JDK 24+ there is no -flag that helps. For DBeaver, add this to `dbeaver.ini` **before** `-vmargs` (a DBeaver update -overwrites the file): +**If you cannot upgrade yet,** run the host process on **JDK 21** (or any JDK ≤ 22); on JDK 23 you +may instead add `-Djava.security.manager=allow`. For DBeaver, add this to `dbeaver.ini` **before** +`-vmargs` (a DBeaver update overwrites the file): ``` -vm /path/to/jdk-21/Contents/Home/lib/libjli.dylib ``` -Lifting the Parquet / Delta / remote restriction depends on an upstream Hadoop release that no -longer calls `Subject.getSubject`. Tracked as -[SoftClient4ES#183](https://github.com/SOFTNETWORK-APP/SoftClient4ES/issues/183). +Tracked as [SoftClient4ES#183](https://github.com/SOFTNETWORK-APP/SoftClient4ES/issues/183). **Local path notes:** `~` is **not** expanded — pass an absolute path. Relative paths resolve against the working directory of the process running the query. A `file://` URI may percent-encode diff --git a/documentation/sql/known_limitations.md b/documentation/sql/known_limitations.md index e022f6ea..9dbb6a3d 100644 --- a/documentation/sql/known_limitations.md +++ b/documentation/sql/known_limitations.md @@ -51,16 +51,6 @@ WHERE department_id IN (SELECT id FROM departments WHERE region = 'EU'); The parser rejects this — `IN` accepts only literal value lists today, not a nested `SELECT`. Rewrite it as an explicit JOIN (fully supported), or wait for the next release where the subquery form lands as-is. -## Runtime / JVM limitations - -These are constraints of the host JVM, not unimplemented SQL features — they have no delivery date because they depend on upstream projects. - -- **`COPY INTO` with `PARQUET`, `DELTA_LAKE`, or a remote URI (`s3a://`, `gs://`, `abfs://`, - `hdfs://`) does not work on JDK 23 or newer.** Local `JSON` / `JSON_ARRAY` files do, on every JDK. - This is an Apache Hadoop limitation (JDK 23 re-specified, and JEP 486 in JDK 24 removed, the API - Hadoop's `UserGroupInformation` depends on); run the host process on JDK 21 for those sources. See - [DML statements → COPY INTO → JVM compatibility](dml_statements.md#jvm-compatibility--jdk-23-and-newer). - ## Coming in the upcoming release (Quarter 1 2027) - **Heterogeneous federation**: JOIN or correlate Elasticsearch with PostgreSQL, MySQL, ClickHouse, Snowflake, and more — plus cross-cluster subqueries (e.g. correlate one cluster's data against another's). diff --git a/project/Versions.scala b/project/Versions.scala index 900a9359..3abb36b4 100644 --- a/project/Versions.scala +++ b/project/Versions.scala @@ -61,7 +61,7 @@ object Versions { val fansi = "0.5.1" // Cloud storage connectors (provided scope — must be on classpath at runtime) - val hadoop = "3.4.2" // must match hadoop-client in core/build.sbt + val hadoop = "3.4.3" // must match hadoop-client in core/build.sbt val gcsConnector = "hadoop3-2.2.24" }