fix: read local files without Hadoop in COPY INTO (JDK 23+) — R1FIX.4 / #183

[fupelaqu]

Story R1FIX.4 (epic R1FIX, R1 defect closure). Closes #183.

Docs mirror PR: SOFTNETWORK-APP/softclient4es-web#30 — must merge together (dual-docs rule).

Problem

Every local file read in COPY INTO went through Hadoop. FileSystem.get →
UserGroupInformation.getCurrentUser() → Subject.getSubject(AccessControlContext).

The break starts at JDK 23, not 24 — the issue title under-states it:

JDK	Default behaviour	Escape hatch
≤ 22	works	n/a
23	throws getSubject is supported only if a security manager is allowed (default)	-Djava.security.manager=allow
24+	throws getSubject is not supported unconditionally (JEP 486)	none — the launcher refuses to start

Upgrading Hadoop does not help: 3.4.2 is already the pin and still takes that path.

Fix

New LocalPath — a pure classifier with zero Hadoop imports, so it is unit-testable on a JVM
where Hadoop cannot initialise — plus an openStream dispatcher and a validateFile split.
Local paths (schemeless, or file: with an empty/localhost authority) go through
java.nio.file; every other scheme is byte-for-byte unchanged.

🔴 Seven call sites, not the five the issue listed

The two missing ones matter: FileSource.validateFile runs before every read, so a fix that
patched only the five listed sites would still have thrown on JDK 23+.

#	Site	In #183's list?
1	FileSource.validateFile	❌ missing
2–5	JsonFileSource.fromFile, JsonArrayFileSource.fromFile / fromFileInMemory / getMetadata	✅
6	FileFormatDetector.detectJsonType	✅
7	FileFormatDetector.isDeltaTable	❌ missing

Parquet, Delta Lake and all remote schemes stay Hadoop-bound on purpose and remain unsupported
on JDK 23+ (AC 3).

Also: unified scheme detection (AD-10, lead-approved)

HadoopConfigurationFactory.forPath had its own classifier
(Try(new URI(path).getScheme)) that disagreed with LocalPath's. Two silent credential losses:

• S3A://bucket/x.jsonl — uppercase matched no lowercase literal ⇒ localConf(), no S3 credentials
• s3a://bucket/my file.jsonl — new URI throws on the space ⇒ scheme null ⇒ same

Both now reach s3aConf(). file:, schemeless and C:/data/x.jsonl still resolve to localConf().

Evidence

AC 1b — JDK 25 (Zulu 25.0.4, arm64) — with its mandatory CONTROL

java=25.0.4
CONTROL: Hadoop failed as expected -> java.lang.UnsupportedOperationException: getSubject is not supported
resolve(local)   = Some(/var/folders/.../probe18315913422219033464641.jsonl)
resolve(file://) = Some(/var/folders/.../probe18315913422219033464641.jsonl)
resolve(s3a://)  = None
openStream bytes = {"id":1}
detect           =  FILE_FORMAT = JSON

The CONTROL fails with getSubject on the same JVM where the fixed path reads the file — a
green run on a JVM that never had the bug would not be evidence.

• core/testOnly *LocalPathSpec on 25 → 41/41
• core/testOnly *FileSourceSpec on 25 → 19 pass, 7 fail — the 7 are exactly the Parquet +
  Delta tests, each UnsupportedOperationException: getSubject is not supported. That is the
  documented limitation, not a regression, and it makes the run self-controlling.

OQ-4 — Akka 2.6.20 does start on JDK 25

Only WARNING: sun.misc.Unsafe::objectFieldOffset has been called by akka.util.Unsafe — not a
failure. All 16 JSON/detection tests run inside Source.unfoldResource in an ActorSystem and pass.
⇒ the docs ✔ for JDK 24/25 is earned, not deferred, and no product-wide Akka blocker exists.

AC 1a — the regression guard is armed (JDK-independent)

Poisoned fs.file.impl + fs.file.impl.disable.cache (without the latter Hadoop's static
FileSystem cache — keyed on scheme/authority/ugi, not the Configuration — makes the guard
vacuous). Each suite carries a self-check asserting the poisoning really throws.

Sanity-checked: reverting one call site to HadoopInputFile… turns the suite RED with
ClassNotFoundException: Class does.not.Exist not found. A guard that cannot fail is worse than none.

AC 2 / AC 3 / integration

• zulu-11 / zulu-17 / zulu-21 → 67/67 each (also discharges AC 3: Parquet + Delta green on ≤ 22)
• + core/compile and + core/Test/compile (2.12.20 + 2.13.16) — both green
• es8java/testOnly *JavaClient8ReplGateway* → 56/56 against live ES 8.18.3, including the new
  file:// + spaced-path COPY INTO case
• headerCheck, scalafmtCheck, test:scalafmtCheck, scalafmtSbtCheck — all green

AC 4 — fat-jar size: measured, no reduction, and here is why

Both jars assembled at the same 0.20.3-SNAPSHOT coordinate:

Artefact	Before	After	Δ
softclient4es8-cli-…-assembly.jar	222 427 032 B	222 434 125 B	+7 093 B (+0.003 %)

The +7 KB is just the two new LocalPath class files. Δ = 0 by construction — no dependency is
added, removed or re-scoped; hadoop-client and parquet-avro stay compile deps because Parquet,
Delta and remote schemes still need them. Follow-up: #187 (make Hadoop optional for JSON-only
deployments) — that is what would move the ~314 MB driver figure. The JDBC-driver number is
R1FIX.7's, since it needs artefacts only the project lead can release.

Notable deviations from the spec (all verified)

1. The spec's expected message for the non-local-authority test was wrong. Hadoop's
   LocalFileSystem rejects a foreign authority in checkPath before stat'ing →
   Wrong FS: …, expected: file:///, not does not exist; and on JDK ≥ 23 the same call throws
   getSubject. Since AC 1b forks this suite onto 25, the test now accepts either — both prove
   validateHadoopPath ran, which is the real invariant.
2. deleteOnExit ordering. DeleteOnExitHook deletes in reverse registration order, so the
   child must be registered last. The spec's comment stated it inverted; both snippets leaked a
   temp dir every run. Fixed in two tests.
3. io.file.buffer.size = 0 deliberately left unguarded — Hadoop throws identically, so a
   max(1, …) would create a divergence. Parity beats defensiveness.

Not done, deliberately

• Tilde expansion — out of scope, no issue filed (lead's decision). ~ stays literal, pinned by
  a test.
• AD-4 drive-by, unfixed and unfiled (lead's decision): isDeltaTable's remote branch calls
  FileSystem.get(conf) with no URI, so it resolves the default filesystem, not the one implied
  by the path — remote Delta detection has always been wrong. Recorded so it is not lost.

Release protocol

build.sbt:23 already read 0.20.3-SNAPSHOT (set by R1FIX.3 / #186) — verified and left
untouched; it is absent from this diff. 🔴 Nothing was published: no publish, no
publishLocal, no release.yml dispatch, no tag. The project lead merges and releases.