Skip to content

Make Hadoop optional for JSON-only COPY INTO deployments #187

Description

@fupelaqu

Follow-up from #183 / story R1FIX.4.

Context

R1FIX.4 removed Hadoop from the runtime path for local JSON / JSON_ARRAY reads and format
detection (LocalPath + openStream, java.nio.file only). It deliberately changed nothing in
the dependency graph, so the artefact size is unchanged.

Measured for AC 4 (both jars assembled at the same 0.20.3-SNAPSHOT coordinate):

Artefact Before After Δ
softclient4es8-cli-…-assembly.jar 222 427 032 B 222 434 125 B +7 093 B (+0.003 %)

The +7 KB is just the two new LocalPath class files. Δ = 0 by construction: no dependency was
added, removed or re-scoped.

Why the number cannot move today

hadoop-client (core/build.sbt:37) and parquet-avro (core/build.sbt:35) remain compile
dependencies of core because three things still genuinely need them:

  • PARQUETAvroParquetReader / ParquetFileReader consume a Hadoop InputFile
  • DELTA_LAKEDeltaLog.forTable takes a Hadoop Configuration
  • every remote scheme (s3a, s3, gs, abfs*, wasb*, hdfs) — Hadoop is the remote FS layer,
    and HadoopConfigurationFactory resolves their credentials

Proposal

Offer a JSON-only deployment in which Hadoop is not on the classpath at all — e.g. move
hadoop-client / parquet-avro to Provided behind a json-only build variant, or split the
Parquet/Delta/remote sources into a separate optional module.

This is what would actually move the ~314 MB JDBC-driver figure quoted in #183. It is a packaging
and module-boundary decision, not a bug fix, hence its own issue.

Notes

  • Requires deciding the behaviour when a user asks for PARQUET/DELTA_LAKE/a remote URI in a
    JSON-only build — it should fail with a clear "not available in this distribution" message, not a
    NoClassDefFoundError.
  • The JDBC-driver before/after measurement belongs to story R1FIX.7, which re-assembles it against
    released upstreams.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions