Follow-up from #183 / story R1FIX.4.
Context
R1FIX.4 removed Hadoop from the runtime path for local JSON / JSON_ARRAY reads and format
detection (LocalPath + openStream, java.nio.file only). It deliberately changed nothing in
the dependency graph, so the artefact size is unchanged.
Measured for AC 4 (both jars assembled at the same 0.20.3-SNAPSHOT coordinate):
| Artefact |
Before |
After |
Δ |
softclient4es8-cli-…-assembly.jar |
222 427 032 B |
222 434 125 B |
+7 093 B (+0.003 %) |
The +7 KB is just the two new LocalPath class files. Δ = 0 by construction: no dependency was
added, removed or re-scoped.
Why the number cannot move today
hadoop-client (core/build.sbt:37) and parquet-avro (core/build.sbt:35) remain compile
dependencies of core because three things still genuinely need them:
PARQUET — AvroParquetReader / ParquetFileReader consume a Hadoop InputFile
DELTA_LAKE — DeltaLog.forTable takes a Hadoop Configuration
- every remote scheme (
s3a, s3, gs, abfs*, wasb*, hdfs) — Hadoop is the remote FS layer,
and HadoopConfigurationFactory resolves their credentials
Proposal
Offer a JSON-only deployment in which Hadoop is not on the classpath at all — e.g. move
hadoop-client / parquet-avro to Provided behind a json-only build variant, or split the
Parquet/Delta/remote sources into a separate optional module.
This is what would actually move the ~314 MB JDBC-driver figure quoted in #183. It is a packaging
and module-boundary decision, not a bug fix, hence its own issue.
Notes
- Requires deciding the behaviour when a user asks for
PARQUET/DELTA_LAKE/a remote URI in a
JSON-only build — it should fail with a clear "not available in this distribution" message, not a
NoClassDefFoundError.
- The JDBC-driver before/after measurement belongs to story R1FIX.7, which re-assembles it against
released upstreams.
Follow-up from #183 / story R1FIX.4.
Context
R1FIX.4 removed Hadoop from the runtime path for local
JSON/JSON_ARRAYreads and formatdetection (
LocalPath+openStream,java.nio.fileonly). It deliberately changed nothing inthe dependency graph, so the artefact size is unchanged.
Measured for AC 4 (both jars assembled at the same
0.20.3-SNAPSHOTcoordinate):softclient4es8-cli-…-assembly.jarThe +7 KB is just the two new
LocalPathclass files. Δ = 0 by construction: no dependency wasadded, removed or re-scoped.
Why the number cannot move today
hadoop-client(core/build.sbt:37) andparquet-avro(core/build.sbt:35) remain compiledependencies of
corebecause three things still genuinely need them:PARQUET—AvroParquetReader/ParquetFileReaderconsume a HadoopInputFileDELTA_LAKE—DeltaLog.forTabletakes a HadoopConfigurations3a,s3,gs,abfs*,wasb*,hdfs) — Hadoop is the remote FS layer,and
HadoopConfigurationFactoryresolves their credentialsProposal
Offer a JSON-only deployment in which Hadoop is not on the classpath at all — e.g. move
hadoop-client/parquet-avrotoProvidedbehind ajson-onlybuild variant, or split theParquet/Delta/remote sources into a separate optional module.
This is what would actually move the ~314 MB JDBC-driver figure quoted in #183. It is a packaging
and module-boundary decision, not a bug fix, hence its own issue.
Notes
PARQUET/DELTA_LAKE/a remote URI in aJSON-only build — it should fail with a clear "not available in this distribution" message, not a
NoClassDefFoundError.released upstreams.