[701] Keep exactly one LICENSE and NOTICE in every published jar (branch-0.4) - #875
Open
vinishjail97 wants to merge 2 commits into
Open
[701] Keep exactly one LICENSE and NOTICE in every published jar (branch-0.4)#875vinishjail97 wants to merge 2 commits into
vinishjail97 wants to merge 2 commits into
Conversation
Every jar carried its licensing information twice. META-INF/LICENSE-bundled and META-INF/NOTICE-bundled are inputs for the shade plugin, which writes their content out as META-INF/LICENSE and META-INF/NOTICE, but they were also packaged verbatim because they live in src/main/resources. In the non-shaded jars that second copy was not just redundant, it described dependencies that are not in the jar at all. On top of that, DontIncludeResourceTransformer only drops the exact resource names LICENSE, NOTICE and NOTICE.txt, so the shaded jars additionally carried each bundled dependency's own license file wherever that dependency happened to put it. xtable-hive-metastore-bundled shipped 19 such files across LICENSE.txt, about_files/, license/, META-INF/license/, META-INF/ASM_LICENSE.txt and META-INF/CLI-LICENSE.txt, which made the licensing of the bundle very hard to review. maven-jar-plugin now excludes the two -bundled files from the jar. They stay in target/classes, which is where IncludeResourceTransformer reads them from, so the shaded jars still get the bundled listing as META-INF/LICENSE while the non-shaded jars keep the plain Apache-2.0 text. The shade filters drop the non-canonical dependency license files. The source files under src/main/resources/META-INF are untouched, so validate_shaded_license_coverage.sh keeps working against them unchanged. Adds validate_jar_license_layout.sh, wired into the License Check workflow, to assert the invariant against the built jars: exactly one META-INF/LICENSE, a META-INF/NOTICE, and third party texts only under META-INF/licenses/. Run against the 0.4.0-incubating-rc1 artifacts it reports all 21 offending entries, which is how this went unnoticed for several releases. xtable-utilities is skipped, matching validate_shaded_license_coverage.sh: its deploy is skipped and create_source_release.sh excludes it, so it is not part of the published set. Its shaded jar has no META-INF/LICENSE at all, which is worth fixing separately.
…ndent Verifying the previous commit on main surfaced LICENSE-junit.txt in the root of xtable-hive-metastore-bundled, which branch-0.4 does not produce: the two branches resolve different junit versions, and only one of them ships that file. Enumerating the suffixes seen on one branch is therefore not enough, so the excludes now cover LICENSE*, NOTICE* and COPYING* at the jar root and META-INF/*LICENSE* / META-INF/*NOTICE* rather than a fixed list of extensions. Our own files are unaffected: META-INF/LICENSE and META-INF/NOTICE are written by IncludeResourceTransformer after filtering, and META-INF/licenses/ sits in a path segment of its own so none of these patterns can match it. Confirmed by rebuilding every shaded module from clean and checking that the bundled jars still carry the dependency listing (56 coordinates for aws, 183 for hive-metastore, 13 for hudi-support-extensions, 8 for spark-runtime) while the non-shaded jars still carry the plain 202 line Apache-2.0 text.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
branch-0.4copy of #876, so that 0.4.0-incubating RC2 ships the fixed jar layout: exactly oneMETA-INF/LICENSE, oneMETA-INF/NOTICE, third party texts only underMETA-INF/licenses/.Refs #701. See #876 for the full root cause writeup.
Why
Both binding voters hit this in the RC1 vote: Stamatis cited #701 as
[4]in his-1, noting these issues "persist across releases", and JB noted that META-INF carries both LICENSE and LICENSE-bundled and "seems to be redundant". RC2 is cut from this branch, so the fix has to land here too.Changes
Identical to #876:
maven-jar-pluginexcludesMETA-INF/LICENSE-bundledandMETA-INF/NOTICE-bundledfrom the jar. They stay intarget/classes, whereIncludeResourceTransformerreads them, so bundles keep the dependency listing and non-shaded jars keep the plain Apache-2.0 text.release/scripts/validate_jar_license_layout.sh, wired into the License Check workflow.Files under
src/main/resources/META-INFare not moved, sovalidate_shaded_license_coverage.shis unaffected and this does not overlap #857.Verification
Rebuilt every shaded module from clean. Non-shaded jars: bare Apache-2.0, 202 lines, 0 dependency coordinates. Bundles: Apache-2.0 plus the listing, 56 coordinates for aws, 183 for hive-metastore, 13 for hudi-support-extensions, 8 for spark-runtime. No
-bundledfile in any jar,META-INF/NOTICEintact, andhive-metastore-bundledwent from 19 non-canonical license files to zero. Against the real 0.4.0-incubating-rc1 artifacts the new script reports all 21 offending entries.Known gaps
Non-shaded jars still carry
META-INF/licenses/, andxtable-utilities-bundledhas noMETA-INF/LICENSEat all. Both are described in #876 and are better handled after #857 lands.