Repository navigation
MLE-30002 MLE-30003 Add CLI and API support for sidecar collections and permissions inheritance - #685
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved test correctness and CLI argument-handling issues need fixes.
3 open findings
What changed in this PR
Exposes sidecar metadata inheritance through Flux’s CLI and fluent API, allowing chunk documents to retain source collections and permissions.
Changes:
- Adds collection and permission inheritance options.
- Adds CLI, API, import, and copy tests.
- Documents inheritance defaults and metadata union behavior.
| File | Description |
|---|---|
| flux-cli/src/test/java/com/marklogic/flux/impl/importdata/SidecarInheritanceE2ETest.java | Adds end-to-end inheritance scenarios. |
| flux-cli/src/test/java/com/marklogic/flux/impl/importdata/ImportFilesOptionsTest.java | Checks CLI option mapping. |
| flux-cli/src/test/java/com/marklogic/flux/impl/importdata/ImportAndSplitFilesTest.java | Tests inheritance, unions, and defaults. |
| flux-cli/src/test/java/com/marklogic/flux/impl/copy/CopyTest.java | Tests inheritance during copying. |
| flux-cli/src/test/java/com/marklogic/flux/api/SplitWithApiTest.java | Exercises fluent API inheritance. |
| flux-cli/src/main/java/com/marklogic/flux/impl/importdata/SplitterParams.java | Wires inheritance flags to connector options. |
| flux-cli/src/main/java/com/marklogic/flux/api/SplitterOptions.java | Declares fluent inheritance methods. |
| docs/import/splitting.md | Explains sidecar metadata behavior. |
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
rjrudin
left a comment
rjrudin
left a comment
There was a problem hiding this comment.
A thought on the tests - the beefier tests should be in the Spark connector, where the real work actually happens. Flux just collects CLI args and passes them to the connector.
Generally, I use the following approach for Flux tests, having already added a bunch of integration tests to the Spark connector to fully test it:
- Use "options tests" to make sure that setting the CLI options results in the correct connector options being set.
- Write a single integration test that uses the new options and verifies the data in MarkLogic.
I try to keep 2 to a minimum, as the tests should be redundant with what the Spark connector tests are already doing.
dd00be0 to
484d35b
Compare
Moved the detailed coverage to the connector (SplitWithInheritedMetadataTest, 6 tests). |
…nd permissions inheritance MLE-30002 MLE-30003 Add CLI and API support for sidecar collections and permissions inheritance - Added CLI options --splitter-sidecar-inherit-collections and --splitter-sidecar-inherit-permissions in SplitterParams.java. - Added fluent builder methods outputInheritCollections() and outputInheritPermissions() in SplitterOptions.java. - Corrected documentation errata in docs/import/splitting.md and added reference documentation for the new inheritance options. - Added CLI parsing tests in ImportFilesOptionsTest.java. - Added integration tests in ImportAndSplitFilesTest.java and CopyTest.java covering file import and database document copying. - Added API tests in SplitWithApiTest.java. - Added dedicated end-to-end multi-tenant test suite in SidecarInheritanceE2ETest.java with injected sample data verifying collection partitioning and RBAC isolation.
5f73882 to
370ca8b
Compare
Summary
Implements Jira stories MLE-30002 (sidecar collections inheritance) and MLE-30003 (sidecar permissions inheritance) in Flux. The inheritance logic lives in the Spark connector (marklogic/marklogic-spark-connector#702); Flux exposes it via CLI and API and passes it to the connector.
Changes
SplitterParams/SplitterOptions: new flags and fluent API methods. Both are off by default; include the flag to enable it.--splitter-sidecar-inherit-collections/outputInheritCollections(boolean)→spark.marklogic.write.splitter.sidecar.inheritCollections--splitter-sidecar-inherit-permissions/outputInheritPermissions(boolean)→spark.marklogic.write.splitter.sidecar.inheritPermissionsdocs/import/splitting.md:--permissionsand no collections, not the source document's metadata).--splitter-sidecar-collections/--splitter-sidecar-permissions.--collections/--permissions(or--output-*forcopy) overrides, and that permission inheritance is mainly useful forcopyand archive imports.Tests
Detailed coverage is in the connector PR (
SplitWithInheritedMetadataTest). Flux tests:ImportFilesOptionsTestandCopyOptionsTest: both CLI flags map to the connector options.SplitterOptionsTest: both fluent API methods map to the connector options.CopyTest.splitWithInheritedCollectionsAndPermissionsverifies sidecar collections and permissions in MarkLogic.Dependency
Requires marklogic-spark-connector
3.2-SNAPSHOTwith marklogic/marklogic-spark-connector#702. Merge the connector PR first.