[727] Fix demo notebook to run against current main

[rangareddy]

What is the purpose of the pull request

The demo notebook (demo/notebook/demo.ipynb) no longer runs on current main. This PR fixes the issues reported in #727 plus several more found while getting it running end-to-end, and removes the foot-guns that let the demo rot unnoticed in the first place.

It also silences the noisy Presto RMI warning reported in #316.

Closes #727
Closes #316

This description was rewritten after review. An earlier revision described a demo/jupyter/setup-jdk11.sh startup hook that provisioned OpenJDK 11 on first container start, and notebooks that pinned dependency versions inline. Neither exists any more — JDK 11 and every dependency are baked into the image at build time, and all versions are derived from the root pom. The old text also referenced xtable-core_2.12-0.2.0-SNAPSHOT.jar by name; the predef now loads whatever is in demo/jars so the version is no longer stated anywhere. Squashed to a single commit at the same time.

Changes

Notebook correctness — demo/notebook/demo.ipynb

1. API drift: ported all sync cells from the removed PerTableConfigImpl (deleted in [297] Refactor config classes #480) to the current SourceTable/TargetTable/ConversionConfig builders, and ConversionSourceProvider#init to its single-Configuration signature. Explicit empty Properties are passed on the builders because HudiSourceConfig.fromProperties throws an NPE when additionalProperties is null.
2. Removed the hoodie.write.schema override: under Hudi 1.2's lazy record serialization the rows are encoded with the nullable dataframe-derived schema but were decoded with the non-null override schema, so every record failed with AvroRuntimeException: Malformed data. Length is negative. With the override removed one consistent schema is used.
3. Re-runnability: the seed Hudi table is no longer upgraded in place, and the Delta join cell can be executed more than once.
4. The demo namespace is created in the metastore automatically. The array is constructed as scala.Array(namespace) — the notebook imports java.util._, so a bare Array(namespace) resolves against java.lang.reflect.Array under Ammonite and fails to compile.
5. All dependency-loading code is gone from the notebooks (see below).

Versions come from the root pom — demo/build_demo.sh (new), demo/jupyter/predef.sc (new)

The root cause of #727 is that the demo stated its own versions. build_demo.sh now builds the demo jars and writes demo/jars/versions.properties from the root pom, and both the image's dependency pre-fetch and the kernel predef read that file. There is one place a version is stated, so the demo cannot silently drift from what xtable is built against.

Covers scala.binary.version, spark.version, spark.version.prefix, hudi.version, delta.version, delta.kernel.version, iceberg.hive.runtime.version, log4j.version, plus trino.jdbc.version / presto.jdbc.version (literals in build_demo.sh, since xtable does not depend on the JDBC drivers).

delta-kernel-api/delta-kernel-defaults are included because they are provided-scope dependencies of xtable-core, and without them the ConversionTarget ServiceLoader fails on DeltaKernelConversionTarget before any sync can run.

Dependencies baked into the image — demo/jupyter/Dockerfile (new)

Hudi 1.x classes are Java 11 bytecode but almondsh/almond only ships Java 8 (there is no JDK 11 tag), which failed with UnsupportedClassVersionError. The image now installs OpenJDK 11, pre-fetches every notebook dependency into the coursier cache at build time, and re-points the Scala kernels at that JDK with a bounded heap and the predef wired in — so the first cell does not spend minutes provisioning, and there is no one-time download on first start.

demo/jupyter/verify_dependencies.sh (new) re-checks the cache before Jupyter starts and fetches anything missing, so a version bump without an image rebuild still works.

Demo lifecycle — demo/start_demo.sh, demo/stop_demo.sh (new)

The demo runs in the background, the jar build is split out into build_demo.sh, and stop_demo.sh --reset-data restores the seed datasets.

Presto log noise — demo/presto/log.properties (new) — fixes #316

The presto container repeatedly logged WARN RMI TCP Accept-0 sun.rmi.transport.tcp … accept loop for ServerSocket[…] throws java.net.SocketTimeoutException. The JMX agent's RMI accept loop emits it whenever its accept() times out — harmless, but it floods the compose logs and reads like a startup failure. The new log.properties raises that logger to ERROR and is mounted at etc/log.properties, which the presto entrypoint already passes as -Dlog.levels-file.

Hardening added in review

Thanks @vinishjail97 — all five points are addressed, each with its own reply on the thread.

• The notebook server is published on 127.0.0.1 only. It runs with authentication disabled and 8888 was bound to every interface, so anyone able to reach the host had an unauthenticated Scala kernel: arbitrary code execution with read/write access to the mounted demo/data. The other three services are unchanged — pre-existing, and not unauthenticated code execution.
• The base image is pinned by tag and digest (almondsh/almond:0.14.1@sha256:0407359b…). The kernel-patching step depends on the base image's kernel path, on argv[0] being java, and on -Xmx4g being safe to insert at argv[1], so a moving :latest could break the demo at build time — the same class of failure this PR exists to fix. 0.14.1 is what :latest resolved to (identical manifest digest), so this changes nothing today.
• The kernel-patching step asserts instead of silently doing nothing. Previously, if the glob matched no kernel the comprehension iterated over an empty list and the build succeeded with an unpatched image that fails only when a notebook is run. Both the kernel path and the argv shape are now checked.
• prefetch_dependencies.sh fails on a missing property rather than building a coordinate like org.apache.spark:spark-sql_: and surfacing it as a coursier resolution error. Values are read with cut -f2- so a value containing = is not truncated.
• log4j moves from 2.17.2 to the root pom's 2.22.0. 2.17.2 was carried over from the original demo commit (5d40c6f), not required by almond or Spark.
• Stale xtable jars are removed from demo/jars before copying new ones. predef.sc puts every jar in that directory on the kernel classpath, so after a version bump two copies of the same classes were loaded.

Verification

Correction. An earlier revision of this description claimed every cell ran end-to-end on pristine seed data. That does not currently hold, and the claim was wrong rather than merely outdated — see Known failure below. The earlier run almost certainly reused a demo/data directory from a previous run, which masks the bug. @vinishjail97 also validated the notebooks independently, presumably under the same condition.

What does still pass, on the current revision:

• Hudi → Iceberg initial sync
• Delta → Hudi sync and read-back as Hudi
• Cross-format Delta join of the two tables
• Delta → Iceberg sync
• Trino/Presto connectivity and catalogs

For #316: recreated the presto container with the new mount and confirmed the server reaches ======== SERVER STARTED ======== with all four catalogs loaded and no sun.rmi.transport.tcp lines in docker logs presto.

Hardening round — the notebook cells are untouched by these changes, so rather than a full re-run I verified each changed mechanism directly:

Check	Result
docker port jupyter	127.0.0.1:8888; loopback → HTTP 200, host LAN address → refused; trino still 0.0.0.0 for contrast
Image build with the pinned base	success — patched kernels: [scala212, scala213, scala36]
Kernel start + predef	compiles clean, prints XTable demo kernel ready: …
log4j actually on the classpath	2.22.0, with Java 11.0.30 and Spark 3.4.2, org.apache.xtable.model.InternalTable resolving
Both new asserts, negative tests	fail loudly (no scala kernels found…, kernel argv[0] is not java: ['python'])
prop() with a missing key	fails naming the key
./demo/build_demo.sh	BUILD SUCCESS, all ten properties emitted

To be explicit: I did not re-run the whole notebook in this round. Cell 6 needs the Hive Metastore, and a local HMS could not be brought up on this machine — apache/hive:4.0.0-alpha-2 is amd64-only and sat in emulated derby initOrUpgradeSchema without progressing. A confirming end-to-end run on x86 would be welcome before merge.

Known failure — incremental Hudi → Iceberg sync on pristine data

The round-2 append cell fails, deterministically, on a freshly reset demo/data:

ERROR TableFormatSync: Failed to sync table changes
java.lang.IllegalArgumentException: Cannot add field CustomerKey as an identifier field: not a required field
    at org.apache.iceberg.Schema.validateIdentifierField(Schema.java:159)
    at org.apache.iceberg.SchemaUpdate.applyChanges(SchemaUpdate.java:554)
    at org.apache.xtable.iceberg.IcebergSchemaSync.sync(IcebergSchemaSync.java:60)

Evidence chain:

step	CustomerKey
tracked seed commit 20231105172136406.commit	"string" → required
initial sync → Iceberg metadata v1	identifier-field-ids=[2], required=true
round-2 append commit	["null","string"] → optional
incremental sync	fails as above

The round-2 CSV is read with header=true and no schema, so Spark types every column as a nullable string, flipping CustomerKey from required to optional. IcebergSchemaExtractor.toIceberg then correctly declines to set identifier fields for a nullable record key (the guard added for #366), so XTable's target schema has none — but the table still carries identifier-field-ids=[2] from the initial sync, and SchemaUpdate.apply() re-validates those against the new schema.

grep -rn 'setIdentifierFields' xtable-core/src/main/java returns nothing: IcebergSchemaSync never reconciles identifier fields, so once a record key goes required → optional the Iceberg target is permanently wedged.

This is a distinct gap from #261 / #366, which fixed table creation with an already-nullable key, not the transition on an existing table. It reproduces outside the demo and is an XTable bug, not a demo bug — but the demo is where it surfaces, because the seed's required key plus a schema-less round-2 read makes it deterministic, and stop_demo.sh --reset-data restores exactly that state.

How XTable should behave is a semantics call (drop the identifier fields, refuse the nullability change, or hold the key required), so it needs a decision rather than a quick patch. Filing separately; this PR should probably not be merged claiming a working demo until it is resolved, since the round-2 cell is part of the notebook.

Note that inferSchema=true on the round-2 read is not a fix: the existing schema is all required strings, so inference additionally yields CustomerKey: int and YearlyIncome: double, adding a string → int change that Iceberg rejects outright.

Notes for reviewers

• almondsh/almond publishes amd64 only, so the whole demo runs under emulation on Apple Silicon and is noticeably slower there. Pre-existing, not changed here.
• docker-compose.yaml runs trino 428 while the driver is trino-jdbc 431 (presto matches at 0.283). Left alone since the driver is version-tolerant and the notebooks were validated on it, but both now live in one place if you want them aligned.
• Starting presto appends node.id=presto to the tracked demo/presto/node.properties, so running the demo dirties the working tree. Pre-existing; worth a separate fix.
• container_name: hive-metastore is a hardcoded global name, so the demo collides with any other local compose project using it. Pre-existing; worth a separate fix.

Verify this pull request

No automated test coverage — this is demo tooling. Reviewers should run cd demo && ./start_demo.sh and execute demo/notebook/demo.ipynb.

[vinishjail97]

@rangareddy Can you validate if all demo notebooks are okay now?

[rangareddy]

Yes @vinishjail97 Executed the demo.ipynb notebook end-to-end after debugging and resolving several issues encountered during the runs.

[vinishjail97]

@rangareddy Thanks for working on this! I have validated the demo notebooks locally and added few comments on making the demo more robust.

[rangareddy]

@vinishjail97 all five comments are addressed and answered inline. Force-pushed as a single commit (d6dd5fd) — the 19 commits are squashed, and I verified the squash is byte-identical to the previous diff (same git diff checksum against the merge base) so nothing was lost or reordered in content.

The PR description needed a rewrite, not just an update: it documented a demo/jupyter/setup-jdk11.sh startup hook and inline notebook version pins, neither of which exists any more — JDK 11 and every dependency are baked into the image and all versions come from the root pom. It also named xtable-core_2.12-0.2.0-SNAPSHOT.jar explicitly, which the predef no longer does. There is a note at the top of the description flagging what changed, so anyone arriving mid-thread is not misled.

Two of your points found things worth calling out beyond the fix itself:

• The :latest pin was hiding three assumptions, not one. I added a second assert on argv[0] == 'java' alongside assert paths: if argv[0] were ever a wrapper script rather than the JVM, replacing index 0 and inserting -Xmx4g at index 1 would produce a broken kernel that the glob assert passes happily. Both fire correctly on negative tests. Also, 0.14.0-scala-2.12.17 does not exist — the scala-suffixed tags are single-kernel images, so the unsuffixed 0.14.1 is the right one for the scala* glob. It has the same manifest digest as latest, so the pin changes nothing today.
• log4j 2.17.2 was carried over, not deliberate — git log -S'2.17.2' traces it to 5d40c6f, the original demo commit, and the pre-PR notebook has the same coordinates inline. Now tracks the root pom's 2.22.0, verified as 2.22.0 on a live kernel classpath.

I also fixed one thing you did not raise but which sits in the same area: build_demo.sh never cleared demo/jars, and predef.sc puts every jar there on the kernel classpath. After this branch picked up main's 0.5.0-SNAPSHOT bump, my local demo/jars held both 0.2.0 and 0.5.0 copies of all three jars — two copies of the same classes, resolved in filesystem order. Added rm -f demo/jars/xtable-*.jar before the copies.

On verification, to be straight with you: I did not re-run the whole notebook this round. Cell 6 needs the Hive Metastore and I could not get one up locally — apache/hive:4.0.0-alpha-2 is amd64-only and sat in emulated derby initOrUpgradeSchema without progressing on Apple Silicon. What I did verify is every mechanism these changes touch: the image builds with the pinned base and patches all three Scala kernels; the kernel starts, the predef compiles, and it reports log4j 2.22.0 / Java 11 / Spark 3.4.2 with org.apache.xtable.model.InternalTable resolving; docker port shows 127.0.0.1:8888 with the host's LAN address refused; and both asserts plus the missing-property check fail loudly when broken. The notebook cells themselves are untouched by this round. A confirming end-to-end run on x86 before merge would be welcome.

Three pre-existing things I noticed and deliberately did not change, each probably worth its own issue:

1. almondsh/almond is amd64 only, so the whole demo runs emulated on Apple Silicon.
2. Starting presto appends node.id=presto to the tracked demo/presto/node.properties, so running the demo dirties the working tree.
3. container_name: hive-metastore is a hardcoded global name, so the demo collides with any other local compose project using that name.