Skip to content

build(deps): keep in-process mode off the RPC and HTTP stacks - #527

Merged
anoop-narang merged 1 commit into
datafusion-contrib:mainfrom
hotdata-dev:upstream/local-without-rpc-deps
Oct 16, 2026
Merged

anoop-narang merged 1 commit into
datafusion-contrib:mainfrom
hotdata-dev:upstream/local-without-rpc-deps

Conversation

@anoop-narang

Copy link
Copy Markdown
Collaborator

Closes #522.

In-process mode (liquid-cache-datafusion-local) no longer pulls in the client/server network stack. Its normal dependency graph drops from 387 to 343 crates; arrow-flight, tonic, axum, hyper, h2, reqwest and rustls are all gone.

Changes

  • liquid-cache-common: new rpc feature, off by default. It gates pub mod rpc and makes arrow-flight and prost optional. liquid-cache-datafusion-client and liquid-cache-datafusion-server enable it.
  • ExecutionMetricsResponse moves to liquid_cache_common::metrics. It is a plain serde struct and the only part of rpc the in-process crates used. extract_execution_metrics stays where it is, with the same signature.
  • liquid-cache-datafusion no longer enables object_store's http feature. Nothing in that crate uses it, and it was what pulled in reqwest/hyper/h2/rustls, so gating rpc alone would leave those in place. liquid-cache-datafusion-client calls object_store::parse_url_opts and was relying on this transitively, so it now enables http itself. The server, benchmark and examples already did.
  • liquid-cache-common gets tokio with macros as a dev-dependency. Its #[tokio::test]s only compiled before because tonic enabled that feature.
  • docs.rs builds liquid-cache-common with rpc, and its README notes the feature.

prost remains in the graph through vortex; that is outside this change.

Breaking changes

  • The metrics struct has a new path. liquid_cache_common::rpc::ExecutionMetricsResponse is now liquid_cache_common::metrics::ExecutionMetricsResponse. The serialized JSON is unchanged.
  • rpc is now opt-in. Code that uses liquid_cache_common::rpc directly, or through the common re-exports of liquid-cache / liquid-cache-datafusion / liquid-cache-datafusion-local, needs liquid-cache-common = { ..., features = ["rpc"] }.
  • No more transitive http. Users of liquid-cache-datafusion or -local who registered http(s):// object stores and got object_store's http feature through this crate need to enable it themselves.

Testing

  • Workspace checks. cargo test --workspace --exclude dev-tools gives 236 passed, 0 failed. The test list is identical to main (239 entries). cargo clippy --workspace --all-targets --all-features -D warnings, cargo fmt --check, cargo doc and cargo shear are clean.
  • Each crate built on its own. cargo check -p <crate> --all-targets passes for every crate. Workspace feature unification would otherwise hide a missing feature declaration.
  • Standalone in-process crate. A crate outside the workspace depending only on liquid-cache-datafusion-local:
    • It builds and runs the examples/datafusion-local.rs flow on nano_hits.parquet; the second run is served from cache and extract_execution_metrics reports correctly.
    • cargo tree shows none of the crates listed above.
  • Standalone client crate. A crate depending only on liquid-cache-datafusion-client registers http:// and https:// object stores.
    • With the client's new http line removed, the same build fails at runtime with feature for Http not enabled, so that line is required.
  • End-to-end client/server.
    • bench_server with clickbench_client returns correct query results.
    • The example client/server reads an https:// parquet file.
    • The admin /execution_metrics endpoint returns the same JSON keys as before.

liquid-cache-datafusion-local pulled in arrow-flight, tonic, axum,
hyper, h2, reqwest and rustls although in-process mode never uses the
network.

- Gate liquid_cache_common::rpc behind a new `rpc` feature, off by
  default; client and server enable it.
- Move ExecutionMetricsResponse, a plain serde struct and the only rpc
  item the in-process crates used, to liquid_cache_common::metrics.
- Drop object_store's `http` feature from liquid-cache-datafusion,
  which does not use it. The client calls parse_url_opts and now
  enables `http` itself.
- Give liquid-cache-common's tests tokio `macros` directly; they had
  relied on tonic enabling it.

Closes datafusion-contrib#522
@codecov-commenter

codecov-commenter commented Oct 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@25358ff). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #527   +/-   ##
=======================================
  Coverage        ?   81.38%           
=======================================
  Files           ?       81           
  Lines           ?    13666           
  Branches        ?    13666           
=======================================
  Hits            ?    11122           
  Misses          ?     2216           
  Partials        ?      328           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@anoop-narang
anoop-narang merged commit ead34d2 into datafusion-contrib:main Oct 16, 2026
12 checks passed
@anoop-narang
anoop-narang deleted the upstream/local-without-rpc-deps branch October 16, 2026 08:07
anoop-narang added a commit to hotdata-dev/liquid-cache that referenced this pull request Oct 16, 2026
@alamb

alamb commented Oct 16, 2026

Copy link
Copy Markdown

🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make the in-process mode (liquid-cache-datafusion-local) usable without the client/server (gRPC/Flight) dependencies

3 participants