Skip to content

refactor(api): split ClickPipes models by source kind (#1173) - #1180

Open
sdairs wants to merge 5 commits into
claude/1173-api-split-client-testsfrom
claude/1173-api-split-clickpipes-models
Open

sdairs wants to merge 5 commits into
claude/1173-api-split-client-testsfrom
claude/1173-api-split-clickpipes-models

Conversation

@sdairs

@sdairs sdairs commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Part of #1173 (5 of 6: models/clickpipes). Stacked on #1179.

Summary

This splits src/models/clickpipes.rs (4.7k lines) into src/models/clickpipes/. Behaviour and the public surface are unchanged. models.rs still has pub use clickpipes::*, and mod.rs glob re-exports every submodule, so every crate-root and models::* path resolves as before.

File Lines Contents
mod.rs 640 ClickPipe, ClickPipeState, ClickPipePostRequest/ClickPipePatchRequest, ClickPipeSource/ClickPipePostSource/ClickPipePatchSource, destination, field-mapping, settings, scaling (including CDC) and state-patch types, ClickPipesServiceContext, and the credentials several kinds share (MskIamUser, PLAIN, ServiceAccount)
kafka.rs 723 Kafka source, Post/Patch/Mutate variants, offsets, Confluent/Glue schema registries and their unions, AzureEventHub, MutualTLS
kinesis.rs 266 Kinesis source, Post/Patch variants, schema registry
pubsub.rs 313 Pub/Sub source, Post/Patch variants, the ClickPipePostPubSubSource union
object_storage.rs 398 Object storage source, Post/Patch variants
postgres.rs 512 Postgres source, Patch/Mutate variants, pipe settings, table mappings
mysql.rs 546 MySQL source, Patch/Mutate variants, pipe settings, table mappings
mongodb.rs 513 MongoDB source, Patch/Mutate variants, pipe settings, table mappings and configs, structured ingestion
bigquery.rs 457 BigQuery source unions (ClickPipeBigQuerySource, ClickPipeMutateBigQuerySource), their variants, pipe settings, table mappings
schema_discovery.rs 47 ClickPipeSchemaDiscovery*
reverse_private_endpoints.rs 303 ReversePrivateEndpoint*, CreateReversePrivateEndpoint*, UpdateReversePrivateEndpoint, CustomPrivateDnsMapping*

Each discriminated_union! invocation, Default/Display impl and From conversion sits next to the enum it belongs to, so enum declarations stay literal source for the analyzer.

Commits

  1. Pure git mv. clickpipes.rs → clickpipes/mod.rs (100% similarity), so blame follows the file.
  2. Moves only. A script cut each top-level item, with its docs and attributes, into its file in the original order. I compared the sorted line multisets before and after: no line was removed, and the only additions are mod, pub use <kind>::* and use super::{...} lines. The 190 pub struct/enum/type names match. Review it with git show --color-moved=dimmed-zebra.
  3. Classifier. scripts/classify-cloud-integration.py maps each new models/clickpipes/*.rs to {"clickpipes"}, the same as the parent. The old path stays mapped, so diffs that name it remain classified. The test fixture lists the new paths. The install classifier only inventories crates/clickhousectl, so it needs no entry.
  4. .git-blame-ignore-revs entry for commit 2. This SHA must be updated if the stack is rebased before merge.
  5. Docs. The crates/clickhouse-cloud-api/AGENTS.md layout section and the crate README's structure table now say a large model domain may be a directory.

Visibility widenings

None. Every moved item was already pub.

Tests

  • Counts match before and after: 887 passed, 7 ignored (916 listed). The full sorted --list output is byte-identical.
  • model_facade_test and spec_coverage_test pass untouched. python3 scripts/check-openapi-drift.py --dry-run gives the same output before and after (no actionable drift).
  • cargo fmt --all passes, as does clippy for the library crates with --all-targets.
  • Both clickhousectl clippy configurations pass, as do cargo check -p clickhousectl --no-default-features and cargo check --workspace --all-features.
  • cargo test -p clickhouse-cloud-api -p clickhouse-openapi-analyzer and the Python classifier tests pass.

Merge with a merge commit, not a squash. The .git-blame-ignore-revs entry names this PR's move commit. A squash merge would leave it pointing at a commit that isn't on main, and git would silently ignore it.

🤖 Generated with Claude Code


Note

Low Risk
Mechanical file split with preserved public re-exports; no runtime or serde shape changes intended beyond moving code.

Overview
Replaces the ~4.7k-line src/models/clickpipes.rs with a src/models/clickpipes/ tree: shared pipe types stay in mod.rs, and source-specific models move into per-kind modules (Kafka, Kinesis, Postgres, etc.) plus schema_discovery.rs and reverse_private_endpoints.rs. pub use clickpipes::* in models.rs and glob re-exports in mod.rs keep all existing crate-root and models::* paths unchanged—this is layout only, not an API change.

Supporting updates: .git-blame-ignore-revs entry for the move commit, AGENTS.md / README notes that large model domains can be directories, and scripts/classify-cloud-integration.py (and its test) map each new file to the clickpipes integration suite.

Reviewed by Cursor Bugbot for commit 91c7d39. Bugbot is set up for automated code reviews on this repo. Configure here.

sdairs and others added 5 commits October 30, 2026 20:54
…1173)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move the per-source-kind ClickPipes models out of models/clickpipes/mod.rs
into kafka, kinesis, pubsub, object_storage, postgres, mysql, mongodb and
bigquery submodules, plus schema_discovery and reverse_private_endpoints.
mod.rs keeps the top-level pipe types, the shared destination, field
mapping, settings, scaling and state types, the source unions, and the
credentials shared across kinds (MskIamUser, PLAIN, ServiceAccount), and
glob re-exports every submodule so models::* paths are unchanged.

Moves only: the item text is unchanged; only mod/use lines were added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each models/clickpipes/*.rs module maps to the clickpipes suite, like
models/clickpipes.rs did. The old path stays mapped so diffs that name
it remain classified. The install classifier only inventories
crates/clickhousectl and never mapped models/clickpipes.rs, so it needs
no entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sdairs sdairs added enhancement New feature or request client Affects the HTTP client rust Pull requests that update rust code labels Oct 30, 2026
@sdairs
sdairs requested a review from iskakaushik as a code owner October 30, 2026 19:57
@sdairs sdairs added enhancement New feature or request client Affects the HTTP client rust Pull requests that update rust code labels Oct 30, 2026
@sdairs
sdairs added this pull request to stack #1153 October 30, 2026 19:57

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

client Affects the HTTP client enhancement New feature or request rust Pull requests that update rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant