PR 2864 target naming updated - #3357
Open
midianyoseph wants to merge 4 commits into
Open
Conversation
…ource configs Previously, `generate_missing_materialization_configs` delegated resource config generation to the generic `stub_config` path, which always derived x-schema-name from the 2nd-to-last collection name component regardless of the materialization's configured strategy. Now resource stubs are created via `update_materialization_resource_spec`, which populates x-schema-name and x-collection-name according to the materialization's `target_naming` and `source` settings. This means `flowctl generate` produces resource configs that match what the runtime and auto-discover would produce for the same materialization.
Adds `flowctl raw migrate-target-naming` to analyze all materializations and determine the appropriate `TargetNamingStrategy` for each, based on the legacy `source.targetNaming` field and endpoint configuration. For each materialization, the tool: * Looks up x-schema-name support from `connector_tags.resource_spec_schema` * Maps the legacy `TargetNaming` enum to the new `TargetNamingStrategy` (`MatchSourceStructure`, `SingleSchema`, `PrefixTableNames`) * Detects the endpoint schema from connector config, falling back to the common schema across existing resource paths * Analyzes each binding to determine whether filling in x-schema-name would change the resource path (requiring manual intervention) or target a different database schema * Falls back from `MatchSourceStructure` to `SingleSchema` when collection names don't match existing resource path schemas * Handles Snowflake's backwards-compat behavior where 1-element paths are preserved when the schema matches the endpoint default The report classifies each materialization as MIGRATE (safe to auto-migrate), MANUAL (needs human intervention due to resource path changes or ambiguous schema), or various SKIP reasons. Disabled tasks with synthetic binding-N resource paths are classified as MIGRATE since they'll backfill on re-enable. Disabled materializations without a built spec are skipped entirely.
…nt API Rebasing onto master surfaced that the flowctl client API was refactored: crate::Client, ctx.client, and crate::api_exec were removed in favor of ctx.pg (postgrest::Postgrest) plus per-call access tokens run through flow_client_next::postgrest::exec. Update all call sites in migrate_target_naming.rs to the new idiom. No behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased copy of #2864 onto current
master— the original branch was~500 commits behind and no longer built. Same changes as #2864, with the
call sites updated to master's refactored flowctl client API so it compiles.
Original PR: #2864