Profile-less composition of modules?

[RonaldHensbergen]

This tool is setup from the start with modules, profiles and contracts. However, would it be possible to create a data platform without a profile? Or that the profile is created runtime? That would make this tool so much more flexible! All possible dependencies will then be stored in contracts, pretty much as done right now.
I would love to hear your thoughts on this!

[SemTiOne]

I'd rather keep profiles as the compatibility boundary, because going fully profile-less means giving up the guarantee that a combination's actually tested, since contracts only confirm kind matches, not whether two modules were ever meant to work together.

I'm fine with the runtime-generated-profile idea though, since build_plan() only takes a profile_path right now. Letting it take a profile dict instead (path becomes a thin wrapper) would give in-memory composition without losing that check.

I'd pull dependsOn from the contractRef graph first, since it already repeats what analyticsDatabase.contractRef: postgres.sql-database shows.

[RonaldHensbergen]

Interesting thought! I thought something like only few relations are allowed, but within that realm you should be allowed to choose. Otherwise I think there will be many profiles. If there are like 10 categories with each 2 or 3 choices, you get many possible profiles, because you can skip categories also.

[SemTiOne]

Exactly, the pairwise check you're describing ("few relations allowed, choose freely within them") already exists via contractRef kind-matching, so the 10x3 explosion isn't blocked by the contract system, it's blocked by needing a hand-authored profile.yaml file per concrete combination before it can be validated. Fix that by letting build_plan() take a dict instead of only a path, and any combination satisfying the contract constraints gets checked on demand without pre-enumerating every permutation.

Though, validate_contract_bindings checks kind as a plain string match with nothing about versions or whether that pairing was actually tested together, so that gap stays even once combinations aren't hand-curated anymore.