diff --git a/.github/.copilot/breadcrumbs/2026-08-17-2356-gep-1748-gateway-api.md b/.github/.copilot/breadcrumbs/2026-08-17-2356-gep-1748-gateway-api.md new file mode 100644 index 00000000..011ea9e7 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-08-17-2356-gep-1748-gateway-api.md @@ -0,0 +1,189 @@ +# GEP-1748 Gateway API Design + +## Requirements + +- Produce a fresh design focused on Gateway API and the GEP-1748 interaction model. +- Keep `networking.fleet.azure.com/v1alpha1 ServiceImport` unchanged. +- Use optional annotations for Azure Front Door, WAF, and Private Link configuration. +- Identify feasibility and repository ownership. +- Produce a detailed, tests-first implementation plan. +- Commit the documentation on a completely new pull request branch using Conventional Commits. + +## Additional comments from user + +- The design must not migrate to `multicluster.x-k8s.io/ServiceImport`. +- The public ingress API must not depend on the prior `FrontDoor*` CRD proposal. +- The working branch is `rchinchani/gep-1748-gateway-api`, created from the latest `origin/main`. +- The architecture should align with GKE's multi-cluster Gateway requirements where the concepts + apply to Fleet and Azure. +- The user approved proceeding with implementation after the design commit. + +## Plan + +### Phase 1: Verify current contracts + +- [x] **Task 1.1: Review repository instructions and documentation conventions.** + - Success criteria: repository-specific design, planning, and breadcrumb requirements are + reflected in the documentation. +- [x] **Task 1.2: Verify ServiceImport and internal export behavior.** + - Success criteria: the design accurately preserves the existing ServiceImport group, version, + schema, status, and aggregation behavior. + +### Phase 2: Create design documentation + +- [x] **Task 2.1: Write the architecture design.** + - Success criteria: the document covers API boundaries, GEP compatibility, annotations, + controllers, AFD/WAF/PLS behavior, status, ownership, security, feasibility, and repository + placement. +- [x] **Task 2.2: Write the implementation plan.** + - Success criteria: the plan is phased, tests-first, independently reviewable, and includes + measurable completion criteria. + +### Phase 3: Validate and commit + +- [x] **Task 3.1: Review documentation consistency and git diff.** + - Success criteria: links, examples, terminology, and referenced repository paths are coherent, + with no unrelated changes. +- [x] **Task 3.2: Commit the design package.** + - Success criteria: the design, plan, and breadcrumb are committed with a detailed Conventional + Commit message and required trailers. + +### Phase 4: Implement the Gateway foundation + +- [x] **Task 4.1: Add Gateway API dependency and scheme tests.** + - Success criteria: Gateway, HTTPRoute, ReferenceGrant, and the unchanged Fleet ServiceImport GVK + are registered. +- [x] **Task 4.2: Add typed AFD annotation parsing.** + - Success criteria: defaults, supported values, misspellings, malformed values, reserved keys, + and cross-resource compatibility are tested. +- [x] **Task 4.3: Add normalized model primitives.** + - Success criteria: listeners, routes, filters, Fleet ServiceImport backends, origins, probes, + WAF, deterministic ordering, and weight validation are represented. +- [x] **Task 4.4: Add the separate hub Gateway manager skeleton.** + - Success criteria: the binary registers schemes, validates controller-wide Azure configuration, + exposes health/readiness, uses repository-standard structured logging, and registers no + reconcilers while the feature is disabled. +- [x] **Task 4.5: Add public and private configuration documentation.** + - Success criteria: separate subsections identify actors and placement for public AFD origins, + optional WAF, internal load balancers, PLS, approval, and cleanup. +- [x] **Task 4.6: Run focused validation.** + - Success criteria: unit tests, build, vet, dependency tidy, and diff checks pass. Race validation + is delegated to CI because this Windows host does not have a C compiler. + +### Detailed checklist + +- [x] Phase 1 / Task 1.1 completed. +- [x] Phase 1 / Task 1.2 completed. +- [x] Phase 2 / Task 2.1 completed. +- [x] Phase 2 / Task 2.2 completed. +- [x] Phase 3 / Task 3.1 completed. +- [x] Phase 3 / Task 3.2 completed. +- [x] Phase 4 / Task 4.1 completed. +- [x] Phase 4 / Task 4.2 completed. +- [x] Phase 4 / Task 4.3 completed. +- [x] Phase 4 / Task 4.4 completed. +- [x] Phase 4 / Task 4.5 completed. +- [x] Phase 4 / Task 4.6 completed. + +### Overall success criteria + +- The design is independent of prior AFD CRD branches. +- Fleet ServiceImport remains unchanged. +- Gateway API is the only public AFD routing API. +- Azure-specific options are optional annotations. +- Formal upstream GEP conformance limitations are explicit. +- The implementation plan can be delivered as focused follow-up pull requests. + +## Decisions + +- Keep the Fleet ServiceImport API because existing controllers, users, and status consumers already + depend on its group and cluster list. +- Follow GEP-1748 semantics without claiming upstream Extended conformance, because the GEP names + the upstream MCS API group. +- Use annotations rather than Azure policy CRDs, accepting weaker API-server validation in exchange + for a smaller and more portable public API. +- Compensate for annotation limitations with typed parsing, admission validation, status + conditions, and warning events. +- Put all controller implementation in `fleet-networking`; `fleet` needs only optional examples or + placement documentation. +- Use a separate hub Gateway controller manager to isolate Azure permissions, rollout, failures, + and ARM throttling. +- Start with direct per-Service member origins; shared per-cluster ingress gateways are deferred. +- Treat the Fleet hub as the configuration cluster and install the multi-cluster GatewayClass there. +- Accept only Fleet ServiceImport backends for the multi-cluster GatewayClass, matching GKE's + separation between single-cluster and multi-cluster GatewayClasses. +- Require explicit feature enablement, Fleet/MCS health, managed identity, Azure provider + registration, quota readiness, and controller readiness. +- Fail static during hub/configuration-plane outages or migrations when Azure resource ownership + cannot be proven. + +## Implementation Details + +- `docs/design/gep-1748-gateway-api.md` defines the architecture and compatibility contract. +- `docs/design/gep-1748-implementation-plan.md` defines the tests-first delivery phases and pull + request sequence. +- Gateway API `v1.2.1` is the selected dependency because it matches Kubernetes `v0.31.1`; newer + Gateway API releases require a repository-wide Kubernetes/controller-runtime upgrade. +- The first implementation slice covers dependency and scheme registration, a separate manager + skeleton, typed annotations, and normalized model primitives. +- The design uses `networking.fleet.azure.com/afd-*` annotations on Gateway and ServiceImport + resources. +- `HTTPRoute.backendRefs` targets the unchanged Fleet ServiceImport group. +- Per-logical-backend weight uses `HTTPRoute.backendRefs.weight`; per-member-cluster weight retains + the existing ServiceExport weight annotation. +- PLS discovery is transported through internal resources rather than public ServiceImport status. +- `cmd/hub-gateway-controller-manager` is a separate process with health, readiness, leader + election, Gateway API schemes, feature gating, and controller-wide Azure configuration. +- `pkg/annotations` rejects unknown reserved AFD annotations and invalid explicit values rather + than silently applying defaults. +- `pkg/controllers/hub/gatewaymodel` provides a provider-neutral, deeply copied, deterministic + model with distinct Gateway API backend weights and Fleet ServiceExport origin weights. + +## Changes Made + +- Added a new design directory for the GEP-1748 Gateway API proposal. +- Added the architecture design. +- Added the detailed implementation plan. +- Added a configuration guide with separate public-origin and private + internal-load-balancer/PLS subsections, each including optional WAF. +- Added an actor-and-placement matrix and labeled each configuration step with + who applies it and whether it belongs in the Fleet hub, member clusters, or Azure. +- Documented the private AFD + optional WAF + PLS topology's SFI-NS253 alignment and clarified that + public member load balancers are not the SFI-NS253 topology. The foundation PR does not claim + compliance until Private Link reconciliation, enforcement, and end-to-end evidence are complete. +- Added the Gateway API dependency, scheme registration tests, manager skeleton, annotation parser, + normalized model, Makefile build target, and focused unit tests. +- Focused tests pass with 81.8% coverage for the manager, 95.1% for annotation parsing, and 89.7% + for the normalized model. Race mode could not run locally because `gcc` is not installed; CI + will provide race validation. +- Added this breadcrumb as the decision and documentation trail. + +## Before/After Comparison + +- **Before:** The main branch had Service export/import and Traffic Manager documentation but no + Gateway API global-ingress design. +- **After:** The branch has a standalone Gateway API design and phased plan for AFD, optional WAF, + and optional PLS while preserving Fleet ServiceImport. + +## References + +- `api/v1alpha1/serviceimport_types.go`: Existing ServiceImport contract. +- `api/v1alpha1/serviceexport_types.go`: Existing ServiceExport and cluster weight annotation. +- `api/v1alpha1/internalserviceexport_types.go`: Existing member-to-hub transport. +- `pkg/controllers/hub/serviceimport/controller.go`: ServiceImport aggregation behavior. +- `pkg/controllers/hub/trafficmanagerbackend/controller.go`: Existing Azure reconciliation + patterns. +- `cmd/hub-net-controller-manager/main.go`: Existing hub controller process boundary. +- `docs/concepts/ExportingService/README.md`: Service export/import user model. +- `docs/concepts/DNSBasedGlobalLoadBalancing/README.md`: Existing global-routing ownership model. +- `docs/howtos/gateway-api-afd-configuration.md`: Proposed public and Private Link configuration + workflow. +- GEP-1748: +- Gateway API v1.2.1 type definitions: + +- GKE multi-cluster Gateway requirements: + +- Azure Front Door Private Link: +- Azure Front Door WAF: +- Repository domain knowledge: no files were present under `.github/.copilot/domain_knowledge`. +- Repository specifications: no files were present under `.github/.copilot/specifications`. diff --git a/Makefile b/Makefile index d8bc01b6..1ef91a0a 100644 --- a/Makefile +++ b/Makefile @@ -183,9 +183,14 @@ generate: $(CONTROLLER_GEN) .PHONY: build build: generate fmt vet ## Build binaries. go build -o bin/hub-net-controller-manager cmd/hub-net-controller-manager/main.go + go build -o bin/hub-gateway-controller-manager cmd/hub-gateway-controller-manager/main.go go build -o bin/member-net-controller-manager cmd/member-net-controller-manager/main.go go build -o bin/mcs-controller-manager cmd/mcs-controller-manager/main.go +.PHONY: run-hub-gateway-controller-manager +run-hub-gateway-controller-manager: manifests generate fmt vet ## Run the hub Gateway controller from your host. + go run ./cmd/hub-gateway-controller-manager/main.go + .PHONY: run-hub-net-controller-manager run-hub-net-controller-manager: manifests generate fmt vet ## Run a controllers from your host. go run ./cmd/hub-net-controller-manager/main.go diff --git a/cmd/hub-gateway-controller-manager/main.go b/cmd/hub-gateway-controller-manager/main.go new file mode 100644 index 00000000..7778f799 --- /dev/null +++ b/cmd/hub-gateway-controller-manager/main.go @@ -0,0 +1,187 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +// Binary hub-gateway-controller-manager watches Gateway API and Fleet +// ServiceImport resources in the hub cluster. +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "os" + + "k8s.io/apimachinery/pkg/runtime" + clientgoscheme "k8s.io/client-go/kubernetes/scheme" + _ "k8s.io/client-go/plugin/pkg/client/auth" + "k8s.io/client-go/rest" + "k8s.io/klog/v2" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/healthz" + "sigs.k8s.io/controller-runtime/pkg/log/zap" + metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" + gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" + gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1" + + "go.goms.io/fleet/pkg/utils/cloudconfig/azure" + + fleetnetv1alpha1 "go.goms.io/fleet-networking/api/v1alpha1" +) + +var ( + metricsAddr = flag.String("metrics-bind-address", ":8082", "The address the metrics endpoint binds to.") + probeAddr = flag.String("health-probe-bind-address", ":8083", "The address the health probe endpoint binds to.") + + enableLeaderElection = flag.Bool("leader-elect", true, "Enable leader election for the controller manager.") + leaderElectionNamespace = flag.String("leader-election-namespace", "fleet-system", "The namespace used for leader election.") + + enableAFD = flag.Bool("enable-afd", false, "Enable Azure Front Door reconciliation.") + cloudConfigFile = flag.String("cloud-config", "/etc/kubernetes/provider/azure.json", "The Azure cloud configuration file.") + afdResourceGroup = flag.String("afd-resource-group", "", "The resource group in which the controller manages Azure Front Door resources.") +) + +func init() { + // Register klog flags before flag.Parse so production verbosity and output + // settings are available consistently with the existing controller binaries. + klog.InitFlags(nil) +} + +func main() { + flag.Parse() + defer klog.Flush() + + ctrl.SetLogger(zap.New(zap.UseDevMode(true))) + + if err := run(currentOptions(), productionDependencies()); err != nil { + klog.ErrorS(err, "Problem running hub Gateway controller manager") + os.Exit(1) + } +} + +func currentOptions() managerOptions { + return managerOptions{ + metricsAddress: *metricsAddr, + probeAddress: *probeAddr, + leaderElection: *enableLeaderElection, + leaderElectionNamespace: *leaderElectionNamespace, + enableAFD: *enableAFD, + cloudConfigFile: *cloudConfigFile, + afdResourceGroup: *afdResourceGroup, + } +} + +type managerOptions struct { + metricsAddress string + probeAddress string + leaderElection bool + leaderElectionNamespace string + enableAFD bool + cloudConfigFile string + afdResourceGroup string +} + +type controllerManager interface { + AddHealthzCheck(string, healthz.Checker) error + AddReadyzCheck(string, healthz.Checker) error + Start(context.Context) error +} + +type dependencies struct { + getConfig func() *rest.Config + newManager func(*rest.Config, ctrl.Options) (controllerManager, error) + signalHandler func() context.Context + loadAFDConfig func(string, string) (*azure.CloudConfig, error) + newScheme func() (*runtime.Scheme, error) +} + +func productionDependencies() dependencies { + return dependencies{ + getConfig: ctrl.GetConfigOrDie, + newManager: func(config *rest.Config, options ctrl.Options) (controllerManager, error) { + return ctrl.NewManager(config, options) + }, + signalHandler: ctrl.SetupSignalHandler, + loadAFDConfig: loadAFDConfiguration, + newScheme: newScheme, + } +} + +func run(options managerOptions, deps dependencies) error { + scheme, err := deps.newScheme() + if err != nil { + return fmt.Errorf("register controller schemes: %w", err) + } + + if options.enableAFD { + if _, err := deps.loadAFDConfig(options.cloudConfigFile, options.afdResourceGroup); err != nil { + return fmt.Errorf("load Azure Front Door configuration: %w", err) + } + // The feature flag intentionally performs configuration validation only + // until the Gateway reconcilers are introduced in the next slice. + klog.InfoS("Azure Front Door configuration is valid; no reconcilers are registered in the foundation release") + } else { + klog.InfoS("Azure Front Door reconciliation is disabled") + } + + mgr, err := deps.newManager(deps.getConfig(), ctrl.Options{ + Scheme: scheme, + Metrics: metricsserver.Options{ + BindAddress: options.metricsAddress, + }, + HealthProbeBindAddress: options.probeAddress, + LeaderElection: options.leaderElection, + LeaderElectionNamespace: options.leaderElectionNamespace, + LeaderElectionID: "hub-gateway-controller-manager.networking.fleet.azure.com", + }) + if err != nil { + return fmt.Errorf("create hub Gateway controller manager: %w", err) + } + + if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil { + return fmt.Errorf("set up health check: %w", err) + } + if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil { + return fmt.Errorf("set up readiness check: %w", err) + } + + klog.InfoS("Starting hub Gateway controller manager") + if err := mgr.Start(deps.signalHandler()); err != nil { + return fmt.Errorf("start hub Gateway controller manager: %w", err) + } + return nil +} + +func newScheme() (*runtime.Scheme, error) { + scheme := runtime.NewScheme() + installers := []func(*runtime.Scheme) error{ + clientgoscheme.AddToScheme, + gatewayv1.Install, + // ReferenceGrant remains v1beta1 in Gateway API v1.2.1. + gatewayv1beta1.Install, + fleetnetv1alpha1.AddToScheme, + } + for _, install := range installers { + if err := install(scheme); err != nil { + return nil, err + } + } + return scheme, nil +} + +func loadAFDConfiguration(cloudConfigPath, resourceGroup string) (*azure.CloudConfig, error) { + if resourceGroup == "" { + return nil, errors.New("afd-resource-group must be configured when Azure Front Door reconciliation is enabled") + } + cloudConfig, err := azure.NewCloudConfigFromFile(cloudConfigPath) + if err != nil { + return nil, err + } + if cloudConfig.SubscriptionID == "" { + return nil, errors.New("Azure subscription ID must be configured") + } + cloudConfig.SetUserAgent("fleet-hub-gateway-controller-manager") + return cloudConfig, nil +} diff --git a/cmd/hub-gateway-controller-manager/main_test.go b/cmd/hub-gateway-controller-manager/main_test.go new file mode 100644 index 00000000..4d8f5d14 --- /dev/null +++ b/cmd/hub-gateway-controller-manager/main_test.go @@ -0,0 +1,348 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +package main + +import ( + "context" + "errors" + "os" + "path/filepath" + "strings" + "testing" + + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/rest" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/healthz" + gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" + gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1" + + "go.goms.io/fleet/pkg/utils/cloudconfig/azure" + + fleetnetv1alpha1 "go.goms.io/fleet-networking/api/v1alpha1" +) + +func TestNewScheme_RegistersGatewayAndFleetTypes(t *testing.T) { + scheme, err := newScheme() + if err != nil { + t.Fatalf("newScheme() error = %v", err) + } + + tests := []struct { + name string + gvk schema.GroupVersionKind + }{ + { + name: "Gateway", + gvk: gatewayv1.SchemeGroupVersion.WithKind("Gateway"), + }, + { + name: "HTTPRoute", + gvk: gatewayv1.SchemeGroupVersion.WithKind("HTTPRoute"), + }, + { + name: "ReferenceGrant", + gvk: gatewayv1beta1.SchemeGroupVersion.WithKind("ReferenceGrant"), + }, + { + name: "Fleet ServiceImport", + gvk: fleetnetv1alpha1.GroupVersion.WithKind("ServiceImport"), + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if !scheme.Recognizes(tt.gvk) { + t.Errorf("scheme does not recognize %s", tt.gvk) + } + }) + } +} + +func TestNewScheme_PreservesFleetServiceImportGVK(t *testing.T) { + scheme, err := newScheme() + if err != nil { + t.Fatalf("newScheme() error = %v", err) + } + + gvks, _, err := scheme.ObjectKinds(&fleetnetv1alpha1.ServiceImport{}) + if err != nil { + t.Fatalf("scheme.ObjectKinds(ServiceImport) error = %v", err) + } + + want := fleetnetv1alpha1.GroupVersion.WithKind("ServiceImport") + for _, got := range gvks { + if got == want { + return + } + } + t.Errorf("ServiceImport GVKs = %v, want to contain %s", gvks, want) +} + +func TestCurrentOptions_UsesCommandLineConfiguration(t *testing.T) { + got := currentOptions() + + if got.metricsAddress != *metricsAddr || + got.probeAddress != *probeAddr || + got.leaderElection != *enableLeaderElection || + got.leaderElectionNamespace != *leaderElectionNamespace || + got.enableAFD != *enableAFD || + got.cloudConfigFile != *cloudConfigFile || + got.afdResourceGroup != *afdResourceGroup { + t.Errorf("currentOptions() = %#v, want values from command-line flags", got) + } +} + +func TestProductionDependencies_AreConfigured(t *testing.T) { + got := productionDependencies() + + if got.getConfig == nil || got.newManager == nil || got.signalHandler == nil || got.loadAFDConfig == nil || got.newScheme == nil { + t.Errorf("productionDependencies() = %#v, want all dependencies configured", got) + } + + // Constructing a manager exercises the production adapter without starting + // it or contacting a Kubernetes API server. + if _, err := got.newManager(&rest.Config{Host: "https://127.0.0.1"}, ctrl.Options{}); err != nil { + t.Fatalf("production newManager() error = %v", err) + } +} + +func TestRun_ManagesStartupLifecycle(t *testing.T) { + tests := []struct { + name string + options managerOptions + schemeError error + loadConfigError error + newManagerError error + healthError error + readyError error + startError error + wantError string + wantConfigLoads int + wantManagerStarts int + }{ + { + name: "scheme registration failure stops startup", + schemeError: errors.New("scheme failed"), + wantError: "register controller schemes", + }, + { + name: "disabled AFD starts manager without loading Azure configuration", + options: managerOptions{ + metricsAddress: ":8082", + probeAddress: ":8083", + leaderElection: true, + leaderElectionNamespace: "fleet-system", + }, + wantManagerStarts: 1, + }, + { + name: "enabled AFD validates configuration and starts manager", + options: managerOptions{ + enableAFD: true, + cloudConfigFile: "provider.json", + afdResourceGroup: "afd-rg", + }, + wantConfigLoads: 1, + wantManagerStarts: 1, + }, + { + name: "configuration failure stops startup", + options: managerOptions{ + enableAFD: true, + }, + loadConfigError: errors.New("invalid cloud config"), + wantError: "load Azure Front Door configuration", + wantConfigLoads: 1, + }, + { + name: "manager creation failure is returned", + newManagerError: errors.New("manager failed"), + wantError: "create hub Gateway controller manager", + }, + { + name: "health registration failure is returned", + healthError: errors.New("health failed"), + wantError: "set up health check", + }, + { + name: "readiness registration failure is returned", + readyError: errors.New("ready failed"), + wantError: "set up readiness check", + }, + { + name: "manager start failure is returned", + startError: errors.New("start failed"), + wantError: "start hub Gateway controller manager", + wantManagerStarts: 1, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + manager := &fakeControllerManager{ + healthError: tt.healthError, + readyError: tt.readyError, + startError: tt.startError, + } + configLoads := 0 + var receivedOptions ctrl.Options + deps := dependencies{ + getConfig: func() *rest.Config { + return &rest.Config{Host: "https://hub.example.com"} + }, + newManager: func(_ *rest.Config, options ctrl.Options) (controllerManager, error) { + receivedOptions = options + if tt.newManagerError != nil { + return nil, tt.newManagerError + } + return manager, nil + }, + signalHandler: func() context.Context { + return context.Background() + }, + loadAFDConfig: func(_, _ string) (*azure.CloudConfig, error) { + configLoads++ + if tt.loadConfigError != nil { + return nil, tt.loadConfigError + } + return &azure.CloudConfig{SubscriptionID: "00000000-0000-0000-0000-000000000000"}, nil + }, + newScheme: func() (*runtime.Scheme, error) { + if tt.schemeError != nil { + return nil, tt.schemeError + } + return newScheme() + }, + } + + err := run(tt.options, deps) + if tt.wantError != "" { + if err == nil { + t.Fatalf("run() error = nil, want containing %q", tt.wantError) + } + if !strings.Contains(err.Error(), tt.wantError) { + t.Errorf("run() error = %q, want containing %q", err, tt.wantError) + } + } else if err != nil { + t.Fatalf("run() error = %v", err) + } + + if configLoads != tt.wantConfigLoads { + t.Errorf("Azure configuration loads = %d, want %d", configLoads, tt.wantConfigLoads) + } + if manager.starts != tt.wantManagerStarts { + t.Errorf("manager starts = %d, want %d", manager.starts, tt.wantManagerStarts) + } + if tt.schemeError == nil && tt.newManagerError == nil && tt.loadConfigError == nil { + if receivedOptions.Scheme == nil { + t.Error("manager options Scheme = nil, want registered scheme") + } + if receivedOptions.Metrics.BindAddress != tt.options.metricsAddress { + t.Errorf("metrics address = %q, want %q", receivedOptions.Metrics.BindAddress, tt.options.metricsAddress) + } + } + }) + } +} + +func TestLoadAFDConfiguration_RejectsInvalidInput(t *testing.T) { + tests := []struct { + name string + cloudConfig string + resourceGroup string + wantError string + }{ + { + name: "missing resource group", + cloudConfig: "provider.json", + wantError: "afd-resource-group", + }, + { + name: "missing cloud config file", + cloudConfig: "does-not-exist.json", + resourceGroup: "afd-rg", + wantError: "does-not-exist.json", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + _, err := loadAFDConfiguration(tt.cloudConfig, tt.resourceGroup) + if err == nil { + t.Fatalf("loadAFDConfiguration() error = nil, want containing %q", tt.wantError) + } + if !strings.Contains(err.Error(), tt.wantError) { + t.Errorf("loadAFDConfiguration() error = %q, want containing %q", err, tt.wantError) + } + }) + } +} + +func TestLoadAFDConfiguration_LoadsValidConfiguration(t *testing.T) { + path := filepath.Join(t.TempDir(), "provider.json") + const subscriptionID = "00000000-0000-0000-0000-000000000000" + config := `{ + "cloud":"AzurePublicCloud", + "location":"eastus", + "subscriptionID":"` + subscriptionID + `", + "resourceGroup":"fleet-rg", + "useManagedIdentityExtension":true + }` + if err := os.WriteFile(path, []byte(config), 0o600); err != nil { + t.Fatalf("os.WriteFile() error = %v", err) + } + + got, err := loadAFDConfiguration(path, "afd-rg") + if err != nil { + t.Fatalf("loadAFDConfiguration() error = %v", err) + } + if got.SubscriptionID != subscriptionID { + t.Errorf("SubscriptionID = %q, want %q", got.SubscriptionID, subscriptionID) + } +} + +func TestLoadAFDConfiguration_RejectsMissingSubscription(t *testing.T) { + path := filepath.Join(t.TempDir(), "provider.json") + config := `{ + "cloud":"AzurePublicCloud", + "location":"eastus", + "resourceGroup":"fleet-rg", + "useManagedIdentityExtension":true + }` + if err := os.WriteFile(path, []byte(config), 0o600); err != nil { + t.Fatalf("os.WriteFile() error = %v", err) + } + + _, err := loadAFDConfiguration(path, "afd-rg") + if err == nil { + t.Fatal("loadAFDConfiguration() error = nil, want missing subscription error") + } + if !strings.Contains(err.Error(), "subscription ID") { + t.Errorf("loadAFDConfiguration() error = %q, want missing subscription ID", err) + } +} + +type fakeControllerManager struct { + healthError error + readyError error + startError error + starts int +} + +func (f *fakeControllerManager) AddHealthzCheck(_ string, _ healthz.Checker) error { + return f.healthError +} + +func (f *fakeControllerManager) AddReadyzCheck(_ string, _ healthz.Checker) error { + return f.readyError +} + +func (f *fakeControllerManager) Start(_ context.Context) error { + f.starts++ + return f.startError +} diff --git a/docs/design/gep-1748-gateway-api.md b/docs/design/gep-1748-gateway-api.md new file mode 100644 index 00000000..e6f8aa9a --- /dev/null +++ b/docs/design/gep-1748-gateway-api.md @@ -0,0 +1,927 @@ +# GEP-1748 Gateway API for Fleet Global Ingress + +## Document status + +- **Status:** Proposed +- **Date:** 2026-08-17 +- **Target repository:** `Azure/fleet-networking` +- **Primary API:** Kubernetes Gateway API +- **Gateway API dependency:** `sigs.k8s.io/gateway-api v1.2.1` +- **Multi-cluster backend API:** `networking.fleet.azure.com/v1alpha1`, kind `ServiceImport` +- **Initial Azure provider:** Azure Front Door Standard/Premium + +## Summary + +This design adds HTTP(S) global ingress to Fleet by combining Kubernetes Gateway API routing with +Fleet's existing multi-cluster `ServiceImport` API. It follows the interaction model described by +[GEP-1748](https://gateway-api.sigs.k8s.io/geps/gep-1748/): an `HTTPRoute` can reference a +`ServiceImport` as a backend, and the Gateway implementation resolves that logical service into +endpoints across the Fleet. + +The design intentionally keeps the existing Fleet `ServiceImport` group, version, kind, schema, and +controller behavior unchanged: + +```text +networking.fleet.azure.com/v1alpha1, kind ServiceImport +``` + +Azure-specific behavior is configured through optional annotations instead of new user-facing +Front Door CRDs. The portable routing contract remains `GatewayClass`, `Gateway`, `HTTPRoute`, and +`ServiceImport`. When another Gateway implementation ignores the Azure annotations, the portable +listener and routing intent remains understandable, although the Fleet-specific `ServiceImport` +group requires explicit support from that implementation. + +Azure Front Door (AFD) is the first provider implementation. Web Application Firewall (WAF) and +Private Link Service (PLS) are optional capabilities layered onto the same Gateway API model. + +## Motivation + +Fleet already provides: + +- Member-cluster `ServiceExport` resources. +- A hub-cluster `ServiceImport` representing one logical service exported by multiple clusters. +- Internal transport resources that identify the contributing member clusters. +- Per-cluster traffic weights through the `networking.fleet.azure.com/weight` annotation. +- Azure global routing integration through Traffic Manager for DNS-based scenarios. + +Fleet does not currently provide a Gateway API implementation for HTTP(S) global ingress. Users +need a model that supports: + +- Host, path, header, and method routing. +- Multiple multi-cluster backends. +- Weighted backend references. +- Edge TLS termination. +- Optional WAF attachment. +- Optional private connectivity from AFD to AKS through PLS. +- Standard Gateway API status and ownership behavior. + +Creating a second set of Front Door-specific routing CRDs would duplicate concepts already defined +by Gateway API. This design instead makes Gateway API the public ingress API and limits Azure +extensions to annotations. + +## Goals + +1. Use `GatewayClass`, `Gateway`, and `HTTPRoute` as the only user-facing AFD routing resources. +2. Allow `HTTPRoute.backendRefs` to target the existing Fleet `ServiceImport`. +3. Preserve the existing Fleet `ServiceImport` API without schema, group, version, or semantic + changes. +4. Implement AFD Standard/Premium as a Gateway controller. +5. Support optional existing WAF policy attachment. +6. Support optional AFD-to-PLS private origin connectivity. +7. Preserve portable Gateway API listener and route configuration. +8. Report standard Gateway API conditions and actionable validation failures. +9. Keep Traffic Manager behavior and APIs independent. +10. Isolate AFD reconciliation from the existing hub networking controller manager. + +## Non-goals + +1. Claim upstream GEP-1748 Extended conformance for the Fleet `ServiceImport` group. +2. Replace or migrate Fleet's `ServiceExport` or `ServiceImport` APIs. +3. Introduce `FrontDoorProfile`, `FrontDoorBackend`, or `FrontDoorCustomDomain` APIs. +4. Model every Azure Front Door property in Kubernetes. +5. Provision WAF policy definitions or managed rule sets. +6. Provision the member-cluster internal load balancer or PLS directly from the hub controller. +7. Implement Gateway API mesh/GAMMA behavior. +8. Make Azure Traffic Manager a GatewayClass. +9. Support arbitrary Azure resource adoption in the initial release. +10. Support shared per-cluster ingress gateways in the initial release. + +## Compatibility with GEP-1748 + +### Dependency baseline + +| Component | Version | Rationale | +|---|---|---| +| Go | Repository baseline | No toolchain upgrade is introduced by Gateway API. | +| Kubernetes libraries | `v0.31.1` | Existing fleet-networking dependency baseline. | +| controller-runtime | `v0.19.0` | Existing fleet-networking dependency baseline. | +| Gateway API | `v1.2.1` | Uses Kubernetes `v0.31.1` and is compatible with the repository baseline. | + +Gateway API `v1.3.0` and later require newer Kubernetes libraries and controller-runtime versions. +They are deferred to avoid coupling this feature to a repository-wide dependency upgrade. + +GEP-1748 defines an `HTTPRoute` backend reference to: + +```yaml +group: multicluster.x-k8s.io +kind: ServiceImport +``` + +Fleet currently owns and reconciles: + +```yaml +group: networking.fleet.azure.com +kind: ServiceImport +``` + +This design adopts the GEP's behavior but not its upstream API group. The Fleet Gateway controller +recognizes the following backend reference as an implementation-specific Extended capability: + +```yaml +backendRefs: +- group: networking.fleet.azure.com + kind: ServiceImport + name: store + port: 8080 +``` + +Consequences: + +- Fleet `ServiceImport` support must be documented as an implementation-specific Extended + capability. +- The implementation must not advertise upstream GEP-1748 Extended conformance unless it also + supports `multicluster.x-k8s.io/ServiceImport`. +- The backend behavior should otherwise follow GEP-1748: routes, filters, weights, namespace + authorization, and status apply consistently to `Service` and `ServiceImport` backends. +- The initial multi-cluster GatewayClass accepts only Fleet `ServiceImport` backend references, + matching the separation used by GKE multi-cluster GatewayClasses. A `Service` backend is rejected + for this class. +- Gateway API conformance claims must describe this class-specific backend restriction. A future + single-cluster GatewayClass may support `Service`, but it is outside this design. + +## Alignment with the GKE multi-cluster Gateway model + +The GKE multi-cluster Gateway setup is a useful operational reference even though the Azure +resource model and network topology differ. This design intentionally follows its major control +plane patterns: + +| GKE multi-cluster Gateway requirement or behavior | Fleet/AFD equivalent | Alignment | +|---|---|---| +| All workload clusters are registered to one fleet | All target clusters are joined to one Fleet hub and have healthy `MemberCluster` state | Aligned | +| Multi-cluster Services is enabled | Fleet member and hub networking controllers reconcile `ServiceExport` and `ServiceImport` | Aligned | +| A selected configuration cluster hosts Gateway resources | The Fleet hub is the configuration cluster for `Gateway`, `HTTPRoute`, and `ServiceImport` | Aligned | +| The platform installs multi-cluster GatewayClasses | The Fleet Gateway chart installs `azure-fleet-afd` | Aligned | +| A hosted controller programs global infrastructure | A dedicated hub Gateway controller manager programs AFD | Same responsibility; different hosting model | +| Workload identity is required | The controller uses Azure workload identity or another approved managed identity mechanism | Aligned | +| Multi-cluster Gateway supports only `ServiceImport` backends | `azure-fleet-afd` accepts only Fleet `ServiceImport` backends | Aligned | +| MCS requirements also apply to Gateway backends | A ServiceImport must be healthy and have resolvable member exports before a route is programmed | Aligned | +| Controller/API enablement is explicit and observable | The feature is enabled explicitly and GatewayClass acceptance confirms readiness | Aligned | +| Configuration-cluster changes can orphan resources | Hub/configuration-plane migration uses an explicit handoff procedure and fails static without ownership proof | Aligned risk treatment | +| Regional control-plane failure causes fail-static behavior | Existing AFD data-plane state remains unchanged while the hub controller is unavailable | Aligned | +| Load balancer quotas apply | AFD profile, endpoint, route, origin, WAF, and Private Link quotas are preflighted and monitored | Aligned | +| Clusters must share supported project/VPC topology | Azure topology is defined by AFD origin reachability and identity authorization, not a same-VNet rule | Intentionally different | + +The Azure implementation does not copy GKE-specific requirements such as VPC-native clusters, +proxy-only subnets, Google APIs, Shared VPC roles, or the `HttpLoadBalancing` add-on. Their Azure +equivalents are AKS/Fleet registration, Azure resource-provider registration, managed identity, +AFD reachability, and PLS readiness. + +## Portability model + +Annotations are used to keep Azure configuration outside the portable Gateway API routing model. +They do not make Azure-specific behavior portable by themselves. Portability is achieved through +graceful degradation: + +1. Listener and routing intent stays in Gateway API fields. +2. Azure annotations are optional and use documented defaults. +3. A non-Azure controller can ignore the annotations. +4. Removing the Azure annotations does not alter host, path, header, method, or backend routing + intent. +5. No Azure resource identifier appears in `HTTPRoute.spec`. + +There is one explicit limitation: Fleet's custom `ServiceImport` group is not directly portable to +controllers that only support the upstream MCS API. Such controllers must add support for Fleet's +group or translate the reference outside this design. + +## User-facing resource model + +### Hub cluster + +The Fleet hub acts as the Gateway API configuration cluster. It contains: + +- One platform-managed `GatewayClass` for the AFD implementation. +- One or more user-created `Gateway` resources. +- User-created `HTTPRoute` resources. +- Existing Fleet `ServiceImport` resources. +- `ReferenceGrant` resources when a route references a backend in another namespace. + +The initial implementation supports exactly one active configuration plane: the Fleet hub where +the Gateway controller is installed. Gateway resources are not copied to member clusters. + +### Member clusters + +Each member cluster contains: + +- The application workload. +- A Kubernetes `Service`. +- A Fleet `ServiceExport`. +- For private origins, an internal `LoadBalancer` Service configured to request PLS creation. + +### Azure + +The Gateway controller owns: + +- AFD profile and endpoint. +- Origin groups and origins. +- Routes and custom domains. +- Security-policy attachment to an existing WAF policy. +- Private Link origin configuration and approval status observation. + +The controller does not own: + +- The AKS-managed internal load balancer. +- The PLS created for the member Service. +- An externally managed WAF policy. +- Public DNS records unless a later design explicitly adds DNS integration. + +## Environment requirements + +These requirements are the Fleet/AFD counterpart of the GKE multi-cluster Gateway preparation +requirements. + +### Fleet and hub requirements + +- All target AKS clusters are registered as healthy members of the same Fleet. +- Fleet member and hub networking controllers are installed and healthy. +- Gateway API CRDs for the selected supported version are installed on the hub. +- The dedicated hub Gateway controller manager and its `azure-fleet-afd` GatewayClass are installed. +- The GatewayClass reports `Accepted=True` before users create Gateways. +- The hub has a durable lifecycle and backup/recovery process because it is the configuration + cluster. + +### Member-cluster requirements + +- The Fleet member networking agent is installed and can publish internal export state to the hub. +- Each multi-cluster backend has a valid `Service` and matching `ServiceExport`. +- Service names, namespaces, and ports satisfy current Fleet ServiceImport conflict rules. +- Public-origin Services expose the public endpoint information required by the controller. +- Private-origin Services use a supported internal load balancer and PLS configuration. + +Gateway API CRDs are not required on member clusters for the initial architecture because Gateway +and HTTPRoute resources live only on the hub. + +### Azure requirements + +- Required Azure resource providers, including `Microsoft.Cdn` and `Microsoft.Network`, are + registered in every subscription used by controller-owned resources or origins. +- The controller has an approved managed identity and scoped Azure RBAC. +- The identity can manage AFD resources in the configured resource group. +- The identity can read externally managed WAF policies that users attach. +- The member or hub identity used for origin discovery can read required public IP, load balancer, + and PLS state. +- Subscription and AFD quotas are sufficient for the requested Gateways, routes, origins, custom + domains, WAF associations, and private endpoints. +- Private Link origins use supported AFD SKU, origin type, Azure region, and approval topology. + +### Enablement and readiness + +Feature enablement is explicit: + +1. Install Gateway API CRDs on the hub. +2. Install the hub Gateway controller with AFD disabled. +3. Configure identity, subscription, resource group, and Azure cloud. +4. Enable the AFD feature. +5. Verify controller health and readiness. +6. Verify `GatewayClass/azure-fleet-afd` reports `Accepted=True`. +7. Verify Fleet ServiceExport/ServiceImport reconciliation is healthy. +8. Create Gateway and HTTPRoute resources. + +## Example configuration + +### GatewayClass + +The platform installs the class. Users reference it but do not modify it. + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: GatewayClass +metadata: + name: azure-fleet-afd +spec: + controllerName: networking.fleet.azure.com/afd +``` + +Provider-wide defaults such as subscription, default resource group, identity, tags, and Azure +cloud environment are controller deployment configuration. They are not repeated on every +Gateway. + +### Gateway without WAF + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: global-ingress + namespace: store +spec: + gatewayClassName: azure-fleet-afd + listeners: + - name: http + protocol: HTTP + port: 80 + hostname: store.example.com + allowedRoutes: + namespaces: + from: Same +``` + +### HTTPRoute to Fleet ServiceImport + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: store + namespace: store +spec: + parentRefs: + - name: global-ingress + hostnames: + - store.example.com + rules: + - matches: + - path: + type: PathPrefix + value: /api + backendRefs: + - group: networking.fleet.azure.com + kind: ServiceImport + name: store-api + port: 8080 + weight: 100 +``` + +The referenced `ServiceImport` remains unchanged: + +```yaml +apiVersion: networking.fleet.azure.com/v1alpha1 +kind: ServiceImport +metadata: + name: store-api + namespace: store +``` + +### Cross-namespace backend + +Gateway API `ReferenceGrant` controls cross-namespace access: + +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: ReferenceGrant +metadata: + name: allow-store-route + namespace: store-backends +spec: + from: + - group: gateway.networking.k8s.io + kind: HTTPRoute + namespace: store + to: + - group: networking.fleet.azure.com + kind: ServiceImport +``` + +## Annotation contract + +All annotations are optional unless stated otherwise. Unknown annotations under the +`networking.fleet.azure.com` prefix are rejected only when they use the reserved AFD annotation +family. Other Fleet annotations remain unaffected. + +### Gateway annotations + +| Annotation | Values | Default | Purpose | +|---|---|---|---| +| `networking.fleet.azure.com/afd-sku` | `Standard_AzureFrontDoor`, `Premium_AzureFrontDoor` | Controller default | Selects the AFD SKU. | +| `networking.fleet.azure.com/afd-resource-group` | Reserved | None | Rejected until a separate per-Gateway ownership design is approved. | +| `networking.fleet.azure.com/afd-waf-policy-id` | Full Azure resource ID | None | Attaches an existing AFD WAF policy. | + +The controller generates stable Azure resource names from the Gateway UID. User-selected Azure +profile names are not part of the initial annotation contract because they complicate ownership, +adoption, collision handling, and deletion. + +The initial implementation does not allow a per-Gateway resource-group override. The resource +group is controller-wide configuration so that Azure RBAC, ownership, quota accounting, and +deletion remain bounded. The `afd-resource-group` annotation is reserved but rejected until a +separate ownership design enables it. + +Annotation keys and values are an implementation API. Existing meanings and defaults cannot change +in place. A replacement must use a new annotation key, support a documented overlap period, and +emit deprecation warnings before the old key is removed. + +### ServiceImport annotations + +| Annotation | Values | Default | Purpose | +|---|---|---|---| +| `networking.fleet.azure.com/afd-origin-connectivity` | `auto`, `public`, `private-link` | `auto` | Selects how AFD reaches all member origins for the logical service. | +| `networking.fleet.azure.com/afd-health-probe-path` | Absolute HTTP path | `/` | Selects the origin-group health probe path. | +| `networking.fleet.azure.com/afd-origin-host-header` | Valid DNS hostname | Derived endpoint hostname | Overrides the Host header sent to origins. | + +The connectivity annotation belongs on `ServiceImport`, rather than `HTTPRoute`, because: + +- Multiple routes can share one logical backend. +- Connectivity is an origin property, not an HTTP match property. +- The same backend cannot safely be public in one route and private in another while sharing an + origin group. +- Backend-level configuration avoids repeating annotations across routes. + +### Existing ServiceExport weight annotation + +The existing annotation remains the source of per-cluster origin weight: + +```text +networking.fleet.azure.com/weight +``` + +`HTTPRoute.backendRefs[*].weight` controls traffic between logical backends. `ServiceExport` +weight controls traffic between member-cluster origins behind one logical `ServiceImport`. + +## Annotation validation + +Annotations are strings and do not receive CRD OpenAPI validation. Misspelled keys and invalid +values may be accepted by the Kubernetes API server. The implementation compensates with: + +1. A single typed annotation parsing package. +2. Table-driven unit tests for all defaults, accepted values, malformed values, and conflicts. +3. A validating admission webhook for deterministic failures where practical. +4. Reconciliation-time validation as the authoritative fallback. +5. Gateway API conditions and Kubernetes warning events with actionable messages. +6. No silent fallback from an explicitly requested capability. + +Examples: + +- An unknown SKU sets the Gateway `Accepted` condition to `False`. +- A malformed WAF policy ID sets `Accepted=False`. +- `private-link` with a non-compatible SKU sets `Accepted=False`. +- A private backend missing PLS information sets the route `ResolvedRefs=False`. +- A backend containing a mixture of public and private member origins sets + `ResolvedRefs=False`. + +The controller must never interpret a malformed value as the default. Defaults apply only when an +annotation is absent. + +## Controller architecture + +### Process boundary + +AFD support is implemented in a separate hub-side controller manager: + +```text +cmd/hub-gateway-controller-manager +``` + +This binary runs in the hub cluster and contains the Gateway API controllers and Azure Front Door +clients. It remains separate from `cmd/hub-net-controller-manager` to provide: + +- Independent feature enablement and rollout. +- Separate Azure permissions. +- Independent leader election and scaling. +- Failure isolation from Service export/import and Traffic Manager. +- Clear ownership of ARM rate limits and metrics. + +### Proposed packages + +```text +cmd/hub-gateway-controller-manager/ +pkg/annotations/ +pkg/controllers/hub/gatewayclass/ +pkg/controllers/hub/gateway/ +pkg/controllers/hub/httproute/ +pkg/controllers/hub/gatewaymodel/ +pkg/providers/azure/frontdoor/ +pkg/providers/azure/origin/ +``` + +### Reconciliation flow + +1. The existing controllers aggregate member `ServiceExport` resources into a hub + `ServiceImport`. +2. The Gateway controller watches `GatewayClass`, `Gateway`, `HTTPRoute`, `ReferenceGrant`, + `ServiceImport`, and relevant internal export resources. +3. It validates the GatewayClass controller name and listener support. +4. It resolves each accepted `HTTPRoute` backend reference. +5. A Fleet `ServiceImport` resolves to its contributing member clusters. +6. Internal export state resolves each member cluster to a public endpoint or PLS. +7. The controller builds a provider-neutral normalized model. +8. The AFD provider reconciles the normalized model into Azure resources. +9. The controller updates Gateway API status only after observing the desired Azure state. + +### Normalized model + +Gateway API objects should not be translated directly into imperative ARM calls. A normalized +model separates Kubernetes validation from provider operations: + +```text +GlobalGateway + listeners[] + routes[] + matches[] + filters[] + backends[] + serviceImport + routeWeight + origins[] + cluster + endpoint + clusterWeight + connectivity + privateLinkResourceID + privateLinkLocation + healthProbe + wafPolicyID +``` + +The model is internal Go code, not a CRD. It allows deterministic comparison, unit testing, and +future provider implementations without changing the public API. + +## ServiceImport resolution + +The public `ServiceImport` schema remains unchanged. The Gateway controller uses: + +- `ServiceImport.status.ports` to validate the backend port. +- `ServiceImport.status.clusters` to identify contributing member clusters. +- `InternalServiceExport.spec.serviceReference` to map an export to its source cluster. +- Existing member-networking state for public IP information and per-cluster weight. +- New internal-only PLS discovery fields when Private Link is implemented. + +Adding PLS information to `InternalServiceExport` or another internal transport resource is allowed +because this design's compatibility promise applies to the public `ServiceImport` API. Internal +schema changes must remain backward-compatible during mixed-version upgrades. + +## HTTPRoute behavior + +### Initial supported features + +- Fleet `ServiceImport` is the only supported `backendRef` kind for the multi-cluster GatewayClass. +- Hostname matching. +- Exact and path-prefix matching. +- Header matching supported by AFD. +- HTTP method matching supported by AFD. +- Weighted backend references. +- Request redirects supported by AFD. +- URL rewrite and header modification where AFD behavior matches Gateway API semantics. +- Same-namespace and `ReferenceGrant`-authorized cross-namespace ServiceImport references. + +### Unsupported features + +An unsupported listener, match, or filter must produce the applicable Gateway API condition and +must not be silently ignored. The initial implementation publishes its supported Gateway API +features through `GatewayClass.status.supportedFeatures` when supported by the selected Gateway API +version. + +A core Kubernetes `Service` backend is unsupported by `azure-fleet-afd`. This is a deliberate +multi-cluster class restriction, consistent with the GKE multi-cluster Gateway model, rather than +an attempt to reinterpret `Service` as a multi-cluster backend. + +### Backend validation + +A Fleet ServiceImport backend is resolved only when: + +- Group is exactly `networking.fleet.azure.com`. +- Kind is exactly `ServiceImport`. +- The object exists and is authorized. +- The requested port exists in `ServiceImport.status.ports`. +- At least one valid member origin can be resolved. +- All origins are compatible with the selected connectivity mode. + +## AFD resource model + +The initial mapping is: + +| Gateway API concept | AFD resource | +|---|---| +| `Gateway` | Profile and endpoint | +| Gateway listener hostname | Custom domain and route domain association | +| `HTTPRoute` | One or more AFD routes/rule sets | +| `ServiceImport` backend | Origin group | +| Member-cluster export | Origin | +| `backendRefs.weight` | Logical backend distribution | +| `ServiceExport` weight | Per-cluster origin weight | +| WAF annotation | Security policy association | + +One Gateway owns one AFD profile in the initial implementation. This provides a simple ownership +and deletion boundary. Profile sharing can be considered separately after the controller's +isolation and quota behavior are understood. + +All owned Azure resources receive tags containing: + +- Fleet hub identity. +- Kubernetes namespace and name. +- Kubernetes UID. +- Controller identifier. + +The Kubernetes UID, not only the object name, is used for ownership checks. + +## Optional WAF + +WAF is enabled by adding a full Azure resource ID to the Gateway: + +```yaml +metadata: + annotations: + networking.fleet.azure.com/afd-waf-policy-id: >- + /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/security-rg/providers/Microsoft.Network/frontdoorWebApplicationFirewallPolicies/store-waf +``` + +Behavior: + +- Absence means no WAF security-policy association. +- The controller attaches the existing policy; it does not create or modify policy rules. +- The controller validates the resource ID shape, policy accessibility, and SKU compatibility. +- Removing the annotation removes only the association, not the WAF policy. +- Deleting the Gateway never deletes an externally managed WAF policy. +- Policy detection/prevention mode remains owned by the WAF policy resource. + +This keeps security policy lifecycle separate from application routing lifecycle. + +## SFI-NS253 alignment for AFD, WAF, and PLS + +The private-backend topology is designed to support SFI-NS253: + +```text +Internet + → Azure Front Door Premium + → optional Azure Front Door WAF policy + → Azure Private Link + → member-cluster Private Link Service + → internal Azure Load Balancer + → application +``` + +In this topology: + +- Member workloads do not require public IP addresses or public load balancer frontends. +- AFD is the public edge and reaches every member origin through Private Link. +- WAF is optional in the API contract; environments that require WAF for SFI or another security + baseline must enforce an approved policy through platform admission or deployment policy. +- Explicit `private-link` connectivity cannot silently downgrade to a public origin. +- A backend remains unresolved until every required PLS and private endpoint connection is ready. + +The public-backend topology described by this design is not the SFI-NS253 topology because member +Services have public load balancer origins. It remains available only for environments where that +exposure is permitted. + +This PR establishes the design, configuration contract, validation primitives, and controller +foundation. It does not by itself demonstrate SFI-NS253 compliance: PLS discovery, AFD Private Link +origin reconciliation, policy enforcement, deployment evidence, and end-to-end validation remain +implementation-plan work. + +## Optional Private Link Service connectivity + +### Member Service + +The member-cluster Service requests an internal load balancer and PLS through supported AKS cloud +provider annotations: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: store-api + namespace: store + annotations: + service.beta.kubernetes.io/azure-load-balancer-internal: "true" + service.beta.kubernetes.io/azure-pls-create: "true" +spec: + type: LoadBalancer + ports: + - name: http + port: 8080 + targetPort: 8080 +``` + +The matching `ServiceExport` remains unchanged. + +### Connectivity modes + +`auto`: + +- Use Private Link when every valid member origin has discoverable PLS state. +- Otherwise use public connectivity when every valid member origin has a valid public endpoint. +- Reject mixed public/private resolution instead of guessing. + +`public`: + +- Require all valid member origins to expose a usable public endpoint. +- Ignore PLS state for origin creation. + +`private-link`: + +- Require an AFD SKU that supports Private Link origins. +- Require all valid member origins to expose discoverable PLS resource IDs and locations. +- Reject the backend until required private endpoint approvals are complete. + +### Discovery + +The member networking controller discovers: + +- PLS resource ID. +- Azure location. +- Provisioning state. +- Any stable information required to correlate the PLS with the exported Service. + +The information is transported to the hub through an internal Fleet networking resource. It is +not added to public `ServiceImport.status`. + +### Constraints + +- Public and Private Link origins are not mixed within one origin group. +- The controller does not silently downgrade explicitly requested `private-link` connectivity. +- Private endpoint approval may be asynchronous; status remains not programmed until Azure reports + a usable connection. +- PLS deletion or replacement triggers origin reconciliation without changing the ServiceImport. +- Direct per-Service origins are the initial model. Shared per-cluster Gateway origins are deferred. + +## TLS and certificates + +The initial milestone should support HTTP so that routing and origin reconciliation can be +validated independently. HTTPS requires a separate implementation decision because AFD certificate +sources do not map perfectly to Kubernetes TLS Secrets. + +The preferred first HTTPS option is an AFD-managed certificate for a listener hostname. Bring-your- +own certificate support, including Key Vault/LUMA-managed certificates, requires a documented +reference mechanism and identity permissions. Certificate identifiers must not be stored as secret +values in annotations. + +HTTPS design completion is required before declaring the feature production ready, but it does not +block the initial controller and public-origin implementation. + +## Status and events + +The implementation follows Gateway API status conventions. + +### GatewayClass + +- `Accepted=True` when `controllerName` is recognized and controller configuration is valid. +- `Accepted=False` for invalid platform configuration. + +### Gateway + +- `Accepted` reflects API, annotation, and listener validation. +- `Programmed` becomes true only after the AFD profile, endpoint, domains, and required security + associations match the desired state. +- `addresses` reports the AFD endpoint hostname. +- Listener conditions report unsupported protocols, hostname conflicts, and route attachment state. + +### HTTPRoute + +For each parent: + +- `Accepted` reports listener attachment. +- `ResolvedRefs` reports ServiceImport, port, namespace authorization, and origin resolution. +- Implementation-specific conditions may report Azure programming failures without replacing + standard conditions. + +### Events + +Warning events are emitted for: + +- Invalid annotations. +- Unsupported Gateway API features. +- Missing or incompatible ServiceImport ports. +- Missing public endpoint or PLS information. +- Azure authorization or quota failures. +- Private endpoint approval requirements. + +Repeated events must be rate limited. + +## Ownership, deletion, and drift + +- The Gateway receives a finalizer only before the controller creates an owned Azure resource. +- Deleting a route removes only route-owned configuration. +- Deleting a Gateway removes its owned AFD resources before removing the finalizer. +- Externally managed WAF policies and member-cluster PLS resources are never deleted. +- Azure resources not tagged with the expected Gateway UID are not adopted or deleted. +- Reconciliation is idempotent and compares normalized desired state with observed Azure state. +- Out-of-band edits to owned properties are corrected. +- Out-of-band edits to unowned external resources are observed but not overwritten. + +### Configuration-plane migration + +Changing or rebuilding the hub configuration plane is an explicit migration, not an ordinary +controller restart: + +1. Preserve all Gateway, HTTPRoute, ReferenceGrant, ServiceImport, and ownership identity data. +2. Start the replacement controller in observation-only mode. +3. Verify every Azure resource is tagged with and attributable to the expected Gateway UID. +4. Transfer reconciliation ownership. +5. Disable the old controller only after the replacement reports complete observation. + +If ownership cannot be proven, the replacement controller fails static: it reports status and does +not create, adopt, update, or delete Azure resources. Disabling the feature while Gateways still +exist is rejected or prominently warned because it can leave intentionally persistent Azure +resources without an active reconciler. + +### Control-plane outage behavior + +AFD continues serving the last successfully programmed configuration when the Fleet hub, +Kubernetes API, controller, or Azure management plane is unavailable. During the outage: + +- No speculative changes are made. +- Kubernetes desired-state changes remain pending. +- Existing AFD data-plane traffic continues subject to Azure Front Door availability. +- Reconciliation resumes from observed state after recovery. + +## Security and authorization + +- The controller uses workload identity or another supported managed identity mechanism. +- Azure permissions are scoped to the configured AFD resource group where practical. +- WAF policy access is read plus association; WAF policy mutation is not required. +- Member networking requires read access to load balancer and PLS state needed for discovery. +- Full Azure resource IDs are identifiers, not credentials. +- Secrets, private keys, tokens, and connection strings are forbidden in annotations. +- `ReferenceGrant` is mandatory for cross-namespace ServiceImport references. +- Gateway and route status must not expose credentials or sensitive Azure responses. + +## Coexistence with Traffic Manager + +Traffic Manager remains the DNS-based global routing solution for scenarios such as: + +- Non-HTTP protocols. +- DNS-level failover. +- Existing public endpoints. +- Workloads that do not need an edge proxy, WAF, or HTTP routing. + +AFD handles HTTP(S), edge proxying, application routing, WAF, and optional Private Link. + +The controllers are independent. A ServiceImport may be referenced by both products, but each +controller owns only its Azure resources. The implementation should emit a warning when +configuration creates an unsupported or ambiguous endpoint topology, but it must not modify +Traffic Manager resources. + +## Repository ownership + +### `fleet-networking` + +All implementation changes belong in this repository: + +- Gateway API dependencies and scheme registration. +- Gateway controller manager binary and chart. +- GatewayClass, Gateway, HTTPRoute, and ReferenceGrant reconcilers. +- Annotation parsing and validation. +- ServiceImport and internal export resolution. +- AFD, WAF association, and PLS origin clients. +- Status, events, metrics, tests, examples, and documentation. + +### `fleet` + +No controller or API changes are required. Optional follow-up changes may add: + +- Placement examples that deploy Services and ServiceExports. +- Documentation that identifies the hub as the Gateway configuration cluster. +- Cluster capability labels if placement later depends on Private Link support. + +### AKS cloud provider + +The existing Service annotation contract remains responsible for internal load balancer and PLS +provisioning. Changes are required only if current status does not expose enough stable information +for Fleet member networking to discover the PLS. + +## Feasibility + +| Capability | Feasibility | Key consideration | +|---|---|---| +| GatewayClass/Gateway/HTTPRoute controller | High | Standard controller-runtime integration. | +| Existing Fleet ServiceImport backend | High | Group is implementation-specific and cannot claim upstream GEP Extended conformance. | +| GKE-style hub/config-cluster model | High | Fleet hub already provides a central configuration and membership plane. | +| Public AFD origins | High | Existing internal export state already includes public IP information. | +| Host/path routing | High | Directly maps to AFD routes and rule sets for a defined supported subset. | +| Existing WAF policy attachment | High | Requires resource ID validation, permissions, and SKU checks. | +| PLS discovery | Medium | Internal transport must carry PLS resource ID, location, and readiness. | +| AFD Private Link origins | Medium | Requires Premium capability, asynchronous approval handling, and strict topology validation. | +| Annotation validation | Medium-high | Webhook and reconciliation validation compensate for lack of CRD schema. | +| HTTPS with AFD-managed certificate | Medium-high | Requires domain ownership and provisioning lifecycle handling. | +| Key Vault/LUMA certificate integration | Medium | Requires a separate certificate reference and authorization design. | +| Formal GEP-1748 Extended conformance | Not targeted | Requires upstream `multicluster.x-k8s.io/ServiceImport`. | + +The design is technically feasible. The largest delivery risks are Private Link lifecycle handling, +certificate integration, and maintaining clear status through asynchronous ARM operations. Keeping +the public ServiceImport unchanged is not a technical blocker, but it is a deliberate conformance +tradeoff. + +## Rollout + +1. Ship the controller and chart behind an explicit feature flag. +2. Require healthy Fleet membership, Service export/import, managed identity, and Azure provider + registration before accepting the GatewayClass. +3. Start with HTTP and public AFD origins. +4. Add WAF attachment. +5. Add member PLS discovery and private origins. +6. Add HTTPS and certificate lifecycle. +7. Run Gateway API conformance tests for the features applicable to the multi-cluster class. +8. Publish Fleet ServiceImport behavior as implementation-specific Extended support. +9. Promote only after upgrade, configuration-plane migration, deletion, Azure throttling, quota, + and control-plane failure testing. + +## Open questions + +1. Which HTTPRoute filters map precisely enough to AFD to advertise as supported? +2. Should AFD-managed certificates be the only initial HTTPS mode? +3. How should Private Link approval ownership be represented when manual approval is required? +4. Should one Gateway always own one AFD profile, or should profile sharing be designed later? +5. What mixed-version compatibility window is required for internal PLS transport fields? + +## References + +- [Configure Gateway API with Azure Front Door](../howtos/gateway-api-afd-configuration.md) +- [GEP-1748: Gateway API Interaction with Multi-Cluster Services](https://gateway-api.sigs.k8s.io/geps/gep-1748/) +- [Gateway API v1.2.1 type definitions](https://github.com/kubernetes-sigs/gateway-api/blob/v1.2.1/apis/v1/gateway_types.go) +- [Gateway API cross-namespace routing](https://gateway-api.sigs.k8s.io/guides/multiple-ns/) +- [GKE multi-cluster Gateway requirements](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/prepare-environment-multi-cluster-gateways#requirements) +- [Azure Front Door Private Link](https://learn.microsoft.com/azure/frontdoor/private-link) +- [Azure Web Application Firewall on Azure Front Door](https://learn.microsoft.com/azure/web-application-firewall/afds/afds-overview) +- `api/v1alpha1/serviceimport_types.go`: Existing Fleet ServiceImport contract. +- `api/v1alpha1/serviceexport_types.go`: Existing Fleet ServiceExport and weight contract. +- `api/v1alpha1/internalserviceexport_types.go`: Current member-to-hub exported Service state. +- `pkg/controllers/hub/serviceimport/controller.go`: Existing ServiceImport aggregation. +- `pkg/controllers/hub/trafficmanagerbackend/controller.go`: Existing Azure global-routing reconciliation patterns. +- `cmd/hub-net-controller-manager/main.go`: Existing hub networking controller process. diff --git a/docs/design/gep-1748-implementation-plan.md b/docs/design/gep-1748-implementation-plan.md new file mode 100644 index 00000000..12bd6990 --- /dev/null +++ b/docs/design/gep-1748-implementation-plan.md @@ -0,0 +1,602 @@ +# GEP-1748 Gateway API Implementation Plan + +## Plan status + +- **Status:** Proposed +- **Date:** 2026-08-17 +- **Design:** [GEP-1748 Gateway API for Fleet Global Ingress](gep-1748-gateway-api.md) +- **Target repository:** `Azure/fleet-networking` +- **Branch:** `rchinchani/gep-1748-gateway-api` + +## Delivery principles + +1. Preserve `networking.fleet.azure.com/v1alpha1 ServiceImport` without API changes. +2. Use Gateway API resources as the public global-ingress API. +3. Use optional annotations for Azure-specific configuration. +4. Write unit tests before implementation whenever practical. +5. Build a provider-neutral normalized model before ARM reconciliation. +6. Deliver public AFD origins before WAF, PLS, and HTTPS. +7. Keep AFD in a separate hub controller manager. +8. Fail explicitly through Gateway API conditions; never silently downgrade requested behavior. +9. Keep each implementation pull request independently reviewable and testable. + +## Phase 0: Confirm API and dependency baselines + +### Task 0.0: Define environment prerequisites + +- [x] Map GKE multi-cluster Gateway prerequisites to Fleet, AKS, and Azure. +- [x] Define Fleet membership and ServiceExport/ServiceImport health requirements. +- [x] Define hub/configuration-plane selection, enablement, readiness, and migration procedures. +- [x] Define required Azure resource providers, identity, RBAC, quotas, and network reachability. + +**Success criteria** + +- A preflight checklist can determine whether the environment is ready. +- The `azure-fleet-afd` GatewayClass remains unaccepted when controller prerequisites are invalid. +- GKE-specific VPC and API requirements are not incorrectly applied to Azure. + +### Task 0.1: Select the Gateway API version + +- [x] Choose Gateway API `v1.2.1`, which uses the repository's Kubernetes `v0.31.1` + dependency baseline. +- [x] Record the supported Core and Extended feature set. +- [x] Confirm the selected release includes the status fields used by the controller. + +**Success criteria** + +- The selected module version builds with the repository. +- A compatibility table identifies supported Gateway API and Kubernetes versions. +- The design's status behavior matches the selected API version. + +### Task 0.2: Freeze the annotation contract + +- [x] Review annotation names, values, defaults, and ownership. +- [x] Keep resource-group configuration controller-wide; reserve and reject the annotation. +- [x] Define additive versioning and deprecation rules for annotations. + +**Success criteria** + +- Every annotation has a documented parser, default, validation rule, and failure condition. +- No credential or secret values are permitted. +- Removing Azure annotations leaves portable Gateway routing intent. + +### Task 0.3: Define the supported AFD mapping + +- [x] Map Gateway listeners, HTTPRoute matches, filters, and backends to AFD resources. +- [x] Enforce Fleet ServiceImport as the only backend kind for the multi-cluster GatewayClass. +- [x] Identify unsupported or semantically incompatible features. +- [x] Define resource and quota limits that must be preflighted before production enablement. + +**Success criteria** + +- A reviewed mapping table exists. +- Unsupported features have explicit Gateway API condition behavior. +- The initial conformance profile is testable. + +## Phase 1: Add test infrastructure and dependencies + +### Task 1.1: Add dependency compile tests + +- [x] Add a minimal test that registers Gateway API and Fleet networking schemes. +- [x] Verify existing Fleet ServiceImport GVK registration remains unchanged. + +**Success criteria** + +- The test fails before dependencies and scheme registration are added. +- The test passes after registration. +- No generated Fleet CRD changes are produced. + +### Task 1.2: Add Gateway API dependencies + +- [x] Add `sigs.k8s.io/gateway-api v1.2.1`. +- [x] Run `go mod tidy`. +- [x] Register Gateway API types with the new manager scheme. + +**Success criteria** + +- Target packages compile. +- Existing dependency replacements remain valid. +- `ServiceImport` continues using `networking.fleet.azure.com/v1alpha1`. + +### Task 1.3: Create controller-manager skeleton + +- [x] Add `cmd/hub-gateway-controller-manager`. +- [x] Add health, readiness, metrics, leader election, repository-standard logging, and signal + handling. +- [x] Add a feature-enable flag for AFD reconciliation. +- [x] Add placeholder Azure configuration loading without creating resources. + +**Success criteria** + +- The binary builds and starts against envtest. +- Health and readiness endpoints pass. +- No controllers run when the feature is disabled. + +## Phase 2: Implement typed annotation handling + +### Task 2.1: Write annotation parser tests + +- [x] Cover absent values and defaults. +- [x] Cover every accepted SKU and connectivity value. +- [x] Cover malformed Azure resource IDs, hostnames, and probe paths. +- [x] Cover unknown reserved AFD annotation names. +- [x] Cover incompatible combinations. + +**Success criteria** + +- Tests are table-driven. +- Explicit invalid values never fall back to defaults. +- Error messages identify object, key, value, and expected format. + +### Task 2.2: Implement annotation package + +- [x] Add constants and typed configuration structures. +- [x] Add Gateway annotation parsing. +- [x] Add ServiceImport annotation parsing. +- [x] Add focused Azure Front Door WAF policy resource ID validation. + +**Success criteria** + +- All parser tests pass. +- Reconciler packages consume typed values instead of reading annotation maps directly. +- No annotation parsing logic is duplicated. + +### Task 2.3: Add admission validation + +- [ ] Add validating webhook handlers for Gateway and annotated ServiceImport resources. +- [ ] Keep reconciliation validation as the authoritative fallback. +- [ ] Add webhook chart and certificate configuration. + +**Success criteria** + +- Deterministic annotation errors are rejected at admission when the webhook is enabled. +- Reconciliation reports the same errors when the webhook is unavailable. +- Existing non-AFD Fleet annotations remain unaffected. + +## Phase 3: Build ServiceImport origin resolution + +### Task 3.1: Write public-origin resolver tests + +- [ ] Resolve a ServiceImport port. +- [ ] Resolve contributing clusters. +- [ ] Resolve public IP/DNS information from internal exports. +- [ ] Apply existing ServiceExport cluster weights. +- [ ] Reject missing ports, empty clusters, invalid exports, and mixed endpoint states. + +**Success criteria** + +- Resolver behavior is deterministic regardless of Kubernetes list order. +- Tests cover member deletion and conflicting exports. +- Public `ServiceImport` objects are not mutated. + +### Task 3.2: Implement origin resolver + +- [ ] Add indexes for ServiceImport-to-route and internal-export-to-ServiceImport relationships. +- [ ] Resolve public origins from existing internal resources. +- [ ] Normalize cluster ordering and weights. +- [ ] Expose resolver errors suitable for `ResolvedRefs` conditions. + +**Success criteria** + +- ServiceImport updates enqueue affected routes. +- Internal export updates enqueue only affected routes. +- No polling is required for Kubernetes state changes. + +### Task 3.3: Define the internal normalized model + +- [ ] Write model validation and equality tests. +- [x] Add listeners, routes, matches, filters, logical backends, origins, WAF, and probes. +- [ ] Add stable sorting and deterministic naming inputs. + +**Success criteria** + +- Equal Kubernetes intent produces an equal normalized model. +- Model serialization used in tests is stable. +- Provider packages do not read Kubernetes resources directly. + +## Phase 4: Implement GatewayClass and Gateway reconciliation + +### Task 4.1: Write GatewayClass controller tests + +- [ ] Test recognized and unrecognized controller names. +- [ ] Test invalid controller Azure configuration. +- [ ] Test missing Azure resource-provider registration and insufficient quota preflight results. +- [ ] Test status update conflict retries. + +**Success criteria** + +- `Accepted` follows Gateway API semantics. +- The controller does not modify classes owned by another implementation. + +### Task 4.2: Implement GatewayClass controller + +- [ ] Watch `GatewayClass`. +- [ ] Publish accepted status and supported features. +- [ ] Enqueue Gateways when class acceptance changes. + +**Success criteria** + +- Only `networking.fleet.azure.com/afd` is accepted. +- Status is idempotent and generation-aware where supported. + +### Task 4.3: Write Gateway validation tests + +- [ ] Test listener protocols, ports, hostnames, and route namespace policy. +- [ ] Test AFD SKU and WAF annotation validation. +- [ ] Test unsupported and conflicting listener configurations. + +**Success criteria** + +- Invalid configuration produces expected listener and Gateway conditions. +- Unsupported listeners never create Azure resources. + +### Task 4.4: Implement Gateway controller foundation + +- [ ] Resolve GatewayClass and typed annotations. +- [ ] Generate stable Azure resource names from Gateway UID. +- [ ] Add ownership tags and finalizer rules. +- [ ] Update Gateway and listener status. + +**Success criteria** + +- A valid Gateway reaches `Accepted=True`. +- A finalizer is added only when Azure resource ownership begins. +- Deleting a never-programmed Gateway does not block. + +## Phase 5: Add Azure Front Door provider and public Gateway + +### Task 5.1: Define provider interfaces and fakes + +- [ ] Add narrow interfaces for AFD profile, endpoint, origin group, origin, route, domain, rule + set, and security policy operations. +- [ ] Add deterministic fake clients. +- [ ] Add Azure error classification tests. + +**Success criteria** + +- Controller tests do not require live Azure. +- Retryable, authorization, conflict, not-found, and terminal errors are distinguishable. + +### Task 5.2: Add Azure SDK clients + +- [ ] Add required `armcdn` SDK dependencies. +- [ ] Initialize clients from the controller's Azure configuration. +- [ ] Apply repository-standard retry, rate limit, and user-agent behavior. + +**Success criteria** + +- Client initialization tests pass. +- No credentials are logged. +- Azure calls use bounded contexts and actionable wrapped errors. + +### Task 5.3: Write Gateway programming tests + +- [ ] Test create, update, no-op, drift correction, and deletion. +- [ ] Test ownership collision and foreign-resource rejection. +- [ ] Test partial ARM failure and retry. + +**Success criteria** + +- Repeated reconciliation is idempotent. +- Foreign resources are not adopted or deleted. +- `Programmed=True` is set only after observing desired state. + +### Task 5.4: Implement AFD profile and endpoint reconciliation + +- [ ] Create one profile and endpoint per Gateway. +- [ ] Apply stable names and ownership tags. +- [ ] Publish the AFD hostname in Gateway addresses. +- [ ] Implement cleanup through the Gateway finalizer. + +**Success criteria** + +- A valid HTTP Gateway becomes programmed. +- Deletion removes only owned resources. +- Out-of-band drift in owned fields is corrected. + +## Phase 6: Implement HTTPRoute and public origins + +### Task 6.1: Write route attachment tests + +- [ ] Test parent references and listener selection. +- [ ] Test hostname intersection. +- [ ] Test allowed route namespaces. +- [ ] Test `ReferenceGrant` for cross-namespace ServiceImport. +- [ ] Test unsupported filters and matches. + +**Success criteria** + +- Route parent status matches Gateway API semantics. +- Unauthorized references set `ResolvedRefs=False`. + +### Task 6.2: Write ServiceImport backend tests + +- [ ] Test the exact Fleet group and kind. +- [ ] Test that core Kubernetes Service backends are rejected for the multi-cluster GatewayClass. +- [ ] Test port resolution. +- [ ] Test weighted logical backends and per-cluster origin weights. +- [ ] Test missing and deleted ServiceImports. +- [ ] Test references to unsupported groups and kinds. + +**Success criteria** + +- Fleet ServiceImport behavior follows the GEP-1748 routing model. +- The controller does not claim support for upstream MCS ServiceImport. + +### Task 6.3: Implement HTTPRoute controller + +- [ ] Build normalized routes from accepted Gateway parents. +- [ ] Resolve Fleet ServiceImport backends. +- [ ] Reconcile AFD origin groups, origins, routes, and supported rule sets. +- [ ] Update `Accepted` and `ResolvedRefs`. + +**Success criteria** + +- Host and path traffic routes to every valid member origin. +- Route updates do not recreate the AFD profile. +- Removing a route cleans only its owned route resources. + +### Task 6.4: Add public-origin integration tests + +- [ ] Run controllers with envtest and fake Azure clients. +- [ ] Create ServiceImport/internal export/Gateway/HTTPRoute fixtures. +- [ ] Verify desired Azure operations and final statuses. + +**Success criteria** + +- End-to-end controller reconciliation succeeds without live Azure. +- Changes to member endpoints update the origin set. + +## Phase 7: Add optional WAF attachment + +### Task 7.1: Write WAF validation and ownership tests + +- [ ] Test absent policy, valid policy ID, malformed ID, inaccessible policy, and incompatible + SKU. +- [ ] Test annotation removal and Gateway deletion. + +**Success criteria** + +- WAF policy objects are never created, modified, or deleted. +- Invalid explicit configuration never falls back to no WAF. + +### Task 7.2: Implement WAF association + +- [ ] Read and validate the existing WAF policy. +- [ ] Create or update the AFD security-policy association. +- [ ] Remove only the association when the annotation is removed. +- [ ] Reflect Azure errors in Gateway status. + +**Success criteria** + +- WAF is optional. +- The correct custom domains are associated. +- Gateway deletion leaves the external policy intact. + +## Phase 8: Add Private Link Service discovery + +### Task 8.1: Define backward-compatible internal transport + +- [ ] Select `InternalServiceExport` fields or a dedicated internal resource for PLS state. +- [ ] Define resource ID, location, provisioning state, and readiness fields. +- [ ] Define mixed-version defaulting and compatibility. + +**Success criteria** + +- Public ServiceImport API and generated CRD remain unchanged. +- Old member agents and new hub agents can coexist safely. +- Missing PLS fields mean unknown, not public. + +### Task 8.2: Write member PLS discovery tests + +- [ ] Test internal load balancer with PLS. +- [ ] Test public Service, missing PLS, provisioning PLS, replacement, and deletion. +- [ ] Test Azure authorization and transient errors. + +**Success criteria** + +- Discovery reports stable state without leaking credentials. +- PLS replacement triggers a hub update. + +### Task 8.3: Implement member PLS discovery + +- [ ] Read Service and Azure load balancer/PLS state. +- [ ] Publish PLS resource ID, location, and readiness through internal transport. +- [ ] Add required member-controller Azure permissions and documentation. + +**Success criteria** + +- Hub state converges after PLS creation, approval, replacement, and deletion. +- Public-origin behavior remains unchanged for non-PLS Services. + +## Phase 9: Add AFD Private Link origins + +### Task 9.1: Write connectivity-mode tests + +- [ ] Test `auto`, `public`, and `private-link`. +- [ ] Test all-public, all-private, mixed, incomplete, and empty origin sets. +- [ ] Test incompatible SKU and asynchronous approval. + +**Success criteria** + +- Mixed topology is rejected. +- Explicit private mode never downgrades to public. +- `auto` selects only a fully valid topology. + +### Task 9.2: Implement Private Link origin reconciliation + +- [ ] Add PLS resource ID and location to normalized origins. +- [ ] Configure AFD Private Link origins. +- [ ] Observe private endpoint connection and approval state. +- [ ] Update route and Gateway status through asynchronous transitions. + +**Success criteria** + +- A fully private ServiceImport becomes programmed. +- Pending approval is visible and retryable. +- PLS deletion removes or disables the affected origin without impacting unrelated backends. + +### Task 9.3: Add live Azure PLS end-to-end tests + +- [ ] Provision two member-cluster internal Services with PLS. +- [ ] Program an AFD Premium Gateway and HTTPRoute. +- [ ] Approve connections using the documented ownership workflow. +- [ ] Verify traffic, failover, weight changes, and cleanup. + +**Success criteria** + +- Traffic reaches both private member origins. +- No public origin path is created. +- Cleanup leaves no controller-owned AFD resources. + +## Phase 10: Add HTTPS and certificate lifecycle + +### Task 10.1: Finalize certificate design + +- [ ] Select the first supported certificate mode. +- [ ] Define listener validation and status. +- [ ] Define domain ownership validation and renewal behavior. +- [ ] Document future Key Vault/LUMA integration boundaries. + +**Success criteria** + +- Certificate identifiers do not expose secret material. +- Ownership and deletion behavior are explicit. +- The design supports safe rotation. + +### Task 10.2: Implement AFD-managed certificate support + +- [ ] Add HTTPS listener translation. +- [ ] Create and associate custom domains. +- [ ] Observe domain validation and certificate deployment. +- [ ] Publish listener status throughout provisioning. + +**Success criteria** + +- HTTPS becomes programmed only after the certificate is deployed. +- HTTP-only Gateways remain unaffected. +- Domain or certificate failure is actionable. + +## Phase 11: Package, observe, and document + +### Task 11.1: Add Helm chart and image targets + +- [ ] Add controller deployment, service account, workload identity, RBAC, webhook, service, + metrics, and configuration. +- [ ] Add Makefile, Dockerfile, and CI targets. +- [ ] Keep installation explicitly disabled by default until promoted. + +**Success criteria** + +- Chart rendering and installation tests pass. +- Controller has only required Kubernetes and Azure permissions. +- Existing controller charts remain unchanged unless shared helpers require updates. + +### Task 11.2: Add metrics and diagnostics + +- [ ] Add reconciliation latency, status, Azure operation, throttling, and resource-count metrics. +- [ ] Add structured logs keyed by Kubernetes and Azure resource identity. +- [ ] Add rate-limited events. + +**Success criteria** + +- Operators can identify validation, Kubernetes, ARM, quota, authorization, and approval failures. +- Logs do not expose secrets. + +### Task 11.3: Add examples and operational documentation + +- [ ] Add public AFD example. +- [ ] Add optional WAF example. +- [ ] Add private PLS example. +- [ ] Add required Azure roles, limits, troubleshooting, and cleanup documentation. +- [ ] Document the implementation-specific GEP-1748 compatibility statement. +- [ ] Add a GKE-to-Fleet prerequisite comparison and an environment preflight procedure. + +**Success criteria** + +- Each supported topology has a runnable example. +- Limitations and non-conformance are prominent. +- Troubleshooting maps status reasons to operator actions. + +## Phase 12: Conformance, upgrade, and release + +### Task 12.1: Run Gateway API conformance tests + +- [ ] Run conformance tests applicable to the selected multi-cluster GatewayClass feature set. +- [ ] Add implementation-specific tests for Fleet ServiceImport backend references. +- [ ] Verify core Kubernetes Service backends are rejected as documented. +- [ ] Publish supported and unsupported features. + +**Success criteria** + +- All claimed Gateway API features pass. +- Fleet ServiceImport support is tested without claiming upstream MCS Extended conformance. + +### Task 12.2: Test upgrades and deletion + +- [ ] Test controller upgrade with existing programmed Gateways. +- [ ] Test configuration-plane migration using observation-only ownership verification. +- [ ] Test fail-static behavior when ownership cannot be proven. +- [ ] Test annotation default changes are prohibited or safely versioned. +- [ ] Test member/hub mixed versions for PLS. +- [ ] Test deletion during partial Azure outages. + +**Success criteria** + +- Upgrades do not recreate stable AFD resources. +- Finalizers do not become permanently stuck on terminal authorization failures without operator + guidance. +- Mixed-version behavior matches the compatibility contract. + +### Task 12.3: Complete production-readiness review + +- [ ] Review quotas, scale, rate limits, retry budgets, and disaster recovery. +- [ ] Review identity, RBAC, WAF ownership, and private connectivity. +- [ ] Review regional and Azure control-plane failure modes. +- [ ] Define support and rollback procedures. + +**Success criteria** + +- Release criteria, rollback, monitoring, and ownership are approved. +- The feature flag can be enabled for the target environment. + +## Proposed pull request sequence + +1. **Design:** design document, implementation plan, and examples of the API contract. +2. **Foundation:** Gateway API dependency, annotation parser, normalized model, and manager skeleton. +3. **Gateway status:** GatewayClass and Gateway validation/status without Azure mutation. +4. **Public AFD:** provider clients, profile/endpoint, HTTPRoute, ServiceImport public origins. +5. **WAF:** existing policy attachment. +6. **PLS discovery:** internal transport and member discovery. +7. **Private origins:** AFD Premium Private Link reconciliation. +8. **HTTPS:** domain and AFD-managed certificate lifecycle. +9. **Packaging:** chart, permissions, observability, examples, and conformance evidence. + +Each PR must include focused unit or integration tests and must leave the repository buildable. + +## Detailed checklist + +- [ ] Phase 0: API, annotation, and AFD mapping decisions complete. +- [ ] Phase 1: Dependencies and manager skeleton complete. +- [ ] Phase 2: Typed annotation handling complete. +- [ ] Phase 3: ServiceImport origin resolution and normalized model complete. +- [ ] Phase 4: GatewayClass and Gateway reconciliation complete. +- [ ] Phase 5: Public AFD Gateway programming complete. +- [ ] Phase 6: HTTPRoute and public ServiceImport origins complete. +- [ ] Phase 7: Optional WAF attachment complete. +- [ ] Phase 8: Member PLS discovery complete. +- [ ] Phase 9: AFD Private Link origins complete. +- [ ] Phase 10: HTTPS and certificates complete. +- [ ] Phase 11: Packaging, observability, and documentation complete. +- [ ] Phase 12: Conformance and production readiness complete. + +## Overall success criteria + +- Users configure global HTTP(S) ingress with Gateway API resources. +- `HTTPRoute` references the unchanged Fleet `ServiceImport` API. +- Public, WAF-protected, and Private Link origin modes work as documented. +- Invalid annotations and unsupported Gateway features fail explicitly. +- Standard Gateway API status accurately represents Kubernetes and Azure state. +- Existing Service export/import and Traffic Manager functionality remains compatible. +- All claimed Gateway API conformance tests pass. +- Upgrade, deletion, throttling, identity, and failure scenarios are documented and tested. diff --git a/docs/howtos/gateway-api-afd-configuration.md b/docs/howtos/gateway-api-afd-configuration.md new file mode 100644 index 00000000..432a3fd4 --- /dev/null +++ b/docs/howtos/gateway-api-afd-configuration.md @@ -0,0 +1,581 @@ +# Configure Gateway API with Azure Front Door + +> [!IMPORTANT] +> This guide describes the proposed GEP-1748 implementation on the +> `rchinchani/gep-1748-gateway-api` branch. The controller is not yet released. +> The manifests define the intended user contract and will become runnable as +> the implementation phases are completed. + +This guide configures a Fleet multi-cluster HTTP application behind Azure Front +Door (AFD). It covers two independent backend topologies: + +1. [AFD with public backends](#configure-afd-with-public-backends) +2. [AFD with Private Link to internal load balancers](#configure-afd-with-private-link-to-internal-load-balancers) + +Both topologies use: + +- Gateway API resources in the Fleet hub. +- A Fleet `ServiceImport` as the only supported `HTTPRoute` backend. +- A matching `Service` and `ServiceExport` in every participating member + cluster. +- Optional attachment to an existing Azure Front Door WAF policy. + +The examples configure HTTP listeners. HTTPS and certificate lifecycle are a +later implementation phase. + +## Common prerequisites + +Before configuring either topology: + +1. Register every workload cluster as a healthy member of the same Fleet. +2. Install the Fleet hub and member networking controllers. +3. Install Gateway API `v1.2.1` CRDs in the Fleet hub. +4. Install the proposed `hub-gateway-controller-manager` in the Fleet hub. +5. Configure the controller with: + - An Azure subscription. + - A controller-wide AFD resource group. + - An approved workload or managed identity. + - Permission to manage AFD resources in the configured resource group. +6. Confirm the platform-installed GatewayClass is accepted: + + ```bash + kubectl get gatewayclass azure-fleet-afd + ``` + + Expected status: + + ```text + NAME CONTROLLER ACCEPTED + azure-fleet-afd networking.fleet.azure.com/afd True + ``` + +The examples use these placeholders: + +| Placeholder | Description | +|---|---| +| `${MEMBER_CONTEXT}` | `kubectl` context for one Fleet member cluster. | +| `${HUB_CONTEXT}` | `kubectl` context for the Fleet hub. | +| `${NAMESPACE}` | Application namespace, such as `store`. | +| `${SERVICE_NAME}` | Service and ServiceExport name, such as `store-api`. | +| `${APP_HOSTNAME}` | Public application hostname, such as `store.example.com`. | +| `${WAF_POLICY_ID}` | Optional full resource ID of an existing AFD WAF policy. | + +Repeat the member-cluster steps for every cluster that should contribute an +origin to the multi-cluster service. + +## Responsibility and resource placement + +| Configuration or action | Applied or performed by | Where | +|---|---|---| +| Install Gateway API CRDs | Fleet platform operator | Fleet hub | +| Install and configure `hub-gateway-controller-manager` | Fleet platform operator | Fleet hub | +| Create `GatewayClass/azure-fleet-afd` | Fleet platform operator, normally through the controller Helm chart | Fleet hub; cluster-scoped | +| Configure controller subscription, AFD resource group, identity, and feature flag | Fleet platform operator | Gateway controller deployment in the Fleet hub | +| Grant AFD and origin-discovery Azure permissions | Azure subscription or resource-group owner | Azure RBAC | +| Deploy application workload | Application team or Fleet placement controller | Every selected member cluster | +| Create the application `Service` and `ServiceExport` | Application team or Fleet placement controller | Every selected member cluster | +| Create the public Load Balancer | AKS cloud provider, in response to the member `Service` | Azure resources associated with each member cluster | +| Create the internal Load Balancer and PLS | AKS cloud provider, in response to the member `Service` annotations | Azure resources associated with each member cluster | +| Create and aggregate `ServiceImport` | Fleet networking controllers | Fleet hub | +| Add AFD backend annotations to `ServiceImport` | Application or networking owner | Existing generated `ServiceImport` in the Fleet hub | +| Create `Gateway` and `HTTPRoute` | Application ingress owner | Fleet hub | +| Create and maintain WAF policy | Security owner | Azure | +| Add the optional WAF policy annotation to `Gateway` | Application ingress owner after authorization from the security owner | Fleet hub | +| Create and reconcile AFD resources | Hub Gateway controller | Azure, in the configured AFD resource group | +| Approve AFD-to-PLS private endpoint connections | Network or security owner | Azure PLS for each member cluster | +| Create application DNS records | DNS owner | Authoritative public DNS zone | + +Users do not create `ServiceImport` directly. Fleet creates it from matching +member-cluster exports. Users annotate the generated hub object after it +appears. A production GitOps workflow should use a controller or patch that +waits for the generated object instead of attempting to own its full manifest. + +## Configure AFD with public backends + +In this topology, every member cluster exposes the application through a public +Azure Load Balancer. AFD connects to those public origins. + +### 1. Create a public Service in each member cluster + +**Actor:** Application team or Fleet placement controller + +**Target:** Every participating member cluster + +Use a unique Azure DNS label in each member cluster. The label gives AFD a +stable origin hostname and must be unique within the Azure region. + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: store-api + namespace: store + annotations: + service.beta.kubernetes.io/azure-dns-label-name: store-api-member-east +spec: + type: LoadBalancer + selector: + app: store-api + ports: + - name: http + protocol: TCP + port: 8080 + targetPort: 8080 +--- +apiVersion: networking.fleet.azure.com/v1alpha1 +kind: ServiceExport +metadata: + name: store-api + namespace: store +``` + +Apply the manifest to each member cluster: + +```bash +kubectl --context "${MEMBER_CONTEXT}" apply -f public-backend.yaml +``` + +Verify that the Service has a public address and that the export is valid: + +```bash +kubectl --context "${MEMBER_CONTEXT}" \ + --namespace "${NAMESPACE}" get service "${SERVICE_NAME}" + +kubectl --context "${MEMBER_CONTEXT}" \ + --namespace "${NAMESPACE}" get serviceexport "${SERVICE_NAME}" -o yaml +``` + +Do not continue until each `ServiceExport` reports `Valid=True` and +`Conflict=False`. + +### 2. Select public origin connectivity on the ServiceImport + +**Actor:** Application or networking owner + +**Target:** Generated `ServiceImport` in the Fleet hub + +Fleet creates the `ServiceImport` in the hub after at least one valid export is +observed. Keep its API unchanged and add only the AFD connectivity annotation: + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" annotate serviceimport "${SERVICE_NAME}" \ + networking.fleet.azure.com/afd-origin-connectivity=public \ + networking.fleet.azure.com/afd-health-probe-path=/healthz \ + --overwrite +``` + +The `public` value is explicit: if any member origin lacks a usable public +endpoint, the route receives `ResolvedRefs=False`. The controller must not +silently omit the invalid member or switch the backend to Private Link. + +### 3. Create the public AFD Gateway and route + +**Actor:** Application ingress owner + +**Target:** Fleet hub + +AFD Standard is sufficient for public origins. Premium is also valid. + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: store-global + namespace: store + annotations: + networking.fleet.azure.com/afd-sku: Standard_AzureFrontDoor +spec: + gatewayClassName: azure-fleet-afd + listeners: + - name: http + protocol: HTTP + port: 80 + hostname: store.example.com + allowedRoutes: + namespaces: + from: Same +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: store-api + namespace: store +spec: + parentRefs: + - name: store-global + sectionName: http + hostnames: + - store.example.com + rules: + - matches: + - path: + type: PathPrefix + value: / + backendRefs: + - group: networking.fleet.azure.com + kind: ServiceImport + name: store-api + port: 8080 + weight: 100 +``` + +Apply the resources to the hub: + +```bash +kubectl --context "${HUB_CONTEXT}" apply -f public-gateway.yaml +``` + +The Fleet group in `backendRefs` is intentional. This implementation follows +the GEP-1748 behavior while preserving the existing +`networking.fleet.azure.com/v1alpha1 ServiceImport` API. + +### 4. Optionally attach WAF to the public Gateway + +**Actors:** Security owner creates and maintains the policy; application +ingress owner attaches the approved policy + +**Targets:** WAF policy in Azure; annotation on the `Gateway` in the Fleet hub + +WAF is optional and is attached at the Gateway, not at an individual route. +The referenced policy must already exist and must be compatible with the +selected AFD SKU. + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" annotate gateway store-global \ + networking.fleet.azure.com/afd-waf-policy-id="${WAF_POLICY_ID}" \ + --overwrite +``` + +The controller creates only the AFD security-policy association. It does not +create, modify, or delete the external WAF policy. + +To remove WAF while retaining the Gateway: + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" annotate gateway store-global \ + networking.fleet.azure.com/afd-waf-policy-id- +``` + +### 5. Verify the public configuration + +**Actor:** Application ingress owner or Fleet platform operator + +**Targets:** Gateway API status in the Fleet hub and AFD resources in Azure + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" get gateway store-global -o yaml + +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" get httproute store-api -o yaml +``` + +Verify: + +- Gateway `Accepted=True`. +- Gateway `Programmed=True`. +- Listener `Programmed=True`. +- HTTPRoute parent `Accepted=True`. +- HTTPRoute parent `ResolvedRefs=True`. +- Gateway `status.addresses` contains the AFD endpoint hostname. + +Create the required public DNS record for `${APP_HOSTNAME}` only after the +Gateway reports its AFD hostname. DNS ownership and custom-domain validation +must follow the status and instructions emitted by the controller. The DNS +owner performs this step in the authoritative public DNS zone. + +## Configure AFD with Private Link to internal load balancers + +In this topology, each member application is exposed through an internal Azure +Load Balancer and an Azure Private Link Service (PLS). AFD Premium creates a +private endpoint connection to each PLS. The member Services are not exposed as +public origins. + +This is the topology intended to support SFI-NS253 because the member workloads +do not require public IP addresses. The public backend configuration in the +preceding section is not the SFI-NS253 topology. The current PR documents and +validates the contract but does not yet implement or certify the complete +Private Link data path. + +### 1. Create an internal LoadBalancer Service and PLS in each member cluster + +**Actor:** Application team or Fleet placement controller + +**Target:** Every participating member cluster + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: store-api + namespace: store + annotations: + service.beta.kubernetes.io/azure-load-balancer-internal: "true" + service.beta.kubernetes.io/azure-pls-create: "true" + service.beta.kubernetes.io/azure-pls-name: store-api-pls +spec: + type: LoadBalancer + selector: + app: store-api + ports: + - name: http + protocol: TCP + port: 8080 + targetPort: 8080 +--- +apiVersion: networking.fleet.azure.com/v1alpha1 +kind: ServiceExport +metadata: + name: store-api + namespace: store +``` + +Apply the manifest to every private-backend member cluster: + +```bash +kubectl --context "${MEMBER_CONTEXT}" apply -f private-backend.yaml +``` + +Verify that the Service address is private and the export is valid: + +```bash +kubectl --context "${MEMBER_CONTEXT}" \ + --namespace "${NAMESPACE}" get service "${SERVICE_NAME}" -o wide + +kubectl --context "${MEMBER_CONTEXT}" \ + --namespace "${NAMESPACE}" get serviceexport "${SERVICE_NAME}" -o yaml +``` + +Also verify that Azure created the PLS in the AKS node resource group: + +```bash +az network private-link-service show \ + --resource-group "${AKS_NODE_RESOURCE_GROUP}" \ + --name store-api-pls \ + --query "{id:id,provisioningState:provisioningState}" \ + --output yaml +``` + +The AKS cloud provider creates the internal load balancer and PLS. The member +networking controller then discovers the PLS resource ID, Azure location, and +readiness and transports them to the hub through internal Fleet networking +state. These fields are not added to public `ServiceImport.status`. + +### 2. Require Private Link connectivity on the ServiceImport + +**Actor:** Application or networking owner + +**Target:** Generated `ServiceImport` in the Fleet hub + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" annotate serviceimport "${SERVICE_NAME}" \ + networking.fleet.azure.com/afd-origin-connectivity=private-link \ + networking.fleet.azure.com/afd-health-probe-path=/healthz \ + --overwrite +``` + +The explicit `private-link` value prevents unsafe downgrade. If any member +origin lacks a ready PLS, the route remains unresolved rather than becoming +public. + +All origins represented by one ServiceImport must use the same connectivity +mode. Public and Private Link origins cannot be mixed in one AFD origin group. + +### 3. Create the Premium AFD Gateway and route + +**Actor:** Application ingress owner + +**Target:** Fleet hub + +Private Link origins require the Premium AFD SKU: + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: store-private-global + namespace: store + annotations: + networking.fleet.azure.com/afd-sku: Premium_AzureFrontDoor +spec: + gatewayClassName: azure-fleet-afd + listeners: + - name: http + protocol: HTTP + port: 80 + hostname: store.example.com + allowedRoutes: + namespaces: + from: Same +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: store-private-api + namespace: store +spec: + parentRefs: + - name: store-private-global + sectionName: http + hostnames: + - store.example.com + rules: + - matches: + - path: + type: PathPrefix + value: / + backendRefs: + - group: networking.fleet.azure.com + kind: ServiceImport + name: store-api + port: 8080 + weight: 100 +``` + +Apply the resources to the hub: + +```bash +kubectl --context "${HUB_CONTEXT}" apply -f private-gateway.yaml +``` + +Using `private-link` with `Standard_AzureFrontDoor` is invalid and must produce +an explicit condition rather than a fallback. + +### 4. Optionally attach WAF to the private Gateway + +**Actors:** Security owner creates and maintains the policy; application +ingress owner attaches the approved policy + +**Targets:** WAF policy in Azure; annotation on the `Gateway` in the Fleet hub + +Attach an existing AFD Premium-compatible WAF policy: + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" annotate gateway store-private-global \ + networking.fleet.azure.com/afd-waf-policy-id="${WAF_POLICY_ID}" \ + --overwrite +``` + +As with the public topology, the WAF policy lifecycle remains external to the +Gateway. Removing the annotation removes only the AFD association. + +### 5. Approve the AFD private endpoint connections + +**Actor:** Network or security owner + +**Target:** Each member cluster's PLS in Azure + +AFD Private Link approval is asynchronous. Repeat this process for the PLS in +every participating member cluster. + +Get the PLS resource ID: + +```bash +export PLS_ID=$(az network private-link-service show \ + --resource-group "${AKS_NODE_RESOURCE_GROUP}" \ + --name store-api-pls \ + --query id \ + --output tsv) +``` + +List pending private endpoint connections: + +```bash +az network private-endpoint-connection list \ + --id "${PLS_ID}" \ + --query "[?privateLinkServiceConnectionState.status=='Pending'].{name:name,id:id}" \ + --output table +``` + +Approve each connection after verifying that its request belongs to the +controller-owned AFD origin: + +```bash +az network private-endpoint-connection approve \ + --id "${PRIVATE_ENDPOINT_CONNECTION_ID}" \ + --description "Approve Fleet Gateway AFD origin" +``` + +Do not configure broad PLS auto-approval unless the platform security owner has +approved that trust boundary. + +### 6. Verify the private configuration + +**Actor:** Application ingress owner, network owner, or Fleet platform operator + +**Targets:** Gateway API status in the Fleet hub, AFD resources, and member PLS +connections in Azure + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" get gateway store-private-global -o yaml + +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" get httproute store-private-api -o yaml +``` + +During approval, the route can remain `ResolvedRefs=False` or the Gateway can +remain `Programmed=False` with a reason identifying pending Private Link +approval. After every required connection is approved, verify: + +- Gateway `Accepted=True`. +- Gateway `Programmed=True`. +- HTTPRoute parent `Accepted=True`. +- HTTPRoute parent `ResolvedRefs=True`. +- Every AFD origin uses Private Link. +- No public AFD origin was created for the ServiceImport. + +## Configure per-cluster traffic weights + +**Actor:** Application or traffic-management owner + +**Target:** `ServiceExport` in each member cluster + +For either topology, the existing ServiceExport annotation controls traffic +between member-cluster origins: + +```bash +kubectl --context "${MEMBER_CONTEXT}" \ + --namespace "${NAMESPACE}" annotate serviceexport "${SERVICE_NAME}" \ + networking.fleet.azure.com/weight=100 \ + --overwrite +``` + +Use: + +- `HTTPRoute.backendRefs[*].weight` to split traffic between logical + ServiceImport backends. +- `ServiceExport` weight to split traffic between member clusters behind one + ServiceImport. + +## Remove the configuration + +**Actors:** Application ingress owner removes hub routing; application team or +Fleet placement controller removes member resources + +**Order:** Fleet hub first, then member clusters + +Delete hub routing resources before deleting member Services: + +```bash +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" delete httproute --all + +kubectl --context "${HUB_CONTEXT}" \ + --namespace "${NAMESPACE}" delete gateway --all +``` + +Wait for Gateway finalizers to finish deleting controller-owned AFD resources. +Then delete ServiceExports and Services from each member cluster. Externally +managed WAF policies are never deleted by the Gateway controller. + +## Related design documents + +- [GEP-1748 Gateway API for Fleet Global Ingress](../design/gep-1748-gateway-api.md) +- [GEP-1748 Gateway API Implementation Plan](../design/gep-1748-implementation-plan.md) +- [Exporting Services](../concepts/ExportingService/README.md) diff --git a/go.mod b/go.mod index c698ae16..2bc0afb9 100644 --- a/go.mod +++ b/go.mod @@ -4,6 +4,8 @@ go 1.25.12 require go.goms.io/fleet v0.14.0 +require sigs.k8s.io/gateway-api v1.2.1 + require ( github.com/Azure/azure-sdk-for-go v68.0.0+incompatible // indirect github.com/Azure/azure-sdk-for-go/sdk/azcore v1.18.0 diff --git a/go.sum b/go.sum index a4654e6b..72d0435f 100644 --- a/go.sum +++ b/go.sum @@ -315,6 +315,8 @@ sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.50 h1:l9igMANNptVwYmZrqGS51oW sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.50/go.mod h1:1M90A+akyTabHVnveSKlvIO/Kk9kEr1LjRx+08twKVU= sigs.k8s.io/controller-runtime v0.19.0 h1:nWVM7aq+Il2ABxwiCizrVDSlmDcshi9llbaFbC0ji/Q= sigs.k8s.io/controller-runtime v0.19.0/go.mod h1:iRmWllt8IlaLjvTTDLhRBXIEtkCK6hwVBJJsYS9Ajf4= +sigs.k8s.io/gateway-api v1.2.1 h1:fZZ/+RyRb+Y5tGkwxFKuYuSRQHu9dZtbjenblleOLHM= +sigs.k8s.io/gateway-api v1.2.1/go.mod h1:EpNfEXNjiYfUJypf0eZ0P5iXA9ekSGWaS1WgPaM42X0= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0= sigs.k8s.io/structured-merge-diff/v4 v4.4.1 h1:150L+0vs/8DA78h1u02ooW1/fFq/Lwr+sGiqlzvrtq4= diff --git a/pkg/annotations/afd.go b/pkg/annotations/afd.go new file mode 100644 index 00000000..f229efcc --- /dev/null +++ b/pkg/annotations/afd.go @@ -0,0 +1,211 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +// Package annotations parses the Azure Front Door annotations supported by the +// Fleet Gateway controller. +package annotations + +import ( + "fmt" + "net/url" + "regexp" + "strings" + + "k8s.io/apimachinery/pkg/util/validation" +) + +const ( + annotationPrefix = "networking.fleet.azure.com/" + afdAnnotationPrefix = annotationPrefix + "afd-" + + // AFDSKUAnnotation selects the Azure Front Door SKU for a Gateway. + AFDSKUAnnotation = afdAnnotationPrefix + "sku" + // AFDResourceGroupAnnotation is reserved for a future ownership model. + AFDResourceGroupAnnotation = afdAnnotationPrefix + "resource-group" + // AFDWAFPolicyIDAnnotation attaches an existing Azure Front Door WAF policy to a Gateway. + AFDWAFPolicyIDAnnotation = afdAnnotationPrefix + "waf-policy-id" + + // AFDOriginConnectivityAnnotation selects how Azure Front Door reaches a ServiceImport. + AFDOriginConnectivityAnnotation = afdAnnotationPrefix + "origin-connectivity" + // AFDHealthProbePathAnnotation configures the origin-group HTTP health probe path. + AFDHealthProbePathAnnotation = afdAnnotationPrefix + "health-probe-path" + // AFDOriginHostHeaderAnnotation overrides the Host header sent to origins. + AFDOriginHostHeaderAnnotation = afdAnnotationPrefix + "origin-host-header" +) + +var subscriptionIDPattern = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`) + +// SKU is an Azure Front Door Standard/Premium SKU name. +type SKU string + +const ( + // SKUStandard is the Azure Front Door Standard SKU. + SKUStandard SKU = "Standard_AzureFrontDoor" + // SKUPremium is the Azure Front Door Premium SKU. + SKUPremium SKU = "Premium_AzureFrontDoor" +) + +// Connectivity selects the origin connectivity mode for every member origin +// represented by one ServiceImport. +type Connectivity string + +const ( + // ConnectivityAuto selects a fully private topology when available and otherwise a fully public topology. + ConnectivityAuto Connectivity = "auto" + // ConnectivityPublic requires all origins to expose public endpoints. + ConnectivityPublic Connectivity = "public" + // ConnectivityPrivateLink requires all origins to expose Azure Private Link Services. + ConnectivityPrivateLink Connectivity = "private-link" +) + +// GatewayConfig is the typed Azure configuration derived from Gateway annotations. +type GatewayConfig struct { + SKU SKU + WAFPolicyID string +} + +// ServiceImportConfig is the typed Azure configuration derived from ServiceImport annotations. +type ServiceImportConfig struct { + Connectivity Connectivity + HealthProbePath string + OriginHostHeader string +} + +// ParseGatewayConfig parses annotations that are valid on a Gateway. +func ParseGatewayConfig(annotations map[string]string, defaultSKU SKU) (GatewayConfig, error) { + if !defaultSKU.valid() { + return GatewayConfig{}, fmt.Errorf("default SKU %q is not supported", defaultSKU) + } + + allowed := map[string]struct{}{ + AFDSKUAnnotation: {}, + AFDResourceGroupAnnotation: {}, + AFDWAFPolicyIDAnnotation: {}, + } + if err := rejectUnsupportedAFDAnnotations(annotations, allowed); err != nil { + return GatewayConfig{}, err + } + + // Resource-group selection is intentionally controller-wide until resource + // adoption, RBAC scoping, and deletion ownership are designed together. + if value, found := annotations[AFDResourceGroupAnnotation]; found { + return GatewayConfig{}, fmt.Errorf("annotation %q with value %q is reserved and not supported", AFDResourceGroupAnnotation, value) + } + + config := GatewayConfig{SKU: defaultSKU} + if value, found := annotations[AFDSKUAnnotation]; found { + config.SKU = SKU(value) + if !config.SKU.valid() { + return GatewayConfig{}, fmt.Errorf("annotation %q has unsupported value %q; expected %q or %q", AFDSKUAnnotation, value, SKUStandard, SKUPremium) + } + } + + if value, found := annotations[AFDWAFPolicyIDAnnotation]; found { + if err := validateWAFPolicyID(value); err != nil { + return GatewayConfig{}, fmt.Errorf("annotation %q has invalid value %q: %w", AFDWAFPolicyIDAnnotation, value, err) + } + config.WAFPolicyID = value + } + return config, nil +} + +// ParseServiceImportConfig parses annotations that are valid on a Fleet ServiceImport. +func ParseServiceImportConfig(annotations map[string]string) (ServiceImportConfig, error) { + allowed := map[string]struct{}{ + AFDOriginConnectivityAnnotation: {}, + AFDHealthProbePathAnnotation: {}, + AFDOriginHostHeaderAnnotation: {}, + } + if err := rejectUnsupportedAFDAnnotations(annotations, allowed); err != nil { + return ServiceImportConfig{}, err + } + + config := ServiceImportConfig{ + Connectivity: ConnectivityAuto, + HealthProbePath: "/", + } + if value, found := annotations[AFDOriginConnectivityAnnotation]; found { + config.Connectivity = Connectivity(value) + if !config.Connectivity.valid() { + return ServiceImportConfig{}, fmt.Errorf("annotation %q has unsupported value %q; expected %q, %q, or %q", AFDOriginConnectivityAnnotation, value, ConnectivityAuto, ConnectivityPublic, ConnectivityPrivateLink) + } + } + if value, found := annotations[AFDHealthProbePathAnnotation]; found { + if err := validateHealthProbePath(value); err != nil { + return ServiceImportConfig{}, fmt.Errorf("annotation %q has invalid value %q: %w", AFDHealthProbePathAnnotation, value, err) + } + config.HealthProbePath = value + } + if value, found := annotations[AFDOriginHostHeaderAnnotation]; found { + if errs := validation.IsDNS1123Subdomain(value); len(errs) != 0 { + return ServiceImportConfig{}, fmt.Errorf("annotation %q has invalid value %q: expected a lowercase DNS hostname: %s", AFDOriginHostHeaderAnnotation, value, strings.Join(errs, "; ")) + } + config.OriginHostHeader = value + } + return config, nil +} + +// ValidateCompatibility checks settings that span a Gateway and one of its ServiceImport backends. +func ValidateCompatibility(gateway GatewayConfig, serviceImport ServiceImportConfig) error { + if serviceImport.Connectivity == ConnectivityPrivateLink && gateway.SKU != SKUPremium { + return fmt.Errorf("connectivity %q requires Azure Front Door SKU %q", ConnectivityPrivateLink, SKUPremium) + } + return nil +} + +func (s SKU) valid() bool { + return s == SKUStandard || s == SKUPremium +} + +func (c Connectivity) valid() bool { + return c == ConnectivityAuto || c == ConnectivityPublic || c == ConnectivityPrivateLink +} + +func rejectUnsupportedAFDAnnotations(annotations map[string]string, allowed map[string]struct{}) error { + for key, value := range annotations { + if !strings.HasPrefix(key, afdAnnotationPrefix) { + continue + } + if _, found := allowed[key]; !found { + // Rejecting unknown reserved keys prevents misspellings from becoming + // silent no-ops, which is the primary validation risk of annotations. + return fmt.Errorf("annotation %q with value %q is not supported on this resource", key, value) + } + } + return nil +} + +func validateHealthProbePath(value string) error { + if !strings.HasPrefix(value, "/") { + return fmt.Errorf("expected an absolute path beginning with '/'") + } + parsed, err := url.ParseRequestURI(value) + if err != nil { + return fmt.Errorf("expected a valid URI path: %w", err) + } + if parsed.IsAbs() || parsed.Host != "" || parsed.RawQuery != "" || parsed.Fragment != "" { + return fmt.Errorf("expected a path without scheme, host, query, or fragment") + } + return nil +} + +func validateWAFPolicyID(value string) error { + parts := strings.Split(strings.Trim(value, "/"), "/") + if len(parts) != 8 || + !strings.EqualFold(parts[0], "subscriptions") || + !strings.EqualFold(parts[2], "resourceGroups") || + !strings.EqualFold(parts[4], "providers") || + !strings.EqualFold(parts[5], "Microsoft.Network") || + !strings.EqualFold(parts[6], "frontdoorWebApplicationFirewallPolicies") { + return fmt.Errorf("expected an Azure Front Door WAF policy resource ID") + } + if !subscriptionIDPattern.MatchString(parts[1]) { + return fmt.Errorf("subscription ID %q is not a GUID", parts[1]) + } + if parts[3] == "" || parts[7] == "" { + return fmt.Errorf("resource group and policy name must not be empty") + } + return nil +} diff --git a/pkg/annotations/afd_test.go b/pkg/annotations/afd_test.go new file mode 100644 index 00000000..a38674a3 --- /dev/null +++ b/pkg/annotations/afd_test.go @@ -0,0 +1,271 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +package annotations + +import ( + "strings" + "testing" +) + +func TestParseGatewayConfig(t *testing.T) { + const validWAFPolicyID = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/security-rg/providers/Microsoft.Network/frontdoorWebApplicationFirewallPolicies/store-waf" + + tests := []struct { + name string + annotations map[string]string + defaultSKU SKU + want GatewayConfig + wantErr string + }{ + { + name: "uses controller default", + defaultSKU: SKUPremium, + want: GatewayConfig{ + SKU: SKUPremium, + }, + }, + { + name: "parses explicit supported values", + annotations: map[string]string{ + AFDSKUAnnotation: string(SKUStandard), + AFDWAFPolicyIDAnnotation: validWAFPolicyID, + }, + defaultSKU: SKUStandard, + want: GatewayConfig{ + SKU: SKUStandard, + WAFPolicyID: validWAFPolicyID, + }, + }, + { + name: "rejects invalid explicit SKU", + annotations: map[string]string{ + AFDSKUAnnotation: "premium", + }, + defaultSKU: SKUStandard, + wantErr: AFDSKUAnnotation, + }, + { + name: "rejects invalid controller default", + defaultSKU: SKU("invalid"), + wantErr: "default SKU", + }, + { + name: "rejects malformed WAF resource ID", + annotations: map[string]string{ + AFDWAFPolicyIDAnnotation: "/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Network/notWaf/name", + }, + defaultSKU: SKUPremium, + wantErr: AFDWAFPolicyIDAnnotation, + }, + { + name: "rejects reserved resource group override", + annotations: map[string]string{ + AFDResourceGroupAnnotation: "application-rg", + }, + defaultSKU: SKUPremium, + wantErr: AFDResourceGroupAnnotation, + }, + { + name: "rejects unknown reserved annotation", + annotations: map[string]string{ + annotationPrefix + "afd-skku": "Premium_AzureFrontDoor", + }, + defaultSKU: SKUPremium, + wantErr: annotationPrefix + "afd-skku", + }, + { + name: "rejects backend annotation on Gateway", + annotations: map[string]string{ + AFDOriginConnectivityAnnotation: string(ConnectivityPublic), + }, + defaultSKU: SKUPremium, + wantErr: AFDOriginConnectivityAnnotation, + }, + { + name: "ignores unrelated annotation", + annotations: map[string]string{ + "example.com/owner": "team-a", + }, + defaultSKU: SKUStandard, + want: GatewayConfig{ + SKU: SKUStandard, + }, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := ParseGatewayConfig(tt.annotations, tt.defaultSKU) + if tt.wantErr != "" { + if err == nil { + t.Fatalf("ParseGatewayConfig() error = nil, want containing %q", tt.wantErr) + } + if !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("ParseGatewayConfig() error = %q, want containing %q", err, tt.wantErr) + } + return + } + if err != nil { + t.Fatalf("ParseGatewayConfig() error = %v", err) + } + if got != tt.want { + t.Errorf("ParseGatewayConfig() = %#v, want %#v", got, tt.want) + } + }) + } +} + +func TestParseServiceImportConfig(t *testing.T) { + tests := []struct { + name string + annotations map[string]string + want ServiceImportConfig + wantErr string + }{ + { + name: "uses defaults", + want: ServiceImportConfig{ + Connectivity: ConnectivityAuto, + HealthProbePath: "/", + }, + }, + { + name: "parses explicit values", + annotations: map[string]string{ + AFDOriginConnectivityAnnotation: string(ConnectivityPrivateLink), + AFDHealthProbePathAnnotation: "/healthz", + AFDOriginHostHeaderAnnotation: "api.example.com", + }, + want: ServiceImportConfig{ + Connectivity: ConnectivityPrivateLink, + HealthProbePath: "/healthz", + OriginHostHeader: "api.example.com", + }, + }, + { + name: "rejects invalid connectivity", + annotations: map[string]string{ + AFDOriginConnectivityAnnotation: "private", + }, + wantErr: AFDOriginConnectivityAnnotation, + }, + { + name: "rejects relative health probe", + annotations: map[string]string{ + AFDHealthProbePathAnnotation: "healthz", + }, + wantErr: AFDHealthProbePathAnnotation, + }, + { + name: "rejects health probe query", + annotations: map[string]string{ + AFDHealthProbePathAnnotation: "/healthz?deep=true", + }, + wantErr: AFDHealthProbePathAnnotation, + }, + { + name: "rejects invalid origin host header", + annotations: map[string]string{ + AFDOriginHostHeaderAnnotation: "https://api.example.com", + }, + wantErr: AFDOriginHostHeaderAnnotation, + }, + { + name: "rejects Gateway annotation on ServiceImport", + annotations: map[string]string{ + AFDSKUAnnotation: string(SKUPremium), + }, + wantErr: AFDSKUAnnotation, + }, + { + name: "ignores existing Fleet weight annotation", + annotations: map[string]string{ + annotationPrefix + "weight": "100", + }, + want: ServiceImportConfig{ + Connectivity: ConnectivityAuto, + HealthProbePath: "/", + }, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := ParseServiceImportConfig(tt.annotations) + if tt.wantErr != "" { + if err == nil { + t.Fatalf("ParseServiceImportConfig() error = nil, want containing %q", tt.wantErr) + } + if !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("ParseServiceImportConfig() error = %q, want containing %q", err, tt.wantErr) + } + return + } + if err != nil { + t.Fatalf("ParseServiceImportConfig() error = %v", err) + } + if got != tt.want { + t.Errorf("ParseServiceImportConfig() = %#v, want %#v", got, tt.want) + } + }) + } +} + +func TestValidateCompatibility(t *testing.T) { + tests := []struct { + name string + gateway GatewayConfig + service ServiceImportConfig + wantErr string + }{ + { + name: "private link with Premium is valid", + gateway: GatewayConfig{ + SKU: SKUPremium, + }, + service: ServiceImportConfig{ + Connectivity: ConnectivityPrivateLink, + }, + }, + { + name: "private link with Standard is invalid", + gateway: GatewayConfig{ + SKU: SKUStandard, + }, + service: ServiceImportConfig{ + Connectivity: ConnectivityPrivateLink, + }, + wantErr: string(SKUPremium), + }, + { + name: "public with Standard is valid", + gateway: GatewayConfig{ + SKU: SKUStandard, + }, + service: ServiceImportConfig{ + Connectivity: ConnectivityPublic, + }, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := ValidateCompatibility(tt.gateway, tt.service) + if tt.wantErr == "" { + if err != nil { + t.Fatalf("ValidateCompatibility() error = %v", err) + } + return + } + if err == nil { + t.Fatalf("ValidateCompatibility() error = nil, want containing %q", tt.wantErr) + } + if !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("ValidateCompatibility() error = %q, want containing %q", err, tt.wantErr) + } + }) + } +} diff --git a/pkg/controllers/hub/gatewaymodel/model.go b/pkg/controllers/hub/gatewaymodel/model.go new file mode 100644 index 00000000..767ed195 --- /dev/null +++ b/pkg/controllers/hub/gatewaymodel/model.go @@ -0,0 +1,206 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +// Package gatewaymodel defines the provider-neutral desired state produced from +// Gateway API and Fleet ServiceImport resources. +package gatewaymodel + +import ( + "fmt" + "sort" +) + +const maxServiceExportWeight = 1000 +const maxHTTPRouteBackendWeight = 1000000 + +// GlobalGateway is the normalized desired state for one Gateway. +type GlobalGateway struct { + Namespace string + Name string + UID string + Listeners []Listener + Routes []Route + WAFPolicyID string +} + +// Listener is a normalized Gateway listener. +type Listener struct { + Name string + Protocol string + Port int32 + Hostname string +} + +// Route is a normalized HTTPRoute attached to a Gateway. +type Route struct { + Namespace string + Name string + Hostnames []string + Matches []HTTPMatch + Filters []HTTPFilter + Backends []Backend +} + +// HTTPMatch contains the portable HTTP match fields supported by the initial provider. +type HTTPMatch struct { + PathType string + Path string + Method string +} + +// HTTPFilter contains the provider-neutral representation of one supported HTTPRoute filter. +type HTTPFilter struct { + Type string + StatusCode int32 + Hostname string + Path string +} + +// Backend represents one Fleet ServiceImport referenced by an HTTPRoute. +type Backend struct { + Namespace string + Name string + Port int32 + // RouteWeight splits traffic between logical ServiceImport backends and + // follows the Gateway API HTTPBackendRef range [0, 1,000,000]. + RouteWeight int32 + HealthProbePath string + Origins []Origin +} + +// Origin represents one member-cluster endpoint behind a ServiceImport. +type Origin struct { + Cluster string + Endpoint string + // Weight splits traffic between member clusters behind one ServiceImport + // and follows the existing Fleet ServiceExport range [0, 1000]. + Weight int64 + Connectivity string + PrivateLinkResourceID string + PrivateLinkLocation string +} + +// Normalize validates a model and returns a deeply copied, deterministically +// ordered representation suitable for equality checks and provider reconciliation. +func Normalize(model GlobalGateway) (GlobalGateway, error) { + normalized := clone(model) + if err := validate(normalized); err != nil { + return GlobalGateway{}, err + } + + sort.Slice(normalized.Listeners, func(i, j int) bool { + return normalized.Listeners[i].Name < normalized.Listeners[j].Name + }) + sort.Slice(normalized.Routes, func(i, j int) bool { + left, right := normalized.Routes[i], normalized.Routes[j] + if left.Namespace != right.Namespace { + return left.Namespace < right.Namespace + } + return left.Name < right.Name + }) + for routeIndex := range normalized.Routes { + route := &normalized.Routes[routeIndex] + sort.Strings(route.Hostnames) + sort.Slice(route.Backends, func(i, j int) bool { + left, right := route.Backends[i], route.Backends[j] + if left.Namespace != right.Namespace { + return left.Namespace < right.Namespace + } + if left.Name != right.Name { + return left.Name < right.Name + } + return left.Port < right.Port + }) + for backendIndex := range route.Backends { + sort.Slice(route.Backends[backendIndex].Origins, func(i, j int) bool { + return route.Backends[backendIndex].Origins[i].Cluster < route.Backends[backendIndex].Origins[j].Cluster + }) + } + } + return normalized, nil +} + +func clone(model GlobalGateway) GlobalGateway { + result := model + result.Listeners = append([]Listener(nil), model.Listeners...) + result.Routes = make([]Route, len(model.Routes)) + for routeIndex := range model.Routes { + result.Routes[routeIndex] = model.Routes[routeIndex] + result.Routes[routeIndex].Hostnames = append([]string(nil), model.Routes[routeIndex].Hostnames...) + result.Routes[routeIndex].Matches = append([]HTTPMatch(nil), model.Routes[routeIndex].Matches...) + result.Routes[routeIndex].Filters = append([]HTTPFilter(nil), model.Routes[routeIndex].Filters...) + result.Routes[routeIndex].Backends = make([]Backend, len(model.Routes[routeIndex].Backends)) + for backendIndex := range model.Routes[routeIndex].Backends { + result.Routes[routeIndex].Backends[backendIndex] = model.Routes[routeIndex].Backends[backendIndex] + result.Routes[routeIndex].Backends[backendIndex].Origins = append([]Origin(nil), model.Routes[routeIndex].Backends[backendIndex].Origins...) + } + } + return result +} + +func validate(model GlobalGateway) error { + if model.Namespace == "" || model.Name == "" { + return fmt.Errorf("Gateway identity requires namespace and name") + } + + listeners := make(map[string]struct{}, len(model.Listeners)) + for _, listener := range model.Listeners { + if listener.Name == "" { + return fmt.Errorf("listener name must not be empty") + } + if _, found := listeners[listener.Name]; found { + return fmt.Errorf("duplicate listener %q", listener.Name) + } + listeners[listener.Name] = struct{}{} + if listener.Port < 1 || listener.Port > 65535 { + return fmt.Errorf("listener %q port %d is outside [1, 65535]", listener.Name, listener.Port) + } + } + + routes := make(map[string]struct{}, len(model.Routes)) + for _, route := range model.Routes { + if route.Namespace == "" || route.Name == "" { + return fmt.Errorf("route identity requires namespace and name") + } + routeKey := route.Namespace + "/" + route.Name + if _, found := routes[routeKey]; found { + return fmt.Errorf("duplicate route %q", routeKey) + } + routes[routeKey] = struct{}{} + + backends := make(map[string]struct{}, len(route.Backends)) + for _, backend := range route.Backends { + if backend.Namespace == "" || backend.Name == "" { + return fmt.Errorf("route %q backend identity requires namespace and name", routeKey) + } + backendKey := fmt.Sprintf("%s/%s:%d", backend.Namespace, backend.Name, backend.Port) + if _, found := backends[backendKey]; found { + return fmt.Errorf("route %q has duplicate backend %q", routeKey, backendKey) + } + backends[backendKey] = struct{}{} + if backend.Port < 1 || backend.Port > 65535 { + return fmt.Errorf("route %q backend %q port %d is outside [1, 65535]", routeKey, backendKey, backend.Port) + } + if backend.RouteWeight < 0 || backend.RouteWeight > maxHTTPRouteBackendWeight { + return fmt.Errorf("route %q backend %q route weight %d is outside [0, %d]", routeKey, backendKey, backend.RouteWeight, maxHTTPRouteBackendWeight) + } + + origins := make(map[string]struct{}, len(backend.Origins)) + for _, origin := range backend.Origins { + if origin.Cluster == "" { + return fmt.Errorf("backend %q origin cluster must not be empty", backendKey) + } + if _, found := origins[origin.Cluster]; found { + return fmt.Errorf("backend %q has duplicate origin cluster %q", backendKey, origin.Cluster) + } + origins[origin.Cluster] = struct{}{} + if origin.Weight < 0 || origin.Weight > maxServiceExportWeight { + return fmt.Errorf("backend %q origin %q weight %d is outside [0, %d]", backendKey, origin.Cluster, origin.Weight, maxServiceExportWeight) + } + } + } + } + return nil +} diff --git a/pkg/controllers/hub/gatewaymodel/model_test.go b/pkg/controllers/hub/gatewaymodel/model_test.go new file mode 100644 index 00000000..248418f0 --- /dev/null +++ b/pkg/controllers/hub/gatewaymodel/model_test.go @@ -0,0 +1,212 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +package gatewaymodel + +import ( + "reflect" + "strings" + "testing" +) + +func TestNormalize_SortsNestedResources(t *testing.T) { + model := GlobalGateway{ + Namespace: "store", + Name: "global", + Listeners: []Listener{ + {Name: "https", Port: 443}, + {Name: "http", Port: 80}, + }, + Routes: []Route{ + { + Namespace: "store", + Name: "z-route", + Backends: []Backend{ + { + Namespace: "store", + Name: "z-api", + Port: 8080, + Origins: []Origin{ + {Cluster: "west", Weight: 20}, + {Cluster: "east", Weight: 10}, + }, + }, + {Namespace: "store", Name: "a-api", Port: 8080}, + }, + }, + {Namespace: "store", Name: "a-route"}, + }, + } + + got, err := Normalize(model) + if err != nil { + t.Fatalf("Normalize() error = %v", err) + } + + if got.Listeners[0].Name != "http" || got.Listeners[1].Name != "https" { + t.Errorf("listener order = %v, want http then https", got.Listeners) + } + if got.Routes[0].Name != "a-route" || got.Routes[1].Name != "z-route" { + t.Errorf("route order = %v, want a-route then z-route", got.Routes) + } + if got.Routes[1].Backends[0].Name != "a-api" || got.Routes[1].Backends[1].Name != "z-api" { + t.Errorf("backend order = %v, want a-api then z-api", got.Routes[1].Backends) + } + origins := got.Routes[1].Backends[1].Origins + if origins[0].Cluster != "east" || origins[1].Cluster != "west" { + t.Errorf("origin order = %v, want east then west", origins) + } + + if reflect.DeepEqual(model, got) { + t.Error("Normalize() returned a model equal to unsorted input") + } + if model.Listeners[0].Name != "https" { + t.Error("Normalize() mutated the input model") + } +} + +func TestNormalize_RejectsInvalidModels(t *testing.T) { + tests := []struct { + name string + model GlobalGateway + wantErr string + }{ + { + name: "missing Gateway identity", + model: GlobalGateway{ + Listeners: []Listener{{Name: "http", Port: 80}}, + }, + wantErr: "Gateway identity", + }, + { + name: "duplicate listener", + model: GlobalGateway{ + Namespace: "store", + Name: "global", + Listeners: []Listener{ + {Name: "http", Port: 80}, + {Name: "http", Port: 8080}, + }, + }, + wantErr: "duplicate listener", + }, + { + name: "duplicate route", + model: GlobalGateway{ + Namespace: "store", + Name: "global", + Routes: []Route{ + {Namespace: "store", Name: "api"}, + {Namespace: "store", Name: "api"}, + }, + }, + wantErr: "duplicate route", + }, + { + name: "duplicate backend", + model: GlobalGateway{ + Namespace: "store", + Name: "global", + Routes: []Route{{ + Namespace: "store", + Name: "api", + Backends: []Backend{ + {Namespace: "store", Name: "backend", Port: 8080}, + {Namespace: "store", Name: "backend", Port: 8080}, + }, + }}, + }, + wantErr: "duplicate backend", + }, + { + name: "invalid backend port", + model: GlobalGateway{ + Namespace: "store", + Name: "global", + Routes: []Route{{ + Namespace: "store", + Name: "api", + Backends: []Backend{{ + Namespace: "store", + Name: "backend", + Port: 70000, + }}, + }}, + }, + wantErr: "port", + }, + { + name: "invalid route weight", + model: GlobalGateway{ + Namespace: "store", + Name: "global", + Routes: []Route{{ + Namespace: "store", + Name: "api", + Backends: []Backend{{ + Namespace: "store", + Name: "backend", + Port: 8080, + RouteWeight: 1000001, + }}, + }}, + }, + wantErr: "route weight", + }, + { + name: "duplicate origin cluster", + model: GlobalGateway{ + Namespace: "store", + Name: "global", + Routes: []Route{{ + Namespace: "store", + Name: "api", + Backends: []Backend{{ + Namespace: "store", + Name: "backend", + Port: 8080, + Origins: []Origin{ + {Cluster: "east", Weight: 1}, + {Cluster: "east", Weight: 2}, + }, + }}, + }}, + }, + wantErr: "duplicate origin", + }, + { + name: "invalid origin weight", + model: GlobalGateway{ + Namespace: "store", + Name: "global", + Routes: []Route{{ + Namespace: "store", + Name: "api", + Backends: []Backend{{ + Namespace: "store", + Name: "backend", + Port: 8080, + Origins: []Origin{ + {Cluster: "east", Weight: 1001}, + }, + }}, + }}, + }, + wantErr: "weight", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + _, err := Normalize(tt.model) + if err == nil { + t.Fatalf("Normalize() error = nil, want containing %q", tt.wantErr) + } + if !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("Normalize() error = %q, want containing %q", err, tt.wantErr) + } + }) + } +}