diff --git a/CHANGELOG.md b/CHANGELOG.md index b5b8bfc50..805e2ccb7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,70 @@ # Changelog +## Release PR [#1485](https://github.com/cobaltcore-dev/cortex/pull/1485) — 2026-11-05 + +### Bumped Versions + +| Package | Previous | New | +|---|---|---| +| cortex | 0.7.3 | 0.8.0 | +| cortex-postgres | 0.6.30 | 0.6.31 | +| cortex-shim | 0.1.35 | 0.1.36 | +| bundles | 0.0.110 | 0.0.111 | +| cortex-placement-shim | 0.1.35 | 0.1.36 | + +### ⚠️ Breaking Changes + +- **`Decision` CRD removed:** The `cortex.cloud_decisions` custom resource definition has been fully deleted (type file, CRD YAML, all controller references). Any tooling using `kubectl get decisions` or importing the Decision type must be removed. Decision state KPI metrics for Cinder, Manila, and Nova are no longer emitted. +- **`CommittedResource` CRD schema change** ([#1438](https://github.com/cobaltcore-dev/cortex/pull/1438)) — Allocation tracking has moved from `Spec.CommittedResourceReservation.Allocations` to `Status.CommittedResourceReservation.Allocation`. Billing fields have been consolidated into a new `Status.Billing` sub-struct (`AssignedInstances`, `UsedResources`, `LastReconcileAt`, `ObservedGeneration`). The flat billing fields on `CommittedResourceStatus` (`AssignedInstances`, `UsedResources`, `LastUsageReconcileAt`, `UsageObservedGeneration`) are retained but deprecated. Generation tracking has been migrated from `ParentGeneration`/`ObservedParentGeneration` to standard Kubernetes conditions with `ObservedGeneration` signaling. +- **`Reservation` CRD schema expanded** ([#1438](https://github.com/cobaltcore-dev/cortex/pull/1438)) — New `Status.CommittedResourceReservation.Allocation` field replaces spec-level allocations. `PrimaryResource` field added to `CommittedResourceReservationSpec`. `CreationTimestamp` on `CommittedResourceAllocation` is deprecated. Resource keys are normalised to canonical `hv1.ResourceName` constants. +- **Pipeline CRD trimmed** — Several `spec` fields have been removed from the Pipeline CRD. +- **Helm values** — New required key `kvm.crStateFilter` in `cortex-nova/values.yaml` (defaults to `["confirmed"]`). Removed KPI alert templates for Cinder/Manila/Nova decision-state metrics. + +### Bug Fixes + +- **Quota: unify CR state filter to confirmed-only default** ([#1461](https://github.com/cobaltcore-dev/cortex/pull/1461)) — Aligned `FilterQuotaEnforcement` and the quota controller to count only `confirmed` CRs toward capacity, dropping `guaranteed` from the default filter. This prepares for the future removal of the guaranteed state. + +- **Move CR reservation allocations from spec to status** ([#1438](https://github.com/cobaltcore-dev/cortex/pull/1438)) — Major CRD refactor: allocation tracking moved to `Status.Allocation`, allocation accounting rewritten for multi-slot spanning, grace period replaced with live `VMSource.GetVM()` lookup, and billing fields grouped into `Status.Billing`. + +- **Fix location of flavor group client** ([#1481](https://github.com/cobaltcore-dev/cortex/pull/1481)) — Corrected the import path for the flavor group client after the Nova API directory reorganisation. + +- **Polish failover host controller & fix probe reservation handling** ([#1484](https://github.com/cobaltcore-dev/cortex/pull/1484)) — The path finder now ignores *all* reservation types during capacity probes (not just failover), so capacity is no longer understated when probing. Alert window bumped to 60 minutes; partition descriptions now show the full flavor list; progress logging added for long passes. + +- **Don't enforce image properties when unset in compute pipeline** ([#1483](https://github.com/cobaltcore-dev/cortex/pull/1483)) — Boot-from-volume VMs no longer cause spurious image property alignment errors. Image property checks are now only applied when an image is actually specified, simplifying failover scheduling calls. + +### Chores & Refactoring + +- **Clean up scheduling pipeline** ([#1464](https://github.com/cobaltcore-dev/cortex/pull/1464)) — General scheduling pipeline code clean-up: the `Decision` type is fully removed, test helpers are deduplicated, and pipeline controller signatures are simplified to use the new `compute.Request`/`compute.Placement` types. + +- **Drive CR state filter from a single Helm value** ([#1472](https://github.com/cobaltcore-dev/cortex/pull/1472)) — Eliminated duplicated Go-level defaults for the CR state filter. `kvm.crStateFilter` is now the single source of truth, wired into both the quota controller and the enforcement filter via a YAML anchor. + +- **Remove deprecated failover package** ([#1479](https://github.com/cobaltcore-dev/cortex/pull/1479)) — Removed the deprecated failover package as a dependency from the failover host controller; reuses shared message-parsing code for pipeline name inference. + +- **Move Nova-specific types into `api/external/nova/`** ([#1480](https://github.com/cobaltcore-dev/cortex/pull/1480)) — Introduced `placement_types.go` (typed `FlavorSpec`, `Capability`, `HypervisorFamily`, `RequestedDestination`, `InstanceGroupSpec`, and trait/capability parsing helpers) and `messages.go` (Nova wire types, intent constants, request/response structs) with full test coverage. This separates Nova wire-format types from the internal scheduling API. + +### Dependency Updates + +| Dependency | Old | New | +|---|---|---| +| `kube-prometheus-stack` | — | **v92.17.1** | +| `github.com/sapcc/go-bits` | — | digest `0cf0269` | +| `github.com/cobaltcore-dev/openstack-hypervisor-operator` | — | **v1.4.0** | +| External dependencies | — | batch update | +| `debian:trixie-slim` | — | digest `a72b443` | + +### ⚠️ Operational Notes + +1. **CRD migration — `cortex.cloud_decisions` deleted:** The Decision CRD YAML has been removed from the Helm CRDs folder. Ensure any existing Decision resources are cleaned up before or during upgrade. If your CI/CD pipeline applies CRDs separately, remove the old `cortex.cloud_decisions` CRD manifest. +2. **CRD migration — `cortex.cloud_committedresources`:** The new `Status.Allocation` and `Status.Billing` fields will be empty after upgrade until the controllers reconcile. The deprecated flat billing fields are still populated during a transition period — migrate consumers to `Status.Billing` before the next major release. +3. **CRD migration — `cortex.cloud_reservations`:** Spec-level `Allocations` are deprecated; the controller now writes to `Status.CommittedResourceReservation.Allocation`. Existing spec-level allocations will be migrated automatically. +4. **Helm values — `kvm.crStateFilter`:** A new required key. The default is `["confirmed"]`, which changes behavior if your deployment previously counted `guaranteed` CRs toward quota. Review quota behavior in staging before rollout. +5. **Dashboard/alert updates:** KPI templates for Cinder, Manila, and Nova decision-state metrics have been removed. Remove any dashboards or alert rules referencing these metrics. +6. **Major version bump (0.7.x → 0.8.0):** This release carries a minor version bump reflecting the breaking CRD schema changes and Decision type removal. + +**Contributors:** @SoWieMarkus, @PhilippMatthes, @mblos, @renovate[bot], @cortex-ai-agents[bot] + +**PR:** [#1485](https://github.com/cobaltcore-dev/cortex/pull/1485) | **Commits:** 14 | **Files changed:** 100 | **+3,728 / −7,181** + ## Release PR [#1458](https://github.com/cobaltcore-dev/cortex/pull/1458) — 2026-11-02 ### Bumped Versions diff --git a/api/external/nova/messages.go b/api/external/nova/messages.go index 33ed4c66f..a6c41c249 100644 --- a/api/external/nova/messages.go +++ b/api/external/nova/messages.go @@ -165,9 +165,19 @@ const ( FlavorTypeHANA FlavorType = "hana" ) -// GetFlavorType determines the flavor type based on the requested flavor's extra specs. -func (r ExternalSchedulerRequest) GetFlavorType() (FlavorType, error) { - extraSpecs := r.Spec.Data.Flavor.Data.ExtraSpecs +// FlavorTypeFromExtraSpecs determines the flavor type from a flavor's extra specs. +// +// The trait:CUSTOM_HANA_EXCLUSIVE_HOST extra spec is the canonical signal that a +// flavor targets HANA workloads: when set to "required" the VM must land on a +// HANA-exclusive host, so the flavor is a HANA flavor; when set to "forbidden" (or +// absent) it is a general-purpose flavor. The value is compared case-insensitively. +// Any other explicit value is rejected as an error, since it indicates a malformed +// flavor rather than a type we can safely default. +// +// Callers that only hold the raw extra specs map (e.g. failover reservation +// scheduling) should use this helper instead of re-implementing the trait check, so +// the HANA-detection logic stays in one place. +func FlavorTypeFromExtraSpecs(extraSpecs map[string]string) (FlavorType, error) { val, ok := extraSpecs["trait:CUSTOM_HANA_EXCLUSIVE_HOST"] if !ok { return FlavorTypeGeneralPurpose, nil @@ -183,6 +193,11 @@ func (r ExternalSchedulerRequest) GetFlavorType() (FlavorType, error) { } } +// GetFlavorType determines the flavor type based on the requested flavor's extra specs. +func (r ExternalSchedulerRequest) GetFlavorType() (FlavorType, error) { + return FlavorTypeFromExtraSpecs(r.Spec.Data.Flavor.Data.ExtraSpecs) +} + type HypervisorType string const ( diff --git a/api/external/nova/messages_test.go b/api/external/nova/messages_test.go index b76cddf03..9f02d583c 100644 --- a/api/external/nova/messages_test.go +++ b/api/external/nova/messages_test.go @@ -422,6 +422,39 @@ func TestGetFlavorType(t *testing.T) { } } +func TestFlavorTypeFromExtraSpecs(t *testing.T) { + tests := []struct { + name string + extraSpecs map[string]string + expected FlavorType + expectError bool + }{ + {name: "hana required", extraSpecs: map[string]string{"trait:CUSTOM_HANA_EXCLUSIVE_HOST": "required"}, expected: FlavorTypeHANA}, + {name: "hana required mixed case", extraSpecs: map[string]string{"trait:CUSTOM_HANA_EXCLUSIVE_HOST": "Required"}, expected: FlavorTypeHANA}, + {name: "general purpose forbidden", extraSpecs: map[string]string{"trait:CUSTOM_HANA_EXCLUSIVE_HOST": "forbidden"}, expected: FlavorTypeGeneralPurpose}, + {name: "general purpose trait absent", extraSpecs: map[string]string{}, expected: FlavorTypeGeneralPurpose}, + {name: "general purpose nil specs", extraSpecs: nil, expected: FlavorTypeGeneralPurpose}, + {name: "unsupported value", extraSpecs: map[string]string{"trait:CUSTOM_HANA_EXCLUSIVE_HOST": "optional"}, expectError: true}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := FlavorTypeFromExtraSpecs(tt.extraSpecs) + if tt.expectError { + if err == nil { + t.Fatalf("expected error but got none") + } + return + } + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if got != tt.expected { + t.Errorf("expected %q, got %q", tt.expected, got) + } + }) + } +} + func TestNovaSpecUnmarshal(t *testing.T) { var jsonData = `{ "spec": { diff --git a/api/external/nova/placement_types.go b/api/external/nova/placement_types.go new file mode 100644 index 000000000..e24eea4ba --- /dev/null +++ b/api/external/nova/placement_types.go @@ -0,0 +1,195 @@ +// Copyright SAP SE +// SPDX-License-Identifier: Apache-2.0 + +package api + +import "strings" + +// HypervisorFamily is the coarse hypervisor family a request targets. It is the +// distinction the pipeline and request routing care about. The adapter resolves +// the family from Nova's "capabilities:hypervisor_type" flavor extra spec. +type HypervisorFamily string + +const ( + // HypervisorFamilyKVM covers KVM-based hypervisors (QEMU and CH). + HypervisorFamilyKVM HypervisorFamily = "kvm" + // HypervisorFamilyVMware covers VMware hypervisors. + HypervisorFamilyVMware HypervisorFamily = "vmware" +) + +// ImageHypervisorType is the hypervisor type declared by the image. The adapter +// resolves it from Nova's image properties ("img_hv_type"/"hypervisor_type"). +type ImageHypervisorType string + +const ( + // ImageHypervisorTypeVMware indicates a VMware image. + ImageHypervisorTypeVMware ImageHypervisorType = "vmware" + // ImageHypervisorTypeBaremetal indicates a baremetal image. + ImageHypervisorTypeBaremetal ImageHypervisorType = "baremetal" + // ImageHypervisorTypeKVM indicates a KVM image. + ImageHypervisorTypeKVM ImageHypervisorType = "kvm" + // ImageHypervisorTypeUnset indicates no image hypervisor type was declared. + ImageHypervisorTypeUnset ImageHypervisorType = "" + // ImageHypervisorTypeUnparseable indicates an image hypervisor type was declared but could not be parsed. + ImageHypervisorTypeUnparseable ImageHypervisorType = "unparseable" +) + +// Capability is a single required hypervisor capability as a typed key/value +// equality, e.g. {Name: "cpu_arch", Value: "x86_64"}. The adapter resolves these +// from Nova's "capabilities:" flavor extra specs and rejects any entry that +// uses an unsupported comparison operator before it reaches the pipeline. +type Capability struct { + // Name is the capability key without the "capabilities:" prefix. + Name string + // Value is the required exact value. + Value string +} + +// FlavorSpec is the typed, Nova-agnostic description of the requested flavor. +// It carries no extra-spec map: the adapter resolves every extra spec the +// pipeline needs into the typed fields on this struct and on PlacementRequest. +type FlavorSpec struct { + // MemoryMB is the requested memory per instance. 0 means a zero-memory + // instance to capacity/bin-pack steps (fits anywhere), so the adapter sets it. + MemoryMB uint64 + // VCPUs is the requested vCPUs per instance. 0 means a zero-CPU instance to + // capacity/bin-pack steps (fits anywhere), so the adapter sets it. + VCPUs uint64 +} + +// RequestedDestination mirrors the subset of Nova's requested_destination that +// the pipeline reads. A nil *RequestedDestination on the request means no +// destination constraint was requested. +type RequestedDestination struct { + // Host is the requested target host. Empty means no host was pinned (only an + // aggregate constraint applies). + Host string + // Aggregates are the host-aggregate names the destination must belong to. + // Empty/nil means no aggregate constraint. + Aggregates []string +} + +// InstanceGroupSpec is the typed subset of a Nova server group the pipeline +// reads. A nil *InstanceGroupSpec on the request means the instance is not part +// of a server group, so group affinity/anti-affinity filters are skipped. +type InstanceGroupSpec struct { + // UUID identifies the server group. Empty if unknown. + UUID string + // Policy is the group policy (e.g. "anti-affinity", "soft-affinity"). Empty + // means no policy-specific filter/weigher applies. + Policy string + // Hosts are the hosts already occupied by members of this group. Empty/nil + // means the group currently occupies no hosts. + Hosts []string + // Members are the instance UUIDs already in the group. Empty/nil means the + // group has no members yet, so anti-affinity has nothing to exclude. + Members []string + // MaxServerPerHost caps how many group members may share a host. 0 means + // unset, which the anti-affinity filter treats as the Nova default of 1. + MaxServerPerHost int +} + +// The helpers below resolve Nova's loosely-typed flavor extra-spec string +// conventions into the typed values the scheduling pipeline consumes. They live +// here, alongside the typed values they produce, because the +// "trait:"/"capabilities:"/"accel:" key conventions are Nova's — not the +// pipeline's. Both the Nova spec adapter and the in-process reservation +// controllers (which hold a flavor's extra-spec map directly) call them, so the +// conventions are understood in exactly one place. + +// unsupportedCapabilityOps are the Nova extra-spec comparison operators the +// capabilities filter does not implement. A capability value containing any of +// these means the request uses an advanced match the pipeline can't honor, so +// SplitCapabilities drops capabilities entirely (the filter then becomes a +// no-op), matching Nova's "skip" behavior. +// See: https://github.com/sapcc/nova/blob/3ebf80/nova/scheduler/filters/extra_specs_ops.py#L23 +var unsupportedCapabilityOps = []string{ + "=", "", "", "==", "!=", ">=", "<=", + "s==", "s!=", "s<", "s<=", "s>", "s>=", "", +} + +// SplitTraits splits a flavor's "trait:" extra specs into required and +// forbidden trait names, classified by the "required"/"forbidden" value. +func SplitTraits(extraSpecs map[string]string) (required, forbidden []string) { + for key, value := range extraSpecs { + name, ok := strings.CutPrefix(key, "trait:") + if !ok { + continue + } + switch value { + case "required": + required = append(required, name) + case "forbidden": + forbidden = append(forbidden, name) + } + } + return required, forbidden +} + +// SplitCapabilities extracts a flavor's "capabilities:" extra specs as typed +// Capability equalities (prefix stripped). The "hypervisor_type" value is +// canonicalized (qemu->QEMU, ch->CH) so it matches what hypervisors advertise +// (filter_capabilities compares case-sensitively); all other values pass through +// unchanged. If any value uses an unsupported comparison operator it returns nil so +// the capabilities filter is skipped entirely (matching Nova behavior). +func SplitCapabilities(extraSpecs map[string]string) []Capability { + var caps []Capability + for key, value := range extraSpecs { + name, ok := strings.CutPrefix(key, "capabilities:") + if !ok { + continue + } + for _, op := range unsupportedCapabilityOps { + if strings.Contains(value, op) { + return nil // Unsupported operator: skip the whole filter. + } + } + if name == "hypervisor_type" { + value = CanonicalHypervisorType(value) + } + caps = append(caps, Capability{Name: name, Value: value}) + } + return caps +} + +// CanonicalHypervisorType normalizes a flavor's capabilities:hypervisor_type value +// to the casing hypervisors advertise (QEMU/CH), since filter_capabilities compares +// values case-sensitively. Unrecognized values pass through unchanged. +func CanonicalHypervisorType(value string) string { + switch strings.ToLower(value) { + case "qemu": + return "QEMU" + case "ch": + return "CH" + default: + return value + } +} + +// RequestsAccelerators reports whether a flavor requests a hardware accelerator, +// resolved from the presence of the "accel:device_profile" extra spec. +func RequestsAccelerators(extraSpecs map[string]string) bool { + _, ok := extraSpecs["accel:device_profile"] + return ok +} + +// HypervisorTypeCapability returns the required (key=value) hypervisor capabilities +// for a reservation scheduler request, so filter_capabilities keeps a workload off +// incompatible hosts. The reservation controllers build the typed placement request +// directly (not via the Nova wire adapter), yet their flavors still carry Nova's +// extra-spec string conventions, so they reuse SplitCapabilities here and layer the +// reservation-specific defaultQEMU seeding on top. It extracts ALL of the flavor's +// "capabilities:" extra specs (canonicalizing the hypervisor_type casing and +// dropping everything on an unsupported operator) — not just hypervisor_type. +// +// When the flavor names no hypervisor type and defaultQEMU is set (the failover +// controller seeds from running VMs whose flavor may omit it), the canonical "QEMU" +// hypervisor_type is added. Without defaultQEMU, a missing hypervisor type adds no +// hypervisor_type constraint. +func HypervisorTypeCapability(extraSpecs map[string]string, defaultQEMU bool) []Capability { + caps := SplitCapabilities(extraSpecs) + if extraSpecs["capabilities:hypervisor_type"] == "" && defaultQEMU { + caps = append(caps, Capability{Name: "hypervisor_type", Value: "QEMU"}) + } + return caps +} diff --git a/api/external/nova/placement_types_test.go b/api/external/nova/placement_types_test.go new file mode 100644 index 000000000..335e90a93 --- /dev/null +++ b/api/external/nova/placement_types_test.go @@ -0,0 +1,146 @@ +// Copyright SAP SE +// SPDX-License-Identifier: Apache-2.0 + +package api + +import ( + "slices" + "strings" + "testing" +) + +func TestSplitTraits(t *testing.T) { + required, forbidden := SplitTraits(map[string]string{ + "trait:CUSTOM_HANA_EXCLUSIVE_HOST": "required", + "trait:CUSTOM_NO_GPU": "forbidden", + "trait:IGNORED": "preferred", // neither required nor forbidden + "capabilities:hypervisor_type": "qemu", // not a trait + }) + if !slices.Equal(required, []string{"CUSTOM_HANA_EXCLUSIVE_HOST"}) { + t.Errorf("required = %v, want [CUSTOM_HANA_EXCLUSIVE_HOST]", required) + } + if !slices.Equal(forbidden, []string{"CUSTOM_NO_GPU"}) { + t.Errorf("forbidden = %v, want [CUSTOM_NO_GPU]", forbidden) + } +} + +func TestSplitCapabilities(t *testing.T) { + sortCaps := func(c []Capability) { + slices.SortFunc(c, func(a, b Capability) int { return strings.Compare(a.Name, b.Name) }) + } + cases := []struct { + name string + extraSpecs map[string]string + want []Capability + }{ + { + name: "canonicalizes lowercase hypervisor_type to match hosts", + extraSpecs: map[string]string{"capabilities:hypervisor_type": "qemu"}, + want: []Capability{{Name: "hypervisor_type", Value: "QEMU"}}, + }, + { + name: "ch canonicalized", + extraSpecs: map[string]string{"capabilities:hypervisor_type": "ch"}, + want: []Capability{{Name: "hypervisor_type", Value: "CH"}}, + }, + { + name: "non-hypervisor_type capabilities preserved verbatim", + extraSpecs: map[string]string{ + "capabilities:hypervisor_type": "QEMU", + "capabilities:cpu_arch": "aarch64", + }, + want: []Capability{ + {Name: "cpu_arch", Value: "aarch64"}, + {Name: "hypervisor_type", Value: "QEMU"}, + }, + }, + { + name: "unsupported operator drops all capabilities", + extraSpecs: map[string]string{ + "capabilities:hypervisor_type": "QEMU", + "capabilities:cpu_arch": " aarch64 x86_64", + }, + want: nil, + }, + { + name: "non-capability extra specs ignored", + extraSpecs: map[string]string{"trait:CUSTOM_X": "required", "quota:hw_version": "2101"}, + want: nil, + }, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + got := SplitCapabilities(c.extraSpecs) + sortCaps(got) + if !slices.Equal(got, c.want) { + t.Errorf("got %v, want %v", got, c.want) + } + }) + } +} + +func TestRequestsAccelerators(t *testing.T) { + if !RequestsAccelerators(map[string]string{"accel:device_profile": "gpu-v100"}) { + t.Error("expected accelerators requested when accel:device_profile is present") + } + if RequestsAccelerators(map[string]string{"capabilities:cpu_arch": "x86_64"}) { + t.Error("expected no accelerators requested without accel:device_profile") + } +} + +func TestHypervisorTypeCapability(t *testing.T) { + tests := []struct { + name string + extraSpecs map[string]string + defaultQEMU bool + want []Capability + }{ + { + name: "value passed through", + extraSpecs: map[string]string{"capabilities:hypervisor_type": "CH"}, + want: []Capability{{Name: "hypervisor_type", Value: "CH"}}, + }, + { + name: "lowercase value canonicalized to match hosts", + extraSpecs: map[string]string{"capabilities:hypervisor_type": "qemu"}, + want: []Capability{{Name: "hypervisor_type", Value: "QEMU"}}, + }, + { + name: "missing without default -> nil", + extraSpecs: map[string]string{}, + want: nil, + }, + { + name: "missing with default -> canonical QEMU", + extraSpecs: map[string]string{}, + defaultQEMU: true, + want: []Capability{{Name: "hypervisor_type", Value: "QEMU"}}, + }, + { + name: "explicit value wins over default", + extraSpecs: map[string]string{"capabilities:hypervisor_type": "CH"}, + defaultQEMU: true, + want: []Capability{{Name: "hypervisor_type", Value: "CH"}}, + }, + { + name: "non-hypervisor_type capabilities preserved", + extraSpecs: map[string]string{ + "capabilities:hypervisor_type": "qemu", + "capabilities:cpu_arch": "aarch64", + }, + want: []Capability{ + {Name: "cpu_arch", Value: "aarch64"}, + {Name: "hypervisor_type", Value: "QEMU"}, + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := HypervisorTypeCapability(tt.extraSpecs, tt.defaultQEMU) + slices.SortFunc(got, func(a, b Capability) int { return strings.Compare(a.Name, b.Name) }) + if !slices.Equal(got, tt.want) { + t.Errorf("got %v, want %v", got, tt.want) + } + }) + } +} diff --git a/api/v1alpha1/committed_resource_types.go b/api/v1alpha1/committed_resource_types.go index 2443dce80..f6b2b2b3e 100644 --- a/api/v1alpha1/committed_resource_types.go +++ b/api/v1alpha1/committed_resource_types.go @@ -104,6 +104,19 @@ type CommittedResourceSpec struct { AllowRejection bool `json:"allowRejection,omitempty"` } +// BillingStatus holds the computed billing assignment for a CommittedResource. +// Written by the UsageReconciler. +type BillingStatus struct { + // AssignedInstances lists the VM UUIDs assigned to this commitment for billing. + AssignedInstances []string `json:"assignedInstances,omitempty"` + // UsedResources is the total resource consumption from assigned instances. + UsedResources map[string]resource.Quantity `json:"usedResources,omitempty"` + // LastReconcileAt is the time of the last successful billing reconcile. + LastReconcileAt *metav1.Time `json:"lastReconcileAt,omitempty"` + // ObservedGeneration is the CommittedResource generation seen during the last billing reconcile. + ObservedGeneration *int64 `json:"observedGeneration,omitempty"` +} + // CommittedResourceStatus defines the observed state of CommittedResource. type CommittedResourceStatus struct { // AcceptedSpec is a snapshot of Spec from the last successful reconcile. @@ -120,20 +133,29 @@ type CommittedResourceStatus struct { // +kubebuilder:validation:Optional LastReconcileAt *metav1.Time `json:"lastReconcileAt,omitempty"` + // Billing holds the computed billing assignment for this commitment. + // Written by the UsageReconciler. Use this instead of the deprecated flat billing fields. + // +kubebuilder:validation:Optional + Billing *BillingStatus `json:"billing,omitempty"` + + // Deprecated: use Status.Billing.AssignedInstances // AssignedInstances holds the UUIDs of VM instances deterministically assigned to this committed resource. // Populated by the usage reconciler; used to compute UsedResources and drive the quota controller. // +kubebuilder:validation:Optional AssignedInstances []string `json:"assignedInstances,omitempty"` + // Deprecated: use Status.Billing.UsedResources // UsedResources is the total resource consumption of assigned VM instances, keyed by resource type // (e.g. "memory" in MiB binary SI, "cpu" as core count). Populated by the usage reconciler. // +kubebuilder:validation:Optional UsedResources map[string]resource.Quantity `json:"usedResources,omitempty"` + // Deprecated: use Status.Billing.LastReconcileAt // LastUsageReconcileAt is when the usage reconciler last updated AssignedInstances and UsedResources. // +kubebuilder:validation:Optional LastUsageReconcileAt *metav1.Time `json:"lastUsageReconcileAt,omitempty"` + // Deprecated: use Status.Billing.ObservedGeneration // UsageObservedGeneration is the CR generation that the usage reconciler last processed. // Follows the Kubernetes observedGeneration pattern: when this differs from // metadata.generation the cooldown is bypassed so spec changes (e.g. shrink) are reflected diff --git a/api/v1alpha1/decision_types.go b/api/v1alpha1/decision_types.go deleted file mode 100644 index fb1e07ca9..000000000 --- a/api/v1alpha1/decision_types.go +++ /dev/null @@ -1,160 +0,0 @@ -// Copyright SAP SE -// SPDX-License-Identifier: Apache-2.0 - -package v1alpha1 - -import ( - corev1 "k8s.io/api/core/v1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - runtime "k8s.io/apimachinery/pkg/runtime" -) - -// SchedulingIntent defines the intent of a scheduling decision. -type SchedulingIntent string - -// Other intents can be defined by the operators. -const ( - // Used as default intent if the operator does not specify one. - SchedulingIntentUnknown SchedulingIntent = "Unknown" -) - -type DecisionSpec struct { - // SchedulingDomain defines in which scheduling domain this decision - // was or is processed (e.g., nova, cinder, manila). - SchedulingDomain SchedulingDomain `json:"schedulingDomain"` - - // A reference to the pipeline that should be used for this decision. - // This reference can be used to look up the pipeline definition and its - // scheduler step configuration for additional context. - PipelineRef corev1.ObjectReference `json:"pipelineRef"` - - // An identifier for the underlying resource to be scheduled. - // For example, this can be the UUID of a nova instance or cinder volume. - // This can be used to correlate multiple decisions for the same resource. - ResourceID string `json:"resourceID"` - - // If the type is "nova", this field contains the raw nova decision request. - // +kubebuilder:validation:Optional - NovaRaw *runtime.RawExtension `json:"novaRaw,omitempty"` - // If the type is "cinder", this field contains the raw cinder decision request. - // +kubebuilder:validation:Optional - CinderRaw *runtime.RawExtension `json:"cinderRaw,omitempty"` - // If the type is "manila", this field contains the raw manila decision request. - // +kubebuilder:validation:Optional - ManilaRaw *runtime.RawExtension `json:"manilaRaw,omitempty"` - // If the type is "machine", this field contains the machine reference. - // +kubebuilder:validation:Optional - MachineRef *corev1.ObjectReference `json:"machineRef,omitempty"` - // If the type is "pod", this field contains the pod reference. - // +kubebuilder:validation:Optional - PodRef *corev1.ObjectReference `json:"podRef,omitempty"` - - // The intent of the scheduling decision (e.g., initial scheduling, rescheduling, etc.). - // +kubebuilder:validation:Optional - Intent SchedulingIntent `json:"intent,omitempty"` -} - -type StepResult struct { - // object reference to the scheduler step. - StepName string `json:"stepName"` - // Activations of the step for each host. - Activations map[string]float64 `json:"activations"` -} - -type DecisionResult struct { - // Raw input weights to the pipeline. - // +kubebuilder:validation:Optional - RawInWeights map[string]float64 `json:"rawInWeights"` - // Normalized input weights to the pipeline. - // +kubebuilder:validation:Optional - NormalizedInWeights map[string]float64 `json:"normalizedInWeights"` - // Outputs of the decision pipeline including the activations used - // to make the final ordering of compute hosts. - // +kubebuilder:validation:Optional - StepResults []StepResult `json:"stepResults,omitempty"` - // Aggregated output weights from the pipeline. - // +kubebuilder:validation:Optional - AggregatedOutWeights map[string]float64 `json:"aggregatedOutWeights"` - // Final ordered list of hosts from most preferred to least preferred. - // +kubebuilder:validation:Optional - OrderedHosts []string `json:"orderedHosts,omitempty"` - // The first element of the ordered hosts is considered the target host. - // +kubebuilder:validation:Optional - TargetHost *string `json:"targetHost,omitempty"` -} - -const ( - // The decision was successfully processed. - DecisionConditionReady = "Ready" -) - -type DecisionStatus struct { - // The result of this decision. - // +kubebuilder:validation:Optional - Result *DecisionResult `json:"result,omitempty"` - - // If there were previous decisions for the underlying resource, they can - // be resolved here to provide historical context for the decision. - // +kubebuilder:validation:Optional - History *[]corev1.ObjectReference `json:"history,omitempty"` - - // The number of decisions that preceded this one for the same resource. - // +kubebuilder:validation:Optional - Precedence *int `json:"precedence,omitempty"` - - // A human-readable explanation of the decision result. - // +kubebuilder:validation:Optional - Explanation string `json:"explanation,omitempty"` - - // The current status conditions of the decision. - // +kubebuilder:validation:Optional - Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"` -} - -// +kubebuilder:object:root=true -// +kubebuilder:subresource:status -// +kubebuilder:resource:scope=Cluster -// +kubebuilder:printcolumn:name="Domain",type="string",JSONPath=".spec.schedulingDomain" -// +kubebuilder:printcolumn:name="Resource ID",type="string",JSONPath=".spec.resourceID" -// +kubebuilder:printcolumn:name="#",type="string",JSONPath=".status.precedence" -// +kubebuilder:printcolumn:name="Created",type="date",JSONPath=".metadata.creationTimestamp" -// +kubebuilder:printcolumn:name="Pipeline",type="string",JSONPath=".spec.pipelineRef.name" -// +kubebuilder:printcolumn:name="TargetHost",type="string",JSONPath=".status.result.targetHost" -// +kubebuilder:printcolumn:name="Ready",type="string",JSONPath=".status.conditions[?(@.type=='Ready')].status" -// +kubebuilder:selectablefield:JSONPath=".spec.resourceID" - -// Currently the Decision CRD is an in-memory scheduling object used by the external scheduler API -// and filter-weigher pipelines to compute a placement result. It is currently NOT persisted -// to etcd — the scheduling outcome (including a human-readable explanation) is written onto the -// scheduled resource instead, e.g. the nova VM's spec.cortex fields. -// The long-term shape of this CRD is still under discussion; until that is -// settled the Decision serves only as a transient carrier within a single -// scheduling run. -type Decision struct { - metav1.TypeMeta `json:",inline"` - - // metadata is a standard object metadata - // +optional - metav1.ObjectMeta `json:"metadata,omitempty,omitzero"` - - // spec defines the desired state of Decision - // +required - Spec DecisionSpec `json:"spec"` - - // status defines the observed state of Decision - // +optional - Status DecisionStatus `json:"status,omitempty,omitzero"` -} - -// +kubebuilder:object:root=true - -// DecisionList contains a list of Decision -type DecisionList struct { - metav1.TypeMeta `json:",inline"` - metav1.ListMeta `json:"metadata,omitempty"` - Items []Decision `json:"items"` -} - -func init() { - SchemeBuilder.Register(&Decision{}, &DecisionList{}) -} diff --git a/api/v1alpha1/pipeline_types.go b/api/v1alpha1/pipeline_types.go index 1e5cfdca3..252718a8d 100644 --- a/api/v1alpha1/pipeline_types.go +++ b/api/v1alpha1/pipeline_types.go @@ -78,12 +78,6 @@ type PipelineSpec struct { // +kubebuilder:validation:Optional Description string `json:"description,omitempty"` - // If this pipeline should ignore host preselection and gather all - // available placement candidates before applying filters, instead of - // relying on a pre-filtered set and weights. - // +kubebuilder:default=false - IgnorePreselection bool `json:"ignorePreselection,omitempty"` - // The type of the pipeline, used to differentiate between // filter-weigher and detector pipelines within the same // scheduling domain. diff --git a/api/v1alpha1/reservation_types.go b/api/v1alpha1/reservation_types.go index 34ee24f10..0847c27ca 100644 --- a/api/v1alpha1/reservation_types.go +++ b/api/v1alpha1/reservation_types.go @@ -5,6 +5,7 @@ package v1alpha1 import ( hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/meta" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -81,18 +82,27 @@ const ( // AnnotationCreatorRequestID tracks the request ID that created this reservation. // Used for end-to-end traceability across API calls, controller reconciles, and scheduler invocations. AnnotationCreatorRequestID = "reservations.cortex.cloud/creator-request-id" + + // AnnotationCandidateAllocation signals that this PAYG pre-allocated reservation slot is intended + // for a specific workload object (e.g. VM UUID). Set by the commitment reconciler on creation. + // The reservation controller reads this annotation to discover and confirm the assignment. + AnnotationCandidateAllocation = "reservations.cortex.cloud/candidate-allocation" ) // CommittedResourceAllocation represents a workload's assignment to a committed resource reservation slot. // The workload could be a VM (Nova/IronCore), Pod (Kubernetes), or other resource. type CommittedResourceAllocation struct { - // Timestamp when this workload was assigned to the reservation. - // +kubebuilder:validation:Required - CreationTimestamp metav1.Time `json:"creationTimestamp"` + // ObjectRef is a reference to the workload object (e.g. VM CRD). + // +kubebuilder:validation:Optional + ObjectRef corev1.ObjectReference `json:"objectRef,omitempty"` - // Resources consumed by this instance. + // Resources consumed by this workload. // +kubebuilder:validation:Required Resources map[hv1.ResourceName]resource.Quantity `json:"resources"` + + // Deprecated: CreationTimestamp is no longer populated. Will be removed once old pods are fully rolled out. + // +kubebuilder:validation:Optional + CreationTimestamp metav1.Time `json:"creationTimestamp,omitempty"` } // CommittedResourceReservationSpec defines the spec fields specific to committed resource reservations. @@ -126,12 +136,22 @@ type CommittedResourceReservationSpec struct { // once it has processed the reservation, allowing the CR controller to wait until // all child reservations are up-to-date before accepting. // Zero means the field is not set (syncer-created reservations, no parent CR). + // + // Deprecated: replaced by conditions[Ready].ObservedGeneration signaling. Will be removed once old pods are fully rolled out. // +kubebuilder:validation:Optional ParentGeneration int64 `json:"parentGeneration,omitempty"` + // PrimaryResource is the resource dimension that drives allocation tracking + // (e.g. hv1.ResourceMemory for memory commitments). Other dimensions in + // alloc.Resources are recorded for bookkeeping but do not gate allocation decisions. + // +kubebuilder:validation:Optional + PrimaryResource hv1.ResourceName `json:"primaryResource,omitempty"` + // Allocations maps workload identifiers to their allocation details. // Key: Workload UUID (VM UUID for Nova, Pod UID for Pods, Machine UID for IronCore, etc.) // Value: allocation state and metadata + // + // Deprecated: use status.committedResourceReservation.allocation instead. Will be removed once old pods are fully rolled out. // +kubebuilder:validation:Optional Allocations map[string]CommittedResourceAllocation `json:"allocations,omitempty"` } @@ -259,13 +279,16 @@ type CommittedResourceReservationStatus struct { // ObservedParentGeneration is the Spec.CommittedResourceReservation.ParentGeneration value // that this Reservation controller last processed. When it matches ParentGeneration in spec, // the CR controller knows this reservation is up-to-date for the current CR spec version. + // + // Deprecated: replaced by conditions[Ready].ObservedGeneration signaling. Will be removed once old pods are fully rolled out. // +kubebuilder:validation:Optional ObservedParentGeneration int64 `json:"observedParentGeneration,omitempty"` - // Allocations maps VM/instance UUIDs to the host they are currently running on. - // Key: VM/instance UUID, Value: Host name where the VM is currently running. + // Allocation maps workload identifiers to their allocation details. + // Key: workload UUID (vmUUID for virtual machines) + // Value: allocation details including resource consumption and object reference. // +kubebuilder:validation:Optional - Allocations map[string]string `json:"allocations,omitempty"` + Allocation map[string]CommittedResourceAllocation `json:"allocation,omitempty"` } // FailoverReservationStatus defines the status fields specific to failover reservations. @@ -390,17 +413,22 @@ type ReservationStatus struct { // +kubebuilder:printcolumn:name="Type",type="string",JSONPath=".metadata.labels['reservations\\.cortex\\.cloud/type']" // +kubebuilder:printcolumn:name="Host",type="string",JSONPath=".status.host" // +kubebuilder:printcolumn:name="Ready",type="string",JSONPath=".status.conditions[?(@.type=='Ready')].status" -// +kubebuilder:printcolumn:name="ResourceGroup",type="string",JSONPath=".spec.committedResourceReservation.resourceGroup",priority=1 -// +kubebuilder:printcolumn:name="HA ResourceGroup",type="string",JSONPath=".spec.failoverReservation.resourceGroup",priority=1 -// +kubebuilder:printcolumn:name="Project",type="string",JSONPath=".spec.committedResourceReservation.projectID",priority=1 -// +kubebuilder:printcolumn:name="AZ",type="string",JSONPath=".spec.availabilityZone",priority=1 +// +kubebuilder:printcolumn:name="ObservedGen",type="integer",JSONPath=".status.conditions[?(@.type=='Ready')].observedGeneration",priority=1 +// +kubebuilder:printcolumn:name="Gen",type="integer",JSONPath=".metadata.generation",priority=1 +// +kubebuilder:printcolumn:name="ResourceGroup",type="string",JSONPath=".spec.*.resourceGroup" +// +kubebuilder:printcolumn:name="ResourceName",type="string",JSONPath=".spec.*.resourceName" +// +kubebuilder:printcolumn:name="Project",type="string",JSONPath=".spec.*.projectID",priority=1 +// +kubebuilder:printcolumn:name="CommitmentID",type="string",JSONPath=".spec.committedResourceReservation.commitmentUUID",priority=1 +// +kubebuilder:printcolumn:name="AZ",type="string",JSONPath=".spec.availabilityZone" // +kubebuilder:printcolumn:name="StartTime",type="string",JSONPath=".spec.startTime",priority=1 // +kubebuilder:printcolumn:name="EndTime",type="string",JSONPath=".spec.endTime",priority=1 -// +kubebuilder:printcolumn:name="Resources",type="string",JSONPath=".spec.resources",priority=1 +// +kubebuilder:printcolumn:name="Resources",type="string",JSONPath=".spec.resources" // +kubebuilder:printcolumn:name="LastChanged",type="date",JSONPath=".status.failoverReservation.lastChanged",priority=1 // +kubebuilder:printcolumn:name="AcknowledgedAt",type="date",JSONPath=".status.failoverReservation.acknowledgedAt",priority=1 -// +kubebuilder:printcolumn:name="CR Allocations",type="string",JSONPath=".status.committedResourceReservation.allocations",priority=1 +// +kubebuilder:printcolumn:name="CR alloc VMs",type="string",JSONPath=".status.committedResourceReservation.allocation.*.objectRef.name",priority=1 +// +kubebuilder:printcolumn:name="CR alloc resources",type="string",JSONPath=".status.committedResourceReservation.allocation.*.resources",priority=1 // +kubebuilder:printcolumn:name="HA Allocations",type="string",JSONPath=".status.failoverReservation.allocations",priority=1 +// +kubebuilder:printcolumn:name="CR CandidateVM",type="string",JSONPath=".metadata.annotations['reservations\\.cortex\\.cloud/candidate-allocation']",priority=1 // Reservation is the Schema for the reservations API type Reservation struct { diff --git a/api/v1alpha1/scheduling_intent.go b/api/v1alpha1/scheduling_intent.go new file mode 100644 index 000000000..03eff7f86 --- /dev/null +++ b/api/v1alpha1/scheduling_intent.go @@ -0,0 +1,14 @@ +// Copyright SAP SE +// SPDX-License-Identifier: Apache-2.0 + +package v1alpha1 + +// SchedulingIntent defines the intent of a scheduling decision (e.g. initial +// scheduling, rescheduling, failover). It is the base type for the Nova intent +// constants in api/external/nova and is carried on Reservation.Spec.Intent. +type SchedulingIntent string + +const ( + // SchedulingIntentUnknown is the default intent when none is specified. + SchedulingIntentUnknown SchedulingIntent = "Unknown" +) diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index a60d5a1f4..557d76c18 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -15,6 +15,42 @@ import ( "k8s.io/apimachinery/pkg/runtime" ) +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *BillingStatus) DeepCopyInto(out *BillingStatus) { + *out = *in + if in.AssignedInstances != nil { + in, out := &in.AssignedInstances, &out.AssignedInstances + *out = make([]string, len(*in)) + copy(*out, *in) + } + if in.UsedResources != nil { + in, out := &in.UsedResources, &out.UsedResources + *out = make(map[string]resource.Quantity, len(*in)) + for key, val := range *in { + (*out)[key] = val.DeepCopy() + } + } + if in.LastReconcileAt != nil { + in, out := &in.LastReconcileAt, &out.LastReconcileAt + *out = (*in).DeepCopy() + } + if in.ObservedGeneration != nil { + in, out := &in.ObservedGeneration, &out.ObservedGeneration + *out = new(int64) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BillingStatus. +func (in *BillingStatus) DeepCopy() *BillingStatus { + if in == nil { + return nil + } + out := new(BillingStatus) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *CinderDatasource) DeepCopyInto(out *CinderDatasource) { *out = *in @@ -60,7 +96,7 @@ func (in *CommittedResource) DeepCopyObject() runtime.Object { // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *CommittedResourceAllocation) DeepCopyInto(out *CommittedResourceAllocation) { *out = *in - in.CreationTimestamp.DeepCopyInto(&out.CreationTimestamp) + out.ObjectRef = in.ObjectRef if in.Resources != nil { in, out := &in.Resources, &out.Resources *out = make(map[apiv1.ResourceName]resource.Quantity, len(*in)) @@ -68,6 +104,7 @@ func (in *CommittedResourceAllocation) DeepCopyInto(out *CommittedResourceAlloca (*out)[key] = val.DeepCopy() } } + in.CreationTimestamp.DeepCopyInto(&out.CreationTimestamp) } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new CommittedResourceAllocation. @@ -270,11 +307,11 @@ func (in *CommittedResourceReservationSpec) DeepCopy() *CommittedResourceReserva // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *CommittedResourceReservationStatus) DeepCopyInto(out *CommittedResourceReservationStatus) { *out = *in - if in.Allocations != nil { - in, out := &in.Allocations, &out.Allocations - *out = make(map[string]string, len(*in)) + if in.Allocation != nil { + in, out := &in.Allocation, &out.Allocation + *out = make(map[string]CommittedResourceAllocation, len(*in)) for key, val := range *in { - (*out)[key] = val + (*out)[key] = *val.DeepCopy() } } } @@ -333,6 +370,11 @@ func (in *CommittedResourceStatus) DeepCopyInto(out *CommittedResourceStatus) { in, out := &in.LastReconcileAt, &out.LastReconcileAt *out = (*in).DeepCopy() } + if in.Billing != nil { + in, out := &in.Billing, &out.Billing + *out = new(BillingStatus) + (*in).DeepCopyInto(*out) + } if in.AssignedInstances != nil { in, out := &in.AssignedInstances, &out.AssignedInstances *out = make([]string, len(*in)) @@ -484,200 +526,6 @@ func (in *DatasourceStatus) DeepCopy() *DatasourceStatus { return out } -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *Decision) DeepCopyInto(out *Decision) { - *out = *in - out.TypeMeta = in.TypeMeta - in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) - in.Spec.DeepCopyInto(&out.Spec) - in.Status.DeepCopyInto(&out.Status) -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Decision. -func (in *Decision) DeepCopy() *Decision { - if in == nil { - return nil - } - out := new(Decision) - in.DeepCopyInto(out) - return out -} - -// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. -func (in *Decision) DeepCopyObject() runtime.Object { - if c := in.DeepCopy(); c != nil { - return c - } - return nil -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *DecisionList) DeepCopyInto(out *DecisionList) { - *out = *in - out.TypeMeta = in.TypeMeta - in.ListMeta.DeepCopyInto(&out.ListMeta) - if in.Items != nil { - in, out := &in.Items, &out.Items - *out = make([]Decision, len(*in)) - for i := range *in { - (*in)[i].DeepCopyInto(&(*out)[i]) - } - } -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new DecisionList. -func (in *DecisionList) DeepCopy() *DecisionList { - if in == nil { - return nil - } - out := new(DecisionList) - in.DeepCopyInto(out) - return out -} - -// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. -func (in *DecisionList) DeepCopyObject() runtime.Object { - if c := in.DeepCopy(); c != nil { - return c - } - return nil -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *DecisionResult) DeepCopyInto(out *DecisionResult) { - *out = *in - if in.RawInWeights != nil { - in, out := &in.RawInWeights, &out.RawInWeights - *out = make(map[string]float64, len(*in)) - for key, val := range *in { - (*out)[key] = val - } - } - if in.NormalizedInWeights != nil { - in, out := &in.NormalizedInWeights, &out.NormalizedInWeights - *out = make(map[string]float64, len(*in)) - for key, val := range *in { - (*out)[key] = val - } - } - if in.StepResults != nil { - in, out := &in.StepResults, &out.StepResults - *out = make([]StepResult, len(*in)) - for i := range *in { - (*in)[i].DeepCopyInto(&(*out)[i]) - } - } - if in.AggregatedOutWeights != nil { - in, out := &in.AggregatedOutWeights, &out.AggregatedOutWeights - *out = make(map[string]float64, len(*in)) - for key, val := range *in { - (*out)[key] = val - } - } - if in.OrderedHosts != nil { - in, out := &in.OrderedHosts, &out.OrderedHosts - *out = make([]string, len(*in)) - copy(*out, *in) - } - if in.TargetHost != nil { - in, out := &in.TargetHost, &out.TargetHost - *out = new(string) - **out = **in - } -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new DecisionResult. -func (in *DecisionResult) DeepCopy() *DecisionResult { - if in == nil { - return nil - } - out := new(DecisionResult) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *DecisionSpec) DeepCopyInto(out *DecisionSpec) { - *out = *in - out.PipelineRef = in.PipelineRef - if in.NovaRaw != nil { - in, out := &in.NovaRaw, &out.NovaRaw - *out = new(runtime.RawExtension) - (*in).DeepCopyInto(*out) - } - if in.CinderRaw != nil { - in, out := &in.CinderRaw, &out.CinderRaw - *out = new(runtime.RawExtension) - (*in).DeepCopyInto(*out) - } - if in.ManilaRaw != nil { - in, out := &in.ManilaRaw, &out.ManilaRaw - *out = new(runtime.RawExtension) - (*in).DeepCopyInto(*out) - } - if in.MachineRef != nil { - in, out := &in.MachineRef, &out.MachineRef - *out = new(v1.ObjectReference) - **out = **in - } - if in.PodRef != nil { - in, out := &in.PodRef, &out.PodRef - *out = new(v1.ObjectReference) - **out = **in - } -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new DecisionSpec. -func (in *DecisionSpec) DeepCopy() *DecisionSpec { - if in == nil { - return nil - } - out := new(DecisionSpec) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *DecisionStatus) DeepCopyInto(out *DecisionStatus) { - *out = *in - if in.Result != nil { - in, out := &in.Result, &out.Result - *out = new(DecisionResult) - (*in).DeepCopyInto(*out) - } - if in.History != nil { - in, out := &in.History, &out.History - *out = new([]v1.ObjectReference) - if **in != nil { - in, out := *in, *out - *out = make([]v1.ObjectReference, len(*in)) - copy(*out, *in) - } - } - if in.Precedence != nil { - in, out := &in.Precedence, &out.Precedence - *out = new(int) - **out = **in - } - if in.Conditions != nil { - in, out := &in.Conditions, &out.Conditions - *out = make([]metav1.Condition, len(*in)) - for i := range *in { - (*in)[i].DeepCopyInto(&(*out)[i]) - } - } -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new DecisionStatus. -func (in *DecisionStatus) DeepCopy() *DecisionStatus { - if in == nil { - return nil - } - out := new(DecisionStatus) - in.DeepCopyInto(out) - return out -} - // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *Descheduling) DeepCopyInto(out *Descheduling) { *out = *in @@ -1951,28 +1799,6 @@ func (in *ReservationStatus) DeepCopy() *ReservationStatus { return out } -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *StepResult) DeepCopyInto(out *StepResult) { - *out = *in - if in.Activations != nil { - in, out := &in.Activations, &out.Activations - *out = make(map[string]float64, len(*in)) - for key, val := range *in { - (*out)[key] = val - } - } -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StepResult. -func (in *StepResult) DeepCopy() *StepResult { - if in == nil { - return nil - } - out := new(StepResult) - in.DeepCopyInto(out) - return out -} - // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *WeigherSpec) DeepCopyInto(out *WeigherSpec) { *out = *in diff --git a/cmd/manager/main.go b/cmd/manager/main.go index 9ba24d770..a567fd7f4 100644 --- a/cmd/manager/main.go +++ b/cmd/manager/main.go @@ -54,6 +54,7 @@ import ( "github.com/cobaltcore-dev/cortex/internal/scheduling/external" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" schedulinglib "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" "github.com/cobaltcore-dev/cortex/internal/scheduling/machines" "github.com/cobaltcore-dev/cortex/internal/scheduling/manila" @@ -415,33 +416,43 @@ func main() { commitmentsAPI.Init(mux, metrics.Registry, ctrl.Log.WithName("commitments-api")) } + // novaScheduler is the shared in-process scheduler. Built when the + // nova-pipeline-controllers are enabled and consumed in-process by the Nova + // HTTP API and the reservation/commitment/capacity/failover controllers (which + // must run in the same process — see the cortex-nova-scheduling deployment). + var novaScheduler *compute.Placement + if slices.Contains(mainConfig.EnabledControllers, "nova-pipeline-controllers") { featureGates := conf.GetConfigOrDie[nova.FeatureGates]() noHostFoundCounter := crs.NewNoHostFoundCounter() placementCounter := crs.NewPlacementCounter() // Filter-weigher pipeline controller setup. filterWeigherController := &nova.FilterWeigherPipelineController{ - Monitor: filterWeigherPipelineMonitor, - FeatureGates: featureGates, - CRRecorder: crs.Recorder{ - NoHostFoundCounter: noHostFoundCounter, - PlacementCounter: placementCounter, - }, + Monitor: filterWeigherPipelineMonitor, } metrics.Registry.MustRegister(noHostFoundCounter) metrics.Registry.MustRegister(placementCounter) // Inferred through the base controller. filterWeigherController.Client = multiclusterClient - filterWeigherController.CRRecorder.Client = multiclusterClient if err := filterWeigherController.SetupWithManager(mgr, multiclusterClient); err != nil { setupLog.Error(err, "unable to create controller", "controller", "nova FilterWeigherPipelineController") os.Exit(1) } + // Shared in-process scheduler: owns the scheduling lock and runs the + // pipeline. The Nova HTTP API and the reservation controllers call it + // directly (no self-directed HTTP). + novaScheduler = compute.NewPlacement(filterWeigherController, nova.NewCandidateGatherer(multiclusterClient)) + vmHandler := &nova.VirtualMachineHandler{Client: multiclusterClient} + crRecorder := crs.Recorder{ + Client: multiclusterClient, + NoHostFoundCounter: noHostFoundCounter, + PlacementCounter: placementCounter, + } novaAPIConfig := conf.GetConfigOrDie[nova.HTTPAPIConfig]() setupLog.Info("loaded nova API config", "evacuationShuffleK", novaAPIConfig.EvacuationShuffleK, "novaLimitHostsToRequest", novaAPIConfig.NovaLimitHostsToRequest) - nova.NewAPI(novaAPIConfig, filterWeigherController).Init(mux) + nova.NewAPI(novaAPIConfig, novaScheduler, vmHandler, crRecorder, featureGates).Init(mux) // Detector pipeline controller setup. novaClient := nova.NewNovaClient() @@ -604,10 +615,16 @@ func main() { if slices.Contains(mainConfig.EnabledControllers, "committed-resource-reservations-controller") { setupLog.Info("enabling controller", "controller", "committed-resource-reservations-controller") + if novaScheduler == nil { + setupLog.Error(nil, "committed-resource-reservations-controller requires nova-pipeline-controllers (in-process scheduler) to be enabled") + os.Exit(1) + } if err := (&commitments.CommitmentReservationController{ - Client: multiclusterClient, - Scheme: mgr.GetScheme(), - Conf: commitmentsConfig.ReservationController, + Client: multiclusterClient, + Scheme: mgr.GetScheme(), + Conf: commitmentsConfig.ReservationController, + Scheduler: novaScheduler, + VMSource: commitmentsVMSource, }).SetupWithManager(mgr, multiclusterClient); err != nil { setupLog.Error(err, "unable to create controller", "controller", "CommitmentReservation") os.Exit(1) @@ -725,9 +742,10 @@ func main() { setupLog.Error(nil, "failover-reservations-controller requires datasourceName to be configured") os.Exit(1) } - - // The scheduler client calls the nova external scheduler API to get placement decisions - schedulerClient := reservations.NewSchedulerClient(failoverConfig.SchedulerURL) + if novaScheduler == nil { + setupLog.Error(nil, "failover-reservations-controller requires nova-pipeline-controllers (in-process scheduler) to be enabled") + os.Exit(1) + } failoverMonitor := failover.NewFailoverMonitor() if err := metrics.Registry.Register(failoverMonitor); err != nil { @@ -759,7 +777,7 @@ func main() { multiclusterClient, vmSource, failoverConfig, - schedulerClient, + novaScheduler, failoverMonitor, ) @@ -771,7 +789,6 @@ func main() { setupLog.Info("failover-reservations-controller starting", "datasourceName", failoverConfig.DatasourceName, - "schedulerURL", failoverConfig.SchedulerURL, "reconcileInterval", failoverConfig.ReconcileInterval, "revalidationInterval", failoverConfig.RevalidationInterval) @@ -783,7 +800,6 @@ func main() { } setupLog.Info("failover-reservations-controller registered", "datasourceName", failoverConfig.DatasourceName, - "schedulerURL", failoverConfig.SchedulerURL, "reconcileInterval", failoverConfig.ReconcileInterval, "revalidationInterval", failoverConfig.RevalidationInterval, "trustHypervisorLocation", failoverConfig.TrustHypervisorLocation, @@ -798,10 +814,15 @@ func main() { setupLog.Error(err, "failed to register failover host monitor metrics, continuing without metrics") failoverHostMonitor = nil } + if novaScheduler == nil { + setupLog.Error(nil, "failover-host-reservations-controller requires nova-pipeline-controllers (in-process scheduler) to be enabled") + os.Exit(1) + } controller := &failoverhost.Controller{ - Client: multiclusterClient, - Config: failoverHostConfig, - Monitor: failoverHostMonitor, + Client: multiclusterClient, + Config: failoverHostConfig, + Scheduler: novaScheduler, + Monitor: failoverHostMonitor, } if err := controller.SetupWithManager(mgr); err != nil { setupLog.Error(err, "unable to create controller", @@ -811,8 +832,7 @@ func main() { setupLog.Info("failover-host-reservations-controller registered", "ratio", failoverHostConfig.Ratio, "minimum", failoverHostConfig.Minimum, - "generationInterval", failoverHostConfig.GenerationInterval, - "externalSchedulerURL", failoverHostConfig.ExternalSchedulerURL) + "generationInterval", failoverHostConfig.GenerationInterval) } if slices.Contains(mainConfig.EnabledControllers, "capacity-controller") { setupLog.Info("enabling controller", "controller", "capacity-controller") @@ -823,18 +843,21 @@ func main() { os.Exit(1) } + if novaScheduler == nil { + setupLog.Error(nil, "capacity-controller requires nova-pipeline-controllers (in-process scheduler) to be enabled") + os.Exit(1) + } capacityMonitor := capacity.NewMonitor(multiclusterClient) if err := metrics.Registry.Register(&capacityMonitor); err != nil { setupLog.Error(err, "failed to register capacity monitor metrics, continuing without metrics") } - if err := capacity.NewController(multiclusterClient, capacityConfig, commitmentsVMSource). + if err := capacity.NewController(multiclusterClient, capacityConfig, commitmentsVMSource, novaScheduler). SetupWithManager(mgr, multiclusterClient); err != nil { setupLog.Error(err, "unable to create controller", "controller", "capacity") os.Exit(1) } setupLog.Info("capacity-controller registered", - "schedulerURL", capacityConfig.SchedulerURL, "reconcileInterval", capacityConfig.ReconcileInterval, "minReconcileInterval", capacityConfig.MinReconcileInterval, "totalPipeline", capacityConfig.TotalPipeline, diff --git a/docs/03-reservations-and-inventory/03-b-committed-resource-transaction-architecture.md b/docs/03-reservations-and-inventory/03-b-committed-resource-transaction-architecture.md index c35c5a6ae..3e5827bd1 100644 --- a/docs/03-reservations-and-inventory/03-b-committed-resource-transaction-architecture.md +++ b/docs/03-reservations-and-inventory/03-b-committed-resource-transaction-architecture.md @@ -59,6 +59,10 @@ sequenceDiagram CRC->>CRCD: read CR.status.Ready + observedGeneration, advance to Accepted or RollingBack ``` +## CR Reservation slot model + +Each CR reservation slot has a fixed capacity. Each reservation slots tracks VM usage in an allocation map. Each entry records the resources *this slot* covers for that VM — which may be less than the VM's full size. This supports multi-slot spanning of the same VM, and also partially covered VMs by a slot. + ## CRC state machine ```mermaid diff --git a/go.mod b/go.mod index 1430500a3..0b88abdfa 100644 --- a/go.mod +++ b/go.mod @@ -3,20 +3,20 @@ module github.com/cobaltcore-dev/cortex go 1.27 require ( - github.com/cobaltcore-dev/openstack-hypervisor-operator v1.3.0 + github.com/cobaltcore-dev/openstack-hypervisor-operator v1.4.0 github.com/cobaltcore-dev/vm-operator v0.2.10 github.com/go-gorp/gorp v2.2.0+incompatible github.com/go-logr/logr v1.4.4 github.com/google/uuid v1.6.0 github.com/gophercloud/gophercloud/v2 v2.15.0 - github.com/ironcore-dev/ironcore v0.8.0 + github.com/ironcore-dev/ironcore v0.9.0 github.com/lib/pq v1.12.3 github.com/mattn/go-sqlite3 v1.14.52 github.com/ory/dockertest v3.3.5+incompatible github.com/prometheus/client_golang v1.25.0 github.com/prometheus/client_model v0.6.3 github.com/sapcc/go-api-declarations v1.25.1 - github.com/sapcc/go-bits v0.0.0-20261029170309-6e36605dd3a1 + github.com/sapcc/go-bits v0.0.0-20261103143358-0cf02691bfb8 go.uber.org/zap v1.28.0 go.xyrillian.de/gg v1.19.0 golang.org/x/sync v0.24.0 @@ -29,7 +29,7 @@ require ( ) require ( - cel.dev/expr v0.25.2 // indirect + cel.dev/expr v0.25.3 // indirect github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect github.com/Microsoft/go-winio v0.6.2 // indirect github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 // indirect @@ -72,7 +72,6 @@ require ( github.com/gorilla/mux v1.8.1 // indirect github.com/gotestyourself/gotestyourself v2.2.0+incompatible // indirect github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect - github.com/hashicorp/go-uuid v1.0.3 // indirect github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/json-iterator/go v1.1.12 // indirect @@ -84,11 +83,11 @@ require ( github.com/opencontainers/go-digest v1.0.0 // indirect github.com/opencontainers/image-spec v1.1.1 // indirect github.com/opencontainers/runc v1.3.6 // indirect - github.com/pandatix/go-abnf v0.5.1 // indirect + github.com/pandatix/go-abnf v0.5.3 // indirect github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect github.com/poy/onpar v0.3.5 // indirect - github.com/prometheus/common v0.73.0 // indirect + github.com/prometheus/common v0.73.1 // indirect github.com/prometheus/procfs v0.22.0 // indirect github.com/sirupsen/logrus v1.9.4 // indirect github.com/spf13/cobra v1.10.2 // indirect @@ -117,7 +116,7 @@ require ( gomodules.xyz/jsonpatch/v2 v2.5.0 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect - google.golang.org/grpc v1.83.2 // indirect + google.golang.org/grpc v1.85.0 // indirect google.golang.org/protobuf v1.36.12 // indirect gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect @@ -128,7 +127,7 @@ require ( k8s.io/klog/v2 v2.140.0 // indirect k8s.io/kube-openapi v0.0.0-20260721132016-d427ff9ee9ad // indirect k8s.io/streaming v0.37.2 // indirect - k8s.io/utils v0.0.0-20260707023825-cf1189d6abe3 // indirect + k8s.io/utils v0.0.0-20261023123148-ca04ba58ecab // indirect sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.36.0 // indirect sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect sigs.k8s.io/randfill v1.0.0 // indirect diff --git a/go.sum b/go.sum index 863f1172e..e82fb852b 100644 --- a/go.sum +++ b/go.sum @@ -1,5 +1,5 @@ -cel.dev/expr v0.25.2 h1:K6j46C81hXtZQfuX60cVWQFBJahKSE2gfRbNuvr5bFs= -cel.dev/expr v0.25.2/go.mod h1:hrXvqGP6G6gyx8UAHSHJ5RGk//1Oj5nXQ2NI02Nrsg4= +cel.dev/expr v0.25.3 h1:A2jO8jwOugrrovveCWfj0KEZOfqiLgAcwjpHPhzIGw0= +cel.dev/expr v0.25.3/go.mod h1:hrXvqGP6G6gyx8UAHSHJ5RGk//1Oj5nXQ2NI02Nrsg4= filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo= filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc= github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c h1:udKWzYgxTojEKWjV8V+WSxDXJ4NFATAsZjh8iIbsQIg= @@ -22,8 +22,8 @@ github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1x github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw= github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/cobaltcore-dev/openstack-hypervisor-operator v1.3.0 h1:WhvPRxNzZCL0X+/k+H2VZPO2jtzGXQ/SspIuYqpryzQ= -github.com/cobaltcore-dev/openstack-hypervisor-operator v1.3.0/go.mod h1:vKyl1a9EJDJs24Uh8GcSfij0WwNXBo9/wtkTG5SOVwI= +github.com/cobaltcore-dev/openstack-hypervisor-operator v1.4.0 h1:lAb+qBhyfgf77YpDtMBhSH1v3YVHI2K2eWRltbLMoaE= +github.com/cobaltcore-dev/openstack-hypervisor-operator v1.4.0/go.mod h1:VIbkpZpnTeLQrEesadTeANk0MB0t+enfIGfJIX/a2vY= github.com/cobaltcore-dev/vm-operator v0.2.10 h1:W/cNZBDva2pq9kn4QYTcfn4VPxyNEoYqxJBU9hc8Ulg= github.com/cobaltcore-dev/vm-operator v0.2.10/go.mod h1:0Lanmhb7onfOIHX+4h1lGLC7vga/nNYabJ6a/y/APk4= github.com/containerd/continuity v0.4.5 h1:ZRoN1sXq9u7V6QoHMcVWGhOwDFqZ4B9i5H6un1Wh0x4= @@ -121,14 +121,12 @@ github.com/gotestyourself/gotestyourself v2.2.0+incompatible h1:AQwinXlbQR2HvPjQ github.com/gotestyourself/gotestyourself v2.2.0+incompatible/go.mod h1:zZKM6oeNM8k+FRljX1mnzVYeS8wiGgQyvST1/GafPbY= github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw= github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY= -github.com/hashicorp/go-uuid v1.0.3 h1:2gKiV6YVmrJ1i2CKKa9obLvRieoRGviZFL26PcT/Co8= -github.com/hashicorp/go-uuid v1.0.3/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro= github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= -github.com/ironcore-dev/ironcore v0.8.0 h1:llh4S1ZK9gLdUPBAUb/dODHaiYk69x4Vi+gpF6aM0sg= -github.com/ironcore-dev/ironcore v0.8.0/go.mod h1:1Jz0yG2Hpt838Rg5WJayj8NjiiSaIwETVogLP/VOynk= +github.com/ironcore-dev/ironcore v0.9.0 h1:hwdvI0pN3y/PGGd/BUxBY4vYW6gfxatBKgsojDBk/j4= +github.com/ironcore-dev/ironcore v0.9.0/go.mod h1:T4a/vnTvdtK7mpfGi1IzLnUFT6BgxyD2jcTGz0ipuuY= github.com/itchyny/gojq v0.12.19 h1:ttXA0XCLEMoaLOz5lSeFOZ6u6Q3QxmG46vfgI4O0DEs= github.com/itchyny/gojq v0.12.19/go.mod h1:5galtVPDywX8SPSOrqjGxkBeDhSxEW1gSxoy7tn1iZY= github.com/itchyny/timefmt-go v0.1.8 h1:1YEo1JvfXeAHKdjelbYr/uCuhkybaHCeTkH8Bo791OI= @@ -169,8 +167,8 @@ github.com/opencontainers/runc v1.3.6 h1:SLGIymCtsk80iNPWgbc8dtjI30r+5mTVV+4dN8/ github.com/opencontainers/runc v1.3.6/go.mod h1:o1wyv76EDlTkcf0KTFgN8bMWLPvgF/HfX709lDv+rr4= github.com/ory/dockertest v3.3.5+incompatible h1:iLLK6SQwIhcbrG783Dghaaa3WPzGc+4Emza6EbVUUGA= github.com/ory/dockertest v3.3.5+incompatible/go.mod h1:1vX4m9wsvi00u5bseYwXaSnhNrne+V0E6LAcBILJdPs= -github.com/pandatix/go-abnf v0.5.1 h1:iyLvSu2Aq12iE0r4n948Qw21BHDuckLq/8WIXAAF+HI= -github.com/pandatix/go-abnf v0.5.1/go.mod h1:VBmbtpFgQNEaQpyAvmi+Ecd9wAYJKzA3TmiB3L93sVg= +github.com/pandatix/go-abnf v0.5.3 h1:LA9AaxAqwzdxfDe7+HIGUB6tzx2N6wYVEAChmfsoPgo= +github.com/pandatix/go-abnf v0.5.3/go.mod h1:AE0KD3/VNoCjGzc1gqetRKshbI4SYErXQ/8kPenR5mc= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= @@ -182,8 +180,8 @@ github.com/prometheus/client_golang v1.25.0 h1:cDtW1TcFsMi1RJc8khwCKB4VhzwV04aKq github.com/prometheus/client_golang v1.25.0/go.mod h1:j2X/4/kaQHLPnmyEU8zjmkndo1T317VpCIQEml8LTKk= github.com/prometheus/client_model v0.6.3 h1:O0jaTVAYNxTHYInEPFJt5I3+sN8zqBtVMPTB1qyxiEo= github.com/prometheus/client_model v0.6.3/go.mod h1:gpN5P9S7Rr6Yr92PiQ+Ixvhf6JZEkF1dnxsYL2aPBEM= -github.com/prometheus/common v0.73.0 h1:NGIOmVi8jrgiiPQgXxD6rP0FfxFsuZxnMQkel0qBy/g= -github.com/prometheus/common v0.73.0/go.mod h1:Nt+CsVkf/mj4wDAmtSwq1HTPohHSX6fMr6+/f9B5UqM= +github.com/prometheus/common v0.73.1 h1:SZWh/j4AvhCdhX8jvYSnxvPxhWlC9QZkjJ+tOrEr7kU= +github.com/prometheus/common v0.73.1/go.mod h1:I9aA/Ukz9s1olOobh8KtCkU7B1mQPQsB9UvQPsUQ1Dk= github.com/prometheus/procfs v0.22.0 h1:6q9+/JL9IKAPbCmBrv9n5O5Ty3NKnciV5X7YGw0oics= github.com/prometheus/procfs v0.22.0/go.mod h1:CvmFr/GVhIjIvWJZW3tgkODBQMRIf0EyWMQLHCHab58= github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= @@ -191,8 +189,8 @@ github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/sapcc/go-api-declarations v1.25.1 h1:Lrbf8TZY/ARg0J9IrlrObqnXjrzSzmYKQd/L/a71Lsg= github.com/sapcc/go-api-declarations v1.25.1/go.mod h1:bJkK8wphzlmUmN5lxBv8BbfxUXMOkW+UiC10xBHXsFs= -github.com/sapcc/go-bits v0.0.0-20261029170309-6e36605dd3a1 h1:bgfB935Ox8K6Fa75T/VlSIAQ0M4v9mKdbHWBYRrRKPE= -github.com/sapcc/go-bits v0.0.0-20261029170309-6e36605dd3a1/go.mod h1:qV1JUJLrG2keKgLcPfI407VwjrYjCm6C9bwx6jX5C1g= +github.com/sapcc/go-bits v0.0.0-20261103143358-0cf02691bfb8 h1:/BVU+P8qLSsO5Tth28f+tJ4sWLnY/Qfe22pDl27CLXE= +github.com/sapcc/go-bits v0.0.0-20261103143358-0cf02691bfb8/go.mod h1:pAWINCvaeqwPz14ySCpreshOet1DXR7ubEntYmD8bCY= github.com/sirupsen/logrus v1.9.4 h1:TsZE7l11zFCLZnZ+teH4Umoq5BhEIfIzfRDZ1Uzql2w= github.com/sirupsen/logrus v1.9.4/go.mod h1:ftWc9WdOfJ0a92nsE2jF5u5ZwH8Bv2zdeOC42RjbV2g= github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU= @@ -274,8 +272,8 @@ google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 h1: google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688/go.mod h1:1RJ9BQGyNdZwkGc1eTqkErfRZ6RJyYPHZo73BZ1vQqI= google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 h1:cYNAzI2sUwhmCcoj9TxvihSrqsxt6uIkj3rDRhSDmW4= google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688/go.mod h1:DjtHYE8FKJLivXcBEjGwndXfIC23G0VpXiXKqG179uA= -google.golang.org/grpc v1.83.2 h1:EManeRomTObA0BU7I8vXgg/78uE5MJ9M8B39EX2WscU= -google.golang.org/grpc v1.83.2/go.mod h1:YPI1hK3kDked6iHvgX3tR0y+nX/qpMFKhPgFsokw1S8= +google.golang.org/grpc v1.85.0 h1:BPBCfuVRQ1qvBhKDOq9uvQllMvO59rKsYhXpp7Dd8vw= +google.golang.org/grpc v1.85.0/go.mod h1:0rQrOD58yjal/0EaVbGBANindixsCZNQPDV3VKtaKuQ= google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc= google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= @@ -307,8 +305,8 @@ k8s.io/kube-openapi v0.0.0-20260721132016-d427ff9ee9ad h1:oXImqH8mQNk7PmvzKhmN3d k8s.io/kube-openapi v0.0.0-20260721132016-d427ff9ee9ad/go.mod h1:0/mqHCVhlumdJ3BhCfnjSZQE037nAhNodh1/hK0T8/I= k8s.io/streaming v0.37.2 h1:xmzp9nbqIbjvsRcqrBYa+K5XPQ6Jy/Dm3zZ8mN916AI= k8s.io/streaming v0.37.2/go.mod h1:APlJR26ZWRcVy5bIEj0QRrKUXROtBHPcxl2NT7EAzPU= -k8s.io/utils v0.0.0-20260707023825-cf1189d6abe3 h1:jVkFFVfXdXP74B/zbO3hM3hpSFD0xvhQ5U686DPurkE= -k8s.io/utils v0.0.0-20260707023825-cf1189d6abe3/go.mod h1:M2s5JB1lIYP3jzZdorPLHXIPJzt9vv2muW5a6L9DtNM= +k8s.io/utils v0.0.0-20261023123148-ca04ba58ecab h1:pjQrNLaF//ytZtWwCAHtW+pqwSKjtWDqzA8+TCX37RI= +k8s.io/utils v0.0.0-20261023123148-ca04ba58ecab/go.mod h1:DdQ0BJxlQtuMlKDqdg7X5IMMQCXIk2b3d1/4M9OhUYo= sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.36.0 h1:/YpDJ4vReG7ZmzSpBGxduXgywWkJU9zHubgJG03MT+Y= sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.36.0/go.mod h1:tJo1aepTXyR+8Xs3sUsGBDk4Ub2AM5dPAPKJx0mpm5c= sigs.k8s.io/controller-runtime v0.25.2 h1:bEkK3PVOIVK9X8QWLGVhJgmFc++47vfT6wakSzAOLsQ= diff --git a/helm/bundles/cortex-cinder/Chart.yaml b/helm/bundles/cortex-cinder/Chart.yaml index a903223b3..77e9c044e 100644 --- a/helm/bundles/cortex-cinder/Chart.yaml +++ b/helm/bundles/cortex-cinder/Chart.yaml @@ -5,23 +5,23 @@ apiVersion: v2 name: cortex-cinder description: A Helm chart deploying Cortex for Cinder. type: application -version: 0.0.110 +version: 0.0.111 appVersion: 0.1.0 dependencies: # from: file://../../library/cortex-postgres - name: cortex-postgres repository: oci://ghcr.io/cobaltcore-dev/cortex/charts - version: 0.6.30 + version: 0.6.31 # from: file://../../library/cortex - name: cortex repository: oci://ghcr.io/cobaltcore-dev/cortex/charts - version: 0.7.3 + version: 0.8.0 alias: cortex-knowledge-controllers # from: file://../../library/cortex - name: cortex repository: oci://ghcr.io/cobaltcore-dev/cortex/charts - version: 0.7.3 + version: 0.8.0 alias: cortex-scheduling-controllers # Owner info adds a configmap to the kubernetes cluster with information on diff --git a/helm/bundles/cortex-cinder/templates/alerts.yaml b/helm/bundles/cortex-cinder/templates/alerts.yaml index f15cfb755..b0c6c6557 100644 --- a/helm/bundles/cortex-cinder/templates/alerts.yaml +++ b/helm/bundles/cortex-cinder/templates/alerts.yaml @@ -252,46 +252,6 @@ spec: configuration. It is recommended to investigate the knowledge status and logs for more details. - - alert: CortexCinderDecisionsWithErrors - expr: cortex_decision_state{domain="cinder",state="error"} > 0 - for: 5m - labels: -{{- if .Values.alerts.persesDashboardBaseURL }} - persesDashboard: "{{ .Values.alerts.persesDashboardBaseURL }}/cortex-status" -{{- end }} - context: decisions - service: cortex - severity: warning - support_group: workload-management - annotations: - summary: "Some decisions are in error state for operator `{{ "{{" }} $labels.operator {{ "}}" }}`" - description: > - The cortex scheduling pipeline generated decisions that are in error state. - This may indicate issues with the decision logic or the underlying infrastructure. - It is recommended to investigate the decision logs and the state of the - VMs being processed. - - - alert: CortexCinderTooManyDecisionsWaiting - expr: cortex_decision_state{domain="cinder",state="waiting"} > 10 - for: 5m - labels: -{{- if .Values.alerts.persesDashboardBaseURL }} - persesDashboard: "{{ .Values.alerts.persesDashboardBaseURL }}/cortex-status" -{{- end }} - context: decisions - service: cortex - severity: warning - support_group: workload-management - annotations: - summary: "Too many decisions are in waiting state for operator `{{ "{{" }} $labels.operator {{ "}}" }}`" - description: > - The cortex scheduling pipeline has a high number of decisions for which - no target host has been assigned yet. - - This may indicate a backlog in processing or issues with the decision logic. - It is recommended to investigate the decision logs and the state of the - VMs being processed. - - alert: CortexCinderKPIUnready expr: | cortex_kpi_state{domain="cinder",state!="ready"} != 0 diff --git a/helm/bundles/cortex-cinder/templates/kpis.yaml b/helm/bundles/cortex-cinder/templates/kpis.yaml index fb3958c08..78fd25dd9 100644 --- a/helm/bundles/cortex-cinder/templates/kpis.yaml +++ b/helm/bundles/cortex-cinder/templates/kpis.yaml @@ -25,18 +25,6 @@ spec: --- apiVersion: cortex.cloud/v1alpha1 kind: KPI -metadata: - name: cortex-cinder-decision-state -spec: - schedulingDomain: cinder - impl: decision_state_kpi - opts: - decisionSchedulingDomain: cinder - description: | - This aggregated KPI tracks the state of decision resources managed by cortex. ---- -apiVersion: cortex.cloud/v1alpha1 -kind: KPI metadata: name: cortex-cinder-kpi-state spec: diff --git a/helm/bundles/cortex-cinder/values.yaml b/helm/bundles/cortex-cinder/values.yaml index b11cbfc38..98d8e5f4b 100644 --- a/helm/bundles/cortex-cinder/values.yaml +++ b/helm/bundles/cortex-cinder/values.yaml @@ -90,8 +90,6 @@ cortex: &cortex apiservers: home: gvks: - - cortex.cloud/v1alpha1/Decision - - cortex.cloud/v1alpha1/DecisionList - cortex.cloud/v1alpha1/Descheduling - cortex.cloud/v1alpha1/DeschedulingList - cortex.cloud/v1alpha1/Pipeline diff --git a/helm/bundles/cortex-crds/Chart.yaml b/helm/bundles/cortex-crds/Chart.yaml index 18f44b407..273303371 100644 --- a/helm/bundles/cortex-crds/Chart.yaml +++ b/helm/bundles/cortex-crds/Chart.yaml @@ -5,13 +5,13 @@ apiVersion: v2 name: cortex-crds description: A Helm chart deploying Cortex CRDs. type: application -version: 0.0.110 +version: 0.0.111 appVersion: 0.1.0 dependencies: # from: file://../../library/cortex - name: cortex repository: oci://ghcr.io/cobaltcore-dev/cortex/charts - version: 0.7.3 + version: 0.8.0 # Owner info adds a configmap to the kubernetes cluster with information on # the service owner. This makes it easier to find out who to contact in case diff --git a/helm/bundles/cortex-ironcore/Chart.yaml b/helm/bundles/cortex-ironcore/Chart.yaml index 1848560dd..3e8c10549 100644 --- a/helm/bundles/cortex-ironcore/Chart.yaml +++ b/helm/bundles/cortex-ironcore/Chart.yaml @@ -5,13 +5,13 @@ apiVersion: v2 name: cortex-ironcore description: A Helm chart deploying Cortex for IronCore. type: application -version: 0.0.110 +version: 0.0.111 appVersion: 0.1.0 dependencies: # from: file://../../library/cortex - name: cortex repository: oci://ghcr.io/cobaltcore-dev/cortex/charts - version: 0.7.3 + version: 0.8.0 # Owner info adds a configmap to the kubernetes cluster with information on # the service owner. This makes it easier to find out who to contact in case diff --git a/helm/bundles/cortex-ironcore/values.yaml b/helm/bundles/cortex-ironcore/values.yaml index 6f577b04b..e54147f15 100644 --- a/helm/bundles/cortex-ironcore/values.yaml +++ b/helm/bundles/cortex-ironcore/values.yaml @@ -33,8 +33,6 @@ cortex: apiservers: home: gvks: - - cortex.cloud/v1alpha1/Decision - - cortex.cloud/v1alpha1/DecisionList - cortex.cloud/v1alpha1/Descheduling - cortex.cloud/v1alpha1/DeschedulingList - cortex.cloud/v1alpha1/Pipeline diff --git a/helm/bundles/cortex-manila/Chart.yaml b/helm/bundles/cortex-manila/Chart.yaml index 0f2f3df64..eddac24bf 100644 --- a/helm/bundles/cortex-manila/Chart.yaml +++ b/helm/bundles/cortex-manila/Chart.yaml @@ -5,23 +5,23 @@ apiVersion: v2 name: cortex-manila description: A Helm chart deploying Cortex for Manila. type: application -version: 0.0.110 +version: 0.0.111 appVersion: 0.1.0 dependencies: # from: file://../../library/cortex-postgres - name: cortex-postgres repository: oci://ghcr.io/cobaltcore-dev/cortex/charts - version: 0.6.30 + version: 0.6.31 # from: file://../../library/cortex - name: cortex repository: oci://ghcr.io/cobaltcore-dev/cortex/charts - version: 0.7.3 + version: 0.8.0 alias: cortex-knowledge-controllers # from: file://../../library/cortex - name: cortex repository: oci://ghcr.io/cobaltcore-dev/cortex/charts - version: 0.7.3 + version: 0.8.0 alias: cortex-scheduling-controllers # Owner info adds a configmap to the kubernetes cluster with information on diff --git a/helm/bundles/cortex-manila/templates/kpis.yaml b/helm/bundles/cortex-manila/templates/kpis.yaml index d05910f3b..6289d0262 100644 --- a/helm/bundles/cortex-manila/templates/kpis.yaml +++ b/helm/bundles/cortex-manila/templates/kpis.yaml @@ -38,18 +38,6 @@ spec: --- apiVersion: cortex.cloud/v1alpha1 kind: KPI -metadata: - name: cortex-manila-decision-state -spec: - schedulingDomain: manila - impl: decision_state_kpi - opts: - decisionSchedulingDomain: manila - description: | - This aggregated KPI tracks the state of decision resources managed by cortex. ---- -apiVersion: cortex.cloud/v1alpha1 -kind: KPI metadata: name: cortex-manila-kpi-state spec: diff --git a/helm/bundles/cortex-manila/values.yaml b/helm/bundles/cortex-manila/values.yaml index 5d48da8f8..39f177f6d 100644 --- a/helm/bundles/cortex-manila/values.yaml +++ b/helm/bundles/cortex-manila/values.yaml @@ -90,8 +90,6 @@ cortex: &cortex apiservers: home: gvks: - - cortex.cloud/v1alpha1/Decision - - cortex.cloud/v1alpha1/DecisionList - cortex.cloud/v1alpha1/Descheduling - cortex.cloud/v1alpha1/DeschedulingList - cortex.cloud/v1alpha1/Pipeline diff --git a/helm/bundles/cortex-nova/Chart.yaml b/helm/bundles/cortex-nova/Chart.yaml index 938021e99..94864e2ec 100644 --- a/helm/bundles/cortex-nova/Chart.yaml +++ b/helm/bundles/cortex-nova/Chart.yaml @@ -5,23 +5,23 @@ apiVersion: v2 name: cortex-nova description: A Helm chart deploying Cortex for Nova. type: application -version: 0.0.110 +version: 0.0.111 appVersion: 0.1.0 dependencies: # from: file://../../library/cortex-postgres - name: cortex-postgres repository: oci://ghcr.io/cobaltcore-dev/cortex/charts - version: 0.6.30 + version: 0.6.31 # from: file://../../library/cortex - name: cortex repository: oci://ghcr.io/cobaltcore-dev/cortex/charts - version: 0.7.3 + version: 0.8.0 alias: cortex-knowledge-controllers # from: file://../../library/cortex - name: cortex repository: oci://ghcr.io/cobaltcore-dev/cortex/charts - version: 0.7.3 + version: 0.8.0 alias: cortex-scheduling-controllers # Owner info adds a configmap to the kubernetes cluster with information on diff --git a/helm/bundles/cortex-nova/templates/alerts.yaml b/helm/bundles/cortex-nova/templates/alerts.yaml index bb82e889a..696446391 100644 --- a/helm/bundles/cortex-nova/templates/alerts.yaml +++ b/helm/bundles/cortex-nova/templates/alerts.yaml @@ -278,46 +278,6 @@ spec: configuration. It is recommended to investigate the knowledge status and logs for more details. - - alert: CortexNovaDecisionsWithErrors - expr: cortex_decision_state{domain="nova",state="error"} > 0 - for: 5m - labels: -{{- if .Values.alerts.persesDashboardBaseURL }} - persesDashboard: "{{ .Values.alerts.persesDashboardBaseURL }}/cortex_status?var-service=cortex-nova-metrics" -{{- end }} - context: decisions - service: cortex - severity: warning - support_group: workload-management - annotations: - summary: "Some decisions are in error state for operator `{{ "{{" }} $labels.operator {{ "}}" }}`" - description: > - The cortex scheduling pipeline generated decisions that are in error state. - This may indicate issues with the decision logic or the underlying infrastructure. - It is recommended to investigate the decision logs and the state of the - VMs being processed. - - - alert: CortexNovaTooManyDecisionsWaiting - expr: cortex_decision_state{domain="nova",state="waiting"} > 10 - for: 5m - labels: -{{- if .Values.alerts.persesDashboardBaseURL }} - persesDashboard: "{{ .Values.alerts.persesDashboardBaseURL }}/cortex_status?var-service=cortex-nova-metrics" -{{- end }} - context: decisions - service: cortex - severity: warning - support_group: workload-management - annotations: - summary: "Too many decisions are in waiting state for operator `{{ "{{" }} $labels.operator {{ "}}" }}`" - description: > - The cortex scheduling pipeline has a high number of decisions for which - no target host has been assigned yet. - - This may indicate a backlog in processing or issues with the decision logic. - It is recommended to investigate the decision logs and the state of the - VMs being processed. - - alert: CortexNovaKPIUnready expr: | cortex_kpi_state{domain="nova",state!="ready"} != 0 @@ -396,31 +356,6 @@ spec: Investigate the hypervisor operator, the nova build pipeline, and the in-flight reservation controller logs. - - alert: CortexNovaImagePropertiesHypervisorTypeUndetermined - expr: | - sum by (pipeline, step, intent) (rate(cortex_filter_weigher_pipeline_step_events_total{service="cortex-nova-metrics", event="image_properties_hv_type_undetermined"}[5m])) > 0.1 - for: 15m - labels: -{{- if .Values.alerts.persesDashboardBaseURL }} - persesDashboard: "{{ .Values.alerts.persesDashboardBaseURL }}/cortex_status?var-service=cortex-nova-metrics" -{{- end }} - context: scheduling - service: cortex - severity: warning - support_group: workload-management - playbook: docs/support/playbook/cortex/alerts/scheduling - annotations: - summary: "Nova scheduling frequently cannot determine the hypervisor type from image properties" - description: > - The `filter_image_properties` step in pipeline `{{ "{{" }} $labels.pipeline {{ "}}" }}` - is frequently unable to determine the hypervisor type from the image properties - of incoming scheduling requests for intent `{{ "{{" }} $labels.intent {{ "}}" }}`. - In this case the filter is skipped and all hosts are returned, so kvm-only - images may be placed on non-kvm hypervisors (or vice versa). This may indicate - that images are missing the expected hypervisor type property, or that the - property format has changed. Investigate the image metadata of the affected - requests. - - alert: CortexNovaCapabilitiesUnknownHypervisorType expr: | sum by (pipeline, step, hypervisor_type) (rate(cortex_filter_weigher_pipeline_step_events_total{service="cortex-nova-metrics", event="filter_capabilities_unknown_hypervisor_type"}[5m])) > 0.1 @@ -1037,10 +972,12 @@ spec: # completes — in all cases the failover host reservations are no longer # being maintained and will silently drift from the cluster. A pod that is # down is covered by CortexNovaSchedulingDown, so require the series to - # still be exported (the metric is present but flat). + # still be exported (the metric is present but flat). The window is 60m so + # a slow pass (e.g. partitioning over a large, flavor-rich fleet) does not + # trip the alert. expr: | - increase(cortex_failover_host_calculation_runs_total{service="cortex-nova-metrics"}[30m]) == 0 - for: 30m + increase(cortex_failover_host_calculation_runs_total{service="cortex-nova-metrics"}[60m]) == 0 + for: 60m labels: {{- if .Values.alerts.persesDashboardBaseURL }} persesDashboard: "{{ .Values.alerts.persesDashboardBaseURL }}/cortex_status?var-service=cortex-nova-metrics" @@ -1054,7 +991,7 @@ spec: summary: "Failover host calculation loop is not completing passes" description: > The failover host controller has not completed a calculation-loop pass in - the last 30 minutes. Its failover host reservations are no longer being + the last 60 minutes. Its failover host reservations are no longer being recomputed and will drift from the actual cluster state. Check the failover-host-controller logs for a stuck loop or repeated aborts. - alert: CortexNovaFailoverHostCalculationAborting diff --git a/helm/bundles/cortex-nova/templates/kpis.yaml b/helm/bundles/cortex-nova/templates/kpis.yaml index 4c8dee11a..d52b6099e 100644 --- a/helm/bundles/cortex-nova/templates/kpis.yaml +++ b/helm/bundles/cortex-nova/templates/kpis.yaml @@ -110,18 +110,6 @@ spec: --- apiVersion: cortex.cloud/v1alpha1 kind: KPI -metadata: - name: cortex-nova-decision-state -spec: - schedulingDomain: nova - impl: decision_state_kpi - opts: - decisionSchedulingDomain: nova - description: | - This aggregated KPI tracks the state of decision resources managed by cortex. ---- -apiVersion: cortex.cloud/v1alpha1 -kind: KPI metadata: name: cortex-nova-kpi-state spec: diff --git a/helm/bundles/cortex-nova/templates/pipelines_kvm.yaml b/helm/bundles/cortex-nova/templates/pipelines_kvm.yaml index 817e75ab6..fd631a303 100644 --- a/helm/bundles/cortex-nova/templates/pipelines_kvm.yaml +++ b/helm/bundles/cortex-nova/templates/pipelines_kvm.yaml @@ -15,7 +15,6 @@ spec: Specifically, this pipeline is used for general purpose workloads. type: filter-weigher # Fetch all placement candidates, ignoring nova's preselection. - ignorePreselection: true filters: - name: filter_status_conditions description: | @@ -121,6 +120,8 @@ spec: requests without headroom, add `params: [{key: enforce, boolValue: true}]` to this step. The shadow default is intentional so that newly-rolled-out releases never silently start rejecting requests. + params: + - {key: crStateFilter, stringListValue: {{ .Values.kvm.crStateFilter | toJson }}} weighers: - name: kvm_prefer_smaller_hosts params: @@ -189,7 +190,6 @@ spec: Specifically, this pipeline is used for hana virtual machines. type: filter-weigher # Fetch all placement candidates, ignoring nova's preselection. - ignorePreselection: true filters: - name: filter_status_conditions description: | @@ -295,6 +295,8 @@ spec: requests without headroom, add `params: [{key: enforce, boolValue: true}]` to this step. The shadow default is intentional so that newly-rolled-out releases never silently start rejecting requests. + params: + - {key: crStateFilter, stringListValue: {{ .Values.kvm.crStateFilter | toJson }}} weighers: - name: kvm_prefer_smaller_hosts params: @@ -355,7 +357,6 @@ spec: Deprecated: no longer used by the failover reservation controller. Kept for backwards compatibility with deployments that reference this pipeline by name in Helm values. type: filter-weigher - ignorePreselection: true filters: - name: filter_status_conditions description: | @@ -637,7 +638,6 @@ spec: considered. type: filter-weigher # Fetch all placement candidates, ignoring nova's preselection. - ignorePreselection: true filters: - name: filter_status_conditions description: | diff --git a/helm/bundles/cortex-nova/values.yaml b/helm/bundles/cortex-nova/values.yaml index 755bb8f42..0ef1ace0b 100644 --- a/helm/bundles/cortex-nova/values.yaml +++ b/helm/bundles/cortex-nova/values.yaml @@ -109,6 +109,12 @@ kvm: # Domain names that are explicitly treated as non-external-customer domains and # skipped by the filter, even if they match one of the prefixes above. ignoredDomainNames: [] + # CommittedResource states that count toward the CR portion of project quota. + # Shared by the quota controller (PaygUsage derivation) and the + # filter_quota_enforcement pipeline step (CR deduction) — the two MUST agree, + # so this single value feeds both. Empty means no CR states count. + crStateFilter: &crStateFilter + - confirmed cortex: &cortex crd: {enable: false} @@ -118,11 +124,12 @@ cortex: &cortex prometheus: {enable: false} conf: &cortexConf schedulingDomain: nova + # Consumed by the quota controller; mirrors the filter_quota_enforcement + # step's crStateFilter. See kvm.crStateFilter for the shared source. + crStateFilter: *crStateFilter apiservers: home: gvks: - - cortex.cloud/v1alpha1/Decision - - cortex.cloud/v1alpha1/DecisionList - cortex.cloud/v1alpha1/Descheduling - cortex.cloud/v1alpha1/DeschedulingList - cortex.cloud/v1alpha1/Pipeline @@ -230,14 +237,9 @@ cortex-scheduling-controllers: "*": "kvm-general-purpose-load-balancing" pipelineDefault: "kvm-general-purpose-load-balancing" # How often to re-verify active Reservation CRDs (healthy state) - requeueIntervalActive: "5m" + requeueIntervalActive: "30m" # Back-off interval when knowledge is unavailable requeueIntervalRetry: "1m" - # Back-off interval while a VM allocation is still within allocationGracePeriod - requeueIntervalGracePeriod: "1m" - # How long after a VM is allocated to a reservation before it is expected to appear - # on the target host; allocations not confirmed within this window are removed - allocationGracePeriod: "15m" # How long to wait after detecting host over-subscription before evicting reservation slots. # Gives other controllers (e.g. failover) time to self-heal. oversubscriptionGracePeriod: "3m" @@ -392,10 +394,6 @@ cortex-scheduling-controllers: # How often the failover partitions are recalculated and the reservations # updated to match. Must be at least 1m. generationInterval: 5m - # URL of the nova external scheduler API, used to discover which hosts a - # given host's workload could be evacuated onto. Typically the same - # scheduler endpoint the flavor-based failover controller uses. - externalSchedulerURL: "http://localhost:8080/scheduler/nova/external" cortex-knowledge-controllers: <<: *cortex diff --git a/helm/bundles/cortex-placement-shim/Chart.yaml b/helm/bundles/cortex-placement-shim/Chart.yaml index 10cfd29ac..8da34e5fe 100644 --- a/helm/bundles/cortex-placement-shim/Chart.yaml +++ b/helm/bundles/cortex-placement-shim/Chart.yaml @@ -5,13 +5,13 @@ apiVersion: v2 name: cortex-placement-shim description: A Helm chart deploying the Cortex placement shim. type: application -version: 0.1.35 +version: 0.1.36 appVersion: 0.1.0 dependencies: # from: file://../../library/cortex-shim - name: cortex-shim repository: oci://ghcr.io/cobaltcore-dev/cortex/charts - version: 0.1.35 + version: 0.1.36 # Owner info adds a configmap to the kubernetes cluster with information on # the service owner. This makes it easier to find out who to contact in case # of issues. See: https://github.com/sapcc/helm-charts/pkgs/container/helm-charts%2Fowner-info diff --git a/helm/bundles/cortex-pods/Chart.yaml b/helm/bundles/cortex-pods/Chart.yaml index 707912dd2..0448c2d92 100644 --- a/helm/bundles/cortex-pods/Chart.yaml +++ b/helm/bundles/cortex-pods/Chart.yaml @@ -5,13 +5,13 @@ apiVersion: v2 name: cortex-pods description: A Helm chart deploying Cortex for Pods. type: application -version: 0.0.110 +version: 0.0.111 appVersion: 0.1.0 dependencies: # from: file://../../library/cortex - name: cortex repository: oci://ghcr.io/cobaltcore-dev/cortex/charts - version: 0.7.3 + version: 0.8.0 # Owner info adds a configmap to the kubernetes cluster with information on # the service owner. This makes it easier to find out who to contact in case diff --git a/helm/bundles/cortex-pods/values.yaml b/helm/bundles/cortex-pods/values.yaml index efd5ee9b1..3815f67b8 100644 --- a/helm/bundles/cortex-pods/values.yaml +++ b/helm/bundles/cortex-pods/values.yaml @@ -33,8 +33,6 @@ cortex: apiservers: home: gvks: - - cortex.cloud/v1alpha1/Decision - - cortex.cloud/v1alpha1/DecisionList - cortex.cloud/v1alpha1/Descheduling - cortex.cloud/v1alpha1/DeschedulingList - cortex.cloud/v1alpha1/Pipeline diff --git a/helm/dev/cortex-prometheus-operator/Chart.yaml b/helm/dev/cortex-prometheus-operator/Chart.yaml index 01a302f80..f1dfa9090 100644 --- a/helm/dev/cortex-prometheus-operator/Chart.yaml +++ b/helm/dev/cortex-prometheus-operator/Chart.yaml @@ -10,4 +10,4 @@ dependencies: # CRDs of the prometheus operator, such as PrometheusRule, ServiceMonitor, etc. - name: kube-prometheus-stack repository: oci://ghcr.io/prometheus-community/charts - version: 92.17.0 + version: 92.18.0 diff --git a/helm/library/cortex-postgres/Chart.yaml b/helm/library/cortex-postgres/Chart.yaml index 15082460c..14584b312 100644 --- a/helm/library/cortex-postgres/Chart.yaml +++ b/helm/library/cortex-postgres/Chart.yaml @@ -5,5 +5,6 @@ apiVersion: v2 name: cortex-postgres description: Postgres setup for Cortex. type: application -version: 0.6.30 -appVersion: "sha-7f53e4df" +version: 0.6.31 +appVersion: "sha-e2e2980d" + diff --git a/helm/library/cortex-shim/Chart.yaml b/helm/library/cortex-shim/Chart.yaml index c0e7bbb03..0c04131e2 100644 --- a/helm/library/cortex-shim/Chart.yaml +++ b/helm/library/cortex-shim/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: cortex-shim description: A Helm chart to distribute cortex shims. type: application -version: 0.1.35 -appVersion: "sha-63f36907" +version: 0.1.36 +appVersion: "sha-821050ce" icon: "https://example.com/icon.png" dependencies: [] diff --git a/helm/library/cortex-shim/templates/clusterrole.yaml b/helm/library/cortex-shim/templates/clusterrole.yaml index 74f8e7ad4..867b45ede 100644 --- a/helm/library/cortex-shim/templates/clusterrole.yaml +++ b/helm/library/cortex-shim/templates/clusterrole.yaml @@ -14,7 +14,6 @@ rules: - knowledges - datasources - reservations - - decisions - deschedulings - pipelines - kpis @@ -33,7 +32,6 @@ rules: - knowledges/finalizers - datasources/finalizers - reservations/finalizers - - decisions/finalizers - deschedulings/finalizers - pipelines/finalizers - kpis/finalizers @@ -46,7 +44,6 @@ rules: - knowledges/status - datasources/status - reservations/status - - decisions/status - deschedulings/status - pipelines/status - kpis/status diff --git a/helm/library/cortex/Chart.yaml b/helm/library/cortex/Chart.yaml index fa068a3db..20180b536 100644 --- a/helm/library/cortex/Chart.yaml +++ b/helm/library/cortex/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: cortex description: A Helm chart to distribute cortex. type: application -version: 0.7.3 -appVersion: "sha-b74c4f13" +version: 0.8.0 +appVersion: "sha-06024c69" icon: "https://example.com/icon.png" dependencies: [] diff --git a/helm/library/cortex/files/crds/cortex.cloud_committedresources.yaml b/helm/library/cortex/files/crds/cortex.cloud_committedresources.yaml index f14556733..0ece2ce2a 100644 --- a/helm/library/cortex/files/crds/cortex.cloud_committedresources.yaml +++ b/helm/library/cortex/files/crds/cortex.cloud_committedresources.yaml @@ -270,11 +270,44 @@ spec: type: object assignedInstances: description: |- + Deprecated: use Status.Billing.AssignedInstances AssignedInstances holds the UUIDs of VM instances deterministically assigned to this committed resource. Populated by the usage reconciler; used to compute UsedResources and drive the quota controller. items: type: string type: array + billing: + description: |- + Billing holds the computed billing assignment for this commitment. + Written by the UsageReconciler. Use this instead of the deprecated flat billing fields. + properties: + assignedInstances: + description: AssignedInstances lists the VM UUIDs assigned to + this commitment for billing. + items: + type: string + type: array + lastReconcileAt: + description: LastReconcileAt is the time of the last successful + billing reconcile. + format: date-time + type: string + observedGeneration: + description: ObservedGeneration is the CommittedResource generation + seen during the last billing reconcile. + format: int64 + type: integer + usedResources: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: UsedResources is the total resource consumption from + assigned instances. + type: object + type: object conditions: description: Conditions holds the current status conditions. items: @@ -338,8 +371,9 @@ spec: format: date-time type: string lastUsageReconcileAt: - description: LastUsageReconcileAt is when the usage reconciler last - updated AssignedInstances and UsedResources. + description: |- + Deprecated: use Status.Billing.LastReconcileAt + LastUsageReconcileAt is when the usage reconciler last updated AssignedInstances and UsedResources. format: date-time type: string statusSummary: @@ -349,6 +383,7 @@ spec: type: string usageObservedGeneration: description: |- + Deprecated: use Status.Billing.ObservedGeneration UsageObservedGeneration is the CR generation that the usage reconciler last processed. Follows the Kubernetes observedGeneration pattern: when this differs from metadata.generation the cooldown is bypassed so spec changes (e.g. shrink) are reflected @@ -364,6 +399,7 @@ spec: pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: |- + Deprecated: use Status.Billing.UsedResources UsedResources is the total resource consumption of assigned VM instances, keyed by resource type (e.g. "memory" in MiB binary SI, "cpu" as core count). Populated by the usage reconciler. type: object diff --git a/helm/library/cortex/files/crds/cortex.cloud_decisions.yaml b/helm/library/cortex/files/crds/cortex.cloud_decisions.yaml deleted file mode 100644 index ec4c99649..000000000 --- a/helm/library/cortex/files/crds/cortex.cloud_decisions.yaml +++ /dev/null @@ -1,412 +0,0 @@ ---- -apiVersion: apiextensions.k8s.io/v1 -kind: CustomResourceDefinition -metadata: - annotations: - controller-gen.kubebuilder.io/version: v0.23.0 - name: decisions.cortex.cloud -spec: - group: cortex.cloud - names: - kind: Decision - listKind: DecisionList - plural: decisions - singular: decision - scope: Cluster - versions: - - additionalPrinterColumns: - - jsonPath: .spec.schedulingDomain - name: Domain - type: string - - jsonPath: .spec.resourceID - name: Resource ID - type: string - - jsonPath: .status.precedence - name: '#' - type: string - - jsonPath: .metadata.creationTimestamp - name: Created - type: date - - jsonPath: .spec.pipelineRef.name - name: Pipeline - type: string - - jsonPath: .status.result.targetHost - name: TargetHost - type: string - - jsonPath: .status.conditions[?(@.type=='Ready')].status - name: Ready - type: string - name: v1alpha1 - schema: - openAPIV3Schema: - description: |- - Currently the Decision CRD is an in-memory scheduling object used by the external scheduler API - and filter-weigher pipelines to compute a placement result. It is currently NOT persisted - to etcd — the scheduling outcome (including a human-readable explanation) is written onto the - scheduled resource instead, e.g. the nova VM's spec.cortex fields. - The long-term shape of this CRD is still under discussion; until that is - settled the Decision serves only as a transient carrier within a single - scheduling run. - properties: - apiVersion: - description: |- - APIVersion defines the versioned schema of this representation of an object. - Servers should convert recognized schemas to the latest internal value, and - may reject unrecognized values. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - type: string - kind: - description: |- - Kind is a string value representing the REST resource this object represents. - Servers may infer this from the endpoint the client submits requests to. - Cannot be updated. - In CamelCase. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - type: string - metadata: - type: object - spec: - description: spec defines the desired state of Decision - properties: - cinderRaw: - description: If the type is "cinder", this field contains the raw - cinder decision request. - type: object - x-kubernetes-preserve-unknown-fields: true - intent: - description: The intent of the scheduling decision (e.g., initial - scheduling, rescheduling, etc.). - type: string - machineRef: - description: If the type is "machine", this field contains the machine - reference. - properties: - apiVersion: - description: API version of the referent. - type: string - fieldPath: - description: |- - If referring to a piece of an object instead of an entire object, this string - should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. - For example, if the object reference is to a container within a pod, this would take on a value like: - "spec.containers{name}" (where "name" refers to the name of the container that triggered - the event) or if no container name is specified "spec.containers[2]" (container with - index 2 in this pod). This syntax is chosen only to have some well-defined way of - referencing a part of an object. - type: string - kind: - description: |- - Kind of the referent. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - type: string - name: - description: |- - Name of the referent. - More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names - type: string - namespace: - description: |- - Namespace of the referent. - More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ - type: string - resourceVersion: - description: |- - Specific resourceVersion to which this reference is made, if any. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency - type: string - uid: - description: |- - UID of the referent. - More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids - type: string - type: object - x-kubernetes-map-type: atomic - manilaRaw: - description: If the type is "manila", this field contains the raw - manila decision request. - type: object - x-kubernetes-preserve-unknown-fields: true - novaRaw: - description: If the type is "nova", this field contains the raw nova - decision request. - type: object - x-kubernetes-preserve-unknown-fields: true - pipelineRef: - description: |- - A reference to the pipeline that should be used for this decision. - This reference can be used to look up the pipeline definition and its - scheduler step configuration for additional context. - properties: - apiVersion: - description: API version of the referent. - type: string - fieldPath: - description: |- - If referring to a piece of an object instead of an entire object, this string - should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. - For example, if the object reference is to a container within a pod, this would take on a value like: - "spec.containers{name}" (where "name" refers to the name of the container that triggered - the event) or if no container name is specified "spec.containers[2]" (container with - index 2 in this pod). This syntax is chosen only to have some well-defined way of - referencing a part of an object. - type: string - kind: - description: |- - Kind of the referent. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - type: string - name: - description: |- - Name of the referent. - More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names - type: string - namespace: - description: |- - Namespace of the referent. - More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ - type: string - resourceVersion: - description: |- - Specific resourceVersion to which this reference is made, if any. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency - type: string - uid: - description: |- - UID of the referent. - More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids - type: string - type: object - x-kubernetes-map-type: atomic - podRef: - description: If the type is "pod", this field contains the pod reference. - properties: - apiVersion: - description: API version of the referent. - type: string - fieldPath: - description: |- - If referring to a piece of an object instead of an entire object, this string - should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. - For example, if the object reference is to a container within a pod, this would take on a value like: - "spec.containers{name}" (where "name" refers to the name of the container that triggered - the event) or if no container name is specified "spec.containers[2]" (container with - index 2 in this pod). This syntax is chosen only to have some well-defined way of - referencing a part of an object. - type: string - kind: - description: |- - Kind of the referent. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - type: string - name: - description: |- - Name of the referent. - More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names - type: string - namespace: - description: |- - Namespace of the referent. - More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ - type: string - resourceVersion: - description: |- - Specific resourceVersion to which this reference is made, if any. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency - type: string - uid: - description: |- - UID of the referent. - More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids - type: string - type: object - x-kubernetes-map-type: atomic - resourceID: - description: |- - An identifier for the underlying resource to be scheduled. - For example, this can be the UUID of a nova instance or cinder volume. - This can be used to correlate multiple decisions for the same resource. - type: string - schedulingDomain: - description: |- - SchedulingDomain defines in which scheduling domain this decision - was or is processed (e.g., nova, cinder, manila). - type: string - required: - - pipelineRef - - resourceID - - schedulingDomain - type: object - status: - description: status defines the observed state of Decision - properties: - conditions: - description: The current status conditions of the decision. - items: - description: Condition contains details for one aspect of the current - state of this API Resource. - properties: - lastTransitionTime: - description: |- - lastTransitionTime is the last time the condition transitioned from one status to another. - This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. - format: date-time - type: string - message: - description: |- - message is a human readable message indicating details about the transition. - This may be an empty string. - maxLength: 32768 - type: string - observedGeneration: - description: |- - observedGeneration represents the .metadata.generation that the condition was set based upon. - For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date - with respect to the current state of the instance. - format: int64 - minimum: 0 - type: integer - reason: - description: |- - reason contains a programmatic identifier indicating the reason for the condition's last transition. - Producers of specific condition types may define expected values and meanings for this field, - and whether the values are considered a guaranteed API. - The value should be a CamelCase string. - This field may not be empty. - maxLength: 1024 - minLength: 1 - pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ - type: string - status: - description: status of the condition, one of True, False, Unknown. - enum: - - "True" - - "False" - - Unknown - type: string - type: - description: type of condition in CamelCase or in foo.example.com/CamelCase. - maxLength: 316 - pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ - type: string - required: - - lastTransitionTime - - message - - reason - - status - - type - type: object - type: array - explanation: - description: A human-readable explanation of the decision result. - type: string - history: - description: |- - If there were previous decisions for the underlying resource, they can - be resolved here to provide historical context for the decision. - items: - description: ObjectReference contains enough information to let - you inspect or modify the referred object. - properties: - apiVersion: - description: API version of the referent. - type: string - fieldPath: - description: |- - If referring to a piece of an object instead of an entire object, this string - should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. - For example, if the object reference is to a container within a pod, this would take on a value like: - "spec.containers{name}" (where "name" refers to the name of the container that triggered - the event) or if no container name is specified "spec.containers[2]" (container with - index 2 in this pod). This syntax is chosen only to have some well-defined way of - referencing a part of an object. - type: string - kind: - description: |- - Kind of the referent. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - type: string - name: - description: |- - Name of the referent. - More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names - type: string - namespace: - description: |- - Namespace of the referent. - More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ - type: string - resourceVersion: - description: |- - Specific resourceVersion to which this reference is made, if any. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency - type: string - uid: - description: |- - UID of the referent. - More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids - type: string - type: object - x-kubernetes-map-type: atomic - type: array - precedence: - description: The number of decisions that preceded this one for the - same resource. - type: integer - result: - description: The result of this decision. - properties: - aggregatedOutWeights: - additionalProperties: - type: number - description: Aggregated output weights from the pipeline. - type: object - normalizedInWeights: - additionalProperties: - type: number - description: Normalized input weights to the pipeline. - type: object - orderedHosts: - description: Final ordered list of hosts from most preferred to - least preferred. - items: - type: string - type: array - rawInWeights: - additionalProperties: - type: number - description: Raw input weights to the pipeline. - type: object - stepResults: - description: |- - Outputs of the decision pipeline including the activations used - to make the final ordering of compute hosts. - items: - properties: - activations: - additionalProperties: - type: number - description: Activations of the step for each host. - type: object - stepName: - description: object reference to the scheduler step. - type: string - required: - - activations - - stepName - type: object - type: array - targetHost: - description: The first element of the ordered hosts is considered - the target host. - type: string - type: object - type: object - required: - - spec - type: object - selectableFields: - - jsonPath: .spec.resourceID - served: true - storage: true - subresources: - status: {} diff --git a/helm/library/cortex/files/crds/cortex.cloud_pipelines.yaml b/helm/library/cortex/files/crds/cortex.cloud_pipelines.yaml index 21a69b62f..f77625aaf 100644 --- a/helm/library/cortex/files/crds/cortex.cloud_pipelines.yaml +++ b/helm/library/cortex/files/crds/cortex.cloud_pipelines.yaml @@ -180,13 +180,6 @@ spec: - name type: object type: array - ignorePreselection: - default: false - description: |- - If this pipeline should ignore host preselection and gather all - available placement candidates before applying filters, instead of - relying on a pre-filtered set and weights. - type: boolean schedulingDomain: description: |- SchedulingDomain defines in which scheduling domain this pipeline diff --git a/helm/library/cortex/files/crds/cortex.cloud_reservations.yaml b/helm/library/cortex/files/crds/cortex.cloud_reservations.yaml index 29152f333..24d7a2b41 100644 --- a/helm/library/cortex/files/crds/cortex.cloud_reservations.yaml +++ b/helm/library/cortex/files/crds/cortex.cloud_reservations.yaml @@ -24,21 +24,30 @@ spec: - jsonPath: .status.conditions[?(@.type=='Ready')].status name: Ready type: string - - jsonPath: .spec.committedResourceReservation.resourceGroup - name: ResourceGroup + - jsonPath: .status.conditions[?(@.type=='Ready')].observedGeneration + name: ObservedGen priority: 1 - type: string - - jsonPath: .spec.failoverReservation.resourceGroup - name: HA ResourceGroup + type: integer + - jsonPath: .metadata.generation + name: Gen priority: 1 + type: integer + - jsonPath: .spec.*.resourceGroup + name: ResourceGroup type: string - - jsonPath: .spec.committedResourceReservation.projectID + - jsonPath: .spec.*.resourceName + name: ResourceName + type: string + - jsonPath: .spec.*.projectID name: Project priority: 1 type: string + - jsonPath: .spec.committedResourceReservation.commitmentUUID + name: CommitmentID + priority: 1 + type: string - jsonPath: .spec.availabilityZone name: AZ - priority: 1 type: string - jsonPath: .spec.startTime name: StartTime @@ -50,7 +59,6 @@ spec: type: string - jsonPath: .spec.resources name: Resources - priority: 1 type: string - jsonPath: .status.failoverReservation.lastChanged name: LastChanged @@ -60,14 +68,22 @@ spec: name: AcknowledgedAt priority: 1 type: date - - jsonPath: .status.committedResourceReservation.allocations - name: CR Allocations + - jsonPath: .status.committedResourceReservation.allocation.*.objectRef.name + name: CR alloc VMs + priority: 1 + type: string + - jsonPath: .status.committedResourceReservation.allocation.*.resources + name: CR alloc resources priority: 1 type: string - jsonPath: .status.failoverReservation.allocations name: HA Allocations priority: 1 type: string + - jsonPath: .metadata.annotations['reservations\.cortex\.cloud/candidate-allocation'] + name: CR CandidateVM + priority: 1 + type: string name: v1alpha1 schema: openAPIV3Schema: @@ -109,10 +125,55 @@ spec: The workload could be a VM (Nova/IronCore), Pod (Kubernetes), or other resource. properties: creationTimestamp: - description: Timestamp when this workload was assigned to - the reservation. + description: 'Deprecated: CreationTimestamp is no longer + populated. Will be removed once old pods are fully rolled + out.' format: date-time type: string + objectRef: + description: ObjectRef is a reference to the workload object + (e.g. VM CRD). + properties: + apiVersion: + description: API version of the referent. + type: string + fieldPath: + description: |- + If referring to a piece of an object instead of an entire object, this string + should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. + For example, if the object reference is to a container within a pod, this would take on a value like: + "spec.containers{name}" (where "name" refers to the name of the container that triggered + the event) or if no container name is specified "spec.containers[2]" (container with + index 2 in this pod). This syntax is chosen only to have some well-defined way of + referencing a part of an object. + type: string + kind: + description: |- + Kind of the referent. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + name: + description: |- + Name of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + type: string + namespace: + description: |- + Namespace of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ + type: string + resourceVersion: + description: |- + Specific resourceVersion to which this reference is made, if any. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency + type: string + uid: + description: |- + UID of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids + type: string + type: object + x-kubernetes-map-type: atomic resources: additionalProperties: anyOf: @@ -120,16 +181,17 @@ spec: - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true - description: Resources consumed by this instance. + description: Resources consumed by this workload. type: object required: - - creationTimestamp - resources type: object description: |- Allocations maps workload identifiers to their allocation details. Key: Workload UUID (VM UUID for Nova, Pod UID for Pods, Machine UID for IronCore, etc.) Value: allocation state and metadata + + Deprecated: use status.committedResourceReservation.allocation instead. Will be removed once old pods are fully rolled out. type: object commitmentUUID: description: CommitmentUUID is the UUID of the commitment that @@ -150,8 +212,16 @@ spec: once it has processed the reservation, allowing the CR controller to wait until all child reservations are up-to-date before accepting. Zero means the field is not set (syncer-created reservations, no parent CR). + + Deprecated: replaced by conditions[Ready].ObservedGeneration signaling. Will be removed once old pods are fully rolled out. format: int64 type: integer + primaryResource: + description: |- + PrimaryResource is the resource dimension that drives allocation tracking + (e.g. hv1.ResourceMemory for memory commitments). Other dimensions in + alloc.Resources are recorded for bookkeeping but do not gate allocation decisions. + type: string projectID: type: string resourceGroup: @@ -273,18 +343,86 @@ spec: CommittedResourceReservation contains status fields specific to committed resource reservations. Only used when Type is CommittedResourceReservation. properties: - allocations: + allocation: additionalProperties: - type: string + description: |- + CommittedResourceAllocation represents a workload's assignment to a committed resource reservation slot. + The workload could be a VM (Nova/IronCore), Pod (Kubernetes), or other resource. + properties: + creationTimestamp: + description: 'Deprecated: CreationTimestamp is no longer + populated. Will be removed once old pods are fully rolled + out.' + format: date-time + type: string + objectRef: + description: ObjectRef is a reference to the workload object + (e.g. VM CRD). + properties: + apiVersion: + description: API version of the referent. + type: string + fieldPath: + description: |- + If referring to a piece of an object instead of an entire object, this string + should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. + For example, if the object reference is to a container within a pod, this would take on a value like: + "spec.containers{name}" (where "name" refers to the name of the container that triggered + the event) or if no container name is specified "spec.containers[2]" (container with + index 2 in this pod). This syntax is chosen only to have some well-defined way of + referencing a part of an object. + type: string + kind: + description: |- + Kind of the referent. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + name: + description: |- + Name of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + type: string + namespace: + description: |- + Namespace of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ + type: string + resourceVersion: + description: |- + Specific resourceVersion to which this reference is made, if any. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency + type: string + uid: + description: |- + UID of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids + type: string + type: object + x-kubernetes-map-type: atomic + resources: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: Resources consumed by this workload. + type: object + required: + - resources + type: object description: |- - Allocations maps VM/instance UUIDs to the host they are currently running on. - Key: VM/instance UUID, Value: Host name where the VM is currently running. + Allocation maps workload identifiers to their allocation details. + Key: workload UUID (vmUUID for virtual machines) + Value: allocation details including resource consumption and object reference. type: object observedParentGeneration: description: |- ObservedParentGeneration is the Spec.CommittedResourceReservation.ParentGeneration value that this Reservation controller last processed. When it matches ParentGeneration in spec, the CR controller knows this reservation is up-to-date for the current CR spec version. + + Deprecated: replaced by conditions[Ready].ObservedGeneration signaling. Will be removed once old pods are fully rolled out. format: int64 type: integer type: object diff --git a/helm/library/cortex/templates/rbac/role.yaml b/helm/library/cortex/templates/rbac/role.yaml index 54991d0b4..16e6b9012 100644 --- a/helm/library/cortex/templates/rbac/role.yaml +++ b/helm/library/cortex/templates/rbac/role.yaml @@ -17,7 +17,6 @@ rules: - committedresourcechanges - projectquotas - flavorgroupcapacities - - decisions - deschedulings - pipelines - kpis @@ -40,7 +39,6 @@ rules: - committedresourcechanges/finalizers - projectquotas/finalizers - flavorgroupcapacities/finalizers - - decisions/finalizers - deschedulings/finalizers - pipelines/finalizers - kpis/finalizers @@ -57,7 +55,6 @@ rules: - committedresourcechanges/status - projectquotas/status - flavorgroupcapacities/status - - decisions/status - deschedulings/status - pipelines/status - kpis/status diff --git a/internal/knowledge/extractor/plugins/compute/flavor_groups_knowledge.go b/internal/knowledge/extractor/plugins/compute/flavor_groups_knowledge.go new file mode 100644 index 000000000..88595821a --- /dev/null +++ b/internal/knowledge/extractor/plugins/compute/flavor_groups_knowledge.go @@ -0,0 +1,72 @@ +// Copyright SAP SE +// SPDX-License-Identifier: Apache-2.0 + +package compute + +import ( + "context" + "errors" + "fmt" + + "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "k8s.io/apimachinery/pkg/api/meta" + "k8s.io/apimachinery/pkg/types" + "sigs.k8s.io/controller-runtime/pkg/client" +) + +// GetFlavorGroupsKnowledge retrieves the flavor groups Knowledge CRD and returns +// it if ready. Returns nil, nil if the CRD exists but is not ready yet. +func GetFlavorGroupsKnowledge(ctx context.Context, c client.Client) (*v1alpha1.Knowledge, error) { + knowledge := &v1alpha1.Knowledge{} + // Namespace is empty as Knowledge is cluster-scoped. + if err := c.Get(ctx, types.NamespacedName{Name: "flavor-groups"}, knowledge); err != nil { + return nil, fmt.Errorf("failed to get flavor groups knowledge: %w", err) + } + if meta.IsStatusConditionTrue(knowledge.Status.Conditions, v1alpha1.KnowledgeConditionReady) { + return knowledge, nil + } + return nil, nil +} + +// GetAllFlavorGroups returns all flavor groups as a map keyed by group name, +// skipping any that fail validation. If knowledgeCRD is nil the Knowledge CRD is +// fetched first (erroring if it is not ready); otherwise the provided CRD is used. +func GetAllFlavorGroups(ctx context.Context, c client.Client, knowledgeCRD *v1alpha1.Knowledge) (map[string]FlavorGroupFeature, error) { + if knowledgeCRD == nil { + var err error + knowledgeCRD, err = GetFlavorGroupsKnowledge(ctx, c) + if err != nil { + return nil, err + } + if knowledgeCRD == nil { + return nil, errors.New("flavor groups knowledge is not ready") + } + } + features, err := v1alpha1.UnboxFeatureList[FlavorGroupFeature](knowledgeCRD.Status.Raw) + if err != nil { + return nil, fmt.Errorf("failed to unbox flavor group features: %w", err) + } + flavorGroupMap := make(map[string]FlavorGroupFeature, len(features)) + for _, feature := range features { + if err := feature.Validate(); err != nil { + flavorGroupLog.Error(err, "skipping invalid flavor group from Knowledge CRD") + continue + } + flavorGroupMap[feature.Name] = feature + } + return flavorGroupMap, nil +} + +// FindFlavorInGroups searches all flavor groups for a flavor by name, returning +// the flavor group name and flavor details, or an error if the flavor is not +// found in any group. +func FindFlavorInGroups(flavorName string, flavorGroups map[string]FlavorGroupFeature) (groupName string, flavor *FlavorInGroup, err error) { + for gName, fg := range flavorGroups { + for i, f := range fg.Flavors { + if f.Name == flavorName { + return gName, &fg.Flavors[i], nil + } + } + } + return "", nil, fmt.Errorf("flavor %q not found in any flavor group", flavorName) +} diff --git a/internal/knowledge/kpis/plugins/deployment/decision_state.go b/internal/knowledge/kpis/plugins/deployment/decision_state.go deleted file mode 100644 index 1d07ac7f9..000000000 --- a/internal/knowledge/kpis/plugins/deployment/decision_state.go +++ /dev/null @@ -1,97 +0,0 @@ -// Copyright SAP SE -// SPDX-License-Identifier: Apache-2.0 - -package deployment - -import ( - "context" - - "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "github.com/cobaltcore-dev/cortex/internal/knowledge/db" - "github.com/cobaltcore-dev/cortex/internal/knowledge/kpis/plugins" - "github.com/cobaltcore-dev/cortex/pkg/conf" - "github.com/prometheus/client_golang/prometheus" - "k8s.io/apimachinery/pkg/api/meta" - "sigs.k8s.io/controller-runtime/pkg/client" -) - -type DecisionStateKPIOpts struct { - // The scheduling domain to filter decisions by. - DecisionSchedulingDomain v1alpha1.SchedulingDomain `json:"decisionSchedulingDomain"` -} - -// KPI observing the state of decision resources managed by cortex. -type DecisionStateKPI struct { - // Common base for all KPIs that provides standard functionality. - plugins.BaseKPI[DecisionStateKPIOpts] - - // Prometheus descriptor for the decision state metric. - counter *prometheus.Desc -} - -func (DecisionStateKPI) GetName() string { return "decision_state_kpi" } - -// Initialize the KPI. -func (k *DecisionStateKPI) Init(db *db.DB, client client.Client, opts conf.RawOpts) error { - if err := k.BaseKPI.Init(db, client, opts); err != nil { - return err - } - k.counter = prometheus.NewDesc( - "cortex_decision_state", - "State of cortex managed decisions (aggregated)", - []string{"domain", "state"}, - nil, - ) - return nil -} - -// Conform to the prometheus collector interface by providing the descriptor. -func (k *DecisionStateKPI) Describe(ch chan<- *prometheus.Desc) { ch <- k.counter } - -// Collect the decision state metrics. -func (k *DecisionStateKPI) Collect(ch chan<- prometheus.Metric) { - // Get all decisions with the specified decision operator. - decisionList := &v1alpha1.DecisionList{} - if err := k.Client.List(context.Background(), decisionList); err != nil { - return - } - var decisions []v1alpha1.Decision - for _, d := range decisionList.Items { - if d.Spec.SchedulingDomain != k.Options.DecisionSchedulingDomain { - continue - } - decisions = append(decisions, d) - } - // For each decision, categorize by state: error, waiting, or success - var errorCount, waitingCount, successCount float64 - for _, d := range decisions { - switch { - // Error state: decision has a false Ready condition - case meta.IsStatusConditionFalse(d.Status.Conditions, v1alpha1.DecisionConditionReady): - errorCount++ - // Waiting state: decision has a target host set (waiting for migration/placement) - case d.Status.Result != nil && d.Status.Result.TargetHost != nil: - waitingCount++ - // Success state: decision is complete (has result with ordered hosts or no result needed) - default: - successCount++ - } - } - // Emit metrics for all three states - ch <- prometheus.MustNewConstMetric( - k.counter, prometheus.GaugeValue, errorCount, - string(k.Options.DecisionSchedulingDomain), "error", - ) - ch <- prometheus.MustNewConstMetric( - k.counter, prometheus.GaugeValue, waitingCount, - string(k.Options.DecisionSchedulingDomain), "waiting", - ) - ch <- prometheus.MustNewConstMetric( - k.counter, prometheus.GaugeValue, successCount, - string(k.Options.DecisionSchedulingDomain), "success", - ) -} - -func init() { - plugins.Index["decision_state_kpi"] = func() plugins.KPI { return &DecisionStateKPI{} } -} diff --git a/internal/knowledge/kpis/plugins/deployment/decision_state_test.go b/internal/knowledge/kpis/plugins/deployment/decision_state_test.go deleted file mode 100644 index 0d6bd4568..000000000 --- a/internal/knowledge/kpis/plugins/deployment/decision_state_test.go +++ /dev/null @@ -1,327 +0,0 @@ -// Copyright SAP SE -// SPDX-License-Identifier: Apache-2.0 - -package deployment - -import ( - "testing" - - "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "github.com/cobaltcore-dev/cortex/pkg/conf" - "github.com/prometheus/client_golang/prometheus" - v1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "sigs.k8s.io/controller-runtime/pkg/client/fake" -) - -func TestDecisionStateKPI_Init(t *testing.T) { - kpi := &DecisionStateKPI{} - if err := kpi.Init(nil, nil, conf.NewRawOpts(`{"decisionSchedulingDomain": "test-operator"}`)); err != nil { - t.Fatalf("expected no error, got %v", err) - } -} - -func TestDecisionStateKPI_Collect(t *testing.T) { - scheme, err := v1alpha1.SchemeBuilder.Build() - if err != nil { - t.Fatalf("expected no error, got %v", err) - } - - targetHost := "host1" - - tests := []struct { - name string - decisions []v1alpha1.Decision - operator string - expectedCount int - description string - expectedError int - expectedWaiting int - expectedSuccess int - }{ - { - name: "no decisions", - decisions: []v1alpha1.Decision{}, - operator: "test-operator", - expectedCount: 3, // always emits 3 metrics: error, waiting, success - description: "should collect metrics with zero counts when no decisions exist", - expectedError: 0, - expectedWaiting: 0, - expectedSuccess: 0, - }, - { - name: "single decision in error state", - decisions: []v1alpha1.Decision{ - { - ObjectMeta: v1.ObjectMeta{Name: "dec1"}, - Spec: v1alpha1.DecisionSpec{SchedulingDomain: "test-operator"}, - Status: v1alpha1.DecisionStatus{ - Conditions: []v1.Condition{ - { - Type: v1alpha1.DecisionConditionReady, - Status: v1.ConditionFalse, - }, - }, - }, - }, - }, - operator: "test-operator", - expectedCount: 3, - description: "should count decision in error state", - expectedError: 1, - expectedWaiting: 0, - expectedSuccess: 0, - }, - { - name: "single decision in waiting state", - decisions: []v1alpha1.Decision{ - { - ObjectMeta: v1.ObjectMeta{Name: "dec2"}, - Spec: v1alpha1.DecisionSpec{SchedulingDomain: "test-operator"}, - Status: v1alpha1.DecisionStatus{ - Result: &v1alpha1.DecisionResult{ - TargetHost: &targetHost, - }, - }, - }, - }, - operator: "test-operator", - expectedCount: 3, - description: "should count decision with target host as waiting", - expectedError: 0, - expectedWaiting: 1, - expectedSuccess: 0, - }, - { - name: "single decision in success state", - decisions: []v1alpha1.Decision{ - { - ObjectMeta: v1.ObjectMeta{Name: "dec3"}, - Spec: v1alpha1.DecisionSpec{SchedulingDomain: "test-operator"}, - Status: v1alpha1.DecisionStatus{ - Result: &v1alpha1.DecisionResult{ - OrderedHosts: []string{"host1", "host2"}, - }, - }, - }, - }, - operator: "test-operator", - expectedCount: 3, - description: "should count decision without target host and no error as success", - expectedError: 0, - expectedWaiting: 0, - expectedSuccess: 1, - }, - { - name: "multiple decisions in different states", - decisions: []v1alpha1.Decision{ - { - ObjectMeta: v1.ObjectMeta{Name: "dec-error"}, - Spec: v1alpha1.DecisionSpec{SchedulingDomain: "test-operator"}, - Status: v1alpha1.DecisionStatus{ - Conditions: []v1.Condition{ - { - Type: v1alpha1.DecisionConditionReady, - Status: v1.ConditionFalse, - }, - }, - }, - }, - { - ObjectMeta: v1.ObjectMeta{Name: "dec-waiting"}, - Spec: v1alpha1.DecisionSpec{SchedulingDomain: "test-operator"}, - Status: v1alpha1.DecisionStatus{ - Result: &v1alpha1.DecisionResult{ - TargetHost: &targetHost, - }, - }, - }, - { - ObjectMeta: v1.ObjectMeta{Name: "dec-success"}, - Spec: v1alpha1.DecisionSpec{SchedulingDomain: "test-operator"}, - Status: v1alpha1.DecisionStatus{ - Result: &v1alpha1.DecisionResult{ - OrderedHosts: []string{"host1"}, - }, - }, - }, - }, - operator: "test-operator", - expectedCount: 3, - description: "should correctly count decisions across all states", - expectedError: 1, - expectedWaiting: 1, - expectedSuccess: 1, - }, - { - name: "filter by operator", - decisions: []v1alpha1.Decision{ - { - ObjectMeta: v1.ObjectMeta{Name: "dec-correct-operator"}, - Spec: v1alpha1.DecisionSpec{SchedulingDomain: "test-operator"}, - Status: v1alpha1.DecisionStatus{ - Conditions: []v1.Condition{ - { - Type: v1alpha1.DecisionConditionReady, - Status: v1.ConditionFalse, - }, - }, - }, - }, - { - ObjectMeta: v1.ObjectMeta{Name: "dec-wrong-operator"}, - Spec: v1alpha1.DecisionSpec{SchedulingDomain: "other-operator"}, - Status: v1alpha1.DecisionStatus{ - Conditions: []v1.Condition{ - { - Type: v1alpha1.DecisionConditionReady, - Status: v1.ConditionFalse, - }, - }, - }, - }, - }, - operator: "test-operator", - expectedCount: 3, - description: "should only count decisions with matching operator", - expectedError: 1, - expectedWaiting: 0, - expectedSuccess: 0, - }, - { - name: "multiple decisions same state", - decisions: []v1alpha1.Decision{ - { - ObjectMeta: v1.ObjectMeta{Name: "dec-error-1"}, - Spec: v1alpha1.DecisionSpec{SchedulingDomain: "test-operator"}, - Status: v1alpha1.DecisionStatus{ - Conditions: []v1.Condition{ - { - Type: v1alpha1.DecisionConditionReady, - Status: v1.ConditionFalse, - }, - }, - }, - }, - { - ObjectMeta: v1.ObjectMeta{Name: "dec-error-2"}, - Spec: v1alpha1.DecisionSpec{SchedulingDomain: "test-operator"}, - Status: v1alpha1.DecisionStatus{ - Conditions: []v1.Condition{ - { - Type: v1alpha1.DecisionConditionReady, - Status: v1.ConditionFalse, - }, - }, - }, - }, - }, - operator: "test-operator", - expectedCount: 3, - description: "should correctly aggregate multiple decisions in same state", - expectedError: 2, - expectedWaiting: 0, - expectedSuccess: 0, - }, - { - name: "decision with no result", - decisions: []v1alpha1.Decision{ - { - ObjectMeta: v1.ObjectMeta{Name: "dec-no-result"}, - Spec: v1alpha1.DecisionSpec{SchedulingDomain: "test-operator"}, - Status: v1alpha1.DecisionStatus{}, - }, - }, - operator: "test-operator", - expectedCount: 3, - description: "should count decision with no result as success", - expectedError: 0, - expectedWaiting: 0, - expectedSuccess: 1, - }, - { - name: "error condition takes precedence", - decisions: []v1alpha1.Decision{ - { - ObjectMeta: v1.ObjectMeta{Name: "dec-error-with-target"}, - Spec: v1alpha1.DecisionSpec{SchedulingDomain: "test-operator"}, - Status: v1alpha1.DecisionStatus{ - Result: &v1alpha1.DecisionResult{ - TargetHost: &targetHost, - }, - Conditions: []v1.Condition{ - { - Type: v1alpha1.DecisionConditionReady, - Status: v1.ConditionFalse, - }, - }, - }, - }, - }, - operator: "test-operator", - expectedCount: 3, - description: "should count as error even if target host is present", - expectedError: 1, - expectedWaiting: 0, - expectedSuccess: 0, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - objects := make([]v1alpha1.Decision, len(tt.decisions)) - copy(objects, tt.decisions) - - clientBuilder := fake.NewClientBuilder().WithScheme(scheme) - for i := range objects { - clientBuilder = clientBuilder.WithObjects(&objects[i]) - } - client := clientBuilder.Build() - - kpi := &DecisionStateKPI{} - if err := kpi.Init(nil, client, conf.NewRawOpts(`{"decisionSchedulingDomain": "`+tt.operator+`"}`)); err != nil { - t.Fatalf("expected no error, got %v", err) - } - - ch := make(chan prometheus.Metric, 10) - kpi.Collect(ch) - close(ch) - - metricsCount := 0 - for range ch { - metricsCount++ - } - - if metricsCount != tt.expectedCount { - t.Errorf("%s: expected %d metrics, got %d", tt.description, tt.expectedCount, metricsCount) - } - }) - } -} - -func TestDecisionStateKPI_GetName(t *testing.T) { - kpi := &DecisionStateKPI{} - expectedName := "decision_state_kpi" - if name := kpi.GetName(); name != expectedName { - t.Errorf("expected name %q, got %q", expectedName, name) - } -} - -func TestDecisionStateKPI_Describe(t *testing.T) { - kpi := &DecisionStateKPI{} - if err := kpi.Init(nil, nil, conf.NewRawOpts(`{"decisionSchedulingDomain": "test-operator"}`)); err != nil { - t.Fatalf("expected no error, got %v", err) - } - - ch := make(chan *prometheus.Desc, 1) - kpi.Describe(ch) - close(ch) - - descCount := 0 - for range ch { - descCount++ - } - - if descCount != 1 { - t.Errorf("expected 1 descriptor, got %d", descCount) - } -} diff --git a/internal/knowledge/kpis/plugins/infrastructure/kvm_host_capacity.go b/internal/knowledge/kpis/plugins/infrastructure/kvm_host_capacity.go index da7c2b824..0885f24cb 100644 --- a/internal/knowledge/kpis/plugins/infrastructure/kvm_host_capacity.go +++ b/internal/knowledge/kpis/plugins/infrastructure/kvm_host_capacity.go @@ -93,30 +93,25 @@ func aggregateReservationsByHost(reservations []v1alpha1.Reservation) ( failoverByHost[host] = entry case v1alpha1.ReservationTypeCommittedResource: - // Total reserved resources for this reservation. - cpuTotal := reservation.Spec.Resources[hv1.ResourceCPU] - memTotal := reservation.Spec.Resources[hv1.ResourceMemory] - - // Sum allocated resources across all workloads. - var cpuAllocated, memAllocated resource.Quantity - if reservation.Spec.CommittedResourceReservation != nil { - for _, alloc := range reservation.Spec.CommittedResourceReservation.Allocations { - cpuAllocated.Add(alloc.Resources[hv1.ResourceCPU]) - memAllocated.Add(alloc.Resources[hv1.ResourceMemory]) + // Compute remaining slot capacity: slot - sum of confirmed VM resources. + cpuNotInUse := reservation.Spec.Resources[hv1.ResourceCPU].DeepCopy() + memNotInUse := reservation.Spec.Resources[hv1.ResourceMemory].DeepCopy() + if reservation.Status.CommittedResourceReservation != nil { + zero := resource.MustParse("0") + for _, alloc := range reservation.Status.CommittedResourceReservation.Allocation { + if cpu, ok := alloc.Resources[hv1.ResourceCPU]; ok { + cpuNotInUse.Sub(cpu) + } + if mem, ok := alloc.Resources[hv1.ResourceMemory]; ok { + memNotInUse.Sub(mem) + } + } + if cpuNotInUse.Cmp(zero) < 0 { + cpuNotInUse = zero.DeepCopy() + } + if memNotInUse.Cmp(zero) < 0 { + memNotInUse = zero.DeepCopy() } - } - - // Not yet in use = total - allocated, clamped to zero. - cpuNotInUse := cpuTotal.DeepCopy() - cpuNotInUse.Sub(cpuAllocated) - if cpuNotInUse.Cmp(resource.MustParse("0")) < 0 { - cpuNotInUse = resource.MustParse("0") - } - - memNotInUse := memTotal.DeepCopy() - memNotInUse.Sub(memAllocated) - if memNotInUse.Cmp(resource.MustParse("0")) < 0 { - memNotInUse = resource.MustParse("0") } entry := committedNotInUseByHost[host] diff --git a/internal/knowledge/kpis/plugins/infrastructure/kvm_host_capacity_test.go b/internal/knowledge/kpis/plugins/infrastructure/kvm_host_capacity_test.go index 317b478cc..1a921a8c8 100644 --- a/internal/knowledge/kpis/plugins/infrastructure/kvm_host_capacity_test.go +++ b/internal/knowledge/kpis/plugins/infrastructure/kvm_host_capacity_test.go @@ -12,6 +12,7 @@ import ( hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "github.com/prometheus/client_golang/prometheus" prometheusgo "github.com/prometheus/client_model/go" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -556,22 +557,24 @@ func TestKVMResourceCapacityKPI_Collect(t *testing.T) { hv1.ResourceCPU: resource.MustParse("32"), hv1.ResourceMemory: resource.MustParse("128Gi"), }, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - "vm-uuid-1": { - Resources: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceCPU: resource.MustParse("8"), - hv1.ResourceMemory: resource.MustParse("32Gi"), - }, - }, - }, - }, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{}, }, Status: v1alpha1.ReservationStatus{ Host: "node001-bb088", Conditions: []v1.Condition{ {Type: v1alpha1.ReservationConditionReady, Status: v1.ConditionTrue}, }, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + "vm-uuid-1": { + ObjectRef: corev1.ObjectReference{Name: "vm-uuid-1"}, + Resources: map[hv1.ResourceName]resource.Quantity{ + hv1.ResourceCPU: resource.MustParse("32"), + hv1.ResourceMemory: resource.MustParse("128Gi"), + }, + }, + }, + }, }, }, }, @@ -580,13 +583,13 @@ func TestKVMResourceCapacityKPI_Collect(t *testing.T) { kvmTotalMetric("node001-bb088", "ram", "qa-1a", 549755813888), // 512Gi kvmUsageMetric("node001-bb088", "cpu", "utilized", "qa-1a", 64), kvmUsageMetric("node001-bb088", "ram", "utilized", "qa-1a", 274877906944), // 256Gi - // reserved = 32-8=24 CPU, 128Gi-32Gi=96Gi RAM (not in use) - kvmUsageMetric("node001-bb088", "cpu", "reserved", "qa-1a", 24), - kvmUsageMetric("node001-bb088", "ram", "reserved", "qa-1a", 103079215104), // 96Gi + // status.allocations non-empty → slot fully in use → reserved = 0 + kvmUsageMetric("node001-bb088", "cpu", "reserved", "qa-1a", 0), + kvmUsageMetric("node001-bb088", "ram", "reserved", "qa-1a", 0), kvmUsageMetric("node001-bb088", "cpu", "failover", "qa-1a", 0), kvmUsageMetric("node001-bb088", "ram", "failover", "qa-1a", 0), - kvmUsageMetric("node001-bb088", "cpu", "available", "qa-1a", 40), // 128-64-24-0 - kvmUsageMetric("node001-bb088", "ram", "available", "qa-1a", 171798691840), // 512Gi-256Gi-96Gi-0 = 160Gi + kvmUsageMetric("node001-bb088", "cpu", "available", "qa-1a", 64), // 128-64-0-0 + kvmUsageMetric("node001-bb088", "ram", "available", "qa-1a", 274877906944), // 512Gi-256Gi-0-0 = 256Gi }, }, { diff --git a/internal/scheduling/cinder/external_scheduler_api.go b/internal/scheduling/cinder/external_scheduler_api.go index 96bea5590..8ae12e3de 100644 --- a/internal/scheduling/cinder/external_scheduler_api.go +++ b/internal/scheduling/cinder/external_scheduler_api.go @@ -7,29 +7,24 @@ import ( "bytes" "context" "encoding/json" - "errors" "fmt" "io" "net/http" api "github.com/cobaltcore-dev/cortex/api/external/cinder" - "github.com/cobaltcore-dev/cortex/api/v1alpha1" scheduling "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" "github.com/go-logr/logr" - corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/api/meta" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/runtime" logf "sigs.k8s.io/controller-runtime/pkg/log" "sigs.k8s.io/controller-runtime/pkg/metrics" ) var log = logf.Log.WithName("cinder-scheduler-api") -type HTTPAPIDelegate interface { - // Process the decision from the API. Should create and return the updated decision. - ProcessNewDecisionFromAPI(ctx context.Context, decision *v1alpha1.Decision) error +// Scheduler runs a cinder pipeline for a request and returns the result. +// *FilterWeigherPipelineController satisfies it. +type Scheduler interface { + Schedule(ctx context.Context, pipelineName string, request api.ExternalSchedulerRequest) (scheduling.PipelineResult, error) } type HTTPAPI interface { @@ -38,14 +33,14 @@ type HTTPAPI interface { } type httpAPI struct { - monitor scheduling.APIMonitor - delegate HTTPAPIDelegate + monitor scheduling.APIMonitor + scheduler Scheduler } -func NewAPI(delegate HTTPAPIDelegate) HTTPAPI { +func NewAPI(scheduler Scheduler) HTTPAPI { return &httpAPI{ - monitor: scheduling.NewSchedulerMonitor(), - delegate: delegate, + monitor: scheduling.NewSchedulerMonitor(), + scheduler: scheduler, } } @@ -109,7 +104,6 @@ func (httpAPI *httpAPI) CinderExternalScheduler(w http.ResponseWriter, r *http.R c.Respond(logr.Logger{}, http.StatusInternalServerError, err, "failed to read request body") return } - raw := runtime.RawExtension{Raw: body} var requestData api.ExternalSchedulerRequest // Copy the raw body to a io.Reader for json deserialization. cp := body @@ -138,36 +132,12 @@ func (httpAPI *httpAPI) CinderExternalScheduler(w http.ResponseWriter, r *http.R logger.V(1).Info("inferred pipeline name", "pipeline", requestData.Pipeline) } - // Create the decision object in kubernetes. - decision := &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - GenerateName: "cinder-", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainCinder, - PipelineRef: corev1.ObjectReference{ - Name: requestData.Pipeline, - }, - ResourceID: "", // TODO - CinderRaw: &raw, - Intent: v1alpha1.SchedulingIntentUnknown, - }, - } - ctx := r.Context() - if err := httpAPI.delegate.ProcessNewDecisionFromAPI(ctx, decision); err != nil { + result, err := httpAPI.scheduler.Schedule(r.Context(), requestData.Pipeline, requestData) + if err != nil { c.Respond(logger, http.StatusInternalServerError, err, fmt.Sprintf("failed to process scheduling decision: %v", err)) return } - // Check if the decision contains status conditions indicating an error. - if meta.IsStatusConditionFalse(decision.Status.Conditions, v1alpha1.DecisionConditionReady) { - c.Respond(logger, http.StatusInternalServerError, errors.New("decision contains error condition"), "decision failed") - return - } - if decision.Status.Result == nil { - c.Respond(logger, http.StatusInternalServerError, errors.New("decision didn't produce a result"), "decision failed") - return - } - hosts := decision.Status.Result.OrderedHosts + hosts := result.OrderedHosts // Ensure we always return an array, never null. The schema requires hosts // to be an array, and a nil slice would serialize to null. if hosts == nil { diff --git a/internal/scheduling/cinder/external_scheduler_api_test.go b/internal/scheduling/cinder/external_scheduler_api_test.go index d5f7f394f..ec8c34feb 100644 --- a/internal/scheduling/cinder/external_scheduler_api_test.go +++ b/internal/scheduling/cinder/external_scheduler_api_test.go @@ -14,26 +14,24 @@ import ( "testing" cinderapi "github.com/cobaltcore-dev/cortex/api/external/cinder" - "github.com/cobaltcore-dev/cortex/api/v1alpha1" - - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" ) -type mockHTTPAPIDelegate struct { - processDecisionFunc func(ctx context.Context, decision *v1alpha1.Decision) error +type mockScheduler struct { + scheduleFunc func(ctx context.Context, pipelineName string, request cinderapi.ExternalSchedulerRequest) (lib.PipelineResult, error) } -func (m *mockHTTPAPIDelegate) ProcessNewDecisionFromAPI(ctx context.Context, decision *v1alpha1.Decision) error { - if m.processDecisionFunc != nil { - return m.processDecisionFunc(ctx, decision) +func (m *mockScheduler) Schedule(ctx context.Context, pipelineName string, request cinderapi.ExternalSchedulerRequest) (lib.PipelineResult, error) { + if m.scheduleFunc != nil { + return m.scheduleFunc(ctx, pipelineName, request) } - return nil + return lib.PipelineResult{}, nil } func TestNewAPI(t *testing.T) { - delegate := &mockHTTPAPIDelegate{} + scheduler := &mockScheduler{} - api := NewAPI(delegate) + api := NewAPI(scheduler) if api == nil { t.Fatal("NewAPI returned nil") @@ -44,8 +42,8 @@ func TestNewAPI(t *testing.T) { t.Fatal("NewAPI did not return httpAPI type") } - if httpAPI.delegate != delegate { - t.Error("Delegate not set correctly") + if httpAPI.scheduler != scheduler { + t.Error("Scheduler not set correctly") } if httpAPI.monitor.ApiRequestsTimer == nil { @@ -54,8 +52,7 @@ func TestNewAPI(t *testing.T) { } func TestHTTPAPI_Init(t *testing.T) { - delegate := &mockHTTPAPIDelegate{} - api := NewAPI(delegate) + api := NewAPI(&mockScheduler{}) mux := http.NewServeMux() api.Init(mux) @@ -72,8 +69,7 @@ func TestHTTPAPI_Init(t *testing.T) { } func TestHTTPAPI_canRunScheduler(t *testing.T) { - delegate := &mockHTTPAPIDelegate{} - api := NewAPI(delegate).(*httpAPI) + api := NewAPI(&mockScheduler{}).(*httpAPI) tests := []struct { name string @@ -141,14 +137,27 @@ func TestHTTPAPI_canRunScheduler(t *testing.T) { } func TestHTTPAPI_CinderExternalScheduler(t *testing.T) { + validBody := func(t *testing.T) string { + req := cinderapi.ExternalSchedulerRequest{ + Hosts: []cinderapi.ExternalSchedulerHost{{VolumeHost: "host1"}}, + Weights: map[string]float64{"host1": 1.0}, + Pipeline: "test-pipeline", + } + data, err := json.Marshal(req) + if err != nil { + t.Fatalf("Failed to marshal request data: %v", err) + } + return string(data) + } + tests := []struct { - name string - method string - body string - processDecisionErr error - decisionResult *v1alpha1.Decision - expectedStatus int - expectedHosts []string + name string + method string + body string + scheduleResult lib.PipelineResult + scheduleErr error + expectedStatus int + expectedHosts []string }{ { name: "invalid method", @@ -162,106 +171,31 @@ func TestHTTPAPI_CinderExternalScheduler(t *testing.T) { expectedStatus: http.StatusBadRequest, }, { - name: "valid request with successful processing", - method: http.MethodPost, - body: func() string { - req := cinderapi.ExternalSchedulerRequest{ - Hosts: []cinderapi.ExternalSchedulerHost{ - {VolumeHost: "host1"}, - }, - Weights: map[string]float64{ - "host1": 1.0, - }, - Pipeline: "test-pipeline", - } - data, err := json.Marshal(req) - if err != nil { - t.Fatalf("Failed to marshal request data: %v", err) - } - return string(data) - }(), - decisionResult: &v1alpha1.Decision{ - Status: v1alpha1.DecisionStatus{ - Result: &v1alpha1.DecisionResult{ - OrderedHosts: []string{"host1"}, - }, - }, - }, + name: "valid request with successful processing", + method: http.MethodPost, + body: validBody(t), + scheduleResult: lib.PipelineResult{OrderedHosts: []string{"host1"}}, expectedStatus: http.StatusOK, expectedHosts: []string{"host1"}, }, { - name: "processing error", - method: http.MethodPost, - body: func() string { - req := cinderapi.ExternalSchedulerRequest{ - Hosts: []cinderapi.ExternalSchedulerHost{ - {VolumeHost: "host1"}, - }, - Weights: map[string]float64{ - "host1": 1.0, - }, - Pipeline: "test-pipeline", - } - data, err := json.Marshal(req) - if err != nil { - t.Fatalf("Failed to marshal request data: %v", err) - } - return string(data) - }(), - processDecisionErr: errors.New("processing failed"), - expectedStatus: http.StatusInternalServerError, - }, - { - name: "decision failed", - method: http.MethodPost, - body: func() string { - req := cinderapi.ExternalSchedulerRequest{ - Hosts: []cinderapi.ExternalSchedulerHost{ - {VolumeHost: "host1"}, - }, - Weights: map[string]float64{ - "host1": 1.0, - }, - Pipeline: "test-pipeline", - } - data, err := json.Marshal(req) - if err != nil { - t.Fatalf("Failed to marshal request data: %v", err) - } - return string(data) - }(), - decisionResult: &v1alpha1.Decision{ - Status: v1alpha1.DecisionStatus{ - Conditions: []metav1.Condition{ - { - Type: v1alpha1.DecisionConditionReady, - Status: metav1.ConditionFalse, - Reason: "SchedulingError", - }, - }, - }, - }, + name: "processing error", + method: http.MethodPost, + body: validBody(t), + scheduleErr: errors.New("processing failed"), expectedStatus: http.StatusInternalServerError, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - delegate := &mockHTTPAPIDelegate{ - processDecisionFunc: func(ctx context.Context, decision *v1alpha1.Decision) error { - if tt.processDecisionErr != nil { - return tt.processDecisionErr - } - if tt.decisionResult != nil { - decision.Status = tt.decisionResult.Status - return nil - } - return nil + scheduler := &mockScheduler{ + scheduleFunc: func(ctx context.Context, pipelineName string, request cinderapi.ExternalSchedulerRequest) (lib.PipelineResult, error) { + return tt.scheduleResult, tt.scheduleErr }, } - api := NewAPI(delegate).(*httpAPI) + api := NewAPI(scheduler).(*httpAPI) var body *strings.Reader if tt.body != "" { @@ -300,8 +234,7 @@ func TestHTTPAPI_CinderExternalScheduler(t *testing.T) { } func TestHTTPAPI_inferPipelineName(t *testing.T) { - delegate := &mockHTTPAPIDelegate{} - api := NewAPI(delegate).(*httpAPI) + api := NewAPI(&mockScheduler{}).(*httpAPI) tests := []struct { name string @@ -347,31 +280,25 @@ func TestHTTPAPI_inferPipelineName(t *testing.T) { } } -func TestHTTPAPI_CinderExternalScheduler_DecisionCreation(t *testing.T) { - var capturedDecision *v1alpha1.Decision - delegate := &mockHTTPAPIDelegate{ - processDecisionFunc: func(ctx context.Context, decision *v1alpha1.Decision) error { - capturedDecision = decision - // Set a successful result to avoid "decision didn't produce a result" error - decision.Status.Result = &v1alpha1.DecisionResult{ - OrderedHosts: []string{"host1"}, - } - return nil +// When the request carries no pipeline, the handler infers the default name and +// passes it to the scheduler. +func TestHTTPAPI_CinderExternalScheduler_InfersPipeline(t *testing.T) { + var capturedPipeline string + var capturedRequest cinderapi.ExternalSchedulerRequest + scheduler := &mockScheduler{ + scheduleFunc: func(ctx context.Context, pipelineName string, request cinderapi.ExternalSchedulerRequest) (lib.PipelineResult, error) { + capturedPipeline = pipelineName + capturedRequest = request + return lib.PipelineResult{OrderedHosts: []string{"host1"}}, nil }, } - api := NewAPI(delegate).(*httpAPI) + api := NewAPI(scheduler).(*httpAPI) requestData := cinderapi.ExternalSchedulerRequest{ - Hosts: []cinderapi.ExternalSchedulerHost{ - {VolumeHost: "host1"}, - }, - Weights: map[string]float64{ - "host1": 1.0, - }, - Pipeline: "test-pipeline", + Hosts: []cinderapi.ExternalSchedulerHost{{VolumeHost: "host1"}}, + Weights: map[string]float64{"host1": 1.0}, } - body, err := json.Marshal(requestData) if err != nil { t.Fatalf("Failed to marshal request data: %v", err) @@ -384,25 +311,10 @@ func TestHTTPAPI_CinderExternalScheduler_DecisionCreation(t *testing.T) { if w.Code != http.StatusOK { t.Errorf("Expected status %d, got %d", http.StatusOK, w.Code) } - - if capturedDecision == nil { - t.Fatal("Decision was not captured") + if capturedPipeline != "cinder-external-scheduler" { + t.Errorf("Expected inferred pipeline 'cinder-external-scheduler', got %q", capturedPipeline) } - - // Verify decision fields - if capturedDecision.Spec.SchedulingDomain != v1alpha1.SchedulingDomainCinder { - t.Errorf("Expected scheduling domain %s, got %s", v1alpha1.SchedulingDomainCinder, capturedDecision.Spec.SchedulingDomain) - } - - if capturedDecision.Spec.PipelineRef.Name != "test-pipeline" { - t.Errorf("Expected pipeline 'test-pipeline', got %s", capturedDecision.Spec.PipelineRef.Name) - } - - if capturedDecision.GenerateName != "cinder-" { - t.Errorf("Expected generate name 'cinder-', got %s", capturedDecision.GenerateName) - } - - if capturedDecision.Spec.CinderRaw == nil { - t.Error("CinderRaw should not be nil") + if len(capturedRequest.Hosts) != 1 || capturedRequest.Hosts[0].VolumeHost != "host1" { + t.Errorf("Request not passed through to scheduler: %+v", capturedRequest) } } diff --git a/internal/scheduling/cinder/filter_weigher_pipeline_controller.go b/internal/scheduling/cinder/filter_weigher_pipeline_controller.go index 6d2593147..815571e4c 100644 --- a/internal/scheduling/cinder/filter_weigher_pipeline_controller.go +++ b/internal/scheduling/cinder/filter_weigher_pipeline_controller.go @@ -5,16 +5,12 @@ package cinder import ( "context" - "encoding/json" - "errors" "fmt" "sync" "time" api "github.com/cobaltcore-dev/cortex/api/external/cinder" "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "k8s.io/apimachinery/pkg/api/meta" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "github.com/cobaltcore-dev/cortex/internal/scheduling/cinder/plugins/filters" "github.com/cobaltcore-dev/cortex/internal/scheduling/cinder/plugins/weighers" @@ -27,17 +23,15 @@ import ( "sigs.k8s.io/controller-runtime/pkg/predicate" ) -// The decision pipeline controller takes decision resources containing a -// placement request spec and runs the scheduling pipeline to make a decision. -// This decision is then written back to the decision resource status. -// -// Additionally, the controller watches for pipeline and step changes to -// reconfigure the pipelines as needed. +// FilterWeigherPipelineController owns the cinder filter-weigher pipelines: it +// watches Pipeline and Knowledge resources to (re)configure the pipelines and +// runs a pipeline on demand via Schedule, called directly by the external +// scheduler HTTP API. type FilterWeigherPipelineController struct { // Toolbox shared between all pipeline controllers. lib.BasePipelineController[lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]] - // Mutex to only allow one process at a time + // Mutex to only allow one scheduling run at a time. processMu sync.Mutex // Monitor to pass down to all pipelines. @@ -49,83 +43,34 @@ func (c *FilterWeigherPipelineController) PipelineType() v1alpha1.PipelineType { return v1alpha1.PipelineTypeFilterWeigher } -// Callback executed when kubernetes asks to reconcile a decision resource. -func (c *FilterWeigherPipelineController) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { - c.processMu.Lock() - defer c.processMu.Unlock() - - decision := &v1alpha1.Decision{} - if err := c.Get(ctx, req.NamespacedName, decision); err != nil { - return ctrl.Result{}, client.IgnoreNotFound(err) - } - old := decision.DeepCopy() - if err := c.process(ctx, decision); err != nil { - return ctrl.Result{}, err - } - patch := client.MergeFrom(old) - if err := c.Status().Patch(ctx, decision, patch); err != nil { - return ctrl.Result{}, err - } +// Reconcile is a no-op: this controller reconfigures pipelines through its +// Pipeline/Knowledge watch handlers and runs placements via Schedule. A +// Reconcile method is required to satisfy the controller builder. +func (c *FilterWeigherPipelineController) Reconcile(context.Context, ctrl.Request) (ctrl.Result, error) { return ctrl.Result{}, nil } -// Process the decision from the API. Should create and return the updated decision. -func (c *FilterWeigherPipelineController) ProcessNewDecisionFromAPI(ctx context.Context, decision *v1alpha1.Decision) error { +// Schedule runs the named pipeline for the given request and returns the +// result. It serializes runs with the process lock. +func (c *FilterWeigherPipelineController) Schedule(ctx context.Context, pipelineName string, request api.ExternalSchedulerRequest) (lib.PipelineResult, error) { c.processMu.Lock() defer c.processMu.Unlock() - - err := c.process(ctx, decision) - if err != nil { - meta.SetStatusCondition(&decision.Status.Conditions, metav1.Condition{ - Type: v1alpha1.DecisionConditionReady, - Status: metav1.ConditionFalse, - Reason: "PipelineRunFailed", - Message: "pipeline run failed: " + err.Error(), - }) - } else { - meta.SetStatusCondition(&decision.Status.Conditions, metav1.Condition{ - Type: v1alpha1.DecisionConditionReady, - Status: metav1.ConditionTrue, - Reason: "PipelineRunSucceeded", - Message: "pipeline run succeeded", - }) - } - return err -} - -func (c *FilterWeigherPipelineController) process(ctx context.Context, decision *v1alpha1.Decision) error { log := ctrl.LoggerFrom(ctx) startedAt := time.Now() // So we can measure sync duration. - pipeline, ok := c.GetPipeline(decision.Spec.PipelineRef.Name) + pipeline, ok := c.GetPipeline(pipelineName) if !ok { - log.Error(nil, "pipeline not found or not ready", "pipelineName", decision.Spec.PipelineRef.Name) - return fmt.Errorf("pipeline not found or not ready: %q", decision.Spec.PipelineRef.Name) - } - if decision.Spec.CinderRaw == nil { - log.Error(nil, "skipping decision, no cinderRaw spec defined") - return errors.New("no cinderRaw spec defined") - } - var request api.ExternalSchedulerRequest - if err := json.Unmarshal(decision.Spec.CinderRaw.Raw, &request); err != nil { - log.Error(err, "failed to unmarshal cinderRaw spec") - return err + log.Error(nil, "pipeline not found or not ready", "pipelineName", pipelineName) + return lib.PipelineResult{}, fmt.Errorf("pipeline not found or not ready: %q", pipelineName) } result, err := pipeline.Run(request) - if err == nil { - decision.Status.Result = &result - } - // Record a human-readable explanation of the decision for downstream - // consumers. On error the result is nil and the explanation carries the - // failure reason. - decision.Status.Explanation = lib.GenerateExplanation(decision.Status.Result, err) if err != nil { log.Error(err, "failed to run pipeline") - return err + return lib.PipelineResult{}, err } log.Info("decision processed successfully", "duration", time.Since(startedAt)) - return nil + return result, nil } // The base controller will delegate the pipeline creation down to this method. @@ -186,25 +131,6 @@ func (c *FilterWeigherPipelineController) SetupWithManager(mgr manager.Manager, if err != nil { return err } - // Watch decision changes across all clusters. - bldr, err = bldr.WatchesMulticluster( - &v1alpha1.Decision{}, - &handler.EnqueueRequestForObject{}, - predicate.NewPredicateFuncs(func(obj client.Object) bool { - decision := obj.(*v1alpha1.Decision) - if decision.Spec.SchedulingDomain != v1alpha1.SchedulingDomainCinder { - return false - } - // Ignore already decided schedulings. - if decision.Status.Result != nil { - return false - } - return true - }), - ) - if err != nil { - return err - } - return bldr.Named("cortex-cinder-decisions"). + return bldr.Named("cortex-cinder-pipelines"). Complete(c) } diff --git a/internal/scheduling/cinder/filter_weigher_pipeline_controller_test.go b/internal/scheduling/cinder/filter_weigher_pipeline_controller_test.go index 8ec56c40f..4daf8fee0 100644 --- a/internal/scheduling/cinder/filter_weigher_pipeline_controller_test.go +++ b/internal/scheduling/cinder/filter_weigher_pipeline_controller_test.go @@ -4,15 +4,8 @@ package cinder import ( - "context" - "encoding/json" "testing" - corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/runtime" - "k8s.io/apimachinery/pkg/types" - ctrl "sigs.k8s.io/controller-runtime" - "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client/fake" api "github.com/cobaltcore-dev/cortex/api/external/cinder" @@ -21,25 +14,16 @@ import ( "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" ) -func TestFilterWeigherPipelineController_Reconcile(t *testing.T) { +func TestFilterWeigherPipelineController_Schedule(t *testing.T) { scheme := runtime.NewScheme() if err := v1alpha1.AddToScheme(scheme); err != nil { t.Fatalf("Failed to add v1alpha1 scheme: %v", err) } - cinderRequest := api.ExternalSchedulerRequest{ - Spec: map[string]any{ - "volume_id": "test-volume-id", - "size": 10, - }, - Context: api.CinderRequestContext{ - ProjectID: "test-project", - UserID: "test-user", - RequestID: "req-123", - GlobalRequestID: "global-req-123", - }, + request := api.ExternalSchedulerRequest{ Hosts: []api.ExternalSchedulerHost{ {VolumeHost: "cinder-volume-1"}, {VolumeHost: "cinder-volume-2"}, @@ -49,127 +33,41 @@ func TestFilterWeigherPipelineController_Reconcile(t *testing.T) { Options: scheduling.Options{}, } - cinderRaw, err := json.Marshal(cinderRequest) - if err != nil { - t.Fatalf("Failed to marshal cinder request: %v", err) - } - tests := []struct { name string - decision *v1alpha1.Decision - pipeline *v1alpha1.Pipeline + pipelineName string + configured bool expectError bool - expectResult bool }{ { - name: "successful cinder decision processing", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainCinder, - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - CinderRaw: &runtime.RawExtension{ - Raw: cinderRaw, - }, - }, - }, - pipeline: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainCinder, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, + name: "successful scheduling run", + pipelineName: "test-pipeline", + configured: true, expectError: false, - expectResult: true, }, { - name: "decision without cinderRaw spec", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-raw", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainCinder, - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - CinderRaw: nil, - }, - }, - pipeline: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainCinder, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - expectError: true, - expectResult: false, - }, - { - name: "pipeline not found", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-pipeline", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainCinder, - PipelineRef: corev1.ObjectReference{ - Name: "nonexistent-pipeline", - }, - CinderRaw: &runtime.RawExtension{ - Raw: cinderRaw, - }, - }, - }, - pipeline: nil, + name: "pipeline not found", + pipelineName: "nonexistent-pipeline", + configured: false, expectError: true, - expectResult: false, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - objects := []client.Object{tt.decision} - if tt.pipeline != nil { - objects = append(objects, tt.pipeline) - } - - client := fake.NewClientBuilder(). - WithScheme(scheme). - WithObjects(objects...). - WithStatusSubresource(&v1alpha1.Decision{}). - Build() + cl := fake.NewClientBuilder().WithScheme(scheme).Build() controller := &FilterWeigherPipelineController{ BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]]{ - Client: client, + Client: cl, Pipelines: make(map[string]lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]), }, Monitor: lib.FilterWeigherPipelineMonitor{}, } - if tt.pipeline != nil { + if tt.configured { initResult := controller.InitPipeline(t.Context(), v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, + ObjectMeta: metav1.ObjectMeta{Name: "test-pipeline"}, Spec: v1alpha1.PipelineSpec{ Type: v1alpha1.PipelineTypeFilterWeigher, SchedulingDomain: v1alpha1.SchedulingDomainCinder, @@ -180,238 +78,10 @@ func TestFilterWeigherPipelineController_Reconcile(t *testing.T) { if len(initResult.FilterErrors) > 0 || len(initResult.WeigherErrors) > 0 { t.Fatalf("Failed to init pipeline: %v", initResult) } - controller.Pipelines[tt.pipeline.Name] = initResult.Pipeline - } - - req := ctrl.Request{ - NamespacedName: types.NamespacedName{ - Name: tt.decision.Name, - Namespace: tt.decision.Namespace, - }, - } - - result, err := controller.Reconcile(context.Background(), req) - - if tt.expectError && err == nil { - t.Error("Expected error but got none") - } - if !tt.expectError && err != nil { - t.Errorf("Expected no error but got: %v", err) - } - - if result.RequeueAfter > 0 { - t.Error("Expected no requeue") - } - - var updatedDecision v1alpha1.Decision - if err := client.Get(context.Background(), req.NamespacedName, &updatedDecision); err != nil { - t.Fatalf("Failed to get updated decision: %v", err) - } - - if tt.expectResult && updatedDecision.Status.Result == nil { - t.Error("Expected result to be set but was nil") - } - if !tt.expectResult && updatedDecision.Status.Result != nil { - t.Error("Expected result to be nil but was set") - } - }) - } -} - -func TestFilterWeigherPipelineController_ProcessNewDecisionFromAPI(t *testing.T) { - scheme := runtime.NewScheme() - if err := v1alpha1.AddToScheme(scheme); err != nil { - t.Fatalf("Failed to add v1alpha1 scheme: %v", err) - } - - cinderRequest := api.ExternalSchedulerRequest{ - Spec: map[string]any{ - "volume_id": "test-volume-id", - "size": 10, - }, - Context: api.CinderRequestContext{ - ProjectID: "test-project", - UserID: "test-user", - RequestID: "req-123", - GlobalRequestID: "global-req-123", - }, - Hosts: []api.ExternalSchedulerHost{ - {VolumeHost: "cinder-volume-1"}, - {VolumeHost: "cinder-volume-2"}, - }, - Weights: map[string]float64{"cinder-volume-1": 1.0, "cinder-volume-2": 0.5}, - Pipeline: "test-pipeline", - } - - cinderRaw, err := json.Marshal(cinderRequest) - if err != nil { - t.Fatalf("Failed to marshal cinder request: %v", err) - } - - tests := []struct { - name string - decision *v1alpha1.Decision - pipelineConfig *v1alpha1.Pipeline - expectError bool - expectResult bool - }{ - { - name: "successful decision processing with creation", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - GenerateName: "test-decision-", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainCinder, - ResourceID: "test-uuid-1", - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - CinderRaw: &runtime.RawExtension{ - Raw: cinderRaw, - }, - }, - }, - pipelineConfig: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainCinder, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - expectError: false, - expectResult: true, - }, - { - name: "successful decision processing without creation", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-create", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainCinder, - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - CinderRaw: &runtime.RawExtension{ - Raw: cinderRaw, - }, - }, - }, - pipelineConfig: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainCinder, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - expectError: false, - expectResult: true, - }, - { - name: "pipeline not configured", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-pipeline", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainCinder, - PipelineRef: corev1.ObjectReference{ - Name: "nonexistent-pipeline", - }, - CinderRaw: &runtime.RawExtension{ - Raw: cinderRaw, - }, - }, - }, - pipelineConfig: nil, - expectError: true, - expectResult: false, - }, - { - name: "decision without cinderRaw spec", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-raw", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainCinder, - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - CinderRaw: nil, - }, - }, - pipelineConfig: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainCinder, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - expectError: true, - expectResult: false, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - objects := []client.Object{} - if tt.pipelineConfig != nil { - objects = append(objects, tt.pipelineConfig) - } - - client := fake.NewClientBuilder(). - WithScheme(scheme). - WithObjects(objects...). - WithStatusSubresource(&v1alpha1.Decision{}). - Build() - - controller := &FilterWeigherPipelineController{ - BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]]{ - Client: client, - Pipelines: make(map[string]lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]), - PipelineConfigs: make(map[string]v1alpha1.Pipeline), - }, - Monitor: lib.FilterWeigherPipelineMonitor{}, - } - - if tt.pipelineConfig != nil { - controller.PipelineConfigs[tt.pipelineConfig.Name] = *tt.pipelineConfig - initResult := controller.InitPipeline(t.Context(), *tt.pipelineConfig) - if len(initResult.FilterErrors) > 0 || len(initResult.WeigherErrors) > 0 { - t.Fatalf("Failed to init pipeline: %v", initResult) - } - controller.Pipelines[tt.pipelineConfig.Name] = initResult.Pipeline + controller.Pipelines["test-pipeline"] = initResult.Pipeline } - if tt.decision.Spec.CinderRaw != nil { - req := cinderRequest - req.Options = scheduling.Options{} - raw, marshalErr := json.Marshal(req) - if marshalErr != nil { - t.Fatalf("Failed to marshal request with options: %v", marshalErr) - } - tt.decision.Spec.CinderRaw = &runtime.RawExtension{Raw: raw} - } - - err := controller.ProcessNewDecisionFromAPI(context.Background(), tt.decision) + result, err := controller.Schedule(t.Context(), tt.pipelineName, request) if tt.expectError && err == nil { t.Error("Expected error but got none") @@ -419,10 +89,9 @@ func TestFilterWeigherPipelineController_ProcessNewDecisionFromAPI(t *testing.T) if !tt.expectError && err != nil { t.Errorf("Expected no error but got: %v", err) } - if !tt.expectError { - if tt.expectResult && tt.decision.Status.Result == nil { - t.Error("expected decision result to be set in original decision object") + if len(result.OrderedHosts) == 0 { + t.Error("Expected ordered hosts but got none") } } }) diff --git a/internal/scheduling/compute/placement.go b/internal/scheduling/compute/placement.go new file mode 100644 index 000000000..7244016b2 --- /dev/null +++ b/internal/scheduling/compute/placement.go @@ -0,0 +1,140 @@ +// Copyright SAP SE +// SPDX-License-Identifier: Apache-2.0 + +package compute + +import ( + "context" + "fmt" + "sync" + + "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" + ctrl "sigs.k8s.io/controller-runtime" +) + +// PipelineProvider supplies initialized pipelines by name. The Nova +// filter-weigher pipeline controller satisfies this interface via its embedded +// lib.BasePipelineController. +type PipelineProvider interface { + // GetPipeline returns the initialized pipeline for the given name, if ready. + GetPipeline(name string) (lib.FilterWeigherPipeline[PlacementRequest], bool) +} + +// CandidateGatherer gathers all placement candidates for a request when the +// pipeline is configured to ignore the caller's preselection. +type CandidateGatherer interface { + // MutateWithAllCandidates overrides the request's hosts and weights with all + // available placement candidates. + MutateWithAllCandidates(ctx context.Context, request *PlacementRequest) error +} + +// PreHook is a caller-provided hook executed under the scheduling lock before +// the pipeline runs. It may mutate the request (e.g. to create a pessimistic +// in-flight reservation and narrow the candidate set). Returning an error aborts +// the run before the pipeline executes. A nil PreHook is a no-op. +type PreHook func(ctx context.Context, req *PlacementRequest) error + +// PostHook is a caller-provided side effect executed after the pipeline run, +// under the scheduling lock (e.g. upserting the VM resource, recording the +// decision). It receives the typed pipeline result. Returning an error aborts +// the run; non-fatal side effects should log and return nil so the placement +// decision still stands. A nil PostHook is a no-op. +type PostHook func(ctx context.Context, req PlacementRequest, result PlacementResult) error + +// Placement is the single in-process owner of Nova placement decisions. It holds +// the one shared scheduling lock and runs the pipeline plus a caller-provided +// pre-hook and post-hook. It owns no side-effect policy itself — the caller +// decides, via the hooks it passes, what happens before and after the run. +type Placement struct { + // mu is the single shared scheduling lock, held across the pre-hook, the + // pipeline run, and the post-hook in Run. RunWithoutLock bypasses it. + mu sync.Mutex + + // pipelines resolves pipelines + their config by name. + pipelines PipelineProvider + // gatherer gathers all candidates when a pipeline ignores preselection. + gatherer CandidateGatherer +} + +// NewPlacement builds a Placement over the given pipeline provider and candidate +// gatherer. +func NewPlacement(pipelines PipelineProvider, gatherer CandidateGatherer) *Placement { + return &Placement{pipelines: pipelines, gatherer: gatherer} +} + +// Run executes the pipeline and hooks while holding the shared scheduling lock +// across the whole operation, so concurrent runs observe a consistent view of +// shared scheduling state. The pre-hook runs before the pipeline; the post-hook +// after. Either hook may be nil. Use this for any run that mutates shared state. +func (s *Placement) Run( + ctx context.Context, + req PlacementRequest, + pre PreHook, + post PostHook, +) (PlacementResult, error) { + + s.mu.Lock() + defer s.mu.Unlock() + return s.run(ctx, req, pre, post) +} + +// RunWithoutLock executes the pipeline and hooks WITHOUT acquiring the shared +// lock. Use this only for read-only runs (e.g. capacity probes) that do not +// mutate shared scheduling state and can safely run concurrently with a locked +// run. Either hook may be nil. +func (s *Placement) RunWithoutLock( + ctx context.Context, + req PlacementRequest, + pre PreHook, + post PostHook, +) (PlacementResult, error) { + + return s.run(ctx, req, pre, post) +} + +// run is the shared body of Run and RunWithoutLock. The caller is responsible +// for holding (or deliberately not holding) the lock. +func (s *Placement) run( + ctx context.Context, + req PlacementRequest, + pre PreHook, + post PostHook, +) (PlacementResult, error) { + + log := ctrl.LoggerFrom(ctx) + + pipeline, ok := s.pipelines.GetPipeline(req.Pipeline) + if !ok { + return PlacementResult{}, fmt.Errorf("pipeline not found or not ready: %q", req.Pipeline) + } + + // The pre-hook runs before the pipeline and may mutate the request. + if pre != nil { + if err := pre(ctx, &req); err != nil { + return PlacementResult{}, err + } + } + + // When the request opts into gathering all candidates, override the hosts and + // weights with every available placement candidate before filtering. + if req.GatherAllCandidates { + log.V(1).Info("gathering all placement candidates before filtering") + if err := s.gatherer.MutateWithAllCandidates(ctx, &req); err != nil { + return PlacementResult{}, fmt.Errorf("failed to gather all placement candidates: %w", err) + } + log.V(1).Info("gathered all placement candidates", "numHosts", len(req.Hosts)) + } + + result, runErr := pipeline.Run(req) + if runErr != nil { + return PlacementResult{}, runErr + } + + // The post-hook runs after a successful pipeline run, under the lock. + if post != nil { + if err := post(ctx, req, result); err != nil { + return result, err + } + } + return result, nil +} diff --git a/internal/scheduling/compute/placement_test.go b/internal/scheduling/compute/placement_test.go new file mode 100644 index 000000000..ab58e687a --- /dev/null +++ b/internal/scheduling/compute/placement_test.go @@ -0,0 +1,269 @@ +// Copyright SAP SE +// SPDX-License-Identifier: Apache-2.0 + +package compute + +import ( + "context" + "errors" + "sync" + "testing" + "time" + + "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" +) + +// fakePipeline is a stub lib.FilterWeigherPipeline[PlacementRequest] that records +// the request it was run with and returns a configurable result/error. The run +// function, if set, is invoked inside Run (used to observe lock behavior). +type fakePipeline struct { + result PlacementResult + err error + onRun func(PlacementRequest) + lastReq PlacementRequest +} + +func (p *fakePipeline) Run(req PlacementRequest) (PlacementResult, error) { + p.lastReq = req + if p.onRun != nil { + p.onRun(req) + } + return p.result, p.err +} + +// fakeProvider implements PipelineProvider. A pipeline is returned only for the +// configured name. +type fakeProvider struct { + name string + pipeline lib.FilterWeigherPipeline[PlacementRequest] +} + +func (f *fakeProvider) GetPipeline(name string) (lib.FilterWeigherPipeline[PlacementRequest], bool) { + if name != f.name || f.pipeline == nil { + return nil, false + } + return f.pipeline, true +} + +// fakeGatherer implements CandidateGatherer, recording whether it was called. +type fakeGatherer struct { + called bool + err error +} + +func (g *fakeGatherer) MutateWithAllCandidates(_ context.Context, req *PlacementRequest) error { + g.called = true + if g.err != nil { + return g.err + } + req.Hosts = []Host{{ComputeHost: "gathered", HypervisorHostname: "gathered"}} + return nil +} + +// newTestPlacement builds a Placement whose only ready pipeline is named "p". +func newTestPlacement(p *fakePipeline, g CandidateGatherer) *Placement { + return NewPlacement(&fakeProvider{name: "p", pipeline: p}, g) +} + +func TestRunPipelineNotFound(t *testing.T) { + s := newTestPlacement(&fakePipeline{}, &fakeGatherer{}) + _, err := s.Run(context.Background(), PlacementRequest{Pipeline: "missing"}, nil, nil) + if err == nil { + t.Fatal("expected error for missing pipeline, got nil") + } +} + +func TestRunPreThenPost(t *testing.T) { + s := newTestPlacement(&fakePipeline{}, &fakeGatherer{}) + var order []string + pre := func(_ context.Context, _ *PlacementRequest) error { + order = append(order, "pre") + return nil + } + post := func(_ context.Context, _ PlacementRequest, _ PlacementResult) error { + order = append(order, "post") + return nil + } + if _, err := s.Run(context.Background(), PlacementRequest{Pipeline: "p"}, pre, post); err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(order) != 2 || order[0] != "pre" || order[1] != "post" { + t.Fatalf("hooks ran out of order: %v", order) + } +} + +func TestRunNilHooks(t *testing.T) { + s := newTestPlacement(&fakePipeline{}, &fakeGatherer{}) + if _, err := s.Run(context.Background(), PlacementRequest{Pipeline: "p"}, nil, nil); err != nil { + t.Fatalf("nil hooks should be a no-op, got error: %v", err) + } +} + +func TestRunPostHookErrorAborts(t *testing.T) { + s := newTestPlacement(&fakePipeline{}, &fakeGatherer{}) + boom := errors.New("boom") + post := func(_ context.Context, _ PlacementRequest, _ PlacementResult) error { return boom } + _, err := s.Run(context.Background(), PlacementRequest{Pipeline: "p"}, nil, post) + if !errors.Is(err, boom) { + t.Fatalf("expected boom error, got %v", err) + } +} + +func TestRunPipelineErrorSkipsPostHook(t *testing.T) { + boom := errors.New("pipeline failed") + s := newTestPlacement(&fakePipeline{err: boom}, &fakeGatherer{}) + var ran bool + post := func(_ context.Context, _ PlacementRequest, _ PlacementResult) error { ran = true; return nil } + _, err := s.Run(context.Background(), PlacementRequest{Pipeline: "p"}, nil, post) + if !errors.Is(err, boom) { + t.Fatalf("expected pipeline error, got %v", err) + } + if ran { + t.Fatal("post-hook must not run when the pipeline errors") + } +} + +func TestRunPreHookRunsBeforePipelineAndMutates(t *testing.T) { + var sawHostInPipeline string + p := &fakePipeline{onRun: func(r PlacementRequest) { + if len(r.Hosts) > 0 { + sawHostInPipeline = r.Hosts[0].ComputeHost + } + }} + s := newTestPlacement(p, &fakeGatherer{}) + pre := func(_ context.Context, r *PlacementRequest) error { + r.Hosts = []Host{{ComputeHost: "from-prehook"}} + return nil + } + if _, err := s.Run(context.Background(), PlacementRequest{Pipeline: "p"}, pre, nil); err != nil { + t.Fatalf("unexpected error: %v", err) + } + if sawHostInPipeline != "from-prehook" { + t.Fatalf("pre-hook mutation not visible to pipeline: got %q", sawHostInPipeline) + } +} + +func TestRunPreHookErrorAbortsBeforePipeline(t *testing.T) { + var pipelineRan bool + p := &fakePipeline{onRun: func(PlacementRequest) { pipelineRan = true }} + s := newTestPlacement(p, &fakeGatherer{}) + boom := errors.New("prehook boom") + pre := func(_ context.Context, _ *PlacementRequest) error { return boom } + _, err := s.Run(context.Background(), PlacementRequest{Pipeline: "p"}, pre, nil) + if !errors.Is(err, boom) { + t.Fatalf("expected prehook error, got %v", err) + } + if pipelineRan { + t.Fatal("pipeline must not run after a pre-hook error") + } +} + +func TestRunGathersCandidatesWhenRequested(t *testing.T) { + g := &fakeGatherer{} + p := &fakePipeline{} + s := newTestPlacement(p, g) + req := PlacementRequest{Pipeline: "p", GatherAllCandidates: true} + if _, err := s.Run(context.Background(), req, nil, nil); err != nil { + t.Fatalf("unexpected error: %v", err) + } + if !g.called { + t.Fatal("expected gatherer to be called when GatherAllCandidates is set") + } + if len(p.lastReq.Hosts) != 1 || p.lastReq.Hosts[0].ComputeHost != "gathered" { + t.Fatalf("pipeline did not see gathered candidates: %+v", p.lastReq.Hosts) + } +} + +func TestRunDoesNotGatherWithoutGatherAllCandidates(t *testing.T) { + g := &fakeGatherer{} + s := newTestPlacement(&fakePipeline{}, g) + if _, err := s.Run(context.Background(), PlacementRequest{Pipeline: "p"}, nil, nil); err != nil { + t.Fatalf("unexpected error: %v", err) + } + if g.called { + t.Fatal("gatherer should not be called without GatherAllCandidates") + } +} + +// TestRunHoldsLockAcrossPostHook verifies that a second Run blocks until the +// first Run's post-hook completes — i.e. the lock spans the whole run, not just +// the pipeline. +func TestRunHoldsLockAcrossPostHook(t *testing.T) { + s := newTestPlacement(&fakePipeline{}, &fakeGatherer{}) + + firstInPost := make(chan struct{}) + releaseFirst := make(chan struct{}) + secondDone := make(chan struct{}) + + var slowPost PostHook = func(_ context.Context, _ PlacementRequest, _ PlacementResult) error { + close(firstInPost) + <-releaseFirst + return nil + } + + var wg sync.WaitGroup + wg.Add(1) + go func() { + defer wg.Done() + if _, err := s.Run(context.Background(), PlacementRequest{Pipeline: "p"}, nil, slowPost); err != nil { + t.Errorf("first Run: %v", err) + } + }() + + <-firstInPost // first Run is parked inside its post-hook, still holding the lock. + + go func() { + if _, err := s.Run(context.Background(), PlacementRequest{Pipeline: "p"}, nil, nil); err != nil { + t.Errorf("second Run: %v", err) + } + close(secondDone) + }() + + // The second Run must be blocked on the lock held by the first. + select { + case <-secondDone: + t.Fatal("second Run completed while the first still held the lock") + case <-time.After(50 * time.Millisecond): + } + + close(releaseFirst) // let the first Run finish and release the lock. + wg.Wait() + <-secondDone // the second Run now proceeds. +} + +// TestRunWithoutLockDoesNotBlock verifies that RunWithoutLock proceeds even while +// a locked Run holds the lock inside its post-hook. +func TestRunWithoutLockDoesNotBlock(t *testing.T) { + s := newTestPlacement(&fakePipeline{}, &fakeGatherer{}) + + firstInPost := make(chan struct{}) + releaseFirst := make(chan struct{}) + var slowPost PostHook = func(_ context.Context, _ PlacementRequest, _ PlacementResult) error { + close(firstInPost) + <-releaseFirst + return nil + } + + go func() { + if _, err := s.Run(context.Background(), PlacementRequest{Pipeline: "p"}, nil, slowPost); err != nil { + t.Errorf("locked Run: %v", err) + } + }() + <-firstInPost // locked Run holds the lock. + + done := make(chan struct{}) + go func() { + if _, err := s.RunWithoutLock(context.Background(), PlacementRequest{Pipeline: "p"}, nil, nil); err != nil { + t.Errorf("RunWithoutLock: %v", err) + } + close(done) + }() + + select { + case <-done: + // Expected: lock-free run completes without waiting for the lock. + case <-time.After(time.Second): + t.Fatal("RunWithoutLock blocked on the scheduling lock") + } + close(releaseFirst) +} diff --git a/internal/scheduling/compute/request.go b/internal/scheduling/compute/request.go new file mode 100644 index 000000000..c73b79ec0 --- /dev/null +++ b/internal/scheduling/compute/request.go @@ -0,0 +1,266 @@ +// Copyright SAP SE +// SPDX-License-Identifier: Apache-2.0 + +// Package compute is the in-process entrypoint for Nova placement scheduling. +// +// It defines a well-typed, Nova-agnostic PlacementRequest that the +// filter-weigher pipeline consumes, and a Placement that owns the single shared +// scheduling lock and runs the pipeline plus caller-provided side-effect +// closures. Callers that run in the same process (the Nova HTTP adapter, the +// reservation/commitment/capacity/failover controllers) call the Placement +// directly instead of POSTing to the external HTTP endpoint. +// +// The spec deliberately contains NO loosely-typed maps or Nova string +// conventions (flavor extra-spec keys, scheduler-hint keys, image property +// keys). A Nova spec adapter is responsible for translating Nova's over-the-wire +// shape into these typed fields; the pipeline only ever sees well-specced +// values. The Nova vocabulary types those fields use (capabilities, flavor +// sizing, hypervisor family, …) live in the api/external/nova package. +package compute + +import ( + "strings" + + api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/api/scheduling" + "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" +) + +// Host is a placement candidate. For KVM hosts the compute host name and the +// hypervisor hostname are identical; for others they may differ. +type Host struct { + // ComputeHost is the Nova compute host name, e.g. nova-compute-bb123. + ComputeHost string + // HypervisorHostname is the hypervisor hostname, e.g. domain-c123.. + HypervisorHostname string +} + +// PlacementRequest is the well-typed, Nova-agnostic input to the scheduling +// pipeline. Every field is a resolved scalar, enum, or typed collection — there +// are no maps of Nova-specific string keys and no values the pipeline must parse. +// +// Each field documents what leaving it at its zero value means, so the Nova spec +// adapter (and any other caller) can see exactly which fields are load-bearing. +// +// PlacementRequest implements lib.FilterWeigherPipelineRequest so it can be the +// RequestType of the filter-weigher pipeline. +type PlacementRequest struct { + // Pipeline is the name of the pipeline to execute. Required: an empty name + // resolves no pipeline and the run fails with "pipeline not found". + Pipeline string + // GatherAllCandidates, when true, tells the Placement to ignore the Hosts and + // Weights the caller provided and instead gather ALL available placement + // candidates (via the CandidateGatherer) before filtering. It is set for + // requests where cortex can enumerate the candidate hosts itself — today the + // KVM family, whose hypervisors cortex knows from the hypervisor operator; + // VMware placement leaves it false and relies on the caller's preselection. + // When true, Hosts/Weights below are overwritten before filtering. + GatherAllCandidates bool + // Hosts are the placement candidates. An empty list yields an empty result + // (no host found), unless GatherAllCandidates is set, in which case Hosts is + // overwritten before filtering. + Hosts []Host + // Weights are the per-host input weights, keyed by compute host name. A host + // absent from this map is weighed as 0. Empty means every candidate starts + // equally weighted. + Weights map[string]float64 + // Options configure the pipeline behavior for this run. The zero value is a + // normal (non-read-only) run with no candidate limit; see scheduling.Options + // for per-field defaults. + Options scheduling.Options + + // Intent is the resolved scheduling intent (replaces the "_nova_check_type" + // scheduler hint). The zero value is the empty intent; intent-gated + // filters/weighers treat it like an initial placement (create) and a forced + // destination is honored (see IsForcedDestination). + Intent v1alpha1.SchedulingIntent + + // InstanceUUID identifies the instance being placed. Empty is allowed for + // synthetic runs (e.g. capacity probes); it only appears in logs and side + // effects, so an empty value disables nothing in the pipeline itself. + InstanceUUID string + // ProjectID is the owning OpenStack project. Empty disables project-scoped + // filtering (e.g. allowed-projects, quota) — those steps treat it as "no + // project context". + ProjectID string + // UserID is the requesting user. Empty only affects trace logging and the VM + // ownership written by side effects. + UserID string + // AvailabilityZone is the target AZ. Empty means no AZ constraint, so the + // correct-AZ filter keeps all hosts. + AvailabilityZone string + // NumInstances is the number of instances this request represents. It is a + // MULTIPLIER in capacity and bin-pack math, so 0 makes a host look infinitely + // free and must not be left unset — the adapter always sets it (Nova sends + // >= 1). + NumInstances uint64 + + // IgnoreHosts are hosts to exclude from placement. Empty/nil means no hosts + // are excluded. + IgnoreHosts []string + // ForceHosts pins placement to these compute hosts. Empty/nil means no host + // is forced. When set (and the intent does not require filter validation), + // the pipeline is skipped and only forced hosts are returned. + ForceHosts []string + // ForceNodes pins placement to these hypervisor hostnames. Empty/nil means no + // node is forced. Semantics mirror ForceHosts. + ForceNodes []string + + // HypervisorFamily is the coarse family (kvm/vmware) used by the pipeline and + // routing. Empty means unresolved: KVM-only gates (e.g. VM creation) treat it + // as not-KVM and skip. + HypervisorFamily api.HypervisorFamily + // ImageHypervisorType is the hypervisor type declared by the image. The empty + // value (ImageHypervisorTypeUnset) means the image declared none ("no image + // constraint"); ImageHypervisorTypeUnparseable means it declared one that + // could not be parsed, which the image-properties filter surfaces as a + // diagnostic. + ImageHypervisorType api.ImageHypervisorType + // HardwareVersion is resolved from the "quota:hw_version" extra spec. Empty + // means no hardware-version constraint. + HardwareVersion string + + // Flavor is the requested flavor sizing. A zero FlavorSpec means capacity and + // bin-pack steps see a zero-sized instance (fits anywhere) — the adapter + // always populates it for real placements. + Flavor api.FlavorSpec + + // RequiredTraits are trait names the host must have (adapter stripped the + // "trait:" prefix and kept the "required" ones). Empty/nil means no required + // traits, so the traits filter imposes no inclusion constraint. + RequiredTraits []string + // ForbiddenTraits are trait names the host must NOT have. Empty/nil means no + // forbidden traits. If both trait lists are empty the traits filter is a no-op. + ForbiddenTraits []string + // RequestedCapabilities are required (key=value) hypervisor capabilities + // (adapter stripped the "capabilities:" prefix and rejected unsupported + // operators). Empty/nil means the capabilities filter is a no-op. + RequestedCapabilities []api.Capability + // AcceleratorsRequested reports whether the flavor requests a hardware + // accelerator (adapter resolves it from the presence of the + // "accel:device_profile" extra spec). When true, the accelerators filter keeps + // only hosts carrying the COMPUTE_ACCELERATORS trait; false is a no-op. + AcceleratorsRequested bool + + // SourceHost is the migration source (was hint "source_host"). Empty means + // not a host-to-host move, so the live-migratable filter imposes no source + // constraint. + SourceHost string + // DomainName is the customer domain (was hint "domain_name"). Empty means no + // domain context, so the external-customer filter is skipped. + DomainName string + // ResourceGroup is the failover resource group (was hint + // "_cortex_resource_group"). Empty means no group, so group-consolidation + // weighing is skipped. + ResourceGroup string + + // RequestedDestination is the requested target host/aggregate. Nil means no + // destination constraint (see the type doc). + RequestedDestination *api.RequestedDestination + // InstanceGroup is the server group the instance belongs to. Nil means the + // instance is not grouped, so affinity/anti-affinity steps are skipped (see + // the type doc). + InstanceGroup *api.InstanceGroupSpec + + // RequestID identifies this scheduling request for tracing. Empty only + // affects log correlation. + RequestID string + // GlobalRequestID is the cross-service request ID for tracing. Empty only + // affects log correlation. + GlobalRequestID string +} + +// Compile-time check that PlacementRequest satisfies the pipeline request contract. +var _ lib.FilterWeigherPipelineRequest = PlacementRequest{} + +// GetIntent returns the resolved scheduling intent. +func (r PlacementRequest) GetIntent() v1alpha1.SchedulingIntent { return r.Intent } + +// GetHosts returns the compute host names of all candidates. +func (r PlacementRequest) GetHosts() []string { + hosts := make([]string, len(r.Hosts)) + for i, h := range r.Hosts { + hosts[i] = h.ComputeHost + } + return hosts +} + +// GetWeights returns the per-host input weights. +func (r PlacementRequest) GetWeights() map[string]float64 { return r.Weights } + +// GetOptions returns the call-time options for this run. +func (r PlacementRequest) GetOptions() scheduling.Options { return r.Options } + +// GetTraceLogArgs returns alternating key/value pairs for the step trace log. +func (r PlacementRequest) GetTraceLogArgs() []any { + return []any{ + "greq", r.GlobalRequestID, + "req", r.RequestID, + "user", r.UserID, + "project", r.ProjectID, + } +} + +// Filter returns a copy of the request narrowed to the given hosts, with the +// weights updated to the provided values. Hosts not present in includedHosts are +// dropped and won't be reconsidered by later steps. +func (r PlacementRequest) Filter(includedHosts map[string]float64) lib.FilterWeigherPipelineRequest { + filteredHosts := make([]Host, 0, len(includedHosts)) + for _, h := range r.Hosts { + if _, ok := includedHosts[h.ComputeHost]; ok { + filteredHosts = append(filteredHosts, h) + } + } + r.Hosts = filteredHosts + filteredWeights := make(map[string]float64, len(includedHosts)) + for host, weight := range includedHosts { + filteredWeights[host] = weight + } + r.Weights = filteredWeights + return r +} + +// IsForcedDestination reports whether the request is forced onto specific +// hosts/nodes and the scheduler filters should be skipped entirely. A forced +// request that still carries a non-create intent (rebuild/evacuate/resize) must +// be validated by the filters, so it is not treated as forced here. Intent +// resolution happens in the adapter; this only inspects the typed Intent field. +func (r PlacementRequest) IsForcedDestination() bool { + if len(r.ForceHosts) == 0 && len(r.ForceNodes) == 0 { + return false + } + return r.Intent == "" || r.Intent == v1alpha1.SchedulingIntentUnknown +} + +// ForcedHosts returns the subset of request hosts matching the forced +// destinations. force_hosts matches on compute host name (case-insensitive); +// force_nodes matches on hypervisor hostname (case-sensitive); when both are set +// a host must match both. Order is preserved and compute hosts are deduplicated. +func (r PlacementRequest) ForcedHosts() []string { + forceHostSet := make(map[string]bool, len(r.ForceHosts)) + for _, h := range r.ForceHosts { + forceHostSet[strings.ToLower(h)] = true + } + forceNodeSet := make(map[string]bool, len(r.ForceNodes)) + for _, n := range r.ForceNodes { + forceNodeSet[n] = true + } + matched := make([]string, 0, len(r.Hosts)) + seen := make(map[string]bool, len(r.Hosts)) + for _, h := range r.Hosts { + if len(forceHostSet) > 0 && !forceHostSet[strings.ToLower(h.ComputeHost)] { + continue + } + if len(forceNodeSet) > 0 && !forceNodeSet[h.HypervisorHostname] { + continue + } + key := strings.ToLower(h.ComputeHost) + if seen[key] { + continue + } + seen[key] = true + matched = append(matched, h.ComputeHost) + } + return matched +} diff --git a/internal/scheduling/compute/request_test.go b/internal/scheduling/compute/request_test.go new file mode 100644 index 000000000..8244e7dbe --- /dev/null +++ b/internal/scheduling/compute/request_test.go @@ -0,0 +1,82 @@ +// Copyright SAP SE +// SPDX-License-Identifier: Apache-2.0 + +package compute + +import ( + "slices" + "testing" + + "github.com/cobaltcore-dev/cortex/api/v1alpha1" +) + +func TestIsForcedDestination(t *testing.T) { + cases := []struct { + name string + req PlacementRequest + forced bool + }{ + {"no force", PlacementRequest{}, false}, + {"force host, no intent", PlacementRequest{ForceHosts: []string{"h1"}}, true}, + {"force node, no intent", PlacementRequest{ForceNodes: []string{"n1"}}, true}, + {"force host, unknown intent", PlacementRequest{ForceHosts: []string{"h1"}, Intent: v1alpha1.SchedulingIntentUnknown}, true}, + {"force host, concrete intent must still filter", PlacementRequest{ForceHosts: []string{"h1"}, Intent: "evacuate"}, false}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + if got := c.req.IsForcedDestination(); got != c.forced { + t.Errorf("IsForcedDestination() = %v, want %v", got, c.forced) + } + }) + } +} + +func TestForcedHosts(t *testing.T) { + hosts := []Host{ + {ComputeHost: "Host-A", HypervisorHostname: "node-a"}, + {ComputeHost: "host-b", HypervisorHostname: "node-b"}, + {ComputeHost: "host-b", HypervisorHostname: "node-b2"}, // duplicate compute host + } + cases := []struct { + name string + force []string + nodes []string + want []string + }{ + {"force_hosts case-insensitive", []string{"host-a"}, nil, []string{"Host-A"}}, + {"force_nodes exact", nil, []string{"node-b"}, []string{"host-b"}}, + {"both intersect", []string{"host-b"}, []string{"node-b2"}, []string{"host-b"}}, + {"no match", []string{"nope"}, nil, []string{}}, + {"dedup compute host", []string{"host-b"}, nil, []string{"host-b"}}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + r := PlacementRequest{Hosts: hosts, ForceHosts: c.force, ForceNodes: c.nodes} + got := r.ForcedHosts() + if !slices.Equal(got, c.want) { + t.Errorf("ForcedHosts() = %v, want %v", got, c.want) + } + }) + } +} + +func TestFilterNarrowsHostsAndWeights(t *testing.T) { + r := PlacementRequest{ + Hosts: []Host{ + {ComputeHost: "a"}, {ComputeHost: "b"}, {ComputeHost: "c"}, + }, + Weights: map[string]float64{"a": 1, "b": 2, "c": 3}, + } + filtered := r.Filter(map[string]float64{"a": 10, "c": 30}).(PlacementRequest) + got := filtered.GetHosts() + slices.Sort(got) + if !slices.Equal(got, []string{"a", "c"}) { + t.Fatalf("Filter kept hosts %v, want [a c]", got) + } + if filtered.Weights["a"] != 10 || filtered.Weights["c"] != 30 { + t.Fatalf("Filter weights = %v, want a:10 c:30", filtered.Weights) + } + if _, ok := filtered.Weights["b"]; ok { + t.Fatalf("Filter should have dropped host b from weights: %v", filtered.Weights) + } +} diff --git a/internal/scheduling/compute/result.go b/internal/scheduling/compute/result.go new file mode 100644 index 000000000..56cfde8a0 --- /dev/null +++ b/internal/scheduling/compute/result.go @@ -0,0 +1,17 @@ +// Copyright SAP SE +// SPDX-License-Identifier: Apache-2.0 + +package compute + +import ( + "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" +) + +// PlacementResult is the well-defined output of a scheduling pipeline run: +// ordered hosts, the target host, and the input/output weights plus per-step +// activations used to rank them. It is an alias of lib.PipelineResult, the +// domain-neutral type the generic pipeline returns. The human-readable +// explanation is NOT part of the result — it is a concern of writing the +// decision onto the scheduled resource (e.g. the Nova VM's CortexSpec), so the +// relevant side effect generates it from the result via lib.GenerateExplanation. +type PlacementResult = lib.PipelineResult diff --git a/internal/scheduling/lib/explanation.go b/internal/scheduling/lib/explanation.go index c1deeff0f..4eff7017a 100644 --- a/internal/scheduling/lib/explanation.go +++ b/internal/scheduling/lib/explanation.go @@ -7,8 +7,6 @@ import ( "fmt" "sort" "strings" - - "github.com/cobaltcore-dev/cortex/api/v1alpha1" ) // maxHostsInExplanation caps how many host names are listed per line in a @@ -24,13 +22,12 @@ func joinHostsCapped(hosts []string, maxHosts int) string { return fmt.Sprintf("%s (and %d more)", strings.Join(hosts[:maxHosts], ", "), len(hosts)-maxHosts) } -// GenerateExplanation produces a human-readable explanation from a decision +// GenerateExplanation produces a human-readable explanation from a pipeline // result. On failure it includes the error. On success it describes which // pipeline steps filtered out which hosts and, for the surviving hosts, how the -// weighers ranked them (via ExplainWeighing). The result is written to -// Decision.Status.Explanation so it propagates to the scheduled resource (e.g. -// the VM CortexSpec.Explanation). -func GenerateExplanation(result *v1alpha1.DecisionResult, pipelineErr error) string { +// weighers ranked them (via ExplainWeighing). Consumers propagate the result to +// the scheduled resource (e.g. the Nova VM's CortexSpec.Explanation). +func GenerateExplanation(result *PipelineResult, pipelineErr error) string { if pipelineErr != nil { return fmt.Sprintf("Pipeline run failed: %s.", pipelineErr.Error()) } diff --git a/internal/scheduling/lib/explanation_test.go b/internal/scheduling/lib/explanation_test.go index 2c020f396..8ddcd2af4 100644 --- a/internal/scheduling/lib/explanation_test.go +++ b/internal/scheduling/lib/explanation_test.go @@ -7,8 +7,6 @@ import ( "errors" "fmt" "testing" - - "github.com/cobaltcore-dev/cortex/api/v1alpha1" ) func ptr[T any](v T) *T { return &v } @@ -16,7 +14,7 @@ func ptr[T any](v T) *T { return &v } func TestGenerateExplanation(t *testing.T) { tests := []struct { name string - result *v1alpha1.DecisionResult + result *PipelineResult err error expected string }{ @@ -34,20 +32,20 @@ func TestGenerateExplanation(t *testing.T) { }, { name: "result with target host only no steps", - result: &v1alpha1.DecisionResult{ + result: &PipelineResult{ TargetHost: ptr("host-1"), }, expected: "Selected host: host-1.", }, { name: "result with RawInWeights and filtering steps", - result: &v1alpha1.DecisionResult{ + result: &PipelineResult{ RawInWeights: map[string]float64{ "host-a": 1.0, "host-b": 0.5, "host-c": 0.3, }, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ { StepName: "filter_capacity", Activations: map[string]float64{ @@ -72,12 +70,12 @@ func TestGenerateExplanation(t *testing.T) { }, { name: "uses NormalizedInWeights when RawInWeights empty", - result: &v1alpha1.DecisionResult{ + result: &PipelineResult{ NormalizedInWeights: map[string]float64{ "host-x": 0.6, "host-y": 0.4, }, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ { StepName: "weigher_cpu", Activations: map[string]float64{ @@ -92,8 +90,8 @@ func TestGenerateExplanation(t *testing.T) { }, { name: "no weights no target", - result: &v1alpha1.DecisionResult{ - StepResults: []v1alpha1.StepResult{ + result: &PipelineResult{ + StepResults: []StepResult{ {StepName: "some-step", Activations: map[string]float64{}}, }, }, @@ -101,9 +99,9 @@ func TestGenerateExplanation(t *testing.T) { }, { name: "no weights with target", - result: &v1alpha1.DecisionResult{ + result: &PipelineResult{ TargetHost: ptr("host-1"), - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ {StepName: "some-step", Activations: map[string]float64{}}, }, }, @@ -111,12 +109,12 @@ func TestGenerateExplanation(t *testing.T) { }, { name: "all hosts survive all steps", - result: &v1alpha1.DecisionResult{ + result: &PipelineResult{ RawInWeights: map[string]float64{ "host-a": 1.0, "host-b": 0.5, }, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ { StepName: "noop", Activations: map[string]float64{ @@ -131,7 +129,7 @@ func TestGenerateExplanation(t *testing.T) { }, { name: "large cluster caps host lists", - result: func() *v1alpha1.DecisionResult { + result: func() *PipelineResult { weights := make(map[string]float64, 50) for i := range 50 { weights[fmt.Sprintf("host-%03d", i)] = 1.0 @@ -141,9 +139,9 @@ func TestGenerateExplanation(t *testing.T) { for i := range 30 { surviving[fmt.Sprintf("host-%03d", i)] = 1.0 } - return &v1alpha1.DecisionResult{ + return &PipelineResult{ RawInWeights: weights, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ { StepName: "filter_big", Activations: surviving, diff --git a/internal/scheduling/lib/filter_weigher_pipeline.go b/internal/scheduling/lib/filter_weigher_pipeline.go index 57ab4e321..d29ff2449 100644 --- a/internal/scheduling/lib/filter_weigher_pipeline.go +++ b/internal/scheduling/lib/filter_weigher_pipeline.go @@ -23,7 +23,7 @@ var pipelineLog = logf.Log.WithName("scheduler-pipeline") type FilterWeigherPipeline[RequestType FilterWeigherPipelineRequest] interface { // Run the scheduling pipeline with the given request. // Call-time options are read from request.GetOptions(). - Run(request RequestType) (v1alpha1.DecisionResult, error) + Run(request RequestType) (PipelineResult, error) } // Pipeline of scheduler steps. @@ -142,7 +142,7 @@ func InitNewFilterWeigherPipeline[RequestType FilterWeigherPipelineRequest]( func (p *filterWeigherPipeline[RequestType]) runFilters( traceLog logr.Logger, request RequestType, -) (filteredRequest RequestType, stepResults []v1alpha1.StepResult) { +) (filteredRequest RequestType, stepResults []StepResult) { filteredRequest = request for _, filterName := range p.filtersOrder { @@ -159,7 +159,7 @@ func (p *filterWeigherPipeline[RequestType]) runFilters( continue } stepLog.V(1).Info("scheduler: finished filter") - stepResults = append(stepResults, v1alpha1.StepResult{ + stepResults = append(stepResults, StepResult{ StepName: filterName, Activations: result.Activations, }) @@ -269,10 +269,10 @@ func (s *filterWeigherPipeline[RequestType]) sortHostsByWeights(weights map[stri } // Evaluate the pipeline and return a list of hosts in order of preference. -func (p *filterWeigherPipeline[RequestType]) Run(request RequestType) (v1alpha1.DecisionResult, error) { +func (p *filterWeigherPipeline[RequestType]) Run(request RequestType) (PipelineResult, error) { opts := request.GetOptions() if err := opts.Validate(); err != nil { - return v1alpha1.DecisionResult{}, err + return PipelineResult{}, err } traceLog := pipelineLog.WithValues(request.GetTraceLogArgs()...) @@ -341,13 +341,13 @@ func (p *filterWeigherPipeline[RequestType]) Run(request RequestType) (v1alpha1. if !ok { continue } - stepResults = append(stepResults, v1alpha1.StepResult{ + stepResults = append(stepResults, StepResult{ StepName: weigherName, Activations: activations, }) } - result := v1alpha1.DecisionResult{ + result := PipelineResult{ RawInWeights: request.GetWeights(), NormalizedInWeights: inWeights, StepResults: stepResults, diff --git a/internal/scheduling/lib/pipeline_result.go b/internal/scheduling/lib/pipeline_result.go new file mode 100644 index 000000000..938c08d78 --- /dev/null +++ b/internal/scheduling/lib/pipeline_result.go @@ -0,0 +1,31 @@ +// Copyright SAP SE +// SPDX-License-Identifier: Apache-2.0 + +package lib + +// StepResult holds the per-host activations a single pipeline step produced. +type StepResult struct { + // StepName is the name of the filter or weigher that produced this result. + StepName string + // Activations maps each host to the activation the step assigned it. + Activations map[string]float64 +} + +// PipelineResult is the domain-neutral output of a filter-weigher pipeline run: +// the ordered hosts, the target host, and the input/output weights plus per-step +// activations used to rank them. Domain-facing aliases (e.g. +// compute.PlacementResult) point at this type. +type PipelineResult struct { + // RawInWeights are the input weights to the pipeline, before normalization. + RawInWeights map[string]float64 + // NormalizedInWeights are the input weights after normalization. + NormalizedInWeights map[string]float64 + // StepResults are the activations each pipeline step produced, in order. + StepResults []StepResult + // AggregatedOutWeights are the final weights after applying all weighers. + AggregatedOutWeights map[string]float64 + // OrderedHosts is the final ordering from most to least preferred. + OrderedHosts []string + // TargetHost is the first element of OrderedHosts, if any. + TargetHost *string +} diff --git a/internal/scheduling/lib/weighing_explainer.go b/internal/scheduling/lib/weighing_explainer.go index 2e66bce41..9584afb6f 100644 --- a/internal/scheduling/lib/weighing_explainer.go +++ b/internal/scheduling/lib/weighing_explainer.go @@ -8,8 +8,6 @@ import ( "math" "sort" "strings" - - "github.com/cobaltcore-dev/cortex/api/v1alpha1" ) const ( @@ -55,7 +53,7 @@ type weigherContribution struct { // // Returns an empty string when the result is nil, has fewer than 2 ordered // hosts, or contains no weigher steps. -func ExplainWeighing(result *v1alpha1.DecisionResult) string { +func ExplainWeighing(result *PipelineResult) string { if result == nil || len(result.OrderedHosts) < 2 { return "" } @@ -199,7 +197,7 @@ func ExplainWeighing(result *v1alpha1.DecisionResult) string { // recovery is not possible (e.g., all activations are zero, or the system is // under-determined). It reports only the initial weight bias and raw activation // differentials without exact contribution magnitudes. -func explainWithoutMultipliers(result *v1alpha1.DecisionResult, topHosts []string, _ []v1alpha1.StepResult) string { +func explainWithoutMultipliers(result *PipelineResult, topHosts []string, _ []StepResult) string { topN := len(topHosts) var sb strings.Builder @@ -239,13 +237,13 @@ func explainWithoutMultipliers(result *v1alpha1.DecisionResult, topHosts []strin // weigher (scoring) steps rather than filter steps. A weigher step is one // whose activation map contains entries for ALL hosts in OrderedHosts — // filters reduce the host set while weighers score all remaining hosts. -func identifyWeigherSteps(result *v1alpha1.DecisionResult) []v1alpha1.StepResult { +func identifyWeigherSteps(result *PipelineResult) []StepResult { orderedHostSet := make(map[string]struct{}, len(result.OrderedHosts)) for _, h := range result.OrderedHosts { orderedHostSet[h] = struct{}{} } - var weigherSteps []v1alpha1.StepResult + var weigherSteps []StepResult for _, step := range result.StepResults { if len(step.Activations) == 0 { continue @@ -281,7 +279,7 @@ func identifyWeigherSteps(result *v1alpha1.DecisionResult) []v1alpha1.StepResult // Returns the recovered multipliers and true on success, or nil and false if // the system is under-determined or ill-conditioned. func recoverMultipliers( - weigherSteps []v1alpha1.StepResult, + weigherSteps []StepResult, orderedHosts []string, normalizedIn map[string]float64, aggregatedOut map[string]float64, diff --git a/internal/scheduling/lib/weighing_explainer_test.go b/internal/scheduling/lib/weighing_explainer_test.go index 3e63508d7..f333ecb8b 100644 --- a/internal/scheduling/lib/weighing_explainer_test.go +++ b/internal/scheduling/lib/weighing_explainer_test.go @@ -8,14 +8,12 @@ import ( "sort" "strings" "testing" - - "github.com/cobaltcore-dev/cortex/api/v1alpha1" ) func TestExplainWeighing(t *testing.T) { tests := []struct { name string - result *v1alpha1.DecisionResult + result *PipelineResult contains []string // substrings that must appear in output excludes []string // substrings that must NOT appear in output empty bool // expect empty string @@ -27,7 +25,7 @@ func TestExplainWeighing(t *testing.T) { }, { name: "single host returns empty", - result: &v1alpha1.DecisionResult{ + result: &PipelineResult{ OrderedHosts: []string{"host-a"}, NormalizedInWeights: map[string]float64{"host-a": 0.5}, AggregatedOutWeights: map[string]float64{"host-a": 0.8}, @@ -36,11 +34,11 @@ func TestExplainWeighing(t *testing.T) { }, { name: "only filter steps returns empty", - result: &v1alpha1.DecisionResult{ + result: &PipelineResult{ NormalizedInWeights: map[string]float64{"host-a": 0.5, "host-b": 0.3, "host-c": 0.1}, AggregatedOutWeights: map[string]float64{"host-a": 0.5, "host-b": 0.3}, OrderedHosts: []string{"host-a", "host-b"}, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ // A true filter: activations do NOT include host-b (it was filtered after this step). {StepName: "filter_az", Activations: map[string]float64{"host-a": 1.0}}, }, @@ -49,7 +47,7 @@ func TestExplainWeighing(t *testing.T) { }, { name: "two hosts single weigher positive multiplier", - result: func() *v1alpha1.DecisionResult { + result: func() *PipelineResult { // Simulate: multiplier = 2.0, act[a] = 1.0, act[b] = -0.5 // contrib[a] = 2.0 * tanh(1.0) = 1.5231 // contrib[b] = 2.0 * tanh(-0.5) = -0.9242 @@ -58,11 +56,11 @@ func TestExplainWeighing(t *testing.T) { inA, inB := 0.3, 0.3 outA := inA + mult*math.Tanh(actA) outB := inB + mult*math.Tanh(actB) - return &v1alpha1.DecisionResult{ + return &PipelineResult{ NormalizedInWeights: map[string]float64{"host-a": inA, "host-b": inB}, AggregatedOutWeights: map[string]float64{"host-a": outA, "host-b": outB}, OrderedHosts: []string{"host-a", "host-b"}, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ {StepName: "weigher_cpu", Activations: map[string]float64{"host-a": actA, "host-b": actB}}, }, } @@ -71,7 +69,7 @@ func TestExplainWeighing(t *testing.T) { }, { name: "two hosts single weigher negative multiplier", - result: func() *v1alpha1.DecisionResult { + result: func() *PipelineResult { // Simulate: multiplier = -1.0 (inverts behavior) // A host with HIGH raw activation is penalized. // act[a] = -2.0 (high penalty activates less due to neg mult -> boosts) @@ -83,11 +81,11 @@ func TestExplainWeighing(t *testing.T) { outB := inB + mult*math.Tanh(actB) // outA = 0 + (-1)*tanh(-2) = 0 + (-1)*(-0.964) = +0.964 // outB = 0 + (-1)*tanh(3) = 0 + (-1)*(0.995) = -0.995 - return &v1alpha1.DecisionResult{ + return &PipelineResult{ NormalizedInWeights: map[string]float64{"host-a": inA, "host-b": inB}, AggregatedOutWeights: map[string]float64{"host-a": outA, "host-b": outB}, OrderedHosts: []string{"host-a", "host-b"}, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ {StepName: "kvm_binpack", Activations: map[string]float64{"host-a": actA, "host-b": actB}}, }, } @@ -96,7 +94,7 @@ func TestExplainWeighing(t *testing.T) { }, { name: "three hosts two weighers with counterfactual", - result: func() *v1alpha1.DecisionResult { + result: func() *PipelineResult { // weigher_cpu (mult=3.0) strongly favors host-a // weigher_mem (mult=1.0) slightly favors host-b // Without weigher_cpu, host-b should be #1. @@ -111,11 +109,11 @@ func TestExplainWeighing(t *testing.T) { // Sort to determine OrderedHosts. hosts := []string{"h1", "h2", "h3"} sortByWeight(hosts, out) - return &v1alpha1.DecisionResult{ + return &PipelineResult{ NormalizedInWeights: in, AggregatedOutWeights: out, OrderedHosts: hosts, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ {StepName: "weigher_cpu", Activations: actCPU}, {StepName: "weigher_mem", Activations: actMem}, }, @@ -125,7 +123,7 @@ func TestExplainWeighing(t *testing.T) { }, { name: "opposing weigher reported", - result: func() *v1alpha1.DecisionResult { + result: func() *PipelineResult { // weigher_boost (mult=2.0) pushes host-a up // weigher_penalty (mult=-0.5) opposes host-a // Net: host-a still wins due to weigher_boost dominance. @@ -139,11 +137,11 @@ func TestExplainWeighing(t *testing.T) { } hosts := []string{"x", "y"} sortByWeight(hosts, out) - return &v1alpha1.DecisionResult{ + return &PipelineResult{ NormalizedInWeights: in, AggregatedOutWeights: out, OrderedHosts: hosts, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ {StepName: "weigher_boost", Activations: actBoost}, {StepName: "weigher_penalty", Activations: actPenalty}, }, @@ -153,7 +151,7 @@ func TestExplainWeighing(t *testing.T) { }, { name: "negligible impact weigher reported", - result: func() *v1alpha1.DecisionResult { + result: func() *PipelineResult { // weigher_big (mult=5.0) dominates // weigher_tiny (mult=0.001) is negligible multBig, multTiny := 5.0, 0.001 @@ -166,11 +164,11 @@ func TestExplainWeighing(t *testing.T) { } hosts := []string{"a", "b"} sortByWeight(hosts, out) - return &v1alpha1.DecisionResult{ + return &PipelineResult{ NormalizedInWeights: in, AggregatedOutWeights: out, OrderedHosts: hosts, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ {StepName: "weigher_big", Activations: actBig}, {StepName: "weigher_tiny", Activations: actTiny}, }, @@ -180,11 +178,11 @@ func TestExplainWeighing(t *testing.T) { }, { name: "initial weight bias dominates when activations are zero", - result: &v1alpha1.DecisionResult{ + result: &PipelineResult{ NormalizedInWeights: map[string]float64{"fast": 0.8, "slow": 0.2}, AggregatedOutWeights: map[string]float64{"fast": 0.8, "slow": 0.2}, OrderedHosts: []string{"fast", "slow"}, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ {StepName: "weigher_noop", Activations: map[string]float64{"fast": 0.0, "slow": 0.0}}, }, }, @@ -193,7 +191,7 @@ func TestExplainWeighing(t *testing.T) { }, { name: "mixed filter and weigher steps ignores filters", - result: func() *v1alpha1.DecisionResult { + result: func() *PipelineResult { mult := 1.5 actW := map[string]float64{"a": 1.0, "b": -0.5} in := map[string]float64{"a": 0.0, "b": 0.0, "c": 0.0} @@ -203,11 +201,11 @@ func TestExplainWeighing(t *testing.T) { } hosts := []string{"a", "b"} sortByWeight(hosts, out) - return &v1alpha1.DecisionResult{ + return &PipelineResult{ NormalizedInWeights: map[string]float64{"a": 0.0, "b": 0.0}, AggregatedOutWeights: out, OrderedHosts: hosts, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ // Filter step: removed host c (activations don't include all OrderedHosts... wait, they do include a and b). // Actually a filter that removed c would have activations for a and b only. // Since OrderedHosts = [a, b], this LOOKS like a weigher. @@ -224,11 +222,11 @@ func TestExplainWeighing(t *testing.T) { }, { name: "under-determined system falls back to gap-only explanation", - result: &v1alpha1.DecisionResult{ + result: &PipelineResult{ NormalizedInWeights: map[string]float64{"a": 0.0, "b": 0.0}, AggregatedOutWeights: map[string]float64{"a": 1.0, "b": 0.5}, OrderedHosts: []string{"a", "b"}, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ {StepName: "w1", Activations: map[string]float64{"a": 1.0, "b": 0.5}}, {StepName: "w2", Activations: map[string]float64{"a": 0.8, "b": 0.3}}, {StepName: "w3", Activations: map[string]float64{"a": 0.6, "b": 0.1}}, @@ -239,7 +237,7 @@ func TestExplainWeighing(t *testing.T) { }, { name: "multiplier recovery accuracy", - result: func() *v1alpha1.DecisionResult { + result: func() *PipelineResult { // Known multipliers: [2.5, -1.0, 0.7] mults := []float64{2.5, -1.0, 0.7} acts := []map[string]float64{ @@ -257,11 +255,11 @@ func TestExplainWeighing(t *testing.T) { } } sortByWeight(hosts, out) - return &v1alpha1.DecisionResult{ + return &PipelineResult{ NormalizedInWeights: in, AggregatedOutWeights: out, OrderedHosts: hosts, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ {StepName: "w_big", Activations: acts[0]}, {StepName: "w_neg", Activations: acts[1]}, {StepName: "w_small", Activations: acts[2]}, @@ -272,7 +270,7 @@ func TestExplainWeighing(t *testing.T) { }, { name: "counterfactual not reported when removal does not change #1", - result: func() *v1alpha1.DecisionResult { + result: func() *PipelineResult { // Two weighers both favor host-a. Removing either still leaves a as #1. mult1, mult2 := 2.0, 2.0 act1 := map[string]float64{"a": 1.0, "b": -1.0} @@ -282,11 +280,11 @@ func TestExplainWeighing(t *testing.T) { for _, h := range []string{"a", "b"} { out[h] = in[h] + mult1*math.Tanh(act1[h]) + mult2*math.Tanh(act2[h]) } - return &v1alpha1.DecisionResult{ + return &PipelineResult{ NormalizedInWeights: in, AggregatedOutWeights: out, OrderedHosts: []string{"a", "b"}, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ {StepName: "w1", Activations: act1}, {StepName: "w2", Activations: act2}, }, @@ -340,7 +338,7 @@ func TestRecoverMultipliers(t *testing.T) { } } - steps := []v1alpha1.StepResult{ + steps := []StepResult{ {StepName: "w1", Activations: acts[0]}, {StepName: "w2", Activations: acts[1]}, {StepName: "w3", Activations: acts[2]}, @@ -382,9 +380,9 @@ func TestSolveLinearSystem(t *testing.T) { } func TestIdentifyWeigherSteps(t *testing.T) { - result := &v1alpha1.DecisionResult{ + result := &PipelineResult{ OrderedHosts: []string{"a", "b", "c"}, - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ // Filter: missing host c {StepName: "filter_x", Activations: map[string]float64{"a": 1.0, "b": 1.0}}, // Weigher: has all ordered hosts @@ -482,12 +480,12 @@ func TestExplainWeighingDemo(t *testing.T) { // Sort hosts by aggregated weight to get OrderedHosts. sortByWeight(hosts, aggregatedOut) - result := &v1alpha1.DecisionResult{ + result := &PipelineResult{ NormalizedInWeights: normalizedIn, AggregatedOutWeights: aggregatedOut, OrderedHosts: hosts, TargetHost: &hosts[0], - StepResults: []v1alpha1.StepResult{ + StepResults: []StepResult{ {StepName: "kvm_binpack", Activations: actBinpack}, {StepName: "kvm_failover_evacuation", Activations: actFailover}, {StepName: "kvm_prefer_smaller_hosts", Activations: actSmaller}, diff --git a/internal/scheduling/machines/filter_weigher_pipeline_controller.go b/internal/scheduling/machines/filter_weigher_pipeline_controller.go index 3c542487b..672e287d6 100644 --- a/internal/scheduling/machines/filter_weigher_pipeline_controller.go +++ b/internal/scheduling/machines/filter_weigher_pipeline_controller.go @@ -19,8 +19,6 @@ import ( "github.com/cobaltcore-dev/cortex/internal/scheduling/machines/plugins/filters" "github.com/cobaltcore-dev/cortex/internal/scheduling/machines/plugins/weighers" corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/api/meta" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/util/workqueue" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" @@ -31,9 +29,9 @@ import ( "sigs.k8s.io/controller-runtime/pkg/reconcile" ) -// The decision pipeline controller takes decision resources containing a -// machine ref and runs the scheduling pipeline to make a decision. -// This decision is then written back to the decision resource status. +// The machine pipeline controller watches for new machines and runs the +// scheduling pipeline to assign each a machine pool. The result is written back +// directly to the machine's MachinePoolRef. // // Additionally, the controller watches for pipeline and step changes to // reconfigure the pipelines as needed. @@ -53,75 +51,28 @@ func (c *FilterWeigherPipelineController) PipelineType() v1alpha1.PipelineType { return v1alpha1.PipelineTypeFilterWeigher } -func (c *FilterWeigherPipelineController) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { - c.processMu.Lock() - defer c.processMu.Unlock() - - // Determine if this is a decision or machine reconciliation. - decision := &v1alpha1.Decision{} - if err := c.Get(ctx, req.NamespacedName, decision); err != nil { - return ctrl.Result{}, client.IgnoreNotFound(err) - } - old := decision.DeepCopy() - if err := c.process(ctx, decision); err != nil { - return ctrl.Result{}, err - } - patch := client.MergeFrom(old) - if err := c.Status().Patch(ctx, decision, patch); err != nil { - return ctrl.Result{}, err - } +// Reconcile is a no-op: this controller schedules machines through its Machine +// watch handler and reconfigures pipelines through its Pipeline watch handler. +// A Reconcile method is required to satisfy the controller builder. +func (c *FilterWeigherPipelineController) Reconcile(context.Context, ctrl.Request) (ctrl.Result, error) { return ctrl.Result{}, nil } func (c *FilterWeigherPipelineController) ProcessNewMachine(ctx context.Context, machine *ironcorev1alpha1.Machine) error { c.processMu.Lock() defer c.processMu.Unlock() - - // Create a decision resource to schedule the machine. - decision := &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - GenerateName: "machine-", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainMachines, - ResourceID: machine.Name, - PipelineRef: corev1.ObjectReference{ - Name: "machines-scheduler", - }, - MachineRef: &corev1.ObjectReference{ - Name: machine.Name, - Namespace: machine.Namespace, - }, - }, - } - - err := c.process(ctx, decision) - if err != nil { - meta.SetStatusCondition(&decision.Status.Conditions, metav1.Condition{ - Type: v1alpha1.DecisionConditionReady, - Status: metav1.ConditionFalse, - Reason: "PipelineRunFailed", - Message: "pipeline run failed: " + err.Error(), - }) - } else { - meta.SetStatusCondition(&decision.Status.Conditions, metav1.Condition{ - Type: v1alpha1.DecisionConditionReady, - Status: metav1.ConditionTrue, - Reason: "PipelineRunSucceeded", - Message: "pipeline run succeeded", - }) - } - return err + return c.process(ctx, machine.DeepCopy()) } -func (c *FilterWeigherPipelineController) process(ctx context.Context, decision *v1alpha1.Decision) error { +func (c *FilterWeigherPipelineController) process(ctx context.Context, machine *ironcorev1alpha1.Machine) error { log := ctrl.LoggerFrom(ctx) startedAt := time.Now() // So we can measure sync duration. - pipeline, ok := c.GetPipeline(decision.Spec.PipelineRef.Name) + const pipelineName = "machines-scheduler" + pipeline, ok := c.GetPipeline(pipelineName) if !ok { - log.Error(nil, "pipeline not found or not ready", "pipelineName", decision.Spec.PipelineRef.Name) - return fmt.Errorf("pipeline not found or not ready: %q", decision.Spec.PipelineRef.Name) + log.Error(nil, "pipeline not found or not ready", "pipelineName", pipelineName) + return fmt.Errorf("pipeline not found or not ready: %q", pipelineName) } // Find all available machine pools. @@ -136,28 +87,12 @@ func (c *FilterWeigherPipelineController) process(ctx context.Context, decision // Execute the scheduling pipeline. request := ironcore.MachinePipelineRequest{Pools: pools.Items} result, err := pipeline.Run(request) - if err == nil { - decision.Status.Result = &result - } - // Record a human-readable explanation of the decision for downstream - // consumers. On error the result is nil and the explanation carries the - // failure reason. - decision.Status.Explanation = lib.GenerateExplanation(decision.Status.Result, err) if err != nil { log.V(1).Error(err, "failed to run scheduler pipeline") return errors.New("failed to run scheduler pipeline") } log.Info("decision processed successfully", "duration", time.Since(startedAt)) - // Set the machine pool ref on the machine. - machine := &ironcorev1alpha1.Machine{} - if err := c.Get(ctx, client.ObjectKey{ - Name: decision.Spec.MachineRef.Name, - Namespace: decision.Spec.MachineRef.Namespace, - }, machine); err != nil { - log.Error(err, "failed to fetch machine for decision") - return err - } // Assign the first machine pool returned by the pipeline. old := machine.DeepCopy() machine.Spec.MachinePoolRef = &corev1.LocalObjectReference{Name: *result.TargetHost} @@ -254,25 +189,6 @@ func (c *FilterWeigherPipelineController) SetupWithManager(mgr manager.Manager, if err != nil { return err } - // Watch decision changes across all clusters. - bldr, err = bldr.WatchesMulticluster( - &v1alpha1.Decision{}, - &handler.EnqueueRequestForObject{}, - predicate.NewPredicateFuncs(func(obj client.Object) bool { - decision := obj.(*v1alpha1.Decision) - if decision.Spec.SchedulingDomain != v1alpha1.SchedulingDomainMachines { - return false - } - // Ignore already decided schedulings. - if decision.Status.Result != nil { - return false - } - return true - }), - ) - if err != nil { - return err - } return bldr.Named("cortex-machine-scheduler"). Complete(c) } diff --git a/internal/scheduling/machines/filter_weigher_pipeline_controller_test.go b/internal/scheduling/machines/filter_weigher_pipeline_controller_test.go index af85f6382..e31032bca 100644 --- a/internal/scheduling/machines/filter_weigher_pipeline_controller_test.go +++ b/internal/scheduling/machines/filter_weigher_pipeline_controller_test.go @@ -12,194 +12,12 @@ import ( "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" - corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" - ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client/fake" ) -func TestFilterWeigherPipelineController_Reconcile(t *testing.T) { - scheme := runtime.NewScheme() - if err := v1alpha1.AddToScheme(scheme); err != nil { - t.Fatalf("Failed to add scheduling scheme: %v", err) - } - if err := ironcorev1alpha1.AddToScheme(scheme); err != nil { - t.Fatalf("Failed to add ironcore scheme: %v", err) - } - - tests := []struct { - name string - decision *v1alpha1.Decision - machinePools []ironcorev1alpha1.MachinePool - machine *ironcorev1alpha1.Machine - expectError bool - expectDecision bool - expectTargetHost string - expectMachinePool string - }{ - { - name: "successful machine decision processing", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainMachines, - ResourceID: "test-machine", - PipelineRef: corev1.ObjectReference{ - Name: "machines-scheduler", - }, - MachineRef: &corev1.ObjectReference{ - Name: "test-machine", - Namespace: "default", - }, - }, - }, - machinePools: []ironcorev1alpha1.MachinePool{ - { - ObjectMeta: metav1.ObjectMeta{Name: "pool1"}, - }, - { - ObjectMeta: metav1.ObjectMeta{Name: "pool2"}, - }, - }, - machine: &ironcorev1alpha1.Machine{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-machine", - Namespace: "default", - }, - Spec: ironcorev1alpha1.MachineSpec{ - Scheduler: "", - }, - }, - expectError: false, - expectDecision: true, - expectTargetHost: "pool1", // NoopFilter returns first pool - expectMachinePool: "pool1", - }, - { - name: "no machine pools available", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-pools", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainMachines, - ResourceID: "test-machine", - PipelineRef: corev1.ObjectReference{ - Name: "machines-scheduler", - }, - MachineRef: &corev1.ObjectReference{ - Name: "test-machine", - Namespace: "default", - }, - }, - }, - machinePools: []ironcorev1alpha1.MachinePool{}, - expectError: true, - expectDecision: false, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - objects := []runtime.Object{tt.decision} - for i := range tt.machinePools { - objects = append(objects, &tt.machinePools[i]) - } - if tt.machine != nil { - objects = append(objects, tt.machine) - } - - client := fake.NewClientBuilder(). - WithScheme(scheme). - WithRuntimeObjects(objects...). - WithStatusSubresource(&v1alpha1.Decision{}). - Build() - - controller := &FilterWeigherPipelineController{ - BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[ironcore.MachinePipelineRequest]]{ - Pipelines: map[string]lib.FilterWeigherPipeline[ironcore.MachinePipelineRequest]{ - "machines-scheduler": createMockPipeline(), - }, - }, - Monitor: lib.FilterWeigherPipelineMonitor{}, - } - controller.Client = client - - req := ctrl.Request{ - NamespacedName: types.NamespacedName{ - Name: tt.decision.Name, - }, - } - - result, err := controller.Reconcile(context.Background(), req) - - if tt.expectError && err == nil { - t.Error("expected error but got none") - return - } - - if !tt.expectError && err != nil { - t.Errorf("expected no error, got: %v", err) - return - } - - if result.RequeueAfter > 0 { - t.Errorf("unexpected requeue: %v", result.RequeueAfter) - } - - // Verify decision status if expected - if tt.expectDecision { - var updatedDecision v1alpha1.Decision - err := client.Get(context.Background(), req.NamespacedName, &updatedDecision) - if err != nil { - t.Errorf("Failed to get updated decision: %v", err) - return - } - - if updatedDecision.Status.Result == nil { - t.Error("expected decision result to be set") - return - } - - if updatedDecision.Status.Result.TargetHost == nil { - t.Error("expected target host to be set") - return - } - - if *updatedDecision.Status.Result.TargetHost != tt.expectTargetHost { - t.Errorf("expected target host %q, got %q", tt.expectTargetHost, *updatedDecision.Status.Result.TargetHost) - } - - // Verify machine was updated with machine pool ref - if tt.machine != nil { - var updatedMachine ironcorev1alpha1.Machine - err := client.Get(context.Background(), types.NamespacedName{ - Name: tt.machine.Name, - Namespace: tt.machine.Namespace, - }, &updatedMachine) - if err != nil { - t.Errorf("Failed to get updated machine: %v", err) - return - } - - if updatedMachine.Spec.MachinePoolRef == nil { - t.Error("expected machine pool ref to be set") - return - } - - if updatedMachine.Spec.MachinePoolRef.Name != tt.expectMachinePool { - t.Errorf("expected machine pool %q, got %q", tt.expectMachinePool, updatedMachine.Spec.MachinePoolRef.Name) - } - } - } - }) - } -} - func TestFilterWeigherPipelineController_InitPipeline(t *testing.T) { controller := &FilterWeigherPipelineController{ Monitor: lib.FilterWeigherPipelineMonitor{}, @@ -293,6 +111,36 @@ func TestFilterWeigherPipelineController_ProcessNewMachine(t *testing.T) { expectMachinePoolAssigned bool expectTargetHost string }{ + { + name: "successful machine scheduling", + machine: &ironcorev1alpha1.Machine{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-machine", + Namespace: "default", + }, + Spec: ironcorev1alpha1.MachineSpec{ + Scheduler: "", + }, + }, + machinePools: []ironcorev1alpha1.MachinePool{ + {ObjectMeta: metav1.ObjectMeta{Name: "pool1"}}, + {ObjectMeta: metav1.ObjectMeta{Name: "pool2"}}, + }, + pipelineConfig: &v1alpha1.Pipeline{ + ObjectMeta: metav1.ObjectMeta{ + Name: "machines-scheduler", + }, + Spec: v1alpha1.PipelineSpec{ + Type: v1alpha1.PipelineTypeFilterWeigher, + SchedulingDomain: v1alpha1.SchedulingDomainMachines, + Filters: []v1alpha1.FilterSpec{}, + Weighers: []v1alpha1.WeigherSpec{}, + }, + }, + expectError: false, + expectMachinePoolAssigned: true, + expectTargetHost: "pool1", // mock pipeline returns first pool + }, { name: "pipeline not configured", machine: &ironcorev1alpha1.Machine{ @@ -350,7 +198,6 @@ func TestFilterWeigherPipelineController_ProcessNewMachine(t *testing.T) { client := fake.NewClientBuilder(). WithScheme(scheme). WithRuntimeObjects(objects...). - WithStatusSubresource(&v1alpha1.Decision{}). Build() controller := &FilterWeigherPipelineController{ @@ -411,14 +258,14 @@ func createMockPipeline() lib.FilterWeigherPipeline[ironcore.MachinePipelineRequ type mockMachinePipeline struct{} -func (m *mockMachinePipeline) Run(request ironcore.MachinePipelineRequest) (v1alpha1.DecisionResult, error) { +func (m *mockMachinePipeline) Run(request ironcore.MachinePipelineRequest) (lib.PipelineResult, error) { if len(request.Pools) == 0 { - return v1alpha1.DecisionResult{}, nil + return lib.PipelineResult{}, nil } // Return the first pool as the target host targetHost := request.Pools[0].Name - return v1alpha1.DecisionResult{ + return lib.PipelineResult{ TargetHost: &targetHost, }, nil } diff --git a/internal/scheduling/manila/external_scheduler_api.go b/internal/scheduling/manila/external_scheduler_api.go index 2dc18d13a..273c282dd 100644 --- a/internal/scheduling/manila/external_scheduler_api.go +++ b/internal/scheduling/manila/external_scheduler_api.go @@ -7,29 +7,24 @@ import ( "bytes" "context" "encoding/json" - "errors" "fmt" "io" "net/http" api "github.com/cobaltcore-dev/cortex/api/external/manila" - "github.com/cobaltcore-dev/cortex/api/v1alpha1" scheduling "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" "github.com/go-logr/logr" - corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/api/meta" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/runtime" logf "sigs.k8s.io/controller-runtime/pkg/log" "sigs.k8s.io/controller-runtime/pkg/metrics" ) var log = logf.Log.WithName("manila-scheduler-api") -type HTTPAPIDelegate interface { - // Process the decision from the API. Should create and return the updated decision. - ProcessNewDecisionFromAPI(ctx context.Context, decision *v1alpha1.Decision) error +// Scheduler runs a manila pipeline for a request and returns the result. +// *FilterWeigherPipelineController satisfies it. +type Scheduler interface { + Schedule(ctx context.Context, pipelineName string, request api.ExternalSchedulerRequest) (scheduling.PipelineResult, error) } type HTTPAPI interface { @@ -38,14 +33,14 @@ type HTTPAPI interface { } type httpAPI struct { - monitor scheduling.APIMonitor - delegate HTTPAPIDelegate + monitor scheduling.APIMonitor + scheduler Scheduler } -func NewAPI(delegate HTTPAPIDelegate) HTTPAPI { +func NewAPI(scheduler Scheduler) HTTPAPI { return &httpAPI{ - monitor: scheduling.NewSchedulerMonitor(), - delegate: delegate, + monitor: scheduling.NewSchedulerMonitor(), + scheduler: scheduler, } } @@ -109,7 +104,6 @@ func (httpAPI *httpAPI) ManilaExternalScheduler(w http.ResponseWriter, r *http.R c.Respond(logr.Logger{}, http.StatusInternalServerError, err, "failed to read request body") return } - raw := runtime.RawExtension{Raw: body} var requestData api.ExternalSchedulerRequest // Copy the raw body to a io.Reader for json deserialization. cp := body @@ -138,36 +132,12 @@ func (httpAPI *httpAPI) ManilaExternalScheduler(w http.ResponseWriter, r *http.R logger.V(1).Info("inferred pipeline name", "pipeline", requestData.Pipeline) } - // Create the decision object in kubernetes. - decision := &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - GenerateName: "manila-", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainManila, - PipelineRef: corev1.ObjectReference{ - Name: requestData.Pipeline, - }, - ResourceID: "", // TODO model out the spec. - ManilaRaw: &raw, - Intent: v1alpha1.SchedulingIntentUnknown, - }, - } - ctx := r.Context() - if err := httpAPI.delegate.ProcessNewDecisionFromAPI(ctx, decision); err != nil { + result, err := httpAPI.scheduler.Schedule(r.Context(), requestData.Pipeline, requestData) + if err != nil { c.Respond(logger, http.StatusInternalServerError, err, fmt.Sprintf("failed to process scheduling decision: %v", err)) return } - // Check if the decision contains status conditions indicating an error. - if meta.IsStatusConditionFalse(decision.Status.Conditions, v1alpha1.DecisionConditionReady) { - c.Respond(logger, http.StatusInternalServerError, errors.New("decision contains error condition"), "decision failed") - return - } - if decision.Status.Result == nil { - c.Respond(logger, http.StatusInternalServerError, errors.New("decision didn't produce a result"), "decision failed") - return - } - hosts := decision.Status.Result.OrderedHosts + hosts := result.OrderedHosts // Ensure we always return an array, never null. The schema requires hosts // to be an array, and a nil slice would serialize to null. if hosts == nil { diff --git a/internal/scheduling/manila/external_scheduler_api_test.go b/internal/scheduling/manila/external_scheduler_api_test.go index dfdb3f534..7012984ea 100644 --- a/internal/scheduling/manila/external_scheduler_api_test.go +++ b/internal/scheduling/manila/external_scheduler_api_test.go @@ -14,26 +14,24 @@ import ( "testing" manilaapi "github.com/cobaltcore-dev/cortex/api/external/manila" - "github.com/cobaltcore-dev/cortex/api/v1alpha1" - - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" ) -type mockHTTPAPIDelegate struct { - processDecisionFunc func(ctx context.Context, decision *v1alpha1.Decision) error +type mockScheduler struct { + scheduleFunc func(ctx context.Context, pipelineName string, request manilaapi.ExternalSchedulerRequest) (lib.PipelineResult, error) } -func (m *mockHTTPAPIDelegate) ProcessNewDecisionFromAPI(ctx context.Context, decision *v1alpha1.Decision) error { - if m.processDecisionFunc != nil { - return m.processDecisionFunc(ctx, decision) +func (m *mockScheduler) Schedule(ctx context.Context, pipelineName string, request manilaapi.ExternalSchedulerRequest) (lib.PipelineResult, error) { + if m.scheduleFunc != nil { + return m.scheduleFunc(ctx, pipelineName, request) } - return nil + return lib.PipelineResult{}, nil } func TestNewAPI(t *testing.T) { - delegate := &mockHTTPAPIDelegate{} + scheduler := &mockScheduler{} - api := NewAPI(delegate) + api := NewAPI(scheduler) if api == nil { t.Fatal("NewAPI returned nil") @@ -44,8 +42,8 @@ func TestNewAPI(t *testing.T) { t.Fatal("NewAPI did not return httpAPI type") } - if httpAPI.delegate != delegate { - t.Error("Delegate not set correctly") + if httpAPI.scheduler != scheduler { + t.Error("Scheduler not set correctly") } if httpAPI.monitor.ApiRequestsTimer == nil { @@ -54,8 +52,7 @@ func TestNewAPI(t *testing.T) { } func TestHTTPAPI_Init(t *testing.T) { - delegate := &mockHTTPAPIDelegate{} - api := NewAPI(delegate) + api := NewAPI(&mockScheduler{}) mux := http.NewServeMux() api.Init(mux) @@ -72,8 +69,7 @@ func TestHTTPAPI_Init(t *testing.T) { } func TestHTTPAPI_canRunScheduler(t *testing.T) { - delegate := &mockHTTPAPIDelegate{} - api := NewAPI(delegate).(*httpAPI) + api := NewAPI(&mockScheduler{}).(*httpAPI) tests := []struct { name string @@ -141,14 +137,27 @@ func TestHTTPAPI_canRunScheduler(t *testing.T) { } func TestHTTPAPI_ManilaExternalScheduler(t *testing.T) { + validBody := func(t *testing.T) string { + req := manilaapi.ExternalSchedulerRequest{ + Hosts: []manilaapi.ExternalSchedulerHost{{ShareHost: "host1"}}, + Weights: map[string]float64{"host1": 1.0}, + Pipeline: "test-pipeline", + } + data, err := json.Marshal(req) + if err != nil { + t.Fatalf("Failed to marshal request data: %v", err) + } + return string(data) + } + tests := []struct { - name string - method string - body string - processDecisionErr error - decisionResult *v1alpha1.Decision - expectedStatus int - expectedHosts []string + name string + method string + body string + scheduleResult lib.PipelineResult + scheduleErr error + expectedStatus int + expectedHosts []string }{ { name: "invalid method", @@ -162,106 +171,31 @@ func TestHTTPAPI_ManilaExternalScheduler(t *testing.T) { expectedStatus: http.StatusBadRequest, }, { - name: "valid request with successful processing", - method: http.MethodPost, - body: func() string { - req := manilaapi.ExternalSchedulerRequest{ - Hosts: []manilaapi.ExternalSchedulerHost{ - {ShareHost: "host1"}, - }, - Weights: map[string]float64{ - "host1": 1.0, - }, - Pipeline: "test-pipeline", - } - data, err := json.Marshal(req) - if err != nil { - t.Fatalf("Failed to marshal request data: %v", err) - } - return string(data) - }(), - decisionResult: &v1alpha1.Decision{ - Status: v1alpha1.DecisionStatus{ - Result: &v1alpha1.DecisionResult{ - OrderedHosts: []string{"host1"}, - }, - }, - }, + name: "valid request with successful processing", + method: http.MethodPost, + body: validBody(t), + scheduleResult: lib.PipelineResult{OrderedHosts: []string{"host1"}}, expectedStatus: http.StatusOK, expectedHosts: []string{"host1"}, }, { - name: "processing error", - method: http.MethodPost, - body: func() string { - req := manilaapi.ExternalSchedulerRequest{ - Hosts: []manilaapi.ExternalSchedulerHost{ - {ShareHost: "host1"}, - }, - Weights: map[string]float64{ - "host1": 1.0, - }, - Pipeline: "test-pipeline", - } - data, err := json.Marshal(req) - if err != nil { - t.Fatalf("Failed to marshal request data: %v", err) - } - return string(data) - }(), - processDecisionErr: errors.New("processing failed"), - expectedStatus: http.StatusInternalServerError, - }, - { - name: "decision failed", - method: http.MethodPost, - body: func() string { - req := manilaapi.ExternalSchedulerRequest{ - Hosts: []manilaapi.ExternalSchedulerHost{ - {ShareHost: "host1"}, - }, - Weights: map[string]float64{ - "host1": 1.0, - }, - Pipeline: "test-pipeline", - } - data, err := json.Marshal(req) - if err != nil { - t.Fatalf("Failed to marshal request data: %v", err) - } - return string(data) - }(), - decisionResult: &v1alpha1.Decision{ - Status: v1alpha1.DecisionStatus{ - Conditions: []metav1.Condition{ - { - Type: v1alpha1.DecisionConditionReady, - Status: metav1.ConditionFalse, - Reason: "SchedulingError", - }, - }, - }, - }, + name: "processing error", + method: http.MethodPost, + body: validBody(t), + scheduleErr: errors.New("processing failed"), expectedStatus: http.StatusInternalServerError, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - delegate := &mockHTTPAPIDelegate{ - processDecisionFunc: func(ctx context.Context, decision *v1alpha1.Decision) error { - if tt.processDecisionErr != nil { - return tt.processDecisionErr - } - if tt.decisionResult != nil { - decision.Status = tt.decisionResult.Status - return nil - } - return nil + scheduler := &mockScheduler{ + scheduleFunc: func(ctx context.Context, pipelineName string, request manilaapi.ExternalSchedulerRequest) (lib.PipelineResult, error) { + return tt.scheduleResult, tt.scheduleErr }, } - api := NewAPI(delegate).(*httpAPI) + api := NewAPI(scheduler).(*httpAPI) var body *strings.Reader if tt.body != "" { @@ -300,8 +234,7 @@ func TestHTTPAPI_ManilaExternalScheduler(t *testing.T) { } func TestHTTPAPI_inferPipelineName(t *testing.T) { - delegate := &mockHTTPAPIDelegate{} - api := NewAPI(delegate).(*httpAPI) + api := NewAPI(&mockScheduler{}).(*httpAPI) tests := []struct { name string @@ -347,31 +280,25 @@ func TestHTTPAPI_inferPipelineName(t *testing.T) { } } -func TestHTTPAPI_ManilaExternalScheduler_DecisionCreation(t *testing.T) { - var capturedDecision *v1alpha1.Decision - delegate := &mockHTTPAPIDelegate{ - processDecisionFunc: func(ctx context.Context, decision *v1alpha1.Decision) error { - capturedDecision = decision - // Set a successful result to avoid "decision didn't produce a result" error - decision.Status.Result = &v1alpha1.DecisionResult{ - OrderedHosts: []string{"host1"}, - } - return nil +// When the request carries no pipeline, the handler infers the default name and +// passes it to the scheduler. +func TestHTTPAPI_ManilaExternalScheduler_InfersPipeline(t *testing.T) { + var capturedPipeline string + var capturedRequest manilaapi.ExternalSchedulerRequest + scheduler := &mockScheduler{ + scheduleFunc: func(ctx context.Context, pipelineName string, request manilaapi.ExternalSchedulerRequest) (lib.PipelineResult, error) { + capturedPipeline = pipelineName + capturedRequest = request + return lib.PipelineResult{OrderedHosts: []string{"host1"}}, nil }, } - api := NewAPI(delegate).(*httpAPI) + api := NewAPI(scheduler).(*httpAPI) requestData := manilaapi.ExternalSchedulerRequest{ - Hosts: []manilaapi.ExternalSchedulerHost{ - {ShareHost: "host1"}, - }, - Weights: map[string]float64{ - "host1": 1.0, - }, - Pipeline: "test-pipeline", + Hosts: []manilaapi.ExternalSchedulerHost{{ShareHost: "host1"}}, + Weights: map[string]float64{"host1": 1.0}, } - body, err := json.Marshal(requestData) if err != nil { t.Fatalf("Failed to marshal request data: %v", err) @@ -384,25 +311,10 @@ func TestHTTPAPI_ManilaExternalScheduler_DecisionCreation(t *testing.T) { if w.Code != http.StatusOK { t.Errorf("Expected status %d, got %d", http.StatusOK, w.Code) } - - if capturedDecision == nil { - t.Fatal("Decision was not captured") + if capturedPipeline != "manila-external-scheduler" { + t.Errorf("Expected inferred pipeline 'manila-external-scheduler', got %q", capturedPipeline) } - - // Verify decision fields - if capturedDecision.Spec.SchedulingDomain != v1alpha1.SchedulingDomainManila { - t.Errorf("Expected scheduling domain %s, got %s", v1alpha1.SchedulingDomainManila, capturedDecision.Spec.SchedulingDomain) - } - - if capturedDecision.Spec.PipelineRef.Name != "test-pipeline" { - t.Errorf("Expected pipeline 'test-pipeline', got %s", capturedDecision.Spec.PipelineRef.Name) - } - - if capturedDecision.GenerateName != "manila-" { - t.Errorf("Expected generate name 'manila-', got %s", capturedDecision.GenerateName) - } - - if capturedDecision.Spec.ManilaRaw == nil { - t.Error("ManilaRaw should not be nil") + if len(capturedRequest.Hosts) != 1 || capturedRequest.Hosts[0].ShareHost != "host1" { + t.Errorf("Request not passed through to scheduler: %+v", capturedRequest) } } diff --git a/internal/scheduling/manila/filter_weigher_pipeline_controller.go b/internal/scheduling/manila/filter_weigher_pipeline_controller.go index f6442f697..cc7ced78e 100644 --- a/internal/scheduling/manila/filter_weigher_pipeline_controller.go +++ b/internal/scheduling/manila/filter_weigher_pipeline_controller.go @@ -5,16 +5,12 @@ package manila import ( "context" - "encoding/json" - "errors" "fmt" "sync" "time" api "github.com/cobaltcore-dev/cortex/api/external/manila" "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "k8s.io/apimachinery/pkg/api/meta" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" "github.com/cobaltcore-dev/cortex/internal/scheduling/manila/plugins/filters" @@ -27,17 +23,15 @@ import ( "sigs.k8s.io/controller-runtime/pkg/predicate" ) -// The decision pipeline controller takes decision resources containing a -// placement request spec and runs the scheduling pipeline to make a decision. -// This decision is then written back to the decision resource status. -// -// Additionally, the controller watches for pipeline and step changes to -// reconfigure the pipelines as needed. +// FilterWeigherPipelineController owns the manila filter-weigher pipelines: it +// watches Pipeline and Knowledge resources to (re)configure the pipelines and +// runs a pipeline on demand via Schedule, called directly by the external +// scheduler HTTP API. type FilterWeigherPipelineController struct { // Toolbox shared between all pipeline controllers. lib.BasePipelineController[lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]] - // Mutex to only allow one process at a time + // Mutex to only allow one scheduling run at a time. processMu sync.Mutex // Monitor to pass down to all pipelines. @@ -49,83 +43,34 @@ func (c *FilterWeigherPipelineController) PipelineType() v1alpha1.PipelineType { return v1alpha1.PipelineTypeFilterWeigher } -// Callback executed when kubernetes asks to reconcile a decision resource. -func (c *FilterWeigherPipelineController) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { - c.processMu.Lock() - defer c.processMu.Unlock() - - decision := &v1alpha1.Decision{} - if err := c.Get(ctx, req.NamespacedName, decision); err != nil { - return ctrl.Result{}, client.IgnoreNotFound(err) - } - old := decision.DeepCopy() - if err := c.process(ctx, decision); err != nil { - return ctrl.Result{}, err - } - patch := client.MergeFrom(old) - if err := c.Status().Patch(ctx, decision, patch); err != nil { - return ctrl.Result{}, err - } +// Reconcile is a no-op: this controller reconfigures pipelines through its +// Pipeline/Knowledge watch handlers and runs placements via Schedule. A +// Reconcile method is required to satisfy the controller builder. +func (c *FilterWeigherPipelineController) Reconcile(context.Context, ctrl.Request) (ctrl.Result, error) { return ctrl.Result{}, nil } -// Process the decision from the API. Should create and return the updated decision. -func (c *FilterWeigherPipelineController) ProcessNewDecisionFromAPI(ctx context.Context, decision *v1alpha1.Decision) error { +// Schedule runs the named pipeline for the given request and returns the +// result. It serializes runs with the process lock. +func (c *FilterWeigherPipelineController) Schedule(ctx context.Context, pipelineName string, request api.ExternalSchedulerRequest) (lib.PipelineResult, error) { c.processMu.Lock() defer c.processMu.Unlock() - - err := c.process(ctx, decision) - if err != nil { - meta.SetStatusCondition(&decision.Status.Conditions, metav1.Condition{ - Type: v1alpha1.DecisionConditionReady, - Status: metav1.ConditionFalse, - Reason: "PipelineRunFailed", - Message: "pipeline run failed: " + err.Error(), - }) - } else { - meta.SetStatusCondition(&decision.Status.Conditions, metav1.Condition{ - Type: v1alpha1.DecisionConditionReady, - Status: metav1.ConditionTrue, - Reason: "PipelineRunSucceeded", - Message: "pipeline run succeeded", - }) - } - return err -} - -func (c *FilterWeigherPipelineController) process(ctx context.Context, decision *v1alpha1.Decision) error { log := ctrl.LoggerFrom(ctx) startedAt := time.Now() // So we can measure sync duration. - pipeline, ok := c.GetPipeline(decision.Spec.PipelineRef.Name) + pipeline, ok := c.GetPipeline(pipelineName) if !ok { - log.Error(nil, "skipping decision, pipeline not found or not ready", "pipelineName", decision.Spec.PipelineRef.Name) - return fmt.Errorf("pipeline not found or not ready: %q", decision.Spec.PipelineRef.Name) - } - if decision.Spec.ManilaRaw == nil { - log.Error(nil, "skipping decision, no manilaRaw spec defined") - return errors.New("no manilaRaw spec defined") - } - var request api.ExternalSchedulerRequest - if err := json.Unmarshal(decision.Spec.ManilaRaw.Raw, &request); err != nil { - log.Error(err, "failed to unmarshal manilaRaw spec") - return err + log.Error(nil, "pipeline not found or not ready", "pipelineName", pipelineName) + return lib.PipelineResult{}, fmt.Errorf("pipeline not found or not ready: %q", pipelineName) } result, err := pipeline.Run(request) - if err == nil { - decision.Status.Result = &result - } - // Record a human-readable explanation of the decision for downstream - // consumers. On error the result is nil and the explanation carries the - // failure reason. - decision.Status.Explanation = lib.GenerateExplanation(decision.Status.Result, err) if err != nil { log.Error(err, "failed to run pipeline") - return err + return lib.PipelineResult{}, err } log.Info("decision processed successfully", "duration", time.Since(startedAt)) - return nil + return result, nil } // The base controller will delegate the pipeline creation down to this method. @@ -186,25 +131,6 @@ func (c *FilterWeigherPipelineController) SetupWithManager(mgr manager.Manager, if err != nil { return err } - // Watch decision changes across all clusters. - bldr, err = bldr.WatchesMulticluster( - &v1alpha1.Decision{}, - &handler.EnqueueRequestForObject{}, - predicate.NewPredicateFuncs(func(obj client.Object) bool { - decision := obj.(*v1alpha1.Decision) - if decision.Spec.SchedulingDomain != v1alpha1.SchedulingDomainManila { - return false - } - // Ignore already decided schedulings. - if decision.Status.Result != nil { - return false - } - return true - }), - ) - if err != nil { - return err - } - return bldr.Named("cortex-manila-decisions"). + return bldr.Named("cortex-manila-pipelines"). Complete(c) } diff --git a/internal/scheduling/manila/filter_weigher_pipeline_controller_test.go b/internal/scheduling/manila/filter_weigher_pipeline_controller_test.go index 76dbd6f93..418c9297d 100644 --- a/internal/scheduling/manila/filter_weigher_pipeline_controller_test.go +++ b/internal/scheduling/manila/filter_weigher_pipeline_controller_test.go @@ -4,44 +4,26 @@ package manila import ( - "context" - "encoding/json" "testing" - corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/runtime" - "k8s.io/apimachinery/pkg/types" - ctrl "sigs.k8s.io/controller-runtime" - "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client/fake" api "github.com/cobaltcore-dev/cortex/api/external/manila" "github.com/cobaltcore-dev/cortex/api/scheduling" "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "github.com/sapcc/go-bits/must" - "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/storage" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" ) -func TestFilterWeigherPipelineController_Reconcile(t *testing.T) { +func TestFilterWeigherPipelineController_Schedule(t *testing.T) { scheme := runtime.NewScheme() if err := v1alpha1.AddToScheme(scheme); err != nil { t.Fatalf("Failed to add v1alpha1 scheme: %v", err) } - manilaRequest := api.ExternalSchedulerRequest{ - Spec: map[string]any{ - "share_id": "test-share-id", - "size": 10, - }, - Context: api.ManilaRequestContext{ - ProjectID: "test-project", - UserID: "test-user", - RequestID: "req-123", - GlobalRequestID: "global-req-123", - }, + request := api.ExternalSchedulerRequest{ Hosts: []api.ExternalSchedulerHost{ {ShareHost: "manila-share-1@backend1"}, {ShareHost: "manila-share-2@backend2"}, @@ -51,143 +33,55 @@ func TestFilterWeigherPipelineController_Reconcile(t *testing.T) { Options: scheduling.Options{}, } - manilaRaw, err := json.Marshal(manilaRequest) - if err != nil { - t.Fatalf("Failed to marshal manila request: %v", err) - } - tests := []struct { name string - decision *v1alpha1.Decision - pipeline *v1alpha1.Pipeline + pipelineName string + configured bool expectError bool - expectResult bool }{ { - name: "successful manila decision processing", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainManila, - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - ManilaRaw: &runtime.RawExtension{ - Raw: manilaRaw, - }, - }, - }, - pipeline: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainManila, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, + name: "successful scheduling run", + pipelineName: "test-pipeline", + configured: true, expectError: false, - expectResult: true, }, { - name: "decision without manilaRaw spec", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-raw", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainManila, - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - ManilaRaw: nil, - }, - }, - pipeline: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainManila, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, + name: "pipeline not found", + pipelineName: "nonexistent-pipeline", + configured: false, expectError: true, - expectResult: false, - }, - { - name: "pipeline not found", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-pipeline", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainManila, - PipelineRef: corev1.ObjectReference{ - Name: "nonexistent-pipeline", - }, - ManilaRaw: &runtime.RawExtension{ - Raw: manilaRaw, - }, - }, - }, - pipeline: nil, - expectError: true, - expectResult: false, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - objects := []client.Object{tt.decision} - if tt.pipeline != nil { - objects = append(objects, tt.pipeline) - } - - client := fake.NewClientBuilder(). - WithScheme(scheme). - WithObjects(objects...). - WithStatusSubresource(&v1alpha1.Decision{}). - Build() + cl := fake.NewClientBuilder().WithScheme(scheme).Build() controller := &FilterWeigherPipelineController{ BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]]{ - Client: client, + Client: cl, Pipelines: make(map[string]lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]), }, Monitor: lib.FilterWeigherPipelineMonitor{}, } - if tt.pipeline != nil { + if tt.configured { initResult := controller.InitPipeline(t.Context(), v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: tt.pipeline.Name, + ObjectMeta: metav1.ObjectMeta{Name: "test-pipeline"}, + Spec: v1alpha1.PipelineSpec{ + Type: v1alpha1.PipelineTypeFilterWeigher, + SchedulingDomain: v1alpha1.SchedulingDomainManila, + Filters: []v1alpha1.FilterSpec{}, + Weighers: []v1alpha1.WeigherSpec{}, }, - Spec: tt.pipeline.Spec, }) - if err != nil { - t.Fatalf("Failed to init pipeline: %v", err) + if len(initResult.FilterErrors) > 0 || len(initResult.WeigherErrors) > 0 { + t.Fatalf("Failed to init pipeline: %v", initResult) } - controller.Pipelines[tt.pipeline.Name] = initResult.Pipeline + controller.Pipelines["test-pipeline"] = initResult.Pipeline } - req := ctrl.Request{ - NamespacedName: types.NamespacedName{ - Name: tt.decision.Name, - Namespace: tt.decision.Namespace, - }, - } - - result, err := controller.Reconcile(context.Background(), req) + result, err := controller.Schedule(t.Context(), tt.pipelineName, request) if tt.expectError && err == nil { t.Error("Expected error but got none") @@ -195,227 +89,9 @@ func TestFilterWeigherPipelineController_Reconcile(t *testing.T) { if !tt.expectError && err != nil { t.Errorf("Expected no error but got: %v", err) } - - if result.RequeueAfter > 0 { - t.Error("Expected no requeue") - } - - var updatedDecision v1alpha1.Decision - if err := client.Get(context.Background(), req.NamespacedName, &updatedDecision); err != nil { - t.Fatalf("Failed to get updated decision: %v", err) - } - - if tt.expectResult && updatedDecision.Status.Result == nil { - t.Error("Expected result to be set but was nil") - } - if !tt.expectResult && updatedDecision.Status.Result != nil { - t.Error("Expected result to be nil but was set") - } - }) - } -} - -func TestFilterWeigherPipelineController_ProcessNewDecisionFromAPI(t *testing.T) { - scheme := runtime.NewScheme() - if err := v1alpha1.AddToScheme(scheme); err != nil { - t.Fatalf("Failed to add v1alpha1 scheme: %v", err) - } - - manilaRequest := api.ExternalSchedulerRequest{ - Spec: map[string]any{ - "share_id": "test-share-id", - "size": 10, - }, - Context: api.ManilaRequestContext{ - ProjectID: "test-project", - UserID: "test-user", - RequestID: "req-123", - GlobalRequestID: "global-req-123", - }, - Hosts: []api.ExternalSchedulerHost{ - {ShareHost: "manila-share-1@backend1"}, - {ShareHost: "manila-share-2@backend2"}, - }, - Weights: map[string]float64{"manila-share-1@backend1": 1.0, "manila-share-2@backend2": 0.5}, - Pipeline: "test-pipeline", - } - - manilaRaw, err := json.Marshal(manilaRequest) - if err != nil { - t.Fatalf("Failed to marshal manila request: %v", err) - } - - tests := []struct { - name string - decision *v1alpha1.Decision - pipelineConfig *v1alpha1.Pipeline - expectError bool - expectResult bool - }{ - { - name: "successful decision processing with creation", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - GenerateName: "test-decision-", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainManila, - ResourceID: "test-uuid-1", - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - ManilaRaw: &runtime.RawExtension{ - Raw: manilaRaw, - }, - }, - }, - pipelineConfig: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainManila, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - expectError: false, - expectResult: true, - }, - { - name: "successful decision processing without creation", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-create", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainManila, - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - ManilaRaw: &runtime.RawExtension{ - Raw: manilaRaw, - }, - }, - }, - pipelineConfig: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainManila, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - expectError: false, - expectResult: true, - }, - { - name: "pipeline not configured", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-pipeline", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainManila, - PipelineRef: corev1.ObjectReference{ - Name: "nonexistent-pipeline", - }, - ManilaRaw: &runtime.RawExtension{ - Raw: manilaRaw, - }, - }, - }, - pipelineConfig: nil, - expectError: true, - expectResult: false, - }, - { - name: "decision without manilaRaw spec", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-raw", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainManila, - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - ManilaRaw: nil, - }, - }, - pipelineConfig: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainManila, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - expectError: true, - expectResult: false, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - objects := []client.Object{} - if tt.pipelineConfig != nil { - objects = append(objects, tt.pipelineConfig) - } - - client := fake.NewClientBuilder(). - WithScheme(scheme). - WithObjects(objects...). - WithStatusSubresource(&v1alpha1.Decision{}). - Build() - - controller := &FilterWeigherPipelineController{ - BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]]{ - Client: client, - Pipelines: make(map[string]lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]), - PipelineConfigs: make(map[string]v1alpha1.Pipeline), - }, - Monitor: lib.FilterWeigherPipelineMonitor{}, - } - - if tt.pipelineConfig != nil { - controller.PipelineConfigs[tt.pipelineConfig.Name] = *tt.pipelineConfig - initResult := controller.InitPipeline(t.Context(), *tt.pipelineConfig) - if len(initResult.FilterErrors) > 0 || len(initResult.WeigherErrors) > 0 { - t.Fatalf("Failed to init pipeline: %v", initResult) - } - controller.Pipelines[tt.pipelineConfig.Name] = initResult.Pipeline - } - - if tt.decision.Spec.ManilaRaw != nil { - req := manilaRequest - req.Options = scheduling.Options{} - raw, marshalErr := json.Marshal(req) - if marshalErr != nil { - t.Fatalf("Failed to marshal request with options: %v", marshalErr) - } - tt.decision.Spec.ManilaRaw = &runtime.RawExtension{Raw: raw} - } - - err := controller.ProcessNewDecisionFromAPI(context.Background(), tt.decision) - if !tt.expectError && err != nil { - t.Errorf("Expected no error but got: %v", err) - } - if !tt.expectError { - if tt.expectResult && tt.decision.Status.Result == nil { - t.Error("expected decision result to be set in original decision object") + if len(result.OrderedHosts) == 0 { + t.Error("Expected ordered hosts but got none") } } }) @@ -433,16 +109,14 @@ func TestFilterWeigherPipelineController_PipelineType(t *testing.T) { } func TestFilterWeigherPipelineController_InitPipeline(t *testing.T) { - scheme := runtime.NewScheme() - if err := v1alpha1.AddToScheme(scheme); err != nil { - t.Fatalf("Failed to add v1alpha1 scheme: %v", err) + controller := &FilterWeigherPipelineController{ + Monitor: lib.FilterWeigherPipelineMonitor{}, } tests := []struct { name string filters []v1alpha1.FilterSpec weighers []v1alpha1.WeigherSpec - knowledges []client.Object expectNonCriticalError bool expectCriticalError bool expectUnknownFilter bool @@ -452,53 +126,6 @@ func TestFilterWeigherPipelineController_InitPipeline(t *testing.T) { name: "empty steps", filters: []v1alpha1.FilterSpec{}, weighers: []v1alpha1.WeigherSpec{}, - knowledges: []client.Object{}, - expectNonCriticalError: false, - expectCriticalError: false, - expectUnknownFilter: false, - expectUnknownWeigher: false, - }, - { - name: "supported netapp step", - weighers: []v1alpha1.WeigherSpec{ - { - Name: "netapp_cpu_usage_balancing", - Params: []v1alpha1.Parameter{ - {Key: "AvgCPUUsageLowerBound", FloatValue: new(0.0)}, - {Key: "AvgCPUUsageUpperBound", FloatValue: new(90.0)}, - {Key: "MaxCPUUsageLowerBound", FloatValue: new(0.0)}, - {Key: "MaxCPUUsageUpperBound", FloatValue: new(100.0)}, - }, - }, - }, - knowledges: []client.Object{ - &v1alpha1.Knowledge{ - ObjectMeta: metav1.ObjectMeta{ - Name: "netapp-storage-pool-cpu-usage-manila", - }, - Status: v1alpha1.KnowledgeStatus{ - Conditions: []metav1.Condition{ - { - Type: v1alpha1.KnowledgeConditionReady, - Status: metav1.ConditionTrue, - }, - }, - Raw: must.Return(v1alpha1.BoxFeatureList([]storage.StoragePoolCPUUsage{ - { - StoragePoolName: "manila-share-1@backend1", - AvgCPUUsagePct: 50, - MaxCPUUsagePct: 80, - }, - { - StoragePoolName: "manila-share-2@backend2", - AvgCPUUsagePct: 20, - MaxCPUUsagePct: 40, - }, - })), - RawLength: 2, - }, - }, - }, expectNonCriticalError: false, expectCriticalError: false, expectUnknownFilter: false, @@ -508,7 +135,7 @@ func TestFilterWeigherPipelineController_InitPipeline(t *testing.T) { name: "unsupported step", filters: []v1alpha1.FilterSpec{ { - Name: "unsupported-plugin", + Name: "test-plugin", }, }, expectNonCriticalError: false, @@ -520,16 +147,6 @@ func TestFilterWeigherPipelineController_InitPipeline(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - client := fake.NewClientBuilder(). - WithScheme(scheme). - WithObjects(tt.knowledges...). - WithStatusSubresource(&v1alpha1.Decision{}). - Build() - controller := &FilterWeigherPipelineController{ - Monitor: lib.FilterWeigherPipelineMonitor{}, - } - controller.Client = client // Through basepipelinecontroller - initResult := controller.InitPipeline(t.Context(), v1alpha1.Pipeline{ ObjectMeta: metav1.ObjectMeta{ Name: "test-pipeline", @@ -542,19 +159,19 @@ func TestFilterWeigherPipelineController_InitPipeline(t *testing.T) { }, }) - if !tt.expectCriticalError && len(initResult.FilterErrors) > 0 { - t.Errorf("Expected no critical error but got: %v", initResult.FilterErrors) - } if tt.expectCriticalError && len(initResult.FilterErrors) == 0 { - t.Error("Expected critical error but got none") + t.Error("Expected error but got none") } - - if !tt.expectNonCriticalError && len(initResult.WeigherErrors) > 0 { - t.Errorf("Expected no non-critical error but got: %v", initResult.WeigherErrors) + if !tt.expectCriticalError && len(initResult.FilterErrors) > 0 { + t.Errorf("Expected no error but got: %v", initResult.FilterErrors) } + if tt.expectNonCriticalError && len(initResult.WeigherErrors) == 0 { t.Error("Expected non-critical error but got none") } + if !tt.expectNonCriticalError && len(initResult.WeigherErrors) > 0 { + t.Errorf("Expected no non-critical error but got: %v", initResult.WeigherErrors) + } }) } } diff --git a/internal/scheduling/nova/adapter.go b/internal/scheduling/nova/adapter.go new file mode 100644 index 000000000..72f57f28f --- /dev/null +++ b/internal/scheduling/nova/adapter.go @@ -0,0 +1,194 @@ +// Copyright SAP SE +// SPDX-License-Identifier: Apache-2.0 + +package nova + +import ( + api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" +) + +// ToSchedulerRequest translates the Nova over-the-wire request into the typed, +// Nova-agnostic compute.PlacementRequest the pipeline consumes. This is the one +// place that reads Nova's loosely-typed scheduler hints and flavor/image extra +// specs; every downstream filter/weigher sees only resolved typed fields. +// +// intent and pipeline are resolved by the caller (the HTTP API layer) and passed +// in. +func ToSchedulerRequest( + wire api.ExternalSchedulerRequest, + intent v1alpha1.SchedulingIntent, + pipeline string, +) compute.PlacementRequest { + + spec := wire.Spec.Data + + req := compute.PlacementRequest{ + Pipeline: pipeline, + Hosts: toComputeHosts(wire.Hosts), + Weights: wire.Weights, + Options: wire.Options, + Intent: intent, + + InstanceUUID: spec.InstanceUUID, + ProjectID: firstNonEmpty(spec.ProjectID, wire.Context.ProjectID), + UserID: spec.UserID, + AvailabilityZone: spec.AvailabilityZone, + NumInstances: spec.NumInstances, + + IgnoreHosts: derefStrings(spec.IgnoreHosts), + ForceHosts: derefStrings(spec.ForceHosts), + ForceNodes: derefStrings(spec.ForceNodes), + + HardwareVersion: spec.Flavor.Data.ExtraSpecs["quota:hw_version"], + + SourceHost: hintStr(spec, "source_host"), + DomainName: hintStr(spec, "domain_name"), + ResourceGroup: hintStr(spec, api.HintKeyResourceGroup), + + RequestID: wire.Context.RequestID, + GlobalRequestID: derefString(wire.Context.GlobalRequestID), + } + + // Flavor sizing. + req.Flavor = api.FlavorSpec{ + VCPUs: spec.Flavor.Data.VCPUs, + MemoryMB: spec.Flavor.Data.MemoryMB, + } + + // Hypervisor family. + req.HypervisorFamily = toHypervisorFamily(wire) + + // Whether to gather all placement candidates (ignore the caller's host + // preselection) is determined by the hypervisor family: cortex can enumerate + // KVM hypervisors itself, whereas VMware placement relies on the host list + // Nova sent in the request. + req.GatherAllCandidates = req.HypervisorFamily == api.HypervisorFamilyKVM + + // Image. + req.ImageHypervisorType = toImageHypervisorType(spec.Image.Data) + + // Traits, capabilities, accelerators from flavor extra specs. + req.RequiredTraits, req.ForbiddenTraits = api.SplitTraits(spec.Flavor.Data.ExtraSpecs) + req.RequestedCapabilities = api.SplitCapabilities(spec.Flavor.Data.ExtraSpecs) + req.AcceleratorsRequested = api.RequestsAccelerators(spec.Flavor.Data.ExtraSpecs) + + // Nested objects. + req.RequestedDestination = toRequestedDestination(spec.RequestedDestination) + req.InstanceGroup = toInstanceGroup(spec.InstanceGroup) + + return req +} + +func toComputeHosts(hosts []api.ExternalSchedulerHost) []compute.Host { + out := make([]compute.Host, len(hosts)) + for i, h := range hosts { + out[i] = compute.Host{ComputeHost: h.ComputeHost, HypervisorHostname: h.HypervisorHostname} + } + return out +} + +// toHypervisorFamily resolves the coarse family from the wire hypervisor type. +// It is empty when unresolved. +func toHypervisorFamily(wire api.ExternalSchedulerRequest) api.HypervisorFamily { + hv, err := wire.GetHypervisorType() + if err != nil { + return "" + } + switch hv { + case api.HypervisorTypeQEMU, api.HypervisorTypeCH: + return api.HypervisorFamilyKVM + case api.HypervisorTypeVMware: + return api.HypervisorFamilyVMware + } + return "" +} + +// toImageHypervisorType maps the image-declared hypervisor type. It returns +// ImageHypervisorTypeUnparseable when the image declared a type that could not +// be parsed, distinct from ImageHypervisorTypeUnset when none was declared. +func toImageHypervisorType(image api.NovaImageMeta) api.ImageHypervisorType { + hv, err := image.GetHypervisorType() + if err != nil { + // GetHypervisorType errors both when nothing is declared and when a + // declared value is unrecognized. Distinguish them: if a known key is + // present, the value was unparseable. + if image.Properties.Data != nil { + for _, key := range []string{"img_hv_type", "hypervisor_type"} { + if _, ok := image.Properties.Data[key]; ok { + return api.ImageHypervisorTypeUnparseable + } + } + } + return api.ImageHypervisorTypeUnset + } + switch hv { + case api.NovaImageMetaHVTypeVMware: + return api.ImageHypervisorTypeVMware + case api.NovaImageMetaHVTypeBaremetal: + return api.ImageHypervisorTypeBaremetal + case api.NovaImageMetaHVTypeKVM: + return api.ImageHypervisorTypeKVM + } + return api.ImageHypervisorTypeUnset +} + +func toRequestedDestination(rd *api.NovaObject[api.NovaRequestedDestination]) *api.RequestedDestination { + if rd == nil { + return nil + } + return &api.RequestedDestination{ + Host: rd.Data.Host, + Aggregates: rd.Data.Aggregates, + } +} + +func toInstanceGroup(ig *api.NovaObject[api.NovaInstanceGroup]) *api.InstanceGroupSpec { + if ig == nil { + return nil + } + out := &api.InstanceGroupSpec{ + UUID: ig.Data.UUID, + Policy: ig.Data.Policy, + Hosts: ig.Data.Hosts, + Members: ig.Data.Members, + } + if ig.Data.Rules != nil { + if v, ok := ig.Data.Rules["max_server_per_host"]; ok { + if n, ok := v.(int); ok { + out.MaxServerPerHost = n + } + } + } + return out +} + +func hintStr(spec api.NovaSpec, key string) string { + s, err := spec.GetSchedulerHintStr(key) + if err != nil { + return "" + } + return s +} + +func derefStrings(p *[]string) []string { + if p == nil { + return nil + } + return *p +} + +func derefString(p *string) string { + if p == nil { + return "" + } + return *p +} + +func firstNonEmpty(a, b string) string { + if a != "" { + return a + } + return b +} diff --git a/internal/scheduling/nova/candidate_gatherer.go b/internal/scheduling/nova/candidate_gatherer.go index a493aaf42..b1b869f64 100644 --- a/internal/scheduling/nova/candidate_gatherer.go +++ b/internal/scheduling/nova/candidate_gatherer.go @@ -6,9 +6,9 @@ package nova import ( "context" "fmt" - "strings" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "sigs.k8s.io/controller-runtime/pkg/client" ) @@ -17,32 +17,28 @@ import ( // for a given Nova scheduling request. type CandidateGatherer interface { // Gather all placement candidates and mutate the request accordingly. - MutateWithAllCandidates(ctx context.Context, request *api.ExternalSchedulerRequest) error + MutateWithAllCandidates(ctx context.Context, request *compute.PlacementRequest) error } // candidateGatherer is the default implementation of CandidateGatherer // for Nova scheduling requests. type candidateGatherer struct{ client.Client } +// NewCandidateGatherer builds the default Nova candidate gatherer backed by the +// given client. It is handed to the compute.Placement, which invokes it when a +// request opts into gathering all placement candidates. +func NewCandidateGatherer(c client.Client) CandidateGatherer { + return &candidateGatherer{Client: c} +} + // MutateWithAllCandidates gathers all placement candidates and mutates // the request accordingly. -func (g *candidateGatherer) MutateWithAllCandidates(ctx context.Context, request *api.ExternalSchedulerRequest) error { +func (g *candidateGatherer) MutateWithAllCandidates(ctx context.Context, request *compute.PlacementRequest) error { // Currently we can only get candidates for kvm placements. - hvType, ok := request.Spec.Data.Flavor.Data.ExtraSpecs["capabilities:hypervisor_type"] - if !ok { - return fmt.Errorf( - "missing hypervisor_type in flavor extra specs: %v", - request.Spec.Data.Flavor.Data.ExtraSpecs, - ) - } - switch strings.ToLower(hvType) { - case "qemu", "ch": - // Supported hypervisor type. - default: - // Unsupported hypervisor type, do nothing. + if request.HypervisorFamily != api.HypervisorFamilyKVM { return fmt.Errorf( - "cannot gather all placement candidates for hypervisor type %q", - request.Spec.Data.Flavor.Data.ExtraSpecs["capabilities:hypervisor_type"], + "cannot gather all placement candidates for hypervisor family %q", + request.HypervisorFamily, ) } @@ -51,10 +47,10 @@ func (g *candidateGatherer) MutateWithAllCandidates(ctx context.Context, request if err := g.List(ctx, hypervisorList); err != nil { return err } - hosts := make([]api.ExternalSchedulerHost, 0, len(hypervisorList.Items)) + hosts := make([]compute.Host, 0, len(hypervisorList.Items)) weights := make(map[string]float64, len(hypervisorList.Items)) for _, hv := range hypervisorList.Items { - host := api.ExternalSchedulerHost{ + host := compute.Host{ // For KVM hosts, compute host name and hypervisor hostname is identical. ComputeHost: hv.Name, HypervisorHostname: hv.Name, diff --git a/internal/scheduling/nova/candidate_gatherer_test.go b/internal/scheduling/nova/candidate_gatherer_test.go index 9f2b9a3b4..b40c0ffd2 100644 --- a/internal/scheduling/nova/candidate_gatherer_test.go +++ b/internal/scheduling/nova/candidate_gatherer_test.go @@ -9,6 +9,7 @@ import ( "testing" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -26,69 +27,41 @@ func TestCandidateGatherer_MutateWithAllCandidates(t *testing.T) { tests := []struct { name string hypervisors []client.Object - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectError bool errorContains string expectedHosts []string expectedWeight float64 }{ { - name: "missing hypervisor_type in flavor extra specs", + name: "unresolved hypervisor family", hypervisors: []client.Object{}, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{}, - }, - }, - }, - }, + request: compute.PlacementRequest{ + HypervisorFamily: "", }, expectError: true, - errorContains: "missing hypervisor_type in flavor extra specs", + errorContains: "cannot gather all placement candidates for hypervisor family", }, { - name: "unsupported hypervisor type vmware", + name: "unsupported hypervisor family vmware", hypervisors: []client.Object{}, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "VMware vCenter Server", - }, - }, - }, - }, - }, + request: compute.PlacementRequest{ + HypervisorFamily: api.HypervisorFamilyVMware, }, expectError: true, - errorContains: "cannot gather all placement candidates for hypervisor type", + errorContains: "cannot gather all placement candidates for hypervisor family", }, { - name: "unsupported hypervisor type unknown", + name: "unsupported hypervisor family unknown", hypervisors: []client.Object{}, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "unknown-type", - }, - }, - }, - }, - }, + request: compute.PlacementRequest{ + HypervisorFamily: "unknown-family", }, expectError: true, - errorContains: "cannot gather all placement candidates for hypervisor type", + errorContains: "cannot gather all placement candidates for hypervisor family", }, { - name: "successful gathering with qemu hypervisor type", + name: "successful gathering with kvm family (qemu)", hypervisors: []client.Object{ &hv1.Hypervisor{ ObjectMeta: metav1.ObjectMeta{ @@ -106,25 +79,15 @@ func TestCandidateGatherer_MutateWithAllCandidates(t *testing.T) { }, }, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "qemu", - }, - }, - }, - }, - }, + request: compute.PlacementRequest{ + HypervisorFamily: api.HypervisorFamilyKVM, }, expectError: false, expectedHosts: []string{"host1", "host2", "host3"}, expectedWeight: 0.0, }, { - name: "successful gathering with QEMU uppercase", + name: "successful gathering with kvm family single host", hypervisors: []client.Object{ &hv1.Hypervisor{ ObjectMeta: metav1.ObjectMeta{ @@ -132,25 +95,15 @@ func TestCandidateGatherer_MutateWithAllCandidates(t *testing.T) { }, }, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "QEMU", - }, - }, - }, - }, - }, + request: compute.PlacementRequest{ + HypervisorFamily: api.HypervisorFamilyKVM, }, expectError: false, expectedHosts: []string{"host1"}, expectedWeight: 0.0, }, { - name: "successful gathering with ch hypervisor type", + name: "successful gathering with kvm family (ch)", hypervisors: []client.Object{ &hv1.Hypervisor{ ObjectMeta: metav1.ObjectMeta{ @@ -163,25 +116,15 @@ func TestCandidateGatherer_MutateWithAllCandidates(t *testing.T) { }, }, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "ch", - }, - }, - }, - }, - }, + request: compute.PlacementRequest{ + HypervisorFamily: api.HypervisorFamilyKVM, }, expectError: false, expectedHosts: []string{"ch-host1", "ch-host2"}, expectedWeight: 0.0, }, { - name: "successful gathering with CH uppercase", + name: "successful gathering with kvm family single ch host", hypervisors: []client.Object{ &hv1.Hypervisor{ ObjectMeta: metav1.ObjectMeta{ @@ -189,18 +132,8 @@ func TestCandidateGatherer_MutateWithAllCandidates(t *testing.T) { }, }, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "CH", - }, - }, - }, - }, - }, + request: compute.PlacementRequest{ + HypervisorFamily: api.HypervisorFamilyKVM, }, expectError: false, expectedHosts: []string{"ch-host1"}, @@ -209,49 +142,13 @@ func TestCandidateGatherer_MutateWithAllCandidates(t *testing.T) { { name: "empty hypervisor list", hypervisors: []client.Object{}, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "qemu", - }, - }, - }, - }, - }, + request: compute.PlacementRequest{ + HypervisorFamily: api.HypervisorFamilyKVM, }, expectError: false, expectedHosts: []string{}, expectedWeight: 0.0, }, - { - name: "mixed case Qemu", - hypervisors: []client.Object{ - &hv1.Hypervisor{ - ObjectMeta: metav1.ObjectMeta{ - Name: "host1", - }, - }, - }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "Qemu", - }, - }, - }, - }, - }, - }, - expectError: false, - expectedHosts: []string{"host1"}, - expectedWeight: 0.0, - }, { name: "request with existing hosts and weights are overwritten", hypervisors: []client.Object{ @@ -261,19 +158,9 @@ func TestCandidateGatherer_MutateWithAllCandidates(t *testing.T) { }, }, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "qemu", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + HypervisorFamily: api.HypervisorFamilyKVM, + Hosts: []compute.Host{ {ComputeHost: "old-host", HypervisorHostname: "old-hv"}, }, Weights: map[string]float64{ @@ -321,7 +208,7 @@ func TestCandidateGatherer_MutateWithAllCandidates(t *testing.T) { } // Verify each expected host is present - hostMap := make(map[string]api.ExternalSchedulerHost) + hostMap := make(map[string]compute.Host) for _, host := range tt.request.Hosts { hostMap[host.ComputeHost] = host } diff --git a/internal/scheduling/nova/cr_metrics_test.go b/internal/scheduling/nova/cr_metrics_test.go index 6b0ac83c3..a05013b7c 100644 --- a/internal/scheduling/nova/cr_metrics_test.go +++ b/internal/scheduling/nova/cr_metrics_test.go @@ -4,9 +4,12 @@ package nova import ( + "bytes" "context" "encoding/json" "fmt" + "net/http" + "net/http/httptest" "testing" "github.com/prometheus/client_golang/prometheus" @@ -21,6 +24,7 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" "github.com/cobaltcore-dev/cortex/internal/scheduling/nova/crs" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" @@ -81,8 +85,10 @@ func TestLogNoHostFound(t *testing.T) { }, } if usedMiB > 0 { - cr.Status.UsedResources = map[string]resource.Quantity{ - "memory": *resource.NewQuantity(usedMiB*1024*1024, resource.BinarySI), + cr.Status.Billing = &v1alpha1.BillingStatus{ + UsedResources: map[string]resource.Quantity{ + "memory": *resource.NewQuantity(usedMiB*1024*1024, resource.BinarySI), + }, } } return cr @@ -107,10 +113,13 @@ func TestLogNoHostFound(t *testing.T) { }, } if allocatedMemMiB > 0 { - res.Spec.CommittedResourceReservation.Allocations = map[string]v1alpha1.CommittedResourceAllocation{ - "some-vm": { - Resources: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceMemory: *resource.NewQuantity(allocatedMemMiB*1024*1024, resource.BinarySI), + res.Status.CommittedResourceReservation = &v1alpha1.CommittedResourceReservationStatus{ + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + "some-vm": { + ObjectRef: corev1.ObjectReference{Name: "some-vm"}, + Resources: map[hv1.ResourceName]resource.Quantity{ + hv1.ResourceMemory: *resource.NewQuantity(allocatedMemMiB*1024*1024, resource.BinarySI), + }, }, }, } @@ -182,7 +191,7 @@ func TestLogNoHostFound(t *testing.T) { flavorKnowledge(), makeCRObject(v1alpha1.CommitmentStatusConfirmed, 8192, 4096), makeHV("host-1", 16384), - makeReadyReservationSlot("slot-free", "host-1", 8192, 4096), + makeReadyReservationSlot("slot-free", "host-1", 8192, 0), }, requestHosts: []api.ExternalSchedulerHost{{ComputeHost: "host-1"}}, expectedCase: "slot_blocked", @@ -234,13 +243,10 @@ func TestLogNoHostFound(t *testing.T) { reg := prometheus.NewRegistry() reg.MustRegister(counter) - controller := &FilterWeigherPipelineController{ - CRRecorder: crs.Recorder{ - Client: fakeClient, - NoHostFoundCounter: counter, - }, + recorder := crs.Recorder{ + Client: fakeClient, + NoHostFoundCounter: counter, } - controller.Client = fakeClient requestFlavorName := flavorName if tt.payg { @@ -258,11 +264,7 @@ func TestLogNoHostFound(t *testing.T) { Context: api.NovaRequestContext{ProjectID: projectID}, Hosts: tt.requestHosts, } - decision := &v1alpha1.Decision{ - Spec: v1alpha1.DecisionSpec{Intent: api.CreateIntent}, - } - - controller.CRRecorder.RecordNoHostFound(context.Background(), decision, request) + recorder.RecordNoHostFound(context.Background(), api.CreateIntent, request) if tt.expectedCase == "" { total := testutil.ToFloat64(counter.WithLabelValues("no_cr", flavorGroup, string(api.CreateIntent))) + @@ -359,7 +361,6 @@ func TestFeatureGate_CommittedResourceTracking(t *testing.T) { fakeClient := fake.NewClientBuilder(). WithScheme(scheme). WithObjects(flavorKnowledge). - WithStatusSubresource(&v1alpha1.Decision{}). Build() counter := crs.NewNoHostFoundCounter() @@ -367,16 +368,11 @@ func TestFeatureGate_CommittedResourceTracking(t *testing.T) { reg.MustRegister(counter) controller := &FilterWeigherPipelineController{ - BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]]{ + BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[sched.PlacementRequest]]{ Client: fakeClient, - Pipelines: make(map[string]lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]), + Pipelines: make(map[string]lib.FilterWeigherPipeline[sched.PlacementRequest]), PipelineConfigs: make(map[string]v1alpha1.Pipeline), }, - FeatureGates: FeatureGates{CommittedResourceTracking: enabled}, - CRRecorder: crs.Recorder{ - Client: fakeClient, - NoHostFoundCounter: counter, - }, } controller.PipelineConfigs["test-pipeline"] = pipelineConf initResult := controller.InitPipeline(context.Background(), pipelineConf) @@ -385,18 +381,20 @@ func TestFeatureGate_CommittedResourceTracking(t *testing.T) { } controller.Pipelines["test-pipeline"] = initResult.Pipeline - decision := &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{Name: "test-decision", Namespace: "default"}, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - PipelineRef: corev1.ObjectReference{Name: "test-pipeline"}, - NovaRaw: &runtime.RawExtension{Raw: novaRaw}, - }, - } + // Drive the committed-resource tracking path through the real HTTP API + // handler, where the feature gate now lives (as a post-hook). + scheduler := sched.NewPlacement(controller, passthroughGatherer{}) + crRecorder := crs.Recorder{Client: fakeClient, NoHostFoundCounter: counter} + api := newTestAPIWithScheduler( + HTTPAPIConfig{}, + scheduler, + crRecorder, + FeatureGates{CommittedResourceTracking: enabled}, + ) - if err := controller.ProcessNewDecisionFromAPI(context.Background(), decision); err != nil { - t.Fatalf("ProcessNewDecisionFromAPI: %v", err) - } + req := httptest.NewRequest(http.MethodPost, "/scheduler/nova/external", bytes.NewReader(novaRaw)) + w := httptest.NewRecorder() + api.NovaExternalScheduler(w, req) count := testutil.CollectAndCount(counter) if enabled && count == 0 { diff --git a/internal/scheduling/nova/crs/crs_test.go b/internal/scheduling/nova/crs/crs_test.go index d1a1611a4..60771288a 100644 --- a/internal/scheduling/nova/crs/crs_test.go +++ b/internal/scheduling/nova/crs/crs_test.go @@ -8,6 +8,7 @@ import ( "github.com/cobaltcore-dev/cortex/api/v1alpha1" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) @@ -27,10 +28,13 @@ func makeSlot(projectID, flavorGroup string, totalMemMiB, allocatedMemMiB int64) }, } if allocatedMemMiB > 0 { - res.Spec.CommittedResourceReservation.Allocations = map[string]v1alpha1.CommittedResourceAllocation{ - "some-vm": { - Resources: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceMemory: *resource.NewQuantity(allocatedMemMiB*1024*1024, resource.BinarySI), + res.Status.CommittedResourceReservation = &v1alpha1.CommittedResourceReservationStatus{ + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + "some-vm": { + ObjectRef: corev1.ObjectReference{Name: "some-vm"}, + Resources: map[hv1.ResourceName]resource.Quantity{ + hv1.ResourceMemory: *resource.NewQuantity(allocatedMemMiB*1024*1024, resource.BinarySI), + }, }, }, } @@ -47,8 +51,10 @@ func makeCR(state v1alpha1.CommitmentStatus, amountMiB, usedMiB int64) v1alpha1. }, } if usedMiB > 0 { - cr.Status.UsedResources = map[string]resource.Quantity{ - "memory": *resource.NewQuantity(usedMiB*1024*1024, resource.BinarySI), + cr.Status.Billing = &v1alpha1.BillingStatus{ + UsedResources: map[string]resource.Quantity{ + "memory": *resource.NewQuantity(usedMiB*1024*1024, resource.BinarySI), + }, } } return cr @@ -148,17 +154,17 @@ func TestClassifyNoHostFound(t *testing.T) { activeCRs: []v1alpha1.CommittedResource{ makeCR(v1alpha1.CommitmentStatusConfirmed, 8192, 4096), }, - eval: evalWithSlot(8192, 4096), // 4096 MiB remaining; 16384-8192+4096=12288 >= 4096 + eval: evalWithSlot(8192, 0), // free slot; HV has 16384 MiB free → 8192 MiB available ≥ 4096 inputHosts: []string{"host-1"}, vmMemBytes: 4096 * MiB, expectedCase: "slot_blocked", }, { - name: "slot_blocked: overfill — slot smaller than VM is still usable", + name: "slot_blocked: overfill — empty slot smaller than VM (remaining 2048 < vmMem 4096, but overfill formula passes)", activeCRs: []v1alpha1.CommittedResource{ makeCR(v1alpha1.CommitmentStatusConfirmed, 8192, 4096), }, - eval: evalWithSlot(8192, 6144), // 2048 MiB remaining; 16384-8192+2048=10240 >= 4096 + eval: evalWithSlot(2048, 0), // free 2048 MiB slot; 16384-2048+2048=16384 >= 4096 inputHosts: []string{"host-1"}, vmMemBytes: 4096 * MiB, expectedCase: "slot_blocked", @@ -198,7 +204,7 @@ func TestReservationRemainingMemory(t *testing.T) { wantBytes int64 }{ {"empty slot", 8192, 0, 8192 * 1024 * 1024}, - {"partially used", 8192, 4096, 4096 * 1024 * 1024}, + {"partially used: slot=8192 vm=4096 → remaining=4096", 8192, 4096, 4096 * 1024 * 1024}, {"fully used", 8192, 8192, 0}, {"over-allocated (clamped to zero)", 4096, 8192, 0}, } @@ -218,30 +224,33 @@ func TestPickSlot(t *testing.T) { // vmMemBytes for a 4096 MiB flavor. const vmMemBytes = int64(4096) * 1024 * 1024 + // makePickSlot creates a test slot. usedMemMiB sets confirmed allocation resources precisely. + // usedCPU is kept for fixture symmetry but has no effect on remaining memory. makePickSlot := func(name string, totalMemMiB, totalCPU, usedMemMiB, usedCPU int64) v1alpha1.Reservation { - var allocs map[string]v1alpha1.CommittedResourceAllocation - if usedMemMiB > 0 || usedCPU > 0 { - allocs = map[string]v1alpha1.CommittedResourceAllocation{ - "vm-existing": { - Resources: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceMemory: *resource.NewQuantity(usedMemMiB*1024*1024, resource.BinarySI), - hv1.ResourceCPU: *resource.NewQuantity(usedCPU, resource.DecimalSI), - }, - }, - } - } - return v1alpha1.Reservation{ + _ = usedCPU + res := v1alpha1.Reservation{ ObjectMeta: metav1.ObjectMeta{Name: name}, Spec: v1alpha1.ReservationSpec{ Resources: map[hv1.ResourceName]resource.Quantity{ hv1.ResourceMemory: *resource.NewQuantity(totalMemMiB*1024*1024, resource.BinarySI), hv1.ResourceCPU: *resource.NewQuantity(totalCPU, resource.DecimalSI), }, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: allocs, - }, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{}, }, } + if usedMemMiB > 0 { + res.Status.CommittedResourceReservation = &v1alpha1.CommittedResourceReservationStatus{ + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + "vm-existing": { + ObjectRef: corev1.ObjectReference{Name: "vm-existing"}, + Resources: map[hv1.ResourceName]resource.Quantity{ + hv1.ResourceMemory: *resource.NewQuantity(usedMemMiB*1024*1024, resource.BinarySI), + }, + }, + }, + } + } + return res } tests := []struct { @@ -260,7 +269,7 @@ func TestPickSlot(t *testing.T) { want: "a", }, { - name: "slot with zero remaining memory excluded", + name: "slot with confirmed VM fully consumed (8192/8192 used → 0 remaining → excluded)", candidates: []v1alpha1.Reservation{makePickSlot("a", 8192, 8, 8192, 0)}, want: "", }, @@ -293,28 +302,28 @@ func TestPickSlot(t *testing.T) { want: "", }, { - name: "partially used slot still usable", - candidates: []v1alpha1.Reservation{makePickSlot("partial", 8192, 8, 2048, 2)}, // 6144 MiB remaining + name: "slot partially used (2048/8192 used → 6144 remaining → usable for 4096 MiB VM)", + candidates: []v1alpha1.Reservation{makePickSlot("partial", 8192, 8, 2048, 2)}, want: "partial", }, { - name: "overfill: slot smaller than VM is usable", - candidates: []v1alpha1.Reservation{makePickSlot("a", 4096, 8, 2048, 0)}, // 2048 MiB remaining < vmMemBytes + name: "overfill: small empty slot smaller than VM is still usable", + candidates: []v1alpha1.Reservation{makePickSlot("a", 2048, 8, 0, 0)}, // 2048 MiB remaining < vmMemBytes (4096) want: "a", }, { - name: "overfill: full coverage preferred over partial", + name: "overfill: full coverage preferred over partial fill (partial has 2048 remaining, full has 6144)", candidates: []v1alpha1.Reservation{ - makePickSlot("partial", 4096, 8, 2048, 0), // 2048 MiB remaining, coverage=2048 - makePickSlot("full", 6144, 8, 0, 0), // 6144 MiB remaining, coverage=4096 (full) + makePickSlot("partial", 4096, 8, 2048, 0), // 2048 remaining → coverage=2048 (overfill) + makePickSlot("full", 6144, 8, 0, 0), // 6144 remaining → coverage=4096 (full) }, want: "full", }, { - name: "overfill: highest partial coverage preferred", + name: "overfill: among two empty slots, higher remaining memory preferred (more coverage for overfill)", candidates: []v1alpha1.Reservation{ - makePickSlot("low", 4096, 8, 2048, 0), // 2048 MiB remaining, coverage=2048 - makePickSlot("high", 4096, 8, 1024, 0), // 3072 MiB remaining, coverage=3072 + makePickSlot("low", 2048, 8, 0, 0), // 2048 MiB remaining < vmMemBytes, coverage=2048 + makePickSlot("high", 3072, 8, 0, 0), // 3072 MiB remaining < vmMemBytes, coverage=3072 }, want: "high", }, diff --git a/internal/scheduling/nova/crs/evaluator.go b/internal/scheduling/nova/crs/evaluator.go index 67faef722..869dd29ec 100644 --- a/internal/scheduling/nova/crs/evaluator.go +++ b/internal/scheduling/nova/crs/evaluator.go @@ -141,19 +141,28 @@ func (e *SlotEvaluator) CanAccommodateSlot(hostName string, requiredBytes int64) } // ReservationRemainingMemory returns how many bytes of memory remain -// unallocated in a reservation slot. Returns 0 if the slot is full or nil. +// unallocated in a reservation slot. Returns 0 if the slot is nil or fully consumed. func ReservationRemainingMemory(res v1alpha1.Reservation) int64 { cr := res.Spec.CommittedResourceReservation if cr == nil { return 0 } totalMemQ := res.Spec.Resources[hv1.ResourceMemory] - var usedMem int64 - for _, alloc := range cr.Allocations { - allocMem := alloc.Resources[hv1.ResourceMemory] - usedMem += allocMem.Value() + total := totalMemQ.Value() + if res.Status.CommittedResourceReservation == nil { + return total } - return max(totalMemQ.Value()-usedMem, 0) + var confirmed int64 + for _, alloc := range res.Status.CommittedResourceReservation.Allocation { + if used, ok := alloc.Resources[hv1.ResourceMemory]; ok { + confirmed += used.Value() + } + } + remaining := total - confirmed + if remaining < 0 { + return 0 + } + return remaining } // PickSlot selects the best reservation slot for a new VM. diff --git a/internal/scheduling/nova/crs/metrics.go b/internal/scheduling/nova/crs/metrics.go index 1732451aa..f8bb80318 100644 --- a/internal/scheduling/nova/crs/metrics.go +++ b/internal/scheduling/nova/crs/metrics.go @@ -53,8 +53,10 @@ func ClassifyNoHostFound( totalUsed := resource.Quantity{} for _, cr := range activeCRs { totalCapacity.Add(cr.Spec.Amount) - if used, ok := cr.Status.UsedResources["memory"]; ok { - totalUsed.Add(used) + if cr.Status.Billing != nil { + if used, ok := cr.Status.Billing.UsedResources["memory"]; ok { + totalUsed.Add(used) + } } } if totalUsed.Cmp(totalCapacity) >= 0 { @@ -87,8 +89,10 @@ func ClassifyPlacement( totalUsed := resource.Quantity{} for _, cr := range activeCRs { totalCapacity.Add(cr.Spec.Amount) - if used, ok := cr.Status.UsedResources["memory"]; ok { - totalUsed.Add(used) + if cr.Status.Billing != nil { + if used, ok := cr.Status.Billing.UsedResources["memory"]; ok { + totalUsed.Add(used) + } } } if totalUsed.Cmp(totalCapacity) >= 0 { diff --git a/internal/scheduling/nova/crs/recorder.go b/internal/scheduling/nova/crs/recorder.go index 99cd0d8f4..b2e928bab 100644 --- a/internal/scheduling/nova/crs/recorder.go +++ b/internal/scheduling/nova/crs/recorder.go @@ -6,17 +6,12 @@ package crs import ( "context" "errors" - "fmt" api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" - "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" - hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" + knowledge "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/prometheus/client_golang/prometheus" - "k8s.io/apimachinery/pkg/api/resource" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/client-go/util/retry" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" ) @@ -40,13 +35,12 @@ var errFlavorNotInGroup = errors.New("flavor not in any group") // resolveFlavorGroup looks up which flavor group the given flavor belongs to. // Returns errFlavorNotInGroup (PAYG) if the flavor is not in any group. // Returns a different error for transient failures (Knowledge CRD unavailable, etc). -func (r *Recorder) resolveFlavorGroup(ctx context.Context, flavorName string) (string, *compute.FlavorInGroup, error) { - fgClient := reservations.FlavorGroupKnowledgeClient{Client: r.Client} - flavorGroups, err := fgClient.GetAllFlavorGroups(ctx, nil) +func (r *Recorder) resolveFlavorGroup(ctx context.Context, flavorName string) (string, *knowledge.FlavorInGroup, error) { + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, r.Client, nil) if err != nil { return "", nil, err } - groupName, flavor, err := reservations.FindFlavorInGroups(flavorName, flavorGroups) + groupName, flavor, err := knowledge.FindFlavorInGroups(flavorName, flavorGroups) if err != nil { return "", nil, errFlavorNotInGroup } @@ -55,14 +49,14 @@ func (r *Recorder) resolveFlavorGroup(ctx context.Context, flavorName string) (s // RecordPlacement writes the placed VM UUID into the matching Reservation slot // and emits placement metrics. Called after a successful Nova placement. -func (r *Recorder) RecordPlacement(ctx context.Context, decision *v1alpha1.Decision, request api.ExternalSchedulerRequest) { +func (r *Recorder) RecordPlacement(ctx context.Context, result sched.PlacementResult, intent v1alpha1.SchedulingIntent, request api.ExternalSchedulerRequest) { log := ctrl.LoggerFrom(ctx) instanceUUID := request.Spec.Data.InstanceUUID projectID := request.Context.ProjectID flavorName := request.Spec.Data.Flavor.Data.Name - selectedHost := *decision.Status.Result.TargetHost - intent := string(decision.Spec.Intent) + selectedHost := *result.TargetHost + intentStr := string(intent) flavorGroupName, flavorInGroup, err := r.resolveFlavorGroup(ctx, flavorName) if err != nil { @@ -72,7 +66,7 @@ func (r *Recorder) RecordPlacement(ctx context.Context, decision *v1alpha1.Decis log.Error(err, "CR allocation: failed to resolve flavor group", "flavor", flavorName, "instanceUUID", instanceUUID) if r.PlacementCounter != nil { - r.PlacementCounter.WithLabelValues("unknown", intent, "error").Inc() + r.PlacementCounter.WithLabelValues("unknown", intentStr, "error").Inc() } } return @@ -97,7 +91,7 @@ func (r *Recorder) RecordPlacement(ctx context.Context, decision *v1alpha1.Decis activeCRs = append(activeCRs, cr) } - candidateHosts := decision.Status.Result.OrderedHosts + candidateHosts := result.OrderedHosts crOutcome := ClassifyPlacement(evaluator, activeCRs, candidateHosts, projectID, flavorGroupName) log.V(1).Info("CR allocation: placement classified", @@ -108,7 +102,7 @@ func (r *Recorder) RecordPlacement(ctx context.Context, decision *v1alpha1.Decis "flavorGroup", flavorGroupName, ) if r.PlacementCounter != nil { - r.PlacementCounter.WithLabelValues(flavorGroupName, intent, crOutcome).Inc() + r.PlacementCounter.WithLabelValues(flavorGroupName, intentStr, crOutcome).Inc() } if crOutcome != "slot_used" { @@ -118,15 +112,16 @@ func (r *Recorder) RecordPlacement(ctx context.Context, decision *v1alpha1.Decis slotsOnTarget := evaluator.SlotsForHost(selectedHost, projectID, flavorGroupName) for _, slot := range slotsOnTarget { - if _, exists := slot.Spec.CommittedResourceReservation.Allocations[instanceUUID]; exists { - log.V(1).Info("CR allocation: VM UUID already in reservation, skipping", - "instanceUUID", instanceUUID, "reservation", slot.Name) - return + if slot.Status.CommittedResourceReservation != nil { + if _, exists := slot.Status.CommittedResourceReservation.Allocation[instanceUUID]; exists { + log.V(1).Info("CR allocation: VM UUID already confirmed in reservation, skipping", + "instanceUUID", instanceUUID, "reservation", slot.Name) + return + } } } vmMemoryBytes := int64(flavorInGroup.MemoryMB) * 1024 * 1024 //nolint:gosec // flavor memory bounded by specs - vmCPUs := int64(flavorInGroup.VCPUs) //nolint:gosec // VCPUs bounded by specs slotName := PickSlot(slotsOnTarget, vmMemoryBytes) if slotName == "" { @@ -135,48 +130,40 @@ func (r *Recorder) RecordPlacement(ctx context.Context, decision *v1alpha1.Decis return } - log.V(1).Info("CR allocation: writing VM UUID into reservation", - "instanceUUID", instanceUUID, "reservation", slotName, - "projectID", projectID, "flavorGroup", flavorGroupName, "host", selectedHost) - - vmResources := map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceMemory: *resource.NewQuantity(vmMemoryBytes, resource.BinarySI), - hv1.ResourceCPU: *resource.NewQuantity(vmCPUs, resource.DecimalSI), - } - if retryErr := retry.RetryOnConflict(retry.DefaultRetry, func() error { - latest := &v1alpha1.Reservation{} - if err := r.Get(ctx, client.ObjectKey{Name: slotName}, latest); err != nil { - return err - } - if latest.Spec.CommittedResourceReservation == nil { - return fmt.Errorf("reservation %s lost CommittedResourceReservation spec", slotName) - } - base := latest.DeepCopy() - if latest.Spec.CommittedResourceReservation.Allocations == nil { - latest.Spec.CommittedResourceReservation.Allocations = make(map[string]v1alpha1.CommittedResourceAllocation) + var slot *v1alpha1.Reservation + for i := range slotsOnTarget { + if slotsOnTarget[i].Name == slotName { + slot = &slotsOnTarget[i] + break } - latest.Spec.CommittedResourceReservation.Allocations[instanceUUID] = v1alpha1.CommittedResourceAllocation{ - CreationTimestamp: metav1.Now(), - Resources: vmResources, - } - return r.Patch(ctx, latest, client.MergeFrom(base)) - }); retryErr != nil { - log.Error(retryErr, "CR allocation: failed to patch reservation", - "reservation", slotName, "instanceUUID", instanceUUID) + } + if slot == nil { + log.V(1).Info("CR allocation: picked slot not found in slotsOnTarget (race?)", + "instanceUUID", instanceUUID, "reservation", slotName) return } - log.V(1).Info("CR allocation: done", "instanceUUID", instanceUUID, "reservation", slotName) + old := slot.DeepCopy() + if slot.Annotations == nil { + slot.Annotations = make(map[string]string) + } + slot.Annotations[v1alpha1.AnnotationCandidateAllocation] = instanceUUID + if err := r.Patch(ctx, slot, client.MergeFrom(old)); err != nil { + log.Error(err, "CR allocation: failed to annotate reservation with candidate VM UUID; controller will confirm via HV CRD", + "instanceUUID", instanceUUID, "reservation", slotName) + } else { + log.V(1).Info("CR allocation: reservation annotated with candidate VM UUID", + "instanceUUID", instanceUUID, "reservation", slotName) + } } // RecordNoHostFound classifies a no-host-found result and emits a log line and metric. -func (r *Recorder) RecordNoHostFound(ctx context.Context, decision *v1alpha1.Decision, request api.ExternalSchedulerRequest) { +func (r *Recorder) RecordNoHostFound(ctx context.Context, intent v1alpha1.SchedulingIntent, request api.ExternalSchedulerRequest) { log := ctrl.LoggerFrom(ctx) projectID := request.Context.ProjectID flavorName := request.Spec.Data.Flavor.Data.Name instanceUUID := request.Spec.Data.InstanceUUID - intent := decision.Spec.Intent flavorGroupName, flavorInGroup, err := r.resolveFlavorGroup(ctx, flavorName) if err != nil { diff --git a/internal/scheduling/nova/crs/recorder_test.go b/internal/scheduling/nova/crs/recorder_test.go index dd136784f..90ee32104 100644 --- a/internal/scheduling/nova/crs/recorder_test.go +++ b/internal/scheduling/nova/crs/recorder_test.go @@ -9,6 +9,7 @@ import ( "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/testutil" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -18,6 +19,7 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" ) @@ -66,8 +68,8 @@ func TestRecordCRAllocation(t *testing.T) { } } - makeReservation := func(name string, memMiB, cpus int64, proj, group, host string, allocs map[string]v1alpha1.CommittedResourceAllocation) *v1alpha1.Reservation { - return &v1alpha1.Reservation{ + makeReservation := func(name string, memMiB, cpus int64, proj, group, host string, statusAllocs map[string]v1alpha1.CommittedResourceAllocation) *v1alpha1.Reservation { + res := &v1alpha1.Reservation{ ObjectMeta: metav1.ObjectMeta{ Name: name, Labels: map[string]string{ @@ -84,10 +86,15 @@ func TestRecordCRAllocation(t *testing.T) { CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ ProjectID: proj, ResourceGroup: group, - Allocations: allocs, }, }, } + if len(statusAllocs) > 0 { + res.Status.CommittedResourceReservation = &v1alpha1.CommittedResourceReservationStatus{ + Allocation: statusAllocs, + } + } + return res } makeRequest := func(uuid, proj, flavor string) api.ExternalSchedulerRequest { @@ -104,15 +111,11 @@ func TestRecordCRAllocation(t *testing.T) { } } - makeDecision := func(host string, candidates ...string) *v1alpha1.Decision { + makeResult := func(host string, candidates ...string) sched.PlacementResult { h := host - return &v1alpha1.Decision{ - Status: v1alpha1.DecisionStatus{ - Result: &v1alpha1.DecisionResult{ - TargetHost: &h, - OrderedHosts: candidates, - }, - }, + return sched.PlacementResult{ + TargetHost: &h, + OrderedHosts: candidates, } } @@ -141,47 +144,53 @@ func TestRecordCRAllocation(t *testing.T) { vmAlloc := func() map[string]v1alpha1.CommittedResourceAllocation { return map[string]v1alpha1.CommittedResourceAllocation{ instanceUUID: { - CreationTimestamp: metav1.Now(), + ObjectRef: corev1.ObjectReference{Name: instanceUUID}, Resources: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceMemory: *resource.NewQuantity(int64(4096)*1024*1024, resource.BinarySI), - hv1.ResourceCPU: *resource.NewQuantity(2, resource.DecimalSI), + hv1.ResourceMemory: resource.MustParse("8192Mi"), }, }, } } tests := []struct { - name string - objects []client.Object - request api.ExternalSchedulerRequest - decision *v1alpha1.Decision - checkSlot string - expectAllocation bool - expectedCRSlot string // if non-empty, asserts PlacementCounter cr_slot label + name string + objects []client.Object + request api.ExternalSchedulerRequest + result sched.PlacementResult + intent v1alpha1.SchedulingIntent + expectedCRSlot string // if non-empty, asserts PlacementCounter cr_slot label + expectedFG string // flavor_group label; defaults to "unknown" if empty + // annotateSlot, when non-empty, asserts that the named reservation slot + // was annotated with the candidate VM UUID (the #1438 recorder delegates + // the confirmed write to the reservation controller via this annotation). + annotateSlot string }{ { - name: "writes allocation into matching reservation", + name: "classifies slot_used and annotates reservation (recorder delegates write to reservation controller)", objects: []client.Object{ flavorKnowledge(), makeCRObject(), setReady(makeReservation("slot-1", 8192, 8, projectID, flavorGroup, selectedHost, nil)), }, - request: makeRequest(instanceUUID, projectID, flavorName), - decision: makeDecision(selectedHost, selectedHost), - checkSlot: "slot-1", - expectAllocation: true, + request: makeRequest(instanceUUID, projectID, flavorName), + result: makeResult(selectedHost, selectedHost), + expectedCRSlot: "slot_used", + expectedFG: flavorGroup, + annotateSlot: "slot-1", }, { - name: "idempotent: UUID already in allocations", + // With status.allocs set, ClassifyPlacement sees remaining=0 → "slot_missed". + // The recorder returns early at that point, so no annotation occurs (idempotency maintained via no-op). + name: "idempotent: UUID already confirmed in status.allocation — skips early", objects: []client.Object{ flavorKnowledge(), makeCRObject(), setReady(makeReservation("slot-1", 8192, 8, projectID, flavorGroup, selectedHost, vmAlloc())), }, - request: makeRequest(instanceUUID, projectID, flavorName), - decision: makeDecision(selectedHost, selectedHost), - checkSlot: "slot-1", - expectAllocation: true, + request: makeRequest(instanceUUID, projectID, flavorName), + result: makeResult(selectedHost, selectedHost), + // No expectedCRSlot: with status.allocs set, slot appears exhausted (remaining=0), + // crOutcome="slot_missed" — counter not checked for this idempotent path. }, { name: "PAYG: flavor not in any group", @@ -189,10 +198,8 @@ func TestRecordCRAllocation(t *testing.T) { flavorKnowledge(), setReady(makeReservation("slot-1", 8192, 8, projectID, flavorGroup, selectedHost, nil)), }, - request: makeRequest(instanceUUID, projectID, "unknown-flavor"), - decision: makeDecision(selectedHost, selectedHost), - checkSlot: "slot-1", - expectAllocation: false, + request: makeRequest(instanceUUID, projectID, "unknown-flavor"), + result: makeResult(selectedHost, selectedHost), }, { name: "no matching reservation: host mismatch", @@ -201,33 +208,29 @@ func TestRecordCRAllocation(t *testing.T) { makeCRObject(), setReady(makeReservation("slot-1", 8192, 8, projectID, flavorGroup, "other-host", nil)), }, - request: makeRequest(instanceUUID, projectID, flavorName), - decision: makeDecision(selectedHost, selectedHost), - checkSlot: "slot-1", - expectAllocation: false, + request: makeRequest(instanceUUID, projectID, flavorName), + result: makeResult(selectedHost, selectedHost), }, { - name: "no slot fits: all capacity used", + name: "no slot fits: all capacity confirmed (status.allocs non-empty → 0 remaining memory)", objects: []client.Object{ flavorKnowledge(), makeCRObject(), setReady(makeReservation("slot-full", 4096, 2, projectID, flavorGroup, selectedHost, map[string]v1alpha1.CommittedResourceAllocation{ "other-vm": { + ObjectRef: corev1.ObjectReference{Name: "other-vm"}, Resources: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceMemory: *resource.NewQuantity(int64(4096)*1024*1024, resource.BinarySI), - hv1.ResourceCPU: *resource.NewQuantity(2, resource.DecimalSI), + hv1.ResourceMemory: resource.MustParse("4096Mi"), }, }, })), }, - request: makeRequest(instanceUUID, projectID, flavorName), - decision: makeDecision(selectedHost, selectedHost), - checkSlot: "slot-full", - expectAllocation: false, + request: makeRequest(instanceUUID, projectID, flavorName), + result: makeResult(selectedHost, selectedHost), }, { - name: "inactive CR (pending state) is filtered by IsActive(): no allocation", + name: "inactive CR (pending state) is filtered by IsActive(): no slot_used", objects: []client.Object{ flavorKnowledge(), func() *v1alpha1.CommittedResource { @@ -237,21 +240,17 @@ func TestRecordCRAllocation(t *testing.T) { }(), setReady(makeReservation("slot-1", 8192, 8, projectID, flavorGroup, selectedHost, nil)), }, - request: makeRequest(instanceUUID, projectID, flavorName), - decision: makeDecision(selectedHost, selectedHost), - checkSlot: "slot-1", - expectAllocation: false, + request: makeRequest(instanceUUID, projectID, flavorName), + result: makeResult(selectedHost, selectedHost), }, { name: "no knowledge CRD: logs error, no allocation", objects: []client.Object{ makeReservation("slot-1", 8192, 8, projectID, flavorGroup, selectedHost, nil), }, - request: makeRequest(instanceUUID, projectID, flavorName), - decision: makeDecision(selectedHost, selectedHost), - checkSlot: "slot-1", - expectAllocation: false, - expectedCRSlot: "error", + request: makeRequest(instanceUUID, projectID, flavorName), + result: makeResult(selectedHost, selectedHost), + expectedCRSlot: "error", }, } @@ -271,25 +270,29 @@ func TestRecordCRAllocation(t *testing.T) { PlacementCounter: counter, } - recorder.RecordPlacement(context.Background(), tt.decision, tt.request) + recorder.RecordPlacement(context.Background(), tt.result, tt.intent, tt.request) - var res v1alpha1.Reservation - if err := fakeClient.Get(context.Background(), client.ObjectKey{Name: tt.checkSlot}, &res); err != nil { - t.Fatalf("Get reservation %q: %v", tt.checkSlot, err) - } - _, hasAlloc := res.Spec.CommittedResourceReservation.Allocations[instanceUUID] - if tt.expectAllocation && !hasAlloc { - t.Errorf("expected allocation for UUID %q but none found", instanceUUID) - } - if !tt.expectAllocation && hasAlloc { - t.Errorf("expected no allocation for UUID %q but one was written", instanceUUID) - } if tt.expectedCRSlot != "" { - intent := string(tt.decision.Spec.Intent) - got := testutil.ToFloat64(counter.WithLabelValues("unknown", intent, tt.expectedCRSlot)) + intent := string(tt.intent) + fg := tt.expectedFG + if fg == "" { + fg = "unknown" + } + got := testutil.ToFloat64(counter.WithLabelValues(fg, intent, tt.expectedCRSlot)) if got != 1 { - t.Errorf("PlacementCounter[flavor_group=unknown, intent=%q, cr_slot=%q] = %.0f, want 1", - intent, tt.expectedCRSlot, got) + t.Errorf("PlacementCounter[flavor_group=%q, intent=%q, cr_slot=%q] = %.0f, want 1", + fg, intent, tt.expectedCRSlot, got) + } + } + + if tt.annotateSlot != "" { + var res v1alpha1.Reservation + if err := fakeClient.Get(context.Background(), client.ObjectKey{Name: tt.annotateSlot}, &res); err != nil { + t.Fatalf("Get reservation %q: %v", tt.annotateSlot, err) + } + if got := res.Annotations[v1alpha1.AnnotationCandidateAllocation]; got != instanceUUID { + t.Errorf("annotation %q on %q = %q, want %q", + v1alpha1.AnnotationCandidateAllocation, tt.annotateSlot, got, instanceUUID) } } }) diff --git a/internal/scheduling/nova/external_scheduler_api.go b/internal/scheduling/nova/external_scheduler_api.go index 9e89f5f2a..409a0f7f1 100644 --- a/internal/scheduling/nova/external_scheduler_api.go +++ b/internal/scheduling/nova/external_scheduler_api.go @@ -12,16 +12,14 @@ import ( "io" "math/rand" "net/http" + "slices" api "github.com/cobaltcore-dev/cortex/api/external/nova" - "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" scheduling "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" + "github.com/cobaltcore-dev/cortex/internal/scheduling/nova/crs" "github.com/go-logr/logr" - corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/api/meta" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/runtime" "sigs.k8s.io/controller-runtime/pkg/metrics" ) @@ -46,27 +44,45 @@ func (c HTTPAPIConfig) forcedDestinationEnabled() bool { return c.ForcedDestinationEnabled == nil || *c.ForcedDestinationEnabled } -type HTTPAPIDelegate interface { - // Process the decision from the API. Should create and return the updated decision. - ProcessNewDecisionFromAPI(ctx context.Context, decision *v1alpha1.Decision) error -} - type HTTPAPI interface { // Bind the server handlers. Init(*http.ServeMux) } +// runner is the subset of compute.Placement the HTTP API depends on. It is an +// interface so the handler can be unit-tested with a mock. *compute.Placement +// satisfies it. +type runner interface { + Run(ctx context.Context, req compute.PlacementRequest, pre compute.PreHook, post compute.PostHook) (compute.PlacementResult, error) +} + type httpAPI struct { - monitor scheduling.APIMonitor - delegate HTTPAPIDelegate - config HTTPAPIConfig + monitor scheduling.APIMonitor + scheduler runner + vmHandler *VirtualMachineHandler + crRecorder crs.Recorder + featureGates FeatureGates + config HTTPAPIConfig } -func NewAPI(config HTTPAPIConfig, delegate HTTPAPIDelegate) HTTPAPI { +// NewAPI builds the Nova external-scheduler HTTP API. It calls the shared +// compute.Placement directly; VM-resource creation and committed-resource +// tracking run as post-hooks after the pipeline, under the scheduler's lock. +func NewAPI( + config HTTPAPIConfig, + scheduler *compute.Placement, + vmHandler *VirtualMachineHandler, + crRecorder crs.Recorder, + featureGates FeatureGates, +) HTTPAPI { + return &httpAPI{ - monitor: scheduling.NewSchedulerMonitor(), - delegate: delegate, - config: config, + monitor: scheduling.NewSchedulerMonitor(), + scheduler: scheduler, + vmHandler: vmHandler, + crRecorder: crRecorder, + featureGates: featureGates, + config: config, } } @@ -202,7 +218,6 @@ func (httpAPI *httpAPI) NovaExternalScheduler(w http.ResponseWriter, r *http.Req c.Respond(logr.Logger{}, http.StatusInternalServerError, err, "failed to read request body") return } - raw := runtime.RawExtension{Raw: body} var requestData api.ExternalSchedulerRequest // Copy the raw body to a io.Reader for json deserialization. cp := body @@ -257,43 +272,55 @@ func (httpAPI *httpAPI) NovaExternalScheduler(w http.ResponseWriter, r *http.Req intent = api.CreateIntent } - decision := &v1alpha1.Decision{ - TypeMeta: metav1.TypeMeta{ - Kind: "Decision", - APIVersion: "cortex.cloud/v1alpha1", - }, - ObjectMeta: metav1.ObjectMeta{ - GenerateName: "nova-", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - PipelineRef: corev1.ObjectReference{ - Name: requestData.Pipeline, - }, - ResourceID: requestData.Spec.Data.InstanceUUID, - NovaRaw: &raw, - Intent: intent, - }, - } ctx := r.Context() - if err := httpAPI.delegate.ProcessNewDecisionFromAPI(ctx, decision); err != nil { - c.Respond(logger, http.StatusInternalServerError, err, fmt.Sprintf("failed to process scheduling decision: %v", err)) - return - } - // Check if the decision contains status conditions indicating an error. - if meta.IsStatusConditionFalse(decision.Status.Conditions, v1alpha1.DecisionConditionReady) { - c.Respond(logger, http.StatusInternalServerError, errors.New("decision contains error condition"), "decision failed") - return + + // Translate the Nova wire request into the typed pipeline request. + schedReq := ToSchedulerRequest(requestData, intent, requestData.Pipeline) + + // Side effects run as a single post-hook after the pipeline, under the + // scheduler lock. It operates on the Nova wire request (which carries the + // fields the VM resource and CR tracking need) plus the typed result. + postHook := func(ctx context.Context, _ compute.PlacementRequest, result compute.PlacementResult) error { + // VM-resource creation. + switch { + case requestData.Options.ReadOnly: + logger.V(1).Info("skipping virtual machine creation, request is read-only", "instanceUUID", requestData.Spec.Data.InstanceUUID) + case !httpAPI.featureGates.VirtualMachineCreation: + logger.V(1).Info("skipping virtual machine creation, feature gate disabled", "instanceUUID", requestData.Spec.Data.InstanceUUID) + case !slices.Contains(api.NovaIntents, intent): + logger.V(1).Info("skipping virtual machine creation, intent not triggered by Nova", "instanceUUID", requestData.Spec.Data.InstanceUUID, "intent", intent) + case schedReq.HypervisorFamily != api.HypervisorFamilyKVM: + logger.V(1).Info("skipping virtual machine creation, unsupported hypervisor type", "instanceUUID", requestData.Spec.Data.InstanceUUID) + default: + if upsertErr := httpAPI.vmHandler.Upsert(ctx, result, requestData, intent); upsertErr != nil { + if httpAPI.featureGates.VirtualMachineCreationMandatory { + return fmt.Errorf("failed to upsert virtual machine resource: %w", upsertErr) + } + // Non-fatal: the placement decision itself succeeded. + logger.Error(upsertErr, "failed to upsert virtual machine resource", "instanceUUID", requestData.Spec.Data.InstanceUUID) + } + } + // Committed-resource tracking. + if httpAPI.featureGates.CommittedResourceTracking && !requestData.Options.SkipCommittedResourceTracking { + if result.TargetHost != nil { + httpAPI.crRecorder.RecordPlacement(ctx, result, intent, requestData) + } else { + httpAPI.crRecorder.RecordNoHostFound(ctx, intent, requestData) + } + } + return nil } - if decision.Status.Result == nil { - c.Respond(logger, http.StatusInternalServerError, errors.New("decision didn't produce a result"), "decision failed") + + result, err := httpAPI.scheduler.Run(ctx, schedReq, nil, postHook) + if err != nil { + c.Respond(logger, http.StatusInternalServerError, err, fmt.Sprintf("failed to process scheduling decision: %v", err)) return } - hosts := decision.Status.Result.OrderedHosts + hosts := result.OrderedHosts if httpAPI.config.NovaLimitHostsToRequest { hosts = limitHostsToRequest(requestData, hosts) logger.V(1).Info("limited hosts to request", - "hosts", hosts, "originalHosts", decision.Status.Result.OrderedHosts) + "hosts", hosts, "originalHosts", result.OrderedHosts) } // This is a hack to address the problem that Nova only uses the first host in hosts for evacuation requests. // Only for evacuation we shuffle the first k hosts to ensure that we do not get stuck on a single host diff --git a/internal/scheduling/nova/external_scheduler_api_test.go b/internal/scheduling/nova/external_scheduler_api_test.go index f3dd2c82b..5a2014b27 100644 --- a/internal/scheduling/nova/external_scheduler_api_test.go +++ b/internal/scheduling/nova/external_scheduler_api_test.go @@ -14,48 +14,72 @@ import ( "testing" novaapi "github.com/cobaltcore-dev/cortex/api/external/nova" - "github.com/cobaltcore-dev/cortex/api/v1alpha1" - - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" + scheduling "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" + "github.com/cobaltcore-dev/cortex/internal/scheduling/nova/crs" ) -type mockHTTPAPIDelegate struct { - processDecisionFunc func(ctx context.Context, decision *v1alpha1.Decision) error +// mockRunner is a test double for the scheduler the HTTP API depends on. runFunc +// receives the mapped request and may invoke the post-hook to exercise the +// handler's side-effect path; it returns the result/error the handler sees. +type mockRunner struct { + runFunc func(ctx context.Context, req compute.PlacementRequest, post compute.PostHook) (compute.PlacementResult, error) } -func (m *mockHTTPAPIDelegate) ProcessNewDecisionFromAPI(ctx context.Context, decision *v1alpha1.Decision) error { - if m.processDecisionFunc != nil { - return m.processDecisionFunc(ctx, decision) +func (m *mockRunner) Run(ctx context.Context, req compute.PlacementRequest, _ compute.PreHook, post compute.PostHook) (compute.PlacementResult, error) { + if m.runFunc != nil { + return m.runFunc(ctx, req, post) } - return nil + return compute.PlacementResult{}, nil } -func TestNewAPI(t *testing.T) { - delegate := &mockHTTPAPIDelegate{} - - api := NewAPI(HTTPAPIConfig{}, delegate) - - if api == nil { - t.Fatal("NewAPI returned nil") +// newTestAPI builds an *httpAPI with the given config and mock runner, leaving +// the side-effect handlers nil (the handler tests that need them set runFunc to +// not invoke the post-hook, or provide their own). +func newTestAPI(config HTTPAPIConfig, r runner) *httpAPI { + return &httpAPI{ + monitor: scheduling.NewSchedulerMonitor(), + scheduler: r, + config: config, } +} - httpAPI, ok := api.(*httpAPI) - if !ok { - t.Fatal("NewAPI did not return httpAPI type") +// newTestAPIWithScheduler builds an *httpAPI over a real compute.Placement plus +// the CR recorder and feature gates, for tests that exercise the committed- +// resource post-hook end-to-end. +func newTestAPIWithScheduler(config HTTPAPIConfig, scheduler runner, crRecorder crs.Recorder, featureGates FeatureGates) *httpAPI { + return &httpAPI{ + monitor: scheduling.NewSchedulerMonitor(), + scheduler: scheduler, + crRecorder: crRecorder, + featureGates: featureGates, + config: config, } +} - if httpAPI.delegate != delegate { - t.Error("Delegate not set correctly") - } +// passthroughGatherer is a CandidateGatherer that leaves the request's hosts +// unchanged (tests seed their own host sets). +type passthroughGatherer struct{} +func (passthroughGatherer) MutateWithAllCandidates(context.Context, *compute.PlacementRequest) error { + return nil +} + +func TestNewAPI(t *testing.T) { + r := &mockRunner{} + + httpAPI := newTestAPI(HTTPAPIConfig{}, r) + + if httpAPI.scheduler != r { + t.Error("scheduler (runner) not set correctly") + } if httpAPI.monitor.ApiRequestsTimer == nil { t.Error("Monitor not initialized") } } func TestHTTPAPI_Init(t *testing.T) { - delegate := &mockHTTPAPIDelegate{} - api := NewAPI(HTTPAPIConfig{}, delegate) + var api HTTPAPI = newTestAPI(HTTPAPIConfig{}, &mockRunner{}) mux := http.NewServeMux() api.Init(mux) @@ -72,8 +96,7 @@ func TestHTTPAPI_Init(t *testing.T) { } func TestHTTPAPI_canRunScheduler(t *testing.T) { - delegate := &mockHTTPAPIDelegate{} - api := NewAPI(HTTPAPIConfig{}, delegate).(*httpAPI) + api := newTestAPI(HTTPAPIConfig{}, &mockRunner{}) tests := []struct { name string @@ -146,7 +169,7 @@ func TestHTTPAPI_NovaExternalScheduler(t *testing.T) { method string body string processDecisionErr error - decisionResult *v1alpha1.Decision + result *compute.PlacementResult expectedStatus int expectedHosts []string }{ @@ -185,12 +208,8 @@ func TestHTTPAPI_NovaExternalScheduler(t *testing.T) { } return string(data) }(), - decisionResult: &v1alpha1.Decision{ - Status: v1alpha1.DecisionStatus{ - Result: &v1alpha1.DecisionResult{ - OrderedHosts: []string{"host1"}, - }, - }, + result: &compute.PlacementResult{ + OrderedHosts: []string{"host1"}, }, expectedStatus: http.StatusOK, expectedHosts: []string{"host1"}, @@ -246,37 +265,26 @@ func TestHTTPAPI_NovaExternalScheduler(t *testing.T) { } return string(data) }(), - decisionResult: &v1alpha1.Decision{ - Status: v1alpha1.DecisionStatus{ - Conditions: []metav1.Condition{ - { - Type: v1alpha1.DecisionConditionReady, - Status: metav1.ConditionFalse, - Reason: "SchedulingError", - }, - }, - }, - }, - expectedStatus: http.StatusInternalServerError, + processDecisionErr: errors.New("scheduling failed"), + expectedStatus: http.StatusInternalServerError, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - delegate := &mockHTTPAPIDelegate{ - processDecisionFunc: func(ctx context.Context, decision *v1alpha1.Decision) error { + r := &mockRunner{ + runFunc: func(_ context.Context, _ compute.PlacementRequest, _ compute.PostHook) (compute.PlacementResult, error) { if tt.processDecisionErr != nil { - return tt.processDecisionErr + return compute.PlacementResult{}, tt.processDecisionErr } - if tt.decisionResult != nil { - decision.Status = tt.decisionResult.Status - return nil + if tt.result != nil { + return *tt.result, nil } - return nil + return compute.PlacementResult{}, nil }, } - api := NewAPI(HTTPAPIConfig{}, delegate).(*httpAPI) + api := newTestAPI(HTTPAPIConfig{}, r) var body *strings.Reader if tt.body != "" { @@ -318,15 +326,12 @@ func TestHTTPAPI_NovaExternalScheduler_EmptyHostsSerializedAsArray(t *testing.T) // When the pipeline filters out all hosts, OrderedHosts is nil. The // response must serialize hosts as an empty array, not null, since Nova's // schema requires an array. - delegate := &mockHTTPAPIDelegate{ - processDecisionFunc: func(ctx context.Context, decision *v1alpha1.Decision) error { - decision.Status = v1alpha1.DecisionStatus{ - Result: &v1alpha1.DecisionResult{OrderedHosts: nil}, - } - return nil + r := &mockRunner{ + runFunc: func(context.Context, compute.PlacementRequest, compute.PostHook) (compute.PlacementResult, error) { + return compute.PlacementResult{OrderedHosts: nil}, nil }, } - api := NewAPI(HTTPAPIConfig{}, delegate).(*httpAPI) + api := newTestAPI(HTTPAPIConfig{}, r) req := novaapi.ExternalSchedulerRequest{ Spec: novaapi.NovaObject[novaapi.NovaSpec]{ @@ -357,20 +362,16 @@ func TestHTTPAPI_NovaExternalScheduler_EmptyHostsSerializedAsArray(t *testing.T) } } -func TestHTTPAPI_NovaExternalScheduler_DecisionCreation(t *testing.T) { - var capturedDecision *v1alpha1.Decision - delegate := &mockHTTPAPIDelegate{ - processDecisionFunc: func(ctx context.Context, decision *v1alpha1.Decision) error { - capturedDecision = decision - // Set a successful result to avoid "decision didn't produce a result" error - decision.Status.Result = &v1alpha1.DecisionResult{ - OrderedHosts: []string{"host1"}, - } - return nil +func TestHTTPAPI_NovaExternalScheduler_RequestMapping(t *testing.T) { + var captured compute.PlacementRequest + r := &mockRunner{ + runFunc: func(_ context.Context, req compute.PlacementRequest, _ compute.PostHook) (compute.PlacementResult, error) { + captured = req + return compute.PlacementResult{OrderedHosts: []string{"host1"}}, nil }, } - api := NewAPI(HTTPAPIConfig{}, delegate).(*httpAPI) + api := newTestAPI(HTTPAPIConfig{}, r) requestData := novaapi.ExternalSchedulerRequest{ Spec: novaapi.NovaObject[novaapi.NovaSpec]{ @@ -400,29 +401,16 @@ func TestHTTPAPI_NovaExternalScheduler_DecisionCreation(t *testing.T) { t.Errorf("Expected status %d, got %d", http.StatusOK, w.Code) } - if capturedDecision == nil { - t.Fatal("Decision was not captured") + // Verify the handler mapped the wire request into the typed scheduler request. + if captured.Pipeline != "test-pipeline" { + t.Errorf("Expected pipeline 'test-pipeline', got %s", captured.Pipeline) } - - // Verify decision fields - if capturedDecision.Spec.SchedulingDomain != v1alpha1.SchedulingDomainNova { - t.Errorf("Expected scheduling domain %s, got %s", v1alpha1.SchedulingDomainNova, capturedDecision.Spec.SchedulingDomain) - } - - if capturedDecision.Spec.PipelineRef.Name != "test-pipeline" { - t.Errorf("Expected pipeline 'test-pipeline', got %s", capturedDecision.Spec.PipelineRef.Name) - } - - if capturedDecision.Spec.ResourceID != "test-uuid-123" { - t.Errorf("Expected resource ID 'test-uuid-123', got %s", capturedDecision.Spec.ResourceID) - } - - if capturedDecision.GenerateName != "nova-" { - t.Errorf("Expected generate name 'nova-', got %s", capturedDecision.GenerateName) + if captured.InstanceUUID != "test-uuid-123" { + t.Errorf("Expected instance UUID 'test-uuid-123', got %s", captured.InstanceUUID) } - - if capturedDecision.Spec.NovaRaw == nil { - t.Error("NovaRaw should not be nil") + // No explicit _nova_check_type hint means the handler defaults to the create intent. + if captured.Intent != novaapi.CreateIntent { + t.Errorf("Expected intent %s, got %s", novaapi.CreateIntent, captured.Intent) } } @@ -445,14 +433,14 @@ func TestHTTPAPI_NovaExternalScheduler_ForcedDestination(t *testing.T) { }, } - delegateCalled := false - delegate := &mockHTTPAPIDelegate{ - processDecisionFunc: func(ctx context.Context, decision *v1alpha1.Decision) error { - delegateCalled = true - return nil + schedulerCalled := false + r := &mockRunner{ + runFunc: func(context.Context, compute.PlacementRequest, compute.PostHook) (compute.PlacementResult, error) { + schedulerCalled = true + return compute.PlacementResult{}, nil }, } - api := NewAPI(HTTPAPIConfig{}, delegate).(*httpAPI) + api := newTestAPI(HTTPAPIConfig{}, r) body, err := json.Marshal(requestData) if err != nil { @@ -466,8 +454,8 @@ func TestHTTPAPI_NovaExternalScheduler_ForcedDestination(t *testing.T) { if w.Code != http.StatusOK { t.Fatalf("Expected status %d, got %d", http.StatusOK, w.Code) } - if delegateCalled { - t.Error("delegate should not be called for forced destination requests") + if schedulerCalled { + t.Error("scheduler should not be called for forced destination requests") } var response novaapi.ExternalSchedulerResponse if err := json.NewDecoder(w.Body).Decode(&response); err != nil { @@ -495,14 +483,14 @@ func TestHTTPAPI_NovaExternalScheduler_ForcedDestinationNoWeights(t *testing.T) // for forced-destination requests. } - delegateCalled := false - delegate := &mockHTTPAPIDelegate{ - processDecisionFunc: func(ctx context.Context, decision *v1alpha1.Decision) error { - delegateCalled = true - return nil + schedulerCalled := false + r := &mockRunner{ + runFunc: func(context.Context, compute.PlacementRequest, compute.PostHook) (compute.PlacementResult, error) { + schedulerCalled = true + return compute.PlacementResult{}, nil }, } - api := NewAPI(HTTPAPIConfig{}, delegate).(*httpAPI) + api := newTestAPI(HTTPAPIConfig{}, r) body, err := json.Marshal(requestData) if err != nil { @@ -516,8 +504,8 @@ func TestHTTPAPI_NovaExternalScheduler_ForcedDestinationNoWeights(t *testing.T) if w.Code != http.StatusOK { t.Fatalf("Expected status %d, got %d (forced destination with no weights must not be rejected)", http.StatusOK, w.Code) } - if delegateCalled { - t.Error("delegate should not be called for forced destination requests") + if schedulerCalled { + t.Error("scheduler should not be called for forced destination requests") } var response novaapi.ExternalSchedulerResponse if err := json.NewDecoder(w.Body).Decode(&response); err != nil { @@ -548,15 +536,15 @@ func TestHTTPAPI_NovaExternalScheduler_ForcedDestinationDisabled(t *testing.T) { Pipeline: "test-pipeline", } - delegateCalled := false - delegate := &mockHTTPAPIDelegate{ - processDecisionFunc: func(ctx context.Context, decision *v1alpha1.Decision) error { - delegateCalled = true - return nil + schedulerCalled := false + r := &mockRunner{ + runFunc: func(context.Context, compute.PlacementRequest, compute.PostHook) (compute.PlacementResult, error) { + schedulerCalled = true + return compute.PlacementResult{}, nil }, } disabled := false - api := NewAPI(HTTPAPIConfig{ForcedDestinationEnabled: &disabled}, delegate).(*httpAPI) + api := newTestAPI(HTTPAPIConfig{ForcedDestinationEnabled: &disabled}, r) body, err := json.Marshal(requestData) if err != nil { @@ -567,8 +555,8 @@ func TestHTTPAPI_NovaExternalScheduler_ForcedDestinationDisabled(t *testing.T) { api.NovaExternalScheduler(w, req) - if !delegateCalled { - t.Error("delegate should be called when forced destination is disabled") + if !schedulerCalled { + t.Error("scheduler should be called when forced destination is disabled") } } @@ -804,8 +792,7 @@ func TestShuffleTopHosts(t *testing.T) { } func TestHTTPAPI_inferPipelineName(t *testing.T) { - delegate := &mockHTTPAPIDelegate{} - api := NewAPI(HTTPAPIConfig{}, delegate).(*httpAPI) + api := newTestAPI(HTTPAPIConfig{}, &mockRunner{}) tests := []struct { name string diff --git a/internal/scheduling/nova/filter_weigher_pipeline_controller.go b/internal/scheduling/nova/filter_weigher_pipeline_controller.go index a81074072..7ee25cf55 100644 --- a/internal/scheduling/nova/filter_weigher_pipeline_controller.go +++ b/internal/scheduling/nova/filter_weigher_pipeline_controller.go @@ -5,20 +5,11 @@ package nova import ( "context" - "encoding/json" - "errors" - "fmt" - "slices" - "sync" - "time" - api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "k8s.io/apimachinery/pkg/api/meta" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" - "github.com/cobaltcore-dev/cortex/internal/scheduling/nova/crs" "github.com/cobaltcore-dev/cortex/internal/scheduling/nova/plugins/filters" "github.com/cobaltcore-dev/cortex/internal/scheduling/nova/plugins/weighers" "github.com/cobaltcore-dev/cortex/pkg/multicluster" @@ -30,32 +21,18 @@ import ( "sigs.k8s.io/controller-runtime/pkg/predicate" ) -// The decision pipeline controller takes decision resources containing a -// placement request spec and runs the scheduling pipeline to make a decision. -// This decision is then written back to the decision resource status. -// -// Additionally, the controller watches for pipeline and step changes to -// reconfigure the pipelines as needed. +// FilterWeigherPipelineController owns the Nova filter-weigher pipelines: it +// watches Pipeline and Knowledge resources to (re)configure the pipelines, and +// exposes them via GetPipeline for the compute.Placement to run. Placement +// requests no longer flow through this controller — the compute.Placement, +// driven by the external-scheduler HTTP API and the reservation controllers, is +// the entrypoint. type FilterWeigherPipelineController struct { // Toolbox shared between all pipeline controllers. - lib.BasePipelineController[lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]] + lib.BasePipelineController[lib.FilterWeigherPipeline[compute.PlacementRequest]] - // Mutex to only allow one process at a time. - // Read-only runs (opts.ReadOnly == true) acquire a read lock; write runs acquire the full lock. - processMu sync.RWMutex - - // FeatureGates holds feature flags for this controller. - FeatureGates FeatureGates // Monitor to pass down to all pipelines. Monitor lib.FilterWeigherPipelineMonitor - // Candidate gatherer to get all placement candidates if needed. - gatherer CandidateGatherer - - // vmHandler maps processed decisions into vm-operator VirtualMachine resources. - vmHandler *VirtualMachineHandler - - // CRRecorder receives scheduling events and updates CR reservations and metrics. - CRRecorder crs.Recorder } // The type of pipeline this controller manages. @@ -63,183 +40,10 @@ func (c *FilterWeigherPipelineController) PipelineType() v1alpha1.PipelineType { return v1alpha1.PipelineTypeFilterWeigher } -// Callback executed when kubernetes asks to reconcile a decision resource. -func (c *FilterWeigherPipelineController) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { - // Peek at the decision before acquiring the lock so we can choose the right lock type. - // Read-only runs can proceed concurrently; write runs need the exclusive lock. - decision := &v1alpha1.Decision{} - if err := c.Get(ctx, req.NamespacedName, decision); err != nil { - return ctrl.Result{}, client.IgnoreNotFound(err) - } - if c.peekReadOnly(decision) { - c.processMu.RLock() - defer c.processMu.RUnlock() - } else { - c.processMu.Lock() - defer c.processMu.Unlock() - // Re-fetch after acquiring the exclusive lock to see consistent state. - if err := c.Get(ctx, req.NamespacedName, decision); err != nil { - return ctrl.Result{}, client.IgnoreNotFound(err) - } - } - old := decision.DeepCopy() - if _, err := c.process(ctx, decision); err != nil { - return ctrl.Result{}, err - } - patch := client.MergeFrom(old) - if err := c.Status().Patch(ctx, decision, patch); err != nil { - return ctrl.Result{}, err - } - return ctrl.Result{}, nil -} - -// Process the decision from the API. Should create and return the updated decision. -func (c *FilterWeigherPipelineController) ProcessNewDecisionFromAPI(ctx context.Context, decision *v1alpha1.Decision) error { - // Read-only runs share the cached decision state; no re-fetch needed because they - // don't observe writes from concurrent exclusive-lock runs. - if c.peekReadOnly(decision) { - c.processMu.RLock() - defer c.processMu.RUnlock() - } else { - c.processMu.Lock() - defer c.processMu.Unlock() - } - - request, err := c.process(ctx, decision) - if err != nil { - meta.SetStatusCondition(&decision.Status.Conditions, metav1.Condition{ - Type: v1alpha1.DecisionConditionReady, - Status: metav1.ConditionFalse, - Reason: "PipelineRunFailed", - Message: "pipeline run failed: " + err.Error(), - }) - } else { - meta.SetStatusCondition(&decision.Status.Conditions, metav1.Condition{ - Type: v1alpha1.DecisionConditionReady, - Status: metav1.ConditionTrue, - Reason: "PipelineRunSucceeded", - Message: "pipeline run succeeded", - }) - } - if err == nil && decision.Status.Result != nil && request != nil && c.FeatureGates.CommittedResourceTracking { - if decision.Status.Result.TargetHost != nil { - if request.Options.SkipCommittedResourceTracking { - ctrl.LoggerFrom(ctx).V(1).Info("CR allocation tracking skipped (SkipCommittedResourceTracking=true)", - "instanceUUID", request.Spec.Data.InstanceUUID, - "targetHost", *decision.Status.Result.TargetHost, - "intent", decision.Spec.Intent, - ) - } else { - c.CRRecorder.RecordPlacement(ctx, decision, *request) - } - } - if decision.Status.Result.TargetHost == nil { - c.CRRecorder.RecordNoHostFound(ctx, decision, *request) - } - } - return err -} - -func (c *FilterWeigherPipelineController) process(ctx context.Context, decision *v1alpha1.Decision) (*api.ExternalSchedulerRequest, error) { - log := ctrl.LoggerFrom(ctx) - startedAt := time.Now() // So we can measure sync duration. - - pipeline, ok := c.GetPipeline(decision.Spec.PipelineRef.Name) - if !ok { - log.Error(nil, "pipeline not found or not ready", "pipelineName", decision.Spec.PipelineRef.Name) - return nil, fmt.Errorf("pipeline not found or not ready: %q", decision.Spec.PipelineRef.Name) - } - if decision.Spec.NovaRaw == nil { - log.Error(nil, "skipping decision, no novaRaw spec defined") - return nil, errors.New("no novaRaw spec defined") - } - var request api.ExternalSchedulerRequest - if err := json.Unmarshal(decision.Spec.NovaRaw.Raw, &request); err != nil { - log.Error(err, "failed to unmarshal novaRaw spec") - return nil, err - } - - // If necessary gather all placement candidates before filtering. - // This will override the hosts and weights in the nova request. - pipelineConf, ok := c.GetPipelineConfig(decision.Spec.PipelineRef.Name) - if !ok { - log.Error(nil, "pipeline config not found", "pipelineName", decision.Spec.PipelineRef.Name) - return &request, fmt.Errorf("pipeline config not found: %q", decision.Spec.PipelineRef.Name) - } - if pipelineConf.Spec.IgnorePreselection { - log.V(1).Info("gathering all placement candidates before filtering") - if err := c.gatherer.MutateWithAllCandidates(ctx, &request); err != nil { - log.Error(err, "failed to gather all placement candidates") - return &request, err - } - log.V(1).Info("gathered all placement candidates", "numHosts", len(request.Hosts)) - } - - result, err := pipeline.Run(request) - if err == nil { - decision.Status.Result = &result - } - // Record a human-readable explanation of the decision so it propagates to - // the VM CortexSpec.Explanation. On error the result is nil and the - // explanation carries the failure reason. - decision.Status.Explanation = lib.GenerateExplanation(decision.Status.Result, err) - // We create a virtual machine resource based on the scheduling decision, - // logging the specific reason whenever we skip creation so it can be traced. - instanceUUID := request.Spec.Data.InstanceUUID - switch { - case err != nil: - log.V(1).Info("skipping virtual machine creation, pipeline run failed", "instanceUUID", instanceUUID) - case request.Options.ReadOnly: - log.V(1).Info("skipping virtual machine creation, request is read-only", "instanceUUID", instanceUUID) - case !c.FeatureGates.VirtualMachineCreation: - log.V(1).Info("skipping virtual machine creation, feature gate disabled", "instanceUUID", instanceUUID) - case !slices.Contains(api.NovaIntents, decision.Spec.Intent): - log.V(1).Info("skipping virtual machine creation, intent not triggered by Nova", "instanceUUID", instanceUUID, "intent", decision.Spec.Intent) - case !request.IsKVMRequest(): - log.V(1).Info("skipping virtual machine creation, unsupported hypervisor type", "instanceUUID", instanceUUID) - default: - vmUpsertErr := c.vmHandler.Upsert(ctx, decision, request) - if c.FeatureGates.VirtualMachineCreationMandatory && vmUpsertErr != nil { - log.Error(vmUpsertErr, "failed to upsert virtual machine resource and it is mandatory", "instanceUUID", instanceUUID) - return &request, vmUpsertErr - } - if vmUpsertErr != nil { - // Non-fatal: the placement decision itself succeeded. - log.Error(vmUpsertErr, "failed to upsert virtual machine resource", "instanceUUID", instanceUUID) - } - } - - if err != nil { - log.Error(err, "failed to run pipeline") - return &request, err - } - meta.SetStatusCondition(&decision.Status.Conditions, metav1.Condition{ - Type: v1alpha1.DecisionConditionReady, - Status: metav1.ConditionTrue, - Reason: "PipelineRunSucceeded", - Message: "pipeline run succeeded", - }) - log.Info("decision processed successfully", "duration", time.Since(startedAt)) - return &request, nil -} - -// peekReadOnly determines whether a decision should use a read lock instead of -// the exclusive write lock. Defaults to false (exclusive) on any parse error. -func (c *FilterWeigherPipelineController) peekReadOnly(decision *v1alpha1.Decision) bool { - if decision.Spec.NovaRaw == nil { - return false - } - var request api.ExternalSchedulerRequest - if err := json.Unmarshal(decision.Spec.NovaRaw.Raw, &request); err != nil { - return false - } - return request.Options.ReadOnly -} - func (c *FilterWeigherPipelineController) InitPipeline( ctx context.Context, p v1alpha1.Pipeline, -) lib.PipelineInitResult[lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]] { +) lib.PipelineInitResult[lib.FilterWeigherPipeline[compute.PlacementRequest]] { return lib.InitNewFilterWeigherPipeline( ctx, c.Client, p.Name, @@ -252,8 +56,6 @@ func (c *FilterWeigherPipelineController) InitPipeline( func (c *FilterWeigherPipelineController) SetupWithManager(mgr manager.Manager, mcl *multicluster.Client) error { c.Initializer = c c.SchedulingDomain = v1alpha1.SchedulingDomainNova - c.gatherer = &candidateGatherer{Client: mcl} - c.vmHandler = &VirtualMachineHandler{Client: mcl} if err := mgr.Add(manager.RunnableFunc(c.InitAllPipelines)); err != nil { return err } @@ -310,25 +112,14 @@ func (c *FilterWeigherPipelineController) SetupWithManager(mgr manager.Manager, if err != nil { return err } - // Watch decision changes across all clusters. - bldr, err = bldr.WatchesMulticluster( - &v1alpha1.Decision{}, - &handler.EnqueueRequestForObject{}, - predicate.NewPredicateFuncs(func(obj client.Object) bool { - decision := obj.(*v1alpha1.Decision) - if decision.Spec.SchedulingDomain != v1alpha1.SchedulingDomainNova { - return false - } - // Ignore already decided schedulings. - if decision.Status.Result != nil { - return false - } - return true - }), - ) - if err != nil { - return err - } - return bldr.Named("cortex-nova-decisions"). + return bldr.Named("cortex-nova-pipelines"). Complete(c) } + +// Reconcile is a no-op: this controller reconfigures pipelines through its +// Pipeline/Knowledge watch handlers, and placement runs through the +// compute.Placement, not through object reconciliation. A Reconcile method is +// required to satisfy the controller builder. +func (c *FilterWeigherPipelineController) Reconcile(context.Context, ctrl.Request) (ctrl.Result, error) { + return ctrl.Result{}, nil +} diff --git a/internal/scheduling/nova/filter_weigher_pipeline_controller_test.go b/internal/scheduling/nova/filter_weigher_pipeline_controller_test.go index f7c4f5a41..26576100c 100644 --- a/internal/scheduling/nova/filter_weigher_pipeline_controller_test.go +++ b/internal/scheduling/nova/filter_weigher_pipeline_controller_test.go @@ -5,22 +5,12 @@ package nova import ( "context" - "encoding/json" - "errors" - "strings" "testing" - corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/runtime" - "k8s.io/apimachinery/pkg/types" - ctrl "sigs.k8s.io/controller-runtime" - "sigs.k8s.io/controller-runtime/pkg/client" - "sigs.k8s.io/controller-runtime/pkg/client/fake" - api "github.com/cobaltcore-dev/cortex/api/external/nova" - "github.com/cobaltcore-dev/cortex/api/scheduling" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" ) @@ -28,10 +18,10 @@ import ( type mockCandidateGatherer struct { called bool err error - gatheredHosts []api.ExternalSchedulerHost + gatheredHosts []compute.Host } -func (m *mockCandidateGatherer) MutateWithAllCandidates(ctx context.Context, request *api.ExternalSchedulerRequest) error { +func (m *mockCandidateGatherer) MutateWithAllCandidates(ctx context.Context, request *compute.PlacementRequest) error { m.called = true if m.err != nil { return m.err @@ -46,239 +36,6 @@ func (m *mockCandidateGatherer) MutateWithAllCandidates(ctx context.Context, req return nil } -func TestFilterWeigherPipelineController_Reconcile(t *testing.T) { - scheme := runtime.NewScheme() - if err := v1alpha1.AddToScheme(scheme); err != nil { - t.Fatalf("Failed to add v1alpha1 scheme: %v", err) - } - - novaRequest := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Name: "RequestSpec", - Namespace: "nova_object", - Version: "1.19", - Data: api.NovaSpec{ - ProjectID: "test-project", - UserID: "test-user", - InstanceUUID: "test-instance-uuid", - NumInstances: 1, - }, - }, - Context: api.NovaRequestContext{ - ProjectID: "test-project", - UserID: "test-user", - RequestID: "req-123", - GlobalRequestID: func() *string { s := "global-req-123"; return &s }(), - }, - Hosts: []api.ExternalSchedulerHost{ - {ComputeHost: "compute-1", HypervisorHostname: "hv-1"}, - {ComputeHost: "compute-2", HypervisorHostname: "hv-2"}, - }, - Weights: map[string]float64{"compute-1": 1.0, "compute-2": 0.5}, - Pipeline: "test-pipeline", - Options: scheduling.Options{}, - } - - novaRaw, err := json.Marshal(novaRequest) - if err != nil { - t.Fatalf("Failed to marshal nova request: %v", err) - } - - tests := []struct { - name string - decision *v1alpha1.Decision - pipeline *v1alpha1.Pipeline - expectError bool - expectResult bool - }{ - { - name: "successful nova decision processing", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - NovaRaw: &runtime.RawExtension{ - Raw: novaRaw, - }, - }, - }, - pipeline: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainNova, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - expectError: false, - expectResult: true, - }, - { - name: "decision without novaRaw spec", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-raw", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - NovaRaw: nil, - }, - }, - pipeline: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainNova, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - expectError: true, - expectResult: false, - }, - { - name: "pipeline not found", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-pipeline", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - PipelineRef: corev1.ObjectReference{ - Name: "nonexistent-pipeline", - }, - NovaRaw: &runtime.RawExtension{ - Raw: novaRaw, - }, - }, - }, - pipeline: nil, - expectError: true, - expectResult: false, - }, - { - name: "invalid novaRaw JSON", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-invalid-json", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - NovaRaw: &runtime.RawExtension{ - Raw: []byte("invalid json"), - }, - }, - }, - pipeline: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainNova, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - expectError: true, - expectResult: false, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - objects := []client.Object{tt.decision} - if tt.pipeline != nil { - objects = append(objects, tt.pipeline) - } - - client := fake.NewClientBuilder(). - WithScheme(scheme). - WithObjects(objects...). - WithStatusSubresource(&v1alpha1.Decision{}). - Build() - - controller := &FilterWeigherPipelineController{ - BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]]{ - Client: client, - Pipelines: make(map[string]lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]), - PipelineConfigs: make(map[string]v1alpha1.Pipeline), - }, - Monitor: lib.FilterWeigherPipelineMonitor{}, - } - - if tt.pipeline != nil { - initResult := controller.InitPipeline(t.Context(), v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: tt.pipeline.Name, - }, - Spec: tt.pipeline.Spec, - }) - if len(initResult.FilterErrors) > 0 || len(initResult.WeigherErrors) > 0 { - t.Fatalf("Failed to initialize pipeline: filter errors: %v, weigher errors: %v", initResult.FilterErrors, initResult.WeigherErrors) - } - controller.Pipelines[tt.pipeline.Name] = initResult.Pipeline - controller.PipelineConfigs[tt.pipeline.Name] = *tt.pipeline - } - - req := ctrl.Request{ - NamespacedName: types.NamespacedName{ - Name: tt.decision.Name, - Namespace: tt.decision.Namespace, - }, - } - - result, err := controller.Reconcile(context.Background(), req) - - if tt.expectError && err == nil { - t.Error("Expected error but got none") - } - if !tt.expectError && err != nil { - t.Errorf("Expected no error but got: %v", err) - } - - if result.RequeueAfter > 0 { - t.Error("Expected no requeue") - } - - var updatedDecision v1alpha1.Decision - if err := client.Get(context.Background(), req.NamespacedName, &updatedDecision); err != nil { - if !tt.expectError { - t.Fatalf("Failed to get updated decision: %v", err) - } - return - } - - if tt.expectResult && updatedDecision.Status.Result == nil { - t.Error("Expected result to be set but was nil") - } - if !tt.expectResult && updatedDecision.Status.Result != nil { - t.Error("Expected result to be nil but was set") - } - }) - } -} - func TestFilterWeigherPipelineController_InitPipeline(t *testing.T) { controller := &FilterWeigherPipelineController{ Monitor: lib.FilterWeigherPipelineMonitor{}, @@ -355,609 +112,3 @@ func TestFilterWeigherPipelineController_InitPipeline(t *testing.T) { }) } } - -func TestFilterWeigherPipelineController_ProcessNewDecisionFromAPI(t *testing.T) { - scheme := runtime.NewScheme() - if err := v1alpha1.AddToScheme(scheme); err != nil { - t.Fatalf("Failed to add v1alpha1 scheme: %v", err) - } - - novaRequest := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Name: "RequestSpec", - Namespace: "nova_object", - Version: "1.19", - Data: api.NovaSpec{ - ProjectID: "test-project", - UserID: "test-user", - InstanceUUID: "test-instance-uuid", - NumInstances: 1, - }, - }, - Context: api.NovaRequestContext{ - ProjectID: "test-project", - UserID: "test-user", - RequestID: "req-123", - GlobalRequestID: func() *string { s := "global-req-123"; return &s }(), - }, - Hosts: []api.ExternalSchedulerHost{ - {ComputeHost: "compute-1", HypervisorHostname: "hv-1"}, - {ComputeHost: "compute-2", HypervisorHostname: "hv-2"}, - }, - Weights: map[string]float64{"compute-1": 1.0, "compute-2": 0.5}, - Pipeline: "test-pipeline", - } - - novaRaw, err := json.Marshal(novaRequest) - if err != nil { - t.Fatalf("Failed to marshal nova request: %v", err) - } - - tests := []struct { - name string - decision *v1alpha1.Decision - pipeline *v1alpha1.Pipeline - pipelineConf *v1alpha1.Pipeline - setupPipelineConfigs bool - expectError bool - expectResult bool - expectUpdatedStatus bool - errorContains string - }{ - { - name: "successful processing with decision creation enabled", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-api", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - ResourceID: "test-uuid-1", - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - NovaRaw: &runtime.RawExtension{ - Raw: novaRaw, - }, - }, - }, - pipeline: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainNova, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - pipelineConf: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainNova, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - setupPipelineConfigs: true, - expectError: false, - expectResult: true, - expectUpdatedStatus: true, - }, - { - name: "successful processing with decision creation disabled", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-create", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - ResourceID: "test-uuid-2", - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline-no-create", - }, - NovaRaw: &runtime.RawExtension{ - Raw: novaRaw, - }, - }, - }, - pipeline: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline-no-create", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainNova, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - pipelineConf: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline-no-create", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainNova, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - setupPipelineConfigs: true, - expectError: false, - expectResult: true, - expectUpdatedStatus: false, - }, - { - name: "pipeline not configured", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-config", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - PipelineRef: corev1.ObjectReference{ - Name: "nonexistent-pipeline", - }, - NovaRaw: &runtime.RawExtension{ - Raw: novaRaw, - }, - }, - }, - pipeline: nil, - pipelineConf: nil, - setupPipelineConfigs: false, - expectError: true, - expectResult: false, - expectUpdatedStatus: false, - errorContains: "pipeline not found or not ready", - }, - { - name: "decision without novaRaw spec", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-raw-api", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - NovaRaw: nil, - }, - }, - pipeline: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainNova, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - pipelineConf: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainNova, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - setupPipelineConfigs: true, - expectError: true, - expectResult: false, - expectUpdatedStatus: false, - errorContains: "no novaRaw spec defined", - }, - { - name: "processing fails after decision creation", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-process-fail", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - NovaRaw: &runtime.RawExtension{ - Raw: novaRaw, - }, - }, - }, - pipeline: nil, // This will cause processing to fail after creation - pipelineConf: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainNova, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - setupPipelineConfigs: true, - expectError: true, - expectResult: false, - expectUpdatedStatus: false, - errorContains: "pipeline not found or not ready", - }, - { - name: "pipeline not found in runtime map", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-runtime-pipeline", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - PipelineRef: corev1.ObjectReference{ - Name: "missing-runtime-pipeline", - }, - NovaRaw: &runtime.RawExtension{ - Raw: novaRaw, - }, - }, - }, - pipeline: nil, - pipelineConf: &v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "missing-runtime-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainNova, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - }, - setupPipelineConfigs: true, - expectError: true, - expectResult: false, - expectUpdatedStatus: false, - errorContains: "pipeline not found or not ready", - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - objects := []client.Object{} - if tt.pipeline != nil { - objects = append(objects, tt.pipeline) - } - - client := fake.NewClientBuilder(). - WithScheme(scheme). - WithObjects(objects...). - WithStatusSubresource(&v1alpha1.Decision{}). - Build() - - controller := &FilterWeigherPipelineController{ - BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]]{ - Client: client, - Pipelines: make(map[string]lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]), - PipelineConfigs: make(map[string]v1alpha1.Pipeline), - }, - Monitor: lib.FilterWeigherPipelineMonitor{}, - } - - // Setup pipeline configurations if needed - if tt.setupPipelineConfigs && tt.pipelineConf != nil { - controller.PipelineConfigs[tt.pipelineConf.Name] = *tt.pipelineConf - } - - // Setup runtime pipeline if needed - if tt.pipeline != nil { - initResult := controller.InitPipeline(context.Background(), v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: tt.pipeline.Name, - }, - Spec: tt.pipeline.Spec, - }) - if len(initResult.FilterErrors) > 0 || len(initResult.WeigherErrors) > 0 { - t.Fatalf("Failed to initialize pipeline: filter errors: %v, weigher errors: %v", initResult.FilterErrors, initResult.WeigherErrors) - } - controller.Pipelines[tt.pipeline.Name] = initResult.Pipeline - } - - if tt.decision.Spec.NovaRaw != nil { - req := novaRequest - req.Options = scheduling.Options{} - raw, marshalErr := json.Marshal(req) - if marshalErr != nil { - t.Fatalf("Failed to marshal request with options: %v", marshalErr) - } - tt.decision.Spec.NovaRaw = &runtime.RawExtension{Raw: raw} - } - - // Call the method under test - err := controller.ProcessNewDecisionFromAPI(context.Background(), tt.decision) - - // Validate error expectations - if tt.expectError && err == nil { - t.Error("Expected error but got none") - } - if !tt.expectError && err != nil { - t.Errorf("Expected no error but got: %v", err) - } - if tt.errorContains != "" && (err == nil || !strings.Contains(err.Error(), tt.errorContains)) { - t.Errorf("Expected error to contain %q, got: %v", tt.errorContains, err) - } - - // Validate result expectations - if tt.expectResult && tt.decision.Status.Result == nil { - t.Error("Expected result to be set but was nil") - } - if !tt.expectResult && tt.decision.Status.Result != nil { - t.Error("Expected result to be nil but was set") - } - }) - } -} - -func TestFilterWeigherPipelineController_IgnorePreselection(t *testing.T) { - scheme := runtime.NewScheme() - if err := v1alpha1.AddToScheme(scheme); err != nil { - t.Fatalf("Failed to add v1alpha1 scheme: %v", err) - } - - // Create a request with initial hosts - novaRequest := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Name: "RequestSpec", - Namespace: "nova_object", - Version: "1.19", - Data: api.NovaSpec{ - ProjectID: "test-project", - UserID: "test-user", - InstanceUUID: "test-instance-uuid", - NumInstances: 1, - }, - }, - Context: api.NovaRequestContext{ - ProjectID: "test-project", - UserID: "test-user", - RequestID: "req-123", - GlobalRequestID: func() *string { s := "global-req-123"; return &s }(), - }, - Hosts: []api.ExternalSchedulerHost{ - {ComputeHost: "original-host-1", HypervisorHostname: "hv-1"}, - {ComputeHost: "original-host-2", HypervisorHostname: "hv-2"}, - }, - Weights: map[string]float64{"original-host-1": 1.0, "original-host-2": 0.5}, - Pipeline: "test-pipeline", - Options: scheduling.Options{}, - } - - novaRaw, err := json.Marshal(novaRequest) - if err != nil { - t.Fatalf("Failed to marshal nova request: %v", err) - } - - tests := []struct { - name string - ignorePreselection bool - gathererErr error - gatheredHosts []api.ExternalSchedulerHost - expectGathererCall bool - expectError bool - errorContains string - }{ - { - name: "IgnorePreselection disabled - gatherer not called", - ignorePreselection: false, - gathererErr: nil, - gatheredHosts: nil, - expectGathererCall: false, - expectError: false, - }, - { - name: "IgnorePreselection enabled - gatherer called and succeeds", - ignorePreselection: true, - gathererErr: nil, - gatheredHosts: []api.ExternalSchedulerHost{ - {ComputeHost: "gathered-host-1", HypervisorHostname: "gathered-host-1"}, - {ComputeHost: "gathered-host-2", HypervisorHostname: "gathered-host-2"}, - {ComputeHost: "gathered-host-3", HypervisorHostname: "gathered-host-3"}, - }, - expectGathererCall: true, - expectError: false, - }, - { - name: "IgnorePreselection enabled - gatherer returns error", - ignorePreselection: true, - gathererErr: errGathererFailed, - gatheredHosts: nil, - expectGathererCall: true, - expectError: true, - errorContains: "gatherer failed", - }, - { - name: "IgnorePreselection enabled - gatherer returns empty hosts", - ignorePreselection: true, - gathererErr: nil, - gatheredHosts: []api.ExternalSchedulerHost{}, - expectGathererCall: true, - expectError: false, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - mockGatherer := &mockCandidateGatherer{ - err: tt.gathererErr, - gatheredHosts: tt.gatheredHosts, - } - - fakeClient := fake.NewClientBuilder(). - WithScheme(scheme). - WithStatusSubresource(&v1alpha1.Decision{}). - Build() - - controller := &FilterWeigherPipelineController{ - BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]]{ - Client: fakeClient, - Pipelines: make(map[string]lib.FilterWeigherPipeline[api.ExternalSchedulerRequest]), - PipelineConfigs: make(map[string]v1alpha1.Pipeline), - }, - Monitor: lib.FilterWeigherPipelineMonitor{}, - gatherer: mockGatherer, - } - - // Setup pipeline config with IgnorePreselection flag - pipelineConf := v1alpha1.Pipeline{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pipeline", - }, - Spec: v1alpha1.PipelineSpec{ - Type: v1alpha1.PipelineTypeFilterWeigher, - SchedulingDomain: v1alpha1.SchedulingDomainNova, - IgnorePreselection: tt.ignorePreselection, - Filters: []v1alpha1.FilterSpec{}, - Weighers: []v1alpha1.WeigherSpec{}, - }, - } - controller.PipelineConfigs["test-pipeline"] = pipelineConf - - // Initialize the pipeline - initResult := controller.InitPipeline(context.Background(), pipelineConf) - if len(initResult.FilterErrors) > 0 || len(initResult.WeigherErrors) > 0 { - t.Fatalf("Failed to initialize pipeline: filter errors: %v, weigher errors: %v", initResult.FilterErrors, initResult.WeigherErrors) - } - controller.Pipelines["test-pipeline"] = initResult.Pipeline - - // Create decision - decision := &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-preselection", - Namespace: "default", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - PipelineRef: corev1.ObjectReference{ - Name: "test-pipeline", - }, - NovaRaw: &runtime.RawExtension{ - Raw: novaRaw, - }, - }, - } - - // Process the decision - err := controller.ProcessNewDecisionFromAPI(context.Background(), decision) - - // Verify gatherer was called (or not) as expected - if tt.expectGathererCall && !mockGatherer.called { - t.Error("Expected gatherer to be called but it was not") - } - if !tt.expectGathererCall && mockGatherer.called { - t.Error("Expected gatherer not to be called but it was") - } - - // Verify error expectations - if tt.expectError && err == nil { - t.Error("Expected error but got none") - } - if !tt.expectError && err != nil { - t.Errorf("Expected no error but got: %v", err) - } - if tt.errorContains != "" && (err == nil || !strings.Contains(err.Error(), tt.errorContains)) { - t.Errorf("Expected error to contain %q, got: %v", tt.errorContains, err) - } - - // Verify result is set when no error - if !tt.expectError && decision.Status.Result == nil { - t.Error("Expected result to be set but was nil") - } - }) - } -} - -// Error variable for testing -var errGathererFailed = errors.New("gatherer failed") - -func TestFilterWeigherPipelineController_PeekReadOnly(t *testing.T) { - makeRaw := func(readOnly bool) []byte { - r := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{Data: api.NovaSpec{NumInstances: 1}}, - Options: scheduling.Options{ReadOnly: readOnly}, - } - raw, err := json.Marshal(r) - if err != nil { - panic(err) - } - return raw - } - - c := &FilterWeigherPipelineController{} - - tests := []struct { - name string - decision *v1alpha1.Decision - want bool - }{ - { - name: "nil NovaRaw defaults to exclusive lock", - decision: &v1alpha1.Decision{ - Spec: v1alpha1.DecisionSpec{ - PipelineRef: corev1.ObjectReference{Name: "test-pipeline"}, - }, - }, - want: false, - }, - { - name: "invalid JSON defaults to exclusive lock", - decision: &v1alpha1.Decision{ - Spec: v1alpha1.DecisionSpec{ - PipelineRef: corev1.ObjectReference{Name: "test-pipeline"}, - NovaRaw: &runtime.RawExtension{Raw: []byte("not-json")}, - }, - }, - want: false, - }, - { - name: "ReadOnly=false uses exclusive lock", - decision: &v1alpha1.Decision{ - Spec: v1alpha1.DecisionSpec{ - PipelineRef: corev1.ObjectReference{Name: "test-pipeline"}, - NovaRaw: &runtime.RawExtension{Raw: makeRaw(false)}, - }, - }, - want: false, - }, - { - name: "ReadOnly=true uses read lock", - decision: &v1alpha1.Decision{ - Spec: v1alpha1.DecisionSpec{ - PipelineRef: corev1.ObjectReference{Name: "test-pipeline"}, - NovaRaw: &runtime.RawExtension{Raw: makeRaw(true)}, - }, - }, - want: true, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - got := c.peekReadOnly(tt.decision) - if got != tt.want { - t.Errorf("expected peekReadOnly = %v, got %v", tt.want, got) - } - }) - } -} diff --git a/internal/scheduling/nova/integration_test.go b/internal/scheduling/nova/integration_test.go index f22ff3523..2ff129ff6 100644 --- a/internal/scheduling/nova/integration_test.go +++ b/internal/scheduling/nova/integration_test.go @@ -15,7 +15,9 @@ import ( novaapi "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" + "github.com/cobaltcore-dev/cortex/internal/scheduling/nova/crs" "github.com/cobaltcore-dev/cortex/internal/scheduling/nova/plugins/filters" "github.com/cobaltcore-dev/cortex/internal/scheduling/nova/plugins/weighers" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" @@ -76,7 +78,6 @@ func newCommittedReservation(name, targetHost, observedHost, projectID, flavorNa ProjectID: projectID, ResourceName: flavorName, ResourceGroup: flavorGroup, - Allocations: nil, }, }, Status: v1alpha1.ReservationStatus{ @@ -248,9 +249,9 @@ func NewIntegrationTestServer(t *testing.T, pipelineConfig PipelineConfig, objec // Create the pipeline controller with shared monitor to avoid duplicate registration controller := &FilterWeigherPipelineController{ - BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[novaapi.ExternalSchedulerRequest]]{ + BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[compute.PlacementRequest]]{ Client: k8sClient, - Pipelines: make(map[string]lib.FilterWeigherPipeline[novaapi.ExternalSchedulerRequest]), + Pipelines: make(map[string]lib.FilterWeigherPipeline[compute.PlacementRequest]), PipelineConfigs: make(map[string]v1alpha1.Pipeline), }, Monitor: getSharedMonitor(), @@ -282,12 +283,17 @@ func NewIntegrationTestServer(t *testing.T, pipelineConfig PipelineConfig, objec controller.Pipelines[testPipeline.Name] = result.Pipeline controller.PipelineConfigs[testPipeline.Name] = testPipeline - // Create the HTTP API with the controller as delegate - skip metrics registration - // Set EvacuationShuffleK=0 to disable shuffle for deterministic test results + // Create the HTTP API backed by a real compute.Placement over the controller. + // Set EvacuationShuffleK=0 to disable shuffle for deterministic test results. + // KVM requests trigger candidate gathering; use a passthrough gatherer + // (nil gatheredHosts) so the test's preselected hosts are preserved. + scheduler := compute.NewPlacement(controller, &mockCandidateGatherer{}) api := &httpAPI{ - config: HTTPAPIConfig{EvacuationShuffleK: 0}, - monitor: lib.NewSchedulerMonitor(), // Create new monitor but don't register - delegate: controller, + config: HTTPAPIConfig{EvacuationShuffleK: 0}, + monitor: lib.NewSchedulerMonitor(), // Create new monitor but don't register + scheduler: scheduler, + vmHandler: &VirtualMachineHandler{Client: k8sClient}, + crRecorder: crs.Recorder{Client: k8sClient}, } // Create test server diff --git a/internal/scheduling/nova/plugins/filters/filter_aggregate_metadata.go b/internal/scheduling/nova/plugins/filters/filter_aggregate_metadata.go index d223dfe5e..7a2aa17bc 100644 --- a/internal/scheduling/nova/plugins/filters/filter_aggregate_metadata.go +++ b/internal/scheduling/nova/plugins/filters/filter_aggregate_metadata.go @@ -10,23 +10,24 @@ import ( "strings" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" ) type FilterAggregateMetadata struct { - lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts] + lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts] } // Restrict hosts to specific projects if they are in an aggregate that has // the "filter_tenant_id" metadata key set. -func (s *FilterAggregateMetadata) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *FilterAggregateMetadata) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) // Capacity probe calls are synthetic and not placed on behalf of a tenant // project, so they must ignore tenant pinning. Failover reservations DO respect // tenant pinning: a reservation on a host the VM's project cannot use is dead // weight and would be deleted at validation time. - if intent, err := request.GetIntent(); err == nil && intent == api.CapacityProbeIntent { + if request.Intent == api.CapacityProbeIntent { return result, nil } @@ -71,20 +72,20 @@ func (s *FilterAggregateMetadata) Run(traceLog logr.Logger, request api.External } for host, restrictedProjects := range restrictedProjectsByHost { - if !slices.Contains(restrictedProjects, request.Spec.Data.ProjectID) { + if !slices.Contains(restrictedProjects, request.ProjectID) { // Project is not allowed on this hypervisor, filter it out. delete(result.Activations, host) traceLog.V(1).Info( "filtering host not allowing project based on aggregate metadata", "host", host, - "project", request.Spec.Data.ProjectID, + "project", request.ProjectID, "restricted_projects", restrictedProjects, ) } else { traceLog.V(1).Info( "host allows project based on aggregate metadata, keeping", "host", host, - "project", request.Spec.Data.ProjectID, + "project", request.ProjectID, "restricted_projects", restrictedProjects, ) } diff --git a/internal/scheduling/nova/plugins/filters/filter_aggregate_metadata_test.go b/internal/scheduling/nova/plugins/filters/filter_aggregate_metadata_test.go index 769d06729..c23479858 100644 --- a/internal/scheduling/nova/plugins/filters/filter_aggregate_metadata_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_aggregate_metadata_test.go @@ -9,6 +9,8 @@ import ( "testing" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -20,20 +22,16 @@ import ( func TestFilterAggregateMetadata_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request compute.PlacementRequest hypervisors []hv1.Hypervisor expectedHosts []string filteredHosts []string }{ { name: "No aggregates with filter_tenant_id - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-a", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-a", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -64,13 +62,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "Host with filter_tenant_id matching project - host passes", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-a", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-a", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -94,13 +88,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "Host with filter_tenant_id not matching project - host filtered", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-b", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-b", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -124,13 +114,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "Host in multiple aggregates with different filter_tenant_id - project matches one", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-b", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-b", + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, }, @@ -150,13 +136,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "Host in multiple aggregates - project matches none", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-c", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-c", + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, }, @@ -176,13 +158,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "Mixed hosts - some restricted, some open", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-a", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-a", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -220,13 +198,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "Empty host list", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-a", - }, - }, - Hosts: []api.ExternalSchedulerHost{}, + request: compute.PlacementRequest{ + ProjectID: "project-a", + Hosts: []compute.Host{}, }, hypervisors: []hv1.Hypervisor{ { @@ -241,13 +215,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "All hosts filtered out", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-nonexistent", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-nonexistent", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -271,13 +241,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "Host not in hypervisor list - passes (no restrictions found)", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-a", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-a", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host-unknown"}, }, @@ -295,13 +261,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "Aggregate with nil metadata - treated as no restriction", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-a", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-a", + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, }, @@ -318,13 +280,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "Comma-separated filter_tenant_id - project matches one of many", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-b", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-b", + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, }, @@ -341,13 +299,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "Comma-separated filter_tenant_id - project matches none", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-d", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-d", + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, }, @@ -364,13 +318,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "Comma-separated filter_tenant_id with surrounding whitespace - project matches", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-b", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-b", + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, }, @@ -387,13 +337,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "Numbered filter_tenant_id keys - project matches one of the numbered keys", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-c", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-c", + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, }, @@ -414,13 +360,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "Numbered filter_tenant_id keys with comma-separated values - project matches", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-e", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-e", + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, }, @@ -441,13 +383,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { }, { name: "Numbered filter_tenant_id keys - project matches none", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-z", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-z", + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, }, @@ -510,13 +448,9 @@ func TestFilterAggregateMetadata_Run(t *testing.T) { } func TestFilterAggregateMetadata_Run_ClientError(t *testing.T) { - request := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-a", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request := compute.PlacementRequest{ + ProjectID: "project-a", + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, } @@ -575,14 +509,10 @@ func TestFilterAggregateMetadata_SkipsForCapacityProbeIntent(t *testing.T) { step := &FilterAggregateMetadata{} step.Client = fake.NewClientBuilder().WithScheme(scheme).WithObjects(objects...).Build() - request := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-y", - SchedulerHints: map[string]any{"_nova_check_type": "capacity_probe"}, - }, - }, - Hosts: []api.ExternalSchedulerHost{{ComputeHost: "host1"}, {ComputeHost: "host2"}}, + request := compute.PlacementRequest{ + ProjectID: "project-y", + Intent: api.CapacityProbeIntent, + Hosts: []compute.Host{{ComputeHost: "host1"}, {ComputeHost: "host2"}}, } result, err := step.Run(logr.Discard(), request) if err != nil { @@ -612,18 +542,14 @@ func TestFilterAggregateMetadata_AppliesForFailoverIntents(t *testing.T) { &hv1.Hypervisor{ObjectMeta: metav1.ObjectMeta{Name: "host2"}}, } - for _, intent := range []string{"reserve_for_failover", "reuse_failover_reservation"} { - t.Run(intent, func(t *testing.T) { + for _, intent := range []v1alpha1.SchedulingIntent{api.ReserveForFailoverIntent, api.ReuseFailoverReservationIntent} { + t.Run(string(intent), func(t *testing.T) { step := &FilterAggregateMetadata{} step.Client = fake.NewClientBuilder().WithScheme(scheme).WithObjects(objects...).Build() - request := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-y", - SchedulerHints: map[string]any{"_nova_check_type": intent}, - }, - }, - Hosts: []api.ExternalSchedulerHost{{ComputeHost: "host1"}, {ComputeHost: "host2"}}, + request := compute.PlacementRequest{ + ProjectID: "project-y", + Intent: intent, + Hosts: []compute.Host{{ComputeHost: "host1"}, {ComputeHost: "host2"}}, } result, err := step.Run(logr.Discard(), request) if err != nil { diff --git a/internal/scheduling/nova/plugins/filters/filter_allowed_projects.go b/internal/scheduling/nova/plugins/filters/filter_allowed_projects.go index 5e1e51b5a..ee0cac37f 100644 --- a/internal/scheduling/nova/plugins/filters/filter_allowed_projects.go +++ b/internal/scheduling/nova/plugins/filters/filter_allowed_projects.go @@ -9,26 +9,27 @@ import ( "slices" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" ) type FilterAllowedProjectsStep struct { - lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts] + lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts] } // Lock certain hosts for certain projects, based on the hypervisor spec. // Note that hosts without specified projects are still accessible. -func (s *FilterAllowedProjectsStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *FilterAllowedProjectsStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) // Capacity probe calls are synthetic and not placed on behalf of a tenant // project, so they must ignore tenant pinning. Failover reservations DO respect // tenant pinning: a reservation on a host the VM's project cannot use is dead // weight and would be deleted at validation time. - if intent, err := request.GetIntent(); err == nil && intent == api.CapacityProbeIntent { + if request.Intent == api.CapacityProbeIntent { return result, nil } - if request.Spec.Data.ProjectID == "" { + if request.ProjectID == "" { traceLog.V(1).Info("no project ID in request, skipping filter") return result, nil } @@ -48,13 +49,13 @@ func (s *FilterAllowedProjectsStep) Run(traceLog logr.Logger, request api.Extern traceLog.V(1).Info("host allows all projects, keeping", "host", hv.Name) continue } - if !slices.Contains(hv.Spec.AllowedProjects, request.Spec.Data.ProjectID) { + if !slices.Contains(hv.Spec.AllowedProjects, request.ProjectID) { // Project is not allowed on this hypervisor, filter it out. delete(result.Activations, hv.Name) traceLog.V(1).Info( "filtering host not allowing project", "host", hv.Name, - "project", request.Spec.Data.ProjectID, + "project", request.ProjectID, ) } } diff --git a/internal/scheduling/nova/plugins/filters/filter_allowed_projects_test.go b/internal/scheduling/nova/plugins/filters/filter_allowed_projects_test.go index 755ea9903..3b4e9d60a 100644 --- a/internal/scheduling/nova/plugins/filters/filter_allowed_projects_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_allowed_projects_test.go @@ -8,6 +8,8 @@ import ( "testing" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -67,19 +69,15 @@ func TestFilterAllowedProjectsStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedHosts []string filteredHosts []string }{ { name: "No project ID in request - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -90,13 +88,9 @@ func TestFilterAllowedProjectsStep_Run(t *testing.T) { }, { name: "Project matches allowed projects on host2", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-a", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-a", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -107,13 +101,9 @@ func TestFilterAllowedProjectsStep_Run(t *testing.T) { }, { name: "Project matches allowed projects on host3 only", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-a", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-a", + Hosts: []compute.Host{ {ComputeHost: "host2"}, {ComputeHost: "host3"}, {ComputeHost: "host4"}, @@ -124,13 +114,9 @@ func TestFilterAllowedProjectsStep_Run(t *testing.T) { }, { name: "Project does not match any allowed projects", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-d", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-d", + Hosts: []compute.Host{ {ComputeHost: "host2"}, {ComputeHost: "host3"}, {ComputeHost: "host4"}, @@ -141,13 +127,9 @@ func TestFilterAllowedProjectsStep_Run(t *testing.T) { }, { name: "Hosts without allowed projects are accessible", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-x", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-x", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host5"}, }, @@ -157,13 +139,9 @@ func TestFilterAllowedProjectsStep_Run(t *testing.T) { }, { name: "Mixed hosts - some with empty allowed projects, some with matching project", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-b", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-b", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -175,13 +153,9 @@ func TestFilterAllowedProjectsStep_Run(t *testing.T) { }, { name: "All hosts filtered out", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-nonexistent", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-nonexistent", + Hosts: []compute.Host{ {ComputeHost: "host2"}, {ComputeHost: "host3"}, {ComputeHost: "host4"}, @@ -192,26 +166,18 @@ func TestFilterAllowedProjectsStep_Run(t *testing.T) { }, { name: "Empty host list", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-a", - }, - }, - Hosts: []api.ExternalSchedulerHost{}, + request: compute.PlacementRequest{ + ProjectID: "project-a", + Hosts: []compute.Host{}, }, expectedHosts: []string{}, filteredHosts: []string{}, }, { name: "Project matches on multiple hosts with allowed projects", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-a", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-a", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -224,13 +190,9 @@ func TestFilterAllowedProjectsStep_Run(t *testing.T) { }, { name: "Project matches second item in allowed projects list", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-b", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-b", + Hosts: []compute.Host{ {ComputeHost: "host2"}, }, }, @@ -239,13 +201,9 @@ func TestFilterAllowedProjectsStep_Run(t *testing.T) { }, { name: "All hosts have empty allowed projects", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "any-project", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "any-project", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host5"}, }, @@ -255,13 +213,9 @@ func TestFilterAllowedProjectsStep_Run(t *testing.T) { }, { name: "Mixed allowed projects configuration", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-c", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "project-c", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -274,13 +228,9 @@ func TestFilterAllowedProjectsStep_Run(t *testing.T) { }, { name: "Case sensitive project matching", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "Project-A", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ProjectID: "Project-A", + Hosts: []compute.Host{ {ComputeHost: "host2"}, {ComputeHost: "host3"}, }, @@ -340,14 +290,10 @@ func TestFilterAllowedProjectsStep_SkipsForCapacityProbeIntent(t *testing.T) { step := &FilterAllowedProjectsStep{} step.Client = fake.NewClientBuilder().WithScheme(scheme).WithObjects(objects...).Build() - request := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-y", - SchedulerHints: map[string]any{"_nova_check_type": "capacity_probe"}, - }, - }, - Hosts: []api.ExternalSchedulerHost{{ComputeHost: "host1"}, {ComputeHost: "host2"}}, + request := compute.PlacementRequest{ + ProjectID: "project-y", + Intent: api.CapacityProbeIntent, + Hosts: []compute.Host{{ComputeHost: "host1"}, {ComputeHost: "host2"}}, } result, err := step.Run(logr.Discard(), request) if err != nil { @@ -372,18 +318,14 @@ func TestFilterAllowedProjectsStep_AppliesForFailoverIntents(t *testing.T) { }, } - for _, intent := range []string{"reserve_for_failover", "reuse_failover_reservation"} { - t.Run(intent, func(t *testing.T) { + for _, intent := range []v1alpha1.SchedulingIntent{api.ReserveForFailoverIntent, api.ReuseFailoverReservationIntent} { + t.Run(string(intent), func(t *testing.T) { step := &FilterAllowedProjectsStep{} step.Client = fake.NewClientBuilder().WithScheme(scheme).WithObjects(objects...).Build() - request := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-y", - SchedulerHints: map[string]any{"_nova_check_type": intent}, - }, - }, - Hosts: []api.ExternalSchedulerHost{{ComputeHost: "host1"}, {ComputeHost: "host2"}}, + request := compute.PlacementRequest{ + ProjectID: "project-y", + Intent: intent, + Hosts: []compute.Host{{ComputeHost: "host1"}, {ComputeHost: "host2"}}, } result, err := step.Run(logr.Discard(), request) if err != nil { diff --git a/internal/scheduling/nova/plugins/filters/filter_capabilities.go b/internal/scheduling/nova/plugins/filters/filter_capabilities.go index 31277eeaa..392ef4d06 100644 --- a/internal/scheduling/nova/plugins/filters/filter_capabilities.go +++ b/internal/scheduling/nova/plugins/filters/filter_capabilities.go @@ -7,15 +7,14 @@ import ( "context" "fmt" "github.com/go-logr/logr" - "strings" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" ) type FilterCapabilitiesStep struct { - lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts] + lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts] } // Get the provided capabilities of a hypervisor resource in the format Nova expects. @@ -45,46 +44,14 @@ func hvToNovaCapabilities(hv hv1.Hypervisor) (map[string]string, error) { // Check the capabilities of each host and if they match the extra spec provided // in the request spec flavor. -func (s *FilterCapabilitiesStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *FilterCapabilitiesStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) - extraSpecs := request.Spec.Data.Flavor.Data.ExtraSpecs - if len(extraSpecs) == 0 { - traceLog.V(1).Info("no flavor extra spec capabilities in request, skipping filter") - return result, nil - } - - // Extract only capability-related extra specs into a separate map. - // We must not modify the original ExtraSpecs map as it's shared across filters. - requestedCapabilities := make(map[string]string) - for key, value := range extraSpecs { - if strings.HasPrefix(key, "capabilities:") { - requestedCapabilities[key] = value - } - } + requestedCapabilities := request.RequestedCapabilities if len(requestedCapabilities) == 0 { - traceLog.V(1).Info("no capabilities in flavor extra specs, skipping filter") + traceLog.V(1).Info("no capabilities requested, skipping filter") return result, nil } - // Note: currently none of the advanced operators for capabilities are - // supported because they are not used by any of our flavors in production. - // Ops: https://github.com/sapcc/nova/blob/3ebf80/nova/scheduler/filters/extra_specs_ops.py#L23 - unsupportedOps := []string{ - "=", "", "", "==", "!=", ">=", "<=", - "s==", "s!=", "s<", "s<=", "s>", "s>=", "", // or is special - } - for key, expr := range requestedCapabilities { - for _, op := range unsupportedOps { - if strings.Contains(expr, op) { - traceLog.V(1).Info( - "unsupported extra spec operator in capabilities filter, skipping filter", - "key", key, "expr", expr, "flavor", request.Spec.Data.Flavor, - ) - return result, nil - } - } - } - hvs := &hv1.HypervisorList{} if err := s.Client.List(context.Background(), hvs); err != nil { traceLog.Error(err, "failed to list hypervisors") @@ -118,11 +85,12 @@ func (s *FilterCapabilitiesStep) Run(traceLog logr.Logger, request api.ExternalS continue } // Check if the provided capabilities match the requested ones. - for keyRequested, valueRequested := range requestedCapabilities { - if providedValue, ok := provided[keyRequested]; !ok || providedValue != valueRequested { + for _, requested := range requestedCapabilities { + key := "capabilities:" + requested.Name + if providedValue, ok := provided[key]; !ok || providedValue != requested.Value { traceLog.V(1).Info( "filtering host with mismatched capabilities", "host", host, - "wantKey", keyRequested, "wantValue", valueRequested, + "wantKey", key, "wantValue", requested.Value, "haveKey?", ok, "haveValue", providedValue, ) delete(result.Activations, host) diff --git a/internal/scheduling/nova/plugins/filters/filter_capabilities_test.go b/internal/scheduling/nova/plugins/filters/filter_capabilities_test.go index 622932bb8..669e71e1c 100644 --- a/internal/scheduling/nova/plugins/filters/filter_capabilities_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_capabilities_test.go @@ -8,6 +8,7 @@ import ( "testing" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -225,24 +226,15 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedHosts []string filteredHosts []string expectedEvents []lib.FilterWeigherPipelineStepEvent }{ { name: "No extra specs in request - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -253,20 +245,8 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "Non-capability extra specs - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "hw:mem_page_size": "large", - "hw:cpu_policy": "dedicated", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -276,19 +256,11 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "Match CH hypervisor type", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "CH", - }, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedCapabilities: []api.Capability{ + {Name: "hypervisor_type", Value: "CH"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -299,19 +271,11 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "Match QEMU hypervisor type", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "QEMU", - }, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedCapabilities: []api.Capability{ + {Name: "hypervisor_type", Value: "QEMU"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -323,19 +287,11 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "Match x86_64 CPU architecture", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:cpu_arch": "x86_64", - }, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedCapabilities: []api.Capability{ + {Name: "cpu_arch", Value: "x86_64"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -347,19 +303,11 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "Match aarch64 CPU architecture", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:cpu_arch": "aarch64", - }, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedCapabilities: []api.Capability{ + {Name: "cpu_arch", Value: "aarch64"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -371,20 +319,12 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "Match both hypervisor type and CPU architecture", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "CH", - "capabilities:cpu_arch": "x86_64", - }, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedCapabilities: []api.Capability{ + {Name: "hypervisor_type", Value: "CH"}, + {Name: "cpu_arch", Value: "x86_64"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -396,20 +336,12 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "Match QEMU with aarch64", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "QEMU", - "capabilities:cpu_arch": "aarch64", - }, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedCapabilities: []api.Capability{ + {Name: "hypervisor_type", Value: "QEMU"}, + {Name: "cpu_arch", Value: "aarch64"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -421,19 +353,11 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "Ignore hypervisors not in request even with unknown type", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "CH", - }, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedCapabilities: []api.Capability{ + {Name: "hypervisor_type", Value: "CH"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, }, @@ -443,19 +367,11 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "Candidate hypervisor with empty hypervisor type emits event and is filtered out", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "CH", - }, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedCapabilities: []api.Capability{ + {Name: "hypervisor_type", Value: "CH"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host-empty-type"}, }, }, @@ -467,19 +383,11 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "Candidate hypervisor with unknown hypervisor type emits event and is filtered out", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "CH", - }, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedCapabilities: []api.Capability{ + {Name: "hypervisor_type", Value: "CH"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host-unknown-type"}, }, }, @@ -491,19 +399,11 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "No matching hosts", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "KVM", - }, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedCapabilities: []api.Capability{ + {Name: "hypervisor_type", Value: "KVM"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -513,21 +413,12 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "Mixed capability and non-capability extra specs", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "CH", - "hw:mem_page_size": "large", - "capabilities:cpu_arch": "x86_64", - }, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedCapabilities: []api.Capability{ + {Name: "hypervisor_type", Value: "CH"}, + {Name: "cpu_arch", Value: "x86_64"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -538,38 +429,22 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "Empty host list", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "CH", - }, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedCapabilities: []api.Capability{ + {Name: "hypervisor_type", Value: "CH"}, }, - Hosts: []api.ExternalSchedulerHost{}, + Hosts: []compute.Host{}, }, expectedHosts: []string{}, filteredHosts: []string{}, }, { name: "Case sensitive capability matching", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "ch", // lowercase should not match - }, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedCapabilities: []api.Capability{ + {Name: "hypervisor_type", Value: "ch"}, // lowercase should not match }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -579,19 +454,11 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "Unsupported operator in extra specs - skip filter", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:cpu_arch": "s>=x86_64", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + // An unsupported comparison operator is dropped by the adapter, + // so the filter sees no requested capabilities and is a no-op. + RequestedCapabilities: nil, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -602,19 +469,11 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "Unsupported operator - skip filter", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": " CH QEMU", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + // An unsupported comparison operator is dropped by the adapter, + // so the filter sees no requested capabilities and is a no-op. + RequestedCapabilities: nil, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -624,17 +483,8 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }, { name: "All hosts match when no capabilities requested", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -701,78 +551,3 @@ func TestFilterCapabilitiesStep_Run(t *testing.T) { }) } } - -// TestFilterCapabilitiesStep_DoesNotMutateExtraSpecs verifies that the filter -// does not modify the original ExtraSpecs map from the request. This is important -// because subsequent filters (like filter_has_requested_traits) also need to access -// the full ExtraSpecs including non-capability keys like trait:*. -func TestFilterCapabilitiesStep_DoesNotMutateExtraSpecs(t *testing.T) { - scheme := runtime.NewScheme() - err := hv1.AddToScheme(scheme) - if err != nil { - t.Fatalf("expected no error, got %v", err) - } - - hvs := []client.Object{ - &hv1.Hypervisor{ - ObjectMeta: v1.ObjectMeta{Name: "host1"}, - Status: hv1.HypervisorStatus{ - DomainCapabilities: hv1.DomainCapabilities{HypervisorType: "ch"}, - Capabilities: hv1.Capabilities{HostCpuArch: "x86_64"}, - }, - }, - } - - // Create a request with mixed extra specs (capabilities + traits + other) - originalExtraSpecs := map[string]string{ - "capabilities:hypervisor_type": "CH", - "trait:CUSTOM_HANA_EXCLUSIVE_HOST": "forbidden", - "trait:CUSTOM_HW_SAPPHIRE_RAPIDS": "required", - "hw:mem_page_size": "large", - "hw:numa_nodes": "1", - } - - request := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: originalExtraSpecs, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{{ComputeHost: "host1"}}, - } - - step := &FilterCapabilitiesStep{} - step.Client = fake.NewClientBuilder(). - WithScheme(scheme). - WithObjects(hvs...). - Build() - - _, err = step.Run(logr.Discard(), request) - if err != nil { - t.Fatalf("expected no error, got %v", err) - } - - // Verify that the original ExtraSpecs map was NOT mutated - expectedKeys := []string{ - "capabilities:hypervisor_type", - "trait:CUSTOM_HANA_EXCLUSIVE_HOST", - "trait:CUSTOM_HW_SAPPHIRE_RAPIDS", - "hw:mem_page_size", - "hw:numa_nodes", - } - - if len(request.Spec.Data.Flavor.Data.ExtraSpecs) != len(expectedKeys) { - t.Errorf("ExtraSpecs was mutated: expected %d keys, got %d", - len(expectedKeys), len(request.Spec.Data.Flavor.Data.ExtraSpecs)) - } - - for _, key := range expectedKeys { - if _, ok := request.Spec.Data.Flavor.Data.ExtraSpecs[key]; !ok { - t.Errorf("ExtraSpecs was mutated: key %q was deleted", key) - } - } -} diff --git a/internal/scheduling/nova/plugins/filters/filter_correct_az.go b/internal/scheduling/nova/plugins/filters/filter_correct_az.go index 882cf261c..730ca968d 100644 --- a/internal/scheduling/nova/plugins/filters/filter_correct_az.go +++ b/internal/scheduling/nova/plugins/filters/filter_correct_az.go @@ -7,20 +7,20 @@ import ( "context" "github.com/go-logr/logr" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" corev1 "k8s.io/api/core/v1" ) type FilterCorrectAZStep struct { - lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts] + lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts] } // Only get hosts in the requested az. -func (s *FilterCorrectAZStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *FilterCorrectAZStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) - if request.Spec.Data.AvailabilityZone == "" { + if request.AvailabilityZone == "" { traceLog.V(1).Info("no availability zone requested, skipping filter_correct_az step") return result, nil } @@ -42,7 +42,7 @@ func (s *FilterCorrectAZStep) Run(traceLog logr.Logger, request api.ExternalSche traceLog.V(1).Info("host missing zone label, keeping", "host", hv.Name) continue } - if az == request.Spec.Data.AvailabilityZone { + if az == request.AvailabilityZone { // We always assume the name of the resource corresponds // to the compute host name. computeHostsInAZ[hv.Name] = struct{}{} @@ -51,7 +51,7 @@ func (s *FilterCorrectAZStep) Run(traceLog logr.Logger, request api.ExternalSche traceLog.V(1).Info( "hosts inside requested az", - "availabilityZone", request.Spec.Data.AvailabilityZone, + "availabilityZone", request.AvailabilityZone, "hosts", computeHostsInAZ, ) for host := range result.Activations { diff --git a/internal/scheduling/nova/plugins/filters/filter_correct_az_test.go b/internal/scheduling/nova/plugins/filters/filter_correct_az_test.go index 135766806..8901fa81d 100644 --- a/internal/scheduling/nova/plugins/filters/filter_correct_az_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_correct_az_test.go @@ -7,7 +7,7 @@ import ( "github.com/go-logr/logr" "testing" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -57,19 +57,15 @@ func TestFilterCorrectAZStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedHosts []string filteredHosts []string }{ { name: "Filter hosts in az-1", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - AvailabilityZone: "az-1", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + AvailabilityZone: "az-1", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -82,13 +78,9 @@ func TestFilterCorrectAZStep_Run(t *testing.T) { }, { name: "Filter hosts in az-2", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - AvailabilityZone: "az-2", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + AvailabilityZone: "az-2", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -100,13 +92,9 @@ func TestFilterCorrectAZStep_Run(t *testing.T) { }, { name: "Filter hosts in az-3", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - AvailabilityZone: "az-3", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + AvailabilityZone: "az-3", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -118,13 +106,9 @@ func TestFilterCorrectAZStep_Run(t *testing.T) { }, { name: "No hosts in requested AZ", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - AvailabilityZone: "az-nonexistent", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + AvailabilityZone: "az-nonexistent", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -135,26 +119,18 @@ func TestFilterCorrectAZStep_Run(t *testing.T) { }, { name: "Empty host list", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - AvailabilityZone: "az-1", - }, - }, - Hosts: []api.ExternalSchedulerHost{}, + request: compute.PlacementRequest{ + AvailabilityZone: "az-1", + Hosts: []compute.Host{}, }, expectedHosts: []string{}, filteredHosts: []string{}, }, { name: "Host not in database", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - AvailabilityZone: "az-1", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + AvailabilityZone: "az-1", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host-unknown"}, }, diff --git a/internal/scheduling/nova/plugins/filters/filter_exclude_hosts.go b/internal/scheduling/nova/plugins/filters/filter_exclude_hosts.go index c35a63cff..de1241fa8 100644 --- a/internal/scheduling/nova/plugins/filters/filter_exclude_hosts.go +++ b/internal/scheduling/nova/plugins/filters/filter_exclude_hosts.go @@ -6,7 +6,7 @@ package filters import ( "github.com/go-logr/logr" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" ) @@ -14,7 +14,7 @@ import ( // This step is useful to circumvent scheduling issues on specific hosts by // excluding them from all scheduling decisions. type FilterExcludeHostsStep struct { - lib.BaseFilter[api.ExternalSchedulerRequest, FilterExcludeHostsStepOpts] + lib.BaseFilter[compute.PlacementRequest, FilterExcludeHostsStepOpts] } type FilterExcludeHostsStepOpts struct { @@ -29,7 +29,7 @@ func (opts FilterExcludeHostsStepOpts) Validate() error { return nil } func (s *FilterExcludeHostsStep) Run( traceLog logr.Logger, - request api.ExternalSchedulerRequest, + request compute.PlacementRequest, ) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) diff --git a/internal/scheduling/nova/plugins/filters/filter_exclude_hosts_test.go b/internal/scheduling/nova/plugins/filters/filter_exclude_hosts_test.go index 45676b731..6a4256f4b 100644 --- a/internal/scheduling/nova/plugins/filters/filter_exclude_hosts_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_exclude_hosts_test.go @@ -7,7 +7,7 @@ import ( "github.com/go-logr/logr" "testing" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" ) @@ -15,15 +15,15 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { tests := []struct { name string excludedHosts []string - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedHosts []string filteredHosts []string }{ { name: "No excluded hosts - all hosts pass", excludedHosts: []string{}, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -35,8 +35,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { { name: "Single excluded host", excludedHosts: []string{"host2"}, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -48,8 +48,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { { name: "Multiple excluded hosts", excludedHosts: []string{"host1", "host3"}, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -62,8 +62,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { { name: "All hosts excluded", excludedHosts: []string{"host1", "host2", "host3"}, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -75,8 +75,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { { name: "Excluded host not in request - no effect", excludedHosts: []string{"host-nonexistent"}, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -87,8 +87,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { { name: "Some excluded hosts in request, some not", excludedHosts: []string{"host1", "host-nonexistent", "host3"}, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -100,8 +100,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { { name: "Empty host list in request", excludedHosts: []string{"host1", "host2"}, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{}, + request: compute.PlacementRequest{ + Hosts: []compute.Host{}, }, expectedHosts: []string{}, filteredHosts: []string{}, @@ -109,8 +109,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { { name: "Nil excluded hosts list", excludedHosts: nil, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -121,8 +121,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { { name: "Duplicate excluded hosts", excludedHosts: []string{"host1", "host1", "host2"}, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -134,8 +134,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { { name: "Single host request - excluded", excludedHosts: []string{"host1"}, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, }, @@ -145,8 +145,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { { name: "Single host request - not excluded", excludedHosts: []string{"host2"}, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, }, @@ -156,8 +156,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { { name: "Case sensitive matching", excludedHosts: []string{"Host1", "HOST2"}, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "Host1"}, @@ -169,8 +169,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { { name: "Hosts with special characters", excludedHosts: []string{"nova-compute-bb123.region.example.com"}, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "nova-compute-bb123.region.example.com"}, {ComputeHost: "nova-compute-bb124.region.example.com"}, }, @@ -181,8 +181,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { { name: "Empty string in excluded hosts", excludedHosts: []string{"", "host1"}, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -193,8 +193,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { { name: "Large number of excluded hosts", excludedHosts: []string{"host1", "host2", "host3", "host4", "host5", "host6", "host7", "host8", "host9", "host10"}, - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host5"}, {ComputeHost: "host10"}, @@ -210,8 +210,8 @@ func TestFilterExcludeHostsStep_Run(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { step := &FilterExcludeHostsStep{ - BaseFilter: lib.BaseFilter[api.ExternalSchedulerRequest, FilterExcludeHostsStepOpts]{ - BaseFilterWeigherPipelineStep: lib.BaseFilterWeigherPipelineStep[api.ExternalSchedulerRequest, FilterExcludeHostsStepOpts]{}, + BaseFilter: lib.BaseFilter[compute.PlacementRequest, FilterExcludeHostsStepOpts]{ + BaseFilterWeigherPipelineStep: lib.BaseFilterWeigherPipelineStep[compute.PlacementRequest, FilterExcludeHostsStepOpts]{}, }, } step.Options = FilterExcludeHostsStepOpts{ diff --git a/internal/scheduling/nova/plugins/filters/filter_external_customer.go b/internal/scheduling/nova/plugins/filters/filter_external_customer.go index 19a412797..e724e6c4d 100644 --- a/internal/scheduling/nova/plugins/filters/filter_external_customer.go +++ b/internal/scheduling/nova/plugins/filters/filter_external_customer.go @@ -12,6 +12,7 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" ) @@ -29,29 +30,29 @@ func (opts FilterExternalCustomerStepOpts) Validate() error { } type FilterExternalCustomerStep struct { - lib.BaseFilter[api.ExternalSchedulerRequest, FilterExternalCustomerStepOpts] + lib.BaseFilter[compute.PlacementRequest, FilterExternalCustomerStepOpts] } // Prefix-match the domain name for external customer domains and filter out hosts // that are not intended for external customers. -func (s *FilterExternalCustomerStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *FilterExternalCustomerStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) // Failover and capacity probe calls are not associated with a user domain; // domain-based host restrictions don't apply. CR scheduling does carry a real domain // and must be checked. - if intent, err := request.GetIntent(); err == nil && slices.Contains([]v1alpha1.SchedulingIntent{ + if slices.Contains([]v1alpha1.SchedulingIntent{ api.ReserveForFailoverIntent, api.ReuseFailoverReservationIntent, api.CapacityProbeIntent, - }, intent) { - traceLog.V(1).Info("skipping external customer filter for non-placement intent", "intent", intent) + }, request.Intent) { + traceLog.V(1).Info("skipping external customer filter for non-placement intent", "intent", request.Intent) return result, nil } - domainName, err := request.Spec.Data.GetSchedulerHintStr("domain_name") - if err != nil { - traceLog.Error(err, "failed to get domain_name scheduler hint, skipping filter") + domainName := request.DomainName + if domainName == "" { + traceLog.V(1).Info("no domain name in request, skipping filter") return result, nil } if slices.Contains(s.Options.CustomerIgnoredDomainNames, domainName) { diff --git a/internal/scheduling/nova/plugins/filters/filter_external_customer_test.go b/internal/scheduling/nova/plugins/filters/filter_external_customer_test.go index 4d2ebe784..fe765695e 100644 --- a/internal/scheduling/nova/plugins/filters/filter_external_customer_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_external_customer_test.go @@ -8,6 +8,8 @@ import ( "testing" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -77,7 +79,7 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { tests := []struct { name string opts FilterExternalCustomerStepOpts - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedHosts []string filteredHosts []string expectError bool @@ -87,15 +89,9 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"ext-"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "domain_name": "ext-customer1", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + DomainName: "ext-customer1", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -110,15 +106,9 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"ext-"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "domain_name": "internal-customer", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + DomainName: "internal-customer", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -133,15 +123,9 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"external-", "ext-", "customer-"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "domain_name": "customer-abc", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + DomainName: "customer-abc", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host3"}, {ComputeHost: "host4"}, @@ -156,15 +140,9 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { CustomerDomainNamePrefixes: []string{"ext-"}, CustomerIgnoredDomainNames: []string{"ext-special-domain"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "domain_name": "ext-special-domain", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + DomainName: "ext-special-domain", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -178,15 +156,9 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"ext-"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "domain_name": "ext-customer2", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + DomainName: "ext-customer2", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host5"}, @@ -200,15 +172,9 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"ext-"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "domain_name": "ext-customer3", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + DomainName: "ext-customer3", + Hosts: []compute.Host{ {ComputeHost: "host3"}, {ComputeHost: "host4"}, }, @@ -221,15 +187,9 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"ext-"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "domain_name": "ext-customer", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{}, + request: compute.PlacementRequest{ + DomainName: "ext-customer", + Hosts: []compute.Host{}, }, expectedHosts: []string{}, filteredHosts: []string{}, @@ -239,15 +199,9 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"ext-"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "domain_name": []any{"ext-customer", "other"}, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + DomainName: "ext-customer", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host3"}, }, @@ -260,13 +214,8 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"ext-"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{}, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host3"}, }, @@ -279,13 +228,8 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"ext-"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: nil, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -298,15 +242,9 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"ext-"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "domain_name": "EXT-customer", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + DomainName: "EXT-customer", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -320,15 +258,9 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"ext"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "domain_name": "ext", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + DomainName: "ext", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host3"}, }, @@ -342,15 +274,9 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { CustomerDomainNamePrefixes: []string{"ext-"}, CustomerIgnoredDomainNames: []string{"ext-test", "ext-dev", "ext-staging"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "domain_name": "ext-dev", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + DomainName: "ext-dev", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host3"}, }, @@ -363,16 +289,10 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"ext-"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": string(api.ReserveForFailoverIntent), - "domain_name": "ext-customer1", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Intent: api.ReserveForFailoverIntent, + DomainName: "ext-customer1", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host3"}, }, @@ -385,15 +305,9 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"ext-"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": string(api.ReserveForCommittedResourceIntent), - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Intent: api.ReserveForCommittedResourceIntent, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host3"}, }, @@ -406,16 +320,10 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"ext-"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": string(api.ReserveForCommittedResourceIntent), - "domain_name": "ext-customer1", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Intent: api.ReserveForCommittedResourceIntent, + DomainName: "ext-customer1", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host3"}, {ComputeHost: "host4"}, @@ -429,15 +337,9 @@ func TestFilterExternalCustomerStep_Run(t *testing.T) { opts: FilterExternalCustomerStepOpts{ CustomerDomainNamePrefixes: []string{"ext-"}, }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "domain_name": "ext-customer1", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + DomainName: "ext-customer1", + Hosts: []compute.Host{ {ComputeHost: "host-custom-trait"}, {ComputeHost: "host4"}, }, @@ -565,16 +467,10 @@ func TestFilterExternalCustomerStep_SkipsForNonPlacementIntent(t *testing.T) { for _, intent := range []string{"reserve_for_failover", "reuse_failover_reservation", "capacity_probe"} { t.Run(intent, func(t *testing.T) { - request := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "domain_name": "iaas-customer", - "_nova_check_type": intent, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{{ComputeHost: "host1"}, {ComputeHost: "host2"}}, + request := compute.PlacementRequest{ + Intent: v1alpha1.SchedulingIntent(intent), + DomainName: "iaas-customer", + Hosts: []compute.Host{{ComputeHost: "host1"}, {ComputeHost: "host2"}}, } result, err := step.Run(logr.Discard(), request) if err != nil { diff --git a/internal/scheduling/nova/plugins/filters/filter_has_accelerators.go b/internal/scheduling/nova/plugins/filters/filter_has_accelerators.go index ad3c20288..8b27f50f1 100644 --- a/internal/scheduling/nova/plugins/filters/filter_has_accelerators.go +++ b/internal/scheduling/nova/plugins/filters/filter_has_accelerators.go @@ -8,20 +8,19 @@ import ( "github.com/go-logr/logr" "slices" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" ) type FilterHasAcceleratorsStep struct { - lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts] + lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts] } // If requested, only get hosts with accelerators. -func (s *FilterHasAcceleratorsStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *FilterHasAcceleratorsStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) - extraSpecs := request.Spec.Data.Flavor.Data.ExtraSpecs - if _, ok := extraSpecs["accel:device_profile"]; !ok { + if !request.AcceleratorsRequested { traceLog.V(1).Info("no accelerators requested, skipping filter") return result, nil } diff --git a/internal/scheduling/nova/plugins/filters/filter_has_accelerators_test.go b/internal/scheduling/nova/plugins/filters/filter_has_accelerators_test.go index 2c92cf81b..57bef26fc 100644 --- a/internal/scheduling/nova/plugins/filters/filter_has_accelerators_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_has_accelerators_test.go @@ -7,7 +7,7 @@ import ( "github.com/go-logr/logr" "testing" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -67,23 +67,14 @@ func TestFilterHasAcceleratorsStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedHosts []string filteredHosts []string }{ { name: "No accelerators requested - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -95,19 +86,9 @@ func TestFilterHasAcceleratorsStep_Run(t *testing.T) { }, { name: "Accelerators requested - filter to hosts with COMPUTE_ACCELERATORS trait", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "accel:device_profile": "gpu-profile", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + AcceleratorsRequested: true, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -119,19 +100,9 @@ func TestFilterHasAcceleratorsStep_Run(t *testing.T) { }, { name: "Accelerators requested with different device profile value", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "accel:device_profile": "fpga-profile", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + AcceleratorsRequested: true, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host3"}, {ComputeHost: "host5"}, @@ -142,19 +113,9 @@ func TestFilterHasAcceleratorsStep_Run(t *testing.T) { }, { name: "Accelerators requested - all hosts have the trait", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "accel:device_profile": "gpu-profile", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + AcceleratorsRequested: true, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host5"}, @@ -165,19 +126,9 @@ func TestFilterHasAcceleratorsStep_Run(t *testing.T) { }, { name: "Accelerators requested - no hosts have the trait", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "accel:device_profile": "gpu-profile", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + AcceleratorsRequested: true, + Hosts: []compute.Host{ {ComputeHost: "host3"}, {ComputeHost: "host4"}, }, @@ -187,20 +138,8 @@ func TestFilterHasAcceleratorsStep_Run(t *testing.T) { }, { name: "Other extra specs present but no accelerator request - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "hw:cpu_policy": "dedicated", - "quota:vif_inbound": "100000", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host3"}, {ComputeHost: "host4"}, @@ -211,19 +150,9 @@ func TestFilterHasAcceleratorsStep_Run(t *testing.T) { }, { name: "Accelerators requested with empty device profile value", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "accel:device_profile": "", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + AcceleratorsRequested: true, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -234,55 +163,26 @@ func TestFilterHasAcceleratorsStep_Run(t *testing.T) { }, { name: "Empty host list with accelerators requested", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "accel:device_profile": "gpu-profile", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{}, + request: compute.PlacementRequest{ + AcceleratorsRequested: true, + Hosts: []compute.Host{}, }, expectedHosts: []string{}, filteredHosts: []string{}, }, { name: "Empty host list without accelerators requested", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{}, + request: compute.PlacementRequest{ + Hosts: []compute.Host{}, }, expectedHosts: []string{}, filteredHosts: []string{}, }, { name: "Mixed hosts with and without accelerators trait", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "accel:device_profile": "custom-accelerator", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + AcceleratorsRequested: true, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -295,21 +195,9 @@ func TestFilterHasAcceleratorsStep_Run(t *testing.T) { }, { name: "Accelerators with additional extra specs", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "accel:device_profile": "gpu-profile", - "hw:cpu_policy": "dedicated", - "hw:mem_page_size": "large", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + AcceleratorsRequested: true, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host3"}, }, @@ -319,19 +207,9 @@ func TestFilterHasAcceleratorsStep_Run(t *testing.T) { }, { name: "Host not in database with accelerators requested", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "accel:device_profile": "gpu-profile", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + AcceleratorsRequested: true, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host-unknown"}, }, diff --git a/internal/scheduling/nova/plugins/filters/filter_has_enough_capacity.go b/internal/scheduling/nova/plugins/filters/filter_has_enough_capacity.go index 6400d5362..345c59b90 100644 --- a/internal/scheduling/nova/plugins/filters/filter_has_enough_capacity.go +++ b/internal/scheduling/nova/plugins/filters/filter_has_enough_capacity.go @@ -12,8 +12,9 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" - resv "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" + resv "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/accounting" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "k8s.io/apimachinery/pkg/api/resource" ) @@ -36,7 +37,7 @@ type FilterHasEnoughCapacityOpts struct { func (FilterHasEnoughCapacityOpts) Validate() error { return nil } type FilterHasEnoughCapacity struct { - lib.BaseFilter[api.ExternalSchedulerRequest, FilterHasEnoughCapacityOpts] + lib.BaseFilter[compute.PlacementRequest, FilterHasEnoughCapacityOpts] } // Filter hosts that don't have enough capacity to run the requested flavor. @@ -63,7 +64,7 @@ type FilterHasEnoughCapacity struct { // known at this point. // // Please also note that disk space is currently not considered by this filter. -func (s *FilterHasEnoughCapacity) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *FilterHasEnoughCapacity) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { opts := request.GetOptions() result := s.IncludeAllHostsFromRequest(request) @@ -157,22 +158,21 @@ func (s *FilterHasEnoughCapacity) Run(traceLog logr.Logger, request api.External // Check if this is a CR reservation scheduling request. // If so, we should NOT unlock any CR reservations to prevent overbooking. // CR capacity should only be unlocked for actual VM scheduling. - intent, err := request.GetIntent() switch { - case err == nil && intent == api.ReserveForCommittedResourceIntent: + case request.Intent == api.ReserveForCommittedResourceIntent: traceLog.V(1).Info("keeping CR reservation locked for CR reservation scheduling", "reservation", reservation.Name, - "intent", intent) + "intent", request.Intent) // Don't continue - fall through to block the resources case !s.Options.LockReserved && !opts.LockReservations && // For committed resource reservations: unlock resources only if: // 1. Project ID matches // 2. ResourceGroup matches the flavor's hw_version - reservation.Spec.CommittedResourceReservation.MatchesGroup(request.Spec.Data.ProjectID, request.Spec.Data.Flavor.Data.ExtraSpecs["quota:hw_version"]): + reservation.Spec.CommittedResourceReservation.MatchesGroup(request.ProjectID, request.HardwareVersion): traceLog.V(1).Info("unlocking resources reserved by matching committed resource reservation with allocation", "reservation", reservation.Name, - "instanceUUID", request.Spec.Data.InstanceUUID, - "projectID", request.Spec.Data.ProjectID, + "instanceUUID", request.InstanceUUID, + "projectID", request.ProjectID, "resourceGroup", reservation.Spec.CommittedResourceReservation.ResourceGroup) continue } @@ -184,25 +184,22 @@ func (s *FilterHasEnoughCapacity) Run(traceLog logr.Logger, request api.External // 1. Failover reservations are specifically for HA/evacuation scenarios. // 2. During live migrations or other operations, we don't want to use failover capacity. // Note: we cannot use failover reservations from other VMs, as that can invalidate our HA guarantees. - intent, err := request.GetIntent() - if err == nil { - switch intent { - case api.EvacuateIntent: - if reservation.Status.FailoverReservation != nil { - if _, contained := reservation.Status.FailoverReservation.Allocations[request.Spec.Data.InstanceUUID]; contained { - traceLog.V(1).Info("unlocking resources reserved by failover reservation for VM in allocations (evacuation)", - "reservation", reservation.Name, - "instanceUUID", request.Spec.Data.InstanceUUID) - continue - } + switch request.Intent { + case api.EvacuateIntent: + if reservation.Status.FailoverReservation != nil { + if _, contained := reservation.Status.FailoverReservation.Allocations[request.InstanceUUID]; contained { + traceLog.V(1).Info("unlocking resources reserved by failover reservation for VM in allocations (evacuation)", + "reservation", reservation.Name, + "instanceUUID", request.InstanceUUID) + continue } - case api.ReuseFailoverReservationIntent: - // Reuse check: the reservation already pre-blocks the right capacity for this VM. - // Don't subtract it from free capacity to avoid double-counting. - traceLog.V(1).Info("skipping failover reservation block for reuse compatibility check", - "reservation", reservation.Name) - continue } + case api.ReuseFailoverReservationIntent: + // Reuse check: the reservation already pre-blocks the right capacity for this VM. + // Don't subtract it from free capacity to avoid double-counting. + traceLog.V(1).Info("skipping failover reservation block for reuse compatibility check", + "reservation", reservation.Name) + continue } traceLog.V(1).Info("processing failover reservation", "reservation", reservation.Name) } @@ -224,16 +221,9 @@ func (s *FilterHasEnoughCapacity) Run(traceLog logr.Logger, request api.External continue } - // CommittedResourceReservations: compute the effective block: - // confirmed = sum of resources for VMs present in both Spec and Status allocations - // specOnly = sum of resources for VMs present in Spec but not yet in Status - // remaining = max(0, Spec.Resources - confirmed) [clamped: never negative] - // block = max(remaining, specOnly) [spec-only VM must be fully covered] - // - // Clamping: if confirmed VMs exceed slot size (e.g. after resize), block = 0. - // Oversize spec-only: if a pending VM is larger than the remaining slot, block its full size. - // - // FailoverReservations: block = Spec.Resources (always fully blocked). + // CommittedResourceReservations: block = max(slot − confirmed VM resources, 0). + // Confirmed VMs appear in hv.Status.Allocation; this subtracts only the unconfirmed remainder + // to avoid double-counting. FailoverReservations: block = Spec.Resources (always fully blocked). resourcesToBlock := resv.UnusedReservationCapacity(&reservation, ignoreAllocations) // Block the calculated resources on each host @@ -283,7 +273,7 @@ func (s *FilterHasEnoughCapacity) Run(traceLog logr.Logger, request api.External hostsEncountered[host] = struct{}{} // Check cpu capacity. - if request.Spec.Data.Flavor.Data.VCPUs == 0 { + if request.Flavor.VCPUs == 0 { return nil, errors.New("flavor has 0 vcpus") } freeCPU, ok := free["cpu"] @@ -298,11 +288,11 @@ func (s *FilterHasEnoughCapacity) Run(traceLog logr.Logger, request api.External // Calculate how many instances can fit on this host, based on cpu. //nolint:gosec // We're checking for underflows above (< 0). vcpuSlots := uint64(freeCPU.Value()) / - request.Spec.Data.Flavor.Data.VCPUs - if vcpuSlots < request.Spec.Data.NumInstances { + request.Flavor.VCPUs + if vcpuSlots < request.NumInstances { traceLog.V(1).Info( "filtering host due to insufficient CPU capacity", - "host", host, "requested", request.Spec.Data.Flavor.Data.VCPUs, + "host", host, "requested", request.Flavor.VCPUs, "available", freeCPU.String(), ) delete(result.Activations, host) @@ -310,7 +300,7 @@ func (s *FilterHasEnoughCapacity) Run(traceLog logr.Logger, request api.External } // Check memory capacity. - if request.Spec.Data.Flavor.Data.MemoryMB == 0 { + if request.Flavor.MemoryMB == 0 { return nil, errors.New("flavor has 0 memory") } freeMemory, ok := free["memory"] @@ -327,11 +317,11 @@ func (s *FilterHasEnoughCapacity) Run(traceLog logr.Logger, request api.External // are in mebibytes (1 MiB = 2^20 bytes). Multiply by 1<<20 to convert to bytes. //nolint:gosec // We're checking for underflows above (< 0). memorySlots := uint64(freeMemory.Value()) / - (request.Spec.Data.Flavor.Data.MemoryMB * (1 << 20)) - if memorySlots < request.Spec.Data.NumInstances { + (request.Flavor.MemoryMB * (1 << 20)) + if memorySlots < request.NumInstances { traceLog.V(1).Info( "filtering host due to insufficient RAM capacity", - "host", host, "requested_mb", request.Spec.Data.Flavor.Data.MemoryMB, + "host", host, "requested_mb", request.Flavor.MemoryMB, "available_mb", freeMemory.String(), ) delete(result.Activations, host) @@ -339,9 +329,9 @@ func (s *FilterHasEnoughCapacity) Run(traceLog logr.Logger, request api.External } traceLog.V(1).Info( "host has enough capacity", "host", host, - "requested_cpus", request.Spec.Data.Flavor.Data.VCPUs, + "requested_cpus", request.Flavor.VCPUs, "available_cpus", freeCPU.String(), - "requested_memory_mb", request.Spec.Data.Flavor.Data.MemoryMB, + "requested_memory_mb", request.Flavor.MemoryMB, "available_memory", freeMemory.String(), ) } diff --git a/internal/scheduling/nova/plugins/filters/filter_has_enough_capacity_test.go b/internal/scheduling/nova/plugins/filters/filter_has_enough_capacity_test.go index bdde34cca..7a3db55f4 100644 --- a/internal/scheduling/nova/plugins/filters/filter_has_enough_capacity_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_has_enough_capacity_test.go @@ -10,7 +10,9 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/scheduling" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -121,22 +123,31 @@ func newUnconfirmedReservation(name, targetHost, projectID, flavorGroup, cpu, me } } +// makeAlloc builds a CommittedResourceAllocation for test call sites. +func makeAlloc(name, cpu, memory string) v1alpha1.CommittedResourceAllocation { + return v1alpha1.CommittedResourceAllocation{ + ObjectRef: corev1.ObjectReference{Name: name}, + Resources: map[hv1.ResourceName]resource.Quantity{ + hv1.ResourceCPU: resource.MustParse(cpu), + hv1.ResourceMemory: resource.MustParse(memory), + }, + } +} + // newCommittedReservation creates a reservation where TargetHost and Status.Host are the same. func newCommittedReservation( name, host, projectID, flavorName, flavorGroup, cpu, memory string, - specAllocations map[string]v1alpha1.CommittedResourceAllocation, - statusAllocations map[string]string, + statusAllocations map[string]v1alpha1.CommittedResourceAllocation, ) *v1alpha1.Reservation { - return newMigratingReservation(name, host, host, projectID, flavorName, flavorGroup, cpu, memory, specAllocations, statusAllocations) + return newMigratingReservation(name, host, host, projectID, flavorName, flavorGroup, cpu, memory, statusAllocations) } // newMigratingReservation creates a reservation where TargetHost and Status.Host may differ, // used for in-progress reservation migrations or pending placements. func newMigratingReservation( name, targetHost, observedHost, projectID, flavorName, flavorGroup, cpu, memory string, - specAllocations map[string]v1alpha1.CommittedResourceAllocation, - statusAllocations map[string]string, + statusAllocations map[string]v1alpha1.CommittedResourceAllocation, ) *v1alpha1.Reservation { res := &v1alpha1.Reservation{ @@ -154,7 +165,6 @@ func newMigratingReservation( ProjectID: projectID, ResourceName: flavorName, ResourceGroup: flavorGroup, - Allocations: specAllocations, }, }, Status: v1alpha1.ReservationStatus{ @@ -171,7 +181,7 @@ func newMigratingReservation( if len(statusAllocations) > 0 { res.Status.CommittedResourceReservation = &v1alpha1.CommittedResourceReservationStatus{ - Allocations: statusAllocations, + Allocation: statusAllocations, } } @@ -213,34 +223,6 @@ func newFailoverReservation(name, targetHost, cpu, memory string, allocations ma return res } -type crVmAlloc struct { - uuid string - cpu string - mem string -} - -func crVm(uuid, cpu, mem string) crVmAlloc { - return crVmAlloc{ - uuid: uuid, - cpu: cpu, - mem: mem, - } -} - -func crSpecAllocs(vms ...crVmAlloc) map[string]v1alpha1.CommittedResourceAllocation { - allocs := make(map[string]v1alpha1.CommittedResourceAllocation) - for _, v := range vms { - allocs[v.uuid] = v1alpha1.CommittedResourceAllocation{ - CreationTimestamp: metav1.Now(), - Resources: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceCPU: resource.MustParse(v.cpu), - hv1.ResourceMemory: resource.MustParse(v.mem), - }, - } - } - return allocs -} - // parseMemoryToMB converts a memory string (e.g., "8Gi", "4096Mi") to megabytes. func parseMemoryToMB(memory string) uint64 { q := resource.MustParse(memory) @@ -248,58 +230,46 @@ func parseMemoryToMB(memory string) uint64 { return uint64(bytes / (1024 * 1024)) //nolint:gosec // test code } -func newNovaRequest(instanceUUID, projectID, flavorName, flavorGroup string, vcpus int, memory string, evacuation bool, hosts []string) api.ExternalSchedulerRequest { - return newNovaRequestWithIntent(instanceUUID, projectID, flavorName, flavorGroup, vcpus, memory, "", evacuation, hosts) +func newNovaRequest(instanceUUID, projectID, flavorGroup string, vcpus int, memory string, evacuation bool, hosts []string) compute.PlacementRequest { + return newNovaRequestWithIntent(instanceUUID, projectID, flavorGroup, vcpus, memory, "", evacuation, hosts) } // newNovaRequestWithIntent creates a nova request with a specific intent. // intentHint can be: "evacuate", "reserve_for_committed_resource", "reserve_for_failover", or "" for create. -func newNovaRequestWithIntent(instanceUUID, projectID, flavorName, flavorGroup string, vcpus int, memory, intentHint string, evacuation bool, hosts []string) api.ExternalSchedulerRequest { - hostList := make([]api.ExternalSchedulerHost, len(hosts)) +func newNovaRequestWithIntent(instanceUUID, projectID, flavorGroup string, vcpus int, memory, intentHint string, evacuation bool, hosts []string) compute.PlacementRequest { + hostList := make([]compute.Host, len(hosts)) for i, h := range hosts { - hostList[i] = api.ExternalSchedulerHost{ComputeHost: h} - } - - extraSpecs := map[string]string{ - "capabilities:hypervisor_type": "qemu", - "quota:hw_version": flavorGroup, + hostList[i] = compute.Host{ComputeHost: h} } - var schedulerHints map[string]any - if evacuation { - schedulerHints = map[string]any{ - "_nova_check_type": []any{"evacuate"}, - } - } else if intentHint != "" { - schedulerHints = map[string]any{ - "_nova_check_type": intentHint, - } + var intent v1alpha1.SchedulingIntent + switch { + case evacuation: + intent = api.EvacuateIntent + case intentHint == "reserve_for_committed_resource": + intent = api.ReserveForCommittedResourceIntent + case intentHint == "reserve_for_failover": + intent = api.ReserveForFailoverIntent } memoryMB := parseMemoryToMB(memory) - spec := api.NovaSpec{ - ProjectID: projectID, - InstanceUUID: instanceUUID, - NumInstances: 1, - SchedulerHints: schedulerHints, - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - Name: flavorName, - VCPUs: uint64(vcpus), //nolint:gosec // test code - MemoryMB: memoryMB, - ExtraSpecs: extraSpecs, - }, - }, - } - weights := make(map[string]float64) for _, h := range hosts { weights[h] = 1.0 } - return api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{Data: spec}, + return compute.PlacementRequest{ + ProjectID: projectID, + InstanceUUID: instanceUUID, + NumInstances: 1, + Intent: intent, + HardwareVersion: flavorGroup, + HypervisorFamily: api.HypervisorFamilyKVM, + Flavor: api.FlavorSpec{ + VCPUs: uint64(vcpus), //nolint:gosec // test code + MemoryMB: memoryMB, + }, Hosts: hostList, Weights: weights, } @@ -362,7 +332,7 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { tests := []struct { name string reservations []*v1alpha1.Reservation - request api.ExternalSchedulerRequest + request compute.PlacementRequest opts FilterHasEnoughCapacityOpts expectedHosts []string filteredHosts []string @@ -370,7 +340,7 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { { name: "No reservations - all hosts with capacity pass", reservations: []*v1alpha1.Reservation{}, - request: newNovaRequest("instance-123", "project-A", "m1.small", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host1", "host2", "host3"}, filteredHosts: []string{"host4"}, @@ -378,10 +348,10 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { { name: "CommittedResourceReservation of other project blocks some hosts", reservations: []*v1alpha1.Reservation{ - newCommittedReservation("res-1", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil, nil), - newCommittedReservation("res-2", "host2", "project-A", "m1.large", "gp-1", "4", "8Gi", nil, nil), + newCommittedReservation("res-1", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil), + newCommittedReservation("res-2", "host2", "project-A", "m1.large", "gp-1", "4", "8Gi", nil), }, - request: newNovaRequest("instance-123", "project-B", "m1.small", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-B", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host3"}, filteredHosts: []string{"host1", "host2", "host4"}, @@ -389,10 +359,10 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { { name: "CommittedResourceReservation of other project blocks only unused resources of reservation", reservations: []*v1alpha1.Reservation{ - newCommittedReservation("res-1 half used", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", crSpecAllocs(crVm("vm-1", "4", "8Gi")), map[string]string{"vm-1": "host1"}), - newCommittedReservation("res-2 fully used", "host2", "project-A", "m1.large", "gp-1", "4", "8Gi", crSpecAllocs(crVm("vm-2", "4", "8Gi")), map[string]string{"vm-2": "host2"}), + newCommittedReservation("res-1 half used", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", map[string]v1alpha1.CommittedResourceAllocation{"vm-1": makeAlloc("vm-1", "4", "8Gi")}), + newCommittedReservation("res-2 fully used", "host2", "project-A", "m1.large", "gp-1", "4", "8Gi", map[string]v1alpha1.CommittedResourceAllocation{"vm-2": makeAlloc("vm-2", "4", "8Gi")}), }, - request: newNovaRequest("instance-123", "project-B", "m1.small", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-B", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host1", "host2", "host3"}, filteredHosts: []string{"host4"}, @@ -405,11 +375,13 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { name: "CommittedResourceReservation blocks only remaining capacity when multiple VMs are confirmed in one slot", reservations: []*v1alpha1.Reservation{ newCommittedReservation("res-multi-vm", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", - crSpecAllocs(crVm("vm-1", "3", "6Gi"), crVm("vm-2", "2", "4Gi")), - map[string]string{"vm-1": "host1", "vm-2": "host1"}, + map[string]v1alpha1.CommittedResourceAllocation{ + "vm-1": makeAlloc("vm-1", "3", "6Gi"), + "vm-2": makeAlloc("vm-2", "2", "4Gi"), + }, ), }, - request: newNovaRequest("instance-123", "project-B", "m1.small", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-B", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host1", "host2", "host3"}, // host1: 5cpu/10Gi free → 4cpu/8Gi request passes filteredHosts: []string{"host4"}, @@ -417,11 +389,11 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { { name: "CommittedResourceReservation of other project blocks both source and target host during migration, ignoring used resources", reservations: []*v1alpha1.Reservation{ - newCommittedReservation("res-1", "host1", "project-A", "m1.large", "gp-1", "4", "8Gi", nil, nil), - newMigratingReservation("res-2", "host1", "host2", "project-A", "m1.large", "gp-1", "2", "4Gi", nil, nil), // migration reservation from host1 to host2 - newMigratingReservation("res-3", "host2", "host1", "project-A", "m1.large", "gp-1", "2", "4Gi", crSpecAllocs(crVm("vm-1", "2", "4Gi")), map[string]string{"vm-1": "host1"}), // migration reservation from host2 to host1 + newCommittedReservation("res-1", "host1", "project-A", "m1.large", "gp-1", "4", "8Gi", nil), + newMigratingReservation("res-2", "host1", "host2", "project-A", "m1.large", "gp-1", "2", "4Gi", nil), // migration reservation from host1 to host2 + newMigratingReservation("res-3", "host2", "host1", "project-A", "m1.large", "gp-1", "2", "4Gi", map[string]v1alpha1.CommittedResourceAllocation{"vm-1": makeAlloc("vm-1", "2", "4Gi")}), // migration reservation from host2 to host1 }, - request: newNovaRequest("instance-123", "project-B", "m1.small", "gp-1", 2, "4Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-B", "gp-1", 2, "4Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host3"}, filteredHosts: []string{"host1", "host2", "host4"}, @@ -430,12 +402,12 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { name: "CommittedResourceReservation unlocks for matching project and flavor group", reservations: []*v1alpha1.Reservation{ // all three reservations 1,2,3 are required to have enough capacity for the request - newCommittedReservation("res-1-unused", "host1", "project-A", "some flavor", "gp-1", "2", "4Gi", nil, nil), // fully unused reservation - newCommittedReservation("res-2-pending-used", "host1", "project-A", "some flavor", "gp-1", "2", "4Gi", crSpecAllocs(crVm("vm-1", "2", "4Gi")), nil), // reservation with a pending allocation - newCommittedReservation("res-3-used", "host1", "project-A", "some flavor", "gp-1", "2", "4Gi", crSpecAllocs(crVm("vm-2", "1", "1Gi")), map[string]string{"vm-2": "host1"}), // used reservation - newCommittedReservation("res-4", "host2", "project-A", "some flavor", "gp-2", "4", "8Gi", nil, nil), // different flavor group, should still block + newCommittedReservation("res-1-unused", "host1", "project-A", "some flavor", "gp-1", "2", "4Gi", nil), // fully unused reservation + newCommittedReservation("res-2-pending-used", "host1", "project-A", "some flavor", "gp-1", "2", "4Gi", nil), // reservation with a pending allocation + newCommittedReservation("res-3-used", "host1", "project-A", "some flavor", "gp-1", "2", "4Gi", map[string]v1alpha1.CommittedResourceAllocation{"vm-2": makeAlloc("vm-2", "2", "4Gi")}), // used reservation + newCommittedReservation("res-4", "host2", "project-A", "some flavor", "gp-2", "4", "8Gi", nil), // different flavor group, should still block }, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 8, "16Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 8, "16Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host1", "host3"}, filteredHosts: []string{"host4", "host2"}, @@ -443,10 +415,10 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { { name: "CommittedResourceReservation stays locked when LockReserved is true", reservations: []*v1alpha1.Reservation{ - newCommittedReservation("res-1", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil, nil), - newCommittedReservation("res-2", "host3", "project-A", "m1.large", "gp-1", "16", "32Gi", nil, nil), + newCommittedReservation("res-1", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil), + newCommittedReservation("res-2", "host3", "project-A", "m1.large", "gp-1", "16", "32Gi", nil), }, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: true}, expectedHosts: []string{"host2"}, filteredHosts: []string{"host1", "host3", "host4"}, @@ -454,9 +426,9 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { { name: "Empty reservation type defaults to CommittedResourceReservation behavior", reservations: []*v1alpha1.Reservation{ - newCommittedReservation("legacy-res", "host1", "project-A", "m1.large", "gp-1", "4", "8Gi", nil, nil), + newCommittedReservation("legacy-res", "host1", "project-A", "m1.large", "gp-1", "4", "8Gi", nil), }, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 4, "8Gi", false, []string{"host1", "host2"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host1", "host2"}, filteredHosts: []string{}, @@ -464,11 +436,11 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { { name: "All hosts blocked by reservations - none pass", reservations: []*v1alpha1.Reservation{ - newCommittedReservation("res-1", "host1", "project-X", "m1.xlarge", "gp-1", "8", "16Gi", nil, nil), - newCommittedReservation("res-2", "host2", "project-X", "m1.xlarge", "gp-1", "4", "8Gi", nil, nil), - newCommittedReservation("res-3", "host3", "project-X", "m1.xlarge", "gp-1", "16", "32Gi", nil, nil), + newCommittedReservation("res-1", "host1", "project-X", "m1.xlarge", "gp-1", "8", "16Gi", nil), + newCommittedReservation("res-2", "host2", "project-X", "m1.xlarge", "gp-1", "4", "8Gi", nil), + newCommittedReservation("res-3", "host3", "project-X", "m1.xlarge", "gp-1", "16", "32Gi", nil), }, - request: newNovaRequest("instance-123", "project-A", "m1.small", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{}, filteredHosts: []string{"host1", "host2", "host3", "host4"}, @@ -476,9 +448,9 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { { name: "Pending reservation (only TargetHost set) blocks capacity on desired host", reservations: []*v1alpha1.Reservation{ - newMigratingReservation("pending-res", "host1", "", "project-X", "m1.large", "gp-1", "8", "16Gi", nil, nil), + newMigratingReservation("pending-res", "host1", "", "project-X", "m1.large", "gp-1", "8", "16Gi", nil), }, - request: newNovaRequest("instance-123", "project-A", "m1.small", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host2", "host3"}, // host1 blocked by pending reservation filteredHosts: []string{"host1", "host4"}, @@ -486,10 +458,10 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { { name: "Multiple reservations: pending and placed block different hosts", reservations: []*v1alpha1.Reservation{ - newMigratingReservation("pending-res", "host1", "", "project-X", "m1.large", "gp-1", "8", "16Gi", nil, nil), - newMigratingReservation("placed-res", "host2", "host3", "project-X", "m1.large", "gp-1", "4", "8Gi", nil, nil), + newMigratingReservation("pending-res", "host1", "", "project-X", "m1.large", "gp-1", "8", "16Gi", nil), + newMigratingReservation("placed-res", "host2", "host3", "project-X", "m1.large", "gp-1", "4", "8Gi", nil), }, - request: newNovaRequest("instance-123", "project-A", "m1.small", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host3"}, // host1 blocked by pending, host2 blocked by placed, host3 still has capacity filteredHosts: []string{"host1", "host2", "host4"}, @@ -497,9 +469,9 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { { name: "Reservation with no host is skipped", reservations: []*v1alpha1.Reservation{ - newCommittedReservation("no-host-res", "", "project-X", "m1.large", "gp-1", "8", "16Gi", nil, nil), + newCommittedReservation("no-host-res", "", "project-X", "m1.large", "gp-1", "8", "16Gi", nil), }, - request: newNovaRequest("instance-123", "project-A", "m1.small", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host1", "host2", "host3"}, filteredHosts: []string{"host4"}, @@ -512,34 +484,29 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { name: "Confirmed VMs exceeding reservation size: block clamped to 0", reservations: []*v1alpha1.Reservation{ newCommittedReservation("res-oversized-vm", "host1", "project-X", "m1.large", "gp-1", "4", "8Gi", - crSpecAllocs(crVm("vm-1", "6", "10Gi")), - map[string]string{"vm-1": "host1"}, + map[string]v1alpha1.CommittedResourceAllocation{"vm-1": makeAlloc("vm-1", "6", "10Gi")}, ), }, - request: newNovaRequest("instance-123", "project-A", "m1.small", "gp-1", 9, "17Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 9, "17Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, // host1: 8 free CPU < 9, 16Gi < 17Gi → filtered; host2: 4 < 9 → filtered; host4: no capacity → filtered expectedHosts: []string{"host3"}, filteredHosts: []string{"host1", "host2", "host4"}, }, { - // host1: 8 CPU free, 16Gi free (shared hypervisors). - // Reservation slot=8cpu/16Gi, confirmed vm-1=4cpu/8Gi (remaining={4,8Gi}). - // Spec-only vm-2=6cpu/12Gi EXCEEDS remaining → block must be {6,12Gi}, not {4,8Gi}. - // Without fix: block={4,8Gi} → free={4cpu,8Gi}; 3-cpu/5Gi request passes (wrong). - // With fix: block={6,12Gi} → free={2cpu,4Gi}; filtered for 3-cpu/5Gi request. - name: "Spec-only VM larger than remaining slot: block covers spec-only VM", + // host1: 8 CPU free, 16Gi free. + // Reservation slot=8cpu/16Gi, confirmed vm-1 in status.allocs → block=0 (all confirmed VMs tracked in HV). + // host4: no capacity data → filtered. + name: "confirmed VM in status.allocs: block=0, host not filtered (HV accounting covers it)", reservations: []*v1alpha1.Reservation{ - newCommittedReservation("res-spec-only-oversize", "host1", "project-X", "m1.large", "gp-1", "8", "16Gi", - crSpecAllocs(crVm("vm-1", "4", "8Gi"), crVm("vm-2", "6", "12Gi")), - map[string]string{"vm-1": "host1"}, // vm-2 is spec-only + newCommittedReservation("res-confirmed", "host1", "project-X", "m1.large", "gp-1", "8", "16Gi", + map[string]v1alpha1.CommittedResourceAllocation{"vm-1": makeAlloc("vm-1", "8", "16Gi")}, ), }, - request: newNovaRequest("instance-123", "project-A", "m1.small", "gp-1", 3, "5Gi", false, []string{"host1", "host2", "host3", "host4"}), - opts: FilterHasEnoughCapacityOpts{LockReserved: false}, - // host1: 2 free CPU < 3 requested, 4Gi < 5Gi → filtered; host4: no capacity → filtered - expectedHosts: []string{"host2", "host3"}, - filteredHosts: []string{"host1", "host4"}, + request: newNovaRequest("instance-123", "project-A", "gp-1", 3, "5Gi", false, []string{"host1", "host2", "host3", "host4"}), + opts: FilterHasEnoughCapacityOpts{LockReserved: false}, + expectedHosts: []string{"host1", "host2", "host3"}, + filteredHosts: []string{"host4"}, }, { name: "FailoverReservation blocks hosts for non-evacuation request even when instance is in Allocations", @@ -547,7 +514,7 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { newFailoverReservation("failover-1", "host1", "8", "16Gi", map[string]string{"instance-123": "host5"}), newFailoverReservation("failover-2", "host2", "4", "8Gi", map[string]string{"instance-123": "host6"}), }, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host3"}, filteredHosts: []string{"host1", "host2", "host4"}, @@ -558,7 +525,7 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { newFailoverReservation("failover-1", "host1", "4", "8Gi", map[string]string{"instance-123": "host5"}), newFailoverReservation("failover-2", "host2", "4", "8Gi", map[string]string{"instance-123": "host6"}), }, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 4, "8Gi", true, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", true, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host1", "host2", "host3"}, filteredHosts: []string{"host4"}, @@ -569,7 +536,7 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { newFailoverReservation("failover-1", "host1", "8", "16Gi", map[string]string{"other-instance": "host5"}), newFailoverReservation("failover-2", "host2", "4", "8Gi", map[string]string{"another-instance": "host6"}), }, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 4, "8Gi", true, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", true, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host3"}, filteredHosts: []string{"host1", "host2", "host4"}, @@ -579,7 +546,7 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { reservations: []*v1alpha1.Reservation{ newFailoverReservation("failover-1", "host1", "8", "16Gi", map[string]string{}), }, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 4, "8Gi", true, []string{"host1", "host2", "host3"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", true, []string{"host1", "host2", "host3"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host2", "host3"}, filteredHosts: []string{"host1"}, @@ -589,7 +556,7 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { reservations: []*v1alpha1.Reservation{ newFailoverReservation("failover-1", "host1", "4", "8Gi", map[string]string{"instance-111": "host5", "instance-222": "host6", "instance-333": "host7"}), }, - request: newNovaRequest("instance-222", "project-A", "m1.large", "gp-1", 4, "8Gi", true, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-222", "project-A", "gp-1", 4, "8Gi", true, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host1", "host2", "host3"}, filteredHosts: []string{"host4"}, @@ -602,7 +569,7 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { reservations: []*v1alpha1.Reservation{ newUnconfirmedReservation("unconfirmed-res", "host1", "project-X", "gp-1", "8", "16Gi"), }, - request: newNovaRequest("instance-123", "project-A", "m1.small", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host2", "host3"}, filteredHosts: []string{"host1", "host4"}, @@ -615,7 +582,7 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { newUnconfirmedReservation("unconfirmed-1", "host1", "project-X", "gp-1", "8", "16Gi"), newUnconfirmedReservation("unconfirmed-2", "host2", "project-X", "gp-1", "4", "8Gi"), }, - request: newNovaRequest("instance-123", "project-A", "m1.small", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host3"}, filteredHosts: []string{"host1", "host2", "host4"}, @@ -640,7 +607,7 @@ func TestFilterHasEnoughCapacity_ReservationTypes(t *testing.T) { Status: v1alpha1.ReservationStatus{}, }, }, - request: newNovaRequest("instance-123", "project-A", "m1.small", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host2", "host3"}, filteredHosts: []string{"host1", "host4"}, @@ -681,7 +648,7 @@ func TestFilterHasEnoughCapacity_LockReservations(t *testing.T) { } // CR reservation on host1 for the same project+flavor — would normally be unlocked. reservations := []*v1alpha1.Reservation{ - newCommittedReservation("cr-res", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil, nil), + newCommittedReservation("cr-res", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil), } objects := make([]client.Object, 0, len(hypervisors)+len(reservations)) @@ -730,7 +697,7 @@ func TestFilterHasEnoughCapacity_LockReservations(t *testing.T) { }, } - request := newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 4, "8Gi", false, []string{"host1", "host2"}) + request := newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2"}) for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { step := &FilterHasEnoughCapacity{} @@ -756,7 +723,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { newHypervisor("host2", "16", "8", "32Gi", "16Gi"), } twoHostRes := []*v1alpha1.Reservation{ - newCommittedReservation("cr-res", "host1", "project-X", "m1.large", "gp-1", "4", "8Gi", nil, nil), + newCommittedReservation("cr-res", "host1", "project-X", "m1.large", "gp-1", "4", "8Gi", nil), newFailoverReservation("failover-res", "host2", "4", "8Gi", map[string]string{"other-vm": "host3"}), } @@ -766,7 +733,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { newHypervisor("host1", "12", "0", "24Gi", "0"), } singleHostRes := []*v1alpha1.Reservation{ - newCommittedReservation("cr-res", "host1", "project-X", "m1.large", "gp-1", "4", "8Gi", nil, nil), + newCommittedReservation("cr-res", "host1", "project-X", "m1.large", "gp-1", "4", "8Gi", nil), newFailoverReservation("failover-res", "host1", "2", "4Gi", map[string]string{"other-vm": "host3"}), } @@ -774,7 +741,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name string hypervisors []*hv1.Hypervisor reservations []*v1alpha1.Reservation - request api.ExternalSchedulerRequest + request compute.PlacementRequest ignoredReservationTypes []v1alpha1.ReservationType expectedHosts []string filteredHosts []string @@ -785,7 +752,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Two hosts: No ignore - both hosts blocked by reservations", hypervisors: twoHostHVs, reservations: twoHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 8, "16Gi", false, []string{"host1", "host2"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 8, "16Gi", false, []string{"host1", "host2"}), ignoredReservationTypes: nil, expectedHosts: []string{}, filteredHosts: []string{"host1", "host2"}, @@ -794,7 +761,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Two hosts: Ignore CR only - host1 passes, host2 blocked by failover", hypervisors: twoHostHVs, reservations: twoHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 8, "16Gi", false, []string{"host1", "host2"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 8, "16Gi", false, []string{"host1", "host2"}), ignoredReservationTypes: []v1alpha1.ReservationType{v1alpha1.ReservationTypeCommittedResource}, expectedHosts: []string{"host1"}, filteredHosts: []string{"host2"}, @@ -803,7 +770,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Two hosts: Ignore Failover only - host2 passes, host1 blocked by CR", hypervisors: twoHostHVs, reservations: twoHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 8, "16Gi", false, []string{"host1", "host2"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 8, "16Gi", false, []string{"host1", "host2"}), ignoredReservationTypes: []v1alpha1.ReservationType{v1alpha1.ReservationTypeFailover}, expectedHosts: []string{"host2"}, filteredHosts: []string{"host1"}, @@ -812,7 +779,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Two hosts: Ignore both - both hosts pass", hypervisors: twoHostHVs, reservations: twoHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 8, "16Gi", false, []string{"host1", "host2"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 8, "16Gi", false, []string{"host1", "host2"}), ignoredReservationTypes: []v1alpha1.ReservationType{v1alpha1.ReservationTypeCommittedResource, v1alpha1.ReservationTypeFailover}, expectedHosts: []string{"host1", "host2"}, filteredHosts: []string{}, @@ -825,7 +792,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Single host, Large VM (12 CPU): No ignore - blocked (6 free < 12 needed)", hypervisors: singleHostHVs, reservations: singleHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 12, "24Gi", false, []string{"host1"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 12, "24Gi", false, []string{"host1"}), ignoredReservationTypes: nil, expectedHosts: []string{}, filteredHosts: []string{"host1"}, @@ -834,7 +801,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Single host, Large VM (12 CPU): Ignore CR - blocked (10 free < 12 needed)", hypervisors: singleHostHVs, reservations: singleHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 12, "24Gi", false, []string{"host1"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 12, "24Gi", false, []string{"host1"}), ignoredReservationTypes: []v1alpha1.ReservationType{v1alpha1.ReservationTypeCommittedResource}, expectedHosts: []string{}, filteredHosts: []string{"host1"}, @@ -843,7 +810,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Single host, Large VM (12 CPU): Ignore Failover - blocked (8 free < 12 needed)", hypervisors: singleHostHVs, reservations: singleHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 12, "24Gi", false, []string{"host1"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 12, "24Gi", false, []string{"host1"}), ignoredReservationTypes: []v1alpha1.ReservationType{v1alpha1.ReservationTypeFailover}, expectedHosts: []string{}, filteredHosts: []string{"host1"}, @@ -852,7 +819,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Single host, Large VM (12 CPU): Ignore both - passes (12 free = 12 needed)", hypervisors: singleHostHVs, reservations: singleHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 12, "24Gi", false, []string{"host1"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 12, "24Gi", false, []string{"host1"}), ignoredReservationTypes: []v1alpha1.ReservationType{v1alpha1.ReservationTypeCommittedResource, v1alpha1.ReservationTypeFailover}, expectedHosts: []string{"host1"}, filteredHosts: []string{}, @@ -863,7 +830,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Single host, Failover-size VM (10 CPU): No ignore - blocked (6 free < 10 needed)", hypervisors: singleHostHVs, reservations: singleHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 10, "20Gi", false, []string{"host1"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 10, "20Gi", false, []string{"host1"}), ignoredReservationTypes: nil, expectedHosts: []string{}, filteredHosts: []string{"host1"}, @@ -872,7 +839,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Single host, Failover-size VM (10 CPU): Ignore CR - passes (10 free = 10 needed)", hypervisors: singleHostHVs, reservations: singleHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 10, "20Gi", false, []string{"host1"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 10, "20Gi", false, []string{"host1"}), ignoredReservationTypes: []v1alpha1.ReservationType{v1alpha1.ReservationTypeCommittedResource}, expectedHosts: []string{"host1"}, filteredHosts: []string{}, @@ -881,7 +848,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Single host, Failover-size VM (10 CPU): Ignore Failover - blocked (8 free < 10 needed)", hypervisors: singleHostHVs, reservations: singleHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 10, "20Gi", false, []string{"host1"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 10, "20Gi", false, []string{"host1"}), ignoredReservationTypes: []v1alpha1.ReservationType{v1alpha1.ReservationTypeFailover}, expectedHosts: []string{}, filteredHosts: []string{"host1"}, @@ -890,7 +857,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Single host, Failover-size VM (10 CPU): Ignore both - passes (12 free > 10 needed)", hypervisors: singleHostHVs, reservations: singleHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 10, "20Gi", false, []string{"host1"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 10, "20Gi", false, []string{"host1"}), ignoredReservationTypes: []v1alpha1.ReservationType{v1alpha1.ReservationTypeCommittedResource, v1alpha1.ReservationTypeFailover}, expectedHosts: []string{"host1"}, filteredHosts: []string{}, @@ -901,7 +868,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Single host, CR-size VM (8 CPU): No ignore - blocked (6 free < 8 needed)", hypervisors: singleHostHVs, reservations: singleHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 8, "16Gi", false, []string{"host1"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 8, "16Gi", false, []string{"host1"}), ignoredReservationTypes: nil, expectedHosts: []string{}, filteredHosts: []string{"host1"}, @@ -910,7 +877,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Single host, CR-size VM (8 CPU): Ignore CR - passes (10 free > 8 needed)", hypervisors: singleHostHVs, reservations: singleHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 8, "16Gi", false, []string{"host1"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 8, "16Gi", false, []string{"host1"}), ignoredReservationTypes: []v1alpha1.ReservationType{v1alpha1.ReservationTypeCommittedResource}, expectedHosts: []string{"host1"}, filteredHosts: []string{}, @@ -919,7 +886,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Single host, CR-size VM (8 CPU): Ignore Failover - passes (8 free = 8 needed)", hypervisors: singleHostHVs, reservations: singleHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 8, "16Gi", false, []string{"host1"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 8, "16Gi", false, []string{"host1"}), ignoredReservationTypes: []v1alpha1.ReservationType{v1alpha1.ReservationTypeFailover}, expectedHosts: []string{"host1"}, filteredHosts: []string{}, @@ -928,7 +895,7 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes(t *testing.T) { name: "Single host, CR-size VM (8 CPU): Ignore both - passes (12 free > 8 needed)", hypervisors: singleHostHVs, reservations: singleHostRes, - request: newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 8, "16Gi", false, []string{"host1"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 8, "16Gi", false, []string{"host1"}), ignoredReservationTypes: []v1alpha1.ReservationType{v1alpha1.ReservationTypeCommittedResource, v1alpha1.ReservationTypeFailover}, expectedHosts: []string{"host1"}, filteredHosts: []string{}, @@ -969,7 +936,7 @@ func TestFilterHasEnoughCapacity_AssumeEmptyHosts(t *testing.T) { hypervisors := []*hv1.Hypervisor{ newHypervisor("host1", "8", "6", "32Gi", "0"), } - request := newNovaRequest("vm", "proj", "m1.large", "gp", 4, "1Gi", false, []string{"host1"}) + request := newNovaRequest("vm", "proj", "gp", 4, "1Gi", false, []string{"host1"}) objects := make([]client.Object, 0, len(hypervisors)) for _, h := range hypervisors { @@ -1006,10 +973,10 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes_CallTime(t *testing.T) newHypervisor("host2", "16", "8", "32Gi", "16Gi"), } reservations := []*v1alpha1.Reservation{ - newCommittedReservation("cr-res", "host1", "project-X", "m1.large", "gp-1", "4", "8Gi", nil, nil), + newCommittedReservation("cr-res", "host1", "project-X", "m1.large", "gp-1", "4", "8Gi", nil), newFailoverReservation("failover-res", "host2", "4", "8Gi", map[string]string{"other-vm": "host3"}), } - request := newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 8, "16Gi", false, []string{"host1", "host2"}) + request := newNovaRequest("instance-123", "project-A", "gp-1", 8, "16Gi", false, []string{"host1", "host2"}) objects := make([]client.Object, 0, len(hypervisors)+len(reservations)) for _, h := range hypervisors { @@ -1045,10 +1012,10 @@ func TestFilterHasEnoughCapacity_IgnoredReservationTypes_CallTimeMerge(t *testin newHypervisor("host2", "16", "8", "32Gi", "16Gi"), } reservations := []*v1alpha1.Reservation{ - newCommittedReservation("cr-res", "host1", "project-X", "m1.large", "gp-1", "4", "8Gi", nil, nil), + newCommittedReservation("cr-res", "host1", "project-X", "m1.large", "gp-1", "4", "8Gi", nil), newFailoverReservation("failover-res", "host2", "4", "8Gi", map[string]string{"other-vm": "host3"}), } - request := newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 8, "16Gi", false, []string{"host1", "host2"}) + request := newNovaRequest("instance-123", "project-A", "gp-1", 8, "16Gi", false, []string{"host1", "host2"}) objects := make([]client.Object, 0, len(hypervisors)+len(reservations)) for _, h := range hypervisors { @@ -1087,7 +1054,7 @@ func TestFilterHasEnoughCapacity_ReserveForCommittedResourceIntent(t *testing.T) name string hypervisors []*hv1.Hypervisor reservations []*v1alpha1.Reservation - request api.ExternalSchedulerRequest + request compute.PlacementRequest opts FilterHasEnoughCapacityOpts expectedHosts []string filteredHosts []string @@ -1100,11 +1067,11 @@ func TestFilterHasEnoughCapacity_ReserveForCommittedResourceIntent(t *testing.T) }, reservations: []*v1alpha1.Reservation{ // Existing CR reservation on host1 for same project+flavor group - newCommittedReservation("existing-cr", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil, nil), + newCommittedReservation("existing-cr", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil), }, // Request with reserve_for_committed_resource intent (scheduling a new CR reservation). // The intent causes the filter to keep CR reservations locked (prevents overbooking). - request: newNovaRequestWithIntent("new-reservation-uuid", "project-A", "m1.large", "gp-1", 4, "8Gi", "reserve_for_committed_resource", false, []string{"host1", "host2"}), + request: newNovaRequestWithIntent("new-reservation-uuid", "project-A", "gp-1", 4, "8Gi", "reserve_for_committed_resource", false, []string{"host1", "host2"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host2"}, // host1 blocked because existing-cr stays locked filteredHosts: []string{"host1"}, @@ -1117,10 +1084,10 @@ func TestFilterHasEnoughCapacity_ReserveForCommittedResourceIntent(t *testing.T) }, reservations: []*v1alpha1.Reservation{ // Existing CR reservation on host1 for same project+flavor group - newCommittedReservation("existing-cr", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil, nil), + newCommittedReservation("existing-cr", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil), }, // Normal VM create request (no special intent) - CR reservation should be unlocked - request: newNovaRequest("vm-instance-123", "project-A", "m1.large", "gp-1", 4, "8Gi", false, []string{"host1", "host2"}), + request: newNovaRequest("vm-instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host1", "host2"}, // host1 passes because existing-cr is unlocked for matching project+flavor filteredHosts: []string{}, @@ -1133,10 +1100,10 @@ func TestFilterHasEnoughCapacity_ReserveForCommittedResourceIntent(t *testing.T) }, reservations: []*v1alpha1.Reservation{ // Existing CR reservation on host1 for different project - newCommittedReservation("other-project-cr", "host1", "project-B", "m1.large", "gp-1", "8", "16Gi", nil, nil), + newCommittedReservation("other-project-cr", "host1", "project-B", "m1.large", "gp-1", "8", "16Gi", nil), }, // Request with reserve_for_committed_resource intent - request: newNovaRequestWithIntent("new-reservation-uuid", "project-A", "m1.large", "gp-1", 4, "8Gi", "reserve_for_committed_resource", false, []string{"host1", "host2"}), + request: newNovaRequestWithIntent("new-reservation-uuid", "project-A", "gp-1", 4, "8Gi", "reserve_for_committed_resource", false, []string{"host1", "host2"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host2"}, filteredHosts: []string{"host1"}, // host1 blocked by other project's reservation (would be blocked anyway) @@ -1148,13 +1115,13 @@ func TestFilterHasEnoughCapacity_ReserveForCommittedResourceIntent(t *testing.T) }, reservations: []*v1alpha1.Reservation{ // Three existing CR reservations on host1 for same project+flavor group - newCommittedReservation("cr-1", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil, nil), - newCommittedReservation("cr-2", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil, nil), - newCommittedReservation("cr-3", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil, nil), + newCommittedReservation("cr-1", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil), + newCommittedReservation("cr-2", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil), + newCommittedReservation("cr-3", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil), }, // Request with reserve_for_committed_resource intent, needs 10 CPU // After blocking all 3 reservations (24 CPU), only 8 CPU free -> should fail - request: newNovaRequestWithIntent("new-reservation-uuid", "project-A", "m1.large", "gp-1", 10, "20Gi", "reserve_for_committed_resource", false, []string{"host1"}), + request: newNovaRequestWithIntent("new-reservation-uuid", "project-A", "gp-1", 10, "20Gi", "reserve_for_committed_resource", false, []string{"host1"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{}, filteredHosts: []string{"host1"}, // All reservations stay locked (intent prevents unlock), not enough capacity @@ -1166,13 +1133,13 @@ func TestFilterHasEnoughCapacity_ReserveForCommittedResourceIntent(t *testing.T) }, reservations: []*v1alpha1.Reservation{ // Three existing CR reservations on host1 for same project+flavor group - newCommittedReservation("cr-1", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil, nil), - newCommittedReservation("cr-2", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil, nil), - newCommittedReservation("cr-3", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil, nil), + newCommittedReservation("cr-1", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil), + newCommittedReservation("cr-2", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil), + newCommittedReservation("cr-3", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil), }, // Normal VM create request, needs 10 CPU // All 3 reservations unlocked for matching project+flavor -> 32 CPU free -> should pass - request: newNovaRequest("vm-instance-123", "project-A", "m1.large", "gp-1", 10, "20Gi", false, []string{"host1"}), + request: newNovaRequest("vm-instance-123", "project-A", "gp-1", 10, "20Gi", false, []string{"host1"}), opts: FilterHasEnoughCapacityOpts{LockReserved: false}, expectedHosts: []string{"host1"}, // All reservations unlocked, enough capacity filteredHosts: []string{}, @@ -1184,11 +1151,11 @@ func TestFilterHasEnoughCapacity_ReserveForCommittedResourceIntent(t *testing.T) }, reservations: []*v1alpha1.Reservation{ // Existing CR reservation on host1 blocks all 8 free CPU - newCommittedReservation("existing-cr", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil, nil), + newCommittedReservation("existing-cr", "host1", "project-A", "m1.large", "gp-1", "8", "16Gi", nil), }, // Request with reserve_for_committed_resource intent. // IgnoredReservationTypes is a YAML-level safety override that bypasses the intent check. - request: newNovaRequestWithIntent("new-reservation-uuid", "project-A", "m1.large", "gp-1", 4, "8Gi", "reserve_for_committed_resource", false, []string{"host1"}), + request: newNovaRequestWithIntent("new-reservation-uuid", "project-A", "gp-1", 4, "8Gi", "reserve_for_committed_resource", false, []string{"host1"}), opts: FilterHasEnoughCapacityOpts{ LockReserved: false, IgnoredReservationTypes: []v1alpha1.ReservationType{v1alpha1.ReservationTypeCommittedResource}, @@ -1203,11 +1170,11 @@ func TestFilterHasEnoughCapacity_ReserveForCommittedResourceIntent(t *testing.T) }, reservations: []*v1alpha1.Reservation{ // Existing CR reservation on host1 blocks all 8 free CPU - newCommittedReservation("existing-cr", "host1", "project-B", "m1.large", "gp-1", "8", "16Gi", nil, nil), + newCommittedReservation("existing-cr", "host1", "project-B", "m1.large", "gp-1", "8", "16Gi", nil), }, // Normal VM create request (different project, so unlocking via project match won't work) // But IgnoredReservationTypes should make it work - request: newNovaRequest("vm-instance-123", "project-A", "m1.large", "gp-1", 4, "8Gi", false, []string{"host1"}), + request: newNovaRequest("vm-instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1"}), opts: FilterHasEnoughCapacityOpts{ LockReserved: false, IgnoredReservationTypes: []v1alpha1.ReservationType{v1alpha1.ReservationTypeCommittedResource}, @@ -1280,7 +1247,7 @@ func TestFilterHasEnoughCapacity_PlannedCRDoesNotBlock(t *testing.T) { step.Client = fake.NewClientBuilder().WithScheme(scheme).WithObjects(objects...).Build() step.Options = FilterHasEnoughCapacityOpts{LockReserved: false} - request := newNovaRequest("instance-123", "project-A", "m1.large", "gp-1", 4, "8Gi", false, []string{"host1"}) + request := newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1"}) result, err := step.Run(logr.Discard(), request) if err != nil { t.Fatalf("expected no error, got %v", err) @@ -1296,7 +1263,7 @@ func TestFilterHasEnoughCapacity_NilEffectiveCapacityFallback(t *testing.T) { tests := []struct { name string hypervisors []*hv1.Hypervisor - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedHosts []string filteredHosts []string }{ @@ -1308,7 +1275,7 @@ func TestFilterHasEnoughCapacity_NilEffectiveCapacityFallback(t *testing.T) { newHypervisorWithCapacityOnly("host3", "2", "4Gi"), // nil EffectiveCapacity, uses Capacity: 2 CPU free (not enough) newHypervisorWithCapacityOnly("host4", "16", "32Gi"), // nil EffectiveCapacity, uses Capacity: 16 CPU free, 32Gi free }, - request: newNovaRequest("instance-123", "project-A", "m1.small", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2", "host3", "host4"}), expectedHosts: []string{"host1", "host2", "host4"}, filteredHosts: []string{"host3"}, }, @@ -1318,7 +1285,7 @@ func TestFilterHasEnoughCapacity_NilEffectiveCapacityFallback(t *testing.T) { newHypervisorWithCapacityOnly("host1", "8", "16Gi"), newHypervisorWithCapacityOnly("host2", "4", "8Gi"), }, - request: newNovaRequest("instance-123", "project-A", "m1.small", "gp-1", 4, "8Gi", false, []string{"host1", "host2"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 4, "8Gi", false, []string{"host1", "host2"}), expectedHosts: []string{"host1", "host2"}, filteredHosts: []string{}, }, @@ -1337,7 +1304,7 @@ func TestFilterHasEnoughCapacity_NilEffectiveCapacityFallback(t *testing.T) { // Both: 4 free -> filtered (need 5) newHypervisorWithBothCapacities("host3", "4", "4", "8Gi", "8Gi"), }, - request: newNovaRequest("instance-123", "project-A", "m1.small", "gp-1", 5, "8Gi", false, []string{"host1", "host2", "host3"}), + request: newNovaRequest("instance-123", "project-A", "gp-1", 5, "8Gi", false, []string{"host1", "host2", "host3"}), expectedHosts: []string{"host1", "host2"}, filteredHosts: []string{"host3"}, }, @@ -1469,7 +1436,7 @@ func TestFilterHasEnoughCapacity_CapacityBoundaries(t *testing.T) { step.Client = fake.NewClientBuilder().WithScheme(scheme).WithObjects(tt.hv.DeepCopy()).Build() step.Options = FilterHasEnoughCapacityOpts{LockReserved: false} - req := newNovaRequest("inst-1", "project-A", "m1.small", "gp-1", flavorVCPUs, flavorMem, false, []string{"h"}) + req := newNovaRequest("inst-1", "project-A", "gp-1", flavorVCPUs, flavorMem, false, []string{"h"}) result, err := step.Run(logr.Discard(), req) if err != nil { t.Fatalf("unexpected error: %v", err) @@ -1534,10 +1501,9 @@ func TestFilterHasEnoughCapacity_VMInterReservationMigration(t *testing.T) { hvA: newHypervisor("hv-a", hvCapCPU, vmCPU, hvCapMem, vmMem), hvB: newHypervisor("hv-b", hvCapCPU, "0", hvCapMem, "0"), resA: newCommittedReservation("res-a", "hv-a", owner, "m1.large", flavorGroup, slotCPU, slotMem, - crSpecAllocs(crVm("vm-1", vmCPU, vmMem)), - map[string]string{"vm-1": "hv-a"}, + map[string]v1alpha1.CommittedResourceAllocation{"vm-1": makeAlloc("vm-1", vmCPU, vmMem)}, ), - resB: newCommittedReservation("res-b", "hv-b", owner, "m1.large", flavorGroup, slotCPU, slotMem, nil, nil), + resB: newCommittedReservation("res-b", "hv-b", owner, "m1.large", flavorGroup, slotCPU, slotMem, nil), expectedHosts: []string{"hv-a", "hv-b"}, filteredHosts: []string{}, }, @@ -1552,11 +1518,9 @@ func TestFilterHasEnoughCapacity_VMInterReservationMigration(t *testing.T) { hvA: newHypervisor("hv-a", hvCapCPU, vmCPU, hvCapMem, vmMem), hvB: newHypervisor("hv-b", hvCapCPU, "0", hvCapMem, "0"), resA: newCommittedReservation("res-a", "hv-a", owner, "m1.large", flavorGroup, slotCPU, slotMem, - crSpecAllocs(crVm("vm-1", vmCPU, vmMem)), - map[string]string{"vm-1": "hv-a"}, + map[string]v1alpha1.CommittedResourceAllocation{"vm-1": makeAlloc("vm-1", vmCPU, vmMem)}, ), resB: newCommittedReservation("res-b", "hv-b", owner, "m1.large", flavorGroup, slotCPU, slotMem, - crSpecAllocs(crVm("vm-1", vmCPU, vmMem)), nil, ), expectedHosts: []string{"hv-a", "hv-b"}, @@ -1572,11 +1536,9 @@ func TestFilterHasEnoughCapacity_VMInterReservationMigration(t *testing.T) { hvA: newHypervisor("hv-a", hvCapCPU, vmCPU, hvCapMem, vmMem), hvB: newHypervisor("hv-b", hvCapCPU, vmCPU, hvCapMem, vmMem), resA: newCommittedReservation("res-a", "hv-a", owner, "m1.large", flavorGroup, slotCPU, slotMem, - crSpecAllocs(crVm("vm-1", vmCPU, vmMem)), - map[string]string{"vm-1": "hv-a"}, + map[string]v1alpha1.CommittedResourceAllocation{"vm-1": makeAlloc("vm-1", vmCPU, vmMem)}, ), resB: newCommittedReservation("res-b", "hv-b", owner, "m1.large", flavorGroup, slotCPU, slotMem, - crSpecAllocs(crVm("vm-1", vmCPU, vmMem)), nil, ), expectedHosts: []string{"hv-a"}, @@ -1591,11 +1553,9 @@ func TestFilterHasEnoughCapacity_VMInterReservationMigration(t *testing.T) { hvA: newHypervisor("hv-a", hvCapCPU, "0", hvCapMem, "0"), hvB: newHypervisor("hv-b", hvCapCPU, vmCPU, hvCapMem, vmMem), resA: newCommittedReservation("res-a", "hv-a", owner, "m1.large", flavorGroup, slotCPU, slotMem, - crSpecAllocs(crVm("vm-1", vmCPU, vmMem)), - map[string]string{"vm-1": "hv-a"}, + map[string]v1alpha1.CommittedResourceAllocation{"vm-1": makeAlloc("vm-1", vmCPU, vmMem)}, ), resB: newCommittedReservation("res-b", "hv-b", owner, "m1.large", flavorGroup, slotCPU, slotMem, - crSpecAllocs(crVm("vm-1", vmCPU, vmMem)), nil, ), expectedHosts: []string{"hv-a"}, @@ -1612,12 +1572,10 @@ func TestFilterHasEnoughCapacity_VMInterReservationMigration(t *testing.T) { hvA: newHypervisor("hv-a", hvCapCPU, "0", hvCapMem, "0"), hvB: newHypervisor("hv-b", hvCapCPU, vmCPU, hvCapMem, vmMem), resA: newCommittedReservation("res-a", "hv-a", owner, "m1.large", flavorGroup, slotCPU, slotMem, - crSpecAllocs(crVm("vm-1", vmCPU, vmMem)), - map[string]string{"vm-1": "hv-a"}, + map[string]v1alpha1.CommittedResourceAllocation{"vm-1": makeAlloc("vm-1", vmCPU, vmMem)}, ), resB: newCommittedReservation("res-b", "hv-b", owner, "m1.large", flavorGroup, slotCPU, slotMem, - crSpecAllocs(crVm("vm-1", vmCPU, vmMem)), - map[string]string{"vm-1": "hv-b"}, + map[string]v1alpha1.CommittedResourceAllocation{"vm-1": makeAlloc("vm-1", vmCPU, vmMem)}, ), expectedHosts: []string{"hv-a", "hv-b"}, filteredHosts: []string{}, @@ -1633,12 +1591,10 @@ func TestFilterHasEnoughCapacity_VMInterReservationMigration(t *testing.T) { hvA: newHypervisor("hv-a", hvCapCPU, "0", hvCapMem, "0"), hvB: newHypervisor("hv-b", hvCapCPU, vmCPU, hvCapMem, vmMem), resA: newCommittedReservation("res-a", "hv-a", owner, "m1.large", flavorGroup, slotCPU, slotMem, - nil, // spec allocs cleared - map[string]string{"vm-1": "hv-a"}, // status stale — not consulted when spec is empty + map[string]v1alpha1.CommittedResourceAllocation{"vm-1": makeAlloc("vm-1", vmCPU, vmMem)}, // status stale ), resB: newCommittedReservation("res-b", "hv-b", owner, "m1.large", flavorGroup, slotCPU, slotMem, - crSpecAllocs(crVm("vm-1", vmCPU, vmMem)), - map[string]string{"vm-1": "hv-b"}, + map[string]v1alpha1.CommittedResourceAllocation{"vm-1": makeAlloc("vm-1", vmCPU, vmMem)}, ), expectedHosts: []string{"hv-a", "hv-b"}, filteredHosts: []string{}, @@ -1653,11 +1609,10 @@ func TestFilterHasEnoughCapacity_VMInterReservationMigration(t *testing.T) { hvA: newHypervisor("hv-a", hvCapCPU, "0", hvCapMem, "0"), hvB: newHypervisor("hv-b", hvCapCPU, vmCPU, hvCapMem, vmMem), resA: newCommittedReservation("res-a", "hv-a", owner, "m1.large", flavorGroup, slotCPU, slotMem, - nil, nil, + nil, ), resB: newCommittedReservation("res-b", "hv-b", owner, "m1.large", flavorGroup, slotCPU, slotMem, - crSpecAllocs(crVm("vm-1", vmCPU, vmMem)), - map[string]string{"vm-1": "hv-b"}, + map[string]v1alpha1.CommittedResourceAllocation{"vm-1": makeAlloc("vm-1", vmCPU, vmMem)}, ), expectedHosts: []string{"hv-a", "hv-b"}, filteredHosts: []string{}, @@ -1672,7 +1627,7 @@ func TestFilterHasEnoughCapacity_VMInterReservationMigration(t *testing.T) { step.Client = fake.NewClientBuilder().WithScheme(scheme).WithObjects(objects...).Build() step.Options = FilterHasEnoughCapacityOpts{LockReserved: false} - request := newNovaRequest("instance-new", thirdParty, "m1.small", flavorGroup, 3, "6Gi", false, []string{"hv-a", "hv-b"}) + request := newNovaRequest("instance-new", thirdParty, flavorGroup, 3, "6Gi", false, []string{"hv-a", "hv-b"}) result, err := step.Run(logr.Discard(), request) if err != nil { t.Fatalf("unexpected error: %v", err) diff --git a/internal/scheduling/nova/plugins/filters/filter_has_requested_traits.go b/internal/scheduling/nova/plugins/filters/filter_has_requested_traits.go index d6d322707..a8c00db7c 100644 --- a/internal/scheduling/nova/plugins/filters/filter_has_requested_traits.go +++ b/internal/scheduling/nova/plugins/filters/filter_has_requested_traits.go @@ -7,43 +7,23 @@ import ( "context" "github.com/go-logr/logr" "slices" - "strings" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" ) type FilterHasRequestedTraits struct { - lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts] + lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts] } -// Filter hosts that do not have the requested traits given by the extra spec: -// - "trait:": "forbidden" means the host must not have the specified trait. -// - "trait:": "required" means the host must have the specified trait. -func (s *FilterHasRequestedTraits) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +// Filter hosts based on the traits requested in the flavor: +// - a forbidden trait means the host must not have the specified trait. +// - a required trait means the host must have the specified trait. +func (s *FilterHasRequestedTraits) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) - var requiredTraits, forbiddenTraits []string - for key, value := range request.Spec.Data.Flavor.Data.ExtraSpecs { - if !strings.HasPrefix(key, "trait:") { - traceLog.V(1).Info("ignoring extra spec that does not start with trait:", "key", key) - continue - } - traitName := strings.TrimPrefix(key, "trait:") - switch value { - case "forbidden": - forbiddenTraits = append(forbiddenTraits, traitName) - traceLog.V(1).Info("trait marked forbidden, will filter out hosts with this trait", - "trait", traitName) - case "required": - requiredTraits = append(requiredTraits, traitName) - traceLog.V(1).Info("trait marked required, will filter out hosts without this trait", - "trait", traitName) - default: - traceLog.V(1).Info("ignoring trait with unsupported value", - "trait", traitName, "value", value) - } - } + requiredTraits := request.RequiredTraits + forbiddenTraits := request.ForbiddenTraits if len(requiredTraits) == 0 && len(forbiddenTraits) == 0 { traceLog.V(1).Info("no traits requested, skipping filter") return result, nil diff --git a/internal/scheduling/nova/plugins/filters/filter_has_requested_traits_test.go b/internal/scheduling/nova/plugins/filters/filter_has_requested_traits_test.go index 719aa37fd..e5276054a 100644 --- a/internal/scheduling/nova/plugins/filters/filter_has_requested_traits_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_has_requested_traits_test.go @@ -7,7 +7,7 @@ import ( "github.com/go-logr/logr" "testing" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -75,23 +75,14 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedHosts []string filteredHosts []string }{ { name: "No traits requested - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -102,19 +93,9 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { }, { name: "Single required trait - filter hosts with trait", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "trait:CUSTOM_TRAIT_A": "required", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + RequiredTraits: []string{"CUSTOM_TRAIT_A"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -126,20 +107,9 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { }, { name: "Multiple required traits - filter hosts with all traits", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "trait:CUSTOM_TRAIT_A": "required", - "trait:CUSTOM_TRAIT_B": "required", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + RequiredTraits: []string{"CUSTOM_TRAIT_A", "CUSTOM_TRAIT_B"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -151,19 +121,9 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { }, { name: "Single forbidden trait - filter hosts without trait", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "trait:CUSTOM_TRAIT_A": "forbidden", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ForbiddenTraits: []string{"CUSTOM_TRAIT_A"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -176,20 +136,9 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { }, { name: "Multiple forbidden traits - filter hosts without any of them", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "trait:CUSTOM_TRAIT_A": "forbidden", - "trait:CUSTOM_TRAIT_B": "forbidden", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ForbiddenTraits: []string{"CUSTOM_TRAIT_A", "CUSTOM_TRAIT_B"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -202,20 +151,10 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { }, { name: "Mixed required and forbidden traits", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "trait:CUSTOM_TRAIT_A": "required", - "trait:CUSTOM_TRAIT_D": "forbidden", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + RequiredTraits: []string{"CUSTOM_TRAIT_A"}, + ForbiddenTraits: []string{"CUSTOM_TRAIT_D"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host4"}, @@ -227,19 +166,9 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { }, { name: "Required trait that no host has", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "trait:CUSTOM_TRAIT_NONEXISTENT": "required", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + RequiredTraits: []string{"CUSTOM_TRAIT_NONEXISTENT"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -250,19 +179,9 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { }, { name: "Forbidden trait that no host has - all pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "trait:CUSTOM_TRAIT_NONEXISTENT": "forbidden", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ForbiddenTraits: []string{"CUSTOM_TRAIT_NONEXISTENT"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -273,19 +192,9 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { }, { name: "Host with no traits - required trait filters it out", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "trait:CUSTOM_TRAIT_A": "required", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + RequiredTraits: []string{"CUSTOM_TRAIT_A"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host5"}, }, @@ -295,19 +204,9 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { }, { name: "Host with no traits - forbidden trait lets it pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "trait:CUSTOM_TRAIT_A": "forbidden", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ForbiddenTraits: []string{"CUSTOM_TRAIT_A"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host5"}, }, @@ -317,20 +216,9 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { }, { name: "Non-trait extra specs are ignored", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "hw:cpu_policy": "dedicated", - "trait:CUSTOM_TRAIT_A": "required", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + RequiredTraits: []string{"CUSTOM_TRAIT_A"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host3"}, }, @@ -340,19 +228,8 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { }, { name: "Invalid trait value (not required or forbidden) - ignored", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "trait:CUSTOM_TRAIT_A": "invalid", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -363,38 +240,18 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { }, { name: "Empty host list", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "trait:CUSTOM_TRAIT_A": "required", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{}, + request: compute.PlacementRequest{ + RequiredTraits: []string{"CUSTOM_TRAIT_A"}, + Hosts: []compute.Host{}, }, expectedHosts: []string{}, filteredHosts: []string{}, }, { name: "Host not in database", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "trait:CUSTOM_TRAIT_A": "required", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + RequiredTraits: []string{"CUSTOM_TRAIT_A"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host-unknown"}, }, @@ -404,21 +261,10 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { }, { name: "Complex scenario with many traits", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "trait:CUSTOM_TRAIT_A": "required", - "trait:CUSTOM_TRAIT_B": "required", - "trait:CUSTOM_TRAIT_D": "forbidden", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + RequiredTraits: []string{"CUSTOM_TRAIT_A", "CUSTOM_TRAIT_B"}, + ForbiddenTraits: []string{"CUSTOM_TRAIT_D"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -432,20 +278,10 @@ func TestFilterHasRequestedTraits_Run(t *testing.T) { }, { name: "All hosts match required and forbidden traits", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - ExtraSpecs: map[string]string{ - "trait:CUSTOM_TRAIT_A": "required", - "trait:CUSTOM_TRAIT_E": "forbidden", - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + RequiredTraits: []string{"CUSTOM_TRAIT_A"}, + ForbiddenTraits: []string{"CUSTOM_TRAIT_E"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, diff --git a/internal/scheduling/nova/plugins/filters/filter_host_instructions.go b/internal/scheduling/nova/plugins/filters/filter_host_instructions.go index c4121c22d..88b1b48b6 100644 --- a/internal/scheduling/nova/plugins/filters/filter_host_instructions.go +++ b/internal/scheduling/nova/plugins/filters/filter_host_instructions.go @@ -7,30 +7,30 @@ import ( "github.com/go-logr/logr" "slices" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" ) type FilterHostInstructionsStep struct { - lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts] + lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts] } // Filter hosts based on instructions given in the request spec. Supported are: // - spec.ignore_hosts: Filter out all hosts in this list. // - spec.force_hosts: Include only hosts in this list. -func (s *FilterHostInstructionsStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *FilterHostInstructionsStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) - if request.Spec.Data.IgnoreHosts != nil { - for _, host := range *request.Spec.Data.IgnoreHosts { + if len(request.IgnoreHosts) > 0 { + for _, host := range request.IgnoreHosts { delete(result.Activations, host) traceLog.V(1).Info("filtering host which is ignored", "host", host) } } else { traceLog.V(1).Info("no hosts to ignore specified") } - if request.Spec.Data.ForceHosts != nil { + if len(request.ForceHosts) > 0 { for host := range result.Activations { - if !slices.Contains(*request.Spec.Data.ForceHosts, host) { + if !slices.Contains(request.ForceHosts, host) { delete(result.Activations, host) traceLog.V(1).Info("filtering host which is not forced", "host", host) } diff --git a/internal/scheduling/nova/plugins/filters/filter_host_instructions_test.go b/internal/scheduling/nova/plugins/filters/filter_host_instructions_test.go index 349932c3c..973b40dc1 100644 --- a/internal/scheduling/nova/plugins/filters/filter_host_instructions_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_host_instructions_test.go @@ -7,26 +7,20 @@ import ( "github.com/go-logr/logr" "testing" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" ) func TestFilterHostInstructionsStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedHosts []string filteredHosts []string }{ { name: "No host instructions - no filtering", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: nil, - ForceHosts: nil, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -38,14 +32,9 @@ func TestFilterHostInstructionsStep_Run(t *testing.T) { }, { name: "Ignore single host", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: &[]string{"host2"}, - ForceHosts: nil, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + IgnoreHosts: []string{"host2"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -57,14 +46,9 @@ func TestFilterHostInstructionsStep_Run(t *testing.T) { }, { name: "Ignore multiple hosts", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: &[]string{"host1", "host3", "host4"}, - ForceHosts: nil, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + IgnoreHosts: []string{"host1", "host3", "host4"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -76,14 +60,9 @@ func TestFilterHostInstructionsStep_Run(t *testing.T) { }, { name: "Ignore all hosts", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: &[]string{"host1", "host2", "host3", "host4"}, - ForceHosts: nil, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + IgnoreHosts: []string{"host1", "host2", "host3", "host4"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -95,14 +74,9 @@ func TestFilterHostInstructionsStep_Run(t *testing.T) { }, { name: "Ignore non-existent host", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: &[]string{"host-nonexistent"}, - ForceHosts: nil, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + IgnoreHosts: []string{"host-nonexistent"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -113,14 +87,9 @@ func TestFilterHostInstructionsStep_Run(t *testing.T) { }, { name: "Force single host", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: nil, - ForceHosts: &[]string{"host2"}, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ForceHosts: []string{"host2"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -132,14 +101,9 @@ func TestFilterHostInstructionsStep_Run(t *testing.T) { }, { name: "Force multiple hosts", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: nil, - ForceHosts: &[]string{"host1", "host3"}, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ForceHosts: []string{"host1", "host3"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -151,14 +115,9 @@ func TestFilterHostInstructionsStep_Run(t *testing.T) { }, { name: "Force non-existent host", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: nil, - ForceHosts: &[]string{"host-nonexistent"}, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ForceHosts: []string{"host-nonexistent"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -169,14 +128,10 @@ func TestFilterHostInstructionsStep_Run(t *testing.T) { }, { name: "Force and ignore same host - ignore takes precedence", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: &[]string{"host2"}, - ForceHosts: &[]string{"host2"}, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + IgnoreHosts: []string{"host2"}, + ForceHosts: []string{"host2"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -187,14 +142,10 @@ func TestFilterHostInstructionsStep_Run(t *testing.T) { }, { name: "Force and ignore different hosts", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: &[]string{"host1"}, - ForceHosts: &[]string{"host2", "host3"}, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + IgnoreHosts: []string{"host1"}, + ForceHosts: []string{"host2", "host3"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -206,14 +157,10 @@ func TestFilterHostInstructionsStep_Run(t *testing.T) { }, { name: "Complex scenario - multiple ignore and force", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: &[]string{"host2", "host5"}, - ForceHosts: &[]string{"host1", "host3", "host4"}, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + IgnoreHosts: []string{"host2", "host5"}, + ForceHosts: []string{"host1", "host3", "host4"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -227,28 +174,19 @@ func TestFilterHostInstructionsStep_Run(t *testing.T) { }, { name: "Empty host list", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: &[]string{"host1"}, - ForceHosts: &[]string{"host2"}, - }, - }, - Hosts: []api.ExternalSchedulerHost{}, + request: compute.PlacementRequest{ + IgnoreHosts: []string{"host1"}, + ForceHosts: []string{"host2"}, + Hosts: []compute.Host{}, }, expectedHosts: []string{}, filteredHosts: []string{}, }, { name: "Empty ignore list", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: &[]string{}, - ForceHosts: nil, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + IgnoreHosts: []string{}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -258,48 +196,36 @@ func TestFilterHostInstructionsStep_Run(t *testing.T) { }, { name: "Empty force list", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: nil, - ForceHosts: &[]string{}, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ForceHosts: []string{}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, }, - expectedHosts: []string{}, - filteredHosts: []string{"host1", "host2"}, + // An empty force list is no constraint in the typed spec (it cannot be + // distinguished from "unset"), so all hosts are kept. + expectedHosts: []string{"host1", "host2"}, + filteredHosts: []string{}, }, { name: "Both lists empty", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: &[]string{}, - ForceHosts: &[]string{}, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + IgnoreHosts: []string{}, + ForceHosts: []string{}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, }, - expectedHosts: []string{}, - filteredHosts: []string{"host1", "host2"}, + expectedHosts: []string{"host1", "host2"}, + filteredHosts: []string{}, }, { name: "Duplicate hosts in ignore list", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: &[]string{"host1", "host1", "host2"}, - ForceHosts: nil, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + IgnoreHosts: []string{"host1", "host1", "host2"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -310,14 +236,9 @@ func TestFilterHostInstructionsStep_Run(t *testing.T) { }, { name: "Duplicate hosts in force list", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - IgnoreHosts: nil, - ForceHosts: &[]string{"host1", "host1", "host2"}, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + ForceHosts: []string{"host1", "host1", "host2"}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, diff --git a/internal/scheduling/nova/plugins/filters/filter_image_properties.go b/internal/scheduling/nova/plugins/filters/filter_image_properties.go index f4467c828..537162c78 100644 --- a/internal/scheduling/nova/plugins/filters/filter_image_properties.go +++ b/internal/scheduling/nova/plugins/filters/filter_image_properties.go @@ -5,68 +5,55 @@ package filters import ( "context" - "github.com/go-logr/logr" "slices" + "github.com/go-logr/logr" + api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" ) type FilterImagePropertiesStep struct { - lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts] + lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts] } // Filter hosts based on image properties given in the request spec. -func (s *FilterImagePropertiesStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *FilterImagePropertiesStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) - // Resolve the scheduling intent once and fall back to a readable default - // when it cannot be determined. - intent, intentErr := request.GetIntent() - intentLabel := "unknown" - if intentErr == nil { - intentLabel = string(intent) - } - // Apply this filter to all requests, unless we know from the request's // intent that image metadata is expected to not be set. - if intentErr == nil { - intentsExpectedToNotHaveImageMeta := []v1alpha1.SchedulingIntent{ - // Cortex-internal intents in which scheduling requests are sent - // mainly based on flavor-related metadata, independent of the - // image that'll land there. - api.ReserveForFailoverIntent, - api.ReuseFailoverReservationIntent, - api.ReserveForCommittedResourceIntent, - api.CapacityProbeIntent, - } - if slices.Contains(intentsExpectedToNotHaveImageMeta, intent) { - traceLog.V(1).Info("skipping filter: expected to have no image metadata", - "intent", intent) - return result, nil - } + intentsExpectedToNotHaveImageMeta := []v1alpha1.SchedulingIntent{ + // Cortex-internal intents in which scheduling requests are sent + // mainly based on flavor-related metadata, independent of the + // image that'll land there. + api.ReserveForFailoverIntent, + api.ReuseFailoverReservationIntent, + api.ReserveForCommittedResourceIntent, + api.CapacityProbeIntent, + } + if slices.Contains(intentsExpectedToNotHaveImageMeta, request.Intent) { + traceLog.V(1).Info("skipping filter: expected to have no image metadata", + "intent", request.Intent) + return result, nil } - // If the image properties indicate any other hypervisor type than kvm, - // we filter out all known kvm hypervisors. - hvType, err := request.Spec.Data.Image.Data.GetHypervisorType() - if err != nil { - // Hypervisor type could not be determined. In this case, just - // ignore the filter and return all hosts. - traceLog.Error(err, "could not determine hypervisor type from image properties") - // Expose this event through the step monitor to alert on high-frequency - // occurrences of this situation. Include the request intent as a dynamic - // label so the alert can pinpoint which scheduling intents are affected. - result.Events = append(result.Events, lib.FilterWeigherPipelineStepEvent{ - Name: "image_properties_hv_type_undetermined", - Labels: map[string]string{"intent": intentLabel}, - }) + hvType := request.ImageHypervisorType + + // If the image hypervisor type is unset, we treat it as if no + // constraint is applied. + if hvType == api.ImageHypervisorTypeUnparseable || hvType == api.ImageHypervisorTypeUnset { + traceLog.Info("image hypervisor type is unset or unparseable, skipping filter") return result, nil } - if hvType != api.NovaImageMetaHVTypeKVM { + // If the image properties indicate any other hypervisor type than kvm, + // we filter out all known kvm hypervisors. An unset image hypervisor type + // means the image declared none, so no image constraint applies. + if hvType != api.ImageHypervisorTypeUnset && hvType != api.ImageHypervisorTypeKVM { traceLog.V(1).Info("filtering out all known kvm hypervisors since image properties indicate a different hypervisor type", "image_hypervisor_type", hvType) hvs := new(hv1.HypervisorList) diff --git a/internal/scheduling/nova/plugins/filters/filter_image_properties_test.go b/internal/scheduling/nova/plugins/filters/filter_image_properties_test.go index 0c836a9e2..0b7234f61 100644 --- a/internal/scheduling/nova/plugins/filters/filter_image_properties_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_image_properties_test.go @@ -4,10 +4,13 @@ package filters import ( - "github.com/go-logr/logr" "testing" + "github.com/go-logr/logr" + api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -16,24 +19,25 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client/fake" ) -// requestWith builds an ExternalSchedulerRequest with the given hosts and image -// properties. -func requestWith(hosts []string, properties map[string]any) api.ExternalSchedulerRequest { - schedHosts := make([]api.ExternalSchedulerHost, 0, len(hosts)) +// requestWith builds a SchedulerRequest with the given hosts and image +// hypervisor type. +func requestWith(hosts []string, hvType api.ImageHypervisorType) compute.PlacementRequest { + schedHosts := make([]compute.Host, 0, len(hosts)) for _, h := range hosts { - schedHosts = append(schedHosts, api.ExternalSchedulerHost{ComputeHost: h}) + schedHosts = append(schedHosts, compute.Host{ComputeHost: h}) + } + return compute.PlacementRequest{ + Hosts: schedHosts, + ImageHypervisorType: hvType, } - request := api.ExternalSchedulerRequest{Hosts: schedHosts} - request.Spec.Data.Image.Data.Properties = api.NovaObject[map[string]any]{Data: properties} - return request } -// requestWithIntent builds an ExternalSchedulerRequest like requestWith, but -// additionally sets the _nova_check_type scheduler hint so GetIntent resolves -// to the given intent. -func requestWithIntent(hosts []string, properties map[string]any, checkType string) api.ExternalSchedulerRequest { - request := requestWith(hosts, properties) - request.Spec.Data.SchedulerHints = map[string]any{"_nova_check_type": checkType} +// requestWithIntent builds a SchedulerRequest with the given intent. The image +// hypervisor type is irrelevant to the intent-skip paths these callers exercise, +// so it is fixed to VMware. +func requestWithIntent(hosts []string, intent v1alpha1.SchedulingIntent) compute.PlacementRequest { + request := requestWith(hosts, api.ImageHypervisorTypeVMware) + request.Intent = intent return request } @@ -51,80 +55,74 @@ func TestFilterImagePropertiesStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedHosts []string filteredHosts []string expectedEvents []lib.FilterWeigherPipelineStepEvent }{ { name: "kvm image keeps all hosts", - request: requestWith([]string{"host1", "host2", "host3"}, map[string]any{"img_hv_type": "kvm"}), + request: requestWith([]string{"host1", "host2", "host3"}, api.ImageHypervisorTypeKVM), expectedHosts: []string{"host1", "host2", "host3"}, filteredHosts: []string{}, }, { name: "vmware image filters out known kvm hypervisors", - request: requestWith([]string{"host1", "host2", "host3"}, map[string]any{"hypervisor_type": "vmware"}), + request: requestWith([]string{"host1", "host2", "host3"}, api.ImageHypervisorTypeVMware), expectedHosts: []string{"host3"}, filteredHosts: []string{"host1", "host2"}, }, { name: "baremetal image filters out known kvm hypervisors", - request: requestWith([]string{"host1", "host3"}, map[string]any{"img_hv_type": "baremetal"}), + request: requestWith([]string{"host1", "host3"}, api.ImageHypervisorTypeBaremetal), expectedHosts: []string{"host3"}, filteredHosts: []string{"host1"}, }, { name: "undeterminable hypervisor type keeps all hosts", - request: requestWith([]string{"host1", "host2", "host3"}, nil), + request: requestWith([]string{"host1", "host2", "host3"}, api.ImageHypervisorTypeUnparseable), expectedHosts: []string{"host1", "host2", "host3"}, filteredHosts: []string{}, - expectedEvents: []lib.FilterWeigherPipelineStepEvent{ - {Name: "image_properties_hv_type_undetermined", Labels: map[string]string{"intent": "unknown"}}, - }, }, { name: "unsupported hypervisor type keeps all hosts", - request: requestWith([]string{"host1", "host2"}, map[string]any{"img_hv_type": "xen"}), + request: requestWith([]string{"host1", "host2"}, api.ImageHypervisorTypeUnparseable), expectedHosts: []string{"host1", "host2"}, filteredHosts: []string{}, - expectedEvents: []lib.FilterWeigherPipelineStepEvent{ - {Name: "image_properties_hv_type_undetermined", Labels: map[string]string{"intent": "unknown"}}, - }, }, { name: "empty host list", - request: requestWith([]string{}, map[string]any{"hypervisor_type": "vmware"}), + request: requestWith([]string{}, api.ImageHypervisorTypeVMware), expectedHosts: []string{}, filteredHosts: []string{}, }, { name: "reserve_for_failover intent skips filter and keeps all hosts", - request: requestWithIntent([]string{"host1", "host2", "host3"}, map[string]any{"hypervisor_type": "vmware"}, "reserve_for_failover"), + request: requestWithIntent([]string{"host1", "host2", "host3"}, api.ReserveForFailoverIntent), expectedHosts: []string{"host1", "host2", "host3"}, filteredHosts: []string{}, }, { name: "reuse_failover_reservation intent skips filter and keeps all hosts", - request: requestWithIntent([]string{"host1", "host2", "host3"}, map[string]any{"hypervisor_type": "vmware"}, "reuse_failover_reservation"), + request: requestWithIntent([]string{"host1", "host2", "host3"}, api.ReuseFailoverReservationIntent), expectedHosts: []string{"host1", "host2", "host3"}, filteredHosts: []string{}, }, { name: "reserve_for_committed_resource intent skips filter and keeps all hosts", - request: requestWithIntent([]string{"host1", "host2", "host3"}, map[string]any{"hypervisor_type": "vmware"}, "reserve_for_committed_resource"), + request: requestWithIntent([]string{"host1", "host2", "host3"}, api.ReserveForCommittedResourceIntent), expectedHosts: []string{"host1", "host2", "host3"}, filteredHosts: []string{}, }, { name: "capacity_probe intent skips filter and keeps all hosts", - request: requestWithIntent([]string{"host1", "host2", "host3"}, map[string]any{"hypervisor_type": "vmware"}, "capacity_probe"), + request: requestWithIntent([]string{"host1", "host2", "host3"}, api.CapacityProbeIntent), expectedHosts: []string{"host1", "host2", "host3"}, filteredHosts: []string{}, }, { name: "create intent still filters out known kvm hypervisors", - request: requestWithIntent([]string{"host1", "host2", "host3"}, map[string]any{"hypervisor_type": "vmware"}, "create"), + request: requestWithIntent([]string{"host1", "host2", "host3"}, api.CreateIntent), expectedHosts: []string{"host3"}, filteredHosts: []string{"host1", "host2"}, }, diff --git a/internal/scheduling/nova/plugins/filters/filter_instance_group_affinity.go b/internal/scheduling/nova/plugins/filters/filter_instance_group_affinity.go index a233bd37a..4430139d9 100644 --- a/internal/scheduling/nova/plugins/filters/filter_instance_group_affinity.go +++ b/internal/scheduling/nova/plugins/filters/filter_instance_group_affinity.go @@ -9,49 +9,50 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" ) type FilterInstanceGroupAffinityStep struct { - lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts] + lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts] } // Select hosts in spec.instance_group. func (s *FilterInstanceGroupAffinityStep) Run( traceLog logr.Logger, - request api.ExternalSchedulerRequest, + request compute.PlacementRequest, ) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) // Instance group hints are only present in user-initiated VM placement requests; - if intent, err := request.GetIntent(); err == nil && slices.Contains([]v1alpha1.SchedulingIntent{ + if slices.Contains([]v1alpha1.SchedulingIntent{ api.ReserveForFailoverIntent, api.ReuseFailoverReservationIntent, api.ReserveForCommittedResourceIntent, api.CapacityProbeIntent, - }, intent) { + }, request.Intent) { return result, nil } - ig := request.Spec.Data.InstanceGroup + ig := request.InstanceGroup if ig == nil { traceLog.V(1).Info("no instance group in request, skipping filter") return result, nil } - policy := ig.Data.Policy + policy := ig.Policy if policy != "affinity" { traceLog.V(1).Info("instance group policy is not 'affinity', skipping filter") return result, nil } - if len(ig.Data.Hosts) == 0 { + if len(ig.Hosts) == 0 { // Nothing to do. traceLog.V(1).Info("instance group has no hosts, skipping filter") return result, nil } for host := range result.Activations { - if slices.Contains(ig.Data.Hosts, host) { + if slices.Contains(ig.Hosts, host) { traceLog.V(1).Info("host is in instance group, keeping", "host", host) continue } diff --git a/internal/scheduling/nova/plugins/filters/filter_instance_group_affinity_test.go b/internal/scheduling/nova/plugins/filters/filter_instance_group_affinity_test.go index 5a955d515..c7a8a5888 100644 --- a/internal/scheduling/nova/plugins/filters/filter_instance_group_affinity_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_instance_group_affinity_test.go @@ -8,24 +8,22 @@ import ( "testing" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" ) func TestFilterInstanceGroupAffinityStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedHosts []string filteredHosts []string }{ { name: "No instance group - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceGroup: nil, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceGroup: nil, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -36,18 +34,9 @@ func TestFilterInstanceGroupAffinityStep_Run(t *testing.T) { }, { name: "Instance group with anti-affinity policy - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Hosts: []string{"host1", "host2"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Hosts: []string{"host1", "host2"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -58,18 +47,9 @@ func TestFilterInstanceGroupAffinityStep_Run(t *testing.T) { }, { name: "Instance group with soft-affinity policy - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "soft-affinity", - Hosts: []string{"host1"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceGroup: &api.InstanceGroupSpec{Policy: "soft-affinity", Hosts: []string{"host1"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -79,18 +59,9 @@ func TestFilterInstanceGroupAffinityStep_Run(t *testing.T) { }, { name: "Affinity policy with empty hosts list - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "affinity", - Hosts: []string{}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceGroup: &api.InstanceGroupSpec{Policy: "affinity", Hosts: []string{}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -101,18 +72,9 @@ func TestFilterInstanceGroupAffinityStep_Run(t *testing.T) { }, { name: "Affinity policy - only hosts in instance group pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "affinity", - Hosts: []string{"host1", "host3"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceGroup: &api.InstanceGroupSpec{Policy: "affinity", Hosts: []string{"host1", "host3"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -123,18 +85,9 @@ func TestFilterInstanceGroupAffinityStep_Run(t *testing.T) { }, { name: "Affinity policy - single host in group", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "affinity", - Hosts: []string{"host2"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceGroup: &api.InstanceGroupSpec{Policy: "affinity", Hosts: []string{"host2"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -146,18 +99,9 @@ func TestFilterInstanceGroupAffinityStep_Run(t *testing.T) { }, { name: "Affinity policy - all hosts filtered out", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "affinity", - Hosts: []string{"host5", "host6"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceGroup: &api.InstanceGroupSpec{Policy: "affinity", Hosts: []string{"host5", "host6"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -168,18 +112,9 @@ func TestFilterInstanceGroupAffinityStep_Run(t *testing.T) { }, { name: "Affinity policy - all hosts in group pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "affinity", - Hosts: []string{"host1", "host2", "host3"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceGroup: &api.InstanceGroupSpec{Policy: "affinity", Hosts: []string{"host1", "host2", "host3"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -190,18 +125,9 @@ func TestFilterInstanceGroupAffinityStep_Run(t *testing.T) { }, { name: "Affinity policy - partial overlap", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "affinity", - Hosts: []string{"host2", "host3", "host5"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceGroup: &api.InstanceGroupSpec{Policy: "affinity", Hosts: []string{"host2", "host3", "host5"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -213,36 +139,18 @@ func TestFilterInstanceGroupAffinityStep_Run(t *testing.T) { }, { name: "Affinity policy - empty request hosts", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "affinity", - Hosts: []string{"host1", "host2"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{}, + request: compute.PlacementRequest{ + InstanceGroup: &api.InstanceGroupSpec{Policy: "affinity", Hosts: []string{"host1", "host2"}}, + Hosts: []compute.Host{}, }, expectedHosts: []string{}, filteredHosts: []string{}, }, { name: "Affinity policy - case sensitive host matching", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "affinity", - Hosts: []string{"Host1", "HOST2"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceGroup: &api.InstanceGroupSpec{Policy: "affinity", Hosts: []string{"Host1", "HOST2"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -253,23 +161,11 @@ func TestFilterInstanceGroupAffinityStep_Run(t *testing.T) { }, { name: "Affinity policy - with instance UUID and project ID", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-123", - ProjectID: "project-abc", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - UUID: "ig-uuid-456", - Policy: "affinity", - Hosts: []string{"host1", "host2"}, - Members: []string{"vm-uuid-789"}, - ProjectID: "project-abc", - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-123", + ProjectID: "project-abc", + InstanceGroup: &api.InstanceGroupSpec{UUID: "ig-uuid-456", Policy: "affinity", Hosts: []string{"host1", "host2"}, Members: []string{"vm-uuid-789"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -281,18 +177,9 @@ func TestFilterInstanceGroupAffinityStep_Run(t *testing.T) { }, { name: "Affinity policy - duplicate hosts in group list", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "affinity", - Hosts: []string{"host1", "host2", "host1", "host2"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceGroup: &api.InstanceGroupSpec{Policy: "affinity", Hosts: []string{"host1", "host2", "host1", "host2"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -303,18 +190,9 @@ func TestFilterInstanceGroupAffinityStep_Run(t *testing.T) { }, { name: "Affinity policy - single host scenario", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "affinity", - Hosts: []string{"host1"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceGroup: &api.InstanceGroupSpec{Policy: "affinity", Hosts: []string{"host1"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, }, @@ -357,11 +235,9 @@ func TestFilterInstanceGroupAffinityStep_SkipsForNonPlacementIntent(t *testing.T // Affinity group on host1 only — host2 would normally be filtered. for _, intent := range []string{"reserve_for_failover", "reuse_failover_reservation", "reserve_for_committed_resource", "capacity_probe"} { t.Run(intent, func(t *testing.T) { - request := newNovaRequest("vm", "proj", "m1.small", "gp", 1, "1Gi", false, []string{"host1", "host2"}) - request.Spec.Data.InstanceGroup = &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{Policy: "affinity", Hosts: []string{"host1"}}, - } - request.Spec.Data.SchedulerHints = map[string]any{"_nova_check_type": intent} + request := newNovaRequest("vm", "proj", "gp", 1, "1Gi", false, []string{"host1", "host2"}) + request.InstanceGroup = &api.InstanceGroupSpec{Policy: "affinity", Hosts: []string{"host1"}} + request.Intent = v1alpha1.SchedulingIntent(intent) step := &FilterInstanceGroupAffinityStep{} result, err := step.Run(logr.Discard(), request) diff --git a/internal/scheduling/nova/plugins/filters/filter_instance_group_anti_affinity.go b/internal/scheduling/nova/plugins/filters/filter_instance_group_anti_affinity.go index ad17d8ab2..2840d3f13 100644 --- a/internal/scheduling/nova/plugins/filters/filter_instance_group_anti_affinity.go +++ b/internal/scheduling/nova/plugins/filters/filter_instance_group_anti_affinity.go @@ -10,55 +10,52 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" ) type FilterInstanceGroupAntiAffinityStep struct { - lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts] + lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts] } // Select hosts not in spec_obj.instance_group but only until // max_server_per_host is reached (by default = 1). func (s *FilterInstanceGroupAntiAffinityStep) Run( traceLog logr.Logger, - request api.ExternalSchedulerRequest, + request compute.PlacementRequest, ) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) // Instance group hints are only present in user-initiated VM placement requests; - if intent, err := request.GetIntent(); err == nil && slices.Contains([]v1alpha1.SchedulingIntent{ + if slices.Contains([]v1alpha1.SchedulingIntent{ api.ReserveForFailoverIntent, api.ReuseFailoverReservationIntent, api.ReserveForCommittedResourceIntent, api.CapacityProbeIntent, - }, intent) { + }, request.Intent) { return result, nil } - ig := request.Spec.Data.InstanceGroup + ig := request.InstanceGroup if ig == nil { traceLog.V(1).Info("no instance group in request, skipping filter") return result, nil } - policy := ig.Data.Policy + policy := ig.Policy if policy != "anti-affinity" { traceLog.V(1).Info("instance group policy is not 'anti-affinity', skipping filter") return result, nil } - memberVMs := ig.Data.Members + memberVMs := ig.Members if len(memberVMs) == 0 { // Nothing to do. traceLog.V(1).Info("instance group has no members, skipping filter") return result, nil } - maxServersPerHost := 1 - if ig.Data.Rules != nil { - if maxServersPerHostAny, ok := ig.Data.Rules["max_server_per_host"]; ok { - if maxServersPerHostInt, ok := maxServersPerHostAny.(int); ok { - maxServersPerHost = maxServersPerHostInt - } - } + maxServersPerHost := ig.MaxServerPerHost + if maxServersPerHost == 0 { + maxServersPerHost = 1 } hvs := &hv1.HypervisorList{} @@ -84,7 +81,7 @@ func (s *FilterInstanceGroupAntiAffinityStep) Run( // Check if this host is already running the same vm (resizes). // In this case we should not filter it out. if slices.ContainsFunc(hv.Status.Instances, func(inst hv1.Instance) bool { - return inst.ID == request.Spec.Data.InstanceUUID + return inst.ID == request.InstanceUUID }) { traceLog.V(1).Info("host is running the same VM, not filtering out", "host", host) continue diff --git a/internal/scheduling/nova/plugins/filters/filter_instance_group_anti_affinity_test.go b/internal/scheduling/nova/plugins/filters/filter_instance_group_anti_affinity_test.go index bfbd1c475..45312e362 100644 --- a/internal/scheduling/nova/plugins/filters/filter_instance_group_anti_affinity_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_instance_group_anti_affinity_test.go @@ -8,6 +8,8 @@ import ( "testing" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -79,20 +81,16 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedHosts []string filteredHosts []string }{ { name: "No instance group - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: nil, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: nil, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -103,19 +101,10 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Instance group with affinity policy - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "affinity", - Members: []string{"vm-uuid-1", "vm-uuid-2"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: &api.InstanceGroupSpec{Policy: "affinity", Members: []string{"vm-uuid-1", "vm-uuid-2"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -126,19 +115,10 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Instance group with soft-anti-affinity policy - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "soft-anti-affinity", - Members: []string{"vm-uuid-1"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: &api.InstanceGroupSpec{Policy: "soft-anti-affinity", Members: []string{"vm-uuid-1"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -148,19 +128,10 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Anti-affinity policy with empty members list - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Members: []string{}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -171,19 +142,10 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Anti-affinity policy - default max_server_per_host=1", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{"vm-uuid-1", "vm-uuid-2"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Members: []string{"vm-uuid-1", "vm-uuid-2"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -195,22 +157,10 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Anti-affinity policy - max_server_per_host=2", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{"vm-uuid-1", "vm-uuid-2", "vm-uuid-3"}, - Rules: map[string]any{ - "max_server_per_host": 2, - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Members: []string{"vm-uuid-1", "vm-uuid-2", "vm-uuid-3"}, MaxServerPerHost: 2}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -222,22 +172,10 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Anti-affinity policy - max_server_per_host=3", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{"vm-uuid-4", "vm-uuid-5", "vm-uuid-6"}, - Rules: map[string]any{ - "max_server_per_host": 3, - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Members: []string{"vm-uuid-4", "vm-uuid-5", "vm-uuid-6"}, MaxServerPerHost: 3}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host5"}, }, @@ -247,19 +185,10 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Anti-affinity policy - host running same VM (resize scenario)", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-1", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{"vm-uuid-1", "vm-uuid-2"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-1", + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Members: []string{"vm-uuid-1", "vm-uuid-2"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host4"}, @@ -270,19 +199,10 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Anti-affinity policy - all hosts filtered out", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{"vm-uuid-1", "vm-uuid-2", "vm-uuid-3"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Members: []string{"vm-uuid-1", "vm-uuid-2", "vm-uuid-3"}}, + Hosts: []compute.Host{ {ComputeHost: "host2"}, {ComputeHost: "host3"}, {ComputeHost: "host4"}, @@ -293,19 +213,10 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Anti-affinity policy - mixed hosts", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{"vm-uuid-1", "vm-uuid-2", "vm-uuid-3", "vm-uuid-4", "vm-uuid-5"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Members: []string{"vm-uuid-1", "vm-uuid-2", "vm-uuid-3", "vm-uuid-4", "vm-uuid-5"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -318,38 +229,20 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Anti-affinity policy - empty request hosts", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{"vm-uuid-1"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{}, + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Members: []string{"vm-uuid-1"}}, + Hosts: []compute.Host{}, }, expectedHosts: []string{}, filteredHosts: []string{}, }, { name: "Anti-affinity policy - host with non-member instances", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{"vm-uuid-100", "vm-uuid-101"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Members: []string{"vm-uuid-100", "vm-uuid-101"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -362,22 +255,10 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Anti-affinity policy - max_server_per_host=2 with mixed instances", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{"vm-uuid-1", "vm-uuid-2"}, - Rules: map[string]any{ - "max_server_per_host": 2, - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Members: []string{"vm-uuid-1", "vm-uuid-2"}, MaxServerPerHost: 2}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -390,22 +271,11 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Anti-affinity policy - with instance UUID and project ID", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - ProjectID: "project-abc", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - UUID: "ig-uuid-456", - Policy: "anti-affinity", - Members: []string{"vm-uuid-1"}, - ProjectID: "project-abc", - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + ProjectID: "project-abc", + InstanceGroup: &api.InstanceGroupSpec{UUID: "ig-uuid-456", Policy: "anti-affinity", Members: []string{"vm-uuid-1"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -416,22 +286,10 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Anti-affinity policy - multiple members on same host with max=2", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{"vm-uuid-1", "vm-uuid-2", "vm-uuid-3"}, - Rules: map[string]any{ - "max_server_per_host": 2, - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Members: []string{"vm-uuid-1", "vm-uuid-2", "vm-uuid-3"}, MaxServerPerHost: 2}, + Hosts: []compute.Host{ {ComputeHost: "host3"}, {ComputeHost: "host4"}, }, @@ -441,19 +299,10 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Anti-affinity policy - resize with VM on multiple hosts", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-2", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{"vm-uuid-1", "vm-uuid-2", "vm-uuid-3"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-2", + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Members: []string{"vm-uuid-1", "vm-uuid-2", "vm-uuid-3"}}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -465,19 +314,10 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Anti-affinity policy - single host scenario", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{"vm-uuid-1"}, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Members: []string{"vm-uuid-1"}}, + Hosts: []compute.Host{ {ComputeHost: "host2"}, }, }, @@ -486,22 +326,10 @@ func TestFilterInstanceGroupAntiAffinityStep_Run(t *testing.T) { }, { name: "Anti-affinity policy - high max_server_per_host value", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: "vm-uuid-new", - InstanceGroup: &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{"vm-uuid-1", "vm-uuid-2", "vm-uuid-3"}, - Rules: map[string]any{ - "max_server_per_host": 10, - }, - }, - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + InstanceUUID: "vm-uuid-new", + InstanceGroup: &api.InstanceGroupSpec{Policy: "anti-affinity", Members: []string{"vm-uuid-1", "vm-uuid-2", "vm-uuid-3"}, MaxServerPerHost: 10}, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -566,15 +394,13 @@ func TestFilterInstanceGroupAntiAffinityStep_SkipsForNonPlacementIntent(t *testi for _, intent := range []string{"reserve_for_failover", "reuse_failover_reservation", "reserve_for_committed_resource", "capacity_probe"} { t.Run(intent, func(t *testing.T) { - request := newNovaRequest("vm-new", "proj", "m1.small", "gp", 1, "1Gi", false, []string{"host1", "host2"}) - request.Spec.Data.InstanceGroup = &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - Policy: "anti-affinity", - Members: []string{"vm-existing"}, - Rules: map[string]any{"max_server_per_host": 1}, - }, + request := newNovaRequest("vm-new", "proj", "gp", 1, "1Gi", false, []string{"host1", "host2"}) + request.InstanceGroup = &api.InstanceGroupSpec{ + Policy: "anti-affinity", + Members: []string{"vm-existing"}, + MaxServerPerHost: 1, } - request.Spec.Data.SchedulerHints = map[string]any{"_nova_check_type": intent} + request.Intent = v1alpha1.SchedulingIntent(intent) result, err := step.Run(logr.Discard(), request) if err != nil { diff --git a/internal/scheduling/nova/plugins/filters/filter_live_migratable.go b/internal/scheduling/nova/plugins/filters/filter_live_migratable.go index 94c525f53..a7cbc4f13 100644 --- a/internal/scheduling/nova/plugins/filters/filter_live_migratable.go +++ b/internal/scheduling/nova/plugins/filters/filter_live_migratable.go @@ -10,12 +10,13 @@ import ( "slices" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" ) type FilterLiveMigratableStep struct { - lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts] + lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts] } // Check if a vm can be live migrated from a source to a given target host. @@ -50,25 +51,19 @@ func (s *FilterLiveMigratableStep) checkHasSufficientFeatures( // Ensure the target host of a live migration can accept the migrating VM. func (s *FilterLiveMigratableStep) Run( traceLog logr.Logger, - request api.ExternalSchedulerRequest, + request compute.PlacementRequest, ) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) - intent, err := request.GetIntent() - if err != nil { - traceLog.V(1).Info("failed to determine request intent, skipping filter", "error", err) - return result, nil - } - if intent != api.LiveMigrationIntent { + if request.Intent != api.LiveMigrationIntent { traceLog.V(1).Info("not a live migration request, skipping filter") return result, nil } - sourceHost, err := request.Spec.Data.GetSchedulerHintStr("source_host") - if err != nil || sourceHost == "" { - traceLog.V(1).Info("no source_host scheduler hint, skipping filter") - //nolint:nilerr // Not an error we want to fail the scheduling for. + sourceHost := request.SourceHost + if sourceHost == "" { + traceLog.V(1).Info("no source host set, skipping filter") return result, nil } diff --git a/internal/scheduling/nova/plugins/filters/filter_live_migratable_test.go b/internal/scheduling/nova/plugins/filters/filter_live_migratable_test.go index 8f866b309..5bae46da5 100644 --- a/internal/scheduling/nova/plugins/filters/filter_live_migratable_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_live_migratable_test.go @@ -9,6 +9,7 @@ import ( "testing" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -200,7 +201,7 @@ func TestFilterLiveMigratableStep_checkHasSufficientFeatures(t *testing.T) { func TestFilterLiveMigratableStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request compute.PlacementRequest hypervisors []hv1.Hypervisor expectedHosts []string filteredHosts []string @@ -208,15 +209,9 @@ func TestFilterLiveMigratableStep_Run(t *testing.T) { }{ { name: "Not a live migration - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": "rebuild", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Intent: api.EvacuateIntent, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -241,15 +236,9 @@ func TestFilterLiveMigratableStep_Run(t *testing.T) { }, { name: "Live migration without source_host - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": "live_migrate", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Intent: api.LiveMigrationIntent, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -274,16 +263,10 @@ func TestFilterLiveMigratableStep_Run(t *testing.T) { }, { name: "Live migration with compatible hosts", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": "live_migrate", - "source_host": "host1", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Intent: api.LiveMigrationIntent, + SourceHost: "host1", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -330,16 +313,10 @@ func TestFilterLiveMigratableStep_Run(t *testing.T) { }, { name: "Live migration with incompatible architecture", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": "live_migrate", - "source_host": "host1", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Intent: api.LiveMigrationIntent, + SourceHost: "host1", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -380,16 +357,10 @@ func TestFilterLiveMigratableStep_Run(t *testing.T) { }, { name: "Live migration with missing CPU modes", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": "live_migrate", - "source_host": "host1", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Intent: api.LiveMigrationIntent, + SourceHost: "host1", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -436,16 +407,10 @@ func TestFilterLiveMigratableStep_Run(t *testing.T) { }, { name: "Live migration with missing features", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": "live_migrate", - "source_host": "host1", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Intent: api.LiveMigrationIntent, + SourceHost: "host1", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -476,16 +441,10 @@ func TestFilterLiveMigratableStep_Run(t *testing.T) { }, { name: "Live migration with missing devices", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": "live_migrate", - "source_host": "host1", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Intent: api.LiveMigrationIntent, + SourceHost: "host1", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -516,16 +475,10 @@ func TestFilterLiveMigratableStep_Run(t *testing.T) { }, { name: "Live migration with non-existent target host", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": "live_migrate", - "source_host": "host1", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Intent: api.LiveMigrationIntent, + SourceHost: "host1", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host-nonexistent"}, @@ -551,16 +504,10 @@ func TestFilterLiveMigratableStep_Run(t *testing.T) { }, { name: "Live migration with all hosts filtered", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": "live_migrate", - "source_host": "host1", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Intent: api.LiveMigrationIntent, + SourceHost: "host1", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -592,16 +539,10 @@ func TestFilterLiveMigratableStep_Run(t *testing.T) { }, { name: "Live migration with empty host list", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": "live_migrate", - "source_host": "host1", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{}, + request: compute.PlacementRequest{ + Intent: api.LiveMigrationIntent, + SourceHost: "host1", + Hosts: []compute.Host{}, }, hypervisors: []hv1.Hypervisor{ { @@ -634,8 +575,8 @@ func TestFilterLiveMigratableStep_Run(t *testing.T) { Build() step := &FilterLiveMigratableStep{ - BaseFilter: lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ - BaseFilterWeigherPipelineStep: lib.BaseFilterWeigherPipelineStep[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ + BaseFilter: lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ + BaseFilterWeigherPipelineStep: lib.BaseFilterWeigherPipelineStep[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ Client: fakeClient, }, }, @@ -677,16 +618,10 @@ func TestFilterLiveMigratableStep_Run(t *testing.T) { } func TestFilterLiveMigratableStep_Run_SourceHostNotFound(t *testing.T) { - request := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": "live_migrate", - "source_host": "nonexistent-host", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request := compute.PlacementRequest{ + Intent: api.LiveMigrationIntent, + SourceHost: "nonexistent-host", + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -721,8 +656,8 @@ func TestFilterLiveMigratableStep_Run_SourceHostNotFound(t *testing.T) { Build() step := &FilterLiveMigratableStep{ - BaseFilter: lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ - BaseFilterWeigherPipelineStep: lib.BaseFilterWeigherPipelineStep[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ + BaseFilter: lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ + BaseFilterWeigherPipelineStep: lib.BaseFilterWeigherPipelineStep[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ Client: fakeClient, }, }, @@ -738,16 +673,10 @@ func TestFilterLiveMigratableStep_Run_SourceHostNotFound(t *testing.T) { } func TestFilterLiveMigratableStep_Run_ClientError(t *testing.T) { - request := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - SchedulerHints: map[string]any{ - "_nova_check_type": "live_migrate", - "source_host": "host1", - }, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request := compute.PlacementRequest{ + Intent: api.LiveMigrationIntent, + SourceHost: "host1", + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, } @@ -767,8 +696,8 @@ func TestFilterLiveMigratableStep_Run_ClientError(t *testing.T) { Build() step := &FilterLiveMigratableStep{ - BaseFilter: lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ - BaseFilterWeigherPipelineStep: lib.BaseFilterWeigherPipelineStep[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ + BaseFilter: lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ + BaseFilterWeigherPipelineStep: lib.BaseFilterWeigherPipelineStep[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ Client: fakeClient, }, }, diff --git a/internal/scheduling/nova/plugins/filters/filter_quota_enforcement.go b/internal/scheduling/nova/plugins/filters/filter_quota_enforcement.go index 09bc67fa5..c51a0a9e9 100644 --- a/internal/scheduling/nova/plugins/filters/filter_quota_enforcement.go +++ b/internal/scheduling/nova/plugins/filters/filter_quota_enforcement.go @@ -6,12 +6,15 @@ package filters import ( "context" "fmt" - "github.com/go-logr/logr" "math" + "slices" "time" + "github.com/go-logr/logr" + api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/resource" @@ -29,6 +32,21 @@ type FilterQuotaEnforcementOpts struct { // enforcement, and so that newly-rolled-out pipelines never silently start // rejecting requests. Enforce bool `json:"enforce,omitempty"` + + // CRStateFilter selects which CommittedResource states count toward the CR + // portion of quota, i.e. which CRs are deducted when computing PAYG headroom + // (headroom = Quota - sum(CR amounts) - PaygUsage). It MUST mirror the quota + // controller's crStateFilter (see internal/scheduling/reservations/quota/config.go) + // so the filter's CR deduction and the controller's PaygUsage derivation agree + // on what quota already accounts for. Empty means no CR states count; the + // deployment supplies the default (confirmed) via Helm. + CRStateFilter []v1alpha1.CommitmentStatus `json:"crStateFilter,omitempty"` +} + +// crStateIncluded reports whether a CR in the given state counts toward the CR +// portion of quota. +func (o FilterQuotaEnforcementOpts) crStateIncluded(state v1alpha1.CommitmentStatus) bool { + return slices.Contains(o.CRStateFilter, state) } func (FilterQuotaEnforcementOpts) Validate() error { return nil } @@ -40,7 +58,7 @@ func (FilterQuotaEnforcementOpts) Validate() error { return nil } // // 1. CR headroom (RAM only): checks if any CommittedResource CRD for the project has // unused capacity that can accommodate the request -// (AcceptedSpec.Amount - Status.UsedResources >= requestRAM). +// (AcceptedSpec.Amount - Status.Billing.UsedResources >= requestRAM). // If a matching CR has headroom, the request is accepted immediately — PAYG checks // are skipped entirely. // @@ -71,7 +89,7 @@ func (FilterQuotaEnforcementOpts) Validate() error { return nil } // // Disabled by default; activated by adding "filter_quota_enforcement" to the pipeline config. type FilterQuotaEnforcement struct { - lib.BaseFilter[api.ExternalSchedulerRequest, FilterQuotaEnforcementOpts] + lib.BaseFilter[compute.PlacementRequest, FilterQuotaEnforcementOpts] } // recordDecision appends an accept/reject/skip outcome as a named pipeline @@ -92,7 +110,7 @@ func recordDecision(result *lib.FilterWeigherPipelineStepResult, mode, decision, }) } -func (s *FilterQuotaEnforcement) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *FilterQuotaEnforcement) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) mode := "shadow" @@ -105,34 +123,31 @@ func (s *FilterQuotaEnforcement) Run(traceLog logr.Logger, request api.ExternalS // external non-consuming intents (evacuate / live-migrate — VM already exists, // being moved) apart from cortex-internal reservation intents (failover / // committed-resource reservations — new schedules driven by cortex itself). - intent, err := request.GetIntent() - if err == nil { - switch intent { - case api.EvacuateIntent, api.LiveMigrationIntent: - traceLog.V(1).Info("skipping quota enforcement for non-consuming intent", "intent", intent) - recordDecision(result, mode, "accept_skipped_intent_migration", "", "", "") - return result, nil - case api.ReserveForFailoverIntent, api.ReuseFailoverReservationIntent: - traceLog.V(1).Info("skipping quota enforcement for failover reservation intent") - recordDecision(result, mode, "accept_skipped_intent_internal", "", "", "") - return result, nil - case api.ReserveForCommittedResourceIntent: - traceLog.V(1).Info("skipping quota enforcement for committed resource reservation intent") - recordDecision(result, mode, "accept_skipped_intent_internal", "", "", "") - return result, nil - case api.CapacityProbeIntent: - traceLog.V(1).Info("skipping quota enforcement for capacity probe intent") - recordDecision(result, mode, "accept_skipped", "", "", "") - return result, nil - } + switch request.Intent { + case api.EvacuateIntent, api.LiveMigrationIntent: + traceLog.V(1).Info("skipping quota enforcement for non-consuming intent", "intent", request.Intent) + recordDecision(result, mode, "accept_skipped_intent_migration", "", "", "") + return result, nil + case api.ReserveForFailoverIntent, api.ReuseFailoverReservationIntent: + traceLog.V(1).Info("skipping quota enforcement for failover reservation intent") + recordDecision(result, mode, "accept_skipped_intent_internal", "", "", "") + return result, nil + case api.ReserveForCommittedResourceIntent: + traceLog.V(1).Info("skipping quota enforcement for committed resource reservation intent") + recordDecision(result, mode, "accept_skipped_intent_internal", "", "", "") + return result, nil + case api.CapacityProbeIntent: + traceLog.V(1).Info("skipping quota enforcement for capacity probe intent") + recordDecision(result, mode, "accept_skipped", "", "", "") + return result, nil } // Step 2: Extract project, AZ, and hw_version from the request. // Missing-data skips share a single decision bucket; the specific missing // field is in the trace log. - projectID := request.Spec.Data.ProjectID - az := request.Spec.Data.AvailabilityZone - hwVersion := request.Spec.Data.Flavor.Data.ExtraSpecs["quota:hw_version"] + projectID := request.ProjectID + az := request.AvailabilityZone + hwVersion := request.HardwareVersion if projectID == "" { traceLog.V(1).Info("no project ID in request, skipping quota enforcement") @@ -151,12 +166,12 @@ func (s *FilterQuotaEnforcement) Run(traceLog logr.Logger, request api.ExternalS } // Step 3: Compute resource demands from the request. - numInstances := request.Spec.Data.NumInstances + numInstances := request.NumInstances if numInstances == 0 { numInstances = 1 } - memoryMB := request.Spec.Data.Flavor.Data.MemoryMB - vcpus := request.Spec.Data.Flavor.Data.VCPUs + memoryMB := request.Flavor.MemoryMB + vcpus := request.Flavor.VCPUs // MB to GiB: using 1024 as divisor is slightly conservative (overestimates GiB needed). requestRAM := int64(math.Ceil(float64(memoryMB*numInstances) / 1024.0)) requestCores := int64(vcpus * numInstances) //nolint:gosec // VCPUs from flavor specs, realistically bounded @@ -199,7 +214,8 @@ func (s *FilterQuotaEnforcement) Run(traceLog logr.Logger, request api.ExternalS spec = cr.Status.AcceptedSpec } - // Must match project, AZ, flavor group, resource type, and be active. + // Must match project, AZ, flavor group, resource type, and be in a + // quota-counting state (see Options.CRStateFilter). if spec.ProjectID != projectID { continue } @@ -212,7 +228,7 @@ func (s *FilterQuotaEnforcement) Run(traceLog logr.Logger, request api.ExternalS if spec.ResourceType != v1alpha1.CommittedResourceTypeMemory { continue } - if !spec.IsActive() { + if !s.Options.crStateIncluded(spec.State) { continue } now := time.Now() @@ -230,8 +246,8 @@ func (s *FilterQuotaEnforcement) Run(traceLog logr.Logger, request api.ExternalS // Check if this specific CR has headroom for the request. if !crHasHeadroom { usedGiB := int64(0) - if cr.Status.UsedResources != nil { - if usedQty, ok := cr.Status.UsedResources["memory"]; ok { + if cr.Status.Billing != nil { + if usedQty, ok := cr.Status.Billing.UsedResources["memory"]; ok { usedGiB = quantityToGiB(usedQty) } } diff --git a/internal/scheduling/nova/plugins/filters/filter_quota_enforcement_test.go b/internal/scheduling/nova/plugins/filters/filter_quota_enforcement_test.go index 2543c5e20..5ef907611 100644 --- a/internal/scheduling/nova/plugins/filters/filter_quota_enforcement_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_quota_enforcement_test.go @@ -9,6 +9,7 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" "github.com/go-logr/logr" "k8s.io/apimachinery/pkg/api/resource" @@ -18,27 +19,44 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client/fake" ) -func makeQuotaEnforcementRequest(projectID, az, hwVersion string, memoryMB, numInstances uint64, hints map[string]any) api.ExternalSchedulerRequest { - flavor := api.NovaFlavor{ - MemoryMB: memoryMB, - VCPUs: 4, +// intentFromHint maps an old "_nova_check_type" scheduler hint value to the +// typed scheduling intent the plugin now reads. The hints map carries at most +// the "_nova_check_type" key in these tests. +func intentFromHint(hints map[string]any) v1alpha1.SchedulingIntent { + raw, ok := hints["_nova_check_type"] + if !ok { + return "" } - if hwVersion != "" { - flavor.ExtraSpecs = map[string]string{"quota:hw_version": hwVersion} - } else { - flavor.ExtraSpecs = map[string]string{} + switch raw { + case "evacuate": + return api.EvacuateIntent + case "live_migrate": + return api.LiveMigrationIntent + case "reserve_for_failover": + return api.ReserveForFailoverIntent + case "reserve_for_committed_resource": + return api.ReserveForCommittedResourceIntent + case "resize": + return api.ResizeIntent + case "capacity_probe": + return api.CapacityProbeIntent + default: + return "" } - return api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: projectID, - AvailabilityZone: az, - NumInstances: numInstances, - SchedulerHints: hints, - Flavor: api.NovaObject[api.NovaFlavor]{Data: flavor}, - }, +} + +func makeQuotaEnforcementRequest(projectID, az, hwVersion string, memoryMB, numInstances uint64, hints map[string]any) compute.PlacementRequest { + return compute.PlacementRequest{ + ProjectID: projectID, + AvailabilityZone: az, + NumInstances: numInstances, + Intent: intentFromHint(hints), + HardwareVersion: hwVersion, + Flavor: api.FlavorSpec{ + VCPUs: 4, + MemoryMB: memoryMB, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -55,8 +73,10 @@ func TestFilterQuotaEnforcement_Run(t *testing.T) { type tc struct { name string objects []client.Object - request api.ExternalSchedulerRequest + request compute.PlacementRequest enforce bool + // crStateFilter sets Options.CRStateFilter; nil exercises the default (confirmed). + crStateFilter []v1alpha1.CommitmentStatus expectAccept bool @@ -84,7 +104,7 @@ func TestFilterQuotaEnforcement_Run(t *testing.T) { Amount: resource.MustParse("100Gi"), }, Status: v1alpha1.CommittedResourceStatus{ - UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("50Gi")}, + Billing: &v1alpha1.BillingStatus{UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("50Gi")}}, }, }, }, @@ -98,7 +118,7 @@ func TestFilterQuotaEnforcement_Run(t *testing.T) { expectFG: "hana_v2", }, { - name: "ACCEPT: CR has exact headroom (guaranteed state)", + name: "ACCEPT: CR has exact headroom (confirmed state)", objects: []client.Object{ &v1alpha1.CommittedResource{ ObjectMeta: metav1.ObjectMeta{Name: "cr-1"}, @@ -107,11 +127,11 @@ func TestFilterQuotaEnforcement_Run(t *testing.T) { AvailabilityZone: "az-1", FlavorGroupName: "hana_v2", ResourceType: v1alpha1.CommittedResourceTypeMemory, - State: v1alpha1.CommitmentStatusGuaranteed, + State: v1alpha1.CommitmentStatusConfirmed, Amount: resource.MustParse("100Gi"), }, Status: v1alpha1.CommittedResourceStatus{ - UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("90Gi")}, + Billing: &v1alpha1.BillingStatus{UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("90Gi")}}, }, }, }, @@ -138,7 +158,7 @@ func TestFilterQuotaEnforcement_Run(t *testing.T) { Amount: resource.MustParse("50Gi"), }, Status: v1alpha1.CommittedResourceStatus{ - UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("50Gi")}, + Billing: &v1alpha1.BillingStatus{UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("50Gi")}}, }, }, &v1alpha1.ProjectQuota{ @@ -180,7 +200,7 @@ func TestFilterQuotaEnforcement_Run(t *testing.T) { Amount: resource.MustParse("50Gi"), }, Status: v1alpha1.CommittedResourceStatus{ - UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("50Gi")}, + Billing: &v1alpha1.BillingStatus{UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("50Gi")}}, }, }, &v1alpha1.ProjectQuota{ @@ -342,7 +362,7 @@ func TestFilterQuotaEnforcement_Run(t *testing.T) { Amount: resource.MustParse("100Gi"), }, Status: v1alpha1.CommittedResourceStatus{ - UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("0")}, + Billing: &v1alpha1.BillingStatus{UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("0")}}, }, }, &v1alpha1.ProjectQuota{ @@ -384,7 +404,7 @@ func TestFilterQuotaEnforcement_Run(t *testing.T) { Amount: resource.MustParse("100Gi"), }, Status: v1alpha1.CommittedResourceStatus{ - UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("0")}, + Billing: &v1alpha1.BillingStatus{UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("0")}}, }, }, &v1alpha1.ProjectQuota{ @@ -443,6 +463,87 @@ func TestFilterQuotaEnforcement_Run(t *testing.T) { expectAZ: "az-1", expectFG: "hana_v2", }, + { + name: "confirmed CR is deducted from PAYG headroom (reject)", + objects: []client.Object{ + &v1alpha1.CommittedResource{ + ObjectMeta: metav1.ObjectMeta{Name: "cr-confirmed-full"}, + Spec: v1alpha1.CommittedResourceSpec{ + ProjectID: "project-1", + AvailabilityZone: "az-1", + FlavorGroupName: "hana_v2", + ResourceType: v1alpha1.CommittedResourceTypeMemory, + State: v1alpha1.CommitmentStatusConfirmed, + Amount: resource.MustParse("100Gi"), + }, + // Fully used, so it provides no CR headroom itself — the request + // falls through to the PAYG check where its amount is deducted. + Status: v1alpha1.CommittedResourceStatus{ + Billing: &v1alpha1.BillingStatus{UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("100Gi")}}, + }, + }, + &v1alpha1.ProjectQuota{ + ObjectMeta: metav1.ObjectMeta{Name: "quota-project-1-az-1"}, + Spec: v1alpha1.ProjectQuotaSpec{ + ProjectID: "project-1", + AvailabilityZone: "az-1", + Quota: map[string]int64{"hw_version_hana_v2_ram": 150}, + }, + Status: v1alpha1.ProjectQuotaStatus{ + PaygUsage: map[string]int64{"hw_version_hana_v2_ram": 45}, + }, + }, + }, + // headroom = 150 - 100 (confirmed CR) - 45 = 5 < 10 requested → reject. + request: makeQuotaEnforcementRequest("project-1", "az-1", "hana_v2", 10240, 1, nil), + expectAccept: false, + enforce: true, + expectMode: "enforce", + expectDecision: "reject", + expectResource: "ram", + expectAZ: "az-1", + expectFG: "hana_v2", + }, + { + name: "guaranteed CR is excluded from PAYG headroom under default filter (accept)", + objects: []client.Object{ + &v1alpha1.CommittedResource{ + ObjectMeta: metav1.ObjectMeta{Name: "cr-guaranteed-full"}, + Spec: v1alpha1.CommittedResourceSpec{ + ProjectID: "project-1", + AvailabilityZone: "az-1", + FlavorGroupName: "hana_v2", + ResourceType: v1alpha1.CommittedResourceTypeMemory, + State: v1alpha1.CommitmentStatusGuaranteed, + Amount: resource.MustParse("100Gi"), + }, + Status: v1alpha1.CommittedResourceStatus{ + Billing: &v1alpha1.BillingStatus{UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("100Gi")}}, + }, + }, + &v1alpha1.ProjectQuota{ + ObjectMeta: metav1.ObjectMeta{Name: "quota-project-1-az-1"}, + Spec: v1alpha1.ProjectQuotaSpec{ + ProjectID: "project-1", + AvailabilityZone: "az-1", + Quota: map[string]int64{"hw_version_hana_v2_ram": 150}, + }, + Status: v1alpha1.ProjectQuotaStatus{ + PaygUsage: map[string]int64{"hw_version_hana_v2_ram": 45}, + }, + }, + }, + // Default filter (confirmed only) excludes the guaranteed CR: + // headroom = 150 - 0 - 45 = 105 >= 10 requested → accept via PAYG. + request: makeQuotaEnforcementRequest("project-1", "az-1", "hana_v2", 10240, 1, nil), + expectAccept: true, + enforce: true, + expectMode: "enforce", + expectDecision: "accept_payg", + expectResource: "", + expectAZ: "az-1", + expectFG: "hana_v2", + }, { name: "ACCEPT: multiple instances, enough PAYG headroom", objects: []client.Object{ @@ -741,7 +842,7 @@ func TestFilterQuotaEnforcement_Run(t *testing.T) { Amount: resource.MustParse("100Gi"), }, Status: v1alpha1.CommittedResourceStatus{ - UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("50Gi")}, + Billing: &v1alpha1.BillingStatus{UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("50Gi")}}, }, }, &v1alpha1.ProjectQuota{ @@ -906,6 +1007,13 @@ func TestFilterQuotaEnforcement_Run(t *testing.T) { filter := &FilterQuotaEnforcement{} filter.Client = fakeClient filter.Options.Enforce = tt.enforce + // Production supplies the CR state filter via Helm; mirror that + // default here so cases that don't set one exercise confirmed-only. + crStateFilter := tt.crStateFilter + if crStateFilter == nil { + crStateFilter = []v1alpha1.CommitmentStatus{v1alpha1.CommitmentStatusConfirmed} + } + filter.Options.CRStateFilter = crStateFilter result, err := filter.Run(logr.Discard(), tt.request) if err != nil { diff --git a/internal/scheduling/nova/plugins/filters/filter_requested_destination.go b/internal/scheduling/nova/plugins/filters/filter_requested_destination.go index e4fa2af5d..397cfdd88 100644 --- a/internal/scheduling/nova/plugins/filters/filter_requested_destination.go +++ b/internal/scheduling/nova/plugins/filters/filter_requested_destination.go @@ -11,12 +11,13 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" ) type FilterRequestedDestinationStep struct { - lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts] + lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts] } // processRequestedAggregates filters hosts based on the requested aggregates. @@ -101,23 +102,23 @@ func (s *FilterRequestedDestinationStep) processRequestedHost( // The requested destination can include a specific host, aggregates, or both. // When both are specified, aggregate filtering is applied first, followed by // host filtering. -func (s *FilterRequestedDestinationStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *FilterRequestedDestinationStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) // The requested_destination hint is only set by Nova for user-directed placement; - if intent, err := request.GetIntent(); err == nil && slices.Contains([]v1alpha1.SchedulingIntent{ + if slices.Contains([]v1alpha1.SchedulingIntent{ api.ReserveForFailoverIntent, api.ReuseFailoverReservationIntent, api.ReserveForCommittedResourceIntent, api.CapacityProbeIntent, - }, intent) { + }, request.Intent) { return result, nil } - rd := request.Spec.Data.RequestedDestination + rd := request.RequestedDestination if rd == nil { traceLog.V(1).Info("no requested_destination in request, skipping filter") return result, nil } - if len(rd.Data.Aggregates) == 0 && rd.Data.Host == "" { + if len(rd.Aggregates) == 0 && rd.Host == "" { traceLog.V(1).Info("requested_destination has no host or aggregates, skipping filter") return result, nil } @@ -133,8 +134,8 @@ func (s *FilterRequestedDestinationStep) Run(traceLog logr.Logger, request api.E } hvsByName[hv.Name] = hv } - s.processRequestedAggregates(traceLog, rd.Data.Aggregates, hvsByName, result.Activations) - s.processRequestedHost(traceLog, rd.Data.Host, result.Activations) + s.processRequestedAggregates(traceLog, rd.Aggregates, hvsByName, result.Activations) + s.processRequestedHost(traceLog, rd.Host, result.Activations) return result, nil } diff --git a/internal/scheduling/nova/plugins/filters/filter_requested_destination_test.go b/internal/scheduling/nova/plugins/filters/filter_requested_destination_test.go index a145190b9..233e97108 100644 --- a/internal/scheduling/nova/plugins/filters/filter_requested_destination_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_requested_destination_test.go @@ -9,6 +9,8 @@ import ( "testing" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -21,7 +23,7 @@ import ( func TestFilterRequestedDestinationStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request compute.PlacementRequest hypervisors []hv1.Hypervisor expectedHosts []string filteredHosts []string @@ -29,13 +31,9 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }{ { name: "No requested_destination - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: nil, - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + RequestedDestination: nil, + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -61,18 +59,12 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }, { name: "Empty requested_destination - all hosts pass", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - Aggregates: []string{}, - Host: "", - }, - }, - }, + request: compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + Aggregates: []string{}, + Host: "", }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -93,17 +85,11 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }, { name: "Filter by specific host", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - Host: "host2", - }, - }, - }, + request: compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + Host: "host2", }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -129,17 +115,11 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }, { name: "Filter by single aggregate - hosts in spec aggregates", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - Aggregates: []string{"aggregate1"}, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + Aggregates: []string{"aggregate1"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -165,17 +145,11 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }, { name: "Filter by single aggregate - hosts in status aggregates", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - Aggregates: []string{"aggregate1"}, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + Aggregates: []string{"aggregate1"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -201,18 +175,12 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }, { name: "Filter by multiple aggregates with OR syntax (comma-separated)", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - // "aggregate1,aggregate3" means host must be in aggregate1 OR aggregate3 - Aggregates: []string{"aggregate1,aggregate3"}, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + // "aggregate1,aggregate3" means host must be in aggregate1 OR aggregate3 + Aggregates: []string{"aggregate1,aggregate3"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -243,18 +211,12 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }, { name: "Filter by multiple aggregates with AND logic", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - // ["aggregate1", "aggregate2"] means host must be in aggregate1 AND aggregate2 - Aggregates: []string{"aggregate1", "aggregate2"}, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + // ["aggregate1", "aggregate2"] means host must be in aggregate1 AND aggregate2 + Aggregates: []string{"aggregate1", "aggregate2"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -280,18 +242,12 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }, { name: "Filter by mixed AND/OR aggregates", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - // Host must be in (aggregate1 OR aggregate2) AND aggregate3 - Aggregates: []string{"aggregate1,aggregate2", "aggregate3"}, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + // Host must be in (aggregate1 OR aggregate2) AND aggregate3 + Aggregates: []string{"aggregate1,aggregate2", "aggregate3"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -322,18 +278,12 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }, { name: "Filter by aggregates with OR syntax - hosts in both spec and status", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - // "aggregate1,aggregate2" means host must be in aggregate1 OR aggregate2 - Aggregates: []string{"aggregate1,aggregate2"}, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + // "aggregate1,aggregate2" means host must be in aggregate1 OR aggregate2 + Aggregates: []string{"aggregate1,aggregate2"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -359,18 +309,12 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }, { name: "Filter by both host and aggregates - host takes precedence", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - Aggregates: []string{"aggregate1"}, - Host: "host2", - }, - }, - }, + request: compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + Aggregates: []string{"aggregate1"}, + Host: "host2", }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -396,17 +340,11 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }, { name: "Host not found in hypervisors - filtered out", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - Aggregates: []string{"aggregate1"}, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + Aggregates: []string{"aggregate1"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host-nonexistent"}, {ComputeHost: "host2"}, @@ -428,17 +366,11 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }, { name: "Host with no aggregates - filtered out when aggregates requested", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - Aggregates: []string{"aggregate1"}, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + Aggregates: []string{"aggregate1"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -464,17 +396,11 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }, { name: "All hosts filtered by aggregate", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - Aggregates: []string{"nonexistent-aggregate"}, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + Aggregates: []string{"nonexistent-aggregate"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -495,17 +421,11 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }, { name: "Requested host not in candidates", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - Host: "host-not-in-list", - }, - }, - }, + request: compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + Host: "host-not-in-list", }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, }, @@ -530,17 +450,11 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { }, { name: "Empty host list", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - Aggregates: []string{"aggregate1"}, - }, - }, - }, + request: compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + Aggregates: []string{"aggregate1"}, }, - Hosts: []api.ExternalSchedulerHost{}, + Hosts: []compute.Host{}, }, hypervisors: []hv1.Hypervisor{ { @@ -571,8 +485,8 @@ func TestFilterRequestedDestinationStep_Run(t *testing.T) { Build() step := &FilterRequestedDestinationStep{ - BaseFilter: lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ - BaseFilterWeigherPipelineStep: lib.BaseFilterWeigherPipelineStep[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ + BaseFilter: lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ + BaseFilterWeigherPipelineStep: lib.BaseFilterWeigherPipelineStep[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ Client: fakeClient, }, }, @@ -740,17 +654,11 @@ func TestFilterRequestedDestinationStep_processRequestedHost(t *testing.T) { } func TestFilterRequestedDestinationStep_Run_ClientError(t *testing.T) { - request := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{ - Aggregates: []string{"aggregate1"}, - }, - }, - }, + request := compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{ + Aggregates: []string{"aggregate1"}, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, }, } @@ -770,8 +678,8 @@ func TestFilterRequestedDestinationStep_Run_ClientError(t *testing.T) { Build() step := &FilterRequestedDestinationStep{ - BaseFilter: lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ - BaseFilterWeigherPipelineStep: lib.BaseFilterWeigherPipelineStep[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ + BaseFilter: lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ + BaseFilterWeigherPipelineStep: lib.BaseFilterWeigherPipelineStep[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts]{ Client: fakeClient, }, }, @@ -796,18 +704,17 @@ func TestFilterRequestedDestinationStep_SkipsForNonPlacementIntent(t *testing.T) step := &FilterRequestedDestinationStep{} step.Client = fake.NewClientBuilder().WithScheme(scheme).WithObjects(objects...).Build() - for _, intent := range []string{"reserve_for_failover", "reuse_failover_reservation", "reserve_for_committed_resource", "capacity_probe"} { - t.Run(intent, func(t *testing.T) { - request := api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - RequestedDestination: &api.NovaObject[api.NovaRequestedDestination]{ - Data: api.NovaRequestedDestination{Host: "host1"}, - }, - SchedulerHints: map[string]any{"_nova_check_type": intent}, - }, - }, - Hosts: []api.ExternalSchedulerHost{{ComputeHost: "host1"}, {ComputeHost: "host2"}}, + for _, intent := range []v1alpha1.SchedulingIntent{ + api.ReserveForFailoverIntent, + api.ReuseFailoverReservationIntent, + api.ReserveForCommittedResourceIntent, + api.CapacityProbeIntent, + } { + t.Run(string(intent), func(t *testing.T) { + request := compute.PlacementRequest{ + RequestedDestination: &api.RequestedDestination{Host: "host1"}, + Intent: intent, + Hosts: []compute.Host{{ComputeHost: "host1"}, {ComputeHost: "host2"}}, } result, err := step.Run(logr.Discard(), request) if err != nil { diff --git a/internal/scheduling/nova/plugins/filters/filter_status_conditions.go b/internal/scheduling/nova/plugins/filters/filter_status_conditions.go index 310a238c2..5bcbfd75f 100644 --- a/internal/scheduling/nova/plugins/filters/filter_status_conditions.go +++ b/internal/scheduling/nova/plugins/filters/filter_status_conditions.go @@ -7,7 +7,7 @@ import ( "context" "github.com/go-logr/logr" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "k8s.io/apimachinery/pkg/api/meta" @@ -15,12 +15,12 @@ import ( ) type FilterStatusConditionsStep struct { - lib.BaseFilter[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts] + lib.BaseFilter[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts] } // Check that all status conditions meet the expected values, for example, // that the hypervisor is ready and not disabled. -func (s *FilterStatusConditionsStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *FilterStatusConditionsStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) hvs := &hv1.HypervisorList{} diff --git a/internal/scheduling/nova/plugins/filters/filter_status_conditions_test.go b/internal/scheduling/nova/plugins/filters/filter_status_conditions_test.go index 7862b6544..b38bebf12 100644 --- a/internal/scheduling/nova/plugins/filters/filter_status_conditions_test.go +++ b/internal/scheduling/nova/plugins/filters/filter_status_conditions_test.go @@ -7,7 +7,7 @@ import ( "github.com/go-logr/logr" "testing" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -214,14 +214,14 @@ func TestFilterStatusConditionsStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedHosts []string filteredHosts []string }{ { name: "Filter hosts with all conditions met", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -233,8 +233,8 @@ func TestFilterStatusConditionsStep_Run(t *testing.T) { }, { name: "Host not ready should be filtered", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host2"}, }, }, @@ -243,8 +243,8 @@ func TestFilterStatusConditionsStep_Run(t *testing.T) { }, { name: "Terminating host should be filtered", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host3"}, }, }, @@ -253,8 +253,8 @@ func TestFilterStatusConditionsStep_Run(t *testing.T) { }, { name: "Tainted host should not be filtered", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host4"}, }, }, @@ -263,8 +263,8 @@ func TestFilterStatusConditionsStep_Run(t *testing.T) { }, { name: "Disabled hypervisor should be filtered", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host8"}, }, }, @@ -273,8 +273,8 @@ func TestFilterStatusConditionsStep_Run(t *testing.T) { }, { name: "Host with optional conditions in any state should pass", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host5"}, }, }, @@ -283,8 +283,8 @@ func TestFilterStatusConditionsStep_Run(t *testing.T) { }, { name: "Host missing Ready condition should be kept", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host6"}, }, }, @@ -293,8 +293,8 @@ func TestFilterStatusConditionsStep_Run(t *testing.T) { }, { name: "Host with no conditions should be kept", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host7"}, }, }, @@ -303,16 +303,16 @@ func TestFilterStatusConditionsStep_Run(t *testing.T) { }, { name: "Empty host list", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{}, + request: compute.PlacementRequest{ + Hosts: []compute.Host{}, }, expectedHosts: []string{}, filteredHosts: []string{}, }, { name: "Host not in database", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host-unknown"}, }, @@ -322,8 +322,8 @@ func TestFilterStatusConditionsStep_Run(t *testing.T) { }, { name: "Mixed condition states", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: compute.PlacementRequest{ + Hosts: []compute.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, diff --git a/internal/scheduling/nova/plugins/filters/zz_index.go b/internal/scheduling/nova/plugins/filters/zz_index.go index 0f63f8842..b72812ffd 100644 --- a/internal/scheduling/nova/plugins/filters/zz_index.go +++ b/internal/scheduling/nova/plugins/filters/zz_index.go @@ -4,11 +4,11 @@ package filters import ( - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" ) -type NovaFilter = lib.Filter[api.ExternalSchedulerRequest] +type NovaFilter = lib.Filter[compute.PlacementRequest] // Configuration of filters supported by the nova scheduler. var Index = map[string]func() NovaFilter{} diff --git a/internal/scheduling/nova/plugins/weighers/kvm_binpack.go b/internal/scheduling/nova/plugins/weighers/kvm_binpack.go index a314dc5e8..9f19b47d1 100644 --- a/internal/scheduling/nova/plugins/weighers/kvm_binpack.go +++ b/internal/scheduling/nova/plugins/weighers/kvm_binpack.go @@ -9,7 +9,7 @@ import ( "fmt" "slices" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "github.com/go-logr/logr" @@ -65,11 +65,11 @@ func (o KVMBinpackStepOpts) Validate() error { // Explanation of the algorithm: https://volcano.sh/en/docs/plugins/#binpack type KVMBinpackStep struct { // Base weigher providing common functionality. - lib.BaseWeigher[api.ExternalSchedulerRequest, KVMBinpackStepOpts] + lib.BaseWeigher[compute.PlacementRequest, KVMBinpackStepOpts] } // Run this weigher in the pipeline after filters have been executed. -func (s *KVMBinpackStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *KVMBinpackStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) result.Statistics["binpack score"] = s.PrepareStats(request, "float") @@ -141,14 +141,14 @@ func (s *KVMBinpackStep) Run(traceLog logr.Logger, request api.ExternalScheduler } // calcVMResources calculates the total resource requests for the VM to be scheduled. -func (s *KVMBinpackStep) calcVMResources(req api.ExternalSchedulerRequest) map[hv1.ResourceName]resource.Quantity { +func (s *KVMBinpackStep) calcVMResources(req compute.PlacementRequest) map[hv1.ResourceName]resource.Quantity { resources := make(map[hv1.ResourceName]resource.Quantity) - resourcesMemBytes := int64(req.Spec.Data.Flavor.Data.MemoryMB) * (1024 * 1024) //nolint:gosec // memory values are bounded by Nova - resourcesMemBytes *= int64(req.Spec.Data.NumInstances) //nolint:gosec // instance count is bounded by Nova + resourcesMemBytes := int64(req.Flavor.MemoryMB) * (1024 * 1024) //nolint:gosec // memory values are bounded by Nova + resourcesMemBytes *= int64(req.NumInstances) //nolint:gosec // instance count is bounded by Nova resources[hv1.ResourceMemory] = *resource. NewQuantity(resourcesMemBytes, resource.BinarySI) - resourcesCPU := int64(req.Spec.Data.Flavor.Data.VCPUs) //nolint:gosec // vCPU values are bounded by Nova - resourcesCPU *= int64(req.Spec.Data.NumInstances) //nolint:gosec // instance count is bounded by Nova + resourcesCPU := int64(req.Flavor.VCPUs) //nolint:gosec // vCPU values are bounded by Nova + resourcesCPU *= int64(req.NumInstances) //nolint:gosec // instance count is bounded by Nova resources[hv1.ResourceCPU] = *resource. NewQuantity(resourcesCPU, resource.DecimalSI) return resources diff --git a/internal/scheduling/nova/plugins/weighers/kvm_binpack_test.go b/internal/scheduling/nova/plugins/weighers/kvm_binpack_test.go index 421b53042..021d013db 100644 --- a/internal/scheduling/nova/plugins/weighers/kvm_binpack_test.go +++ b/internal/scheduling/nova/plugins/weighers/kvm_binpack_test.go @@ -9,6 +9,7 @@ import ( "testing" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -34,28 +35,10 @@ func newHypervisor(name, capacityCPU, capacityMem, allocationCPU, allocationMem } } -func newBinpackRequest(memoryMB, vcpus, numInstances uint64, hosts []string) api.ExternalSchedulerRequest { - hostList := make([]api.ExternalSchedulerHost, len(hosts)) +func newBinpackRequest(memoryMB, vcpus, numInstances uint64, hosts []string) compute.PlacementRequest { + hostList := make([]compute.Host, len(hosts)) for i, h := range hosts { - hostList[i] = api.ExternalSchedulerHost{ComputeHost: h} - } - - extraSpecs := map[string]string{ - "capabilities:hypervisor_type": "qemu", - } - - spec := api.NovaSpec{ - ProjectID: "project-A", - InstanceUUID: "instance-123", - NumInstances: numInstances, - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - Name: "m1.large", - VCPUs: vcpus, - MemoryMB: memoryMB, - ExtraSpecs: extraSpecs, - }, - }, + hostList[i] = compute.Host{ComputeHost: h} } weights := make(map[string]float64) @@ -63,8 +46,15 @@ func newBinpackRequest(memoryMB, vcpus, numInstances uint64, hosts []string) api weights[h] = 1.0 } - return api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{Data: spec}, + return compute.PlacementRequest{ + ProjectID: "project-A", + InstanceUUID: "instance-123", + NumInstances: numInstances, + HypervisorFamily: api.HypervisorFamilyKVM, + Flavor: api.FlavorSpec{ + VCPUs: vcpus, + MemoryMB: memoryMB, + }, Hosts: hostList, Weights: weights, } @@ -183,7 +173,7 @@ func TestKVMBinpackStep_Run(t *testing.T) { tests := []struct { name string hypervisors []*hv1.Hypervisor - request api.ExternalSchedulerRequest + request compute.PlacementRequest opts KVMBinpackStepOpts expectedWeights map[string]float64 wantErr bool @@ -485,7 +475,7 @@ func TestKVMBinpackStep_Run(t *testing.T) { func TestKVMBinpackStep_calcVMResources(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedMemBytes int64 expectedCPU int64 }{ diff --git a/internal/scheduling/nova/plugins/weighers/kvm_committed_resource_reservation.go b/internal/scheduling/nova/plugins/weighers/kvm_committed_resource_reservation.go index fc0c8c6b9..2c626b7ad 100644 --- a/internal/scheduling/nova/plugins/weighers/kvm_committed_resource_reservation.go +++ b/internal/scheduling/nova/plugins/weighers/kvm_committed_resource_reservation.go @@ -8,10 +8,10 @@ import ( hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "github.com/go-logr/logr" - "k8s.io/apimachinery/pkg/api/resource" api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" ) @@ -47,31 +47,31 @@ func (o KVMCommittedResourceReservationOpts) GetDefaultHostWeight() float64 { // Hosts that have a ready Reservation matching the request's project and flavor group, // with enough free memory capacity for the requested VM, receive a higher weight. type KVMCommittedResourceReservationStep struct { - lib.BaseWeigher[api.ExternalSchedulerRequest, KVMCommittedResourceReservationOpts] + lib.BaseWeigher[compute.PlacementRequest, KVMCommittedResourceReservationOpts] } // Run the weigher step. // Hosts with a matching CommittedResourceReservation (project + flavor group + free capacity) get a higher weight. -func (s *KVMCommittedResourceReservationStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *KVMCommittedResourceReservationStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) // Skip for failover reservation scheduling — CR reservations should not influence // where failover capacity is placed. - if intent, err := request.GetIntent(); err == nil && intent == api.ReserveForFailoverIntent { + if request.Intent == api.ReserveForFailoverIntent { traceLog.V(1).Info("skipping committed resource reservation weigher for failover reservation intent") return result, nil } - projectID := request.Spec.Data.ProjectID - az := request.Spec.Data.AvailabilityZone - resourceGroup, err := request.Spec.Data.GetSchedulerHintStr(api.HintKeyResourceGroup) - if err != nil || resourceGroup == "" { + projectID := request.ProjectID + az := request.AvailabilityZone + resourceGroup := request.ResourceGroup + if resourceGroup == "" { traceLog.V(1).Info("skipping committed resource reservation weigher: no resource group in request") - return result, nil //nolint:nilerr + return result, nil } // Flavor memory in bytes for free-capacity check. - flavorMemoryBytes := int64(request.Spec.Data.Flavor.Data.MemoryMB) * 1024 * 1024 //nolint:gosec + flavorMemoryBytes := int64(request.Flavor.MemoryMB) * 1024 * 1024 //nolint:gosec var reservations v1alpha1.ReservationList if err := s.Client.List(context.Background(), &reservations); err != nil { @@ -126,19 +126,24 @@ func (s *KVMCommittedResourceReservationStep) Run(traceLog logr.Logger, request return result, nil } -// freeMemoryBytes returns the free memory in bytes for a committed resource reservation: -// total reserved memory minus the sum of memory already allocated to running VMs. +// freeMemoryBytes returns the free memory in bytes for a committed resource reservation. func freeMemoryBytes(res *v1alpha1.Reservation) int64 { - total := res.Spec.Resources[hv1.ResourceMemory] - var used resource.Quantity - if res.Spec.CommittedResourceReservation != nil { - for _, alloc := range res.Spec.CommittedResourceReservation.Allocations { - if mem, ok := alloc.Resources[hv1.ResourceMemory]; ok { - used.Add(mem) - } + totalMemQ := res.Spec.Resources[hv1.ResourceMemory] + total := totalMemQ.Value() + if res.Status.CommittedResourceReservation == nil { + return total + } + var confirmed int64 + for _, alloc := range res.Status.CommittedResourceReservation.Allocation { + if used, ok := alloc.Resources[hv1.ResourceMemory]; ok { + confirmed += used.Value() } } - return total.Value() - used.Value() + remaining := total - confirmed + if remaining < 0 { + return 0 + } + return remaining } func init() { diff --git a/internal/scheduling/nova/plugins/weighers/kvm_committed_resource_reservation_test.go b/internal/scheduling/nova/plugins/weighers/kvm_committed_resource_reservation_test.go index 421c6b75b..74d361abf 100644 --- a/internal/scheduling/nova/plugins/weighers/kvm_committed_resource_reservation_test.go +++ b/internal/scheduling/nova/plugins/weighers/kvm_committed_resource_reservation_test.go @@ -9,7 +9,9 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/controller-runtime/pkg/client" @@ -17,17 +19,7 @@ import ( ) func newCRReservation(name, host, projectID, resourceGroup, az, totalMemory string, allocatedMemory ...string) *v1alpha1.Reservation { - allocs := make(map[string]v1alpha1.CommittedResourceAllocation) - for i, mem := range allocatedMemory { - vmUUID := "vm-" + string(rune('a'+i)) - allocs[vmUUID] = v1alpha1.CommittedResourceAllocation{ - CreationTimestamp: metav1.Now(), - Resources: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceMemory: resource.MustParse(mem), - }, - } - } - return &v1alpha1.Reservation{ + res := &v1alpha1.Reservation{ ObjectMeta: metav1.ObjectMeta{Name: name}, Spec: v1alpha1.ReservationSpec{ Type: v1alpha1.ReservationTypeCommittedResource, @@ -39,7 +31,6 @@ func newCRReservation(name, host, projectID, resourceGroup, az, totalMemory stri CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ ProjectID: projectID, ResourceGroup: resourceGroup, - Allocations: allocs, }, }, Status: v1alpha1.ReservationStatus{ @@ -51,30 +42,40 @@ func newCRReservation(name, host, projectID, resourceGroup, az, totalMemory stri Host: host, }, } + if len(allocatedMemory) > 0 { + statusAllocs := make(map[string]v1alpha1.CommittedResourceAllocation, len(allocatedMemory)) + for i := range allocatedMemory { + vmUUID := "vm-" + string(rune('a'+i)) + statusAllocs[vmUUID] = v1alpha1.CommittedResourceAllocation{ + ObjectRef: corev1.ObjectReference{Name: vmUUID}, + Resources: map[hv1.ResourceName]resource.Quantity{ + hv1.ResourceMemory: resource.MustParse(allocatedMemory[i]), + }, + } + } + res.Status.CommittedResourceReservation = &v1alpha1.CommittedResourceReservationStatus{ + Allocation: statusAllocs, + } + } + return res } -func newCRRequest(hosts []string) api.ExternalSchedulerRequest { - hostList := make([]api.ExternalSchedulerHost, len(hosts)) +func newCRRequest(hosts []string) compute.PlacementRequest { + hostList := make([]compute.Host, len(hosts)) for i, h := range hosts { - hostList[i] = api.ExternalSchedulerHost{ComputeHost: h} - } - hints := map[string]any{ - api.HintKeyResourceGroup: []any{"group-2101"}, - } - spec := api.NovaSpec{ - ProjectID: "project-A", - AvailabilityZone: "qa-de-1a", - SchedulerHints: hints, - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{MemoryMB: 8 * 1024}, - }, + hostList[i] = compute.Host{ComputeHost: h} } weights := make(map[string]float64, len(hosts)) for _, h := range hosts { weights[h] = 1.0 } - return api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{Data: spec}, + return compute.PlacementRequest{ + ProjectID: "project-A", + AvailabilityZone: "qa-de-1a", + ResourceGroup: "group-2101", + Flavor: api.FlavorSpec{ + MemoryMB: 8 * 1024, + }, Hosts: hostList, Weights: weights, } @@ -88,7 +89,7 @@ func TestKVMCommittedResourceReservationStep_Run(t *testing.T) { tests := []struct { name string reservations []*v1alpha1.Reservation - request api.ExternalSchedulerRequest + request compute.PlacementRequest opts KVMCommittedResourceReservationOpts expectedWeights map[string]float64 }{ @@ -111,7 +112,7 @@ func TestKVMCommittedResourceReservationStep_Run(t *testing.T) { expectedWeights: map[string]float64{"host1": 0.1, "host2": 0.1}, }, { - name: "reservation partially allocated with enough free capacity", + name: "reservation with confirmed VM: free capacity proportional to allocation size", reservations: []*v1alpha1.Reservation{ newCRReservation("res-1", "host1", "project-A", "group-2101", "qa-de-1a", "16Gi", "4Gi"), }, @@ -171,9 +172,9 @@ func TestKVMCommittedResourceReservationStep_Run(t *testing.T) { { name: "no resource group hint skips weigher", reservations: []*v1alpha1.Reservation{newCRReservation("res-1", "host1", "project-A", "group-2101", "qa-de-1a", "16Gi")}, - request: func() api.ExternalSchedulerRequest { + request: func() compute.PlacementRequest { r := newCRRequest([]string{"host1", "host2"}) - r.Spec.Data.SchedulerHints = nil + r.ResourceGroup = "" return r }(), opts: KVMCommittedResourceReservationOpts{ReservationHostWeight: floatPtr(1.0), DefaultHostWeight: floatPtr(0.1)}, @@ -210,19 +211,15 @@ func TestKVMCommittedResourceReservationStep_Run(t *testing.T) { reservations: []*v1alpha1.Reservation{ newCRReservation("res-1", "host1", "project-A", "group-2101", "qa-de-1a", "16Gi"), }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project-A", - AvailabilityZone: "qa-de-1a", - Flavor: api.NovaObject[api.NovaFlavor]{Data: api.NovaFlavor{MemoryMB: 4096}}, - SchedulerHints: map[string]any{ - "_nova_check_type": string(api.ReserveForFailoverIntent), - api.HintKeyResourceGroup: "group-2101", - }, - }, + request: compute.PlacementRequest{ + ProjectID: "project-A", + AvailabilityZone: "qa-de-1a", + Intent: api.ReserveForFailoverIntent, + ResourceGroup: "group-2101", + Flavor: api.FlavorSpec{ + MemoryMB: 4096, }, - Hosts: []api.ExternalSchedulerHost{{ComputeHost: "host1"}, {ComputeHost: "host2"}}, + Hosts: []compute.Host{{ComputeHost: "host1"}, {ComputeHost: "host2"}}, }, opts: KVMCommittedResourceReservationOpts{ReservationHostWeight: floatPtr(1.0), DefaultHostWeight: floatPtr(0.1)}, expectedWeights: map[string]float64{"host1": 0.0, "host2": 0.0}, diff --git a/internal/scheduling/nova/plugins/weighers/kvm_cr_migration_slot.go b/internal/scheduling/nova/plugins/weighers/kvm_cr_migration_slot.go index 9c897a61c..92ac4c057 100644 --- a/internal/scheduling/nova/plugins/weighers/kvm_cr_migration_slot.go +++ b/internal/scheduling/nova/plugins/weighers/kvm_cr_migration_slot.go @@ -11,6 +11,7 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" "github.com/cobaltcore-dev/cortex/internal/scheduling/nova/crs" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" @@ -58,24 +59,23 @@ func (o KVMCRMigrationSlotOpts) GetDefaultHostWeight() float64 { // // Only activates for LiveMigrationIntent. type KVMCRMigrationSlotStep struct { - lib.BaseWeigher[api.ExternalSchedulerRequest, KVMCRMigrationSlotOpts] + lib.BaseWeigher[compute.PlacementRequest, KVMCRMigrationSlotOpts] } func (s *KVMCRMigrationSlotStep) Run( traceLog logr.Logger, - request api.ExternalSchedulerRequest, + request compute.PlacementRequest, ) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) - intent, err := request.GetIntent() - if err != nil || intent != api.LiveMigrationIntent { + if request.Intent != api.LiveMigrationIntent { traceLog.V(1).Info("not a live migration, skipping CR migration slot weigher") - return result, nil //nolint:nilerr + return result, nil } - instanceUUID := request.Spec.Data.InstanceUUID - projectID := request.Spec.Data.ProjectID + instanceUUID := request.InstanceUUID + projectID := request.ProjectID var allReservations v1alpha1.ReservationList if err := s.Client.List(context.Background(), &allReservations, @@ -94,7 +94,7 @@ func (s *KVMCRMigrationSlotStep) Run( if res.Status.CommittedResourceReservation == nil || res.Spec.CommittedResourceReservation == nil { continue } - if _, ok := res.Status.CommittedResourceReservation.Allocations[instanceUUID]; ok { + if _, ok := res.Status.CommittedResourceReservation.Allocation[instanceUUID]; ok { sourceSlot = res break } diff --git a/internal/scheduling/nova/plugins/weighers/kvm_cr_migration_slot_test.go b/internal/scheduling/nova/plugins/weighers/kvm_cr_migration_slot_test.go index 3156c6699..d3f73416d 100644 --- a/internal/scheduling/nova/plugins/weighers/kvm_cr_migration_slot_test.go +++ b/internal/scheduling/nova/plugins/weighers/kvm_cr_migration_slot_test.go @@ -9,7 +9,9 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/controller-runtime/pkg/client" @@ -34,22 +36,16 @@ func newCRMigrationSlotWeigher(t *testing.T, opts KVMCRMigrationSlotOpts, objs . } // migrationRequest builds a live-migration request from the given candidate hosts. -func migrationRequest(hosts ...string) api.ExternalSchedulerRequest { - hostList := make([]api.ExternalSchedulerHost, len(hosts)) +func migrationRequest(hosts ...string) compute.PlacementRequest { + hostList := make([]compute.Host, len(hosts)) for i, h := range hosts { - hostList[i] = api.ExternalSchedulerHost{ComputeHost: h} + hostList[i] = compute.Host{ComputeHost: h} } - return api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: testInstanceUUID, - ProjectID: testProjectID, - SchedulerHints: map[string]any{ - "_nova_check_type": "live_migrate", - }, - }, - }, - Hosts: hostList, + return compute.PlacementRequest{ + InstanceUUID: testInstanceUUID, + ProjectID: testProjectID, + Intent: api.LiveMigrationIntent, + Hosts: hostList, } } @@ -79,7 +75,14 @@ func confirmedSourceSlot(memory string) *v1alpha1.Reservation { {Type: v1alpha1.ReservationConditionReady, Status: metav1.ConditionTrue, Reason: "ReservationActive"}, }, CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - Allocations: map[string]string{testInstanceUUID: testSourceHost}, + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + testInstanceUUID: { + ObjectRef: corev1.ObjectReference{Name: testInstanceUUID}, + Resources: map[hv1.ResourceName]resource.Quantity{ + hv1.ResourceMemory: resource.MustParse(memory), + }, + }, + }, }, }, } @@ -135,7 +138,7 @@ func TestKVMCRMigrationSlotStep_Run(t *testing.T) { tests := []struct { name string objects []client.Object - request api.ExternalSchedulerRequest + request compute.PlacementRequest opts KVMCRMigrationSlotOpts expectedWeights map[string]float64 }{ @@ -144,15 +147,11 @@ func TestKVMCRMigrationSlotStep_Run(t *testing.T) { objects: []client.Object{ confirmedSourceSlot("16Gi"), }, - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: testInstanceUUID, - ProjectID: testProjectID, - // no _nova_check_type → CreateIntent - }, - }, - Hosts: []api.ExternalSchedulerHost{{ComputeHost: "host-a"}, {ComputeHost: "host-b"}}, + request: compute.PlacementRequest{ + InstanceUUID: testInstanceUUID, + ProjectID: testProjectID, + // no Intent → CreateIntent + Hosts: []compute.Host{{ComputeHost: "host-a"}, {ComputeHost: "host-b"}}, }, opts: defaultOpts, expectedWeights: map[string]float64{"host-a": 0.0, "host-b": 0.0}, diff --git a/internal/scheduling/nova/plugins/weighers/kvm_failover_evacuation.go b/internal/scheduling/nova/plugins/weighers/kvm_failover_evacuation.go index 3088496b0..f213ae40b 100644 --- a/internal/scheduling/nova/plugins/weighers/kvm_failover_evacuation.go +++ b/internal/scheduling/nova/plugins/weighers/kvm_failover_evacuation.go @@ -10,6 +10,7 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" ) @@ -45,22 +46,21 @@ func (o KVMFailoverEvacuationOpts) GetDefaultHostWeight() float64 { // Hosts that match a failover reservation for the VM being scheduled get a higher weight, // encouraging placement on pre-reserved failover capacity. type KVMFailoverEvacuationStep struct { - lib.BaseWeigher[api.ExternalSchedulerRequest, KVMFailoverEvacuationOpts] + lib.BaseWeigher[compute.PlacementRequest, KVMFailoverEvacuationOpts] } // Run the weigher step. // For evacuation requests, hosts matching a failover reservation where the VM is in Allocations get a higher weight. // For non-evacuation requests (e.g., live migration, rebuild), this weigher has no effect. -func (s *KVMFailoverEvacuationStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *KVMFailoverEvacuationStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) - intent, err := request.GetIntent() - if err != nil || intent != api.EvacuateIntent { + if request.Intent != api.EvacuateIntent { traceLog.V(1).Info("skipping failover weigher for non-evacuation request") - return result, nil //nolint:nilerr // intentionally skip weigher on error + return result, nil } - instanceUUID := request.Spec.Data.InstanceUUID + instanceUUID := request.InstanceUUID // Fetch all reservations var reservations v1alpha1.ReservationList diff --git a/internal/scheduling/nova/plugins/weighers/kvm_failover_evacuation_test.go b/internal/scheduling/nova/plugins/weighers/kvm_failover_evacuation_test.go index 16299100b..8667a9ff9 100644 --- a/internal/scheduling/nova/plugins/weighers/kvm_failover_evacuation_test.go +++ b/internal/scheduling/nova/plugins/weighers/kvm_failover_evacuation_test.go @@ -9,6 +9,7 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -112,48 +113,34 @@ func parseMemoryToMB(memory string) uint64 { return uint64(bytes / (1024 * 1024)) //nolint:gosec // test code } -func newNovaRequest(instanceUUID string, evacuation bool, hosts []string) api.ExternalSchedulerRequest { //nolint:unparam // instanceUUID varies in real usage - hostList := make([]api.ExternalSchedulerHost, len(hosts)) +func newNovaRequest(instanceUUID string, evacuation bool, hosts []string) compute.PlacementRequest { //nolint:unparam // instanceUUID varies in real usage + hostList := make([]compute.Host, len(hosts)) for i, h := range hosts { - hostList[i] = api.ExternalSchedulerHost{ComputeHost: h} + hostList[i] = compute.Host{ComputeHost: h} } - extraSpecs := map[string]string{ - "capabilities:hypervisor_type": "qemu", + req := compute.PlacementRequest{ + ProjectID: "project-A", + InstanceUUID: instanceUUID, + NumInstances: 1, + HypervisorFamily: api.HypervisorFamilyKVM, + Flavor: api.FlavorSpec{ + VCPUs: 4, + MemoryMB: parseMemoryToMB("8Gi"), + }, + Hosts: hostList, } - - var schedulerHints map[string]any if evacuation { - schedulerHints = map[string]any{ - "_nova_check_type": []any{"evacuate"}, - } - } - - spec := api.NovaSpec{ - ProjectID: "project-A", - InstanceUUID: instanceUUID, - NumInstances: 1, - SchedulerHints: schedulerHints, - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - Name: "m1.large", - VCPUs: 4, - MemoryMB: parseMemoryToMB("8Gi"), - ExtraSpecs: extraSpecs, - }, - }, + req.Intent = api.EvacuateIntent } weights := make(map[string]float64) for _, h := range hosts { weights[h] = 1.0 } + req.Weights = weights - return api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{Data: spec}, - Hosts: hostList, - Weights: weights, - } + return req } // ============================================================================ @@ -166,7 +153,7 @@ func TestKVMFailoverEvacuationStep_Run(t *testing.T) { tests := []struct { name string reservations []*v1alpha1.Reservation - request api.ExternalSchedulerRequest + request compute.PlacementRequest opts KVMFailoverEvacuationOpts expectedWeights map[string]float64 }{ diff --git a/internal/scheduling/nova/plugins/weighers/kvm_failover_reservation_consolidation.go b/internal/scheduling/nova/plugins/weighers/kvm_failover_reservation_consolidation.go index 270d3c34c..f6b10ee04 100644 --- a/internal/scheduling/nova/plugins/weighers/kvm_failover_reservation_consolidation.go +++ b/internal/scheduling/nova/plugins/weighers/kvm_failover_reservation_consolidation.go @@ -11,6 +11,7 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" ) @@ -73,25 +74,24 @@ func (o KVMFailoverReservationConsolidationOpts) GetSameSpecPenalty() float64 { // // This produces bounded output (~0 to 1) that plays nicely with other weighers. type KVMFailoverReservationConsolidationStep struct { - lib.BaseWeigher[api.ExternalSchedulerRequest, KVMFailoverReservationConsolidationOpts] + lib.BaseWeigher[compute.PlacementRequest, KVMFailoverReservationConsolidationOpts] } // Run the weigher step. // For reserve_for_failover requests, hosts are scored based on existing failover reservation density // and same-spec diversity. For all other request types, this weigher has no effect. -func (s *KVMFailoverReservationConsolidationStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *KVMFailoverReservationConsolidationStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) - intent, err := request.GetIntent() - if err != nil || intent != api.ReserveForFailoverIntent { + if request.Intent != api.ReserveForFailoverIntent { traceLog.V(1).Info("skipping failover reservation consolidation weigher for non-failover-reservation request") - return result, nil //nolint:nilerr // intentionally skip weigher on error + return result, nil } - // Extract the resource group from the scheduler hint. + // Extract the resource group from the request. // This identifies which "spec group" the incoming reservation belongs to. - // If the hint is missing, requestResourceGroup will be empty and the same-group penalty is skipped. - requestResourceGroup, _ := request.Spec.Data.GetSchedulerHintStr(api.HintKeyResourceGroup) //nolint:errcheck // missing hint is fine, same-group penalty is simply skipped + // If it is empty, the same-group penalty is skipped. + requestResourceGroup := request.ResourceGroup // Fetch all reservations. var reservations v1alpha1.ReservationList diff --git a/internal/scheduling/nova/plugins/weighers/kvm_failover_reservation_consolidation_test.go b/internal/scheduling/nova/plugins/weighers/kvm_failover_reservation_consolidation_test.go index 44736e6a4..93825f070 100644 --- a/internal/scheduling/nova/plugins/weighers/kvm_failover_reservation_consolidation_test.go +++ b/internal/scheduling/nova/plugins/weighers/kvm_failover_reservation_consolidation_test.go @@ -10,6 +10,7 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -50,30 +51,10 @@ func newFailoverReservationWithGroup(name, targetHost, resourceGroup string) *v1 } } -func newFailoverReservationRequest(resourceGroup string, hosts []string) api.ExternalSchedulerRequest { - hostList := make([]api.ExternalSchedulerHost, len(hosts)) +func newFailoverReservationRequest(resourceGroup string, hosts []string) compute.PlacementRequest { + hostList := make([]compute.Host, len(hosts)) for i, h := range hosts { - hostList[i] = api.ExternalSchedulerHost{ComputeHost: h} - } - - spec := api.NovaSpec{ - ProjectID: "project-A", - InstanceUUID: "test-instance", - NumInstances: 1, - SchedulerHints: map[string]any{ - "_nova_check_type": string(api.ReserveForFailoverIntent), - api.HintKeyResourceGroup: resourceGroup, - }, - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - Name: "m1.large", - VCPUs: 4, - MemoryMB: 8192, - ExtraSpecs: map[string]string{ - "capabilities:hypervisor_type": "qemu", - }, - }, - }, + hostList[i] = compute.Host{ComputeHost: h} } weights := make(map[string]float64) @@ -81,8 +62,17 @@ func newFailoverReservationRequest(resourceGroup string, hosts []string) api.Ext weights[h] = 1.0 } - return api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{Data: spec}, + return compute.PlacementRequest{ + ProjectID: "project-A", + InstanceUUID: "test-instance", + NumInstances: 1, + Intent: api.ReserveForFailoverIntent, + ResourceGroup: resourceGroup, + HypervisorFamily: api.HypervisorFamilyKVM, + Flavor: api.FlavorSpec{ + VCPUs: 4, + MemoryMB: 8192, + }, Hosts: hostList, Weights: weights, } @@ -98,7 +88,7 @@ func TestKVMFailoverReservationConsolidationStep_Run(t *testing.T) { tests := []struct { name string reservations []*v1alpha1.Reservation - request api.ExternalSchedulerRequest + request compute.PlacementRequest opts KVMFailoverReservationConsolidationOpts expectedWeights map[string]float64 }{ diff --git a/internal/scheduling/nova/plugins/weighers/kvm_instance_group_soft_affinity.go b/internal/scheduling/nova/plugins/weighers/kvm_instance_group_soft_affinity.go index c95ff1b3c..d26947c7c 100644 --- a/internal/scheduling/nova/plugins/weighers/kvm_instance_group_soft_affinity.go +++ b/internal/scheduling/nova/plugins/weighers/kvm_instance_group_soft_affinity.go @@ -7,7 +7,7 @@ import ( "context" "slices" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "github.com/go-logr/logr" @@ -23,19 +23,19 @@ import ( // respectively, for the scheduler to choose that host for new instances of // the same group. type KVMInstanceGroupSoftAffinityStep struct { - lib.BaseWeigher[api.ExternalSchedulerRequest, lib.EmptyFilterWeigherPipelineStepOpts] + lib.BaseWeigher[compute.PlacementRequest, lib.EmptyFilterWeigherPipelineStepOpts] } -func (s *KVMInstanceGroupSoftAffinityStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *KVMInstanceGroupSoftAffinityStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) result.Statistics["affinity"] = s.PrepareStats(request, "float") - ig := request.Spec.Data.InstanceGroup + ig := request.InstanceGroup if ig == nil { traceLog.V(1).Info("no instance group in request, skipping weigher") return result, nil } - policy := ig.Data.Policy + policy := ig.Policy var factor float64 switch policy { case "soft-anti-affinity": @@ -46,7 +46,7 @@ func (s *KVMInstanceGroupSoftAffinityStep) Run(traceLog logr.Logger, request api traceLog.V(1).Info("instance group policy is not 'soft-affinity' or 'soft-anti-affinity', skipping weigher", "policy", policy) return result, nil } - if len(ig.Data.Members) == 0 { + if len(ig.Members) == 0 { traceLog.V(1).Info("instance group has no members, skipping weigher") return result, nil } @@ -72,7 +72,7 @@ func (s *KVMInstanceGroupSoftAffinityStep) Run(traceLog logr.Logger, request api } count := 0 for _, instance := range hv.Status.Instances { - if slices.Contains(ig.Data.Members, instance.ID) { + if slices.Contains(ig.Members, instance.ID) { count++ } } diff --git a/internal/scheduling/nova/plugins/weighers/kvm_instance_group_soft_affinity_test.go b/internal/scheduling/nova/plugins/weighers/kvm_instance_group_soft_affinity_test.go index b839388b2..767e1c7a6 100644 --- a/internal/scheduling/nova/plugins/weighers/kvm_instance_group_soft_affinity_test.go +++ b/internal/scheduling/nova/plugins/weighers/kvm_instance_group_soft_affinity_test.go @@ -8,6 +8,7 @@ import ( "testing" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/controller-runtime/pkg/client" @@ -33,39 +34,30 @@ func newHypervisorWithInstances(name string, instanceIDs ...string) *hv1.Hypervi } } -func newInstanceGroupRequest(policy string, members, hosts []string) api.ExternalSchedulerRequest { - hostList := make([]api.ExternalSchedulerHost, len(hosts)) +func newInstanceGroupRequest(policy string, members, hosts []string) compute.PlacementRequest { + hostList := make([]compute.Host, len(hosts)) for i, h := range hosts { - hostList[i] = api.ExternalSchedulerHost{ComputeHost: h} + hostList[i] = compute.Host{ComputeHost: h} } - extraSpecs := map[string]string{ - "capabilities:hypervisor_type": "qemu", - } - - spec := api.NovaSpec{ - ProjectID: "project-A", - InstanceUUID: "new-instance-uuid", - NumInstances: 1, - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - Name: "m1.large", - VCPUs: 4, - MemoryMB: 8192, - ExtraSpecs: extraSpecs, - }, + req := compute.PlacementRequest{ + ProjectID: "project-A", + InstanceUUID: "new-instance-uuid", + NumInstances: 1, + HypervisorFamily: api.HypervisorFamilyKVM, + Flavor: api.FlavorSpec{ + VCPUs: 4, + MemoryMB: 8192, }, + Hosts: hostList, } // Only set InstanceGroup if policy is provided if policy != "" { - spec.InstanceGroup = &api.NovaObject[api.NovaInstanceGroup]{ - Data: api.NovaInstanceGroup{ - UUID: "instance-group-uuid", - Name: "test-instance-group", - Policy: policy, - Members: members, - }, + req.InstanceGroup = &api.InstanceGroupSpec{ + UUID: "instance-group-uuid", + Policy: policy, + Members: members, } } @@ -73,12 +65,9 @@ func newInstanceGroupRequest(policy string, members, hosts []string) api.Externa for _, h := range hosts { weights[h] = 1.0 } + req.Weights = weights - return api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{Data: spec}, - Hosts: hostList, - Weights: weights, - } + return req } func TestKVMInstanceGroupSoftAffinityStep_Run(t *testing.T) { @@ -87,7 +76,7 @@ func TestKVMInstanceGroupSoftAffinityStep_Run(t *testing.T) { tests := []struct { name string hypervisors []*hv1.Hypervisor - request api.ExternalSchedulerRequest + request compute.PlacementRequest expectedWeights map[string]float64 wantErr bool }{ diff --git a/internal/scheduling/nova/plugins/weighers/kvm_prefer_smaller_hosts.go b/internal/scheduling/nova/plugins/weighers/kvm_prefer_smaller_hosts.go index 52632df77..87bf495e4 100644 --- a/internal/scheduling/nova/plugins/weighers/kvm_prefer_smaller_hosts.go +++ b/internal/scheduling/nova/plugins/weighers/kvm_prefer_smaller_hosts.go @@ -9,7 +9,7 @@ import ( "fmt" "slices" - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "github.com/go-logr/logr" @@ -53,11 +53,11 @@ func (o KVMPreferSmallerHostsStepOpts) Validate() error { // accommodate larger VMs when they need to be scheduled. type KVMPreferSmallerHostsStep struct { // Base weigher providing common functionality. - lib.BaseWeigher[api.ExternalSchedulerRequest, KVMPreferSmallerHostsStepOpts] + lib.BaseWeigher[compute.PlacementRequest, KVMPreferSmallerHostsStepOpts] } // Run this weigher in the pipeline after filters have been executed. -func (s *KVMPreferSmallerHostsStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *KVMPreferSmallerHostsStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) result.Statistics["small host score"] = s.PrepareStats(request, "float") diff --git a/internal/scheduling/nova/plugins/weighers/kvm_prefer_smaller_hosts_test.go b/internal/scheduling/nova/plugins/weighers/kvm_prefer_smaller_hosts_test.go index 1f5a07677..198290870 100644 --- a/internal/scheduling/nova/plugins/weighers/kvm_prefer_smaller_hosts_test.go +++ b/internal/scheduling/nova/plugins/weighers/kvm_prefer_smaller_hosts_test.go @@ -9,6 +9,7 @@ import ( "testing" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -30,28 +31,10 @@ func newHypervisorWithCapacity(name, capacityCPU, capacityMem string) *hv1.Hyper } } -func newPreferSmallerHostsRequest(hosts []string) api.ExternalSchedulerRequest { - hostList := make([]api.ExternalSchedulerHost, len(hosts)) +func newPreferSmallerHostsRequest(hosts []string) compute.PlacementRequest { + hostList := make([]compute.Host, len(hosts)) for i, h := range hosts { - hostList[i] = api.ExternalSchedulerHost{ComputeHost: h} - } - - extraSpecs := map[string]string{ - "capabilities:hypervisor_type": "qemu", - } - - spec := api.NovaSpec{ - ProjectID: "project-A", - InstanceUUID: "instance-123", - NumInstances: 1, - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - Name: "m1.large", - VCPUs: 4, - MemoryMB: 8192, - ExtraSpecs: extraSpecs, - }, - }, + hostList[i] = compute.Host{ComputeHost: h} } weights := make(map[string]float64) @@ -59,8 +42,15 @@ func newPreferSmallerHostsRequest(hosts []string) api.ExternalSchedulerRequest { weights[h] = 1.0 } - return api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{Data: spec}, + return compute.PlacementRequest{ + ProjectID: "project-A", + InstanceUUID: "instance-123", + NumInstances: 1, + HypervisorFamily: api.HypervisorFamilyKVM, + Flavor: api.FlavorSpec{ + VCPUs: 4, + MemoryMB: 8192, + }, Hosts: hostList, Weights: weights, } @@ -181,7 +171,7 @@ func TestKVMPreferSmallerHostsStep_Run(t *testing.T) { tests := []struct { name string hypervisors []*hv1.Hypervisor - request api.ExternalSchedulerRequest + request compute.PlacementRequest opts KVMPreferSmallerHostsStepOpts expectedWeights map[string]float64 wantErr bool diff --git a/internal/scheduling/nova/plugins/weighers/vmware_anti_affinity_noisy_projects.go b/internal/scheduling/nova/plugins/weighers/vmware_anti_affinity_noisy_projects.go index 426526e2d..3b775977b 100644 --- a/internal/scheduling/nova/plugins/weighers/vmware_anti_affinity_noisy_projects.go +++ b/internal/scheduling/nova/plugins/weighers/vmware_anti_affinity_noisy_projects.go @@ -9,9 +9,9 @@ import ( "github.com/go-logr/logr" - api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" corev1 "k8s.io/api/core/v1" "sigs.k8s.io/controller-runtime/pkg/client" @@ -38,7 +38,7 @@ func (o VMwareAntiAffinityNoisyProjectsStepOpts) Validate() error { // Step to avoid noisy projects by downvoting the hosts they are running on. type VMwareAntiAffinityNoisyProjectsStep struct { // BaseStep is a helper struct that provides common functionality for all steps. - lib.BaseWeigher[api.ExternalSchedulerRequest, VMwareAntiAffinityNoisyProjectsStepOpts] + lib.BaseWeigher[sched.PlacementRequest, VMwareAntiAffinityNoisyProjectsStepOpts] } // Initialize the step and validate that all required knowledges are ready. @@ -53,7 +53,7 @@ func (s *VMwareAntiAffinityNoisyProjectsStep) Init(ctx context.Context, client c } // Downvote the hosts a project is currently running on if it's noisy. -func (s *VMwareAntiAffinityNoisyProjectsStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *VMwareAntiAffinityNoisyProjectsStep) Run(traceLog logr.Logger, request sched.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) result.Statistics["avg cpu usage of this project"] = s.PrepareStats(request, "%") @@ -72,7 +72,7 @@ func (s *VMwareAntiAffinityNoisyProjectsStep) Run(traceLog logr.Logger, request return nil, err } for _, p := range projectNoisinessOnHosts { - if p.Project != request.Spec.Data.ProjectID { + if p.Project != request.ProjectID { continue } // Only modify the weight if the host is in the scenario. @@ -92,7 +92,7 @@ func (s *VMwareAntiAffinityNoisyProjectsStep) Run(traceLog logr.Logger, request } func init() { - Index["vmware_anti_affinity_noisy_projects"] = func() lib.Weigher[api.ExternalSchedulerRequest] { + Index["vmware_anti_affinity_noisy_projects"] = func() lib.Weigher[sched.PlacementRequest] { return &VMwareAntiAffinityNoisyProjectsStep{} } } diff --git a/internal/scheduling/nova/plugins/weighers/vmware_anti_affinity_noisy_projects_test.go b/internal/scheduling/nova/plugins/weighers/vmware_anti_affinity_noisy_projects_test.go index 5005bae45..38a28de94 100644 --- a/internal/scheduling/nova/plugins/weighers/vmware_anti_affinity_noisy_projects_test.go +++ b/internal/scheduling/nova/plugins/weighers/vmware_anti_affinity_noisy_projects_test.go @@ -9,9 +9,9 @@ import ( "strings" "testing" - api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/controller-runtime/pkg/client/fake" ) @@ -219,18 +219,14 @@ func TestVMwareAntiAffinityNoisyProjectsStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request sched.PlacementRequest downvotedHosts map[string]struct{} }{ { name: "Noisy project", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project1", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: sched.PlacementRequest{ + ProjectID: "project1", + Hosts: []sched.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -243,13 +239,9 @@ func TestVMwareAntiAffinityNoisyProjectsStep_Run(t *testing.T) { }, { name: "Non-noisy project", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - ProjectID: "project2", - }, - }, - Hosts: []api.ExternalSchedulerHost{ + request: sched.PlacementRequest{ + ProjectID: "project2", + Hosts: []sched.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -259,8 +251,8 @@ func TestVMwareAntiAffinityNoisyProjectsStep_Run(t *testing.T) { }, { name: "No noisy project data", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: sched.PlacementRequest{ + Hosts: []sched.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, diff --git a/internal/scheduling/nova/plugins/weighers/vmware_avoid_long_term_contended_hosts.go b/internal/scheduling/nova/plugins/weighers/vmware_avoid_long_term_contended_hosts.go index a12c35fe1..c52cd921a 100644 --- a/internal/scheduling/nova/plugins/weighers/vmware_avoid_long_term_contended_hosts.go +++ b/internal/scheduling/nova/plugins/weighers/vmware_avoid_long_term_contended_hosts.go @@ -9,9 +9,9 @@ import ( "github.com/go-logr/logr" - api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" corev1 "k8s.io/api/core/v1" "sigs.k8s.io/controller-runtime/pkg/client" @@ -47,7 +47,7 @@ func (o VMwareAvoidLongTermContendedHostsStepOpts) Validate() error { // Step to avoid long term contended hosts by downvoting them. type VMwareAvoidLongTermContendedHostsStep struct { // BaseStep is a helper struct that provides common functionality for all steps. - lib.BaseWeigher[api.ExternalSchedulerRequest, VMwareAvoidLongTermContendedHostsStepOpts] + lib.BaseWeigher[sched.PlacementRequest, VMwareAvoidLongTermContendedHostsStepOpts] } // Initialize the step and validate that all required knowledges are ready. @@ -62,7 +62,7 @@ func (s *VMwareAvoidLongTermContendedHostsStep) Init(ctx context.Context, client } // Downvote hosts that are highly contended. -func (s *VMwareAvoidLongTermContendedHostsStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *VMwareAvoidLongTermContendedHostsStep) Run(traceLog logr.Logger, request sched.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) result.Statistics["avg cpu contention"] = s.PrepareStats(request, "%") @@ -110,7 +110,7 @@ func (s *VMwareAvoidLongTermContendedHostsStep) Run(traceLog logr.Logger, reques } func init() { - Index["vmware_avoid_long_term_contended_hosts"] = func() lib.Weigher[api.ExternalSchedulerRequest] { + Index["vmware_avoid_long_term_contended_hosts"] = func() lib.Weigher[sched.PlacementRequest] { return &VMwareAvoidLongTermContendedHostsStep{} } } diff --git a/internal/scheduling/nova/plugins/weighers/vmware_avoid_long_term_contended_hosts_test.go b/internal/scheduling/nova/plugins/weighers/vmware_avoid_long_term_contended_hosts_test.go index 701cef92e..25f697a4d 100644 --- a/internal/scheduling/nova/plugins/weighers/vmware_avoid_long_term_contended_hosts_test.go +++ b/internal/scheduling/nova/plugins/weighers/vmware_avoid_long_term_contended_hosts_test.go @@ -9,9 +9,9 @@ import ( "strings" "testing" - api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/controller-runtime/pkg/client/fake" ) @@ -218,13 +218,13 @@ func TestVMwareAvoidLongTermContendedHostsStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request sched.PlacementRequest expected map[string]float64 }{ { name: "Avoid contended hosts", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: sched.PlacementRequest{ + Hosts: []sched.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -240,8 +240,8 @@ func TestVMwareAvoidLongTermContendedHostsStep_Run(t *testing.T) { }, { name: "Missing data", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: sched.PlacementRequest{ + Hosts: []sched.Host{ {ComputeHost: "host4"}, {ComputeHost: "host5"}, }, diff --git a/internal/scheduling/nova/plugins/weighers/vmware_avoid_short_term_contended_hosts.go b/internal/scheduling/nova/plugins/weighers/vmware_avoid_short_term_contended_hosts.go index 209eb97ba..6d2f0fc37 100644 --- a/internal/scheduling/nova/plugins/weighers/vmware_avoid_short_term_contended_hosts.go +++ b/internal/scheduling/nova/plugins/weighers/vmware_avoid_short_term_contended_hosts.go @@ -9,9 +9,9 @@ import ( "github.com/go-logr/logr" - api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + extractorcompute "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" corev1 "k8s.io/api/core/v1" "sigs.k8s.io/controller-runtime/pkg/client" @@ -47,7 +47,7 @@ func (o VMwareAvoidShortTermContendedHostsStepOpts) Validate() error { // Step to avoid recently contended hosts by downvoting them. type VMwareAvoidShortTermContendedHostsStep struct { // BaseStep is a helper struct that provides common functionality for all steps. - lib.BaseWeigher[api.ExternalSchedulerRequest, VMwareAvoidShortTermContendedHostsStepOpts] + lib.BaseWeigher[compute.PlacementRequest, VMwareAvoidShortTermContendedHostsStepOpts] } // Initialize the step and validate that all required knowledges are ready. @@ -62,7 +62,7 @@ func (s *VMwareAvoidShortTermContendedHostsStep) Init(ctx context.Context, clien } // Downvote hosts that are highly contended. -func (s *VMwareAvoidShortTermContendedHostsStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *VMwareAvoidShortTermContendedHostsStep) Run(traceLog logr.Logger, request compute.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) result.Statistics["avg cpu contention"] = s.PrepareStats(request, "%") @@ -77,7 +77,7 @@ func (s *VMwareAvoidShortTermContendedHostsStep) Run(traceLog logr.Logger, reque return nil, err } highlyContendedHosts, err := v1alpha1. - UnboxFeatureList[compute.VROpsHostsystemContentionShortTerm](knowledge.Status.Raw) + UnboxFeatureList[extractorcompute.VROpsHostsystemContentionShortTerm](knowledge.Status.Raw) if err != nil { return nil, err } @@ -110,7 +110,7 @@ func (s *VMwareAvoidShortTermContendedHostsStep) Run(traceLog logr.Logger, reque } func init() { - Index["vmware_avoid_short_term_contended_hosts"] = func() lib.Weigher[api.ExternalSchedulerRequest] { + Index["vmware_avoid_short_term_contended_hosts"] = func() lib.Weigher[compute.PlacementRequest] { return &VMwareAvoidShortTermContendedHostsStep{} } } diff --git a/internal/scheduling/nova/plugins/weighers/vmware_avoid_short_term_contended_hosts_test.go b/internal/scheduling/nova/plugins/weighers/vmware_avoid_short_term_contended_hosts_test.go index 624ee7e14..768973f1c 100644 --- a/internal/scheduling/nova/plugins/weighers/vmware_avoid_short_term_contended_hosts_test.go +++ b/internal/scheduling/nova/plugins/weighers/vmware_avoid_short_term_contended_hosts_test.go @@ -9,9 +9,9 @@ import ( "strings" "testing" - api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/controller-runtime/pkg/client/fake" ) @@ -218,13 +218,13 @@ func TestVMwareAvoidShortTermContendedHostsStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request sched.PlacementRequest expected map[string]float64 }{ { name: "Avoid contended hosts", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: sched.PlacementRequest{ + Hosts: []sched.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -240,8 +240,8 @@ func TestVMwareAvoidShortTermContendedHostsStep_Run(t *testing.T) { }, { name: "Missing data", - request: api.ExternalSchedulerRequest{ - Hosts: []api.ExternalSchedulerHost{ + request: sched.PlacementRequest{ + Hosts: []sched.Host{ {ComputeHost: "host4"}, {ComputeHost: "host5"}, // No data for host5 }, diff --git a/internal/scheduling/nova/plugins/weighers/vmware_binpack.go b/internal/scheduling/nova/plugins/weighers/vmware_binpack.go index 3f8b01fd8..27bc33f39 100644 --- a/internal/scheduling/nova/plugins/weighers/vmware_binpack.go +++ b/internal/scheduling/nova/plugins/weighers/vmware_binpack.go @@ -11,9 +11,9 @@ import ( "github.com/go-logr/logr" - api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" @@ -69,7 +69,7 @@ func (o VMwareBinpackStepOpts) Validate() error { // Explanation of the algorithm: https://volcano.sh/en/docs/plugins/#binpack type VMwareBinpackStep struct { // Base weigher providing common functionality. - lib.BaseWeigher[api.ExternalSchedulerRequest, VMwareBinpackStepOpts] + lib.BaseWeigher[sched.PlacementRequest, VMwareBinpackStepOpts] } // Initialize the step and validate that all required knowledges are ready. @@ -86,7 +86,7 @@ func (s *VMwareBinpackStep) Init(ctx context.Context, client client.Client, weig } // Run this weigher in the pipeline after filters have been executed. -func (s *VMwareBinpackStep) Run(traceLog logr.Logger, request api.ExternalSchedulerRequest) (*lib.FilterWeigherPipelineStepResult, error) { +func (s *VMwareBinpackStep) Run(traceLog logr.Logger, request sched.PlacementRequest) (*lib.FilterWeigherPipelineStepResult, error) { result := s.IncludeAllHostsFromRequest(request) result.Statistics["binpack score"] = s.PrepareStats(request, "float") @@ -185,14 +185,14 @@ func (s *VMwareBinpackStep) calcHostAllocation(hostUtilization compute.HostUtili } // calcVMResources calculates the total resource requests for the VM to be scheduled. -func (s *VMwareBinpackStep) calcVMResources(req api.ExternalSchedulerRequest) map[corev1.ResourceName]resource.Quantity { +func (s *VMwareBinpackStep) calcVMResources(req sched.PlacementRequest) map[corev1.ResourceName]resource.Quantity { resources := make(map[corev1.ResourceName]resource.Quantity) - resourcesMemBytes := int64(req.Spec.Data.Flavor.Data.MemoryMB) * (1 << 20) //nolint:gosec // memory values are bounded by Nova - resourcesMemBytes *= int64(req.Spec.Data.NumInstances) //nolint:gosec // instance count is bounded by Nova + resourcesMemBytes := int64(req.Flavor.MemoryMB) * (1 << 20) //nolint:gosec // memory values are bounded by Nova + resourcesMemBytes *= int64(req.NumInstances) //nolint:gosec // instance count is bounded by Nova resources[corev1.ResourceMemory] = *resource. NewQuantity(resourcesMemBytes, resource.BinarySI) - resourcesCPU := int64(req.Spec.Data.Flavor.Data.VCPUs) //nolint:gosec // vCPU values are bounded by Nova - resourcesCPU *= int64(req.Spec.Data.NumInstances) //nolint:gosec // instance count is bounded by Nova + resourcesCPU := int64(req.Flavor.VCPUs) //nolint:gosec // vCPU values are bounded by Nova + resourcesCPU *= int64(req.NumInstances) //nolint:gosec // instance count is bounded by Nova resources[corev1.ResourceCPU] = *resource. NewQuantity(resourcesCPU, resource.DecimalSI) return resources diff --git a/internal/scheduling/nova/plugins/weighers/vmware_binpack_test.go b/internal/scheduling/nova/plugins/weighers/vmware_binpack_test.go index 404722f43..0d7b85e39 100644 --- a/internal/scheduling/nova/plugins/weighers/vmware_binpack_test.go +++ b/internal/scheduling/nova/plugins/weighers/vmware_binpack_test.go @@ -10,6 +10,7 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/controller-runtime/pkg/client/fake" @@ -156,25 +157,19 @@ func TestVMwareBinpackStep_Run(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request sched.PlacementRequest expectedHigherScore string expectedLowerScore string }{ { name: "binpack prefers hosts with higher utilization", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - NumInstances: 1, - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - MemoryMB: 1000, // 1GB - VCPUs: 1, - }, - }, - }, + request: sched.PlacementRequest{ + NumInstances: 1, + Flavor: api.FlavorSpec{ + MemoryMB: 1000, // 1GB + VCPUs: 1, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []sched.Host{ {ComputeHost: "host1"}, {ComputeHost: "host2"}, {ComputeHost: "host3"}, @@ -187,19 +182,13 @@ func TestVMwareBinpackStep_Run(t *testing.T) { }, { name: "handles missing host data gracefully", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - NumInstances: 1, - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - MemoryMB: 1000, - VCPUs: 1, - }, - }, - }, + request: sched.PlacementRequest{ + NumInstances: 1, + Flavor: api.FlavorSpec{ + MemoryMB: 1000, + VCPUs: 1, }, - Hosts: []api.ExternalSchedulerHost{ + Hosts: []sched.Host{ {ComputeHost: "host1"}, {ComputeHost: "unknown_host"}, // No utilization data }, @@ -292,23 +281,17 @@ func TestVMwareBinpackStep_CalcVMResources(t *testing.T) { tests := []struct { name string - request api.ExternalSchedulerRequest + request sched.PlacementRequest expectedMemory int64 expectedCPU int64 }{ { name: "single instance", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - NumInstances: 1, - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - MemoryMB: 2048, - VCPUs: 4, - }, - }, - }, + request: sched.PlacementRequest{ + NumInstances: 1, + Flavor: api.FlavorSpec{ + MemoryMB: 2048, + VCPUs: 4, }, }, expectedMemory: 2048 * (1 << 20), // 2048 MiB in bytes @@ -316,17 +299,11 @@ func TestVMwareBinpackStep_CalcVMResources(t *testing.T) { }, { name: "multiple instances", - request: api.ExternalSchedulerRequest{ - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - NumInstances: 3, - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - MemoryMB: 1024, - VCPUs: 2, - }, - }, - }, + request: sched.PlacementRequest{ + NumInstances: 3, + Flavor: api.FlavorSpec{ + MemoryMB: 1024, + VCPUs: 2, }, }, expectedMemory: 3 * 1024 * (1 << 20), // 3 instances * 1024 MiB in bytes diff --git a/internal/scheduling/nova/plugins/weighers/zz_index.go b/internal/scheduling/nova/plugins/weighers/zz_index.go index d3136e724..45bef3727 100644 --- a/internal/scheduling/nova/plugins/weighers/zz_index.go +++ b/internal/scheduling/nova/plugins/weighers/zz_index.go @@ -4,11 +4,11 @@ package weighers import ( - api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" ) -type NovaWeigher = lib.Weigher[api.ExternalSchedulerRequest] +type NovaWeigher = lib.Weigher[compute.PlacementRequest] // Configuration of weighers supported by the nova scheduler. var Index = map[string]func() NovaWeigher{} diff --git a/internal/scheduling/nova/vm_handler.go b/internal/scheduling/nova/vm_handler.go index 3548122f3..bbc22d635 100644 --- a/internal/scheduling/nova/vm_handler.go +++ b/internal/scheduling/nova/vm_handler.go @@ -12,6 +12,8 @@ import ( api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" + "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" vmv1alpha1 "github.com/cobaltcore-dev/vm-operator/api/v1alpha1" corev1 "k8s.io/api/core/v1" @@ -33,21 +35,26 @@ type VirtualMachineHandler struct { } // Upsert creates or patches the VirtualMachine resource described by the -// decision and request. Region and AZ are immutable on the CRD and only written -// on create; mutable fields (target host, cortex spec, labels) are patched on -// updates. -func (h *VirtualMachineHandler) Upsert(ctx context.Context, decision *v1alpha1.Decision, request api.ExternalSchedulerRequest) error { +// scheduling result and request. Region and AZ are immutable on the CRD and only +// written on create; mutable fields (target host, cortex spec, labels) are +// patched on updates. A human-readable explanation of the result and the intent +// are recorded on the VM's cortex spec. +func (h *VirtualMachineHandler) Upsert(ctx context.Context, result compute.PlacementResult, request api.ExternalSchedulerRequest, intent v1alpha1.SchedulingIntent) error { log := ctrl.LoggerFrom(ctx) - if decision.Status.Result == nil || decision.Status.Result.TargetHost == nil { - log.V(1).Info("skipping virtual machine upsert, no target host in decision") + if result.TargetHost == nil { + log.V(1).Info("skipping virtual machine upsert, no target host in result") return nil } + // The explanation is a VM-creation concern, so it is generated here from the + // pipeline result and recorded on the VM's cortex spec. + explanation := lib.GenerateExplanation(&result, nil) + // We derive hierarchical location information (region and az) from the // target host (i.e. the first host in the list). This assumes that all // hosts in the list share the same region and availability zone. - targetHost := *decision.Status.Result.TargetHost + targetHost := *result.TargetHost targetHv := &hv1.Hypervisor{} if err := h.Get(ctx, client.ObjectKey{Name: targetHost}, targetHv); err != nil { return fmt.Errorf("failed to get hypervisor %q: %w", targetHost, err) @@ -136,20 +143,18 @@ func (h *VirtualMachineHandler) Upsert(ctx context.Context, decision *v1alpha1.D } } - // Read intent from the decision, which was already resolved by the API layer - // (including a graceful fallback to CreateIntent for initial placements). - intent := decision.Spec.Intent - - // Write the cortex spec (our scheduling result). + // Write the cortex spec (our scheduling result). The intent was resolved by + // the API layer (with a graceful fallback to CreateIntent for initial + // placements) and passed in. vm.Spec.Cortex = &vmv1alpha1.CortexSpec{} - for _, host := range decision.Status.Result.OrderedHosts { + for _, host := range result.OrderedHosts { vm.Spec.Cortex.HostCandidates = append(vm.Spec.Cortex.HostCandidates, vmv1alpha1.HostRef{ APIVersion: hv1.GroupVersion.String(), Kind: "Hypervisor", Name: host, }) } - vm.Spec.Cortex.Explanation = decision.Status.Explanation + vm.Spec.Cortex.Explanation = explanation vm.Spec.Cortex.Intent = string(intent) // Write the ConditionTypeScheduled condition to indicate that the VM has diff --git a/internal/scheduling/nova/vm_handler_test.go b/internal/scheduling/nova/vm_handler_test.go index 8c1499c07..6c81f6f60 100644 --- a/internal/scheduling/nova/vm_handler_test.go +++ b/internal/scheduling/nova/vm_handler_test.go @@ -8,7 +8,7 @@ import ( "testing" api "github.com/cobaltcore-dev/cortex/api/external/nova" - "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" vmopv1alpha1 "github.com/cobaltcore-dev/vm-operator/api/v1alpha1" corev1 "k8s.io/api/core/v1" @@ -56,15 +56,8 @@ func vmTestRequest() api.ExternalSchedulerRequest { return req } -func vmTestDecision(targetHost *string) *v1alpha1.Decision { - return &v1alpha1.Decision{ - Spec: v1alpha1.DecisionSpec{ - Intent: api.CreateIntent, - }, - Status: v1alpha1.DecisionStatus{ - Result: &v1alpha1.DecisionResult{TargetHost: targetHost}, - }, - } +func vmTestResult(targetHost *string) compute.PlacementResult { + return compute.PlacementResult{TargetHost: targetHost} } func vmTestHypervisor(region, az string) *hv1.Hypervisor { @@ -88,7 +81,7 @@ func TestVirtualMachineHandlerCreate(t *testing.T) { h := &VirtualMachineHandler{Client: fakeClient} targetHost := "compute-1" - if err := h.Upsert(context.Background(), vmTestDecision(&targetHost), vmTestRequest()); err != nil { + if err := h.Upsert(context.Background(), vmTestResult(&targetHost), vmTestRequest(), api.CreateIntent); err != nil { t.Fatalf("Upsert: %v", err) } @@ -137,7 +130,7 @@ func TestVirtualMachineHandlerUpdate(t *testing.T) { h := &VirtualMachineHandler{Client: fakeClient} targetHost := "compute-1" - if err := h.Upsert(context.Background(), vmTestDecision(&targetHost), vmTestRequest()); err != nil { + if err := h.Upsert(context.Background(), vmTestResult(&targetHost), vmTestRequest(), api.CreateIntent); err != nil { t.Fatalf("Upsert: %v", err) } @@ -162,7 +155,7 @@ func TestVirtualMachineHandlerNoTargetHost(t *testing.T) { fakeClient := fake.NewClientBuilder().WithScheme(scheme).Build() h := &VirtualMachineHandler{Client: fakeClient} - if err := h.Upsert(context.Background(), vmTestDecision(nil), vmTestRequest()); err != nil { + if err := h.Upsert(context.Background(), vmTestResult(nil), vmTestRequest(), api.CreateIntent); err != nil { t.Fatalf("Upsert with nil target host should be a no-op, got: %v", err) } var vmList vmopv1alpha1.VirtualMachineList @@ -183,7 +176,7 @@ func TestVirtualMachineHandlerMissingRegion(t *testing.T) { h := &VirtualMachineHandler{Client: fakeClient} targetHost := "compute-1" - if err := h.Upsert(context.Background(), vmTestDecision(&targetHost), vmTestRequest()); err == nil { + if err := h.Upsert(context.Background(), vmTestResult(&targetHost), vmTestRequest(), api.CreateIntent); err == nil { t.Fatal("expected error for missing region label, got nil") } vm := &vmopv1alpha1.VirtualMachine{} diff --git a/internal/scheduling/pods/filter_weigher_pipeline_controller.go b/internal/scheduling/pods/filter_weigher_pipeline_controller.go index 50afde197..ee140526c 100644 --- a/internal/scheduling/pods/filter_weigher_pipeline_controller.go +++ b/internal/scheduling/pods/filter_weigher_pipeline_controller.go @@ -18,7 +18,6 @@ import ( "github.com/cobaltcore-dev/cortex/internal/scheduling/pods/plugins/filters" "github.com/cobaltcore-dev/cortex/internal/scheduling/pods/plugins/weighers" corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/util/workqueue" ctrl "sigs.k8s.io/controller-runtime" @@ -30,9 +29,9 @@ import ( "sigs.k8s.io/controller-runtime/pkg/reconcile" ) -// The decision pipeline controller takes decision resources containing a -// pod ref and runs the scheduling pipeline to make a decision. -// This decision is then written back to the decision resource status. +// The pod pipeline controller watches for new pods and runs the scheduling +// pipeline to assign each a node. The result is written back directly via a +// Binding. // // Additionally, the controller watches for pipeline and step changes to // reconfigure the pipelines as needed. @@ -52,87 +51,30 @@ func (c *FilterWeigherPipelineController) PipelineType() v1alpha1.PipelineType { return v1alpha1.PipelineTypeFilterWeigher } -func (c *FilterWeigherPipelineController) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { - c.processMu.Lock() - defer c.processMu.Unlock() - - // Determine if this is a decision or pod reconciliation. - decision := &v1alpha1.Decision{} - if err := c.Get(ctx, req.NamespacedName, decision); err != nil { - return ctrl.Result{}, client.IgnoreNotFound(err) - } - old := decision.DeepCopy() - if err := c.process(ctx, decision); err != nil { - return ctrl.Result{}, err - } - patch := client.MergeFrom(old) - if err := c.Status().Patch(ctx, decision, patch); err != nil { - return ctrl.Result{}, err - } +// Reconcile is a no-op: this controller schedules pods through its Pod watch +// handler and reconfigures pipelines through its Pipeline watch handler. A +// Reconcile method is required to satisfy the controller builder. +func (c *FilterWeigherPipelineController) Reconcile(context.Context, ctrl.Request) (ctrl.Result, error) { return ctrl.Result{}, nil } func (c *FilterWeigherPipelineController) ProcessNewPod(ctx context.Context, pod *corev1.Pod) error { c.processMu.Lock() defer c.processMu.Unlock() - - // Create a decision resource to schedule the pod. - decision := &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - GenerateName: "pod-", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainPods, - ResourceID: pod.Namespace + "--" + pod.Name, - PipelineRef: corev1.ObjectReference{ - Name: "pods-scheduler", - }, - PodRef: &corev1.ObjectReference{ - Name: pod.Name, - Namespace: pod.Namespace, - }, - Intent: v1alpha1.SchedulingIntentUnknown, - }, - } - - err := c.process(ctx, decision) - if err != nil { - meta.SetStatusCondition(&decision.Status.Conditions, metav1.Condition{ - Type: v1alpha1.DecisionConditionReady, - Status: metav1.ConditionFalse, - Reason: "PipelineRunFailed", - Message: "pipeline run failed: " + err.Error(), - }) - } else { - meta.SetStatusCondition(&decision.Status.Conditions, metav1.Condition{ - Type: v1alpha1.DecisionConditionReady, - Status: metav1.ConditionTrue, - Reason: "PipelineRunSucceeded", - Message: "pipeline run succeeded", - }) - } - return err + return c.process(ctx, pod) } -func (c *FilterWeigherPipelineController) process(ctx context.Context, decision *v1alpha1.Decision) error { +func (c *FilterWeigherPipelineController) process(ctx context.Context, pod *corev1.Pod) error { log := ctrl.LoggerFrom(ctx) startedAt := time.Now() // So we can measure sync duration. - pipeline, ok := c.GetPipeline(decision.Spec.PipelineRef.Name) + const pipelineName = "pods-scheduler" + pipeline, ok := c.GetPipeline(pipelineName) if !ok { - log.Error(nil, "pipeline not found or not ready", "pipelineName", decision.Spec.PipelineRef.Name) - return fmt.Errorf("pipeline not found or not ready: %q", decision.Spec.PipelineRef.Name) + log.Error(nil, "pipeline not found or not ready", "pipelineName", pipelineName) + return fmt.Errorf("pipeline not found or not ready: %q", pipelineName) } - // Check if the pod is already assigned to a node. - pod := &corev1.Pod{} - if err := c.Get(ctx, client.ObjectKey{ - Name: decision.Spec.PodRef.Name, - Namespace: decision.Spec.PodRef.Namespace, - }, pod); err != nil { - log.Error(err, "failed to fetch pod for decision") - return err - } if pod.Spec.NodeName != "" { log.V(1).Info("pod is already assigned to a node", "node", pod.Spec.NodeName) return nil @@ -150,14 +92,6 @@ func (c *FilterWeigherPipelineController) process(ctx context.Context, decision // Execute the scheduling pipeline. request := pods.PodPipelineRequest{Nodes: nodes.Items, Pod: *pod} result, err := pipeline.Run(request) - - if err == nil { - decision.Status.Result = &result - } - // Record a human-readable explanation of the decision for downstream - // consumers. On error the result is nil and the explanation carries the - // failure reason. - decision.Status.Explanation = lib.GenerateExplanation(decision.Status.Result, err) if err != nil { log.V(1).Error(err, "failed to run scheduler pipeline") return errors.New("failed to run scheduler pipeline") @@ -167,8 +101,8 @@ func (c *FilterWeigherPipelineController) process(ctx context.Context, decision // Assign the first node returned by the pipeline using a Binding. binding := &corev1.Binding{ ObjectMeta: metav1.ObjectMeta{ - Name: decision.Spec.PodRef.Name, - Namespace: decision.Spec.PodRef.Namespace, + Name: pod.Name, + Namespace: pod.Namespace, }, Target: corev1.ObjectReference{ Kind: "Node", @@ -263,25 +197,6 @@ func (c *FilterWeigherPipelineController) SetupWithManager(mgr manager.Manager, if err != nil { return err } - // Watch decision changes across all clusters. - bldr, err = bldr.WatchesMulticluster( - &v1alpha1.Decision{}, - &handler.EnqueueRequestForObject{}, - predicate.NewPredicateFuncs(func(obj client.Object) bool { - decision := obj.(*v1alpha1.Decision) - if decision.Spec.SchedulingDomain != v1alpha1.SchedulingDomainPods { - return false - } - // Ignore already decided schedulings. - if decision.Status.Result != nil { - return false - } - return true - }), - ) - if err != nil { - return err - } return bldr.Named("cortex-pod-scheduler"). Complete(c) } diff --git a/internal/scheduling/pods/filter_weigher_pipeline_controller_test.go b/internal/scheduling/pods/filter_weigher_pipeline_controller_test.go index 0c9ad6fb8..e9a8f93c3 100644 --- a/internal/scheduling/pods/filter_weigher_pipeline_controller_test.go +++ b/internal/scheduling/pods/filter_weigher_pipeline_controller_test.go @@ -15,166 +15,9 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" - ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client/fake" ) -func TestFilterWeigherPipelineController_Reconcile(t *testing.T) { - scheme := runtime.NewScheme() - if err := v1alpha1.AddToScheme(scheme); err != nil { - t.Fatalf("Failed to add scheduling scheme: %v", err) - } - if err := corev1.AddToScheme(scheme); err != nil { - t.Fatalf("Failed to add corev1 scheme: %v", err) - } - - tests := []struct { - name string - decision *v1alpha1.Decision - nodes []corev1.Node - pod *corev1.Pod - expectError bool - expectDecision bool - expectTargetHost string - }{ - { - name: "successful pod decision processing", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainPods, - ResourceID: "test-pod", - PipelineRef: corev1.ObjectReference{ - Name: "pods-scheduler", - }, - PodRef: &corev1.ObjectReference{ - Name: "test-pod", - Namespace: "default", - }, - }, - }, - nodes: []corev1.Node{ - { - ObjectMeta: metav1.ObjectMeta{Name: "node1"}, - }, - { - ObjectMeta: metav1.ObjectMeta{Name: "node2"}, - }, - }, - pod: &corev1.Pod{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-pod", - Namespace: "default", - }, - Spec: corev1.PodSpec{ - SchedulerName: "", - }, - }, - expectError: false, - expectDecision: true, - expectTargetHost: "node1", // NoopFilter returns first node - }, - { - name: "no nodes available", - decision: &v1alpha1.Decision{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-decision-no-nodes", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainPods, - ResourceID: "test-pod", - PipelineRef: corev1.ObjectReference{ - Name: "pods-scheduler", - }, - PodRef: &corev1.ObjectReference{ - Name: "test-pod", - Namespace: "default", - }, - }, - }, - nodes: []corev1.Node{}, - expectError: true, - expectDecision: false, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - objects := []runtime.Object{tt.decision} - for i := range tt.nodes { - objects = append(objects, &tt.nodes[i]) - } - if tt.pod != nil { - objects = append(objects, tt.pod) - } - - client := fake.NewClientBuilder(). - WithScheme(scheme). - WithRuntimeObjects(objects...). - WithStatusSubresource(&v1alpha1.Decision{}). - Build() - - controller := &FilterWeigherPipelineController{ - BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[pods.PodPipelineRequest]]{ - Pipelines: map[string]lib.FilterWeigherPipeline[pods.PodPipelineRequest]{ - "pods-scheduler": createMockPodPipeline(), - }, - }, - Monitor: lib.FilterWeigherPipelineMonitor{}, - } - controller.Client = client - - req := ctrl.Request{ - NamespacedName: types.NamespacedName{ - Name: tt.decision.Name, - }, - } - - result, err := controller.Reconcile(context.Background(), req) - - if tt.expectError && err == nil { - t.Error("expected error but got none") - return - } - - if !tt.expectError && err != nil { - t.Errorf("expected no error, got: %v", err) - return - } - - if result.RequeueAfter > 0 { - t.Errorf("unexpected requeue: %v", result.RequeueAfter) - } - - // Verify decision status if expected - if tt.expectDecision { - var updatedDecision v1alpha1.Decision - err := client.Get(context.Background(), req.NamespacedName, &updatedDecision) - if err != nil { - t.Errorf("Failed to get updated decision: %v", err) - return - } - - if updatedDecision.Status.Result == nil { - t.Error("expected decision result to be set") - return - } - - if updatedDecision.Status.Result.TargetHost == nil { - t.Error("expected target host to be set") - return - } - - if *updatedDecision.Status.Result.TargetHost != tt.expectTargetHost { - t.Errorf("expected target host %q, got %q", tt.expectTargetHost, *updatedDecision.Status.Result.TargetHost) - } - } - }) - } -} - func TestFilterWeigherPipelineController_InitPipeline(t *testing.T) { controller := &FilterWeigherPipelineController{ Monitor: lib.FilterWeigherPipelineMonitor{}, @@ -393,7 +236,6 @@ func TestFilterWeigherPipelineController_ProcessNewPod(t *testing.T) { client := fake.NewClientBuilder(). WithScheme(scheme). WithRuntimeObjects(objects...). - WithStatusSubresource(&v1alpha1.Decision{}). Build() controller := &FilterWeigherPipelineController{ @@ -452,14 +294,14 @@ func createMockPodPipeline() lib.FilterWeigherPipeline[pods.PodPipelineRequest] type mockPodPipeline struct{} -func (m *mockPodPipeline) Run(request pods.PodPipelineRequest) (v1alpha1.DecisionResult, error) { +func (m *mockPodPipeline) Run(request pods.PodPipelineRequest) (lib.PipelineResult, error) { if len(request.Nodes) == 0 { - return v1alpha1.DecisionResult{}, nil + return lib.PipelineResult{}, nil } // Return the first node as the target host targetHost := request.Nodes[0].Name - return v1alpha1.DecisionResult{ + return lib.PipelineResult{ TargetHost: &targetHost, }, nil } diff --git a/internal/scheduling/reservations/capacity_accounting.go b/internal/scheduling/reservations/accounting/accounting.go similarity index 74% rename from internal/scheduling/reservations/capacity_accounting.go rename to internal/scheduling/reservations/accounting/accounting.go index f73debff0..f896800e9 100644 --- a/internal/scheduling/reservations/capacity_accounting.go +++ b/internal/scheduling/reservations/accounting/accounting.go @@ -1,7 +1,12 @@ // Copyright SAP SE // SPDX-License-Identifier: Apache-2.0 -package reservations +// Package accounting holds the reservation capacity-accounting math and flavor +// group lookup. Both are pure, self-contained helpers over Reservation/Knowledge +// CRDs with no dependency on the scheduling pipeline, so they can live below both +// the reservation controllers and the Nova scheduler tree without forming an +// import cycle. +package accounting import ( hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" @@ -124,9 +129,9 @@ func HostFreeCapacity(hostReservations []v1alpha1.Reservation, hv hv1.Hypervisor // filter_has_enough_capacity to ensure consistent accounting. // // CommittedResourceReservations: confirmed VMs already appear in hv.Status.Allocation, -// so blocking the full slot would double-count them. The effective block is: -// max(slot − confirmedVMs, specOnlyVMs), clamped to zero. Skipped (full slot used) when -// ignoreAllocations is true or when mid-migration (TargetHost != Status.Host). +// so blocking the full slot would double-count them. The remaining block is +// max(slot − sum(confirmed VM resources), 0) per resource. Skipped when ignoreAllocations +// is true or when mid-migration (TargetHost != Status.Host). // // FailoverReservations: always block the full Spec.Resources. // @@ -142,52 +147,25 @@ func UnusedReservationCapacity(res *v1alpha1.Reservation, ignoreAllocations bool if res.Spec.Type == v1alpha1.ReservationTypeCommittedResource && !ignoreAllocations && res.Spec.TargetHost == res.Status.Host && - res.Spec.CommittedResourceReservation != nil && - len(res.Spec.CommittedResourceReservation.Allocations) > 0 { - confirmedResources := make(map[hv1.ResourceName]resource.Quantity) - specOnlyResources := make(map[hv1.ResourceName]resource.Quantity) - - statusAllocs := map[string]string{} - if res.Status.CommittedResourceReservation != nil { - statusAllocs = res.Status.CommittedResourceReservation.Allocations - } - - for instanceUUID, allocation := range res.Spec.CommittedResourceReservation.Allocations { - _, isConfirmed := statusAllocs[instanceUUID] - for resourceName, quantity := range allocation.Resources { - if isConfirmed { - existing := confirmedResources[resourceName] - existing.Add(quantity) - confirmedResources[resourceName] = existing - } else { - existing := specOnlyResources[resourceName] - existing.Add(quantity) - specOnlyResources[resourceName] = existing + res.Status.CommittedResourceReservation != nil && + len(res.Status.CommittedResourceReservation.Allocation) > 0 { + // Compute remaining slot capacity: slot - sum of confirmed VM resources. + result := make(map[hv1.ResourceName]resource.Quantity, len(res.Spec.Resources)) + zero := resource.MustParse("0") + for rn, slotQty := range res.Spec.Resources { + remaining := slotQty.DeepCopy() + for _, alloc := range res.Status.CommittedResourceReservation.Allocation { + if used, ok := alloc.Resources[rn]; ok { + remaining.Sub(used) } } - } - - result := make(map[hv1.ResourceName]resource.Quantity) - zero := resource.Quantity{} - for resourceName, slotSize := range res.Spec.Resources { - confirmed := confirmedResources[resourceName] - specOnly := specOnlyResources[resourceName] - - remaining := slotSize.DeepCopy() - remaining.Sub(confirmed) if remaining.Cmp(zero) < 0 { remaining = zero.DeepCopy() } - - if specOnly.Cmp(remaining) > 0 { - result[resourceName] = specOnly.DeepCopy() - } else { - result[resourceName] = remaining - } + result[rn] = remaining } return result - } else { - // FailoverReservations are always fully blocked and unused. - return res.Spec.Resources } + // No confirmed allocations, or ignoreAllocations, or mid-migration: block the full slot. + return res.Spec.Resources } diff --git a/internal/scheduling/reservations/capacity_accounting_test.go b/internal/scheduling/reservations/accounting/accounting_test.go similarity index 82% rename from internal/scheduling/reservations/capacity_accounting_test.go rename to internal/scheduling/reservations/accounting/accounting_test.go index 140b58f8f..1d00239e0 100644 --- a/internal/scheduling/reservations/capacity_accounting_test.go +++ b/internal/scheduling/reservations/accounting/accounting_test.go @@ -1,12 +1,13 @@ // Copyright SAP SE // SPDX-License-Identifier: Apache-2.0 -package reservations +package accounting import ( "testing" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -61,59 +62,59 @@ func TestUnusedReservationCapacity(t *testing.T) { wantMemoryBytes: 480 * 1024 * 1024 * 1024, }, { - name: "CR 1 confirmed VM (240Gi), slot=480Gi: remaining = 240Gi blocked", + name: "CR 1 confirmed VM in status.allocations, slot=480Gi: block = max(480Gi-480Gi, 0) = 0", res: &v1alpha1.Reservation{ Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(480)}, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - "vm-1": {Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(240)}}, - }, - }, + Type: v1alpha1.ReservationTypeCommittedResource, + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(480)}, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{}, }, Status: v1alpha1.ReservationStatus{ CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - Allocations: map[string]string{"vm-1": "host-a"}, + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + "vm-1": { + ObjectRef: corev1.ObjectReference{Name: "vm-1"}, + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(480)}, + }, + }, }, }, }, - wantMemoryBytes: 240 * 1024 * 1024 * 1024, + wantMemoryBytes: 0, }, { - name: "CR slot fully consumed by confirmed VMs: block = 0", + name: "CR slot fully consumed by 2 confirmed VMs: block = max(480Gi-240Gi-240Gi, 0) = 0", res: &v1alpha1.Reservation{ Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(480)}, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - "vm-1": {Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(240)}}, - "vm-2": {Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(240)}}, - }, - }, + Type: v1alpha1.ReservationTypeCommittedResource, + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(480)}, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{}, }, Status: v1alpha1.ReservationStatus{ CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - Allocations: map[string]string{"vm-1": "host-a", "vm-2": "host-a"}, + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + "vm-1": { + ObjectRef: corev1.ObjectReference{Name: "vm-1"}, + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(240)}, + }, + "vm-2": { + ObjectRef: corev1.ObjectReference{Name: "vm-2"}, + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(240)}, + }, + }, }, }, }, wantMemoryBytes: 0, }, { - name: "CR spec-only VM (240Gi), slot=480Gi, no confirmed: specOnly < remaining → full slot blocked", + name: "CR slot with no confirmed VMs: full slot blocked (conservative — per-VM sizes not tracked yet)", res: &v1alpha1.Reservation{ Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(480)}, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - "vm-1": {Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(240)}}, - }, - }, + Type: v1alpha1.ReservationTypeCommittedResource, + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(480)}, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{}, }, - // vm-1 not in status → spec-only }, wantMemoryBytes: 480 * 1024 * 1024 * 1024, }, @@ -121,19 +122,20 @@ func TestUnusedReservationCapacity(t *testing.T) { name: "CR mid-migration (TargetHost != Status.Host): full slot blocked despite confirmed VMs", res: &v1alpha1.Reservation{ Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: "new-host", - Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(480)}, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - "vm-1": {Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(240)}}, - }, - }, + Type: v1alpha1.ReservationTypeCommittedResource, + TargetHost: "new-host", + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(480)}, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{}, }, Status: v1alpha1.ReservationStatus{ Host: "old-host", // differs from TargetHost → migration in progress CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - Allocations: map[string]string{"vm-1": "old-host"}, + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + "vm-1": { + ObjectRef: corev1.ObjectReference{Name: "vm-1"}, + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(480)}, + }, + }, }, }, }, @@ -143,17 +145,18 @@ func TestUnusedReservationCapacity(t *testing.T) { name: "CR ignoreAllocations=true: full slot blocked regardless of confirmed VMs", res: &v1alpha1.Reservation{ Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(480)}, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - "vm-1": {Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(240)}}, - }, - }, + Type: v1alpha1.ReservationTypeCommittedResource, + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(480)}, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{}, }, Status: v1alpha1.ReservationStatus{ CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - Allocations: map[string]string{"vm-1": "host-a"}, + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + "vm-1": { + ObjectRef: corev1.ObjectReference{Name: "vm-1"}, + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(480)}, + }, + }, }, }, }, @@ -312,30 +315,31 @@ func TestHostFreeCapacity(t *testing.T) { hv.Status.Allocation = map[hv1.ResourceName]resource.Quantity{ hv1.ResourceMemory: gib(256), } - // 512 GiB slot with 256 GiB confirmed VM → block = 512-256 = 256 GiB + // 512 GiB slot with 1 confirmed VM → block = 0 (VM in HV allocation; slot unblocked) slot := v1alpha1.Reservation{ ObjectMeta: metav1.ObjectMeta{Name: "slot-1"}, Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: "host", - Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(512)}, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - "vm-1": {Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(256)}}, - }, - }, + Type: v1alpha1.ReservationTypeCommittedResource, + TargetHost: "host", + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(512)}, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{}, }, Status: v1alpha1.ReservationStatus{ Host: "host", CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - Allocations: map[string]string{"vm-1": "host"}, + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + "vm-1": { + ObjectRef: corev1.ObjectReference{Name: "vm-1"}, + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: gib(512)}, + }, + }, }, }, } free := HostFreeCapacity([]v1alpha1.Reservation{slot}, hv) - // 1024 - 256 (alloc/vm) - 256 (remaining slot block) = 512 - if freeMemGiB(free) != 512 { - t.Errorf("expected 512 GiB free, got %d", freeMemGiB(free)) + // 1024 - 256 (hv.Allocation) - 0 (slot block = 0, VMs confirmed) = 768 + if freeMemGiB(free) != 768 { + t.Errorf("expected 768 GiB free, got %d", freeMemGiB(free)) } }) diff --git a/internal/scheduling/reservations/capacity/config.go b/internal/scheduling/reservations/capacity/config.go index 4b9292042..beab45642 100644 --- a/internal/scheduling/reservations/capacity/config.go +++ b/internal/scheduling/reservations/capacity/config.go @@ -29,9 +29,6 @@ type Config struct { // PlaceablePipeline is the scheduler pipeline used for the current-state probe. // This pipeline considers current VM allocations to determine remaining placement capacity. PlaceablePipeline string `json:"capacityPlaceablePipeline"` - - // SchedulerURL is the endpoint of the nova external scheduler. - SchedulerURL string `json:"schedulerURL"` } // ApplyDefaults fills in any unset values with defaults. @@ -49,9 +46,6 @@ func (c *Config) ApplyDefaults() { if c.PlaceablePipeline == "" { c.PlaceablePipeline = defaults.PlaceablePipeline } - if c.SchedulerURL == "" { - c.SchedulerURL = defaults.SchedulerURL - } } // Validate checks that the config is internally consistent after defaults are applied. @@ -69,6 +63,5 @@ func DefaultConfig() Config { MinReconcileInterval: metav1.Duration{Duration: 30 * time.Second}, TotalPipeline: "kvm-general-purpose-load-balancing", PlaceablePipeline: "kvm-general-purpose-load-balancing", - SchedulerURL: "http://localhost:8080/scheduler/nova/external", } } diff --git a/internal/scheduling/reservations/capacity/controller.go b/internal/scheduling/reservations/capacity/controller.go index caeab9556..16c443e67 100644 --- a/internal/scheduling/reservations/capacity/controller.go +++ b/internal/scheduling/reservations/capacity/controller.go @@ -28,8 +28,10 @@ import ( schedulerapi "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/scheduling" "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + knowledge "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" + "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/accounting" "github.com/cobaltcore-dev/cortex/pkg/multicluster" "github.com/go-logr/logr" ) @@ -102,20 +104,20 @@ func capacityMapsEqual(a, b map[hv1.ResourceName]resource.Quantity) bool { // periodic floor timer. All watch events are coalesced into a single reconcile key so // rapid changes produce at most one queued reconcile. type Reconciler struct { - client client.Client - vmSource reservations.VMSource - schedulerClient *reservations.SchedulerClient - config Config + client client.Client + vmSource reservations.VMSource + scheduler *sched.Placement + config Config lastReconcileAt time.Time } -func NewController(c client.Client, config Config, vmSource reservations.VMSource) *Reconciler { +func NewController(c client.Client, config Config, vmSource reservations.VMSource, scheduler *sched.Placement) *Reconciler { return &Reconciler{ - client: c, - vmSource: vmSource, - schedulerClient: reservations.NewSchedulerClient(config.SchedulerURL), - config: config, + client: c, + vmSource: vmSource, + scheduler: scheduler, + config: config, } } @@ -195,7 +197,7 @@ type vmUsage struct { // probeGroupResult holds the outcome of probing all flavors in one (group × AZ). type probeGroupResult struct { groupName string - groupData compute.FlavorGroupFeature + groupData knowledge.FlavorGroupFeature flavors []v1alpha1.FlavorCapacityStatus // allFresh is false if any scheduler probe failed; the group's CRD is left unchanged. allFresh bool @@ -223,8 +225,7 @@ func (c *Reconciler) reconcileAll(ctx context.Context) error { logger := LoggerFromContext(ctx) startTime := time.Now() - knowledge := &reservations.FlavorGroupKnowledgeClient{Client: c.client} - flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, c.client, nil) if err != nil { return fmt.Errorf("failed to get flavor groups: %w", err) } @@ -272,7 +273,7 @@ func (c *Reconciler) reconcileAll(ctx context.Context) error { // On error returns an empty map with fresh=false — callers must not overwrite running fields. func (c *Reconciler) computeVMUsage( ctx context.Context, - flavorGroups map[string]compute.FlavorGroupFeature, + flavorGroups map[string]knowledge.FlavorGroupFeature, hvs []hv1.Hypervisor, ) map[vmUsageKey]vmUsage { @@ -376,7 +377,7 @@ func hvRemainingResources(hv hv1.Hypervisor, blockedResources map[string]int64) func (c *Reconciler) probeGroup( ctx context.Context, groupName string, - groupData compute.FlavorGroupFeature, + groupData knowledge.FlavorGroupFeature, az string, hvByName map[string]hv1.Hypervisor, blockedByReservations map[string]map[string]int64, @@ -402,7 +403,7 @@ func (c *Reconciler) probeGroup( } // Probe all flavors. Sort for stable CRD output. - flavors := make([]compute.FlavorInGroup, len(groupData.Flavors)) + flavors := make([]knowledge.FlavorInGroup, len(groupData.Flavors)) copy(flavors, groupData.Flavors) sort.Slice(flavors, func(i, j int) bool { return flavors[i].Name < flavors[j].Name }) @@ -520,7 +521,7 @@ func buildSplitInputs( func (c *Reconciler) reconcileAZ( ctx context.Context, az string, - flavorGroups map[string]compute.FlavorGroupFeature, + flavorGroups map[string]knowledge.FlavorGroupFeature, hvByName map[string]hv1.Hypervisor, blockedByReservations map[string]map[string]int64, committedByGroupAZ map[vmUsageKey]map[string]int64, @@ -643,7 +644,7 @@ func sumEffectiveCapacity(hostNames []string, hvByName map[string]hv1.Hypervisor // computeTotalCapacity returns the maximum memory bytes and CPU cores representable // by the flavor with the highest slot count in the group (empty-datacenter view). -func computeTotalCapacity(newFlavors []v1alpha1.FlavorCapacityStatus, flavorSpecByName map[string]compute.FlavorInGroup) (maxMemBytes, maxCPUCores int64) { +func computeTotalCapacity(newFlavors []v1alpha1.FlavorCapacityStatus, flavorSpecByName map[string]knowledge.FlavorInGroup) (maxMemBytes, maxCPUCores int64) { for _, f := range newFlavors { spec, ok := flavorSpecByName[f.FlavorName] if !ok || f.TotalCapacityVMSlots <= 0 { @@ -665,7 +666,7 @@ func computeTotalCapacity(newFlavors []v1alpha1.FlavorCapacityStatus, flavorSpec func (c *Reconciler) writeCRD( ctx context.Context, groupName string, - groupData compute.FlavorGroupFeature, + groupData knowledge.FlavorGroupFeature, az string, newFlavors []v1alpha1.FlavorCapacityStatus, committedCapacity int64, @@ -696,7 +697,7 @@ func (c *Reconciler) writeCRD( return fmt.Errorf("failed to get FlavorGroupCapacity %s: %w", crdName, err) } - flavorSpecByName := make(map[string]compute.FlavorInGroup, len(groupData.Flavors)) + flavorSpecByName := make(map[string]knowledge.FlavorInGroup, len(groupData.Flavors)) for _, f := range groupData.Flavors { flavorSpecByName[f.Name] = f } @@ -783,7 +784,7 @@ func (c *Reconciler) markCRDNotReady(ctx context.Context, groupName, az string) // ignoreAllocations=true (total probe) uses raw effective capacity; false (placeable probe) subtracts allocations. func (c *Reconciler) probeScheduler( ctx context.Context, - flavor compute.FlavorInGroup, + flavor knowledge.FlavorInGroup, az, pipeline string, hvByName map[string]hv1.Hypervisor, ignoreAllocations bool, @@ -814,29 +815,50 @@ func (c *Reconciler) probeScheduler( } } - resp, err := c.schedulerClient.ScheduleReservation(ctx, reservations.ScheduleReservationRequest{ - InstanceUUID: "capacity-" + flavor.Name, - ProjectID: "cortex-capacity-probe", - FlavorName: flavor.Name, - MemoryMB: flavor.MemoryMB, - VCPUs: flavor.VCPUs, - FlavorExtraSpecs: flavor.ExtraSpecs, - AvailabilityZone: az, - Pipeline: pipeline, - EligibleHosts: eligibleHosts, - SchedulerHints: map[string]any{"_nova_check_type": string(schedulerapi.CapacityProbeIntent)}, - }, scheduling.Options{ - ReadOnly: true, - AssumeEmptyHosts: ignoreAllocations, - IgnoredReservationTypes: ignoredReservationTypes, - SkipInflight: true, - SkipCommittedResourceTracking: true, - }) + probeHosts := make([]sched.Host, 0, len(eligibleHosts)) + probeWeights := make(map[string]float64, len(eligibleHosts)) + for _, h := range eligibleHosts { + probeHosts = append(probeHosts, sched.Host{ComputeHost: h.ComputeHost, HypervisorHostname: h.HypervisorHostname}) + probeWeights[h.ComputeHost] = 0.0 + } + // Honor the flavor's trait and hypervisor-type constraints so the probe only + // counts compatible hosts (the typed request is built directly, so + // filter_has_requested_traits/filter_capabilities would otherwise see empty + // slices and become no-ops). No hypervisor-type default — only match when the + // flavor names one. + requiredTraits, forbiddenTraits := schedulerapi.SplitTraits(flavor.ExtraSpecs) + out, err := c.scheduler.RunWithoutLock(ctx, sched.PlacementRequest{ + Pipeline: pipeline, + GatherAllCandidates: true, + HypervisorFamily: schedulerapi.HypervisorFamilyKVM, + Hosts: probeHosts, + Weights: probeWeights, + Intent: schedulerapi.CapacityProbeIntent, + InstanceUUID: "capacity-" + flavor.Name, + NumInstances: 1, // probe a single flavor slot; NumInstances must be >= 1 or capacity/bin-pack math treats the host as free + ProjectID: "cortex-capacity-probe", + AvailabilityZone: az, + RequiredTraits: requiredTraits, + ForbiddenTraits: forbiddenTraits, + RequestedCapabilities: schedulerapi.HypervisorTypeCapability(flavor.ExtraSpecs, false), + AcceleratorsRequested: schedulerapi.RequestsAccelerators(flavor.ExtraSpecs), + Flavor: schedulerapi.FlavorSpec{ + VCPUs: flavor.VCPUs, + MemoryMB: flavor.MemoryMB, + }, + Options: scheduling.Options{ + ReadOnly: true, + AssumeEmptyHosts: ignoreAllocations, + IgnoredReservationTypes: ignoredReservationTypes, + SkipInflight: true, + SkipCommittedResourceTracking: true, + }, + }, nil, nil) if err != nil { return 0, 0, nil, fmt.Errorf("scheduler call failed (pipeline=%s): %w", pipeline, err) } - for _, hostName := range resp.Hosts { + for _, hostName := range out.OrderedHosts { hv, ok := hvByName[hostName] if !ok { continue @@ -897,7 +919,7 @@ func (c *Reconciler) blockedResourcesByHost(ctx context.Context) (map[string]map continue } - resourcesToBlock := reservations.UnusedReservationCapacity(res, false) + resourcesToBlock := accounting.UnusedReservationCapacity(res, false) for host := range hostsToBlock { if blocked[host] == nil { blocked[host] = make(map[string]int64) @@ -921,7 +943,7 @@ func (c *Reconciler) blockedResourcesByHost(ctx context.Context) (map[string]map // Only placed reservations are included, each once. func (c *Reconciler) reservedResourcesByGroupAZ( ctx context.Context, - flavorGroups map[string]compute.FlavorGroupFeature, + flavorGroups map[string]knowledge.FlavorGroupFeature, hvByName map[string]hv1.Hypervisor, ) (committed, failover map[vmUsageKey]map[string]int64, err error) { @@ -968,7 +990,7 @@ func (c *Reconciler) reservedResourcesByGroupAZ( "reservation", res.Name, "type", res.Spec.Type, "az", az) continue } - blockedRes := reservations.UnusedReservationCapacity(res, false) + blockedRes := accounting.UnusedReservationCapacity(res, false) // Route each reservation to exactly one bucket by type; the two are disjoint. bucket := committed if res.Spec.Type == v1alpha1.ReservationTypeFailover { @@ -993,7 +1015,7 @@ func (c *Reconciler) reservedResourcesByGroupAZ( // to its group) when useFlavorGroupResources is disabled. Returns false if none can be determined. func resolveReservationGroup( res *v1alpha1.Reservation, - flavorGroups map[string]compute.FlavorGroupFeature, + flavorGroups map[string]knowledge.FlavorGroupFeature, flavorToGroup map[string]string, ) (string, bool) { diff --git a/internal/scheduling/reservations/capacity/controller_test.go b/internal/scheduling/reservations/capacity/controller_test.go index 1894c859e..5501166ec 100644 --- a/internal/scheduling/reservations/capacity/controller_test.go +++ b/internal/scheduling/reservations/capacity/controller_test.go @@ -5,9 +5,7 @@ package capacity import ( "context" - "encoding/json" - "net/http" - "net/http/httptest" + "errors" "regexp" "sort" "testing" @@ -25,6 +23,8 @@ import ( schedulerapi "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" + "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" ) @@ -99,38 +99,98 @@ func newHypervisor(name, az string, memoryBytes int64) *hv1.Hypervisor { return hv } -// newMockSchedulerServer creates an httptest server that always returns the given host list. -func newMockSchedulerServer(t *testing.T, hosts []string) *httptest.Server { - t.Helper() - return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - resp := schedulerapi.ExternalSchedulerResponse{Hosts: hosts} - if err := json.NewEncoder(w).Encode(resp); err != nil { - t.Errorf("mock scheduler: failed to encode response: %v", err) - } - })) +// fakePipeline returns a fixed ordered-host list (modelling a scheduler run). +type fakePipeline struct{ hosts []string } + +func (p fakePipeline) Run(sched.PlacementRequest) (sched.PlacementResult, error) { + return sched.PlacementResult{OrderedHosts: p.hosts}, nil } -// newMockSchedulerServerPerPipeline returns different candidate hosts depending on the request's -// pipeline, so a test can model a host that passes the total (empty-datacenter) probe but drops -// out of the placeable probe because it is fully occupied. -func newMockSchedulerServerPerPipeline(t *testing.T, hostsByPipeline map[string][]string) *httptest.Server { - t.Helper() - return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - var req schedulerapi.ExternalSchedulerRequest - if err := json.NewDecoder(r.Body).Decode(&req); err != nil { - t.Errorf("mock scheduler: failed to decode request: %v", err) - } - resp := schedulerapi.ExternalSchedulerResponse{Hosts: hostsByPipeline[req.Pipeline]} - if err := json.NewEncoder(w).Encode(resp); err != nil { - t.Errorf("mock scheduler: failed to encode response: %v", err) - } - })) +// fakePipelineProvider serves a fakePipeline per pipeline name, returning the +// per-pipeline host list configured for the test. When allPipelines is set it +// returns that host list for every pipeline name regardless of hostsByPipeline. +type fakePipelineProvider struct { + hostsByPipeline map[string][]string + allPipelines []string + useAll bool +} + +func (f fakePipelineProvider) GetPipeline(name string) (lib.FilterWeigherPipeline[sched.PlacementRequest], bool) { + if f.useAll { + return fakePipeline{hosts: f.allPipelines}, true + } + return fakePipeline{hosts: f.hostsByPipeline[name]}, true +} + +// fakeGatherer is a no-op gatherer (tests preselect their own hosts). +type fakeGatherer struct{} + +func (fakeGatherer) MutateWithAllCandidates(context.Context, *sched.PlacementRequest) error { + return nil +} + +// newMockScheduler builds an in-process compute.Placement whose pipelines return +// the given per-pipeline host lists. Replaces the former httptest mock server. +func newMockScheduler(hostsByPipeline map[string][]string) *sched.Placement { + return sched.NewPlacement(fakePipelineProvider{hostsByPipeline: hostsByPipeline}, fakeGatherer{}) +} + +// newMockSchedulerHosts is a convenience for a scheduler that returns the same +// host list regardless of pipeline. +func newMockSchedulerHosts(hosts []string) *sched.Placement { + return sched.NewPlacement(fakePipelineProvider{allPipelines: hosts, useAll: true}, fakeGatherer{}) +} + +// fakeErrPipeline models a scheduler run that fails (replaces the former +// HTTP 500 mock server). +type fakeErrPipeline struct{} + +func (fakeErrPipeline) Run(sched.PlacementRequest) (sched.PlacementResult, error) { + return sched.PlacementResult{}, errors.New("boom") +} + +type fakeErrPipelineProvider struct{} + +func (fakeErrPipelineProvider) GetPipeline(string) (lib.FilterWeigherPipeline[sched.PlacementRequest], bool) { + return fakeErrPipeline{}, true +} + +// newErrScheduler builds a scheduler whose pipeline always returns an error. +func newErrScheduler() *sched.Placement { + return sched.NewPlacement(fakeErrPipelineProvider{}, fakeGatherer{}) +} + +// capturePipeline records the request it is run with and returns a fixed host list. +type capturePipeline struct { + captured *sched.PlacementRequest + hosts []string +} + +func (p capturePipeline) Run(req sched.PlacementRequest) (sched.PlacementResult, error) { + *p.captured = req + return sched.PlacementResult{OrderedHosts: p.hosts}, nil +} + +type capturePipelineProvider struct { + captured *sched.PlacementRequest + hosts []string +} + +func (f capturePipelineProvider) GetPipeline(string) (lib.FilterWeigherPipeline[sched.PlacementRequest], bool) { + return capturePipeline(f), true } -// newController is a test helper that creates a Reconciler with a nil VMSource. -func newController(t *testing.T, c client.Client, cfg Config) *Reconciler { +// captureScheduler builds a scheduler that records the request it is run with +// into captured and returns the given hosts. +func captureScheduler(captured *sched.PlacementRequest, hosts []string) *sched.Placement { + return sched.NewPlacement(capturePipelineProvider{captured: captured, hosts: hosts}, fakeGatherer{}) +} + +// newController is a test helper that creates a Reconciler with a nil VMSource +// and the given scheduler. +func newController(t *testing.T, c client.Client, cfg Config, scheduler *sched.Placement) *Reconciler { t.Helper() - return NewController(c, cfg, nil) + return NewController(c, cfg, nil, scheduler) } // --- unit tests for pure helper functions --- @@ -224,14 +284,12 @@ func TestReconcileAZ_CreatesCRD(t *testing.T) { Build() // Both probes return host-1 → total capacity = floor(4GiB/4GiB) = 1, placeable = 1. - schedulerServer := newMockSchedulerServer(t, []string{"host-1"}) - defer schedulerServer.Close() + scheduler := newMockSchedulerHosts([]string{"host-1"}) ctrl := newController(t, fakeClient, Config{ - SchedulerURL: schedulerServer.URL, TotalPipeline: "kvm-report-capacity", PlaceablePipeline: "kvm-general-purpose", - }) + }, scheduler) smallFlavor := compute.FlavorInGroup{Name: groupName + "-small", MemoryMB: memMB, VCPUs: 2} groupData := compute.FlavorGroupFeature{ @@ -311,17 +369,15 @@ func TestReconcileAZ_RawCapacityIncludesOccupiedHosts(t *testing.T) { // Total probe sees both hosts (empty-datacenter); placeable probe sees only host-1 because // host-2 is fully occupied and the smallest flavor no longer fits. - schedulerServer := newMockSchedulerServerPerPipeline(t, map[string][]string{ + scheduler := newMockScheduler(map[string][]string{ totalPipe: {"host-1", "host-2"}, placeablePipe: {"host-1"}, }) - defer schedulerServer.Close() ctrl := newController(t, fakeClient, Config{ - SchedulerURL: schedulerServer.URL, TotalPipeline: totalPipe, PlaceablePipeline: placeablePipe, - }) + }, scheduler) smallFlavor := compute.FlavorInGroup{Name: groupName + "-small", MemoryMB: memMB, VCPUs: 2} groupData := compute.FlavorGroupFeature{ @@ -370,14 +426,12 @@ func TestReconcileAZ_WritesReservedCapacity(t *testing.T) { WithStatusSubresource(&v1alpha1.FlavorGroupCapacity{}, &v1alpha1.Knowledge{}). Build() - schedulerServer := newMockSchedulerServer(t, []string{"host-1"}) - defer schedulerServer.Close() + scheduler := newMockSchedulerHosts([]string{"host-1"}) ctrl := newController(t, fakeClient, Config{ - SchedulerURL: schedulerServer.URL, TotalPipeline: "kvm-report-capacity", PlaceablePipeline: "kvm-general-purpose", - }) + }, scheduler) smallFlavor := compute.FlavorInGroup{Name: groupName + "-small", MemoryMB: memMB, VCPUs: flavorCPUs} groupData := compute.FlavorGroupFeature{SmallestFlavor: smallFlavor, Flavors: []compute.FlavorInGroup{smallFlavor}} @@ -495,7 +549,7 @@ func TestReservedResourcesByGroupAZ(t *testing.T) { WithObjects(committedA, failoverB, failoverAByFlavor, pending, inflight). Build() - c := newController(t, fakeClient, Config{}) + c := newController(t, fakeClient, Config{}, newMockSchedulerHosts(nil)) flavorA := compute.FlavorInGroup{Name: "group-a-small", MemoryMB: 4096, VCPUs: flavorCPUs} flavorB := compute.FlavorInGroup{Name: "group-b-small", MemoryMB: 4096, VCPUs: flavorCPUs} @@ -546,17 +600,11 @@ func TestReconcileAZ_SkipsCRDWriteOnSchedulerError(t *testing.T) { WithStatusSubresource(&v1alpha1.FlavorGroupCapacity{}, &v1alpha1.Knowledge{}). Build() - // Scheduler returns 500 to simulate error. - failServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusInternalServerError) - })) - defer failServer.Close() - + // Scheduler returns an error to simulate a failed probe. ctrl := newController(t, fakeClient, Config{ - SchedulerURL: failServer.URL, TotalPipeline: "kvm-report-capacity", PlaceablePipeline: "kvm-general-purpose", - }) + }, newErrScheduler()) smallFlavor := compute.FlavorInGroup{Name: groupName + "-small", MemoryMB: memMB, VCPUs: 2} groupData := compute.FlavorGroupFeature{ @@ -616,17 +664,11 @@ func TestReconcileAZ_MarksExistingCRDNotReadyOnSchedulerError(t *testing.T) { t.Fatalf("failed to set Ready=True: %v", err) } - // Scheduler returns 500 to simulate error. - failServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusInternalServerError) - })) - defer failServer.Close() - + // Scheduler returns an error to simulate a failed probe. ctrl := newController(t, fakeClient, Config{ - SchedulerURL: failServer.URL, TotalPipeline: "kvm-report-capacity", PlaceablePipeline: "kvm-general-purpose", - }) + }, newErrScheduler()) smallFlavor := compute.FlavorInGroup{Name: groupName + "-small", MemoryMB: memMB, VCPUs: 2} groupData := compute.FlavorGroupFeature{ @@ -685,14 +727,12 @@ func TestReconcileAZ_IdempotentUpdate(t *testing.T) { WithStatusSubresource(&v1alpha1.FlavorGroupCapacity{}, &v1alpha1.Knowledge{}). Build() - schedulerServer := newMockSchedulerServer(t, []string{"host-1"}) - defer schedulerServer.Close() + scheduler := newMockSchedulerHosts([]string{"host-1"}) ctrl := newController(t, fakeClient, Config{ - SchedulerURL: schedulerServer.URL, TotalPipeline: "kvm-report-capacity", PlaceablePipeline: "kvm-general-purpose", - }) + }, scheduler) smallFlavor := compute.FlavorInGroup{Name: groupName + "-small", MemoryMB: memMB, VCPUs: 2} groupData := compute.FlavorGroupFeature{ @@ -731,10 +771,9 @@ func TestReconcileAll_SkipsGroupsWithNoAZs(t *testing.T) { Build() ctrl := newController(t, fakeClient, Config{ - SchedulerURL: "http://localhost:9999", // unreachable; not called TotalPipeline: "kvm-report-capacity", PlaceablePipeline: "kvm-general-purpose", - }) + }, newMockSchedulerHosts(nil)) // scheduler not called (no AZs) if err := ctrl.reconcileAll(context.Background()); err != nil { t.Errorf("reconcileAll with no hypervisors returned error: %v", err) @@ -760,10 +799,9 @@ func TestProbeScheduler_CapacityCalculation(t *testing.T) { fakeClient := fake.NewClientBuilder().WithScheme(scheme).Build() // Scheduler returns both hosts. - srv := newMockSchedulerServer(t, []string{"host-1", "host-2"}) - defer srv.Close() + scheduler := newMockSchedulerHosts([]string{"host-1", "host-2"}) - c := newController(t, fakeClient, Config{SchedulerURL: srv.URL}) + c := newController(t, fakeClient, Config{}, scheduler) hvByName := map[string]hv1.Hypervisor{ "host-1": *hv1Obj, "host-2": *hv2Obj, @@ -810,10 +848,9 @@ func TestProbeScheduler_SubtractsAllocationsWhenNotIgnored(t *testing.T) { } fakeClient := fake.NewClientBuilder().WithScheme(scheme).Build() - srv := newMockSchedulerServer(t, []string{"host-1"}) - defer srv.Close() + scheduler := newMockSchedulerHosts([]string{"host-1"}) - c := newController(t, fakeClient, Config{SchedulerURL: srv.URL}) + c := newController(t, fakeClient, Config{}, scheduler) hvByName := map[string]hv1.Hypervisor{"host-1": *hv} flavor := compute.FlavorInGroup{Name: "test-flavor", MemoryMB: memMB} @@ -853,14 +890,12 @@ func TestReconcileAll_MultipleGroupsAndAZs(t *testing.T) { WithStatusSubresource(&v1alpha1.FlavorGroupCapacity{}, &v1alpha1.Knowledge{}). Build() - srv := newMockSchedulerServer(t, []string{}) - defer srv.Close() + scheduler := newMockSchedulerHosts([]string{}) c := newController(t, fakeClient, Config{ - SchedulerURL: srv.URL, TotalPipeline: "kvm-report-capacity", PlaceablePipeline: "kvm-general-purpose", - }) + }, scheduler) if err := c.reconcileAll(context.Background()); err != nil { t.Fatalf("reconcileAll failed: %v", err) @@ -908,10 +943,9 @@ func TestReconcileAll_FlavorGroupsKnowledgeNotReady(t *testing.T) { Build() c := newController(t, fakeClient, Config{ - SchedulerURL: "http://localhost:9999", TotalPipeline: "kvm-report-capacity", PlaceablePipeline: "kvm-general-purpose", - }) + }, newMockSchedulerHosts(nil)) // scheduler not called (knowledge not ready) // Should return an error when knowledge is not ready. if err := c.reconcileAll(context.Background()); err == nil { @@ -925,7 +959,7 @@ func TestReconcileAZ_ZeroMemoryFlavorSkipped(t *testing.T) { WithScheme(scheme). WithStatusSubresource(&v1alpha1.FlavorGroupCapacity{}). Build() - c := newController(t, fakeClient, Config{}) + c := newController(t, fakeClient, Config{}, newMockSchedulerHosts(nil)) groupData := compute.FlavorGroupFeature{ SmallestFlavor: compute.FlavorInGroup{Name: "bad-flavor", MemoryMB: 0}, @@ -1075,12 +1109,6 @@ func TestComputeTotalCapacity(t *testing.T) { } } -// Verify that the module-level log variable from reservations package doesn't -// collide with the one in this package. -func TestPackageLogVar(t *testing.T) { - _ = reservations.NewSchedulerClient("http://localhost") -} - func TestSumCommittedCapacity(t *testing.T) { const ( groupName = "hana-v2" @@ -1131,7 +1159,7 @@ func TestSumCommittedCapacity(t *testing.T) { WithObjects(objects...). Build() - c := newController(t, fakeClient, Config{}) + c := newController(t, fakeClient, Config{}, newMockSchedulerHosts(nil)) // smallestFlavorBytes = 4GiB → cr1 = 8GiB/4GiB = 2 slots, cr2 = 4GiB/4GiB = 1 slot → total = 3. got, err := c.sumCommittedCapacity(context.Background(), groupName, az, memBytes) if err != nil { @@ -1217,17 +1245,15 @@ func TestComputeVMUsage_ZerosOutWhenAllVMsRemoved(t *testing.T) { } // Scheduler returns host-1 so probes succeed. - schedulerServer := newMockSchedulerServer(t, []string{"host-1"}) - defer schedulerServer.Close() + scheduler := newMockSchedulerHosts([]string{"host-1"}) // VMSource returns empty list (all VMs removed). vmSource := &mockVMSource{vms: []reservations.VM{}} ctrl := NewController(fakeClient, Config{ - SchedulerURL: schedulerServer.URL, TotalPipeline: "kvm-report-capacity", PlaceablePipeline: "kvm-general-purpose", - }, vmSource) + }, vmSource, scheduler) smallFlavor := compute.FlavorInGroup{Name: groupName + "-small", MemoryMB: memMB, VCPUs: 2} groupData := compute.FlavorGroupFeature{ @@ -1284,10 +1310,9 @@ func TestProbeScheduler_SubtractsReservationBlocksWhenNotIgnored(t *testing.T) { } fakeClient := fake.NewClientBuilder().WithScheme(scheme).Build() - srv := newMockSchedulerServer(t, []string{"host-1"}) - defer srv.Close() + scheduler := newMockSchedulerHosts([]string{"host-1"}) - c := newController(t, fakeClient, Config{SchedulerURL: srv.URL}) + c := newController(t, fakeClient, Config{}, scheduler) hvByName := map[string]hv1.Hypervisor{"host-1": *hv} flavor := compute.FlavorInGroup{Name: "test-flavor", MemoryMB: memMB} @@ -1317,32 +1342,23 @@ func TestProbeScheduler_SetsCapacityProbeIntent(t *testing.T) { scheme := newTestScheme(t) hv := newHypervisor("host-1", "az-a", 4096*1024*1024) - var capturedReq schedulerapi.ExternalSchedulerRequest - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if err := json.NewDecoder(r.Body).Decode(&capturedReq); err != nil { - http.Error(w, err.Error(), http.StatusBadRequest) - return - } - json.NewEncoder(w).Encode(schedulerapi.ExternalSchedulerResponse{Hosts: []string{"host-1"}}) //nolint:errcheck - })) - defer srv.Close() + // Capture the request the scheduler receives so we can assert the probe + // intent and project are set. + var capturedReq sched.PlacementRequest + captured := captureScheduler(&capturedReq, []string{"host-1"}) - c := NewController(fake.NewClientBuilder().WithScheme(scheme).Build(), Config{SchedulerURL: srv.URL}, nil) + c := NewController(fake.NewClientBuilder().WithScheme(scheme).Build(), Config{}, nil, captured) hvByName := map[string]hv1.Hypervisor{"host-1": *hv} flavor := compute.FlavorInGroup{Name: "test-flavor", MemoryMB: 4096} if _, _, _, err := c.probeScheduler(context.Background(), flavor, "az-a", "test-pipeline", hvByName, true, nil); err != nil { t.Fatalf("probeScheduler failed: %v", err) } - hint, err := capturedReq.Spec.Data.GetSchedulerHintStr("_nova_check_type") - if err != nil { - t.Fatalf("failed to get _nova_check_type hint: %v", err) - } - if hint != string(schedulerapi.CapacityProbeIntent) { - t.Errorf("capacity probe must set _nova_check_type=%q, got %q", schedulerapi.CapacityProbeIntent, hint) + if capturedReq.Intent != schedulerapi.CapacityProbeIntent { + t.Errorf("capacity probe must set Intent=%q, got %q", schedulerapi.CapacityProbeIntent, capturedReq.Intent) } - if capturedReq.Spec.Data.ProjectID != "cortex-capacity-probe" { - t.Errorf("capacity probe must send ProjectID cortex-capacity-probe, got %q", capturedReq.Spec.Data.ProjectID) + if capturedReq.ProjectID != "cortex-capacity-probe" { + t.Errorf("capacity probe must send ProjectID cortex-capacity-probe, got %q", capturedReq.ProjectID) } } @@ -1367,16 +1383,14 @@ func TestReconcile_ReactsToKnowledgeChange(t *testing.T) { WithStatusSubresource(&v1alpha1.FlavorGroupCapacity{}, &v1alpha1.Knowledge{}). Build() - schedulerServer := newMockSchedulerServer(t, []string{"host-1"}) - defer schedulerServer.Close() + scheduler := newMockSchedulerHosts([]string{"host-1"}) r := NewController(fakeClient, Config{ - SchedulerURL: schedulerServer.URL, TotalPipeline: "kvm-report-capacity", PlaceablePipeline: "kvm-general-purpose", ReconcileInterval: metav1.Duration{Duration: 5 * time.Minute}, MinReconcileInterval: metav1.Duration{Duration: 30 * time.Second}, - }, nil) + }, nil, scheduler) req := ctrl.Request{NamespacedName: types.NamespacedName{Name: coalescedKey}} result, err := r.Reconcile(context.Background(), req) @@ -1418,17 +1432,15 @@ func TestReconcile_MinIntervalEarlyReturn(t *testing.T) { WithStatusSubresource(&v1alpha1.FlavorGroupCapacity{}, &v1alpha1.Knowledge{}). Build() - schedulerServer := newMockSchedulerServer(t, []string{"host-1"}) - defer schedulerServer.Close() + scheduler := newMockSchedulerHosts([]string{"host-1"}) minInterval := 30 * time.Second r := NewController(fakeClient, Config{ - SchedulerURL: schedulerServer.URL, TotalPipeline: "kvm-report-capacity", PlaceablePipeline: "kvm-general-purpose", ReconcileInterval: metav1.Duration{Duration: 5 * time.Minute}, MinReconcileInterval: metav1.Duration{Duration: minInterval}, - }, nil) + }, nil, scheduler) req := ctrl.Request{NamespacedName: types.NamespacedName{Name: coalescedKey}} diff --git a/internal/scheduling/reservations/commitments/api/change_commitments.go b/internal/scheduling/reservations/commitments/api/change_commitments.go index e4c04d70a..ab6314851 100644 --- a/internal/scheduling/reservations/commitments/api/change_commitments.go +++ b/internal/scheduling/reservations/commitments/api/change_commitments.go @@ -14,6 +14,7 @@ import ( "time" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + knowledge "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" commitments "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/commitments" "github.com/cobaltcore-dev/cortex/pkg/multicluster" @@ -140,7 +141,7 @@ func (api *HTTPAPI) HandleChangeCommitments(w http.ResponseWriter, r *http.Reque api.changeMutex.Lock() defer api.changeMutex.Unlock() - ctx := reservations.WithGlobalRequestID(context.Background(), "committed-resource-"+requestID) + ctx := reservations.WithGlobalRequestID(context.Background(), requestID) logger := commitments.LoggerFromContext(ctx).WithValues("endpoint", "/commitments/v1/change-commitments") if r.Method != http.MethodPost { @@ -179,23 +180,20 @@ func (api *HTTPAPI) HandleChangeCommitments(w http.ResponseWriter, r *http.Reque return } - { - knowledge := &reservations.FlavorGroupKnowledgeClient{Client: api.client} - if knowledgeCRD, err := knowledge.Get(ctx); err == nil && knowledgeCRD != nil { - var currentVersion int64 = -1 - if !knowledgeCRD.Status.LastContentChange.IsZero() { - currentVersion = knowledgeCRD.Status.LastContentChange.Unix() - } - if req.InfoVersion != currentVersion { - logger.Info("version mismatch in commitment change request", - "requestVersion", req.InfoVersion, - "currentVersion", currentVersion) - statusCode = http.StatusConflict - http.Error(w, fmt.Sprintf("Version mismatch: request version %d, current version %d. Please refresh and retry.", - req.InfoVersion, currentVersion), statusCode) - api.recordMetrics(req, resp, statusCode, startTime) - return - } + if knowledgeCRD, err := knowledge.GetFlavorGroupsKnowledge(ctx, api.client); err == nil && knowledgeCRD != nil { + var currentVersion int64 = -1 + if !knowledgeCRD.Status.LastContentChange.IsZero() { + currentVersion = knowledgeCRD.Status.LastContentChange.Unix() + } + if req.InfoVersion != currentVersion { + logger.Info("version mismatch in commitment change request", + "requestVersion", req.InfoVersion, + "currentVersion", currentVersion) + statusCode = http.StatusConflict + http.Error(w, fmt.Sprintf("Version mismatch: request version %d, current version %d. Please refresh and retry.", + req.InfoVersion, currentVersion), statusCode) + api.recordMetrics(req, resp, statusCode, startTime) + return } } @@ -231,8 +229,7 @@ func (api *HTTPAPI) HandleChangeCommitments(w http.ResponseWriter, r *http.Reque } func (api *HTTPAPI) processCommitmentChanges(ctx context.Context, w http.ResponseWriter, logger logr.Logger, req liquid.CommitmentChangeRequest, resp *liquid.CommitmentChangeResponse) error { - knowledge := &reservations.FlavorGroupKnowledgeClient{Client: api.client} - flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, api.client, nil) if err != nil { logger.Info("failed to get flavor groups from knowledge extractor", "error", err) http.Error(w, "caches not ready, please retry later", http.StatusServiceUnavailable) @@ -456,7 +453,7 @@ ProcessLoop: if len(toWatch) == 1 { sanitisedReason = "internal error on commitment " + strings.TrimPrefix(toWatch[0].name, "commitment-") } else { - sanitisedReason = "internal error (request ID: " + strings.TrimPrefix(reservations.GlobalRequestIDFromContext(ctx), "committed-resource-") + ")" + sanitisedReason = "internal error (request ID: " + reservations.GlobalRequestIDFromContext(ctx) + ")" } api.monitor.timeouts.WithLabelValues("false").Inc() rollback = true @@ -640,8 +637,7 @@ func rollbackCR(ctx context.Context, logger logr.Logger, k8sClient client.Client // performDryRun checks whether the net capacity delta in req can be placed by running // temporary probe CommittedResource CRDs through the controller, then always cleaning them up. func (api *HTTPAPI) performDryRun(ctx context.Context, logger logr.Logger, req liquid.CommitmentChangeRequest, resp *liquid.CommitmentChangeResponse) { - knowledge := &reservations.FlavorGroupKnowledgeClient{Client: api.client} - flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, api.client, nil) if err != nil { resp.RejectionReason = "dry run: caches not ready, please retry later" return @@ -728,7 +724,7 @@ func (api *HTTPAPI) performDryRun(ctx context.Context, logger logr.Logger, req l probeDomainID := firstProjectMeta.Domain.UUID // Probe ID is derived from the HTTP request ID for log traceability. - ctxRequestID := strings.TrimPrefix(reservations.GlobalRequestIDFromContext(ctx), "committed-resource-") + ctxRequestID := reservations.GlobalRequestIDFromContext(ctx) if len(ctxRequestID) > 8 { ctxRequestID = ctxRequestID[:8] } diff --git a/internal/scheduling/reservations/commitments/api/change_commitments_e2e_test.go b/internal/scheduling/reservations/commitments/api/change_commitments_e2e_test.go index b9c1978aa..f4963ef1b 100644 --- a/internal/scheduling/reservations/commitments/api/change_commitments_e2e_test.go +++ b/internal/scheduling/reservations/commitments/api/change_commitments_e2e_test.go @@ -12,7 +12,6 @@ package api import ( "context" - "encoding/json" "net/http" "net/http/httptest" "os" @@ -20,8 +19,9 @@ import ( "testing" "time" - schedulerdelegationapi "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" + "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" commitments "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/commitments" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "github.com/prometheus/client_golang/prometheus" @@ -46,19 +46,18 @@ const ( // e2eEnv is a full end-to-end test environment: real controllers, fake k8s client, // mock scheduler, and a background reconcile driver goroutine. type e2eEnv struct { - t *testing.T - k8sClient client.Client - httpServer *httptest.Server - schedulerSrv *httptest.Server - crCtrl *commitments.CommittedResourceController - resCtrl *commitments.CommitmentReservationController - cancelBg context.CancelFunc - bgDone chan struct{} + t *testing.T + k8sClient client.Client + httpServer *httptest.Server + crCtrl *commitments.CommittedResourceController + resCtrl *commitments.CommitmentReservationController + cancelBg context.CancelFunc + bgDone chan struct{} } // newE2EEnv creates an e2eEnv with the given flavors and scheduler handler. // The scheduler handler controls what the mock Nova scheduler returns. -func newE2EEnv(t *testing.T, flavors []*TestFlavor, infoVersion int64, schedulerHandler http.HandlerFunc) *e2eEnv { +func newE2EEnv(t *testing.T, flavors []*TestFlavor, infoVersion int64, scheduler *sched.Placement) *e2eEnv { t.Helper() log.SetLogger(zap.New(zap.WriteTo(os.Stderr), zap.UseDevMode(true))) @@ -98,8 +97,6 @@ func newE2EEnv(t *testing.T, flavors []*TestFlavor, infoVersion int64, scheduler }). Build() - schedulerSrv := httptest.NewServer(schedulerHandler) - crCtrl := &commitments.CommittedResourceController{ Client: k8sClient, Scheme: scheme, @@ -107,11 +104,10 @@ func newE2EEnv(t *testing.T, flavors []*TestFlavor, infoVersion int64, scheduler } resCtrl := &commitments.CommitmentReservationController{ - Client: k8sClient, - Scheme: scheme, + Client: k8sClient, + Scheme: scheme, + Scheduler: scheduler, Conf: commitments.ReservationControllerConfig{ - SchedulerURL: schedulerSrv.URL, - AllocationGracePeriod: metav1.Duration{Duration: 15 * time.Minute}, RequeueIntervalActive: metav1.Duration{Duration: 5 * time.Minute}, RequeueIntervalRetry: metav1.Duration{Duration: 100 * time.Millisecond}, }, @@ -134,14 +130,13 @@ func newE2EEnv(t *testing.T, flavors []*TestFlavor, infoVersion int64, scheduler ctx, cancel := context.WithCancel(context.Background()) env := &e2eEnv{ - t: t, - k8sClient: k8sClient, - httpServer: httpServer, - schedulerSrv: schedulerSrv, - crCtrl: crCtrl, - resCtrl: resCtrl, - cancelBg: cancel, - bgDone: make(chan struct{}), + t: t, + k8sClient: k8sClient, + httpServer: httpServer, + crCtrl: crCtrl, + resCtrl: resCtrl, + cancelBg: cancel, + bgDone: make(chan struct{}), } go env.driveReconciles(ctx) return env @@ -151,7 +146,6 @@ func (e *e2eEnv) close() { e.cancelBg() <-e.bgDone e.httpServer.Close() - e.schedulerSrv.Close() } // asCRTestEnv wraps e2eEnv as a CRTestEnv to reuse its HTTP-call and assertion helpers. @@ -274,27 +268,35 @@ func e2eIsTerminalReservation(res v1alpha1.Reservation) bool { // Scheduler handlers // ============================================================================ -func e2eAcceptScheduler(t *testing.T) http.HandlerFunc { - t.Helper() - return func(w http.ResponseWriter, r *http.Request) { - resp := &schedulerdelegationapi.ExternalSchedulerResponse{Hosts: []string{"host-1"}} - if err := json.NewEncoder(w).Encode(resp); err != nil { - t.Errorf("scheduler encode: %v", err) - } - } +// e2eFakePipeline models a scheduler run returning a fixed host list. +type e2eFakePipeline struct{ hosts []string } + +func (p e2eFakePipeline) Run(sched.PlacementRequest) (sched.PlacementResult, error) { + return sched.PlacementResult{OrderedHosts: p.hosts}, nil } -func e2eRejectScheduler(t *testing.T) http.HandlerFunc { - t.Helper() - return func(w http.ResponseWriter, r *http.Request) { - // Return an empty hosts list — the reservation controller treats this as NoHostsFound. - resp := &schedulerdelegationapi.ExternalSchedulerResponse{Hosts: []string{}} - if err := json.NewEncoder(w).Encode(resp); err != nil { - t.Errorf("scheduler encode: %v", err) - } - } +type e2eFakeProvider struct{ hosts []string } + +func (f e2eFakeProvider) GetPipeline(string) (lib.FilterWeigherPipeline[sched.PlacementRequest], bool) { + return e2eFakePipeline(f), true +} + +type e2eFakeGatherer struct{} + +func (e2eFakeGatherer) MutateWithAllCandidates(context.Context, *sched.PlacementRequest) error { + return nil } +func e2eScheduler(hosts []string) *sched.Placement { + return sched.NewPlacement(e2eFakeProvider{hosts: hosts}, e2eFakeGatherer{}) +} + +// e2eAcceptScheduler returns a scheduler that always places on host-1. +func e2eAcceptScheduler() *sched.Placement { return e2eScheduler([]string{"host-1"}) } + +// e2eRejectScheduler returns a scheduler that finds no host (NoHostsFound). +func e2eRejectScheduler() *sched.Placement { return e2eScheduler(nil) } + // assertNoDryRunProbes verifies that all probe CommittedResource CRDs and their child // Reservations have been cleaned up. It only checks objects associated with dry-run probes // (name prefix "commitment-dryrun-") rather than asserting zero total objects, so the @@ -334,7 +336,7 @@ var e2eFlavor = &TestFlavor{Name: "m1.small", Group: "hana_1", MemoryMB: 1024, V func TestE2EChangeCommitments(t *testing.T) { testCases := []struct { Name string - Scheduler func(*testing.T) http.HandlerFunc + Scheduler func() *sched.Placement ReqJSON string WantResp APIResponseExpectation WantAbsent []string @@ -522,7 +524,7 @@ func TestE2EChangeCommitments(t *testing.T) { for _, tc := range testCases { t.Run(tc.Name, func(t *testing.T) { - env := newE2EEnv(t, []*TestFlavor{e2eFlavor}, e2eInfoVersion, tc.Scheduler(t)) + env := newE2EEnv(t, []*TestFlavor{e2eFlavor}, e2eInfoVersion, tc.Scheduler()) defer env.close() te := env.asCRTestEnv() diff --git a/internal/scheduling/reservations/commitments/api/change_commitments_test.go b/internal/scheduling/reservations/commitments/api/change_commitments_test.go index a387c60f7..ab29cce5a 100644 --- a/internal/scheduling/reservations/commitments/api/change_commitments_test.go +++ b/internal/scheduling/reservations/commitments/api/change_commitments_test.go @@ -1292,7 +1292,7 @@ func (vm *TestVM) ToVM() VM { AvailabilityZone: vm.AZ, Resources: map[string]int64{ "memory": vm.Flavor.MemoryMB, - "vcpus": vm.Flavor.VCPUs, + "cpu": vm.Flavor.VCPUs, }, FlavorExtraSpecs: map[string]string{ "quota:hw_version": vm.Flavor.Group, diff --git a/internal/scheduling/reservations/commitments/api/info.go b/internal/scheduling/reservations/commitments/api/info.go index 7ab5bcac8..f840c9083 100644 --- a/internal/scheduling/reservations/commitments/api/info.go +++ b/internal/scheduling/reservations/commitments/api/info.go @@ -13,6 +13,7 @@ import ( "strings" "time" + knowledge "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" commitments "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/commitments" "github.com/go-logr/logr" @@ -35,7 +36,7 @@ func (api *HTTPAPI) HandleInfo(w http.ResponseWriter, r *http.Request) { // Set request ID in response header for client correlation w.Header().Set("X-Request-ID", requestID) - ctx := reservations.WithGlobalRequestID(r.Context(), "committed-resource-"+requestID) + ctx := reservations.WithGlobalRequestID(r.Context(), requestID) logger := commitments.LoggerFromContext(ctx).WithValues("endpoint", "/commitments/v1/info") // Only accept GET method @@ -105,8 +106,7 @@ func mibToGiB(v *uint64) *uint64 { // All flavor groups report usage; only those with fixed RAM/core ratio accept commitments. func (api *HTTPAPI) buildServiceInfo(ctx context.Context, logger logr.Logger) (liquid.ServiceInfo, error) { // Get all flavor groups from Knowledge CRDs - knowledge := &reservations.FlavorGroupKnowledgeClient{Client: api.client} - flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, api.client, nil) if err != nil { // Return -1 as version when knowledge is not ready return liquid.ServiceInfo{ @@ -219,7 +219,7 @@ func (api *HTTPAPI) buildServiceInfo(ctx context.Context, logger logr.Logger) (l // Get last content changed from flavor group knowledge and treat it as version var version int64 = -1 - if knowledgeCRD, err := knowledge.Get(ctx); err == nil && knowledgeCRD != nil && !knowledgeCRD.Status.LastContentChange.IsZero() { + if knowledgeCRD, err := knowledge.GetFlavorGroupsKnowledge(ctx, api.client); err == nil && knowledgeCRD != nil && !knowledgeCRD.Status.LastContentChange.IsZero() { version = knowledgeCRD.Status.LastContentChange.Unix() } diff --git a/internal/scheduling/reservations/commitments/api/quota.go b/internal/scheduling/reservations/commitments/api/quota.go index 59d759a9a..c4945d8bc 100644 --- a/internal/scheduling/reservations/commitments/api/quota.go +++ b/internal/scheduling/reservations/commitments/api/quota.go @@ -13,7 +13,7 @@ import ( "time" "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" + knowledge "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" commitments "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/commitments" "github.com/cobaltcore-dev/cortex/pkg/multicluster" "github.com/google/uuid" @@ -97,8 +97,7 @@ func (api *HTTPAPI) HandleQuota(w http.ResponseWriter, r *http.Request) { // Fetch flavor groups to determine per-resource RAM unit. // The ProjectQuota CRD stores RAM values in GiB; Limes sends in the declared unit // (slots for fixed-ratio groups, GiB for variable-ratio). Convert on receipt. - knowledge := &reservations.FlavorGroupKnowledgeClient{Client: api.client} - flavorGroups, err := knowledge.GetAllFlavorGroups(r.Context(), nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(r.Context(), api.client, nil) if err != nil { log.Info("flavor groups not available for quota unit conversion", "error", err.Error()) api.quotaError(w, http.StatusServiceUnavailable, "flavor groups not available: "+err.Error(), startTime) diff --git a/internal/scheduling/reservations/commitments/api/report_capacity.go b/internal/scheduling/reservations/commitments/api/report_capacity.go index dc25fb547..e468576ea 100644 --- a/internal/scheduling/reservations/commitments/api/report_capacity.go +++ b/internal/scheduling/reservations/commitments/api/report_capacity.go @@ -53,7 +53,7 @@ func (api *HTTPAPI) HandleReportCapacity(w http.ResponseWriter, r *http.Request) return } - ctx := reservations.WithGlobalRequestID(r.Context(), "committed-resource-"+requestID) + ctx := reservations.WithGlobalRequestID(r.Context(), requestID) logger := commitments.LoggerFromContext(ctx).WithValues("endpoint", "/commitments/v1/report-capacity") // Only accept POST method diff --git a/internal/scheduling/reservations/commitments/api/report_usage_test.go b/internal/scheduling/reservations/commitments/api/report_usage_test.go index e25dea790..5f8d79b67 100644 --- a/internal/scheduling/reservations/commitments/api/report_usage_test.go +++ b/internal/scheduling/reservations/commitments/api/report_usage_test.go @@ -626,8 +626,8 @@ func (m *mockVMSource) addVM(vm *TestVMUsage) { OSType: osType, ProjectID: vm.ProjectID, Resources: map[string]resource.Quantity{ - "memory": *resource.NewQuantity(vm.Flavor.MemoryMB*1024*1024, resource.BinarySI), - "vcpus": *resource.NewQuantity(vm.Flavor.VCPUs, resource.DecimalSI), + string(hv1.ResourceMemory): *resource.NewQuantity(vm.Flavor.MemoryMB*1024*1024, resource.BinarySI), + string(hv1.ResourceCPU): *resource.NewQuantity(vm.Flavor.VCPUs, resource.DecimalSI), }, } m.vms[vm.ProjectID] = append(m.vms[vm.ProjectID], v) @@ -1104,15 +1104,14 @@ func (tr *UsageTestReservation) toK8sReservation(number int) *v1alpha1.Reservati spec := v1alpha1.ReservationSpec{ Type: v1alpha1.ReservationTypeCommittedResource, Resources: map[hv1.ResourceName]resource.Quantity{ - "memory": resource.MustParse(strconv.FormatInt(memoryMB, 10) + "Mi"), - "cpu": resource.MustParse(strconv.FormatInt(tr.Flavor.VCPUs, 10)), + hv1.ResourceMemory: resource.MustParse(strconv.FormatInt(memoryMB, 10) + "Mi"), + hv1.ResourceCPU: resource.MustParse(strconv.FormatInt(tr.Flavor.VCPUs, 10)), }, CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ CommitmentUUID: tr.CommitmentID, ProjectID: tr.ProjectID, ResourceName: tr.Flavor.Name, ResourceGroup: tr.Flavor.Group, - Allocations: map[string]v1alpha1.CommittedResourceAllocation{}, }, } @@ -1134,7 +1133,7 @@ func (tr *UsageTestReservation) toK8sReservation(number int) *v1alpha1.Reservati }, }, CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - Allocations: map[string]string{}, + Allocation: map[string]v1alpha1.CommittedResourceAllocation{}, }, } diff --git a/internal/scheduling/reservations/commitments/api/usage_test.go b/internal/scheduling/reservations/commitments/api/usage_test.go index eca1f0f9a..fc97c3f34 100644 --- a/internal/scheduling/reservations/commitments/api/usage_test.go +++ b/internal/scheduling/reservations/commitments/api/usage_test.go @@ -52,8 +52,8 @@ func mkVM(id, az, flavorName string, ramMiB, vcpus int64, createdAt time.Time, p FlavorName: flavorName, ProjectID: projectID, Resources: map[string]resource.Quantity{ - "memory": *resource.NewQuantity(ramMiB*1024*1024, resource.BinarySI), - "vcpus": *resource.NewQuantity(vcpus, resource.DecimalSI), + string(hv1.ResourceMemory): *resource.NewQuantity(ramMiB*1024*1024, resource.BinarySI), + string(hv1.ResourceCPU): *resource.NewQuantity(vcpus, resource.DecimalSI), }, } } @@ -533,7 +533,7 @@ func makeUsageTestReservationWithTimes(commitmentUUID, projectID, flavorGroup, a Type: v1alpha1.ReservationTypeCommittedResource, AvailabilityZone: az, Resources: map[hv1.ResourceName]resource.Quantity{ - "memory": *resource.NewQuantity(memoryBytes, resource.BinarySI), + hv1.ResourceMemory: *resource.NewQuantity(memoryBytes, resource.BinarySI), }, CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ CommitmentUUID: commitmentUUID, diff --git a/internal/scheduling/reservations/commitments/capacity.go b/internal/scheduling/reservations/commitments/capacity.go index 78381d0f5..7b6c65162 100644 --- a/internal/scheduling/reservations/commitments/capacity.go +++ b/internal/scheduling/reservations/commitments/capacity.go @@ -15,7 +15,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" + knowledge "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" ) // ErrCapacityNotReady is returned by CalculateCapacity when one or more FlavorGroupCapacity @@ -64,14 +64,13 @@ func NewCapacityCalculator(client client.Client, conf APIConfig) *CapacityCalcul // - Usage: always None. Limes derives project usage from the separate Report-Usage endpoint, so // a usage value here is unused and would only invite misinterpretation of reserved capacity. func (c *CapacityCalculator) CalculateCapacity(ctx context.Context, req liquid.ServiceCapacityRequest) (liquid.ServiceCapacityReport, error) { - knowledge := &reservations.FlavorGroupKnowledgeClient{Client: c.client} - flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, c.client, nil) if err != nil { return liquid.ServiceCapacityReport{}, fmt.Errorf("failed to get flavor groups: %w", err) } var infoVersion int64 = -1 - if knowledgeCRD, err := knowledge.Get(ctx); err == nil && knowledgeCRD != nil && !knowledgeCRD.Status.LastContentChange.IsZero() { + if knowledgeCRD, err := knowledge.GetFlavorGroupsKnowledge(ctx, c.client); err == nil && knowledgeCRD != nil && !knowledgeCRD.Status.LastContentChange.IsZero() { infoVersion = knowledgeCRD.Status.LastContentChange.Unix() } diff --git a/internal/scheduling/reservations/commitments/changes/reconciler.go b/internal/scheduling/reservations/commitments/changes/reconciler.go index 73e244a99..ec2be9b82 100644 --- a/internal/scheduling/reservations/commitments/changes/reconciler.go +++ b/internal/scheduling/reservations/commitments/changes/reconciler.go @@ -19,6 +19,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/handler" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/commitments" "github.com/cobaltcore-dev/cortex/pkg/multicluster" ) @@ -37,8 +38,9 @@ type CommittedResourceChangeReconciler struct { } func (r *CommittedResourceChangeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { - ctx = commitments.WithNewGlobalRequestID(ctx) - logger := commitments.LoggerFromContext(ctx).WithValues("committedResourceChangeRequest", req.Name) + ctx = commitments.WithNewReconcileID(ctx) + ctx = reservations.WithRequestID(ctx, req.Name) + logger := commitments.LoggerFromContext(ctx) var tx v1alpha1.CommittedResourceChange if err := r.Get(ctx, req.NamespacedName, &tx); err != nil { diff --git a/internal/scheduling/reservations/commitments/committed_resource_controller.go b/internal/scheduling/reservations/commitments/committed_resource_controller.go index e5101f0d2..a4e55ad04 100644 --- a/internal/scheduling/reservations/commitments/committed_resource_controller.go +++ b/internal/scheduling/reservations/commitments/committed_resource_controller.go @@ -23,6 +23,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/handler" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + knowledge "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" "github.com/cobaltcore-dev/cortex/pkg/multicluster" ) @@ -52,10 +53,9 @@ func (r *CommittedResourceController) Reconcile(ctx context.Context, req ctrl.Re return ctrl.Result{}, client.IgnoreNotFound(err) } - ctx = WithNewGlobalRequestID(ctx) - logger := LoggerFromContext(ctx).WithValues( - "committedResource", req.Name, - ) + ctx = WithNewReconcileID(ctx) + ctx = reservations.WithRequestID(ctx, req.Name) + logger := LoggerFromContext(ctx) // Timing anchor: shows which generation the cache served when this reconcile fired. // Missing for a specific generation means the CR controller never dequeued that gen. @@ -141,6 +141,7 @@ func (r *CommittedResourceController) reconcilePending(ctx context.Context, logg if rollbackErr := r.deleteChildReservations(ctx, cr); rollbackErr != nil { return ctrl.Result{}, rollbackErr } + // TODO: if firstSlotReason == "NoHostsFound", consider retrying with a smaller slot size before rejecting. return ctrl.Result{}, r.setNotReady(ctx, cr, v1alpha1.CommittedResourceReasonRejected, failReason) } delay := r.retryDelay(cr) @@ -169,7 +170,6 @@ func (r *CommittedResourceController) reconcilePending(ctx context.Context, logg } return ctrl.Result{RequeueAfter: 3 * time.Second}, nil } - logger.Info("committed resource accepted", "generation", cr.Generation, "amount", cr.Spec.Amount.String()) initialGen := cr.Generation if err := r.setAccepted(ctx, cr); err != nil { return ctrl.Result{}, err @@ -304,7 +304,6 @@ func (r *CommittedResourceController) reconcileCommitted(ctx context.Context, lo } return ctrl.Result{RequeueAfter: 3 * time.Second}, nil } - logger.Info("committed resource accepted", "generation", cr.Generation, "amount", cr.Spec.Amount.String()) // setAccepted mutates cr in-place via the Patch response (server returns the latest object, // which may already carry a newer generation from a concurrent spec write). // Capture initialGen before the mutation so the comparison below is correct. @@ -442,8 +441,7 @@ func (r *CommittedResourceController) reconcileCoresHeadroom(ctx context.Context } func (r *CommittedResourceController) applyReservationState(ctx context.Context, logger logr.Logger, cr *v1alpha1.CommittedResource) (*ApplyResult, error) { - knowledge := &reservations.FlavorGroupKnowledgeClient{Client: r.Client} - flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, r.Client, nil) if err != nil { return nil, fmt.Errorf("flavor knowledge not ready: %w", err) } @@ -454,8 +452,6 @@ func (r *CommittedResourceController) applyReservationState(ctx context.Context, } state.NamePrefix = cr.Name + "-" state.CreatorRequestID = reservations.GlobalRequestIDFromContext(ctx) - state.ParentGeneration = cr.Generation - maxSlots := 0 if cr.Spec.AllowRejection { maxSlots = r.Conf.MaxSlotsPerCommitment @@ -507,37 +503,34 @@ func (r *CommittedResourceController) checkChildReservationStatus(ctx context.Co // other slots are still pending. allReady = true for _, res := range list.Items { - obs := int64(0) - if res.Status.CommittedResourceReservation != nil { - obs = res.Status.CommittedResourceReservation.ObservedParentGeneration + cond := meta.FindStatusCondition(res.Status.Conditions, v1alpha1.ReservationConditionReady) + condObsGen := int64(0) + if cond != nil { + condObsGen = cond.ObservedGeneration } ctrl.LoggerFrom(ctx).V(1).Info("checkChildReservationStatus slot", "slot", res.Name, "rv", res.ResourceVersion, - "crGen", cr.Generation, - "obsParentGen", obs, - "genMatch", obs == cr.Generation, + "resGen", res.Generation, + "condObsGen", condObsGen, + "genMatch", condObsGen == res.Generation, ) - if res.Status.CommittedResourceReservation == nil || - res.Status.CommittedResourceReservation.ObservedParentGeneration != cr.Generation { - // parentGen has been written to the Reservation spec (syncReservationMetadata - // already ran for this generation) but the Reservation controller hasn't echoed - // it to status yet. Poke the Reservation to generate a fresh watch event so the - // Reservation controller wakes up and fires the echo. + if cond == nil || cond.ObservedGeneration != res.Generation { + // Reservation controller hasn't processed the current generation yet. + // Poke the Reservation to generate a fresh watch event so the + // Reservation controller wakes up. // Use gen+ms as the annotation value so each nudge attempt is a distinct string: // a same-value patch is a no-op on the API server (no watch event generated), and // in a double-stale scenario the first nudge may itself be processed with a stale // object, so periodic retries (every 3s via the fallback requeue) are needed. // shouldNudge is false on the first cycle (CR condition not yet set for this - // generation) to give the normal echo path ~3s to complete before intervening. - if shouldNudge && - res.Spec.CommittedResourceReservation != nil && - res.Spec.CommittedResourceReservation.ParentGeneration == cr.Generation { + // generation) to give the normal path ~3s to complete before intervening. + if shouldNudge { nudgeVal := strconv.FormatInt(cr.Generation, 10) + "-" + strconv.FormatInt(time.Now().UnixMilli(), 10) - ctrl.LoggerFrom(ctx).V(1).Info("nudging stale reservation: parentGen written to spec but echo pending", + ctrl.LoggerFrom(ctx).V(1).Info("nudging stale reservation: condition ObservedGeneration behind reservation generation", "slot", res.Name, - "crGen", cr.Generation, - "obs", obs, + "resGen", res.Generation, + "condObsGen", condObsGen, ) old := res.DeepCopy() if res.Annotations == nil { @@ -545,14 +538,13 @@ func (r *CommittedResourceController) checkChildReservationStatus(ctx context.Co } res.Annotations["cortex.cloud/echo-nudge"] = nudgeVal if pErr := r.Patch(ctx, &res, client.MergeFrom(old)); client.IgnoreNotFound(pErr) != nil { - ctrl.LoggerFrom(ctx).V(1).Info("failed to nudge stale reservation echo", "slot", res.Name, "err", pErr) + ctrl.LoggerFrom(ctx).V(1).Info("failed to nudge stale reservation", "slot", res.Name, "err", pErr) } } allReady = false continue } - cond := meta.FindStatusCondition(res.Status.Conditions, v1alpha1.ReservationConditionReady) - if cond == nil || cond.Status == metav1.ConditionUnknown { + if cond.Status == metav1.ConditionUnknown { allReady = false continue } @@ -602,6 +594,7 @@ func (r *CommittedResourceController) setAccepted(ctx context.Context, cr *v1alp if err := r.Status().Patch(ctx, cr, client.MergeFrom(old)); err != nil { return client.IgnoreNotFound(err) } + LoggerFromContext(ctx).Info("committed resource accepted", "generation", cr.Generation, "amount", cr.Spec.Amount.String()) return nil } @@ -646,8 +639,7 @@ func (r *CommittedResourceController) rollbackToAccepted(ctx context.Context, lo if cr.Status.AcceptedSpec == nil { return r.deleteChildReservations(ctx, cr) } - knowledge := &reservations.FlavorGroupKnowledgeClient{Client: r.Client} - flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, r.Client, nil) if err != nil { // Can't compute the rollback target — fall back to full delete rather than leaving // a partial state that's inconsistent with the unknown accepted state. @@ -666,7 +658,6 @@ func (r *CommittedResourceController) rollbackToAccepted(ctx context.Context, lo } state.NamePrefix = cr.Name + "-" state.CreatorRequestID = reservations.GlobalRequestIDFromContext(ctx) - state.ParentGeneration = cr.Generation rollbackMgr := NewReservationManager(r.Client, ReservationManagerConfig{ EnablePaygPreAllocation: r.Conf.EnablePaygPreAllocation, VMSource: r.VMSource, @@ -855,7 +846,6 @@ func (r *CommittedResourceController) SetupWithManager(mgr ctrl.Manager, mcl *mu if readyCond != nil && readyCond.Reason == v1alpha1.CommittedResourceReasonReserving && readyCond.ObservedGeneration == cr.Generation && - res.Spec.CommittedResourceReservation.ParentGeneration == cr.Generation && time.Since(readyCond.LastTransitionTime.Time) > maxReservingAgeForSlowdown { LoggerFromContext(ctx).V(1).Info("Reserving state age exceeded threshold, watch re-enqueues suppressed — retrying via backoff timer only", "name", cr.Name, "reservingAge", time.Since(readyCond.LastTransitionTime.Time).Round(time.Second), "threshold", maxReservingAgeForSlowdown) diff --git a/internal/scheduling/reservations/commitments/committed_resource_controller_test.go b/internal/scheduling/reservations/commitments/committed_resource_controller_test.go index d5732f0d0..589f13b85 100644 --- a/internal/scheduling/reservations/commitments/committed_resource_controller_test.go +++ b/internal/scheduling/reservations/commitments/committed_resource_controller_test.go @@ -129,11 +129,11 @@ func newCRTestClient(scheme *runtime.Scheme, objects ...client.Object) client.Cl }). WithIndex(&v1alpha1.Reservation{}, idxReservationByAllocationVMUUID, func(obj client.Object) []string { res, ok := obj.(*v1alpha1.Reservation) - if !ok || res.Spec.CommittedResourceReservation == nil { + if !ok || res.Status.CommittedResourceReservation == nil { return nil } - uuids := make([]string, 0, len(res.Spec.CommittedResourceReservation.Allocations)) - for vmUUID := range res.Spec.CommittedResourceReservation.Allocations { + uuids := make([]string, 0, len(res.Status.CommittedResourceReservation.Allocation)) + for vmUUID := range res.Status.CommittedResourceReservation.Allocation { uuids = append(uuids, vmUUID) } return uuids @@ -208,8 +208,8 @@ func countChildReservations(t *testing.T, k8sClient client.Client, commitmentUUI } // setChildReservationsReady simulates the reservation controller by marking all child -// Reservations for the given commitmentUUID as Ready=True and echoing ParentGeneration -// into ObservedParentGeneration (matching what echoParentGeneration does in production). +// Reservations for the given commitmentUUID as Ready=True with ObservedGeneration set +// to the reservation's own Generation (matching the conditions[Ready].ObservedGeneration contract). func setChildReservationsReady(t *testing.T, k8sClient client.Client, commitmentUUID string) { t.Helper() var list v1alpha1.ReservationList @@ -229,11 +229,8 @@ func setChildReservationsReady(t *testing.T, k8sClient client.Client, commitment Status: metav1.ConditionTrue, Reason: "ReservationActive", LastTransitionTime: metav1.Now(), + ObservedGeneration: res.Generation, }} - if res.Status.CommittedResourceReservation == nil { - res.Status.CommittedResourceReservation = &v1alpha1.CommittedResourceReservationStatus{} - } - res.Status.CommittedResourceReservation.ObservedParentGeneration = res.Spec.CommittedResourceReservation.ParentGeneration if err := k8sClient.Status().Update(context.Background(), res); err != nil { t.Fatalf("set reservation Ready=True: %v", err) } @@ -557,11 +554,10 @@ func TestCommittedResourceController_Rollback(t *testing.T) { hv1.ResourceCPU: resource.MustParse("2"), }, CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - CommitmentUUID: "test-uuid-1234", - ProjectID: "test-project", - DomainID: "test-domain", - ResourceGroup: "test-group", - ParentGeneration: 1, // stale + CommitmentUUID: "test-uuid-1234", + ProjectID: "test-project", + DomainID: "test-domain", + ResourceGroup: "test-group", }, }, } @@ -577,8 +573,8 @@ func TestCommittedResourceController_Rollback(t *testing.T) { if err := k8sClient.Get(context.Background(), types.NamespacedName{Name: "test-cr-0"}, &res); err != nil { t.Fatalf("get reservation: %v", err) } - if got := res.Spec.CommittedResourceReservation.ParentGeneration; got != cr.Generation { - t.Errorf("ParentGeneration: want %d, got %d", cr.Generation, got) + if got := res.Spec.CommittedResourceReservation.CommitmentUUID; got != "test-uuid-1234" { + t.Errorf("CommitmentUUID: want %q, got %q", "test-uuid-1234", got) } } @@ -615,11 +611,10 @@ func TestCommittedResourceController_RollbackUsesAcceptedSpecAZ(t *testing.T) { hv1.ResourceCPU: resource.MustParse("2"), }, CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - CommitmentUUID: "test-uuid-1234", - ProjectID: "test-project", - DomainID: "test-domain", - ResourceGroup: "test-group", - ParentGeneration: 2, + CommitmentUUID: "test-uuid-1234", + ProjectID: "test-project", + DomainID: "test-domain", + ResourceGroup: "test-group", }, }, } @@ -1291,6 +1286,7 @@ func TestCheckChildReservationStatus_GenerationGuard(t *testing.T) { wantAllReady: false, wantAnyFailed: false, wantReadySlots: 0, + wantNudge: true, }, { name: "parentGen written to spec but echo not yet in status: nudge annotation set", @@ -1312,7 +1308,8 @@ func TestCheckChildReservationStatus_GenerationGuard(t *testing.T) { child := &v1alpha1.Reservation{ ObjectMeta: metav1.ObjectMeta{ - Name: "test-cr-0", + Name: "test-cr-0", + Generation: 2, Labels: map[string]string{ v1alpha1.LabelReservationType: v1alpha1.ReservationTypeLabelCommittedResource, }, @@ -1320,16 +1317,12 @@ func TestCheckChildReservationStatus_GenerationGuard(t *testing.T) { Spec: v1alpha1.ReservationSpec{ Type: v1alpha1.ReservationTypeCommittedResource, CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - CommitmentUUID: cr.Spec.CommitmentUUID, - ParentGeneration: cr.Generation, + CommitmentUUID: cr.Spec.CommitmentUUID, }, }, } k8sClient := newCRTestClient(scheme, child) - child.Status.CommittedResourceReservation = &v1alpha1.CommittedResourceReservationStatus{ - ObservedParentGeneration: tt.obsGen, - } if tt.condStatus != "" { child.Status.Conditions = []metav1.Condition{{ Type: v1alpha1.ReservationConditionReady, @@ -1337,6 +1330,7 @@ func TestCheckChildReservationStatus_GenerationGuard(t *testing.T) { Reason: "Test", Message: tt.condMessage, LastTransitionTime: metav1.Now(), + ObservedGeneration: tt.obsGen, }} } if err := k8sClient.Status().Update(context.Background(), child); err != nil { diff --git a/internal/scheduling/reservations/commitments/committed_resource_summary.go b/internal/scheduling/reservations/commitments/committed_resource_summary.go index 63fb18443..d426c8689 100644 --- a/internal/scheduling/reservations/commitments/committed_resource_summary.go +++ b/internal/scheduling/reservations/commitments/committed_resource_summary.go @@ -48,7 +48,10 @@ func computeStatusSummary(spec v1alpha1.CommittedResourceSpec, status v1alpha1.C // VM count — only meaningful once placement is accepted. if reason == v1alpha1.CommittedResourceReasonAccepted { - n := len(status.AssignedInstances) + n := 0 + if status.Billing != nil { + n = len(status.Billing.AssignedInstances) + } if n == 1 { parts = append(parts, "1 VM") } else { diff --git a/internal/scheduling/reservations/commitments/committed_resource_summary_test.go b/internal/scheduling/reservations/commitments/committed_resource_summary_test.go index 56dec3304..8ac585bd9 100644 --- a/internal/scheduling/reservations/commitments/committed_resource_summary_test.go +++ b/internal/scheduling/reservations/commitments/committed_resource_summary_test.go @@ -41,8 +41,9 @@ func makeStatusWithMessage(reason, message string, accepted *v1alpha1.CommittedR } func makeStatus(reason string, accepted *v1alpha1.CommittedResourceSpec, instances []string) v1alpha1.CommittedResourceStatus { - status := v1alpha1.CommittedResourceStatus{ - AssignedInstances: instances, + status := v1alpha1.CommittedResourceStatus{} + if len(instances) > 0 { + status.Billing = &v1alpha1.BillingStatus{AssignedInstances: instances} } if accepted != nil { status.AcceptedSpec = accepted.DeepCopy() diff --git a/internal/scheduling/reservations/commitments/config.go b/internal/scheduling/reservations/commitments/config.go index d82740b53..34e41ab9d 100644 --- a/internal/scheduling/reservations/commitments/config.go +++ b/internal/scheduling/reservations/commitments/config.go @@ -55,13 +55,6 @@ type ReservationControllerConfig struct { RequeueIntervalActive metav1.Duration `json:"requeueIntervalActive"` // RequeueIntervalRetry is the back-off interval when knowledge is unavailable. RequeueIntervalRetry metav1.Duration `json:"requeueIntervalRetry"` - // RequeueIntervalGracePeriod is how often to re-check while a VM allocation - // is still within AllocationGracePeriod. Shorter than RequeueIntervalActive. - RequeueIntervalGracePeriod metav1.Duration `json:"requeueIntervalGracePeriod"` - // AllocationGracePeriod is the time window after a VM is allocated to a - // reservation during which it's expected to appear on the target host. - // VMs not confirmed within this period are considered stale and removed. - AllocationGracePeriod metav1.Duration `json:"allocationGracePeriod"` // OversubscriptionGracePeriod is how long to wait after detecting host over-subscription // before evicting slots. Gives other controllers (e.g. failover) time to self-heal. OversubscriptionGracePeriod metav1.Duration `json:"oversubscriptionGracePeriod,omitempty"` @@ -79,8 +72,6 @@ type ReservationControllerConfig struct { // When false, violations are detected and exposed via metrics but no eviction is performed. // Has no effect when EnableOversubscriptionCheck is false. EnableOversubscriptionReservationEviction bool `json:"enableOversubscriptionReservationEviction,omitempty"` - // SchedulerURL is the endpoint of the nova external scheduler. - SchedulerURL string `json:"schedulerURL"` // PipelineDefault is the fallback pipeline when no FlavorGroupPipelines entry matches. PipelineDefault string `json:"pipelineDefault"` // FlavorGroupPipelines maps flavor group IDs to pipeline names; "*" acts as catch-all. diff --git a/internal/scheduling/reservations/commitments/context.go b/internal/scheduling/reservations/commitments/context.go index 311d5538a..913adbf8e 100644 --- a/internal/scheduling/reservations/commitments/context.go +++ b/internal/scheduling/reservations/commitments/context.go @@ -16,20 +16,14 @@ import ( // Use LoggerFromContext to get a logger with request tracking values. var baseLog = ctrl.Log.WithName("committed-resource").WithValues("module", "committed-resources") -// WithNewGlobalRequestID creates a new context with a committed-resource-prefixed global request ID. -func WithNewGlobalRequestID(ctx context.Context) context.Context { - return reservations.WithGlobalRequestID(ctx, "committed-resource-"+uuid.New().String()) -} - -// WithGlobalRequestID creates a new context with the specified global request ID. -// This is used to propagate existing request IDs (e.g., from the creator annotation). -func WithGlobalRequestID(ctx context.Context, greq string) context.Context { - return reservations.WithGlobalRequestID(ctx, greq) +// WithNewReconcileID creates a new context with a unique per-reconcile correlation ID. +// Use this at the entry point of each Reconcile call so every log line in that run +// carries the same greq, making a single reconcile easy to isolate in logs. +func WithNewReconcileID(ctx context.Context) context.Context { + return reservations.WithGlobalRequestID(ctx, "reconcile-"+uuid.New().String()) } // LoggerFromContext returns a logger with greq and req values from the context. -// This creates a child logger with the request tracking values pre-attached, -// so you don't need to repeat them in every log call. func LoggerFromContext(ctx context.Context) logr.Logger { return baseLog.WithValues( "greq", reservations.GlobalRequestIDFromContext(ctx), diff --git a/internal/scheduling/reservations/commitments/field_index.go b/internal/scheduling/reservations/commitments/field_index.go index 1a4ca1a3b..1ae9c3330 100644 --- a/internal/scheduling/reservations/commitments/field_index.go +++ b/internal/scheduling/reservations/commitments/field_index.go @@ -18,7 +18,7 @@ const idxCommittedResourceByUUID = "spec.commitmentUUID" const idxCommittedResourceByProjectID = "spec.projectID" const idxReservationByCommitmentUUID = "spec.committedResourceReservation.commitmentUUID" const idxProjectQuotaByProjectID = "spec.projectID" -const idxReservationByAllocationVMUUID = "spec.committedResourceReservation.allocations" +const idxReservationByAllocationVMUUID = "status.committedResourceReservation.allocation" // once guards ensure each field index is registered exactly once. // Both CommittedResourceController and UsageReconciler call indexCommittedResourceByUUID; @@ -131,8 +131,8 @@ func indexProjectQuotaByProjectID(ctx context.Context, mcl *multicluster.Client) return err } -// Spec.CommittedResourceReservation.Allocations. This allows the reservation controller -// to efficiently find all other Reservation CRDs carrying a specific VM UUID without +// Status.CommittedResourceReservation.Allocation. This allows the reservation controller +// to efficiently find all Reservation CRDs that have confirmed a specific VM UUID without // scanning every reservation in the cluster. func indexReservationByAllocationVMUUID(ctx context.Context, mcl *multicluster.Client) (err error) { onceIndexReservationByAllocationVMID.Do(func() { @@ -147,11 +147,11 @@ func indexReservationByAllocationVMUUID(ctx context.Context, mcl *multicluster.C log.Error(errors.New("unexpected type"), "expected Reservation", "object", obj) return nil } - if res.Spec.CommittedResourceReservation == nil { + if res.Status.CommittedResourceReservation == nil { return nil } - uuids := make([]string, 0, len(res.Spec.CommittedResourceReservation.Allocations)) - for vmUUID := range res.Spec.CommittedResourceReservation.Allocations { + uuids := make([]string, 0, len(res.Status.CommittedResourceReservation.Allocation)) + for vmUUID := range res.Status.CommittedResourceReservation.Allocation { uuids = append(uuids, vmUUID) } return uuids diff --git a/internal/scheduling/reservations/commitments/host_oversubscription_controller.go b/internal/scheduling/reservations/commitments/host_oversubscription_controller.go index 325640e07..b116465f7 100644 --- a/internal/scheduling/reservations/commitments/host_oversubscription_controller.go +++ b/internal/scheduling/reservations/commitments/host_oversubscription_controller.go @@ -22,6 +22,7 @@ import ( "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" + "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/accounting" "github.com/cobaltcore-dev/cortex/pkg/multicluster" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" ) @@ -267,7 +268,7 @@ func (r *HostOversubscriptionController) SetupWithManager(mgr ctrl.Manager, mcl // computeViolations returns the per-resource excess (positive quantity) for any resource where // the host is over-subscribed. Returns nil if capacity data is unavailable, empty map if no violation. func computeViolations(allReservations []v1alpha1.Reservation, hv hv1.Hypervisor) map[hv1.ResourceName]resource.Quantity { - free := reservations.HostFreeCapacity(allReservations, hv) + free := accounting.HostFreeCapacity(allReservations, hv) if free == nil { return nil } @@ -304,7 +305,7 @@ func selectEvictionTarget( if res.Spec.TargetHost == "" { continue // already evicted, pending status cleanup } - if len(res.Spec.CommittedResourceReservation.Allocations) == 0 { + if res.Status.CommittedResourceReservation == nil || len(res.Status.CommittedResourceReservation.Allocation) == 0 { unallocated = append(unallocated, res) } else { allocated = append(allocated, res) @@ -333,7 +334,7 @@ func selectEvictionTarget( } entries := make([]allocatedEntry, len(allocated)) for i, res := range allocated { - unusedCap := reservations.UnusedReservationCapacity(res, false) + unusedCap := accounting.UnusedReservationCapacity(res, false) buckets := make(map[hv1.ResourceName]int64) for rn, total := range res.Spec.Resources { if t := total.Value(); t != 0 { @@ -371,8 +372,8 @@ func selectEvictionTarget( return selectedRes } -// evictReservation clears Spec.TargetHost and Spec.Allocations, and returns the resources -// freed (full Spec.Resources — the slot is fully evicted). Status cleanup is left to the reconcile loop +// evictReservation clears Spec.TargetHost, and returns the resources +// freed (full Spec.Resources — the slot is fully evicted). Status cleanup is left to the reconcile loop. func evictReservation( ctx context.Context, c client.Client, @@ -381,13 +382,10 @@ func evictReservation( logger := LoggerFromContext(ctx).WithValues("component", "host-oversubscription", "reservation", res.Name) - freed := reservations.UnusedReservationCapacity(res, true) + freed := accounting.UnusedReservationCapacity(res, false) old := res.DeepCopy() res.Spec.TargetHost = "" - if res.Spec.CommittedResourceReservation != nil { - res.Spec.CommittedResourceReservation.Allocations = nil - } logger.Info("evicting reservation", "freed", freed, "previous host", old.Spec.TargetHost, diff --git a/internal/scheduling/reservations/commitments/host_oversubscription_controller_test.go b/internal/scheduling/reservations/commitments/host_oversubscription_controller_test.go index 2bceb0fe9..12bc8fc7b 100644 --- a/internal/scheduling/reservations/commitments/host_oversubscription_controller_test.go +++ b/internal/scheduling/reservations/commitments/host_oversubscription_controller_test.go @@ -8,6 +8,7 @@ import ( "testing" "time" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/controller-runtime/pkg/client" @@ -139,22 +140,21 @@ func TestSelectEvictionTarget(t *testing.T) { } } allocated := func(name string, totalGiB, usedGiB int64, vmID string) v1alpha1.Reservation { + _ = usedGiB return v1alpha1.Reservation{ ObjectMeta: metav1.ObjectMeta{Name: name}, Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: "host-1", - Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(totalGiB)}, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - vmID: {Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(usedGiB)}}, - }, - }, + Type: v1alpha1.ReservationTypeCommittedResource, + TargetHost: "host-1", + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(totalGiB)}, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{}, }, Status: v1alpha1.ReservationStatus{ Host: "host-1", CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - Allocations: map[string]string{vmID: "host-1"}, + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + vmID: {ObjectRef: corev1.ObjectReference{Name: vmID}, Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(totalGiB)}}, + }, }, }, } @@ -177,13 +177,13 @@ func TestSelectEvictionTarget(t *testing.T) { wantName: "small", }, { - name: "3 allocated slots: selects most-idle (validates sort ordering)", + name: "3 allocated slots: all have confirmed VMs — stable sort picks first (most-idle concept requires per-VM sizes)", slots: []v1alpha1.Reservation{ - allocated("mostly-used", 512, 480, "vm-1"), // ~6% unused - allocated("half-used", 512, 256, "vm-2"), // ~50% unused - allocated("idle", 512, 10, "vm-3"), // ~98% unused + allocated("mostly-used", 512, 480, "vm-1"), // status.allocs non-empty → UnusedReservationCapacity = 0 + allocated("half-used", 512, 256, "vm-2"), // same + allocated("idle", 512, 10, "vm-3"), // same — all equal, stable sort picks first }, - wantName: "idle", + wantName: "mostly-used", }, { name: "skips non-CR reservation types", @@ -440,19 +440,17 @@ func TestEvictReservation_ClearsAllocations(t *testing.T) { slot := &v1alpha1.Reservation{ ObjectMeta: metav1.ObjectMeta{Name: "slot-1"}, Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: "host-1", - Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(512)}, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - "vm-1": {Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(256)}}, - }, - }, + Type: v1alpha1.ReservationTypeCommittedResource, + TargetHost: "host-1", + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(512)}, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{}, }, Status: v1alpha1.ReservationStatus{ Host: "host-1", CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - Allocations: map[string]string{"vm-1": "host-1"}, + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + "vm-1": {ObjectRef: corev1.ObjectReference{Name: "vm-1"}, Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(512)}}, + }, }, }, } @@ -463,8 +461,10 @@ func TestEvictReservation_ClearsAllocations(t *testing.T) { t.Fatalf("unexpected error: %v", err) } freedMem := freed[hv1.ResourceMemory] - if freedMem.Cmp(testGiB(512)) != 0 { - t.Errorf("expected freed 512 GiB, got %s", freedMem.String()) + // A fully-confirmed slot has no unfilled capacity: confirmed VMs appear in hv.Status.Allocation, + // so UnusedReservationCapacity returns 0. Only an empty slot's full block is freed. + if !freedMem.IsZero() { + t.Errorf("expected freed 0 (VM still running on host), got %s", freedMem.String()) } var updated v1alpha1.Reservation @@ -474,7 +474,8 @@ func TestEvictReservation_ClearsAllocations(t *testing.T) { if updated.Spec.TargetHost != "" { t.Errorf("expected TargetHost cleared, got %q", updated.Spec.TargetHost) } - if len(updated.Spec.CommittedResourceReservation.Allocations) != 0 { - t.Errorf("expected Spec.Allocations cleared, got %v", updated.Spec.CommittedResourceReservation.Allocations) + // Eviction only clears TargetHost; status.allocation tracks confirmed VMs and is untouched. + if updated.Status.CommittedResourceReservation == nil || len(updated.Status.CommittedResourceReservation.Allocation) == 0 { + t.Errorf("expected status.allocation to remain after eviction") } } diff --git a/internal/scheduling/reservations/commitments/integration_test.go b/internal/scheduling/reservations/commitments/integration_test.go index 154d96137..9bc0ee94b 100644 --- a/internal/scheduling/reservations/commitments/integration_test.go +++ b/internal/scheduling/reservations/commitments/integration_test.go @@ -19,10 +19,7 @@ package commitments import ( "context" - "encoding/json" "fmt" - "net/http" - "net/http/httptest" "strings" "sync/atomic" "testing" @@ -30,8 +27,12 @@ import ( schedulerdelegationapi "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" + "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" + "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/accounting" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/meta" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -298,8 +299,8 @@ func TestCRIntegration(t *testing.T) { if res.Spec.CommittedResourceReservation == nil { t.Fatal("CommittedResourceReservation is nil") } - if _, ok := res.Spec.CommittedResourceReservation.Allocations["vm-payg-1"]; !ok { - t.Error("expected vm-payg-1 in Spec.Allocations") + if res.Annotations[v1alpha1.AnnotationCandidateAllocation] != "vm-payg-1" { + t.Errorf("expected annotation %s=vm-payg-1, got %q", v1alpha1.AnnotationCandidateAllocation, res.Annotations[v1alpha1.AnnotationCandidateAllocation]) } }, }, @@ -323,8 +324,8 @@ func TestCRIntegration(t *testing.T) { if slots[0].Spec.TargetHost == "" { t.Error("expected TargetHost set by scheduler (Phase 5 path)") } - if len(slots[0].Spec.CommittedResourceReservation.Allocations) != 0 { - t.Error("expected no pre-allocations on scheduler-placed slot") + if slots[0].Annotations[v1alpha1.AnnotationCandidateAllocation] != "" { + t.Error("expected no candidate VM annotation on scheduler-placed slot") } }, }, @@ -405,10 +406,30 @@ type intgEnv struct { crController *CommittedResourceController resController *CommitmentReservationController hvController *HostOversubscriptionController - schedulerSrv *httptest.Server } -func newIntgEnv(t *testing.T, initialObjects []client.Object, schedulerFn http.HandlerFunc, vmSource reservations.VMSource) *intgEnv { +// intgSchedulerFn models a scheduler run: given the request it returns the ordered +// hosts (an empty list models NoHostsFound). It replaces the former httptest mock +// Nova scheduler handler. +type intgSchedulerFn func(sched.PlacementRequest) sched.PlacementResult + +type intgPipeline struct{ fn intgSchedulerFn } + +func (p intgPipeline) Run(req sched.PlacementRequest) (sched.PlacementResult, error) { + return p.fn(req), nil +} + +type intgPipelineProvider struct{ fn intgSchedulerFn } + +func (p intgPipelineProvider) GetPipeline(string) (lib.FilterWeigherPipeline[sched.PlacementRequest], bool) { + return intgPipeline(p), true +} + +func newIntgScheduler(fn intgSchedulerFn) *sched.Placement { + return sched.NewPlacement(intgPipelineProvider{fn: fn}, fakeGatherer{}) +} + +func newIntgEnv(t *testing.T, initialObjects []client.Object, schedulerFn intgSchedulerFn, vmSource reservations.VMSource) *intgEnv { t.Helper() scheme := newCRTestScheme(t) @@ -436,11 +457,11 @@ func newIntgEnv(t *testing.T, initialObjects []client.Object, schedulerFn http.H }). WithIndex(&v1alpha1.Reservation{}, idxReservationByAllocationVMUUID, func(obj client.Object) []string { res, ok := obj.(*v1alpha1.Reservation) - if !ok || res.Spec.CommittedResourceReservation == nil { + if !ok || res.Status.CommittedResourceReservation == nil { return nil } - uuids := make([]string, 0, len(res.Spec.CommittedResourceReservation.Allocations)) - for vmUUID := range res.Spec.CommittedResourceReservation.Allocations { + uuids := make([]string, 0, len(res.Status.CommittedResourceReservation.Allocation)) + for vmUUID := range res.Status.CommittedResourceReservation.Allocation { uuids = append(uuids, vmUUID) } return uuids @@ -465,8 +486,6 @@ func newIntgEnv(t *testing.T, initialObjects []client.Object, schedulerFn http.H }). Build() - schedulerSrv := httptest.NewServer(schedulerFn) - crCtrl := &CommittedResourceController{ Client: k8sClient, Scheme: scheme, @@ -478,11 +497,11 @@ func newIntgEnv(t *testing.T, initialObjects []client.Object, schedulerFn http.H } monitor := NewReservationControllerMonitor() resCtrl := &CommitmentReservationController{ - Client: k8sClient, - Scheme: scheme, + Client: k8sClient, + Scheme: scheme, + Scheduler: newIntgScheduler(schedulerFn), + VMSource: vmSource, Conf: ReservationControllerConfig{ - SchedulerURL: schedulerSrv.URL, - AllocationGracePeriod: metav1.Duration{Duration: 15 * time.Minute}, RequeueIntervalActive: metav1.Duration{Duration: 5 * time.Minute}, EnableOversubscriptionCheck: true, EnableOversubscriptionReservationEviction: true, @@ -496,10 +515,10 @@ func newIntgEnv(t *testing.T, initialObjects []client.Object, schedulerFn http.H Monitor: &monitor, Conf: resCtrl.Conf, } - return &intgEnv{k8sClient: k8sClient, crController: crCtrl, resController: resCtrl, hvController: hvCtrl, schedulerSrv: schedulerSrv} + return &intgEnv{k8sClient: k8sClient, crController: crCtrl, resController: resCtrl, hvController: hvCtrl} } -func (e *intgEnv) close() { e.schedulerSrv.Close() } +func (e *intgEnv) close() {} func newDefaultIntgEnv(t *testing.T) *intgEnv { t.Helper() @@ -698,42 +717,38 @@ func intgAssertCRCondition(t *testing.T, k8sClient client.Client, crNames []stri } // ============================================================================ -// Scheduler handlers +// Scheduler functions // ============================================================================ -func intgAcceptScheduler(w http.ResponseWriter, r *http.Request) { - resp := &schedulerdelegationapi.ExternalSchedulerResponse{Hosts: []string{"host-1"}} - json.NewEncoder(w).Encode(resp) //nolint:errcheck +func intgAcceptScheduler(sched.PlacementRequest) sched.PlacementResult { + return sched.PlacementResult{OrderedHosts: []string{"host-1"}} } -func intgRejectScheduler(w http.ResponseWriter, r *http.Request) { - resp := &schedulerdelegationapi.ExternalSchedulerResponse{Hosts: []string{}} - json.NewEncoder(w).Encode(resp) //nolint:errcheck +func intgRejectScheduler(sched.PlacementRequest) sched.PlacementResult { + return sched.PlacementResult{OrderedHosts: []string{}} } -// intgAcceptFirstScheduler returns a handler that accepts the first count placement calls +// intgAcceptFirstScheduler returns a scheduler fn that accepts the first count placement calls // and rejects all subsequent ones. Uses an atomic counter so concurrent calls are safe. -func intgAcceptFirstScheduler(count int) http.HandlerFunc { +func intgAcceptFirstScheduler(count int) intgSchedulerFn { var calls atomic.Int32 - return func(w http.ResponseWriter, r *http.Request) { + return func(req sched.PlacementRequest) sched.PlacementResult { if int(calls.Add(1)) <= count { - intgAcceptScheduler(w, r) - } else { - intgRejectScheduler(w, r) + return intgAcceptScheduler(req) } + return intgRejectScheduler(req) } } -// intgRejectFirstScheduler returns a handler that rejects the first count placement calls +// intgRejectFirstScheduler returns a scheduler fn that rejects the first count placement calls // and accepts all subsequent ones. Used to test AllowRejection=false retry-until-success paths. -func intgRejectFirstScheduler(count int) http.HandlerFunc { +func intgRejectFirstScheduler(count int) intgSchedulerFn { var calls atomic.Int32 - return func(w http.ResponseWriter, r *http.Request) { + return func(req sched.PlacementRequest) sched.PlacementResult { if int(calls.Add(1)) <= count { - intgRejectScheduler(w, r) - } else { - intgAcceptScheduler(w, r) + return intgRejectScheduler(req) } + return intgAcceptScheduler(req) } } @@ -1237,17 +1252,13 @@ func TestCRScheduling_SetsReserveForCommittedResourceIntent(t *testing.T) { // CR slot scheduling has a real project ID and must run tenant-context filters // (filter_allowed_projects, filter_aggregate_metadata, etc.). Verify the intent is // reserve_for_committed_resource so filters can identify the call type correctly. - var capturedReq schedulerdelegationapi.ExternalSchedulerRequest + var capturedReq sched.PlacementRequest var schedulerCalled bool - captureScheduler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + captureScheduler := func(req sched.PlacementRequest) sched.PlacementResult { schedulerCalled = true - if err := json.NewDecoder(r.Body).Decode(&capturedReq); err != nil { - http.Error(w, err.Error(), http.StatusBadRequest) - return - } - resp := &schedulerdelegationapi.ExternalSchedulerResponse{Hosts: []string{"host-1"}} - json.NewEncoder(w).Encode(resp) //nolint:errcheck - }) + capturedReq = req + return sched.PlacementResult{OrderedHosts: []string{"host-1"}} + } env := newIntgEnv(t, []client.Object{newTestFlavorKnowledge(), intgHypervisor("host-1")}, captureScheduler, nil) defer env.close() @@ -1262,12 +1273,8 @@ func TestCRScheduling_SetsReserveForCommittedResourceIntent(t *testing.T) { if !schedulerCalled { t.Fatal("scheduler was never called — test did not exercise the scheduling path") } - hint, err := capturedReq.Spec.Data.GetSchedulerHintStr("_nova_check_type") - if err != nil { - t.Fatalf("failed to get _nova_check_type hint: %v", err) - } - if hint != string(schedulerdelegationapi.ReserveForCommittedResourceIntent) { - t.Errorf("CR slot scheduling must set _nova_check_type=%q, got %q", schedulerdelegationapi.ReserveForCommittedResourceIntent, hint) + if capturedReq.Intent != schedulerdelegationapi.ReserveForCommittedResourceIntent { + t.Errorf("CR slot scheduling must set Intent=%q, got %q", schedulerdelegationapi.ReserveForCommittedResourceIntent, capturedReq.Intent) } } @@ -1325,23 +1332,25 @@ func TestCROversubscriptionRemediation(t *testing.T) { if i < 10 { // slots 0-9: allocated — have a confirmed VM vmID := fmt.Sprintf("vm-%02d", i) - slot.Spec.CommittedResourceReservation.Allocations = map[string]v1alpha1.CommittedResourceAllocation{ - vmID: {Resources: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceMemory: gib(memGiB), - hv1.ResourceCPU: cpu(cores), - }}, - } slot.Status.CommittedResourceReservation = &v1alpha1.CommittedResourceReservationStatus{ - Allocations: map[string]string{vmID: "host-1"}, + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + vmID: { + ObjectRef: corev1.ObjectReference{Name: vmID}, + Resources: map[hv1.ResourceName]resource.Quantity{ + hv1.ResourceMemory: gib(memGiB), + hv1.ResourceCPU: cpu(cores), + }, + }, + }, } } objects = append(objects, slot) } var schedulerCalls atomic.Int32 - schedulerFn := func(w http.ResponseWriter, r *http.Request) { + schedulerFn := func(req sched.PlacementRequest) sched.PlacementResult { schedulerCalls.Add(1) - intgRejectScheduler(w, r) // reject so evicted slots don't land back on host-1 + return intgRejectScheduler(req) // reject so evicted slots don't land back on host-1 } env := newIntgEnv(t, append(objects, newTestFlavorKnowledge()), schedulerFn, nil) defer env.close() @@ -1379,7 +1388,7 @@ func TestCROversubscriptionRemediation(t *testing.T) { placedSlots = append(placedSlots, r) } } - free := reservations.HostFreeCapacity(placedSlots, *hv) + free := accounting.HostFreeCapacity(placedSlots, *hv) zero := resource.MustParse("0") for rn, f := range free { if f.Cmp(zero) < 0 { @@ -1389,9 +1398,9 @@ func TestCROversubscriptionRemediation(t *testing.T) { for _, r := range resList.Items { if r.Spec.TargetHost == "" && - r.Spec.CommittedResourceReservation != nil && - len(r.Spec.CommittedResourceReservation.Allocations) > 0 { - t.Errorf("slot %s has allocations but was evicted", r.Name) + r.Status.CommittedResourceReservation != nil && + len(r.Status.CommittedResourceReservation.Allocation) > 0 { + t.Errorf("slot %s has confirmed allocations but was evicted", r.Name) } } diff --git a/internal/scheduling/reservations/commitments/payg_candidates.go b/internal/scheduling/reservations/commitments/payg_candidates.go index 7894852b5..952fe3cab 100644 --- a/internal/scheduling/reservations/commitments/payg_candidates.go +++ b/internal/scheduling/reservations/commitments/payg_candidates.go @@ -24,7 +24,7 @@ type PAYGCandidate struct { // ScanAZForPaygCandidates returns unallocated PAYG VMs across all HVs in az, grouped by HV name. // Makes exactly two calls regardless of HV or VM count: -// 1. List all CR Reservations → build allocated VM UUID set from Spec.Allocations +// 1. List all CR Reservations → build allocated VM UUID set from Status.Allocation // 2. VMSource.ListVMsByProject → enriched VM data for the project // // HVs are identified by the label "topology.kubernetes.io/zone". @@ -52,8 +52,8 @@ func ScanAZForPaygCandidates( return nil, nil } - // One cache scan: build the set of VM UUIDs already claimed by any CR Reservation. - // Spec.Allocations is the scheduling perspective — exactly what we need here. + // One cache scan: build the set of VM UUIDs already confirmed in any CR Reservation. + // Status.Allocation holds verified VM placements — exactly what we need here. allocatedVMIDs, err := buildAllocatedVMSet(ctx, k8sClient, az) if err != nil { return nil, err @@ -82,8 +82,8 @@ func ScanAZForPaygCandidates( } // buildAllocatedVMSet lists CR Reservations in az and returns the set of VM UUIDs present -// in any Spec.Allocations. Filtering by AZ keeps the set small — only slots relevant -// to the current scan are included. One cache scan shared across all HV filters. +// in any Status.Allocation (confirmed placements). Filtering by AZ keeps the set small — +// only slots relevant to the current scan are included. One cache scan shared across all HV filters. func buildAllocatedVMSet(ctx context.Context, k8sClient client.Client, az string) (map[string]struct{}, error) { var resList v1alpha1.ReservationList if err := k8sClient.List(ctx, &resList, @@ -96,11 +96,11 @@ func buildAllocatedVMSet(ctx context.Context, k8sClient client.Client, az string if res.Spec.AvailabilityZone != az { continue } - if res.Spec.CommittedResourceReservation == nil { - continue - } - for vmUUID := range res.Spec.CommittedResourceReservation.Allocations { - allocated[vmUUID] = struct{}{} + // Include confirmed VMs from status. + if res.Status.CommittedResourceReservation != nil { + for vmUUID := range res.Status.CommittedResourceReservation.Allocation { + allocated[vmUUID] = struct{}{} + } } } return allocated, nil @@ -108,7 +108,7 @@ func buildAllocatedVMSet(ctx context.Context, k8sClient client.Client, az string // filterPaygCandidates returns unallocated PAYG VMs from a pre-fetched list for one HV. // vms must already be filtered to hvName by the caller. -// allocatedVMIDs is a pre-built set of UUIDs already claimed by any CR Reservation.Spec.Allocations. +// allocatedVMIDs is a pre-built set of UUIDs already confirmed in any CR Reservation.Status.Allocation. // Used by reuse sites (ticket #410, #372) where the caller already holds VM data and the allocated set. func filterPaygCandidates( hvName string, diff --git a/internal/scheduling/reservations/commitments/payg_candidates_test.go b/internal/scheduling/reservations/commitments/payg_candidates_test.go index a607d51cf..2ff6fde08 100644 --- a/internal/scheduling/reservations/commitments/payg_candidates_test.go +++ b/internal/scheduling/reservations/commitments/payg_candidates_test.go @@ -11,6 +11,7 @@ import ( "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -36,7 +37,12 @@ func (f *fakeVMSource) ListVMsByProject(_ context.Context, _ string) ([]reservat func (f *fakeVMSource) ListVMsOnHypervisors(_ context.Context, _ *hv1.HypervisorList, _ bool) ([]reservations.VM, error) { return f.vms, f.err } -func (f *fakeVMSource) GetVM(_ context.Context, _ string) (*reservations.VM, error) { +func (f *fakeVMSource) GetVM(_ context.Context, uuid string) (*reservations.VM, error) { + for i := range f.vms { + if f.vms[i].UUID == uuid { + return &f.vms[i], nil + } + } return nil, nil } func (f *fakeVMSource) IsServerActive(_ context.Context, _ string) (bool, error) { @@ -86,7 +92,7 @@ func vmOnHV(uuid, hvName, flavorName string, memMB uint64) reservations.VM { //n FlavorName: flavorName, CurrentHypervisor: hvName, Resources: map[string]resource.Quantity{ - "memory": *resource.NewQuantity(int64(memMB)*1024*1024, resource.BinarySI), //nolint:gosec + string(hv1.ResourceMemory): *resource.NewQuantity(int64(memMB)*1024*1024, resource.BinarySI), //nolint:gosec }, } } @@ -95,11 +101,13 @@ func reservationWithAlloc(name, vmUUID string) *v1alpha1.Reservation { return &v1alpha1.Reservation{ ObjectMeta: metav1.ObjectMeta{Name: name}, Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - CommitmentUUID: "other-cr", - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - vmUUID: {CreationTimestamp: metav1.Now()}, + Type: v1alpha1.ReservationTypeCommittedResource, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{CommitmentUUID: "other-cr"}, + }, + Status: v1alpha1.ReservationStatus{ + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + vmUUID: {ObjectRef: corev1.ObjectReference{Name: vmUUID}}, }, }, }, @@ -113,11 +121,11 @@ func buildPaygClient(t *testing.T, objs ...client.Object) client.Client { WithObjects(objs...). WithIndex(&v1alpha1.Reservation{}, idxReservationByAllocationVMUUID, func(obj client.Object) []string { res, ok := obj.(*v1alpha1.Reservation) - if !ok || res.Spec.CommittedResourceReservation == nil { + if !ok || res.Status.CommittedResourceReservation == nil { return nil } var uuids []string - for vmUUID := range res.Spec.CommittedResourceReservation.Allocations { + for vmUUID := range res.Status.CommittedResourceReservation.Allocation { uuids = append(uuids, vmUUID) } return uuids @@ -147,7 +155,7 @@ func TestFilterPaygCandidates(t *testing.T) { wantVMIDs: []string{"vm-a"}, }, { - name: "VM already in Reservation.Spec.Allocations — excluded", + name: "VM already in Reservation.Status.allocation — excluded", hv: hvWithInstances("host-1", "az-1", activeInstance("vm-a")), vms: []reservations.VM{vmOnHV("vm-a", "host-1", "small", 8192)}, extraObjs: []*v1alpha1.Reservation{reservationWithAlloc("res-1", "vm-a")}, @@ -200,8 +208,10 @@ func TestFilterPaygCandidates(t *testing.T) { t.Run(tc.name, func(t *testing.T) { allocatedVMIDs := make(map[string]struct{}) for _, res := range tc.extraObjs { - for vmUUID := range res.Spec.CommittedResourceReservation.Allocations { - allocatedVMIDs[vmUUID] = struct{}{} + if res.Status.CommittedResourceReservation != nil { + for vmUUID := range res.Status.CommittedResourceReservation.Allocation { + allocatedVMIDs[vmUUID] = struct{}{} + } } } diff --git a/internal/scheduling/reservations/commitments/payg_rollback_test.go b/internal/scheduling/reservations/commitments/payg_rollback_test.go index f1f7efeb0..5a5c39bf2 100644 --- a/internal/scheduling/reservations/commitments/payg_rollback_test.go +++ b/internal/scheduling/reservations/commitments/payg_rollback_test.go @@ -10,6 +10,7 @@ import ( "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" + hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "k8s.io/apimachinery/pkg/api/meta" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -34,6 +35,7 @@ func TestCRLifecycle_PAYGRollback(t *testing.T) { UUID: "vm-payg-rollback", FlavorName: "test-flavor", CurrentHypervisor: "host-1", + Resources: map[string]resource.Quantity{string(hv1.ResourceMemory): resource.MustParse("4Gi")}, }}}) defer env.close() @@ -56,8 +58,8 @@ func TestCRLifecycle_PAYGRollback(t *testing.T) { if slots[0].Spec.TargetHost != "host-1" { t.Errorf("phase 1: expected pre-allocated slot on host-1, got %q", slots[0].Spec.TargetHost) } - if _, ok := slots[0].Spec.CommittedResourceReservation.Allocations["vm-payg-rollback"]; !ok { - t.Error("phase 1: expected vm-payg-rollback in Spec.Allocations") + if slots[0].Annotations[v1alpha1.AnnotationCandidateAllocation] != "vm-payg-rollback" { + t.Errorf("phase 1: expected annotation %s=vm-payg-rollback, got %q", v1alpha1.AnnotationCandidateAllocation, slots[0].Annotations[v1alpha1.AnnotationCandidateAllocation]) } preAllocatedSlotName := slots[0].Name @@ -112,8 +114,8 @@ func TestCRLifecycle_PAYGRollback(t *testing.T) { if surviving.Spec.TargetHost != "host-1" { t.Errorf("rollback: expected TargetHost host-1, got %q", surviving.Spec.TargetHost) } - if _, ok := surviving.Spec.CommittedResourceReservation.Allocations["vm-payg-rollback"]; !ok { - t.Error("rollback: expected vm-payg-rollback in Spec.Allocations of surviving slot") + if surviving.Annotations[v1alpha1.AnnotationCandidateAllocation] != "vm-payg-rollback" { + t.Errorf("rollback: expected annotation %s=vm-payg-rollback, got %q", v1alpha1.AnnotationCandidateAllocation, surviving.Annotations[v1alpha1.AnnotationCandidateAllocation]) } intgAssertCRCondition(t, env.k8sClient, []string{cr.Name}, metav1.ConditionFalse, v1alpha1.CommittedResourceReasonRejected) } diff --git a/internal/scheduling/reservations/commitments/reservation_controller.go b/internal/scheduling/reservations/commitments/reservation_controller.go index 249d10e47..d09d594b5 100644 --- a/internal/scheduling/reservations/commitments/reservation_controller.go +++ b/internal/scheduling/reservations/commitments/reservation_controller.go @@ -8,9 +8,14 @@ import ( "fmt" "net/http" "reflect" + "slices" + "sort" "time" + corev1 "k8s.io/api/core/v1" + apiequality "k8s.io/apimachinery/pkg/api/equality" "k8s.io/apimachinery/pkg/api/meta" + "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" @@ -27,6 +32,8 @@ import ( schedulerdelegationapi "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/scheduling" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + knowledge "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" "github.com/cobaltcore-dev/cortex/pkg/keystone" "github.com/cobaltcore-dev/cortex/pkg/multicluster" @@ -44,25 +51,15 @@ type CommitmentReservationController struct { Scheme *runtime.Scheme // Configuration for the controller. Conf ReservationControllerConfig - // SchedulerClient for making scheduler API calls. - SchedulerClient *reservations.SchedulerClient + // Scheduler is the in-process scheduler used to place reservations. + Scheduler *sched.Placement // DomainResolver resolves OpenStack domain IDs to domain names so the // domain_name scheduler hint can be populated for filter_external_customer. // Nil when KeystoneSecretRef is not configured; hint is omitted in that case. DomainResolver DomainResolver -} - -// echoParentGeneration copies Spec.CommittedResourceReservation.ParentGeneration to -// Status.CommittedResourceReservation.ObservedParentGeneration so the CommittedResource -// controller can confirm this reservation was processed for the current CR generation. -func echoParentGeneration(res *v1alpha1.Reservation) { - if res.Spec.CommittedResourceReservation == nil { - return - } - if res.Status.CommittedResourceReservation == nil { - res.Status.CommittedResourceReservation = &v1alpha1.CommittedResourceReservationStatus{} - } - res.Status.CommittedResourceReservation.ObservedParentGeneration = res.Spec.CommittedResourceReservation.ParentGeneration + // VMSource is the source of truth for VM presence and actual VM resources. + // Abstracts over the Nova DB today; will be swapped for a VM CRD when available. + VMSource reservations.VMSource } // Reconcile is part of the main kubernetes reconciliation loop which aims to @@ -76,23 +73,9 @@ func (r *CommitmentReservationController) Reconcile(ctx context.Context, req ctr return ctrl.Result{}, client.IgnoreNotFound(err) } - ctx = WithNewGlobalRequestID(ctx) + ctx = WithNewReconcileID(ctx) ctx = reservations.WithRequestID(ctx, req.Name) - logger := LoggerFromContext(ctx).WithValues("reservation", req.Name) - - // Debug: timing anchor — log RV and gen/obs so we can correlate watch delivery with Phase 6. - if res.Spec.CommittedResourceReservation != nil { - obs := int64(0) - if res.Status.CommittedResourceReservation != nil { - obs = res.Status.CommittedResourceReservation.ObservedParentGeneration - } - logger.V(1).Info("reconcile entry", - "rv", res.ResourceVersion, - "specGen", res.Generation, - "parentGen", res.Spec.CommittedResourceReservation.ParentGeneration, - "obs", obs, - ) - } + logger := LoggerFromContext(ctx) // filter for CR reservations resourceName := "" @@ -103,12 +86,12 @@ func (r *CommitmentReservationController) Reconcile(ctx context.Context, req ctr logger.Info("reservation has no resource name, skipping") old := res.DeepCopy() meta.SetStatusCondition(&res.Status.Conditions, metav1.Condition{ - Type: v1alpha1.ReservationConditionReady, - Status: metav1.ConditionFalse, - Reason: "MissingResourceName", - Message: "reservation has no resource name", + Type: v1alpha1.ReservationConditionReady, + Status: metav1.ConditionFalse, + Reason: "MissingResourceName", + Message: "reservation has no resource name", + ObservedGeneration: res.Generation, }) - echoParentGeneration(&res) patch := client.MergeFrom(old) if err := r.Status().Patch(ctx, &res, patch); err != nil { // Ignore not-found errors during background deletion @@ -122,101 +105,78 @@ func (r *CommitmentReservationController) Reconcile(ctx context.Context, req ctr return ctrl.Result{}, nil // Don't need to requeue. } - if res.IsReady() { - // Spec.TargetHost was cleared (e.g. oversubscription eviction) — revoke status so the - // slot re-enters the placement flow on the next reconcile. - if res.Spec.TargetHost == "" { - old := res.DeepCopy() - res.Status.Host = "" - if res.Status.CommittedResourceReservation != nil { - res.Status.CommittedResourceReservation.Allocations = nil - } - meta.SetStatusCondition(&res.Status.Conditions, metav1.Condition{ - Type: v1alpha1.ReservationConditionReady, - Status: metav1.ConditionFalse, - Reason: "PlacementRevoked", - Message: "target host was cleared; pending re-placement", - }) - if err := r.Status().Patch(ctx, &res, client.MergeFrom(old)); client.IgnoreNotFound(err) != nil { - return ctrl.Result{}, err - } - logger.Info("revoked ready status after placement eviction, slot re-enters placement flow", - "component", "oversubscription-check") - return ctrl.Result{}, nil + // Spec.TargetHost was cleared (e.g. oversubscription eviction) while the slot was confirmed — + // revoke status so the slot re-enters the placement flow on the next reconcile. + if res.Status.Host != "" && res.Spec.TargetHost == "" { + old := res.DeepCopy() + res.Status.Host = "" + if res.Status.CommittedResourceReservation != nil { + res.Status.CommittedResourceReservation.Allocation = nil } + meta.SetStatusCondition(&res.Status.Conditions, metav1.Condition{ + Type: v1alpha1.ReservationConditionReady, + Status: metav1.ConditionFalse, + Reason: "PlacementRevoked", + Message: "target host was cleared; pending re-placement", + ObservedGeneration: res.Generation, + }) + if err := r.Status().Patch(ctx, &res, client.MergeFrom(old)); client.IgnoreNotFound(err) != nil { + return ctrl.Result{}, err + } + logger.Info("revoked ready status after placement eviction, slot re-enters placement flow", + "component", "oversubscription-check") + return ctrl.Result{}, nil + } - logger.V(1).Info("reservation is active, verifying allocations") - - // Sync ObservedParentGeneration if the CR controller bumped ParentGeneration since - // the last time this reservation was processed (e.g. after a spec update). Without - // this patch the CR controller would spin in Reserving forever for already-ready slots. + // Slot is placed and confirmed — verify allocations and keep ObservedGeneration current. + if res.Spec.TargetHost != "" && res.Status.Host == res.Spec.TargetHost { + // Deprecated: remove when Spec.CommittedResourceReservation.Allocations is removed from the schema. + // One-time upgrade migration: blind-scheduler slots populated Spec.Allocations on the old schema. + // Seed AnnotationCandidateAllocation from the lexically-first Spec entry so the existing + // annotation recovery path re-confirms the VM on the next pass. if res.Spec.CommittedResourceReservation != nil && - (res.Status.CommittedResourceReservation == nil || - res.Status.CommittedResourceReservation.ObservedParentGeneration != res.Spec.CommittedResourceReservation.ParentGeneration) { - obs := int64(0) - if res.Status.CommittedResourceReservation != nil { - obs = res.Status.CommittedResourceReservation.ObservedParentGeneration + res.Annotations[v1alpha1.AnnotationCandidateAllocation] == "" && + len(res.Spec.CommittedResourceReservation.Allocations) > 0 { //nolint:staticcheck + keys := make([]string, 0, len(res.Spec.CommittedResourceReservation.Allocations)) //nolint:staticcheck + for k := range res.Spec.CommittedResourceReservation.Allocations { //nolint:staticcheck + keys = append(keys, k) } - logger.V(1).Info("echoParentGeneration firing", - "parentGen", res.Spec.CommittedResourceReservation.ParentGeneration, - "obs", obs, - ) + sort.Strings(keys) + vmUUID := keys[0] + logger.Info("migrating first Spec allocation to annotation (one-time upgrade path)", + "vm", vmUUID, "reservation", res.Name) old := res.DeepCopy() - echoParentGeneration(&res) - if err := r.Status().Patch(ctx, &res, client.MergeFrom(old)); client.IgnoreNotFound(err) != nil { + if res.Annotations == nil { + res.Annotations = make(map[string]string) + } + res.Annotations[v1alpha1.AnnotationCandidateAllocation] = vmUUID + if err := r.Patch(ctx, &res, client.MergeFrom(old)); client.IgnoreNotFound(err) != nil { return ctrl.Result{}, err } } - // Verify all allocations in Spec against actual VM state - result, err := r.reconcileAllocations(ctx, &res) - if err != nil { + logger.V(1).Info("reservation is active, verifying allocations") + if err := r.reconcileAllocations(ctx, &res); err != nil { logger.Error(err, "failed to reconcile allocations") return ctrl.Result{}, err } - - // Requeue with appropriate interval based on allocation state - // Use shorter interval if there are allocations in grace period for faster verification - if result.HasAllocationsInGracePeriod { - return ctrl.Result{RequeueAfter: r.Conf.RequeueIntervalGracePeriod.Duration}, nil - } - return ctrl.Result{RequeueAfter: r.Conf.RequeueIntervalActive.Duration}, nil - } - - // TODO trigger re-placement of unused reservations over time - - // Check if this is a pre-allocated reservation with allocations - if res.Spec.CommittedResourceReservation != nil && - len(res.Spec.CommittedResourceReservation.Allocations) > 0 && - res.Spec.TargetHost != "" { - // mark as ready without calling the placement API - logger.V(1).Info("detected pre-allocated reservation", - "targetHost", res.Spec.TargetHost, - "allocatedVMs", len(res.Spec.CommittedResourceReservation.Allocations)) - - old := res.DeepCopy() - res.Status.Host = res.Spec.TargetHost - meta.SetStatusCondition(&res.Status.Conditions, metav1.Condition{ - Type: v1alpha1.ReservationConditionReady, - Status: metav1.ConditionTrue, - Reason: "PreAllocated", - Message: "reservation pre-allocated with VM allocations", - }) - echoParentGeneration(&res) - patch := client.MergeFrom(old) - if err := r.Status().Patch(ctx, &res, patch); err != nil { - // Ignore not-found errors during background deletion - if client.IgnoreNotFound(err) != nil { - logger.Error(err, "failed to patch pre-allocated reservation status") + // Refresh ObservedGeneration if the spec changed since the last reconcile + // (e.g. endTime extension or metadata update via syncReservationMetadata). + cond := meta.FindStatusCondition(res.Status.Conditions, v1alpha1.ReservationConditionReady) + if cond == nil || cond.ObservedGeneration < res.Generation { + old := res.DeepCopy() + meta.SetStatusCondition(&res.Status.Conditions, metav1.Condition{ + Type: v1alpha1.ReservationConditionReady, + Status: metav1.ConditionTrue, + Reason: "ReservationActive", + Message: "reservation is successfully scheduled", + ObservedGeneration: res.Generation, + }) + if err := r.Status().Patch(ctx, &res, client.MergeFrom(old)); client.IgnoreNotFound(err) != nil { return ctrl.Result{}, err } - // Object was deleted, no need to continue - return ctrl.Result{}, nil } - - logger.Info("marked pre-allocated reservation as ready", "host", res.Status.Host) - // The status patch triggers a re-reconcile via the Reservation watch; allocation verification runs in that cycle. - return ctrl.Result{}, nil + return ctrl.Result{RequeueAfter: r.Conf.RequeueIntervalActive.Duration}, nil } // Sync Spec values to Status fields for non-pre-allocated reservations @@ -227,12 +187,12 @@ func (r *CommitmentReservationController) Reconcile(ctx context.Context, req ctr old := res.DeepCopy() res.Status.Host = res.Spec.TargetHost meta.SetStatusCondition(&res.Status.Conditions, metav1.Condition{ - Type: v1alpha1.ReservationConditionReady, - Status: metav1.ConditionTrue, - Reason: "ReservationActive", - Message: "reservation is successfully scheduled", + Type: v1alpha1.ReservationConditionReady, + Status: metav1.ConditionTrue, + Reason: "ReservationActive", + Message: "reservation is successfully scheduled", + ObservedGeneration: res.Generation, }) - echoParentGeneration(&res) patch := client.MergeFrom(old) if err := r.Status().Patch(ctx, &res, patch); err != nil { // Ignore not-found errors during background deletion @@ -260,28 +220,25 @@ func (r *CommitmentReservationController) Reconcile(ctx context.Context, req ctr availabilityZone = res.Spec.AvailabilityZone } - // Resolve domain name for the domain_name scheduler hint consumed by - // filter_external_customer to enforce host restrictions for external customers. - // Resolution is best-effort: if the DomainResolver is not configured (no - // KeystoneSecretRef) or the lookup fails, we log and proceed without the hint. - // filter_external_customer already handles a missing hint by skipping the filter, - // so omitting it degrades gracefully rather than blocking scheduling. - schedulerHints := map[string]any{ - "_nova_check_type": string(schedulerdelegationapi.ReserveForCommittedResourceIntent), - } + // Resolve domain name so filter_external_customer can enforce host + // restrictions for external customers. Resolution is best-effort: if the + // DomainResolver is not configured (no KeystoneSecretRef) or the lookup + // fails, we log and proceed without it. filter_external_customer handles a + // missing domain name by skipping the filter, so omitting it degrades + // gracefully rather than blocking scheduling. + domainName := "" if r.DomainResolver != nil && res.Spec.CommittedResourceReservation != nil && res.Spec.CommittedResourceReservation.DomainID != "" { - domainName, err := r.DomainResolver.ResolveDomainName(ctx, res.Spec.CommittedResourceReservation.DomainID) + resolved, err := r.DomainResolver.ResolveDomainName(ctx, res.Spec.CommittedResourceReservation.DomainID) if err != nil { logger.Error(err, "failed to resolve domain name for scheduler hint, proceeding without it", "domainID", res.Spec.CommittedResourceReservation.DomainID) } else { - schedulerHints["domain_name"] = domainName + domainName = resolved } } // Get flavor details from flavor group knowledge CRD - knowledge := &reservations.FlavorGroupKnowledgeClient{Client: r.Client} - flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, r.Client, nil) if err != nil { logger.V(1).Info("flavor knowledge not ready, requeueing", "resourceName", resourceName, @@ -290,7 +247,7 @@ func (r *CommitmentReservationController) Reconcile(ctx context.Context, req ctr } // Search for the flavor across all flavor groups - flavorGroupName, flavorDetails, err := reservations.FindFlavorInGroups(resourceName, flavorGroups) + flavorGroupName, flavorDetails, err := knowledge.FindFlavorInGroups(resourceName, flavorGroups) if err != nil { logger.Error(err, "flavor not found in any flavor group", "resourceName", resourceName) @@ -316,12 +273,12 @@ func (r *CommitmentReservationController) Reconcile(ctx context.Context, req ctr logger.Info("no hypervisors available for scheduling") old := res.DeepCopy() meta.SetStatusCondition(&res.Status.Conditions, metav1.Condition{ - Type: v1alpha1.ReservationConditionReady, - Status: metav1.ConditionFalse, - Reason: "NoHostsAvailable", - Message: "no hypervisors available for scheduling", + Type: v1alpha1.ReservationConditionReady, + Status: metav1.ConditionFalse, + Reason: "NoHostsAvailable", + Message: "no hypervisors available for scheduling", + ObservedGeneration: res.Generation, }) - echoParentGeneration(&res) patch := client.MergeFrom(old) if err := r.Status().Patch(ctx, &res, patch); err != nil { return ctrl.Result{}, client.IgnoreNotFound(err) @@ -343,45 +300,65 @@ func (r *CommitmentReservationController) Reconcile(ctx context.Context, req ctr return "initial" }()) - // Use the SchedulerClient to schedule the reservation - scheduleReq := reservations.ScheduleReservationRequest{ - InstanceUUID: res.Name, - ProjectID: projectID, - FlavorName: flavorDetails.Name, - FlavorExtraSpecs: flavorDetails.ExtraSpecs, - MemoryMB: flavorDetails.MemoryMB, - VCPUs: flavorDetails.VCPUs, - EligibleHosts: eligibleHosts, - Pipeline: pipelineName, - AvailabilityZone: availabilityZone, - SchedulerHints: schedulerHints, - } - scheduleOpts := scheduling.Options{ - ReadOnly: false, // mutates state (reservation placement) - LockReservations: true, // don't unlock CR reservations; finding a slot, not placing a VM - AssumeEmptyHosts: false, - IgnoredReservationTypes: nil, - MaxCandidates: 1, - SkipInflight: false, // TODO pessimistic blocking needed, will be addressed in follow up ticket - SkipCommittedResourceTracking: true, // CR slot scheduling, not a VM placement - } - - scheduleResp, err := r.SchedulerClient.ScheduleReservation(ctx, scheduleReq, scheduleOpts) + // Build the typed scheduler request directly (in-process; no HTTP). KVM + // reservations gather all candidates, so the eligible-host list is a starting + // set; GatherAllCandidates mirrors the previous ignore-preselection behavior. + hosts := make([]sched.Host, 0, len(eligibleHosts)) + weights := make(map[string]float64, len(eligibleHosts)) + for _, h := range eligibleHosts { + hosts = append(hosts, sched.Host{ComputeHost: h.ComputeHost, HypervisorHostname: h.HypervisorHostname}) + weights[h.ComputeHost] = 0.0 + } + // Honor the flavor's trait and hypervisor-type constraints so a + // committed-resource reservation lands on a compatible host (the typed + // request is built directly, so filter_has_requested_traits/filter_capabilities + // would otherwise see empty slices and become no-ops). No hypervisor-type + // default here — only match when the flavor names one. + requiredTraits, forbiddenTraits := schedulerdelegationapi.SplitTraits(flavorDetails.ExtraSpecs) + schedReq := sched.PlacementRequest{ + Pipeline: pipelineName, + GatherAllCandidates: true, + HypervisorFamily: schedulerdelegationapi.HypervisorFamilyKVM, + Hosts: hosts, + Weights: weights, + Intent: schedulerdelegationapi.ReserveForCommittedResourceIntent, + InstanceUUID: res.Name, + NumInstances: 1, // one reservation slot; NumInstances must be >= 1 or capacity/bin-pack math treats the host as free + ProjectID: projectID, + AvailabilityZone: availabilityZone, + DomainName: domainName, + RequiredTraits: requiredTraits, + ForbiddenTraits: forbiddenTraits, + RequestedCapabilities: schedulerdelegationapi.HypervisorTypeCapability(flavorDetails.ExtraSpecs, false), + AcceleratorsRequested: schedulerdelegationapi.RequestsAccelerators(flavorDetails.ExtraSpecs), + Flavor: schedulerdelegationapi.FlavorSpec{ + VCPUs: flavorDetails.VCPUs, + MemoryMB: flavorDetails.MemoryMB, + }, + Options: scheduling.Options{ + ReadOnly: false, // mutates state (reservation placement) + LockReservations: true, // don't unlock CR reservations; finding a slot, not placing a VM + MaxCandidates: 1, + SkipCommittedResourceTracking: true, // CR slot scheduling, not a VM placement + }, + } + + out, err := r.Scheduler.Run(ctx, schedReq, nil, nil) if err != nil { logger.Error(err, "failed to schedule reservation") return ctrl.Result{}, err } - if len(scheduleResp.Hosts) == 0 { + if len(out.OrderedHosts) == 0 { logger.Info("no hosts found for reservation, will retry", "reservation", res.Name, "flavorName", resourceName) old := res.DeepCopy() meta.SetStatusCondition(&res.Status.Conditions, metav1.Condition{ - Type: v1alpha1.ReservationConditionReady, - Status: metav1.ConditionFalse, - Reason: "NoHostsFound", - Message: "no hosts found for reservation", + Type: v1alpha1.ReservationConditionReady, + Status: metav1.ConditionFalse, + Reason: "NoHostsFound", + Message: "no hosts found for reservation", + ObservedGeneration: res.Generation, }) - echoParentGeneration(&res) patch := client.MergeFrom(old) if err := r.Status().Patch(ctx, &res, patch); err != nil { // Ignore not-found errors during background deletion @@ -398,7 +375,7 @@ func (r *CommitmentReservationController) Reconcile(ctx context.Context, req ctr // Update the reservation Spec with the found host (idx 0) // Only update Spec here - the Status will be synced in the next reconcile cycle // This avoids race conditions from doing two patches in one reconcile - host := scheduleResp.Hosts[0] + host := out.OrderedHosts[0] logger.Info("found host for reservation", "host", host, "flavorName", resourceName) old := res.DeepCopy() @@ -418,342 +395,263 @@ func (r *CommitmentReservationController) Reconcile(ctx context.Context, req ctr return ctrl.Result{}, nil } -// reconcileAllocationsResult holds the outcome of allocation reconciliation. -type reconcileAllocationsResult struct { - // HasAllocationsInGracePeriod is true if any allocations are still in grace period. - HasAllocationsInGracePeriod bool -} - -// reconcileAllocations verifies all allocations in Spec against actual VM state using the -// Hypervisor CRD as the sole source of truth. -// -// New allocations within the grace period are skipped — the VM may not yet appear in the -// HV CRD while it is still spawning. Older allocations are verified; VMs no longer present -// on their expected host are handled as follows: -// -// Live migration: when a confirmed VM is found on a different host, the reservation follows -// it only when the reservation has exactly one allocated VM and the new host has capacity. -// In all other cases (multiple VMs, or new host at capacity), the migrated VM is removed -// from the reservation so the slot remains available for re-use on the original host. -// Moving TargetHost when other VMs are present would cause those remaining VMs to appear -// misplaced on the next reconcile cycle. -func (r *CommitmentReservationController) reconcileAllocations(ctx context.Context, res *v1alpha1.Reservation) (*reconcileAllocationsResult, error) { +// reconcileAllocations has two responsibilities: +// 1. Verify existing confirmed allocations via VMSource: VMs still on the expected +// host have their partial allocation recomputed; VMs that have disappeared are removed. +// 2. Promote a PAYG candidate VM (AnnotationCandidateAllocation) into +// status.allocation when it is confirmed active on the expected host. +func (r *CommitmentReservationController) reconcileAllocations(ctx context.Context, res *v1alpha1.Reservation) error { logger := LoggerFromContext(ctx) - result := &reconcileAllocationsResult{} - now := time.Now() - // Skip if no CommittedResourceReservation if res.Spec.CommittedResourceReservation == nil { - return result, nil + return nil } - // Skip if no allocations to verify - if len(res.Spec.CommittedResourceReservation.Allocations) == 0 { - logger.V(1).Info("no allocations to verify", "reservation", res.Name) - return result, nil + primaryResource := res.Spec.CommittedResourceReservation.PrimaryResource + if primaryResource == "" { + primaryResource = hv1.ResourceMemory } - expectedHost := res.Status.Host - - // Fetch the Hypervisor CRD for the expected host. - var hypervisor hv1.Hypervisor - hvInstanceSet := make(map[string]bool) - if expectedHost != "" { - if err := r.Get(ctx, client.ObjectKey{Name: expectedHost}, &hypervisor); err != nil { - if client.IgnoreNotFound(err) != nil { - return nil, fmt.Errorf("failed to get hypervisor %s: %w", expectedHost, err) - } - // Hypervisor not found — treat all post-grace-period VMs as stale. - logger.V(1).Info("hypervisor CRD not found", "host", expectedHost) - } else { - // Build set of all VM UUIDs on this hypervisor for O(1) lookup. - // Include both active and inactive VMs — stopped/shelved VMs still hold the slot. - for _, inst := range hypervisor.Status.Instances { - hvInstanceSet[inst.ID] = true - } - logger.V(1).Info("fetched hypervisor instances", "host", expectedHost, "instanceCount", len(hvInstanceSet)) - } - } - - // Initialize status if res.Status.CommittedResourceReservation == nil { res.Status.CommittedResourceReservation = &v1alpha1.CommittedResourceReservationStatus{} } - // Snapshot existing Status.Allocations before we overwrite it so we can detect - // which VM UUIDs are newly confirmed after the patch. - existingStatusAllocations := make(map[string]string, len(res.Status.CommittedResourceReservation.Allocations)) - for k, v := range res.Status.CommittedResourceReservation.Allocations { - existingStatusAllocations[k] = v - } + expectedHost := res.Status.Host - // allHVs and allReservations are fetched lazily — only needed when a confirmed VM is - // missing from its expected host and we need to scan for a live migration. - var allHVs *hv1.HypervisorList - var allReservations *v1alpha1.ReservationList + newStatusAllocations := make(map[string]v1alpha1.CommittedResourceAllocation) - ensureHVsAndReservations := func() error { - if allHVs != nil && allReservations != nil { - return nil - } - hvs := &hv1.HypervisorList{} - if err := r.List(ctx, hvs); err != nil { - return fmt.Errorf("failed to list hypervisors: %w", err) - } - res := &v1alpha1.ReservationList{} - if err := r.List(ctx, res); err != nil { - return fmt.Errorf("failed to list reservations: %w", err) - } - allHVs = hvs - allReservations = res - return nil + // Pass 1: remove VMs no longer on the expected host. + // Sort lexically for deterministic processing order for the greedy resource assignment logic. + vmUUIDs := make([]string, 0, len(res.Status.CommittedResourceReservation.Allocation)) + for vmUUID := range res.Status.CommittedResourceReservation.Allocation { + vmUUIDs = append(vmUUIDs, vmUUID) } + // add VM id from annotation if present + if candidateVMUUID := res.Annotations[v1alpha1.AnnotationCandidateAllocation]; candidateVMUUID != "" && !slices.Contains(vmUUIDs, candidateVMUUID) { + logger.V(1).Info("adding candidate VM from annotation", + "vm", candidateVMUUID, + "reservation", res.Name) + vmUUIDs = append(vmUUIDs, candidateVMUUID) + } + sort.Strings(vmUUIDs) - // Build new Status.Allocations map based on HV CRD state. - newStatusAllocations := make(map[string]string) - // Track allocations to remove from Spec (stale/leaving VMs). - var allocationsToRemove []string - - // migrationTargetHost is set when the reservation has exactly one VM, that VM - // live-migrated to a new host, and the new host has capacity. - migrationTargetHost := "" - - for vmUUID, allocation := range res.Spec.CommittedResourceReservation.Allocations { - allocationAge := now.Sub(allocation.CreationTimestamp.Time) - isInGracePeriod := allocationAge < r.Conf.AllocationGracePeriod.Duration - - _, isConfirmed := existingStatusAllocations[vmUUID] - - // Always check HV CRD first: if the VM is already present (e.g. a long-running - // PayG VM being absorbed), confirm immediately without waiting for the grace period. - if hvInstanceSet[vmUUID] { - newStatusAllocations[vmUUID] = expectedHost - logger.V(1).Info("verified VM allocation via Hypervisor CRD", - "vm", vmUUID, - "host", expectedHost) - continue - } - - // VM not yet visible in HV CRD. For unconfirmed VMs still within the grace period - // the HV CRD sync may not have caught up yet (newly spawned VM) — defer and requeue. - if !isConfirmed && isInGracePeriod { - result.HasAllocationsInGracePeriod = true - logger.V(1).Info("allocation in grace period, deferring verification", - "vm", vmUUID, - "allocationAge", allocationAge) - continue - } - - // VM not on the expected host, past grace period or already confirmed. - // For unconfirmed post-grace VMs this is a clean stale allocation — remove it. - if !isConfirmed { - allocationsToRemove = append(allocationsToRemove, vmUUID) - logger.Info("removing stale allocation (VM not found on hypervisor)", + survivingVMs := make(map[string]*reservations.VM, len(vmUUIDs)) + for _, vmUUID := range vmUUIDs { + vm, err := r.VMSource.GetVM(ctx, vmUUID) + if err != nil { + // a failure of a VM lookup should not block the reconciliation of the reservation + // instead, remove VM from allocations; if it was temporary failure, the VM will be consumed by some reservation eventually + logger.V(1).Info("failed to get VM, removing from allocations", "vm", vmUUID, "reservation", res.Name, - "expectedHost", expectedHost, - "allocationAge", allocationAge, - "gracePeriod", r.Conf.AllocationGracePeriod.Duration) + "err", err) continue } - - // Confirmed VM missing from expected host — could be a live migration. - // Scan all HVs lazily; the list is shared across any further misses this cycle. - if err := ensureHVsAndReservations(); err != nil { - return nil, err - } - - var foundHost string - var foundHV hv1.Hypervisor - for _, hv := range allHVs.Items { - if hv.Name == expectedHost { - continue // already checked via hvInstanceSet above - } - for _, inst := range hv.Status.Instances { - if inst.ID == vmUUID { - foundHost = hv.Name - foundHV = hv - break - } + if vm == nil || vm.CurrentHypervisor != expectedHost { + if _, ok := res.Status.CommittedResourceReservation.Allocation[vmUUID]; ok { + logger.Info("removing confirmed allocation (VM not found on expected host)", + "vm", vmUUID, + "reservation", res.Name, + "expectedHost", expectedHost) } - if foundHost != "" { - break + if vmUUID == res.Annotations[v1alpha1.AnnotationCandidateAllocation] { + logger.V(1).Info("candidate VM not found on expected host", + "vm", vmUUID, + "reservation", res.Name, + "expectedHost", expectedHost) } - } - - if foundHost == "" { - // VM is not on any known hypervisor. This covers two cases: - // 1. The VM was terminated or evacuated — correct to remove. - // 2. The VM is mid-live-migration: it has left host-old's HV CRD but - // host-new's CRD has not been updated yet. In this window the VM - // is incorrectly treated as gone and removed from the reservation. - // A VM CRD with lifecycle state (migrating/active) would close this - // gap; without one we accept this narrow race as a known limitation. - allocationsToRemove = append(allocationsToRemove, vmUUID) - logger.Info("removing confirmed allocation (VM not found on any hypervisor)", - "vm", vmUUID, - "reservation", res.Name, - "expectedHost", expectedHost) continue } - - // VM found on a different host — live migration detected. - // - // Follow the VM only when this is the sole VM in the reservation and the new - // host has capacity. Moving TargetHost with multiple VMs present would cause - // the remaining VMs to appear misplaced on the next reconcile. When there are - // multiple VMs, or the new host is at capacity, remove this VM so the slot - // on the original host remains available for re-use. - isSingleVM := len(res.Spec.CommittedResourceReservation.Allocations) == 1 - if isSingleVM && reservations.HostHasCapacityForReservation(allReservations.Items, foundHV, res) { - logger.Info("VM live-migrated to host with capacity, updating TargetHost", - "vm", vmUUID, - "reservation", res.Name, - "oldHost", expectedHost, - "newHost", foundHost) - migrationTargetHost = foundHost - newStatusAllocations[vmUUID] = foundHost - } else { - logger.Info("removing VM from reservation after live migration: either multiple VMs present or new host lacks capacity", - "vm", vmUUID, - "reservation", res.Name, - "expectedHost", expectedHost, - "actualHost", foundHost, - "singleVM", isSingleVM) - allocationsToRemove = append(allocationsToRemove, vmUUID) - } + survivingVMs[vmUUID] = vm } - // Patch the reservation - old := res.DeepCopy() - specChanged := false - - // Remove stale allocations from Spec - if len(allocationsToRemove) > 0 { - for _, vmUUID := range allocationsToRemove { - delete(res.Spec.CommittedResourceReservation.Allocations, vmUUID) - } - specChanged = true + // Pass 2: recompute allocation amounts for surviving VMs. + // remainingSpare tracks slot capacity not yet assigned in this pass. + remainingSpare := make(map[hv1.ResourceName]resource.Quantity, len(res.Spec.Resources)) + for rn, qty := range res.Spec.Resources { + remainingSpare[rn] = qty.DeepCopy() } - - // Advance both TargetHost and Status.Host in the same patch cycle to avoid a - // transient state where Status.Host lags behind TargetHost and blocks capacity - // accounting on the old host during the next reconcile. - if migrationTargetHost != "" { - res.Spec.TargetHost = migrationTargetHost - res.Status.Host = migrationTargetHost - specChanged = true + if len(remainingSpare) == 0 { + logger.Info("Spec.Resources is empty — skipping allocation computation; no VMs will be confirmed", "reservation", res.Name) } + zero := resource.MustParse("0") - // Update Status.Allocations - res.Status.CommittedResourceReservation.Allocations = newStatusAllocations - - // Patch Spec if changed (stale allocations removed and/or TargetHost updated) - if specChanged { - if err := r.Patch(ctx, res, client.MergeFrom(old)); err != nil { - if client.IgnoreNotFound(err) == nil { - return result, nil - } - return nil, fmt.Errorf("failed to patch reservation spec: %w", err) + // Iterate in the same lexical order as pass 1. + // assign resources greedily to lexically earlier VMs + for _, vmUUID := range vmUUIDs { + vm, ok := survivingVMs[vmUUID] + if !ok { + continue } - // Re-fetch to get the updated resource version for status patch - if err := r.Get(ctx, client.ObjectKeyFromObject(res), res); err != nil { - if client.IgnoreNotFound(err) == nil { - return result, nil - } - return nil, fmt.Errorf("failed to re-fetch reservation: %w", err) + vmResources := vmResourcesToHV(vm.Resources) + alreadyElsewhere, err := r.computeAlreadyAssignedElsewhere(ctx, res, vmUUID) + if err != nil { + return fmt.Errorf("failed to compute already-assigned for vm %s: %w", vmUUID, err) + } + vmRemaining := subtractResourceQuantitiesClamped(vmResources, alreadyElsewhere) + newAlloc := minResourceQuantities(remainingSpare, vmRemaining) + newAllocPrimary := newAlloc[primaryResource] + if len(newAlloc) == 0 || newAllocPrimary.IsZero() { + logger.V(1).Info("removing VM allocation: fully covered by other slots", + "vm", vmUUID, "reservation", res.Name) + continue } - // Capture the re-fetched state as the patch base BEFORE re-applying - // the status update. Otherwise MergeFrom(old) would see no diff - // and the status patch would be a no-op. - old = res.DeepCopy() - // Re-apply status updates that were overwritten by the re-fetch. - res.Status.CommittedResourceReservation.Allocations = newStatusAllocations - if migrationTargetHost != "" { - res.Status.Host = migrationTargetHost + newStatusAllocations[vmUUID] = v1alpha1.CommittedResourceAllocation{ + // TODO replace with VM CRD ref once the VM CRD is strictly considered as being the mandatory gate + ObjectRef: corev1.ObjectReference{Name: vmUUID}, + Resources: newAlloc, } - } - - // Proactively remove this VM UUID from all other candidate reservations that still - // carry it in their Spec.Allocations. Only do this for VMs that are newly confirmed - // in this reconcile cycle (present in newStatusAllocations but absent in the snapshot - // taken before any patch) to avoid redundant work on subsequent reconciles. - for vmUUID := range newStatusAllocations { - if _, wasAlreadyConfirmed := existingStatusAllocations[vmUUID]; wasAlreadyConfirmed { - continue + for rn, qty := range newAlloc { + if spare, ok := remainingSpare[rn]; ok { + spare.Sub(qty) + if spare.Cmp(zero) < 0 { + spare = zero.DeepCopy() + } + remainingSpare[rn] = spare + } } - if err := r.cleanupCandidateReservations(ctx, res.Name, vmUUID); err != nil { - return nil, fmt.Errorf("failed to cleanup candidate reservations for vm %s: %w", vmUUID, err) + // is the VM fully covered by slots, or do we need more reservations? + if vmRemainingPrimary := vmRemaining[primaryResource]; len(vmRemaining) == 0 || vmRemainingPrimary.Cmp(newAllocPrimary) > 0 { + logger.V(1).Info("VM not fully covered by slots, may need more reservations", + "vm", vmUUID, + "reservation", res.Name, + "vmRemaining", vmRemaining, + "newAlloc", newAlloc, + ) } } - // Patch Status - patch := client.MergeFrom(old) - if err := r.Status().Patch(ctx, res, patch); err != nil { + // Single status patch. + old := res.DeepCopy() + // Skip the status patch when allocations have not changed. + if apiequality.Semantic.DeepEqual(res.Status.CommittedResourceReservation.Allocation, newStatusAllocations) { + logger.V(1).Info("reconciled allocations, nothing has changed") + return nil + } + res.Status.CommittedResourceReservation.Allocation = newStatusAllocations + if err := r.Status().Patch(ctx, res, client.MergeFrom(old)); err != nil { if client.IgnoreNotFound(err) == nil { - return result, nil + logger.Info("reservation not found, nothing to patch, probably deleted in the meantime", "reservation", res.Name) + return nil } - return nil, fmt.Errorf("failed to patch reservation status: %w", err) + return fmt.Errorf("failed to patch reservation status: %w", err) } - - logger.V(1).Info("reconciled allocations", - "specAllocations", len(res.Spec.CommittedResourceReservation.Allocations), - "statusAllocations", len(newStatusAllocations), - "removedAllocations", len(allocationsToRemove), - "hasAllocationsInGracePeriod", result.HasAllocationsInGracePeriod) - - return result, nil + logger.V(1).Info("reconciled allocations", "remainingSpare", remainingSpare, "statusAllocations", len(newStatusAllocations)) + return nil } -// cleanupCandidateReservations removes vmUUID from Spec.Allocations of all Reservation CRDs -// other than the one that just confirmed the VM. This is called once per newly confirmed VM -// so that candidate slots on non-selected hosts are freed immediately rather than waiting -// for those reservations' own grace period or periodic requeue. -func (r *CommitmentReservationController) cleanupCandidateReservations(ctx context.Context, confirmedReservationName, vmUUID string) error { - logger := LoggerFromContext(ctx).WithValues("component", "controller", "vm", vmUUID) +// getPipelineForFlavorGroup returns the pipeline name for a given flavor group. +func (r *CommitmentReservationController) getPipelineForFlavorGroup(flavorGroupName string, logger logr.Logger) string { + // Try exact match first (e.g., "2152" -> "kvm-cr-hana") + if pipeline, ok := r.Conf.FlavorGroupPipelines[flavorGroupName]; ok { + return pipeline + } - var candidates v1alpha1.ReservationList - if err := r.List(ctx, &candidates, client.MatchingFields{idxReservationByAllocationVMUUID: vmUUID}); err != nil { - return fmt.Errorf("failed to list candidate reservations: %w", err) + // Try wildcard fallback + if pipeline, ok := r.Conf.FlavorGroupPipelines["*"]; ok { + return pipeline } - for i := range candidates.Items { - candidate := &candidates.Items[i] - if candidate.Name == confirmedReservationName { - continue - } - if candidate.Spec.CommittedResourceReservation == nil { + logger.V(1).Info("no pipeline configured for flavor group, using default", "flavorGroup", flavorGroupName, "defaultPipeline", r.Conf.PipelineDefault) + return r.Conf.PipelineDefault +} + +// computeAlreadyAssignedElsewhere returns the total resources already assigned to vmUUID +// by OTHER confirmed slots on the same host. +// Used to compute the correct partial amount when a VM may span multiple reservation slots. +func (r *CommitmentReservationController) computeAlreadyAssignedElsewhere( + ctx context.Context, + res *v1alpha1.Reservation, + vmUUID string, +) (map[hv1.ResourceName]resource.Quantity, error) { + + var conflicts v1alpha1.ReservationList + if err := r.List(ctx, &conflicts, client.MatchingFields{idxReservationByAllocationVMUUID: vmUUID}); err != nil { + return nil, fmt.Errorf("failed to query allocation index for vm %s: %w", vmUUID, err) + } + result := make(map[hv1.ResourceName]resource.Quantity) + for i := range conflicts.Items { + other := &conflicts.Items[i] + if other.Name == res.Name && other.Namespace == res.Namespace { continue } - if _, ok := candidate.Spec.CommittedResourceReservation.Allocations[vmUUID]; !ok { + if other.Status.CommittedResourceReservation == nil { + // unreachable path continue } - old := candidate.DeepCopy() - delete(candidate.Spec.CommittedResourceReservation.Allocations, vmUUID) - if err := r.Patch(ctx, candidate, client.MergeFrom(old)); err != nil { - if client.IgnoreNotFound(err) != nil { - return fmt.Errorf("failed to patch candidate reservation %s: %w", candidate.Name, err) + if other.Status.Host != res.Status.Host { + // stale allocation on a different-host slot — skip; it will self-correct when that slot next reconciles. + continue // only same-host slots contribute + } + if alloc, ok := other.Status.CommittedResourceReservation.Allocation[vmUUID]; ok { + for rn, qty := range alloc.Resources { + if cur, exists := result[rn]; exists { + cur.Add(qty) + result[rn] = cur + } else { + result[rn] = qty.DeepCopy() + } } } - logger.Info("removed vm from candidate reservation", "reservation", candidate.Name, "host", candidate.Status.Host) } - return nil + return result, nil } -// getPipelineForFlavorGroup returns the pipeline name for a given flavor group. -func (r *CommitmentReservationController) getPipelineForFlavorGroup(flavorGroupName string, logger logr.Logger) string { - // Try exact match first (e.g., "2152" -> "kvm-cr-hana") - if pipeline, ok := r.Conf.FlavorGroupPipelines[flavorGroupName]; ok { - return pipeline +// vmResourcesToHV converts VM.Resources (plain string keys from Nova DB) +// to the hv1.ResourceName-keyed format used by reservation slots. +func vmResourcesToHV(r map[string]resource.Quantity) map[hv1.ResourceName]resource.Quantity { + out := make(map[hv1.ResourceName]resource.Quantity, len(r)) + for k, v := range r { + out[hv1.ResourceName(k)] = v } + return out +} - // Try wildcard fallback - if pipeline, ok := r.Conf.FlavorGroupPipelines["*"]; ok { - return pipeline +// minResourceQuantities returns the component-wise minimum of two resource maps. +// For resources present in one map but not the other, the missing side defaults to zero. +func minResourceQuantities(a, b map[hv1.ResourceName]resource.Quantity) map[hv1.ResourceName]resource.Quantity { + zero := resource.MustParse("0") + all := make(map[hv1.ResourceName]struct{}) + for rn := range a { + all[rn] = struct{}{} + } + for rn := range b { + all[rn] = struct{}{} + } + result := make(map[hv1.ResourceName]resource.Quantity, len(all)) + for rn := range all { + av, aok := a[rn] + if !aok { + av = zero + } + bv, bok := b[rn] + if !bok { + bv = zero + } + if av.Cmp(bv) <= 0 { + result[rn] = av.DeepCopy() + } else { + result[rn] = bv.DeepCopy() + } } + return result +} - logger.V(1).Info("no pipeline configured for flavor group, using default", "flavorGroup", flavorGroupName, "defaultPipeline", r.Conf.PipelineDefault) - return r.Conf.PipelineDefault +// subtractResourceQuantitiesClamped returns max(a[rn] - b[rn], 0) for each resource in a. +func subtractResourceQuantitiesClamped(a, b map[hv1.ResourceName]resource.Quantity) map[hv1.ResourceName]resource.Quantity { + zero := resource.MustParse("0") + result := make(map[hv1.ResourceName]resource.Quantity, len(a)) + for rn, av := range a { + diff := av.DeepCopy() + if bv, ok := b[rn]; ok { + diff.Sub(bv) + } + if diff.Cmp(zero) < 0 { + diff = zero.DeepCopy() + } + result[rn] = diff + } + return result } // hypervisorToReservations maps a Hypervisor change event to the set of CR reservations @@ -784,9 +682,6 @@ func (r *CommitmentReservationController) hypervisorToReservations(ctx context.C // Init initializes the reconciler with required clients and DB connection. func (r *CommitmentReservationController) Init(ctx context.Context, conf ReservationControllerConfig) error { - r.SchedulerClient = reservations.NewSchedulerClient(conf.SchedulerURL) - logf.FromContext(ctx).Info("scheduler client initialized for commitment reservation controller", "url", conf.SchedulerURL) - if conf.KeystoneSecretRef.Name != "" { var authenticatedHTTP *http.Client if conf.SSOSecretRef != nil { diff --git a/internal/scheduling/reservations/commitments/reservation_controller_test.go b/internal/scheduling/reservations/commitments/reservation_controller_test.go index 98e4cb867..d5439852f 100644 --- a/internal/scheduling/reservations/commitments/reservation_controller_test.go +++ b/internal/scheduling/reservations/commitments/reservation_controller_test.go @@ -5,16 +5,12 @@ package commitments import ( "context" - "encoding/json" "errors" - "fmt" - "net/http" - "net/http/httptest" - "strconv" "testing" "time" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/meta" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -25,9 +21,39 @@ import ( schedulerdelegationapi "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" + "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" ) +// fakePipeline returns a fixed ordered-host list (modelling a scheduler run). +type fakePipeline struct{ hosts []string } + +func (p fakePipeline) Run(sched.PlacementRequest) (sched.PlacementResult, error) { + return sched.PlacementResult{OrderedHosts: p.hosts}, nil +} + +// fakePipelineProvider serves a fakePipeline for every pipeline name, returning +// the configured host list. +type fakePipelineProvider struct{ hosts []string } + +func (f fakePipelineProvider) GetPipeline(string) (lib.FilterWeigherPipeline[sched.PlacementRequest], bool) { + return fakePipeline(f), true +} + +// fakeGatherer is a no-op gatherer (tests supply their own hosts). +type fakeGatherer struct{} + +func (fakeGatherer) MutateWithAllCandidates(context.Context, *sched.PlacementRequest) error { + return nil +} + +// newMockScheduler builds an in-process compute.Placement whose pipeline returns +// the given host list. Replaces the former httptest mock scheduler server. +func newMockScheduler(hosts []string) *sched.Placement { + return sched.NewPlacement(fakePipelineProvider{hosts: hosts}, fakeGatherer{}) +} + // mockDomainResolver is a test double for DomainResolver. type mockDomainResolver struct { // names maps domainID → name returned on success. @@ -46,6 +72,32 @@ func (m *mockDomainResolver) ResolveDomainName(_ context.Context, domainID strin return m.names[domainID], nil } +// vmSourceByUUID is a test VMSource that looks up VMs by UUID from a fixed map. +// Used by reconcileAllocations tests to replace the former HV CRD presence check. +type vmSourceByUUID struct { + vms map[string]*reservations.VM +} + +func (s *vmSourceByUUID) GetVM(_ context.Context, uuid string) (*reservations.VM, error) { + if s.vms == nil { + return nil, nil + } + return s.vms[uuid], nil +} +func (s *vmSourceByUUID) ListVMs(_ context.Context) ([]reservations.VM, error) { return nil, nil } +func (s *vmSourceByUUID) ListVMsByProject(_ context.Context, _ string) ([]reservations.VM, error) { + return nil, nil +} +func (s *vmSourceByUUID) ListVMsOnHypervisors(_ context.Context, _ *hv1.HypervisorList, _ bool) ([]reservations.VM, error) { + return nil, nil +} +func (s *vmSourceByUUID) IsServerActive(_ context.Context, _ string) (bool, error) { + return false, nil +} +func (s *vmSourceByUUID) GetDeletedVMInfo(_ context.Context, _ string) (*reservations.DeletedVMInfo, error) { + return nil, nil +} + func TestCommitmentReservationController_Reconcile(t *testing.T) { scheme := newCRTestScheme(t) @@ -153,313 +205,486 @@ func TestCommitmentReservationController_Reconcile(t *testing.T) { } } +// TestMigrateSpecAllocationsToAnnotation verifies that a blind-scheduler slot that has +// Spec.CommittedResourceReservation.Allocation populated (old schema) but no +// AnnotationCandidateAllocation annotation gets the annotation seeded from the first +// Spec allocation entry on reconcile (one-time upgrade migration path). +func TestMigrateSpecAllocationsToAnnotation(t *testing.T) { + scheme := newCRTestScheme(t) + + res := &v1alpha1.Reservation{ + ObjectMeta: metav1.ObjectMeta{ + Name: "slot-blind", + }, + Spec: v1alpha1.ReservationSpec{ + Type: v1alpha1.ReservationTypeCommittedResource, + TargetHost: "host-1", + Resources: map[hv1.ResourceName]resource.Quantity{ + hv1.ResourceMemory: resource.MustParse("8Gi"), + }, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ + ProjectID: "test-project", + ResourceName: "test-flavor", + Allocations: map[string]v1alpha1.CommittedResourceAllocation{ //nolint:staticcheck + "vm-legacy-b": {ObjectRef: corev1.ObjectReference{Name: "vm-legacy-b"}}, + "vm-legacy-a": {ObjectRef: corev1.ObjectReference{Name: "vm-legacy-a"}}, + }, + }, + }, + Status: v1alpha1.ReservationStatus{ + Host: "host-1", + Conditions: []metav1.Condition{ + { + Type: v1alpha1.ReservationConditionReady, + Status: metav1.ConditionTrue, + Reason: "ReservationActive", + ObservedGeneration: 1, + }, + }, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{}, + }, + } + + k8sClient := newCRTestClient(scheme, res) + reconciler := &CommitmentReservationController{ + Client: k8sClient, + Scheme: scheme, + Conf: ReservationControllerConfig{ + RequeueIntervalActive: metav1.Duration{Duration: 5 * time.Minute}, + }, + VMSource: &vmSourceByUUID{}, + } + + req := ctrl.Request{NamespacedName: types.NamespacedName{Name: res.Name}} + if _, err := reconciler.Reconcile(context.Background(), req); err != nil { + t.Fatalf("Reconcile() error = %v", err) + } + + var updated v1alpha1.Reservation + if err := k8sClient.Get(context.Background(), req.NamespacedName, &updated); err != nil { + t.Fatalf("Get() error = %v", err) + } + + // Lexically first UUID is "vm-legacy-a". + got := updated.Annotations[v1alpha1.AnnotationCandidateAllocation] + if got != "vm-legacy-a" { + t.Errorf("expected annotation %s=vm-legacy-a, got %q", v1alpha1.AnnotationCandidateAllocation, got) + } +} + // ============================================================================ // Test: reconcileAllocations // ============================================================================ -func TestReconcileAllocations_HypervisorCRDPath(t *testing.T) { +func TestReconcileAllocations(t *testing.T) { scheme := newCRTestScheme(t) - now := time.Now() - recentTime := metav1.NewTime(now.Add(-5 * time.Minute)) // 5 minutes ago (within grace period) - oldTime := metav1.NewTime(now.Add(-30 * time.Minute)) // 30 minutes ago (past grace period) - - config := ReservationControllerConfig{AllocationGracePeriod: metav1.Duration{Duration: 15 * time.Minute}} - - tests := []struct { - name string - reservation *v1alpha1.Reservation - hypervisor *hv1.Hypervisor - expectedStatusAllocations map[string]string - expectedSpecAllocations []string // VM UUIDs expected to remain in spec; nil means no check - expectedHasGracePeriodAllocs bool - }{ + type tc struct { + name string + res *v1alpha1.Reservation + siblings []*v1alpha1.Reservation + vms map[string]*reservations.VM + vmSrc reservations.VMSource // overrides vms when set + secondResName string // if set, fetch fresh + reconcile as second pass + wantAllocKeys []string + wantAbsentKeys []string + wantAllocGiB map[string]int64 // vmUUID → GiB for memory resource + wantObjectRef map[string]string // vmUUID → expected ObjectRef.Name + extraCheck func(t *testing.T, k8sClient client.Client) + } + + tests := []tc{ { - name: "old allocation - VM found on hypervisor CRD", - reservation: newTestCRReservation(map[string]metav1.Time{ - "vm-1": oldTime, - }), - hypervisor: newTestHypervisorCRD("host-1", []hv1.Instance{ - {ID: "vm-1", Name: "vm-1", Active: true}, - }), - expectedStatusAllocations: map[string]string{"vm-1": "host-1"}, - expectedSpecAllocations: []string{"vm-1"}, - expectedHasGracePeriodAllocs: false, + name: "confirmed VM present on expected host", + res: withAlloc(crSlot("test-reservation", "host-1", 8, nil), "vm-1", 8), + vms: map[string]*reservations.VM{ + "vm-1": {UUID: "vm-1", CurrentHypervisor: "host-1", Resources: map[string]resource.Quantity{string(hv1.ResourceMemory): testGiB(8)}}, + }, + wantAllocKeys: []string{"vm-1"}, + wantAllocGiB: map[string]int64{"vm-1": 8}, }, { - name: "old allocation - inactive VM still counted (stopped/shelved)", - reservation: newTestCRReservation(map[string]metav1.Time{ - "vm-stopped": oldTime, - }), - hypervisor: newTestHypervisorCRD("host-1", []hv1.Instance{ - {ID: "vm-stopped", Name: "vm-stopped", Active: false}, - }), - expectedStatusAllocations: map[string]string{"vm-stopped": "host-1"}, - expectedSpecAllocations: []string{"vm-stopped"}, - expectedHasGracePeriodAllocs: false, + name: "confirmed inactive VM still counted (stopped/shelved)", + res: withAlloc(crSlot("test-reservation", "host-1", 8, nil), "vm-stopped", 8), + vms: map[string]*reservations.VM{ + "vm-stopped": {UUID: "vm-stopped", CurrentHypervisor: "host-1", Resources: map[string]resource.Quantity{string(hv1.ResourceMemory): testGiB(8)}}, + }, + wantAllocKeys: []string{"vm-stopped"}, }, { - name: "old allocation - VM not on hypervisor CRD (stale, removed)", - reservation: newTestCRReservation(map[string]metav1.Time{ - "vm-1": oldTime, + name: "confirmed VM absent from VMSource — removed", + res: withAlloc(crSlot("test-reservation", "host-1", 8, nil), "vm-gone", 8), + vms: map[string]*reservations.VM{}, + wantAbsentKeys: []string{"vm-gone"}, + }, + { + name: "empty allocations — no work to do", + res: crSlot("test-reservation", "host-1", 8, nil), + vms: map[string]*reservations.VM{}, + }, + { + name: "confirmed VM stays confirmed — idempotent re-reconcile", + res: withAlloc(crSlot("res-winning", "host-1", 8, nil), "vm-confirmed-uuid", 8), + vms: map[string]*reservations.VM{ + "vm-confirmed-uuid": {UUID: "vm-confirmed-uuid", CurrentHypervisor: "host-1", Resources: map[string]resource.Quantity{string(hv1.ResourceMemory): testGiB(8)}}, + }, + wantAllocKeys: []string{"vm-confirmed-uuid"}, + wantAllocGiB: map[string]int64{"vm-confirmed-uuid": 8}, + }, + { + name: "annotation candidate confirmed — ObjectRef.Name set correctly", + res: crSlot("res-candidate", "host-1", 8, map[string]string{ + v1alpha1.AnnotationCandidateAllocation: "vm-new-candidate", }), - hypervisor: newTestHypervisorCRD("host-1", []hv1.Instance{}), - expectedStatusAllocations: map[string]string{}, - expectedSpecAllocations: []string{}, - expectedHasGracePeriodAllocs: false, + vms: map[string]*reservations.VM{ + "vm-new-candidate": {UUID: "vm-new-candidate", CurrentHypervisor: "host-1", Resources: map[string]resource.Quantity{string(hv1.ResourceMemory): testGiB(8)}}, + }, + wantAllocKeys: []string{"vm-new-candidate"}, + wantObjectRef: map[string]string{"vm-new-candidate": "vm-new-candidate"}, }, { - name: "new allocation within grace period - deferred to requeue", - reservation: newTestCRReservation(map[string]metav1.Time{ - "vm-1": recentTime, + // S02: min taken per resource dimension; slot caps cpu even though VM has more. + name: "S02 multi-dimensional resources — min taken per dimension", + res: &v1alpha1.Reservation{ + ObjectMeta: metav1.ObjectMeta{ + Name: "slot-a", + Annotations: map[string]string{v1alpha1.AnnotationCandidateAllocation: "vm-multi"}, + }, + Spec: v1alpha1.ReservationSpec{ + Type: v1alpha1.ReservationTypeCommittedResource, + TargetHost: "host-1", + Resources: map[hv1.ResourceName]resource.Quantity{ + hv1.ResourceMemory: testGiB(8), + hv1.ResourceCPU: resource.MustParse("4"), + }, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ + ProjectID: "p-1", ResourceName: "test-flavor", PrimaryResource: hv1.ResourceMemory, + }, + }, + Status: v1alpha1.ReservationStatus{ + Host: "host-1", + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{}, + }, + }, + vms: map[string]*reservations.VM{ + "vm-multi": {UUID: "vm-multi", CurrentHypervisor: "host-1", Resources: map[string]resource.Quantity{ + string(hv1.ResourceMemory): testGiB(8), + string(hv1.ResourceCPU): resource.MustParse("8"), + }}, + }, + wantAllocKeys: []string{"vm-multi"}, + wantAllocGiB: map[string]int64{"vm-multi": 8}, + extraCheck: func(t *testing.T, k8sClient client.Client) { + t.Helper() + var updated v1alpha1.Reservation + if err := k8sClient.Get(context.Background(), client.ObjectKey{Name: "slot-a"}, &updated); err != nil { + t.Fatalf("get slot-a: %v", err) + } + alloc := updated.Status.CommittedResourceReservation.Allocation["vm-multi"] + got := alloc.Resources[hv1.ResourceCPU] + if got.Cmp(resource.MustParse("4")) != 0 { + t.Errorf("cpu alloc: want 4, got %s", got.String()) + } + }, + }, + { + // S03: VM is larger than the slot; allocation is capped at slot capacity. + name: "S03 VM too large for slot — allocation capped at slot capacity", + res: crSlot("slot-a", "host-1", 8, map[string]string{ + v1alpha1.AnnotationCandidateAllocation: "vm-big", }), - expectedStatusAllocations: map[string]string{}, - expectedSpecAllocations: []string{"vm-1"}, - expectedHasGracePeriodAllocs: true, + vms: map[string]*reservations.VM{"vm-big": memVM("vm-big", "host-1", 24)}, + wantAllocKeys: []string{"vm-big"}, + wantAllocGiB: map[string]int64{"vm-big": 8}, }, { - name: "mixed allocations - old verified via CRD, new in grace period", - reservation: newTestCRReservation(map[string]metav1.Time{ - "vm-new": recentTime, - "vm-old": oldTime, + // S04: Annotated VM is on a different host — not confirmed. + name: "S04 annotation candidate on wrong host — not confirmed", + res: crSlot("slot-a", "host-1", 8, map[string]string{ + v1alpha1.AnnotationCandidateAllocation: "vm-wrong-host", }), - hypervisor: newTestHypervisorCRD("host-1", []hv1.Instance{ - {ID: "vm-old", Name: "vm-old", Active: true}, + vms: map[string]*reservations.VM{"vm-wrong-host": memVM("vm-wrong-host", "host-2", 8)}, + wantAbsentKeys: []string{"vm-wrong-host"}, + }, + { + // S05: Previously confirmed VM disappears from VMSource — removed. + name: "S05 confirmed VM gone — removed from allocations", + res: withAlloc(crSlot("slot-a", "host-1", 8, nil), "vm-gone", 8), + vms: map[string]*reservations.VM{}, + wantAbsentKeys: []string{"vm-gone"}, + }, + { + // S06: Two VMs on a 12 GiB slot; lexical order determines spare assignment. + // vm-aaa (lexically first) gets 8 GiB; vm-zzz gets the remaining 4 GiB. + name: "S06 two VMs on same slot — lexical order determines spare", + res: &v1alpha1.Reservation{ + ObjectMeta: metav1.ObjectMeta{Name: "slot-a"}, + Spec: v1alpha1.ReservationSpec{ + Type: v1alpha1.ReservationTypeCommittedResource, + TargetHost: "host-1", + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(12)}, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ + ProjectID: "p-1", ResourceName: "test-flavor", PrimaryResource: hv1.ResourceMemory, + }, + }, + Status: v1alpha1.ReservationStatus{ + Host: "host-1", + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + "vm-aaa": {ObjectRef: corev1.ObjectReference{Name: "vm-aaa"}, Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(8)}}, + "vm-zzz": {ObjectRef: corev1.ObjectReference{Name: "vm-zzz"}, Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(8)}}, + }, + }, + }, + }, + vms: map[string]*reservations.VM{ + "vm-aaa": memVM("vm-aaa", "host-1", 8), + "vm-zzz": memVM("vm-zzz", "host-1", 8), + }, + wantAllocGiB: map[string]int64{"vm-aaa": 8, "vm-zzz": 4}, + }, + { + // S07: VM (16 GiB) spans two slots; slot-a already covers 8 GiB, slot-b gets the rest. + name: "S07 VM spans two slots — second slot gets remainder", + res: crSlot("slot-b", "host-1", 8, map[string]string{ + v1alpha1.AnnotationCandidateAllocation: "vm-big", }), - expectedStatusAllocations: map[string]string{"vm-old": "host-1"}, - expectedSpecAllocations: []string{"vm-new", "vm-old"}, - expectedHasGracePeriodAllocs: true, + siblings: []*v1alpha1.Reservation{withAlloc(crSlot("slot-a", "host-1", 8, nil), "vm-big", 8)}, + vms: map[string]*reservations.VM{"vm-big": memVM("vm-big", "host-1", 16)}, + wantAllocKeys: []string{"vm-big"}, + wantAllocGiB: map[string]int64{"vm-big": 8}, + }, + { + // S08: Slot is fully occupied; annotation candidate cannot fit. + name: "S08 slot at capacity — annotation candidate not confirmed", + res: &v1alpha1.Reservation{ + ObjectMeta: metav1.ObjectMeta{ + Name: "slot-a", + Annotations: map[string]string{v1alpha1.AnnotationCandidateAllocation: "vm-new"}, + }, + Spec: v1alpha1.ReservationSpec{ + Type: v1alpha1.ReservationTypeCommittedResource, + TargetHost: "host-1", + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(8)}, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ + ProjectID: "p-1", ResourceName: "test-flavor", PrimaryResource: hv1.ResourceMemory, + }, + }, + Status: v1alpha1.ReservationStatus{ + Host: "host-1", + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + "vm-existing": {ObjectRef: corev1.ObjectReference{Name: "vm-existing"}, Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(8)}}, + }, + }, + }, + }, + vms: map[string]*reservations.VM{ + "vm-existing": memVM("vm-existing", "host-1", 8), + "vm-new": memVM("vm-new", "host-1", 8), + }, + wantAllocGiB: map[string]int64{"vm-existing": 8}, + wantAbsentKeys: []string{"vm-new"}, + }, + { + // S09: VMSource returns error for one VM; that VM is dropped, others are unaffected. + name: "S09 VMSource error for one VM — dropped, others unaffected", + res: &v1alpha1.Reservation{ + ObjectMeta: metav1.ObjectMeta{Name: "slot-a"}, + Spec: v1alpha1.ReservationSpec{ + Type: v1alpha1.ReservationTypeCommittedResource, + TargetHost: "host-1", + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(16)}, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ + ProjectID: "p-1", ResourceName: "test-flavor", PrimaryResource: hv1.ResourceMemory, + }, + }, + Status: v1alpha1.ReservationStatus{ + Host: "host-1", + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + "vm-good": {ObjectRef: corev1.ObjectReference{Name: "vm-good"}, Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(8)}}, + "vm-error": {ObjectRef: corev1.ObjectReference{Name: "vm-error"}, Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(8)}}, + }, + }, + }, + }, + vmSrc: &errForUUID{ + errUUID: "vm-error", + err: errors.New("nova unavailable"), + fallback: &vmSourceByUUID{vms: map[string]*reservations.VM{"vm-good": memVM("vm-good", "host-1", 8)}}, + }, + wantAllocGiB: map[string]int64{"vm-good": 8}, + wantAbsentKeys: []string{"vm-error"}, }, { - name: "empty allocations - no work to do", - reservation: newTestCRReservation(map[string]metav1.Time{}), - expectedStatusAllocations: map[string]string{}, - expectedHasGracePeriodAllocs: false, + // ZeroAllocRemoval: VM fully covered by sibling slot — removed from this slot. + name: "zero alloc removal — VM fully covered by sibling slot removed", + res: withAlloc(crSlot("res-b", "host-1", 8, nil), "vm-race", 8), + siblings: []*v1alpha1.Reservation{withAlloc(crSlot("res-a", "host-1", 8, nil), "vm-race", 8)}, + vms: map[string]*reservations.VM{"vm-race": memVM("vm-race", "host-1", 8)}, + wantAbsentKeys: []string{"vm-race"}, }, { - name: "hypervisor CRD not found - post-grace VM removed", - reservation: newTestCRReservation(map[string]metav1.Time{ - "vm-1": oldTime, + // MultiSlotPartialAllocation: sibling covers 3 GiB of vm-1 (8 GiB VM); this slot gets the remaining 5 GiB. + name: "partial allocation — sibling covers part, this slot gets remainder", + res: crSlot("res-b", "host-1", 8, map[string]string{ + v1alpha1.AnnotationCandidateAllocation: "vm-1", }), - expectedStatusAllocations: map[string]string{}, - expectedSpecAllocations: []string{}, - expectedHasGracePeriodAllocs: false, + siblings: []*v1alpha1.Reservation{withAlloc(crSlot("res-a", "host-1", 16, nil), "vm-1", 3)}, + vms: map[string]*reservations.VM{"vm-1": memVM("vm-1", "host-1", 8)}, + wantAllocGiB: map[string]int64{"vm-1": 5}, }, { - name: "hypervisor CRD not found - grace period VM kept", - reservation: newTestCRReservation(map[string]metav1.Time{ - "vm-1": recentTime, + // S01: PAYG VM annotated on two slots simultaneously. + // slot-a reconciles first and claims vm-x; slot-b reconciles second and defers. + // Total assigned must equal VM size (8 GiB), exactly one slot wins. + name: "S01 PAYG annotated on both slots — total = VM size, exactly one slot wins", + res: crSlot("slot-a", "host-1", 8, map[string]string{ + v1alpha1.AnnotationCandidateAllocation: "vm-x", }), - expectedStatusAllocations: map[string]string{}, - expectedSpecAllocations: []string{"vm-1"}, - expectedHasGracePeriodAllocs: true, + siblings: []*v1alpha1.Reservation{crSlot("slot-b", "host-1", 8, map[string]string{ + v1alpha1.AnnotationCandidateAllocation: "vm-x", + })}, + vms: map[string]*reservations.VM{"vm-x": memVM("vm-x", "host-1", 8)}, + secondResName: "slot-b", + wantAllocKeys: []string{"vm-x"}, + extraCheck: func(t *testing.T, k8sClient client.Client) { + t.Helper() + ctx := context.Background() + var updA, updB v1alpha1.Reservation + if err := k8sClient.Get(ctx, client.ObjectKey{Name: "slot-a"}, &updA); err != nil { + t.Fatalf("get slot-a: %v", err) + } + if err := k8sClient.Get(ctx, client.ObjectKey{Name: "slot-b"}, &updB); err != nil { + t.Fatalf("get slot-b: %v", err) + } + allocA := getAllocGiB(&updA, "vm-x") + allocB := getAllocGiB(&updB, "vm-x") + total := allocA.DeepCopy() + total.Add(allocB) + if total.Cmp(testGiB(8)) != 0 { + t.Errorf("total assigned to vm-x = %s, want 8 GiB (slot-a=%s, slot-b=%s)", total.String(), allocA.String(), allocB.String()) + } + if (allocA.Cmp(testGiB(0)) > 0) == (allocB.Cmp(testGiB(0)) > 0) { + t.Errorf("expected exactly one slot to hold vm-x, got slot-a=%s slot-b=%s", allocA.String(), allocB.String()) + } + }, + }, + { + // S10: Both slots start with vm-x confirmed (race state). + // Step 1: slot-a reconciles, sees slot-b has vm-x → drops it. + // Step 2: slot-b reconciles, sees slot-a has nothing → keeps vm-x. + // Total = 8 GiB, exactly one slot wins. + name: "S10 convergence from race state — one slot drops, other keeps", + res: withAlloc(crSlot("slot-a", "host-1", 8, nil), "vm-x", 8), + siblings: []*v1alpha1.Reservation{withAlloc(crSlot("slot-b", "host-1", 8, nil), "vm-x", 8)}, + vms: map[string]*reservations.VM{"vm-x": memVM("vm-x", "host-1", 8)}, + secondResName: "slot-b", + wantAbsentKeys: []string{"vm-x"}, + extraCheck: func(t *testing.T, k8sClient client.Client) { + t.Helper() + ctx := context.Background() + var updA, updB v1alpha1.Reservation + if err := k8sClient.Get(ctx, client.ObjectKey{Name: "slot-a"}, &updA); err != nil { + t.Fatalf("get slot-a: %v", err) + } + if err := k8sClient.Get(ctx, client.ObjectKey{Name: "slot-b"}, &updB); err != nil { + t.Fatalf("get slot-b: %v", err) + } + allocA := getAllocGiB(&updA, "vm-x") + allocB := getAllocGiB(&updB, "vm-x") + total := allocA.DeepCopy() + total.Add(allocB) + if total.Cmp(testGiB(8)) != 0 { + t.Errorf("total assigned to vm-x = %s, want exactly 8 GiB (slot-a=%s, slot-b=%s)", total.String(), allocA.String(), allocB.String()) + } + if (allocA.Cmp(testGiB(0)) > 0) == (allocB.Cmp(testGiB(0)) > 0) { + t.Errorf("expected exactly one slot to hold vm-x after convergence, got slot-a=%s slot-b=%s", allocA.String(), allocB.String()) + } + }, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - objects := []client.Object{tt.reservation} - if tt.hypervisor != nil { - objects = append(objects, tt.hypervisor) + objects := []client.Object{tt.res} + for _, s := range tt.siblings { + objects = append(objects, s) } - k8sClient := newCRTestClient(scheme, objects...) + vmSrc := tt.vmSrc + if vmSrc == nil { + vmSrc = &vmSourceByUUID{vms: tt.vms} + } controller := &CommitmentReservationController{ - Client: k8sClient, - Scheme: scheme, - Conf: config, + Client: k8sClient, + Scheme: scheme, + Conf: ReservationControllerConfig{}, + VMSource: vmSrc, } + ctx := WithNewReconcileID(context.Background()) - ctx := WithNewGlobalRequestID(context.Background()) - result, err := controller.reconcileAllocations(ctx, tt.reservation) - if err != nil { - t.Fatalf("reconcileAllocations() error = %v", err) + if err := controller.reconcileAllocations(ctx, tt.res); err != nil { + t.Fatalf("reconcileAllocations: %v", err) } - // Check grace period result - if result.HasAllocationsInGracePeriod != tt.expectedHasGracePeriodAllocs { - t.Errorf("expected HasAllocationsInGracePeriod=%v, got %v", - tt.expectedHasGracePeriodAllocs, result.HasAllocationsInGracePeriod) + if tt.secondResName != "" { + var second v1alpha1.Reservation + if err := k8sClient.Get(ctx, client.ObjectKey{Name: tt.secondResName}, &second); err != nil { + t.Fatalf("get second %q: %v", tt.secondResName, err) + } + if err := controller.reconcileAllocations(ctx, &second); err != nil { + t.Fatalf("reconcileAllocations second=%q: %v", tt.secondResName, err) + } } - // Re-fetch reservation to check updates var updated v1alpha1.Reservation - if err := k8sClient.Get(ctx, client.ObjectKeyFromObject(tt.reservation), &updated); err != nil { - t.Fatalf("failed to get updated reservation: %v", err) + if err := k8sClient.Get(ctx, client.ObjectKeyFromObject(tt.res), &updated); err != nil { + t.Fatalf("get updated: %v", err) } - - // Check status allocations - actualStatusAllocs := map[string]string{} + allocs := map[string]v1alpha1.CommittedResourceAllocation{} if updated.Status.CommittedResourceReservation != nil { - actualStatusAllocs = updated.Status.CommittedResourceReservation.Allocations + allocs = updated.Status.CommittedResourceReservation.Allocation } - if len(actualStatusAllocs) != len(tt.expectedStatusAllocations) { - t.Errorf("expected %d status allocations, got %d: %v", - len(tt.expectedStatusAllocations), len(actualStatusAllocs), actualStatusAllocs) + for _, key := range tt.wantAllocKeys { + if _, ok := allocs[key]; !ok { + t.Errorf("want %q in allocations, not found", key) + } } - - for vmUUID, expectedHost := range tt.expectedStatusAllocations { - if actualHost, ok := actualStatusAllocs[vmUUID]; !ok { - t.Errorf("expected VM %s in status allocations", vmUUID) - } else if actualHost != expectedHost { - t.Errorf("VM %s: expected host %s, got %s", vmUUID, expectedHost, actualHost) + for _, key := range tt.wantAbsentKeys { + if _, ok := allocs[key]; ok { + t.Errorf("want %q absent from allocations, but found", key) } } - - // Check spec allocations if expected set is specified - if tt.expectedSpecAllocations != nil { - specAllocs := map[string]bool{} - if updated.Spec.CommittedResourceReservation != nil { - for vmUUID := range updated.Spec.CommittedResourceReservation.Allocations { - specAllocs[vmUUID] = true - } + for vmUUID, wantGiB := range tt.wantAllocGiB { + alloc, ok := allocs[vmUUID] + if !ok { + t.Errorf("wantAllocGiB: %q not in allocations", vmUUID) + continue } - if len(specAllocs) != len(tt.expectedSpecAllocations) { - t.Errorf("expected %d spec allocations, got %d: %v", - len(tt.expectedSpecAllocations), len(specAllocs), specAllocs) + got := alloc.Resources[hv1.ResourceMemory] + if got.Cmp(testGiB(wantGiB)) != 0 { + t.Errorf("memory alloc for %q: want %d GiB, got %s", vmUUID, wantGiB, got.String()) } - for _, vmUUID := range tt.expectedSpecAllocations { - if !specAllocs[vmUUID] { - t.Errorf("expected VM %s in spec allocations", vmUUID) - } + } + for vmUUID, wantName := range tt.wantObjectRef { + alloc, ok := allocs[vmUUID] + if !ok { + t.Errorf("wantObjectRef: %q not in allocations", vmUUID) + continue + } + if alloc.ObjectRef.Name != wantName { + t.Errorf("ObjectRef.Name for %q: want %q, got %q", vmUUID, wantName, alloc.ObjectRef.Name) } } - }) - } -} - -// TestReconcileAllocations_ConfirmedVMDeparture verifies that a VM already confirmed in -// Status.Allocations is removed immediately when it disappears from the HV CRD, without -// waiting for the grace period to expire. -func TestReconcileAllocations_ConfirmedVMDeparture(t *testing.T) { - scheme := newCRTestScheme(t) - config := ReservationControllerConfig{AllocationGracePeriod: metav1.Duration{Duration: 15 * time.Minute}} - now := time.Now() - - // VM was written to Spec.Allocations very recently (within grace period) but was already - // confirmed (present in Status.Allocations). It has since disappeared from the HV CRD. - recentTime := metav1.NewTime(now.Add(-2 * time.Minute)) - res := &v1alpha1.Reservation{ - ObjectMeta: metav1.ObjectMeta{Name: "test-reservation"}, - Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: "host-1", - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - ProjectID: "test-project", - ResourceName: "test-flavor", - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - "vm-confirmed": { - CreationTimestamp: recentTime, - Resources: map[hv1.ResourceName]resource.Quantity{ - "memory": resource.MustParse("4Gi"), - }, - }, - }, - }, - }, - Status: v1alpha1.ReservationStatus{ - Host: "host-1", - Conditions: []metav1.Condition{ - {Type: v1alpha1.ReservationConditionReady, Status: metav1.ConditionTrue, Reason: "ReservationActive"}, - }, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - // VM was previously confirmed — it is in Status.Allocations. - Allocations: map[string]string{"vm-confirmed": "host-1"}, - }, - }, - } - - // HV CRD exists but the VM is gone — simulates termination/evacuation. - hv := newTestHypervisorCRD("host-1", []hv1.Instance{}) - - k8sClient := newCRTestClient(scheme, res, hv) - controller := &CommitmentReservationController{Client: k8sClient, Scheme: scheme, Conf: config} - - ctx := WithNewGlobalRequestID(context.Background()) - result, err := controller.reconcileAllocations(ctx, res) - if err != nil { - t.Fatalf("reconcileAllocations() error = %v", err) - } - - // Must not be treated as grace-period — departure of confirmed VM is immediate. - if result.HasAllocationsInGracePeriod { - t.Error("confirmed VM departure must not trigger grace period requeue") - } - - var updated v1alpha1.Reservation - if err := k8sClient.Get(ctx, client.ObjectKeyFromObject(res), &updated); err != nil { - t.Fatalf("get updated reservation: %v", err) - } - - // Spec.Allocations must be empty — VM removed. - if n := len(updated.Spec.CommittedResourceReservation.Allocations); n != 0 { - t.Errorf("expected 0 spec allocations after departure, got %d", n) - } - // Status.Allocations must also be empty — updated in the same reconcile pass. - if updated.Status.CommittedResourceReservation == nil || - len(updated.Status.CommittedResourceReservation.Allocations) != 0 { - t.Errorf("expected 0 status allocations after departure, got %#v", - updated.Status.CommittedResourceReservation) - } -} - -// newTestCRReservation creates a test CR reservation with allocations on "host-1". -func newTestCRReservation(allocations map[string]metav1.Time) *v1alpha1.Reservation { - const host = "host-1" - specAllocs := make(map[string]v1alpha1.CommittedResourceAllocation) - for vmUUID, timestamp := range allocations { - specAllocs[vmUUID] = v1alpha1.CommittedResourceAllocation{ - CreationTimestamp: timestamp, - Resources: map[hv1.ResourceName]resource.Quantity{ - "memory": resource.MustParse("4Gi"), - "cpu": resource.MustParse("2"), - }, - } - } - - return &v1alpha1.Reservation{ - ObjectMeta: metav1.ObjectMeta{ - Name: "test-reservation", - }, - Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: host, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - ProjectID: "test-project", - ResourceName: "test-flavor", - Allocations: specAllocs, - }, - }, - Status: v1alpha1.ReservationStatus{ - Host: host, - Conditions: []metav1.Condition{ - { - Type: v1alpha1.ReservationConditionReady, - Status: metav1.ConditionTrue, - Reason: "ReservationActive", - }, - }, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - Allocations: make(map[string]string), - }, - }, - } -} - -// newTestHypervisorCRD creates a test Hypervisor CRD with instances. -// - -func newTestHypervisorCRD(name string, instances []hv1.Instance) *hv1.Hypervisor { - return &hv1.Hypervisor{ - ObjectMeta: metav1.ObjectMeta{ - Name: name, - }, - Status: hv1.HypervisorStatus{ - Instances: instances, - }, + if tt.extraCheck != nil { + tt.extraCheck(t, k8sClient) + } + }) } } @@ -521,240 +746,6 @@ func TestHypervisorToReservations(t *testing.T) { } } -// ============================================================================ -// Test: cleanupCandidateReservations -// ============================================================================ - -// TestCleanupCandidateReservations_MultiCandidate covers the BLI #409 acceptance -// criterion: VM UUID in 3 reservations, confirmed on 1, cleaned from 2. -func TestCleanupCandidateReservations_MultiCandidate(t *testing.T) { - scheme := newCRTestScheme(t) - now := time.Now() - oldTime := metav1.NewTime(now.Add(-30 * time.Minute)) - - const vmUUID = "vm-candidate-uuid" - - makeAlloc := func() map[string]v1alpha1.CommittedResourceAllocation { - return map[string]v1alpha1.CommittedResourceAllocation{ - vmUUID: { - CreationTimestamp: oldTime, - Resources: map[hv1.ResourceName]resource.Quantity{"memory": resource.MustParse("4Gi")}, - }, - } - } - - winning := &v1alpha1.Reservation{ - ObjectMeta: metav1.ObjectMeta{Name: "res-winning"}, - Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: "host-1", - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: makeAlloc(), - }, - }, - Status: v1alpha1.ReservationStatus{Host: "host-1"}, - } - candidate2 := &v1alpha1.Reservation{ - ObjectMeta: metav1.ObjectMeta{Name: "res-candidate-2"}, - Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: "host-2", - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: makeAlloc(), - }, - }, - Status: v1alpha1.ReservationStatus{Host: "host-2"}, - } - candidate3 := &v1alpha1.Reservation{ - ObjectMeta: metav1.ObjectMeta{Name: "res-candidate-3"}, - Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: "host-3", - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: makeAlloc(), - }, - }, - Status: v1alpha1.ReservationStatus{Host: "host-3"}, - } - - k8sClient := newCRTestClient(scheme, winning, candidate2, candidate3) - ctrl := &CommitmentReservationController{Client: k8sClient, Scheme: scheme} - ctx := WithNewGlobalRequestID(context.Background()) - - if err := ctrl.cleanupCandidateReservations(ctx, "res-winning", vmUUID); err != nil { - t.Fatalf("cleanupCandidateReservations() error = %v", err) - } - - // winning reservation must still carry the VM UUID - var updatedWinning v1alpha1.Reservation - if err := k8sClient.Get(ctx, client.ObjectKeyFromObject(winning), &updatedWinning); err != nil { - t.Fatalf("failed to get winning reservation: %v", err) - } - if _, ok := updatedWinning.Spec.CommittedResourceReservation.Allocations[vmUUID]; !ok { - t.Errorf("winning reservation should still carry vm UUID %s", vmUUID) - } - - // both candidates must have the VM UUID removed - for _, obj := range []*v1alpha1.Reservation{candidate2, candidate3} { - var updated v1alpha1.Reservation - if err := k8sClient.Get(ctx, client.ObjectKeyFromObject(obj), &updated); err != nil { - t.Fatalf("failed to get %s: %v", obj.Name, err) - } - if _, ok := updated.Spec.CommittedResourceReservation.Allocations[vmUUID]; ok { - t.Errorf("candidate reservation %s should have vm UUID %s removed", obj.Name, vmUUID) - } - } -} - -// TestReconcileAllocations_ConfirmTriggersCandidateCleanup verifies the end-to-end flow: -// when reconcileAllocations confirms a VM on the winning reservation, the same VM UUID -// is removed from candidate reservations on other hosts. -func TestReconcileAllocations_ConfirmTriggersCandidateCleanup(t *testing.T) { - scheme := newCRTestScheme(t) - now := time.Now() - oldTime := metav1.NewTime(now.Add(-30 * time.Minute)) - - const vmUUID = "vm-confirmed-uuid" - config := ReservationControllerConfig{AllocationGracePeriod: metav1.Duration{Duration: 15 * time.Minute}} - - makeAlloc := func() map[string]v1alpha1.CommittedResourceAllocation { - return map[string]v1alpha1.CommittedResourceAllocation{ - vmUUID: { - CreationTimestamp: oldTime, - Resources: map[hv1.ResourceName]resource.Quantity{"memory": resource.MustParse("4Gi")}, - }, - } - } - - winning := &v1alpha1.Reservation{ - ObjectMeta: metav1.ObjectMeta{Name: "res-winning"}, - Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: "host-1", - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: makeAlloc(), - }, - }, - Status: v1alpha1.ReservationStatus{ - Host: "host-1", - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - Allocations: map[string]string{}, // not yet confirmed - }, - }, - } - candidate2 := &v1alpha1.Reservation{ - ObjectMeta: metav1.ObjectMeta{Name: "res-candidate-2"}, - Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: "host-2", - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: makeAlloc(), - }, - }, - Status: v1alpha1.ReservationStatus{Host: "host-2"}, - } - hypervisor := newTestHypervisorCRD("host-1", []hv1.Instance{ - {ID: vmUUID, Name: "vm-name", Active: true}, - }) - - k8sClient := newCRTestClient(scheme, winning, candidate2, hypervisor) - controller := &CommitmentReservationController{Client: k8sClient, Scheme: scheme, Conf: config} - ctx := WithNewGlobalRequestID(context.Background()) - - if _, err := controller.reconcileAllocations(ctx, winning); err != nil { - t.Fatalf("reconcileAllocations() error = %v", err) - } - - // VM must be confirmed on winning reservation - var updatedWinning v1alpha1.Reservation - if err := k8sClient.Get(ctx, client.ObjectKeyFromObject(winning), &updatedWinning); err != nil { - t.Fatalf("failed to get winning reservation: %v", err) - } - if updatedWinning.Status.CommittedResourceReservation == nil || - updatedWinning.Status.CommittedResourceReservation.Allocations[vmUUID] != "host-1" { - t.Errorf("expected vm %s confirmed on host-1 in winning reservation status", vmUUID) - } - - // VM UUID must be removed from the candidate reservation - var updatedCandidate v1alpha1.Reservation - if err := k8sClient.Get(ctx, client.ObjectKeyFromObject(candidate2), &updatedCandidate); err != nil { - t.Fatalf("failed to get candidate reservation: %v", err) - } - if _, ok := updatedCandidate.Spec.CommittedResourceReservation.Allocations[vmUUID]; ok { - t.Errorf("expected vm UUID %s to be removed from candidate reservation", vmUUID) - } -} - -// TestCleanupCandidateReservations_NoDoubleCleanup verifies that a VM UUID already -// confirmed in a previous reconcile cycle (present in existingStatusAllocations) does -// not trigger a redundant cleanup call. -func TestReconcileAllocations_NoCleanupForAlreadyConfirmedVM(t *testing.T) { - scheme := newCRTestScheme(t) - now := time.Now() - oldTime := metav1.NewTime(now.Add(-30 * time.Minute)) - - const vmUUID = "vm-already-confirmed" - config := ReservationControllerConfig{AllocationGracePeriod: metav1.Duration{Duration: 15 * time.Minute}} - - // Winning reservation already has the VM in Status.Allocations from a prior cycle. - winning := &v1alpha1.Reservation{ - ObjectMeta: metav1.ObjectMeta{Name: "res-winning"}, - Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: "host-1", - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - vmUUID: { - CreationTimestamp: oldTime, - Resources: map[hv1.ResourceName]resource.Quantity{"memory": resource.MustParse("4Gi")}, - }, - }, - }, - }, - Status: v1alpha1.ReservationStatus{ - Host: "host-1", - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - Allocations: map[string]string{vmUUID: "host-1"}, // already confirmed - }, - }, - } - // This candidate still has the VM UUID — it should NOT be touched because the VM - // was confirmed in a previous cycle (the cleanup already ran then). - staleCandidate := &v1alpha1.Reservation{ - ObjectMeta: metav1.ObjectMeta{Name: "res-stale-candidate"}, - Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: "host-2", - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - vmUUID: {CreationTimestamp: oldTime}, - }, - }, - }, - Status: v1alpha1.ReservationStatus{Host: "host-2"}, - } - hypervisor := newTestHypervisorCRD("host-1", []hv1.Instance{ - {ID: vmUUID, Name: "vm-name", Active: true}, - }) - - k8sClient := newCRTestClient(scheme, winning, staleCandidate, hypervisor) - controller := &CommitmentReservationController{Client: k8sClient, Scheme: scheme, Conf: config} - ctx := WithNewGlobalRequestID(context.Background()) - - if _, err := controller.reconcileAllocations(ctx, winning); err != nil { - t.Fatalf("reconcileAllocations() error = %v", err) - } - - // Stale candidate should be untouched — cleanup only fires on first confirmation. - var updatedCandidate v1alpha1.Reservation - if err := k8sClient.Get(ctx, client.ObjectKeyFromObject(staleCandidate), &updatedCandidate); err != nil { - t.Fatalf("failed to get stale candidate: %v", err) - } - if _, ok := updatedCandidate.Spec.CommittedResourceReservation.Allocations[vmUUID]; !ok { - t.Errorf("stale candidate should not have been touched on a re-reconcile of an already-confirmed VM") - } -} - func TestCommitmentReservationController_reconcileInstanceReservation_Success(t *testing.T) { scheme := newCRTestScheme(t) @@ -780,44 +771,11 @@ func TestCommitmentReservationController_reconcileInstanceReservation_Success(t k8sClient := newCRTestClient(scheme, reservation, newTestFlavorKnowledge(), hypervisor1, hypervisor2) - // Create a mock server that returns a successful response - mockResponse := &schedulerdelegationapi.ExternalSchedulerResponse{ - Hosts: []string{"test-host-1", "test-host-2"}, - } - - server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - // Verify the request body - var req schedulerdelegationapi.ExternalSchedulerRequest - if err := json.NewDecoder(r.Body).Decode(&req); err != nil { - t.Errorf("Failed to decode request: %v", err) - w.WriteHeader(http.StatusBadRequest) - return - } - - // Verify request structure - if req.Spec.Data.NumInstances != 1 { - t.Errorf("Expected NumInstances to be 1, got %d", req.Spec.Data.NumInstances) - } - - if err := json.NewEncoder(w).Encode(mockResponse); err != nil { - t.Fatalf("failed to write response: %v", err) - } - })) - defer server.Close() - - config := ReservationControllerConfig{ - SchedulerURL: server.URL, - } - reconciler := &CommitmentReservationController{ - Client: k8sClient, - Scheme: scheme, - Conf: config, - } - - // Initialize the reconciler (this sets up SchedulerClient) - if err := reconciler.Init(context.Background(), config); err != nil { - t.Fatalf("Failed to initialize reconciler: %v", err) + Client: k8sClient, + Scheme: scheme, + Conf: ReservationControllerConfig{}, + Scheduler: newMockScheduler([]string{"test-host-1", "test-host-2"}), } req := ctrl.Request{ @@ -876,20 +834,19 @@ func TestCommitmentReservationController_reconcileInstanceReservation_Success(t // Test: domain_name scheduler hint // ============================================================================ -// newTestSchedulerServer captures the decoded request and returns a single host. -func newTestSchedulerServer(t *testing.T, captured *schedulerdelegationapi.ExternalSchedulerRequest) *httptest.Server { - t.Helper() - return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if err := json.NewDecoder(r.Body).Decode(captured); err != nil { - t.Errorf("failed to decode scheduler request: %v", err) - w.WriteHeader(http.StatusBadRequest) - return - } - resp := schedulerdelegationapi.ExternalSchedulerResponse{Hosts: []string{"test-host-1"}} - if err := json.NewEncoder(w).Encode(resp); err != nil { - t.Fatalf("failed to write scheduler response: %v", err) - } - })) +// capturingPipeline records the SchedulerRequest it was run with and returns a +// single host, so tests can assert on the request the reconciler built. +type capturingPipeline struct{ captured *sched.PlacementRequest } + +func (p capturingPipeline) Run(req sched.PlacementRequest) (sched.PlacementResult, error) { + *p.captured = req + return sched.PlacementResult{OrderedHosts: []string{"test-host-1"}}, nil +} + +type capturingPipelineProvider struct{ captured *sched.PlacementRequest } + +func (f capturingPipelineProvider) GetPipeline(string) (lib.FilterWeigherPipeline[sched.PlacementRequest], bool) { + return capturingPipeline(f), true } func TestCommitmentReservationController_DomainNameHint(t *testing.T) { @@ -938,16 +895,13 @@ func TestCommitmentReservationController_DomainNameHint(t *testing.T) { k8sClient := newCRTestClient(scheme, reservation, newTestFlavorKnowledge(), &hv1.Hypervisor{ObjectMeta: metav1.ObjectMeta{Name: "test-host-1"}}) - var captured schedulerdelegationapi.ExternalSchedulerRequest - server := newTestSchedulerServer(t, &captured) - defer server.Close() - + var captured sched.PlacementRequest reconciler := &CommitmentReservationController{ - Client: k8sClient, - Scheme: scheme, - Conf: ReservationControllerConfig{SchedulerURL: server.URL}, - SchedulerClient: reservations.NewSchedulerClient(server.URL), - DomainResolver: tt.resolver, + Client: k8sClient, + Scheme: scheme, + Conf: ReservationControllerConfig{}, + Scheduler: sched.NewPlacement(capturingPipelineProvider{captured: &captured}, fakeGatherer{}), + DomainResolver: tt.resolver, } if _, err := reconciler.Reconcile(context.Background(), ctrl.Request{ @@ -964,257 +918,14 @@ func TestCommitmentReservationController_DomainNameHint(t *testing.T) { } } - // Verify domain_name hint. - gotDomain, err := captured.Spec.Data.GetSchedulerHintStr("domain_name") - if tt.wantDomainName == "" { - if err == nil { - t.Errorf("expected domain_name hint absent, got %q", gotDomain) - } - } else { - if err != nil { - t.Fatalf("expected domain_name hint %q, got error: %v", tt.wantDomainName, err) - } - if gotDomain != tt.wantDomainName { - t.Errorf("expected domain_name hint %q, got %q", tt.wantDomainName, gotDomain) - } - } - - // _nova_check_type must always be present regardless of resolver outcome. - gotIntent, err := captured.Spec.Data.GetSchedulerHintStr("_nova_check_type") - if err != nil || gotIntent != string(schedulerdelegationapi.ReserveForCommittedResourceIntent) { - t.Errorf("expected _nova_check_type=%q, got %q (err=%v)", schedulerdelegationapi.ReserveForCommittedResourceIntent, gotIntent, err) + // Verify domain_name hint (now the typed request field). + if captured.DomainName != tt.wantDomainName { + t.Errorf("expected domain name %q, got %q", tt.wantDomainName, captured.DomainName) } - }) - } -} - -// ============================================================================ -// Tests: live migration detection in reconcileAllocations -// ============================================================================ - -// newHVWithCapacity creates a Hypervisor CRD with the given instances and effective capacity. -func newHVWithCapacity(name string, memGiB, cpuCores int64, instances []hv1.Instance) *hv1.Hypervisor { - return &hv1.Hypervisor{ - ObjectMeta: metav1.ObjectMeta{Name: name}, - Status: hv1.HypervisorStatus{ - EffectiveCapacity: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceMemory: resource.MustParse(fmt.Sprintf("%dGi", memGiB)), - hv1.ResourceCPU: resource.MustParse(strconv.FormatInt(cpuCores, 10)), - }, - Instances: instances, - }, - } -} - -// newConfirmedCRReservation creates a ready CR reservation with one confirmed VM on host. -// slotMemGiB/slotCPU define the full reservation slot; vmMemGiB/vmCPU define what the VM -// actually consumes — these may be smaller, leaving an unfilled remainder in the slot. -func newConfirmedCRReservation(name, host, vmUUID string, slotMemGiB, slotCPU, vmMemGiB, vmCPU int64) *v1alpha1.Reservation { - return &v1alpha1.Reservation{ - ObjectMeta: metav1.ObjectMeta{Name: name}, - Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: host, - Resources: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceMemory: resource.MustParse(fmt.Sprintf("%dGi", slotMemGiB)), - hv1.ResourceCPU: resource.MustParse(strconv.FormatInt(slotCPU, 10)), - }, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - ProjectID: "test-project", - ResourceName: "test-flavor", - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - vmUUID: { - CreationTimestamp: metav1.NewTime(time.Now().Add(-1 * time.Hour)), - Resources: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceMemory: resource.MustParse(fmt.Sprintf("%dGi", vmMemGiB)), - hv1.ResourceCPU: resource.MustParse(strconv.FormatInt(vmCPU, 10)), - }, - }, - }, - }, - }, - Status: v1alpha1.ReservationStatus{ - Host: host, - Conditions: []metav1.Condition{ - {Type: v1alpha1.ReservationConditionReady, Status: metav1.ConditionTrue, Reason: "ReservationActive"}, - }, - CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - Allocations: map[string]string{vmUUID: host}, - }, - }, - } -} - -func TestReconcileAllocations_LiveMigration(t *testing.T) { - const ( - vmUUID = "vm-uuid" - vm2UUID = "vm-uuid-2" - oldHost = "host-old" - newHost = "host-new" - ) - - config := ReservationControllerConfig{AllocationGracePeriod: metav1.Duration{Duration: 15 * time.Minute}} - - tests := []struct { - name string - // reservation to use; nil uses the default single-VM reservation - reservation *v1alpha1.Reservation - // extra objects beyond the base reservation and old host HV - extraObjects []client.Object - // expected outcomes after first reconcile pass - wantTargetHost string - wantStatusHost string // expected in Status.Allocations[vmUUID]; "" means absent - wantSpecHasVM bool - // if true, run a second reconcile pass and assert state is stable - assertSecondPass bool - }{ - { - name: "single VM, new host has capacity: follow the VM", - extraObjects: []client.Object{ - newHVWithCapacity(newHost, 960, 80, []hv1.Instance{{ID: vmUUID, Active: true}}), - }, - wantTargetHost: newHost, - wantStatusHost: newHost, - wantSpecHasVM: true, - assertSecondPass: true, - }, - { - name: "single VM, new host at capacity: remove VM, slot stays on old host", - extraObjects: []client.Object{ - // VM (240Gi/20) running on newHost; a full-slot blocker leaves no room for - // the 240Gi/20 slot remainder. - newHVWithCapacity(newHost, 480, 40, []hv1.Instance{{ID: vmUUID, Active: true}}), - &v1alpha1.Reservation{ - ObjectMeta: metav1.ObjectMeta{Name: "res-blocker"}, - Spec: v1alpha1.ReservationSpec{ - Type: v1alpha1.ReservationTypeCommittedResource, - TargetHost: newHost, - Resources: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceMemory: resource.MustParse("480Gi"), - hv1.ResourceCPU: resource.MustParse("40"), - }, - }, - Status: v1alpha1.ReservationStatus{Host: newHost}, - }, - }, - wantTargetHost: oldHost, - wantStatusHost: "", - wantSpecHasVM: false, - }, - { - name: "multiple VMs, one migrated: remove migrated VM, never update TargetHost", - reservation: func() *v1alpha1.Reservation { - res := newConfirmedCRReservation("res-1", oldHost, vmUUID, 480, 40, 240, 20) - // Add a second VM confirmed on oldHost. - res.Spec.CommittedResourceReservation.Allocations[vm2UUID] = v1alpha1.CommittedResourceAllocation{ - CreationTimestamp: metav1.NewTime(time.Now().Add(-1 * time.Hour)), - Resources: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceMemory: resource.MustParse("240Gi"), - hv1.ResourceCPU: resource.MustParse("20"), - }, - } - res.Status.CommittedResourceReservation.Allocations[vm2UUID] = oldHost - return res - }(), - extraObjects: []client.Object{ - // vmUUID has migrated to newHost with plenty of capacity; vm2UUID stays on oldHost. - // Supply oldHost HV explicitly so vm2UUID is present on it. - newTestHypervisorCRD(oldHost, []hv1.Instance{{ID: vm2UUID, Active: true}}), - newHVWithCapacity(newHost, 960, 80, []hv1.Instance{{ID: vmUUID, Active: true}}), - }, - wantTargetHost: oldHost, - wantStatusHost: "", - wantSpecHasVM: false, - }, - { - name: "VM gone from all hosts: remove allocation", - extraObjects: []client.Object{newTestHypervisorCRD("host-other", []hv1.Instance{{ID: "other-vm"}})}, - wantTargetHost: oldHost, - wantStatusHost: "", - wantSpecHasVM: false, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - scheme := newCRTestScheme(t) - res := tt.reservation - if res == nil { - res = newConfirmedCRReservation("res-1", oldHost, vmUUID, 480, 40, 240, 20) - } - - var objects []client.Object - objects = append(objects, res) - - // Add an oldHost HV unless the test provides its own via extraObjects. - addsOldHostHV := false - for _, obj := range tt.extraObjects { - if hv, ok := obj.(*hv1.Hypervisor); ok && hv.Name == oldHost { - addsOldHostHV = true - break - } - } - if !addsOldHostHV { - objects = append(objects, newTestHypervisorCRD(oldHost, []hv1.Instance{})) - } - objects = append(objects, tt.extraObjects...) - - k8sClient := newCRTestClient(scheme, objects...) - controller := &CommitmentReservationController{Client: k8sClient, Scheme: scheme, Conf: config} - ctx := WithNewGlobalRequestID(context.Background()) - - if _, err := controller.reconcileAllocations(ctx, res); err != nil { - t.Fatalf("reconcileAllocations() error = %v", err) - } - - var updated v1alpha1.Reservation - if err := k8sClient.Get(ctx, client.ObjectKeyFromObject(res), &updated); err != nil { - t.Fatalf("failed to get updated reservation: %v", err) - } - - if updated.Spec.TargetHost != tt.wantTargetHost { - t.Errorf("Spec.TargetHost = %q, want %q", updated.Spec.TargetHost, tt.wantTargetHost) - } - _, specHasVM := updated.Spec.CommittedResourceReservation.Allocations[vmUUID] - if specHasVM != tt.wantSpecHasVM { - t.Errorf("VM in Spec.Allocations = %v, want %v", specHasVM, tt.wantSpecHasVM) - } - var statusHost string - if updated.Status.CommittedResourceReservation != nil { - statusHost = updated.Status.CommittedResourceReservation.Allocations[vmUUID] - } - if statusHost != tt.wantStatusHost { - t.Errorf("Status.Allocations[%s] = %q, want %q", vmUUID, statusHost, tt.wantStatusHost) - } - - // For the migration-follow case: run a second reconcile to confirm state is - // stable and Status.Host was advanced to the new host in the first pass. - if tt.assertSecondPass { - if _, err := controller.reconcileAllocations(ctx, &updated); err != nil { - t.Fatalf("second reconcileAllocations() error = %v", err) - } - var updated2 v1alpha1.Reservation - if err := k8sClient.Get(ctx, client.ObjectKeyFromObject(res), &updated2); err != nil { - t.Fatalf("failed to get reservation after second pass: %v", err) - } - if updated2.Spec.TargetHost != tt.wantTargetHost { - t.Errorf("second pass: Spec.TargetHost = %q, want %q", updated2.Spec.TargetHost, tt.wantTargetHost) - } - if updated2.Status.Host != tt.wantTargetHost { - t.Errorf("second pass: Status.Host = %q, want %q", updated2.Status.Host, tt.wantTargetHost) - } - _, specHasVM2 := updated2.Spec.CommittedResourceReservation.Allocations[vmUUID] - if !specHasVM2 { - t.Errorf("second pass: VM unexpectedly removed from Spec.Allocations") - } - var statusHost2 string - if updated2.Status.CommittedResourceReservation != nil { - statusHost2 = updated2.Status.CommittedResourceReservation.Allocations[vmUUID] - } - if statusHost2 != tt.wantStatusHost { - t.Errorf("second pass: Status.Allocations[%s] = %q, want %q", vmUUID, statusHost2, tt.wantStatusHost) - } + // Intent must always be set regardless of resolver outcome. + if captured.Intent != schedulerdelegationapi.ReserveForCommittedResourceIntent { + t.Errorf("expected Intent=%q, got %q", schedulerdelegationapi.ReserveForCommittedResourceIntent, captured.Intent) } }) } @@ -1313,7 +1024,9 @@ func TestReconcile_RevokesReadyWhenTargetHostCleared(t *testing.T) { }, }, CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{ - Allocations: map[string]string{"vm-1": "host-1"}, + Allocation: map[string]v1alpha1.CommittedResourceAllocation{ + "vm-1": {ObjectRef: corev1.ObjectReference{Name: "vm-1"}}, + }, }, }, } @@ -1336,8 +1049,8 @@ func TestReconcile_RevokesReadyWhenTargetHostCleared(t *testing.T) { if updated.Status.Host != "" { t.Errorf("expected Status.Host cleared, got %q", updated.Status.Host) } - if updated.Status.CommittedResourceReservation != nil && len(updated.Status.CommittedResourceReservation.Allocations) != 0 { - t.Errorf("expected Status.Allocations cleared, got %v", updated.Status.CommittedResourceReservation.Allocations) + if updated.Status.CommittedResourceReservation != nil && len(updated.Status.CommittedResourceReservation.Allocation) != 0 { + t.Errorf("expected Status.allocation cleared, got %v", updated.Status.CommittedResourceReservation.Allocation) } cond := meta.FindStatusCondition(updated.Status.Conditions, v1alpha1.ReservationConditionReady) if cond == nil || cond.Status != metav1.ConditionFalse { @@ -1347,3 +1060,91 @@ func TestReconcile_RevokesReadyWhenTargetHostCleared(t *testing.T) { t.Errorf("expected reason PlacementRevoked, got %q", cond.Reason) } } + +// ============================================================================ +// Helpers for reconcileAllocations table test +// ============================================================================ + +// errForUUID is a VMSource that returns a simulated error for one specific UUID +// and delegates all other lookups to a backing vmSourceByUUID map. +type errForUUID struct { + errUUID string + err error + fallback *vmSourceByUUID +} + +func (e *errForUUID) GetVM(_ context.Context, uuid string) (*reservations.VM, error) { + if uuid == e.errUUID { + return nil, e.err + } + return e.fallback.GetVM(context.Background(), uuid) +} +func (e *errForUUID) ListVMs(_ context.Context) ([]reservations.VM, error) { return nil, nil } +func (e *errForUUID) ListVMsByProject(_ context.Context, _ string) ([]reservations.VM, error) { + return nil, nil +} +func (e *errForUUID) ListVMsOnHypervisors(_ context.Context, _ *hv1.HypervisorList, _ bool) ([]reservations.VM, error) { + return nil, nil +} +func (e *errForUUID) IsServerActive(_ context.Context, _ string) (bool, error) { return false, nil } +func (e *errForUUID) GetDeletedVMInfo(_ context.Context, _ string) (*reservations.DeletedVMInfo, error) { + return nil, nil +} + +// crSlot is a compact builder for CommittedResource reservation test objects. +func crSlot(name, host string, memGiB int64, annotations map[string]string) *v1alpha1.Reservation { //nolint:unparam + return &v1alpha1.Reservation{ + ObjectMeta: metav1.ObjectMeta{Name: name, Annotations: annotations}, + Spec: v1alpha1.ReservationSpec{ + Type: v1alpha1.ReservationTypeCommittedResource, + TargetHost: host, + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(memGiB)}, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ + ProjectID: "p-1", + ResourceName: "test-flavor", + PrimaryResource: hv1.ResourceMemory, + }, + }, + Status: v1alpha1.ReservationStatus{ + Host: host, + CommittedResourceReservation: &v1alpha1.CommittedResourceReservationStatus{}, + }, + } +} + +// withAlloc adds a confirmed allocation to a slot for the given VM. +func withAlloc(slot *v1alpha1.Reservation, vmUUID string, memGiB int64) *v1alpha1.Reservation { + if slot.Status.CommittedResourceReservation.Allocation == nil { + slot.Status.CommittedResourceReservation.Allocation = make(map[string]v1alpha1.CommittedResourceAllocation) + } + slot.Status.CommittedResourceReservation.Allocation[vmUUID] = v1alpha1.CommittedResourceAllocation{ + ObjectRef: corev1.ObjectReference{Name: vmUUID}, + Resources: map[hv1.ResourceName]resource.Quantity{hv1.ResourceMemory: testGiB(memGiB)}, + } + return slot +} + +// memVM builds a reservations.VM with the given memory on the given host. +func memVM(uuid, host string, memGiB int64) *reservations.VM { + return &reservations.VM{ + UUID: uuid, + CurrentHypervisor: host, + Resources: map[string]resource.Quantity{string(hv1.ResourceMemory): testGiB(memGiB)}, + } +} + +// getAlloc returns the memory allocation for vmUUID in a slot, or 0 if absent. +func getAllocGiB(slot *v1alpha1.Reservation, vmUUID string) resource.Quantity { //nolint:unparam + if slot.Status.CommittedResourceReservation == nil { + return testGiB(0) + } + alloc, ok := slot.Status.CommittedResourceReservation.Allocation[vmUUID] + if !ok { + return testGiB(0) + } + q, ok := alloc.Resources[hv1.ResourceMemory] + if !ok { + return testGiB(0) + } + return q +} diff --git a/internal/scheduling/reservations/commitments/reservation_manager.go b/internal/scheduling/reservations/commitments/reservation_manager.go index aa6b89384..3df5651d1 100644 --- a/internal/scheduling/reservations/commitments/reservation_manager.go +++ b/internal/scheduling/reservations/commitments/reservation_manager.go @@ -217,7 +217,7 @@ func (m *ReservationManager) ApplyCommitmentState( } if reservationToDelete == nil { for i, res := range existing { - if len(res.Spec.CommittedResourceReservation.Allocations) == 0 { + if res.Status.CommittedResourceReservation == nil || len(res.Status.CommittedResourceReservation.Allocation) == 0 { reservationToDelete = &res existing = append(existing[:i], existing[i+1:]...) // remove from existing list break @@ -258,7 +258,7 @@ func (m *ReservationManager) ApplyCommitmentState( for deltaMemoryBytes > 0 && len(allCandidates) > 0 { // Largest candidate that fits the delta (candidates sorted descending). - // If none fit, pick the smallest to minimise waste on the undersized slot. + // If none fit, pick the smallest to minimize waste on the undersized slot. idx := len(allCandidates) - 1 for i, c := range allCandidates { if int64(c.MemoryMB)*1024*1024 <= deltaMemoryBytes { //nolint:gosec // bounded by flavor specs @@ -408,14 +408,11 @@ func (m *ReservationManager) syncReservationMetadata( if (state.CommitmentUUID != "" && reservation.Spec.CommittedResourceReservation.CommitmentUUID != state.CommitmentUUID) || (state.DomainID != "" && reservation.Spec.CommittedResourceReservation.DomainID != state.DomainID) || (state.StartTime != nil && (reservation.Spec.StartTime == nil || !reservation.Spec.StartTime.Time.Equal(*state.StartTime))) || - (state.EndTime != nil && (reservation.Spec.EndTime == nil || !reservation.Spec.EndTime.Time.Equal(*state.EndTime))) || - (state.ParentGeneration != 0 && reservation.Spec.CommittedResourceReservation.ParentGeneration != state.ParentGeneration) { + (state.EndTime != nil && (reservation.Spec.EndTime == nil || !reservation.Spec.EndTime.Time.Equal(*state.EndTime))) { // Apply patch logger.V(1).Info("syncing reservation metadata", "reservation", reservation.Name, "commitmentUUID", state.CommitmentUUID, - "currentParentGen", reservation.Spec.CommittedResourceReservation.ParentGeneration, - "newParentGen", state.ParentGeneration, ) patch := client.MergeFrom(reservation.DeepCopy()) @@ -426,9 +423,6 @@ func (m *ReservationManager) syncReservationMetadata( if state.DomainID != "" { reservation.Spec.CommittedResourceReservation.DomainID = state.DomainID } - if state.ParentGeneration != 0 { - reservation.Spec.CommittedResourceReservation.ParentGeneration = state.ParentGeneration - } if state.StartTime != nil { reservation.Spec.StartTime = &metav1.Time{Time: *state.StartTime} } @@ -509,14 +503,13 @@ func (m *ReservationManager) newReservation( ), }, CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - ProjectID: state.ProjectID, - CommitmentUUID: state.CommitmentUUID, - DomainID: state.DomainID, - ResourceGroup: state.FlavorGroupName, - ResourceName: flavorInGroup.Name, - Creator: creator, - ParentGeneration: state.ParentGeneration, - Allocations: nil, + ProjectID: state.ProjectID, + CommitmentUUID: state.CommitmentUUID, + DomainID: state.DomainID, + ResourceGroup: state.FlavorGroupName, + ResourceName: flavorInGroup.Name, + Creator: creator, + PrimaryResource: primaryResourceForCommitment(state.ResourceType), }, } @@ -588,22 +581,13 @@ func (m *ReservationManager) newPaygReservation( hv1.ResourceCPU: *resource.NewQuantity(cpus, resource.DecimalSI), }, CommittedResourceReservation: &v1alpha1.CommittedResourceReservationSpec{ - ProjectID: state.ProjectID, - CommitmentUUID: state.CommitmentUUID, - DomainID: state.DomainID, - ResourceGroup: state.FlavorGroupName, - ResourceName: candidate.FlavorName, - Creator: creator, - ParentGeneration: state.ParentGeneration, - Allocations: map[string]v1alpha1.CommittedResourceAllocation{ - candidate.VMID: { - CreationTimestamp: metav1.Now(), - Resources: map[hv1.ResourceName]resource.Quantity{ - hv1.ResourceMemory: *resource.NewQuantity(memoryBytes, resource.BinarySI), - hv1.ResourceCPU: *resource.NewQuantity(cpus, resource.DecimalSI), - }, - }, - }, + ProjectID: state.ProjectID, + CommitmentUUID: state.CommitmentUUID, + DomainID: state.DomainID, + ResourceGroup: state.FlavorGroupName, + ResourceName: candidate.FlavorName, + Creator: creator, + PrimaryResource: primaryResourceForCommitment(state.ResourceType), }, } if state.AvailabilityZone != "" { @@ -623,7 +607,8 @@ func (m *ReservationManager) newPaygReservation( v1alpha1.LabelReservationType: v1alpha1.ReservationTypeLabelCommittedResource, }, Annotations: map[string]string{ - v1alpha1.AnnotationCreatorRequestID: state.CreatorRequestID, + v1alpha1.AnnotationCreatorRequestID: state.CreatorRequestID, + v1alpha1.AnnotationCandidateAllocation: candidate.VMID, }, }, Spec: spec, @@ -635,3 +620,12 @@ func (m *ReservationManager) newPaygReservation( } return reservation, nil } + +func primaryResourceForCommitment(rt v1alpha1.CommittedResourceType) hv1.ResourceName { //nolint:unparam + switch rt { + case v1alpha1.CommittedResourceTypeMemory: + return hv1.ResourceMemory + default: + return hv1.ResourceMemory + } +} diff --git a/internal/scheduling/reservations/commitments/reservation_manager_test.go b/internal/scheduling/reservations/commitments/reservation_manager_test.go index 0ee1209a7..84b798cc4 100644 --- a/internal/scheduling/reservations/commitments/reservation_manager_test.go +++ b/internal/scheduling/reservations/commitments/reservation_manager_test.go @@ -14,6 +14,7 @@ import ( "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "github.com/go-logr/logr" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/meta" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -21,12 +22,9 @@ import ( ) // newTestCRSlot creates a Reservation slot for commitment "abc123" / project "project-1". -// Pass nil allocs for an empty allocation map. -func newTestCRSlot(name string, memGiB int64, targetHost, resourceGroup string, allocs map[string]v1alpha1.CommittedResourceAllocation) v1alpha1.Reservation { - if allocs == nil { - allocs = map[string]v1alpha1.CommittedResourceAllocation{} - } - return v1alpha1.Reservation{ +// Pass nil statusAllocs for a slot with no confirmed VMs. +func newTestCRSlot(name string, memGiB int64, targetHost, resourceGroup string, statusAllocs map[string]v1alpha1.CommittedResourceAllocation) v1alpha1.Reservation { + res := v1alpha1.Reservation{ ObjectMeta: metav1.ObjectMeta{ Name: name, Labels: map[string]string{ @@ -43,10 +41,15 @@ func newTestCRSlot(name string, memGiB int64, targetHost, resourceGroup string, ProjectID: "project-1", ResourceGroup: resourceGroup, Creator: "syncer", - Allocations: allocs, }, }, } + if len(statusAllocs) > 0 { + res.Status.CommittedResourceReservation = &v1alpha1.CommittedResourceReservationStatus{ + Allocation: statusAllocs, + } + } + return res } // withAZ returns a copy of the reservation with the given availability zone set. @@ -176,7 +179,7 @@ func TestApplyCommitmentState(t *testing.T) { { name: "deletion priority: unscheduled (no TargetHost) deleted before scheduled", existingSlots: []v1alpha1.Reservation{ - newTestCRSlot("commitment-abc123-0", 8, "host-1", "test-group", map[string]v1alpha1.CommittedResourceAllocation{"vm-123": {}}), + newTestCRSlot("commitment-abc123-0", 8, "host-1", "test-group", map[string]v1alpha1.CommittedResourceAllocation{"vm-123": {ObjectRef: corev1.ObjectReference{Name: "vm-123"}}}), newTestCRSlot("commitment-abc123-1", 8, "", "test-group", nil), }, desiredMemoryGiB: 8, @@ -190,7 +193,9 @@ func TestApplyCommitmentState(t *testing.T) { if len(remaining) != 1 { t.Fatalf("expected 1 remaining, got %d", len(remaining)) } - if remaining[0].Spec.TargetHost == "" || len(remaining[0].Spec.CommittedResourceReservation.Allocations) == 0 { + if remaining[0].Spec.TargetHost == "" || + remaining[0].Status.CommittedResourceReservation == nil || + len(remaining[0].Status.CommittedResourceReservation.Allocation) == 0 { t.Error("expected scheduled reservation with allocations to remain") } }, @@ -198,13 +203,14 @@ func TestApplyCommitmentState(t *testing.T) { { name: "deletion priority: unused scheduled (no allocations) deleted before allocated", existingSlots: []v1alpha1.Reservation{ - newTestCRSlot("commitment-abc123-0", 8, "host-1", "test-group", map[string]v1alpha1.CommittedResourceAllocation{"vm-123": {}}), + newTestCRSlot("commitment-abc123-0", 8, "host-1", "test-group", map[string]v1alpha1.CommittedResourceAllocation{"vm-123": {ObjectRef: corev1.ObjectReference{Name: "vm-123"}}}), newTestCRSlot("commitment-abc123-1", 8, "host-2", "test-group", nil), }, desiredMemoryGiB: 8, wantRemovedCount: 1, validateRemoved: func(t *testing.T, removed []v1alpha1.Reservation) { - if len(removed[0].Spec.CommittedResourceReservation.Allocations) != 0 { + if removed[0].Status.CommittedResourceReservation != nil && + len(removed[0].Status.CommittedResourceReservation.Allocation) != 0 { t.Error("expected reservation without allocations to be removed") } }, @@ -212,7 +218,8 @@ func TestApplyCommitmentState(t *testing.T) { if len(remaining) != 1 { t.Fatalf("expected 1 remaining, got %d", len(remaining)) } - if len(remaining[0].Spec.CommittedResourceReservation.Allocations) == 0 { + if remaining[0].Status.CommittedResourceReservation == nil || + len(remaining[0].Status.CommittedResourceReservation.Allocation) == 0 { t.Error("expected reservation with allocations to remain") } }, @@ -220,7 +227,7 @@ func TestApplyCommitmentState(t *testing.T) { { name: "deletion priority: unscheduled removed first across mixed set", existingSlots: []v1alpha1.Reservation{ - newTestCRSlot("commitment-abc123-0", 8, "host-1", "test-group", map[string]v1alpha1.CommittedResourceAllocation{"vm-allocated": {}}), + newTestCRSlot("commitment-abc123-0", 8, "host-1", "test-group", map[string]v1alpha1.CommittedResourceAllocation{"vm-allocated": {ObjectRef: corev1.ObjectReference{Name: "vm-allocated"}}}), newTestCRSlot("commitment-abc123-1", 8, "host-2", "test-group", nil), newTestCRSlot("commitment-abc123-2", 8, "", "test-group", nil), newTestCRSlot("commitment-abc123-3", 8, "", "test-group", nil), @@ -632,8 +639,8 @@ func TestApplyCommitmentState_PAYG(t *testing.T) { if res.Spec.CommittedResourceReservation == nil { t.Fatal("CommittedResourceReservation is nil") } - if _, ok := res.Spec.CommittedResourceReservation.Allocations[vmUUID]; !ok { - t.Errorf("expected vm %q in Spec.Allocations", vmUUID) + if res.Annotations[v1alpha1.AnnotationCandidateAllocation] != vmUUID { + t.Errorf("expected annotation %s=%q, got %q", v1alpha1.AnnotationCandidateAllocation, vmUUID, res.Annotations[v1alpha1.AnnotationCandidateAllocation]) } if res.Spec.CommittedResourceReservation.ResourceName != "small" { t.Errorf("ResourceName: want %q, got %q", "small", res.Spec.CommittedResourceReservation.ResourceName) @@ -653,8 +660,8 @@ func TestApplyCommitmentState_PAYG(t *testing.T) { if touched[0].Spec.TargetHost != "" { t.Errorf("expected no TargetHost for blind slot, got %q", touched[0].Spec.TargetHost) } - if len(touched[0].Spec.CommittedResourceReservation.Allocations) != 0 { - t.Error("expected no pre-allocations for blind slot") + if touched[0].Annotations[v1alpha1.AnnotationCandidateAllocation] != "" { + t.Error("expected no candidate VM annotation for blind slot") } }, }, @@ -667,7 +674,7 @@ func TestApplyCommitmentState_PAYG(t *testing.T) { // Different commitment UUID so Phase 1 doesn't count it against our delta. func() v1alpha1.Reservation { s := withAZ(newTestCRSlot("other-cr-0", 8, hvName, "test-group", - map[string]v1alpha1.CommittedResourceAllocation{vmUUID: {}}), az) + map[string]v1alpha1.CommittedResourceAllocation{vmUUID: {ObjectRef: corev1.ObjectReference{Name: vmUUID}}}), az) s.Spec.CommittedResourceReservation.CommitmentUUID = "other-uuid" return s }(), @@ -697,8 +704,8 @@ func TestApplyCommitmentState_PAYG(t *testing.T) { if res.Spec.TargetHost != hvName { t.Errorf("TargetHost: want %q, got %q", hvName, res.Spec.TargetHost) } - if _, ok := res.Spec.CommittedResourceReservation.Allocations[vmUUID]; !ok { - t.Errorf("expected vm %q in Spec.Allocations", vmUUID) + if res.Annotations[v1alpha1.AnnotationCandidateAllocation] != vmUUID { + t.Errorf("expected annotation %s=%q, got %q", v1alpha1.AnnotationCandidateAllocation, vmUUID, res.Annotations[v1alpha1.AnnotationCandidateAllocation]) } wantMem := int64(4) * 1024 * 1024 * 1024 memQty := res.Spec.Resources[hv1.ResourceMemory] @@ -748,8 +755,8 @@ func TestApplyCommitmentState_PAYG(t *testing.T) { if touched[0].Spec.TargetHost != "" { t.Errorf("expected Phase 5 blind slot (no TargetHost), got %q", touched[0].Spec.TargetHost) } - if len(touched[0].Spec.CommittedResourceReservation.Allocations) != 0 { - t.Error("expected no pre-allocations when flag is disabled") + if touched[0].Annotations[v1alpha1.AnnotationCandidateAllocation] != "" { + t.Error("expected no candidate VM annotation when flag is disabled") } }, }, @@ -765,9 +772,9 @@ func TestApplyCommitmentState_PAYG(t *testing.T) { if len(touched) != 1 { t.Fatalf("want 1 slot, got %d", len(touched)) } - if _, ok := touched[0].Spec.CommittedResourceReservation.Allocations["vm-small"]; !ok { - t.Errorf("expected vm-small (8GiB exact fit), got allocations: %v", - touched[0].Spec.CommittedResourceReservation.Allocations) + if touched[0].Annotations[v1alpha1.AnnotationCandidateAllocation] != "vm-small" { + t.Errorf("expected vm-small (8GiB exact fit), got annotation: %v", + touched[0].Annotations[v1alpha1.AnnotationCandidateAllocation]) } }, }, @@ -783,9 +790,9 @@ func TestApplyCommitmentState_PAYG(t *testing.T) { if len(touched) != 1 { t.Fatalf("want 1 slot, got %d", len(touched)) } - if _, ok := touched[0].Spec.CommittedResourceReservation.Allocations["vm-small"]; !ok { - t.Errorf("expected smallest oversized vm-small, got allocations: %v", - touched[0].Spec.CommittedResourceReservation.Allocations) + if touched[0].Annotations[v1alpha1.AnnotationCandidateAllocation] != "vm-small" { + t.Errorf("expected smallest oversized vm-small, got annotation: %v", + touched[0].Annotations[v1alpha1.AnnotationCandidateAllocation]) } wantMem := int64(4) * 1024 * 1024 * 1024 memQty := touched[0].Spec.Resources[hv1.ResourceMemory] @@ -817,7 +824,7 @@ func TestApplyCommitmentState_PAYG(t *testing.T) { for _, res := range touched { if res.Spec.TargetHost != "" { preAllocated++ - for vm := range res.Spec.CommittedResourceReservation.Allocations { + if vm := res.Annotations[v1alpha1.AnnotationCandidateAllocation]; vm != "" { usedVMs[vm] = true } } @@ -932,8 +939,8 @@ func TestApplyCommitmentState_PAYG(t *testing.T) { if res.Spec.TargetHost != hvName { t.Errorf("TargetHost: want %q, got %q", hvName, res.Spec.TargetHost) } - if _, ok := res.Spec.CommittedResourceReservation.Allocations[vmID]; !ok { - t.Errorf("expected PAYG VM %q in Allocations", vmID) + if res.Annotations[v1alpha1.AnnotationCandidateAllocation] != vmID { + t.Errorf("expected annotation %s=%q, got %q", v1alpha1.AnnotationCandidateAllocation, vmID, res.Annotations[v1alpha1.AnnotationCandidateAllocation]) } wantMem := int64(4080) * 1024 * 1024 memQty := res.Spec.Resources[hv1.ResourceMemory] @@ -1343,3 +1350,20 @@ func TestSelectFlavor(t *testing.T) { }) } } + +func TestPrimaryResourceForCommitment(t *testing.T) { + tests := []struct { + rt v1alpha1.CommittedResourceType + want hv1.ResourceName + }{ + {v1alpha1.CommittedResourceTypeMemory, hv1.ResourceMemory}, + {"unknown", hv1.ResourceMemory}, + {"", hv1.ResourceMemory}, + } + for _, tt := range tests { + got := primaryResourceForCommitment(tt.rt) + if got != tt.want { + t.Errorf("primaryResourceForCommitment(%q) = %q, want %q", tt.rt, got, tt.want) + } + } +} diff --git a/internal/scheduling/reservations/commitments/state.go b/internal/scheduling/reservations/commitments/state.go index bf6014dbf..9aa182f1c 100644 --- a/internal/scheduling/reservations/commitments/state.go +++ b/internal/scheduling/reservations/commitments/state.go @@ -11,6 +11,7 @@ import ( "time" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "github.com/sapcc/go-api-declarations/liquid" ) @@ -126,11 +127,6 @@ type CommitmentState struct { // When set (e.g. "-"), Reservation CRDs are named "". // Used by the CommittedResource controller; leave empty for the legacy syncer path. NamePrefix string - // ParentGeneration is the Generation of the parent CommittedResource CRD. Written into - // Reservation spec so the Reservation controller can echo it back in status, letting - // the CR controller detect when all children have been processed for the current spec. - // Zero for syncer-created reservations (no parent CR). - ParentGeneration int64 // State is the lifecycle state from Limes (planned/pending/guaranteed/confirmed/superseded/expired). State v1alpha1.CommitmentStatus } @@ -350,7 +346,7 @@ func FromReservations(reservations []v1alpha1.Reservation) (*CommitmentState, er continue } - memoryQuantity := res.Spec.Resources["memory"] + memoryQuantity := res.Spec.Resources[hv1.ResourceMemory] memoryBytes := memoryQuantity.Value() state.TotalMemoryBytes += memoryBytes } diff --git a/internal/scheduling/reservations/commitments/syncer.go b/internal/scheduling/reservations/commitments/syncer.go index ef5bcf97a..1071b0bd5 100644 --- a/internal/scheduling/reservations/commitments/syncer.go +++ b/internal/scheduling/reservations/commitments/syncer.go @@ -11,7 +11,7 @@ import ( "time" "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + knowledge "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" "github.com/go-logr/logr" "github.com/sapcc/go-api-declarations/liquid" @@ -105,7 +105,7 @@ type getCommitmentStatesResult struct { skippedUUIDs map[string]bool } -func (s *Syncer) getCommitmentStates(ctx context.Context, log logr.Logger, flavorGroups map[string]compute.FlavorGroupFeature) (*getCommitmentStatesResult, error) { +func (s *Syncer) getCommitmentStates(ctx context.Context, log logr.Logger, flavorGroups map[string]knowledge.FlavorGroupFeature) (*getCommitmentStatesResult, error) { allProjects, err := s.ListProjects(ctx) if err != nil { return nil, err @@ -235,8 +235,8 @@ func (s *Syncer) SyncReservations(ctx context.Context) error { // Create context with request ID for this sync execution runID := fmt.Sprintf("sync-%d", time.Now().Unix()) - ctx = WithNewGlobalRequestID(ctx) - logger := LoggerFromContext(ctx).WithValues("runID", runID) + ctx = reservations.WithGlobalRequestID(ctx, runID) + logger := LoggerFromContext(ctx) logger.Info("starting commitment sync") @@ -248,8 +248,7 @@ func (s *Syncer) SyncReservations(ctx context.Context) error { }() // Check if flavor group knowledge is ready - knowledge := &reservations.FlavorGroupKnowledgeClient{Client: s.Client} - knowledgeCRD, err := knowledge.Get(ctx) + knowledgeCRD, err := knowledge.GetFlavorGroupsKnowledge(ctx, s.Client) if err != nil { logger.Error(err, "failed to check flavor group knowledge readiness") if s.monitor != nil { @@ -263,7 +262,7 @@ func (s *Syncer) SyncReservations(ctx context.Context) error { } // Get flavor groups using the CRD we already fetched - flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, knowledgeCRD) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, s.Client, knowledgeCRD) if err != nil { logger.Error(err, "failed to get flavor groups from knowledge") if s.monitor != nil { diff --git a/internal/scheduling/reservations/commitments/usage.go b/internal/scheduling/reservations/commitments/usage.go index 90ab813d9..b28275d1b 100644 --- a/internal/scheduling/reservations/commitments/usage.go +++ b/internal/scheduling/reservations/commitments/usage.go @@ -12,8 +12,9 @@ import ( "time" "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + knowledge "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" + hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "github.com/go-logr/logr" "github.com/sapcc/go-api-declarations/liquid" . "go.xyrillian.de/gg/option" @@ -98,14 +99,13 @@ func (c *UsageCalculator) CalculateUsage( allAZs []liquid.AvailabilityZone, ) (liquid.ServiceUsageReport, error) { - knowledge := &reservations.FlavorGroupKnowledgeClient{Client: c.client} - flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, c.client, nil) if err != nil { return liquid.ServiceUsageReport{}, fmt.Errorf("failed to get flavor groups: %w", err) } var infoVersion int64 = -1 - if knowledgeCRD, err := knowledge.Get(ctx); err == nil && knowledgeCRD != nil && !knowledgeCRD.Status.LastContentChange.IsZero() { + if knowledgeCRD, err := knowledge.GetFlavorGroupsKnowledge(ctx, c.client); err == nil && knowledgeCRD != nil && !knowledgeCRD.Status.LastContentChange.IsZero() { infoVersion = knowledgeCRD.Status.LastContentChange.Unix() } @@ -156,7 +156,10 @@ func (c *UsageCalculator) BuildVMAssignmentsFromStatus(ctx context.Context, proj } assignments := make(map[string]string) for _, cr := range crList.Items { - for _, vmUUID := range cr.Status.AssignedInstances { + if cr.Status.Billing == nil { + continue + } + for _, vmUUID := range cr.Status.Billing.AssignedInstances { assignments[vmUUID] = cr.Spec.CommitmentUUID } } @@ -240,7 +243,7 @@ func getProjectVMs( vmSource reservations.VMSource, log logr.Logger, projectID string, - flavorGroups map[string]compute.FlavorGroupFeature, + flavorGroups map[string]knowledge.FlavorGroupFeature, allAZs []liquid.AvailabilityZone, ) ([]VMUsageInfo, error) { @@ -274,10 +277,10 @@ func getProjectVMs( memoryMB := uint64(0) vcpus := uint64(0) - if qty, ok := vm.Resources["memory"]; ok { + if qty, ok := vm.Resources[string(hv1.ResourceMemory)]; ok { memoryMB = uint64(qty.Value()) / (1024 * 1024) //nolint:gosec } - if qty, ok := vm.Resources["vcpus"]; ok { + if qty, ok := vm.Resources[string(hv1.ResourceCPU)]; ok { vcpus = uint64(qty.Value()) //nolint:gosec } @@ -429,7 +432,7 @@ func buildCombinedQuotaMap(pqs []v1alpha1.ProjectQuota) map[string]map[string]in func (c *UsageCalculator) buildUsageResponse( vms []VMUsageInfo, vmAssignments map[string]string, - flavorGroups map[string]compute.FlavorGroupFeature, + flavorGroups map[string]knowledge.FlavorGroupFeature, allAZs []liquid.AvailabilityZone, infoVersion int64, quotaByResourceAZ map[string]map[string]int64, diff --git a/internal/scheduling/reservations/commitments/usage_reconciler.go b/internal/scheduling/reservations/commitments/usage_reconciler.go index 0e350036a..2f4ae18f6 100644 --- a/internal/scheduling/reservations/commitments/usage_reconciler.go +++ b/internal/scheduling/reservations/commitments/usage_reconciler.go @@ -24,6 +24,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/reconcile" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + knowledge "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" "github.com/cobaltcore-dev/cortex/pkg/multicluster" ) @@ -51,12 +52,9 @@ func (r *UsageReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl // Only active commitments have assigned VMs. Clear stale usage status if present. if !cr.IsActive() { log.V(1).Info("skipping: commitment state is not active", "state", cr.Spec.State) - if len(cr.Status.AssignedInstances) > 0 || len(cr.Status.UsedResources) > 0 { + if cr.Status.Billing != nil { old := cr.DeepCopy() - cr.Status.AssignedInstances = nil - cr.Status.UsedResources = nil - cr.Status.LastUsageReconcileAt = nil - cr.Status.UsageObservedGeneration = nil + cr.Status.Billing = nil cr.Status.StatusSummary = computeStatusSummary(cr.Spec, cr.Status, start) if err := r.Status().Patch(ctx, &cr, client.MergeFrom(old)); err != nil { return ctrl.Result{}, client.IgnoreNotFound(err) @@ -70,12 +68,9 @@ func (r *UsageReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl // so we check EndTime directly to prevent stale assignments from persisting past expiry. if cr.Spec.EndTime != nil && cr.Spec.EndTime.Time.Before(start) { log.V(1).Info("skipping: commitment has expired", "endTime", cr.Spec.EndTime) - if len(cr.Status.AssignedInstances) > 0 || len(cr.Status.UsedResources) > 0 { + if cr.Status.Billing != nil { old := cr.DeepCopy() - cr.Status.AssignedInstances = nil - cr.Status.UsedResources = nil - cr.Status.LastUsageReconcileAt = nil - cr.Status.UsageObservedGeneration = nil + cr.Status.Billing = nil cr.Status.StatusSummary = computeStatusSummary(cr.Spec, cr.Status, start) if err := r.Status().Patch(ctx, &cr, client.MergeFrom(old)); err != nil { return ctrl.Result{}, client.IgnoreNotFound(err) @@ -107,10 +102,10 @@ func (r *UsageReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl // Bypass cooldown when the spec generation has advanced since the last usage reconcile. // This ensures spec changes (e.g. shrink) are reflected immediately rather than waiting // for the next cooldown interval — follows the Kubernetes observedGeneration pattern. - generationAdvanced := cr.Status.UsageObservedGeneration == nil || - *cr.Status.UsageObservedGeneration != cr.Generation - if !generationAdvanced && cr.Status.LastUsageReconcileAt != nil { - if elapsed := time.Since(cr.Status.LastUsageReconcileAt.Time); elapsed < cooldown { + generationAdvanced := cr.Status.Billing == nil || cr.Status.Billing.ObservedGeneration == nil || + *cr.Status.Billing.ObservedGeneration != cr.Generation + if !generationAdvanced && cr.Status.Billing.LastReconcileAt != nil { + if elapsed := time.Since(cr.Status.Billing.LastReconcileAt.Time); elapsed < cooldown { return ctrl.Result{RequeueAfter: cooldown - elapsed}, nil } } @@ -123,8 +118,7 @@ func (r *UsageReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl logger := log logger.V(1).Info("usage reconcile starting", "trigger", trigger, "generation", cr.Generation) - knowledge := &reservations.FlavorGroupKnowledgeClient{Client: r.Client} - flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, r.Client, nil) if err != nil { r.Monitor.reconcileDuration.WithLabelValues("error").Observe(time.Since(start).Seconds()) return ctrl.Result{}, err @@ -185,8 +179,8 @@ func (r *UsageReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl r.Monitor.reconcileDuration.WithLabelValues("success").Observe(time.Since(start).Seconds()) // Observe status age once per reconcile, not once per commitment, to avoid biasing the // histogram toward projects with many commitments. - if written > 0 && cr.Status.LastUsageReconcileAt != nil { - r.Monitor.statusAge.Observe(now.Time.Sub(cr.Status.LastUsageReconcileAt.Time).Seconds()) + if written > 0 && cr.Status.Billing != nil && cr.Status.Billing.LastReconcileAt != nil { + r.Monitor.statusAge.Observe(now.Time.Sub(cr.Status.Billing.LastReconcileAt.Time).Seconds()) } r.Monitor.assignedInstances.WithLabelValues(cr.Spec.ProjectID).Set(float64(totalAssigned)) @@ -217,13 +211,15 @@ func (r *UsageReconciler) writeUsageStatus(ctx context.Context, state *Commitmen usedQty := resource.NewQuantity(usedBytes, resource.BinarySI) usedCores := resource.NewQuantity(state.UsedVCPUs, resource.DecimalSI) - target.Status.AssignedInstances = state.AssignedInstances - target.Status.UsedResources = map[string]resource.Quantity{ - "memory": *usedQty, - "cpu": *usedCores, + target.Status.Billing = &v1alpha1.BillingStatus{ + AssignedInstances: state.AssignedInstances, + UsedResources: map[string]resource.Quantity{ + string(hv1.ResourceMemory): *usedQty, + string(hv1.ResourceCPU): *usedCores, + }, + LastReconcileAt: &now, + ObservedGeneration: &target.Generation, } - target.Status.LastUsageReconcileAt = &now - target.Status.UsageObservedGeneration = &target.Generation target.Status.StatusSummary = computeStatusSummary(target.Spec, target.Status, now.Time) return r.Status().Patch(ctx, target, client.MergeFrom(old)) @@ -347,11 +343,11 @@ func (p acceptedGenerationPredicate) Update(e event.UpdateEvent) bool { return false } // Don't fire if usage is already up to date for the accepted generation. - if newCR.Status.UsageObservedGeneration != nil && *newCR.Status.UsageObservedGeneration >= newCond.ObservedGeneration { + if newCR.Status.Billing != nil && newCR.Status.Billing.ObservedGeneration != nil && *newCR.Status.Billing.ObservedGeneration >= newCond.ObservedGeneration { return false } p.log.V(1).Info("predicate fired: Ready accepted, usage not yet up to date", "name", newCR.Name, "generation", newCR.Generation, - "usageObservedGeneration", newCR.Status.UsageObservedGeneration) + "billing", newCR.Status.Billing) return true } diff --git a/internal/scheduling/reservations/commitments/usage_reconciler_test.go b/internal/scheduling/reservations/commitments/usage_reconciler_test.go index ee07b0632..46bd38da8 100644 --- a/internal/scheduling/reservations/commitments/usage_reconciler_test.go +++ b/internal/scheduling/reservations/commitments/usage_reconciler_test.go @@ -97,8 +97,8 @@ func TestAcceptedGenerationPredicate_Update(t *testing.T) { new: &v1alpha1.CommittedResource{ ObjectMeta: metav1.ObjectMeta{Generation: 1}, Status: v1alpha1.CommittedResourceStatus{ - Conditions: []metav1.Condition{readyCond}, - UsageObservedGeneration: i64ptr(1), + Conditions: []metav1.Condition{readyCond}, + Billing: &v1alpha1.BillingStatus{ObservedGeneration: i64ptr(1)}, }, }, want: false, @@ -109,8 +109,7 @@ func TestAcceptedGenerationPredicate_Update(t *testing.T) { new: &v1alpha1.CommittedResource{ ObjectMeta: metav1.ObjectMeta{Generation: 1}, Status: v1alpha1.CommittedResourceStatus{ - Conditions: []metav1.Condition{readyCond}, - UsageObservedGeneration: nil, + Conditions: []metav1.Condition{readyCond}, }, }, want: true, @@ -121,8 +120,8 @@ func TestAcceptedGenerationPredicate_Update(t *testing.T) { new: &v1alpha1.CommittedResource{ ObjectMeta: metav1.ObjectMeta{Generation: 1}, Status: v1alpha1.CommittedResourceStatus{ - Conditions: []metav1.Condition{readyCond}, - UsageObservedGeneration: i64ptr(0), // lags behind Ready.ObservedGeneration=1 + Conditions: []metav1.Condition{readyCond}, + Billing: &v1alpha1.BillingStatus{ObservedGeneration: i64ptr(0)}, // lags behind Ready.ObservedGeneration=1 }, }, want: true, @@ -174,8 +173,10 @@ func TestUsageReconciler_Reconcile_Gates(t *testing.T) { t.Run("non-active state with stale data: clears AssignedInstances and timestamps", func(t *testing.T) { now := metav1.Now() cr := newTestCommittedResource("test-cr", v1alpha1.CommitmentStatusExpired) - cr.Status.AssignedInstances = []string{"vm-1", "vm-2"} - cr.Status.LastUsageReconcileAt = &now + cr.Status.Billing = &v1alpha1.BillingStatus{ + AssignedInstances: []string{"vm-1", "vm-2"}, + LastReconcileAt: &now, + } k8sClient := newCRTestClient(scheme, cr) r := newUsageReconciler(k8sClient, cooldown) @@ -188,11 +189,11 @@ func TestUsageReconciler_Reconcile_Gates(t *testing.T) { if err := k8sClient.Get(context.Background(), types.NamespacedName{Name: cr.Name}, &updated); err != nil { t.Fatalf("get CR: %v", err) } - if len(updated.Status.AssignedInstances) != 0 { - t.Errorf("AssignedInstances = %v, want nil", updated.Status.AssignedInstances) + if updated.Status.Billing != nil && len(updated.Status.Billing.AssignedInstances) != 0 { + t.Errorf("AssignedInstances = %v, want nil", updated.Status.Billing.AssignedInstances) } - if updated.Status.LastUsageReconcileAt != nil { - t.Errorf("LastUsageReconcileAt = %v, want nil", updated.Status.LastUsageReconcileAt) + if updated.Status.Billing != nil && updated.Status.Billing.LastReconcileAt != nil { + t.Errorf("LastReconcileAt = %v, want nil", updated.Status.Billing.LastReconcileAt) } }) @@ -201,8 +202,10 @@ func TestUsageReconciler_Reconcile_Gates(t *testing.T) { pastTime := metav1.NewTime(time.Now().Add(-1 * time.Minute)) cr := newTestCommittedResource("test-cr", v1alpha1.CommitmentStatusConfirmed) cr.Spec.EndTime = &pastTime - cr.Status.AssignedInstances = []string{"vm-1"} - cr.Status.LastUsageReconcileAt = &now + cr.Status.Billing = &v1alpha1.BillingStatus{ + AssignedInstances: []string{"vm-1"}, + LastReconcileAt: &now, + } k8sClient := newCRTestClient(scheme, cr) r := newUsageReconciler(k8sClient, cooldown) @@ -215,11 +218,11 @@ func TestUsageReconciler_Reconcile_Gates(t *testing.T) { if err := k8sClient.Get(context.Background(), types.NamespacedName{Name: cr.Name}, &updated); err != nil { t.Fatalf("get CR: %v", err) } - if len(updated.Status.AssignedInstances) != 0 { - t.Errorf("AssignedInstances = %v, want nil", updated.Status.AssignedInstances) + if updated.Status.Billing != nil && len(updated.Status.Billing.AssignedInstances) != 0 { + t.Errorf("AssignedInstances = %v, want nil", updated.Status.Billing.AssignedInstances) } - if updated.Status.LastUsageReconcileAt != nil { - t.Errorf("LastUsageReconcileAt = %v, want nil", updated.Status.LastUsageReconcileAt) + if updated.Status.Billing != nil && updated.Status.Billing.LastReconcileAt != nil { + t.Errorf("LastReconcileAt = %v, want nil", updated.Status.Billing.LastReconcileAt) } }) @@ -278,8 +281,10 @@ func TestUsageReconciler_Reconcile_Gates(t *testing.T) { ObservedGeneration: 1, Reason: v1alpha1.CommittedResourceReasonAccepted, }} - cr.Status.LastUsageReconcileAt = &now - cr.Status.UsageObservedGeneration = i64ptr(1) // up to date → generationAdvanced=false + cr.Status.Billing = &v1alpha1.BillingStatus{ + LastReconcileAt: &now, + ObservedGeneration: i64ptr(1), // up to date → generationAdvanced=false + } r := newUsageReconciler(newCRTestClient(scheme, cr), cooldown) result, err := r.Reconcile(context.Background(), reconcileReq(cr.Name)) @@ -466,19 +471,22 @@ func TestUsageReconciler_WriteUsageStatus(t *testing.T) { t.Fatalf("get CR: %v", err) } - if len(updated.Status.AssignedInstances) != 2 { - t.Errorf("AssignedInstances len = %d, want 2", len(updated.Status.AssignedInstances)) + if updated.Status.Billing == nil { + t.Fatal("Billing is nil") + } + if len(updated.Status.Billing.AssignedInstances) != 2 { + t.Errorf("AssignedInstances len = %d, want 2", len(updated.Status.Billing.AssignedInstances)) } - if updated.Status.LastUsageReconcileAt == nil { - t.Errorf("LastUsageReconcileAt not set") + if updated.Status.Billing.LastReconcileAt == nil { + t.Errorf("LastReconcileAt not set") } - if updated.Status.UsageObservedGeneration == nil || *updated.Status.UsageObservedGeneration != 1 { - t.Errorf("UsageObservedGeneration = %v, want 1", updated.Status.UsageObservedGeneration) + if updated.Status.Billing.ObservedGeneration == nil || *updated.Status.Billing.ObservedGeneration != 1 { + t.Errorf("ObservedGeneration = %v, want 1", updated.Status.Billing.ObservedGeneration) } - if _, ok := updated.Status.UsedResources["memory"]; !ok { + if _, ok := updated.Status.Billing.UsedResources[string(hv1.ResourceMemory)]; !ok { t.Errorf("UsedResources[memory] not set") } - if _, ok := updated.Status.UsedResources["cpu"]; !ok { + if _, ok := updated.Status.Billing.UsedResources[string(hv1.ResourceCPU)]; !ok { t.Errorf("UsedResources[cpu] not set") } }) diff --git a/internal/scheduling/reservations/failover/config.go b/internal/scheduling/reservations/failover/config.go index b8bde49a6..3c2f9d3f1 100644 --- a/internal/scheduling/reservations/failover/config.go +++ b/internal/scheduling/reservations/failover/config.go @@ -27,10 +27,6 @@ type FailoverConfig struct { // This is used to read VM data from the Nova database. DatasourceName string `json:"datasourceName"` - // SchedulerURL is the URL of the nova external scheduler API. - // Example: "http://localhost:8080/scheduler/nova/external" - SchedulerURL string `json:"schedulerURL"` - // MaxVMsToProcess limits the number of VMs to process per reconciliation cycle. // Set to negative to process all VMs (default behavior). // Useful for debugging and testing with large VM counts. @@ -99,9 +95,6 @@ func (c *FailoverConfig) ApplyDefaults() { if c.DatasourceName == "" { c.DatasourceName = defaults.DatasourceName } - if c.SchedulerURL == "" { - c.SchedulerURL = defaults.SchedulerURL - } if c.ReconcileInterval.Duration == 0 { c.ReconcileInterval = defaults.ReconcileInterval } @@ -142,7 +135,6 @@ func DefaultConfig() FailoverConfig { MaxVMsToProcess: 30, Creator: "cortex-failover-controller", DatasourceName: "nova-servers", // we have the server and flavor data source (both store in same postgres and same secret but still) - SchedulerURL: "http://localhost:8080/scheduler/nova/external", TrustHypervisorLocation: false, RevalidationInterval: metav1.Duration{Duration: 30 * time.Minute}, LimitOneNewReservationPerHypervisor: true, diff --git a/internal/scheduling/reservations/failover/controller.go b/internal/scheduling/reservations/failover/controller.go index 213708245..25597a260 100644 --- a/internal/scheduling/reservations/failover/controller.go +++ b/internal/scheduling/reservations/failover/controller.go @@ -13,7 +13,8 @@ import ( "time" "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + knowledge "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" "github.com/cobaltcore-dev/cortex/pkg/multicluster" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" @@ -40,21 +41,21 @@ var log = ctrl.Log.WithName("failover-reservation-controller").WithValues("modul // 2. Watch-based per-reservation reconciliation (Reconcile) - handles acknowledgment and validation of individual reservations type FailoverReservationController struct { client.Client - VMSource reservations.VMSource - Config FailoverConfig - SchedulerClient *reservations.SchedulerClient - Recorder events.EventRecorder // Event recorder for emitting Kubernetes events - Monitor *FailoverMonitor - reconcileCount int64 // Track reconciliation count for rotating VM selection + VMSource reservations.VMSource + Config FailoverConfig + Scheduler *sched.Placement + Recorder events.EventRecorder // Event recorder for emitting Kubernetes events + Monitor *FailoverMonitor + reconcileCount int64 // Track reconciliation count for rotating VM selection } -func NewFailoverReservationController(c client.Client, vmSource reservations.VMSource, config FailoverConfig, schedulerClient *reservations.SchedulerClient, monitor *FailoverMonitor) *FailoverReservationController { +func NewFailoverReservationController(c client.Client, vmSource reservations.VMSource, config FailoverConfig, scheduler *sched.Placement, monitor *FailoverMonitor) *FailoverReservationController { return &FailoverReservationController{ - Client: c, - VMSource: vmSource, - Config: config, - SchedulerClient: schedulerClient, - Monitor: monitor, + Client: c, + VMSource: vmSource, + Config: config, + Scheduler: scheduler, + Monitor: monitor, } } @@ -344,10 +345,9 @@ func (c *FailoverReservationController) ReconcilePeriodic(ctx context.Context) ( summary.reservationsDeleted = len(emptyReservationsToDelete) // 6. Fetch flavor groups for reservation sizing (if configured) - var flavorGroups map[string]compute.FlavorGroupFeature + var flavorGroups map[string]knowledge.FlavorGroupFeature if c.Config.UseFlavorGroupResources { - knowledge := &reservations.FlavorGroupKnowledgeClient{Client: c.Client} - fg, err := knowledge.GetAllFlavorGroups(ctx, nil) + fg, err := knowledge.GetAllFlavorGroups(ctx, c.Client, nil) if err != nil { logger.Info("flavor group knowledge not available, will fall back to VM resources for sizing", "error", err) @@ -614,10 +614,10 @@ func (c *FailoverReservationController) selectVMsToProcess( func sortVMsByMemory(vms []vmFailoverNeed) { sort.Slice(vms, func(i, j int) bool { var memI, memJ int64 - if mem, ok := vms[i].VM.Resources["memory"]; ok { + if mem, ok := vms[i].VM.Resources[string(hv1.ResourceMemory)]; ok { memI = mem.Value() } - if mem, ok := vms[j].VM.Resources["memory"]; ok { + if mem, ok := vms[j].VM.Resources[string(hv1.ResourceMemory)]; ok { memJ = mem.Value() } return memI > memJ @@ -630,7 +630,7 @@ func (c *FailoverReservationController) reconcileCreateAndAssignReservations( vms []reservations.VM, failoverReservations []v1alpha1.Reservation, allHypervisors []string, - flavorGroups map[string]compute.FlavorGroupFeature, // passed to resolveVMForScheduling per-VM + flavorGroups map[string]knowledge.FlavorGroupFeature, // passed to resolveVMForScheduling per-VM ) (reconcileSummary, bool) { logger := LoggerFromContext(ctx) @@ -881,7 +881,6 @@ func (c *FailoverReservationController) Start(ctx context.Context) error { "maxFailuresForShortInterval", c.Config.MaxFailuresForShortInterval, "creator", c.Config.Creator, "datasourceName", c.Config.DatasourceName, - "schedulerURL", c.Config.SchedulerURL, "flavorFailoverRequirements", c.Config.FlavorFailoverRequirements, "maxVMsToProcess", c.Config.MaxVMsToProcess, "trustHypervisorLocation", c.Config.TrustHypervisorLocation, diff --git a/internal/scheduling/reservations/failover/controller_test.go b/internal/scheduling/reservations/failover/controller_test.go index 55a09cc03..307cb7a01 100644 --- a/internal/scheduling/reservations/failover/controller_test.go +++ b/internal/scheduling/reservations/failover/controller_test.go @@ -739,8 +739,8 @@ func newTestVMWithResources(uuid, currentHypervisor string) reservations.VM { FlavorName: "m1.large", ProjectID: "test-project", Resources: map[string]resource.Quantity{ - "memory": *resource.NewQuantity(8192*1024*1024, resource.BinarySI), - "vcpus": *resource.NewQuantity(4, resource.DecimalSI), + string(hv1.ResourceMemory): *resource.NewQuantity(8192*1024*1024, resource.BinarySI), + string(hv1.ResourceCPU): *resource.NewQuantity(4, resource.DecimalSI), }, } } @@ -771,8 +771,8 @@ func newTestReservationWithResources(name, host string, allocations map[string]s Type: v1alpha1.ReservationTypeFailover, TargetHost: host, Resources: map[hv1.ResourceName]resource.Quantity{ - "memory": *resource.NewQuantity(8192*1024*1024, resource.BinarySI), - "cpu": *resource.NewQuantity(4, resource.DecimalSI), + hv1.ResourceMemory: *resource.NewQuantity(8192*1024*1024, resource.BinarySI), + hv1.ResourceCPU: *resource.NewQuantity(4, resource.DecimalSI), }, }, Status: v1alpha1.ReservationStatus{ @@ -805,7 +805,7 @@ func TestSelectVMsToProcess(t *testing.T) { UUID: "vm-" + string(rune('a'+i)), CurrentHypervisor: "host" + string(rune('1'+i)), Resources: map[string]resource.Quantity{ - "memory": *resource.NewQuantity(int64((count-i)*1024*1024*1024), resource.BinarySI), // Descending memory + string(hv1.ResourceMemory): *resource.NewQuantity(int64((count-i)*1024*1024*1024), resource.BinarySI), // Descending memory }, }, Count: 1, diff --git a/internal/scheduling/reservations/failover/helpers.go b/internal/scheduling/reservations/failover/helpers.go index 862674f95..b14a14da4 100644 --- a/internal/scheduling/reservations/failover/helpers.go +++ b/internal/scheduling/reservations/failover/helpers.go @@ -10,7 +10,7 @@ import ( "strings" "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + knowledge "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "k8s.io/apimachinery/pkg/api/resource" @@ -79,13 +79,13 @@ func resolveVMSpecForScheduling( ctx context.Context, vm reservations.VM, useFlavorGroupResources bool, - flavorGroups map[string]compute.FlavorGroupFeature, + flavorGroups map[string]knowledge.FlavorGroupFeature, ) resolvedReservationSpec { logger := LoggerFromContext(ctx) if useFlavorGroupResources && flavorGroups != nil { - groupName, _, err := reservations.FindFlavorInGroups(vm.FlavorName, flavorGroups) + groupName, _, err := knowledge.FindFlavorInGroups(vm.FlavorName, flavorGroups) if err == nil { fg := flavorGroups[groupName] largest := fg.LargestFlavor @@ -109,10 +109,10 @@ func resolveVMSpecForScheduling( // Fallback: use VM's own resources var memoryMB, vcpus uint64 - if memory, ok := vm.Resources["memory"]; ok { + if memory, ok := vm.Resources[string(hv1.ResourceMemory)]; ok { memoryMB = uint64(memory.Value() / (1024 * 1024)) //nolint:gosec // memory values won't overflow; binary MiB matches commitments/state.go and vm_source.go } - if v, ok := vm.Resources["vcpus"]; ok { + if v, ok := vm.Resources[string(hv1.ResourceCPU)]; ok { vcpus = uint64(v.Value()) //nolint:gosec // vcpus values won't overflow } return resolvedReservationSpec{ @@ -187,7 +187,7 @@ func ValidateFailoverReservationResources(res *v1alpha1.Reservation) error { return nil // No resources is valid (will be caught elsewhere if needed) } - allowedKeys := map[hv1.ResourceName]bool{"cpu": true, "memory": true} + allowedKeys := map[hv1.ResourceName]bool{hv1.ResourceCPU: true, hv1.ResourceMemory: true} for key := range res.Spec.Resources { if !allowedKeys[key] { return fmt.Errorf("invalid resource key %q: only 'cpu' and 'memory' are allowed", key) diff --git a/internal/scheduling/reservations/failover/integration_test.go b/internal/scheduling/reservations/failover/integration_test.go index fcbdaa7c2..04f9f92e7 100644 --- a/internal/scheduling/reservations/failover/integration_test.go +++ b/internal/scheduling/reservations/failover/integration_test.go @@ -18,6 +18,7 @@ import ( novaapi "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" "github.com/cobaltcore-dev/cortex/internal/scheduling/nova" "github.com/cobaltcore-dev/cortex/internal/scheduling/nova/plugins/filters" @@ -32,6 +33,14 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client/fake" ) +// passthroughGatherer is a nova.CandidateGatherer that leaves the request's +// preselected hosts unchanged (the tests seed their own eligible hosts). +type passthroughGatherer struct{} + +func (passthroughGatherer) MutateWithAllCandidates(context.Context, *compute.PlacementRequest) error { + return nil +} + // ============================================================================ // Integration Tests // ============================================================================ @@ -639,10 +648,10 @@ func (env *IntegrationTestEnv) LogStateSummary() { for _, vm := range vmsByHypervisor[hypervisorName] { memoryMB := int64(0) vcpus := int64(0) - if mem, ok := vm.Resources["memory"]; ok { + if mem, ok := vm.Resources[string(hv1.ResourceMemory)]; ok { memoryMB = mem.Value() / (1024 * 1024) } - if cpu, ok := vm.Resources["vcpus"]; ok { + if cpu, ok := vm.Resources[string(hv1.ResourceCPU)]; ok { vcpus = cpu.Value() } vmParts = append(vmParts, fmt.Sprintf("%s(%dMB,%dCPU)", vm.UUID, memoryMB, vcpus)) @@ -695,7 +704,9 @@ func (env *IntegrationTestEnv) LogStateSummary() { func (env *IntegrationTestEnv) TriggerFailoverReconcile(flavorRequirements map[string]int) error { env.T.Helper() - schedulerClient := reservations.NewSchedulerClient(env.SchedulerBaseURL + "/scheduler/nova/external") + // Drive the same in-process pipeline the testHTTPAPI uses, so the failover + // reservation flow exercises the real scheduler over the env's nova controller. + scheduler := compute.NewPlacement(env.NovaController, passthroughGatherer{}) config := FailoverConfig{ ReconcileInterval: metav1.Duration{Duration: time.Minute}, @@ -708,7 +719,7 @@ func (env *IntegrationTestEnv) TriggerFailoverReconcile(flavorRequirements map[s env.K8sClient, env.VMSource, config, - schedulerClient, + scheduler, nil, ) @@ -1142,9 +1153,9 @@ func newIntegrationTestEnv(t *testing.T, vms []reservations.VM, hypervisors []*h Build() novaController := &nova.FilterWeigherPipelineController{ - BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[novaapi.ExternalSchedulerRequest]]{ + BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[compute.PlacementRequest]]{ Client: k8sClient, - Pipelines: make(map[string]lib.FilterWeigherPipeline[novaapi.ExternalSchedulerRequest]), + Pipelines: make(map[string]lib.FilterWeigherPipeline[compute.PlacementRequest]), PipelineConfigs: make(map[string]v1alpha1.Pipeline), }, Monitor: getSharedMonitor(), @@ -1223,7 +1234,7 @@ func newIntegrationTestEnv(t *testing.T, vms []reservations.VM, hypervisors []*h novaController.PipelineConfigs[pipeline.Name] = pipeline } - api := &testHTTPAPI{delegate: novaController} + api := &testHTTPAPI{scheduler: compute.NewPlacement(novaController, passthroughGatherer{})} mux := http.NewServeMux() mux.HandleFunc("/scheduler/nova/external", api.NovaExternalScheduler) server := httptest.NewServer(mux) @@ -1242,7 +1253,7 @@ func newIntegrationTestEnv(t *testing.T, vms []reservations.VM, hypervisors []*h // testHTTPAPI is a simplified HTTP API for testing that delegates to the controller. // It does NOT shuffle evacuation hosts, ensuring deterministic test results. type testHTTPAPI struct { - delegate nova.HTTPAPIDelegate + scheduler *compute.Placement } // NovaExternalScheduler handles the POST request from the Nova scheduler. @@ -1260,49 +1271,26 @@ func (api *testHTTPAPI) NovaExternalScheduler(w http.ResponseWriter, r *http.Req return } - rawBytes, err := json.Marshal(requestData) - if err != nil { - http.Error(w, "failed to marshal request", http.StatusInternalServerError) - return - } - raw := runtime.RawExtension{Raw: rawBytes} - pipelineName := requestData.Pipeline if pipelineName == "" { pipelineName = "test-filter-pipeline" } - decision := &v1alpha1.Decision{ - TypeMeta: metav1.TypeMeta{ - Kind: "Decision", - APIVersion: "cortex.cloud/v1alpha1", - }, - ObjectMeta: metav1.ObjectMeta{ - GenerateName: "nova-", - }, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - PipelineRef: corev1.ObjectReference{ - Name: pipelineName, - }, - ResourceID: requestData.Spec.Data.InstanceUUID, - NovaRaw: &raw, - }, + // Resolve the scheduling intent from the request hints, mirroring the + // production handler (external_scheduler_api.go). + intent, err := requestData.GetIntent() + if err != nil { + intent = novaapi.CreateIntent } - ctx := r.Context() - if err := api.delegate.ProcessNewDecisionFromAPI(ctx, decision); err != nil { + req := nova.ToSchedulerRequest(requestData, intent, pipelineName) + out, err := api.scheduler.Run(r.Context(), req, nil, nil) + if err != nil { http.Error(w, "failed to process scheduling decision", http.StatusInternalServerError) return } - if decision.Status.Result == nil { - http.Error(w, "decision didn't produce a result", http.StatusInternalServerError) - return - } - - hosts := decision.Status.Result.OrderedHosts - response := novaapi.ExternalSchedulerResponse{Hosts: hosts} + response := novaapi.ExternalSchedulerResponse{Hosts: out.OrderedHosts} w.Header().Set("Content-Type", "application/json") if err := json.NewEncoder(w).Encode(response); err != nil { http.Error(w, "failed to encode response", http.StatusInternalServerError) @@ -1342,9 +1330,9 @@ func newIntegrationTestEnvWithTraitsFilter(t *testing.T, vms []reservations.VM, Build() novaController := &nova.FilterWeigherPipelineController{ - BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[novaapi.ExternalSchedulerRequest]]{ + BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[compute.PlacementRequest]]{ Client: k8sClient, - Pipelines: make(map[string]lib.FilterWeigherPipeline[novaapi.ExternalSchedulerRequest]), + Pipelines: make(map[string]lib.FilterWeigherPipeline[compute.PlacementRequest]), PipelineConfigs: make(map[string]v1alpha1.Pipeline), }, Monitor: getSharedMonitor(), @@ -1424,7 +1412,7 @@ func newIntegrationTestEnvWithTraitsFilter(t *testing.T, vms []reservations.VM, novaController.PipelineConfigs[pipeline.Name] = pipeline } - api := &testHTTPAPI{delegate: novaController} + api := &testHTTPAPI{scheduler: compute.NewPlacement(novaController, passthroughGatherer{})} mux := http.NewServeMux() mux.HandleFunc("/scheduler/nova/external", api.NovaExternalScheduler) server := httptest.NewServer(mux) @@ -1460,8 +1448,8 @@ func newHypervisorWithAZ(name string, cpuCap, memoryGi, cpuAlloc, memoryGiAlloc labels[corev1.LabelTopologyZone] = az } capacity := map[hv1.ResourceName]resource.Quantity{ - "cpu": resource.MustParse(strconv.Itoa(cpuCap)), - "memory": resource.MustParse(strconv.Itoa(memoryGi) + "Gi"), + hv1.ResourceCPU: resource.MustParse(strconv.Itoa(cpuCap)), + hv1.ResourceMemory: resource.MustParse(strconv.Itoa(memoryGi) + "Gi"), } return &hv1.Hypervisor{ ObjectMeta: metav1.ObjectMeta{ @@ -1470,14 +1458,14 @@ func newHypervisorWithAZ(name string, cpuCap, memoryGi, cpuAlloc, memoryGiAlloc }, Spec: hv1.HypervisorSpec{ Overcommit: map[hv1.ResourceName]float64{ - "cpu": 1.0, - "memory": 1.0, + hv1.ResourceCPU: 1.0, + hv1.ResourceMemory: 1.0, }, }, Status: hv1.HypervisorStatus{ Capacity: capacity, EffectiveCapacity: capacity, - Allocation: map[hv1.ResourceName]resource.Quantity{"cpu": resource.MustParse(strconv.Itoa(cpuAlloc)), "memory": resource.MustParse(strconv.Itoa(memoryGiAlloc) + "Gi")}, + Allocation: map[hv1.ResourceName]resource.Quantity{hv1.ResourceCPU: resource.MustParse(strconv.Itoa(cpuAlloc)), hv1.ResourceMemory: resource.MustParse(strconv.Itoa(memoryGiAlloc) + "Gi")}, NumInstances: len(instances), Instances: instances, Traits: traits, @@ -1499,8 +1487,8 @@ func newReservation(name, host string, memoryMB, vcpus uint64, allocations map[s Type: v1alpha1.ReservationTypeFailover, TargetHost: host, Resources: map[hv1.ResourceName]resource.Quantity{ - "memory": resource.MustParse(strconv.FormatUint(memoryMB, 10) + "Mi"), - "cpu": resource.MustParse(strconv.FormatUint(vcpus, 10)), + hv1.ResourceMemory: resource.MustParse(strconv.FormatUint(memoryMB, 10) + "Mi"), + hv1.ResourceCPU: resource.MustParse(strconv.FormatUint(vcpus, 10)), }, FailoverReservation: &v1alpha1.FailoverReservationSpec{ ResourceGroup: "m1.large", @@ -1537,8 +1525,8 @@ func newVMWithAZ(uuid, flavorName, projectID, host string, memoryMB, vcpus uint6 CurrentHypervisor: host, AvailabilityZone: az, Resources: map[string]resource.Quantity{ - "memory": resource.MustParse(strconv.FormatUint(memoryMB, 10) + "Mi"), - "vcpus": resource.MustParse(strconv.FormatUint(vcpus, 10)), + string(hv1.ResourceMemory): resource.MustParse(strconv.FormatUint(memoryMB, 10) + "Mi"), + string(hv1.ResourceCPU): resource.MustParse(strconv.FormatUint(vcpus, 10)), }, FlavorExtraSpecs: make(map[string]string), } diff --git a/internal/scheduling/reservations/failover/reservation_eligibility.go b/internal/scheduling/reservations/failover/reservation_eligibility.go index abcaa636c..5c30d865f 100644 --- a/internal/scheduling/reservations/failover/reservation_eligibility.go +++ b/internal/scheduling/reservations/failover/reservation_eligibility.go @@ -291,7 +291,7 @@ func (g *DependencyGraph) isVMEligibleForReservation(candidateResName string) bo // doesVMFitInReservation checks if a VM's resources fit within a reservation's resources. func doesVMFitInReservation(vm reservations.VM, reservation v1alpha1.Reservation) bool { - if vmMemory, ok := vm.Resources["memory"]; ok { + if vmMemory, ok := vm.Resources[string(hv1.ResourceMemory)]; ok { if resMemory, ok := reservation.Spec.Resources[hv1.ResourceMemory]; ok { if vmMemory.Cmp(resMemory) > 0 { return false @@ -301,7 +301,7 @@ func doesVMFitInReservation(vm reservations.VM, reservation v1alpha1.Reservation } } - if vmVCPUs, ok := vm.Resources["vcpus"]; ok { + if vmVCPUs, ok := vm.Resources[string(hv1.ResourceCPU)]; ok { if resCPU, ok := reservation.Spec.Resources[hv1.ResourceCPU]; ok { if vmVCPUs.Cmp(resCPU) > 0 { return false diff --git a/internal/scheduling/reservations/failover/reservation_eligibility_test.go b/internal/scheduling/reservations/failover/reservation_eligibility_test.go index d66d3322e..e31ea892e 100644 --- a/internal/scheduling/reservations/failover/reservation_eligibility_test.go +++ b/internal/scheduling/reservations/failover/reservation_eligibility_test.go @@ -14,15 +14,15 @@ import ( ) // Default resources for test VMs and reservations (4GB memory, 2 vcpus) -// Note: Reservations use "cpu" as the canonical key, VMs use "vcpus" +// Note: Both reservations and VMs use "cpu" as the canonical CPU resource key. var defaultResources = map[hv1.ResourceName]resource.Quantity{ - "memory": resource.MustParse("4Gi"), - "cpu": resource.MustParse("2"), + hv1.ResourceMemory: resource.MustParse("4Gi"), + hv1.ResourceCPU: resource.MustParse("2"), } var defaultVMResources = map[string]resource.Quantity{ - "memory": resource.MustParse("4Gi"), - "vcpus": resource.MustParse("2"), + string(hv1.ResourceMemory): resource.MustParse("4Gi"), + string(hv1.ResourceCPU): resource.MustParse("2"), } // makeReservation creates a test reservation with the given parameters. @@ -622,48 +622,48 @@ func TestDoesVMFitInReservation(t *testing.T) { { name: "fits: VM fits exactly in reservation", vm: makeVMWithResources("vm-1", "host1", map[string]resource.Quantity{ - "memory": resource.MustParse("4Gi"), - "vcpus": resource.MustParse("2"), + string(hv1.ResourceMemory): resource.MustParse("4Gi"), + string(hv1.ResourceCPU): resource.MustParse("2"), }), reservation: makeReservationWithResources("res-1", "host2", map[string]string{}, map[hv1.ResourceName]resource.Quantity{ - "memory": resource.MustParse("4Gi"), - "cpu": resource.MustParse("2"), + hv1.ResourceMemory: resource.MustParse("4Gi"), + hv1.ResourceCPU: resource.MustParse("2"), }), expected: true, }, { name: "fits: VM is smaller than reservation", vm: makeVMWithResources("vm-1", "host1", map[string]resource.Quantity{ - "memory": resource.MustParse("2Gi"), - "vcpus": resource.MustParse("1"), + string(hv1.ResourceMemory): resource.MustParse("2Gi"), + string(hv1.ResourceCPU): resource.MustParse("1"), }), reservation: makeReservationWithResources("res-1", "host2", map[string]string{}, map[hv1.ResourceName]resource.Quantity{ - "memory": resource.MustParse("4Gi"), - "cpu": resource.MustParse("2"), + hv1.ResourceMemory: resource.MustParse("4Gi"), + hv1.ResourceCPU: resource.MustParse("2"), }), expected: true, }, { name: "exceeds: VM memory exceeds reservation", vm: makeVMWithResources("vm-1", "host1", map[string]resource.Quantity{ - "memory": resource.MustParse("8Gi"), - "vcpus": resource.MustParse("2"), + string(hv1.ResourceMemory): resource.MustParse("8Gi"), + string(hv1.ResourceCPU): resource.MustParse("2"), }), reservation: makeReservationWithResources("res-1", "host2", map[string]string{}, map[hv1.ResourceName]resource.Quantity{ - "memory": resource.MustParse("4Gi"), - "cpu": resource.MustParse("2"), + hv1.ResourceMemory: resource.MustParse("4Gi"), + hv1.ResourceCPU: resource.MustParse("2"), }), expected: false, }, { name: "exceeds: VM vcpus exceeds reservation cpu", vm: makeVMWithResources("vm-1", "host1", map[string]resource.Quantity{ - "memory": resource.MustParse("4Gi"), - "vcpus": resource.MustParse("4"), + string(hv1.ResourceMemory): resource.MustParse("4Gi"), + string(hv1.ResourceCPU): resource.MustParse("4"), }), reservation: makeReservationWithResources("res-1", "host2", map[string]string{}, map[hv1.ResourceName]resource.Quantity{ - "memory": resource.MustParse("4Gi"), - "cpu": resource.MustParse("2"), + hv1.ResourceMemory: resource.MustParse("4Gi"), + hv1.ResourceCPU: resource.MustParse("2"), }), expected: false, }, @@ -671,30 +671,30 @@ func TestDoesVMFitInReservation(t *testing.T) { name: "fits: VM has no resources defined", vm: makeVMWithResources("vm-1", "host1", map[string]resource.Quantity{}), reservation: makeReservationWithResources("res-1", "host2", map[string]string{}, map[hv1.ResourceName]resource.Quantity{ - "memory": resource.MustParse("4Gi"), - "cpu": resource.MustParse("2"), + hv1.ResourceMemory: resource.MustParse("4Gi"), + hv1.ResourceCPU: resource.MustParse("2"), }), expected: true, }, { name: "exceeds: reservation has no memory resource", vm: makeVMWithResources("vm-1", "host1", map[string]resource.Quantity{ - "memory": resource.MustParse("4Gi"), - "vcpus": resource.MustParse("2"), + string(hv1.ResourceMemory): resource.MustParse("4Gi"), + string(hv1.ResourceCPU): resource.MustParse("2"), }), reservation: makeReservationWithResources("res-1", "host2", map[string]string{}, map[hv1.ResourceName]resource.Quantity{ - "cpu": resource.MustParse("2"), + hv1.ResourceCPU: resource.MustParse("2"), }), expected: false, }, { name: "exceeds: reservation has no cpu resource", vm: makeVMWithResources("vm-1", "host1", map[string]resource.Quantity{ - "memory": resource.MustParse("4Gi"), - "vcpus": resource.MustParse("2"), + string(hv1.ResourceMemory): resource.MustParse("4Gi"), + string(hv1.ResourceCPU): resource.MustParse("2"), }), reservation: makeReservationWithResources("res-1", "host2", map[string]string{}, map[hv1.ResourceName]resource.Quantity{ - "memory": resource.MustParse("4Gi"), + hv1.ResourceMemory: resource.MustParse("4Gi"), }), expected: false, }, diff --git a/internal/scheduling/reservations/failover/reservation_scheduling.go b/internal/scheduling/reservations/failover/reservation_scheduling.go index 2cae7105a..22b9c7a7d 100644 --- a/internal/scheduling/reservations/failover/reservation_scheduling.go +++ b/internal/scheduling/reservations/failover/reservation_scheduling.go @@ -8,12 +8,13 @@ import ( "fmt" "slices" "sort" - "strings" api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/scheduling" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" + hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" ) // Pipeline names for failover reservation scheduling @@ -30,47 +31,17 @@ var DefaultFailoverOptions = scheduling.Options{ SkipCommittedResourceTracking: true, } -// inferFailoverPipeline returns the standard pipeline for a failover scheduling call based on +// InferFailoverPipeline returns the standard pipeline for a failover scheduling call based on // the VM's flavor extra specs — the same HANA vs general-purpose split used by Nova placement. -func inferFailoverPipeline(extraSpecs map[string]string) string { - if strings.ToLower(extraSpecs["trait:CUSTOM_HANA_EXCLUSIVE_HOST"]) == "required" { +// An unparseable flavor type falls back to the general-purpose pipeline. +func InferFailoverPipeline(extraSpecs map[string]string) string { + flavorType, err := api.FlavorTypeFromExtraSpecs(extraSpecs) + if err == nil && flavorType == api.FlavorTypeHANA { return "kvm-hana-bin-packing" } return "kvm-general-purpose-load-balancing" } -// imagePropertiesFromFlavor synthesizes Glance-style image properties from a flavor's -// capabilities:hypervisor_type extra spec. The failover controller does not have access -// to the real Glance image metadata of the VM, but for reservation scheduling the VM -// already exists and has passed FilterImagePropertiesStep at original create time, so -// this synthesis is safe: it exists only to keep the filter from emitting spurious -// "image_properties_hv_type_undetermined" events when the failover controller reuses -// the EvacuateIntent to validate reservation hosts. -// -// The extra-spec values below duplicate the canonical mapping in -// nova.Flavor.GetHypervisorType() (see internal/knowledge/datasources/plugins/openstack/nova/nova_types.go); -// keep the two in sync. Real observed fixtures also carry lowercase "qemu"/"ch", so -// those are accepted here too. -// -// Returns nil for unknown/unspecified hypervisor types, which preserves the filter's -// existing tolerant fallback (log a warning event, keep all hosts). -// -// TODO(vm-crd): remove this and populate image properties from real Glance metadata -// once the VM CRD carries them. See also the TODO on validateVMViaSchedulerEvacuation. -func imagePropertiesFromFlavor(flavorExtraSpecs map[string]string) map[string]any { - switch flavorExtraSpecs["capabilities:hypervisor_type"] { - case "QEMU", "CH", "qemu", "ch": - // KVM family; the kvm-flavored key is "img_hv_type". - return map[string]any{"img_hv_type": string(api.NovaImageMetaHVTypeKVM)} - case "VMware vCenter Server": - return map[string]any{"hypervisor_type": string(api.NovaImageMetaHVTypeVMware)} - case "Ironic": - return map[string]any{"hypervisor_type": string(api.NovaImageMetaHVTypeBaremetal)} - default: - return nil - } -} - func (c *FailoverReservationController) queryHypervisorsFromScheduler(ctx context.Context, vm reservations.VM, allHypervisors []string, pipeline string, resSpec resolvedReservationSpec, intent v1alpha1.SchedulingIntent, opts scheduling.Options) ([]string, error) { logger := LoggerFromContext(ctx) @@ -93,38 +64,40 @@ func (c *FailoverReservationController) queryHypervisorsFromScheduler(ctx contex ignoreHypervisors := []string{vm.CurrentHypervisor} - // Build flavor extra specs from VM's extra specs - // Start with the VM's actual extra specs, then ensure required defaults are set - flavorExtraSpecs := make(map[string]string) - for k, v := range vm.FlavorExtraSpecs { - flavorExtraSpecs[k] = v - } - // Ensure hypervisor_type is set for KVM scheduling if not already present - if _, ok := flavorExtraSpecs["capabilities:hypervisor_type"]; !ok { - flavorExtraSpecs["capabilities:hypervisor_type"] = "qemu" + // Build the typed scheduler request directly (in-process; no HTTP). KVM + // failover reservations gather all candidates; the eligible list is a + // starting set. The filter_correct_az filter excludes hosts outside the VM's + // AZ. ImageHypervisorType is irrelevant here: filter_image_properties skips + // the failover/reuse reservation intents. + hosts := make([]compute.Host, 0, len(eligibleHypervisors)) + weights := make(map[string]float64, len(eligibleHypervisors)) + for _, h := range eligibleHypervisors { + hosts = append(hosts, compute.Host{ComputeHost: h.ComputeHost, HypervisorHostname: h.HypervisorHostname}) + weights[h.ComputeHost] = 0.0 } - - // Schedule the reservation using the SchedulerClient. - // Note: We pass all hypervisors (from all AZs) in EligibleHosts. The scheduler pipeline's - // filter_correct_az filter will exclude hosts that are not in the VM's availability zone. - // Use resSpec.FlavorName and reservation spec resources so the scheduler checks capacity for the - // correct flavor size (which may be the LargestFlavor from the flavor group). - scheduleReq := reservations.ScheduleReservationRequest{ - InstanceUUID: vm.UUID, - ProjectID: vm.ProjectID, - FlavorName: resSpec.FlavorName, - FlavorExtraSpecs: flavorExtraSpecs, - MemoryMB: resSpec.MemoryMB, - VCPUs: resSpec.VCPUs, - EligibleHosts: eligibleHypervisors, - IgnoreHosts: ignoreHypervisors, - Pipeline: pipeline, - AvailabilityZone: vm.AvailabilityZone, - SchedulerHints: map[string]any{ - "_nova_check_type": string(intent), - api.HintKeyResourceGroup: resSpec.ResourceGroup(vm.FlavorName), + requiredTraits, forbiddenTraits := api.SplitTraits(vm.FlavorExtraSpecs) + schedReq := compute.PlacementRequest{ + Pipeline: pipeline, + GatherAllCandidates: true, + HypervisorFamily: api.HypervisorFamilyKVM, + Hosts: hosts, + Weights: weights, + Options: opts, + Intent: intent, + InstanceUUID: vm.UUID, + NumInstances: 1, // one reservation per VM; NumInstances must be >= 1 or capacity/bin-pack math treats the host as free + ProjectID: vm.ProjectID, + AvailabilityZone: vm.AvailabilityZone, + IgnoreHosts: ignoreHypervisors, + ResourceGroup: resSpec.ResourceGroup(vm.FlavorName), + RequiredTraits: requiredTraits, + ForbiddenTraits: forbiddenTraits, + RequestedCapabilities: api.HypervisorTypeCapability(vm.FlavorExtraSpecs, true), + AcceleratorsRequested: api.RequestsAccelerators(vm.FlavorExtraSpecs), + Flavor: api.FlavorSpec{ + VCPUs: resSpec.VCPUs, + MemoryMB: resSpec.MemoryMB, }, - ImageProperties: imagePropertiesFromFlavor(flavorExtraSpecs), } logger.V(1).Info("scheduling failover reservation", @@ -133,7 +106,7 @@ func (c *FailoverReservationController) queryHypervisorsFromScheduler(ctx contex "eligibleHypervisors", len(eligibleHypervisors), "ignoreHypervisors", ignoreHypervisors) - scheduleResp, err := c.SchedulerClient.ScheduleReservation(ctx, scheduleReq, opts) + out, err := c.runScheduler(ctx, schedReq) if err != nil { logger.Error(err, "failed to schedule failover reservation", "vmUUID", vm.UUID, "pipeline", pipeline) return nil, fmt.Errorf("failed to schedule failover reservation: %w", err) @@ -144,9 +117,19 @@ func (c *FailoverReservationController) queryHypervisorsFromScheduler(ctx contex "pipeline", pipeline, "eligibleHypervisors", len(eligibleHypervisors), "ignoreHypervisors", ignoreHypervisors, - "returnedHypervisors", scheduleResp.Hosts) + "returnedHypervisors", out.OrderedHosts) + + return out.OrderedHosts, nil +} - return scheduleResp.Hosts, nil +// runScheduler runs the shared scheduler, choosing the lock-free path for +// read-only runs (e.g. reuse/evacuate compatibility checks) and the locked path +// for runs that mutate shared reservation state. +func (c *FailoverReservationController) runScheduler(ctx context.Context, req compute.PlacementRequest) (compute.PlacementResult, error) { + if req.Options.ReadOnly { + return c.Scheduler.RunWithoutLock(ctx, req, nil, nil) + } + return c.Scheduler.Run(ctx, req, nil, nil) } // tryReuseExistingReservation finds an existing reservation that can be reused for a VM. @@ -165,7 +148,7 @@ func (c *FailoverReservationController) tryReuseExistingReservation( reuseOpts := DefaultFailoverOptions reuseOpts.ReadOnly = true - validHypervisors, err := c.queryHypervisorsFromScheduler(ctx, vm, allHypervisors, inferFailoverPipeline(vm.FlavorExtraSpecs), resSpec, api.ReuseFailoverReservationIntent, reuseOpts) + validHypervisors, err := c.queryHypervisorsFromScheduler(ctx, vm, allHypervisors, InferFailoverPipeline(vm.FlavorExtraSpecs), resSpec, api.ReuseFailoverReservationIntent, reuseOpts) if err != nil { logger.Error(err, "failed to get potential hypervisors for VM", "vmUUID", vm.UUID) return nil @@ -225,14 +208,14 @@ func (c *FailoverReservationController) validateVMViaSchedulerEvacuation( // Get memory and vcpus from VM resources var memoryMB uint64 var vcpus uint64 - if memory, ok := vm.Resources["memory"]; ok { + if memory, ok := vm.Resources[string(hv1.ResourceMemory)]; ok { memoryMB = uint64(memory.Value() / (1024 * 1024)) //nolint:gosec // memory values won't overflow } - if vcpusRes, ok := vm.Resources["vcpus"]; ok { + if vcpusRes, ok := vm.Resources[string(hv1.ResourceCPU)]; ok { vcpus = uint64(vcpusRes.Value()) //nolint:gosec // vcpus values won't overflow } - // Build flavor extra specs from VM's extra specs + // Infer the pipeline from the VM's flavor extra specs (HANA vs general purpose). flavorExtraSpecs := make(map[string]string) for k, v := range vm.FlavorExtraSpecs { flavorExtraSpecs[k] = v @@ -241,57 +224,61 @@ func (c *FailoverReservationController) validateVMViaSchedulerEvacuation( flavorExtraSpecs["capabilities:hypervisor_type"] = "qemu" } - // Build a single-host request to validate the VM can use the reservation host - // Use vm.CurrentHypervisor directly instead of a separate parameter to avoid stale data - // Set _nova_check_type to "evacuate" so that filter_has_enough_capacity unlocks - // the failover reservation's resources for this VM (avoiding self-blocking). - // Use the actual VM UUID (not prefixed) so the filter can match it in the reservation's allocations. - scheduleReq := reservations.ScheduleReservationRequest{ - InstanceUUID: vm.UUID, - ProjectID: vm.ProjectID, - FlavorName: vm.FlavorName, - FlavorExtraSpecs: flavorExtraSpecs, - MemoryMB: memoryMB, - VCPUs: vcpus, - EligibleHosts: []api.ExternalSchedulerHost{{ComputeHost: reservationHost}}, - IgnoreHosts: []string{vm.CurrentHypervisor}, - Pipeline: inferFailoverPipeline(flavorExtraSpecs), - AvailabilityZone: vm.AvailabilityZone, - SchedulerHints: map[string]any{"_nova_check_type": string(api.EvacuateIntent)}, - ImageProperties: imagePropertiesFromFlavor(flavorExtraSpecs), + // Validate the VM can use the reservation host. Set the Evacuate intent so + // filter_has_enough_capacity unlocks the failover reservation's resources for + // this VM (avoiding self-blocking). Use the actual VM UUID so the filter can + // match it in the reservation's allocations. KVM gathers all candidates; the + // reservation host must come back ranked for the VM to be compatible. + pipeline := InferFailoverPipeline(flavorExtraSpecs) + requiredTraits, forbiddenTraits := api.SplitTraits(flavorExtraSpecs) + evacuateOpts := DefaultFailoverOptions + evacuateOpts.ReadOnly = true + evacuateOpts.LockReservations = true + schedReq := compute.PlacementRequest{ + Pipeline: pipeline, + GatherAllCandidates: true, + HypervisorFamily: api.HypervisorFamilyKVM, + Hosts: []compute.Host{{ComputeHost: reservationHost, HypervisorHostname: reservationHost}}, + Weights: map[string]float64{reservationHost: 0.0}, + Options: evacuateOpts, + Intent: api.EvacuateIntent, + InstanceUUID: vm.UUID, + NumInstances: 1, // validating one VM; NumInstances must be >= 1 or capacity/bin-pack math treats the host as free + ProjectID: vm.ProjectID, + AvailabilityZone: vm.AvailabilityZone, + IgnoreHosts: []string{vm.CurrentHypervisor}, + RequiredTraits: requiredTraits, + ForbiddenTraits: forbiddenTraits, + RequestedCapabilities: api.HypervisorTypeCapability(flavorExtraSpecs, true), + AcceleratorsRequested: api.RequestsAccelerators(flavorExtraSpecs), + Flavor: api.FlavorSpec{ + VCPUs: vcpus, + MemoryMB: memoryMB, + }, } logger.V(1).Info("validating VM via scheduler evacuation", "vmUUID", vm.UUID, "reservationHost", reservationHost, "vmCurrentHost", vm.CurrentHypervisor, - "pipeline", scheduleReq.Pipeline) + "pipeline", pipeline) - evacuateOpts := DefaultFailoverOptions - evacuateOpts.ReadOnly = true - evacuateOpts.LockReservations = true - resp, err := c.SchedulerClient.ScheduleReservation(ctx, scheduleReq, evacuateOpts) + out, err := c.runScheduler(ctx, schedReq) if err != nil { logger.Error(err, "failed to validate VM for reservation host", "vmUUID", vm.UUID, "reservationHost", reservationHost) return false, fmt.Errorf("failed to validate VM for reservation host: %w", err) } + hosts := out.OrderedHosts // Handle empty response - no hosts returned - if len(resp.Hosts) < 1 { + if len(hosts) < 1 { logger.V(1).Info("scheduler returned no hosts for VM validation", "vmUUID", vm.UUID, "reservationHost", reservationHost) return false, nil } - // Log unexpected scheduler responses - if len(resp.Hosts) > 1 || resp.Hosts[0] != reservationHost { - logger.Error(nil, "scheduler returned unexpected hosts for single-host validation request", - "vmUUID", vm.UUID, - "reservationHost", reservationHost, - "returnedHosts", resp.Hosts) - } - - // If the reservation host is returned, the VM can use it - return resp.Hosts[0] == reservationHost, nil + // The reservation host must be among the returned hosts for the VM to be + // compatible with it. + return slices.Contains(hosts, reservationHost), nil } // scheduleAndBuildNewFailoverReservation schedules a failover reservation for a VM. diff --git a/internal/scheduling/reservations/failover/reservation_scheduling_test.go b/internal/scheduling/reservations/failover/reservation_scheduling_test.go index 506b6033b..275e95cf8 100644 --- a/internal/scheduling/reservations/failover/reservation_scheduling_test.go +++ b/internal/scheduling/reservations/failover/reservation_scheduling_test.go @@ -5,14 +5,14 @@ package failover import ( "context" - "encoding/json" - "net/http" - "net/http/httptest" + "slices" "testing" - novaapi "github.com/cobaltcore-dev/cortex/api/external/nova" + api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" + "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "k8s.io/apimachinery/pkg/api/resource" @@ -178,23 +178,21 @@ func TestBuildNewFailoverReservation(t *testing.T) { } // Verify resources are derived from VM - // Note: VM uses "vcpus" but reservation uses "cpu" as the canonical key. // Memory uses binary MiB (bytes / 1024*1024 → MiB → MiB * 1024*1024), matching // commitments/state.go and vm_source.go conventions. if tt.vm.Resources != nil { - if memory, ok := tt.vm.Resources["memory"]; ok { + if memory, ok := tt.vm.Resources[string(hv1.ResourceMemory)]; ok { if resMemory, ok := result.Spec.Resources[hv1.ResourceMemory]; !ok { t.Error("reservation missing memory resource") } else if !memory.Equal(resMemory) { t.Errorf("memory resource = %v, want %v", resMemory, memory) } } - if vcpus, ok := tt.vm.Resources["vcpus"]; ok { - // VM uses "vcpus" but reservation should use "cpu" + if cpu, ok := tt.vm.Resources[string(hv1.ResourceCPU)]; ok { if resCPU, ok := result.Spec.Resources[hv1.ResourceCPU]; !ok { t.Error("reservation missing cpu resource") - } else if !vcpus.Equal(resCPU) { - t.Errorf("cpu resource = %v, want %v", resCPU, vcpus) + } else if !cpu.Equal(resCPU) { + t.Errorf("cpu resource = %v, want %v", resCPU, cpu) } } } @@ -276,8 +274,8 @@ func TestResolveVMForSchedulingAndNewFailoverReservation(t *testing.T) { FlavorName: "hana_c60_m960", ProjectID: "test-project", Resources: map[string]resource.Quantity{ - "vcpus": *resource.NewQuantity(60, resource.DecimalSI), - "memory": *resource.NewQuantity(983040*1024*1024, resource.BinarySI), + string(hv1.ResourceCPU): *resource.NewQuantity(60, resource.DecimalSI), + string(hv1.ResourceMemory): *resource.NewQuantity(983040*1024*1024, resource.BinarySI), }, }, useFlavorGroupResources: true, @@ -296,8 +294,8 @@ func TestResolveVMForSchedulingAndNewFailoverReservation(t *testing.T) { FlavorName: "hana_c60_m960", ProjectID: "test-project", Resources: map[string]resource.Quantity{ - "vcpus": *resource.NewQuantity(60, resource.DecimalSI), - "memory": *resource.NewQuantity(983040*1024*1024, resource.BinarySI), + string(hv1.ResourceCPU): *resource.NewQuantity(60, resource.DecimalSI), + string(hv1.ResourceMemory): *resource.NewQuantity(983040*1024*1024, resource.BinarySI), }, }, useFlavorGroupResources: false, @@ -316,8 +314,8 @@ func TestResolveVMForSchedulingAndNewFailoverReservation(t *testing.T) { FlavorName: "unknown_flavor", ProjectID: "test-project", Resources: map[string]resource.Quantity{ - "vcpus": *resource.NewQuantity(8, resource.DecimalSI), - "memory": *resource.NewQuantity(16384*1024*1024, resource.BinarySI), + string(hv1.ResourceCPU): *resource.NewQuantity(8, resource.DecimalSI), + string(hv1.ResourceMemory): *resource.NewQuantity(16384*1024*1024, resource.BinarySI), }, }, useFlavorGroupResources: true, @@ -336,8 +334,8 @@ func TestResolveVMForSchedulingAndNewFailoverReservation(t *testing.T) { FlavorName: "hana_c60_m960", ProjectID: "test-project", Resources: map[string]resource.Quantity{ - "vcpus": *resource.NewQuantity(60, resource.DecimalSI), - "memory": *resource.NewQuantity(983040*1024*1024, resource.BinarySI), + string(hv1.ResourceCPU): *resource.NewQuantity(60, resource.DecimalSI), + string(hv1.ResourceMemory): *resource.NewQuantity(983040*1024*1024, resource.BinarySI), }, }, useFlavorGroupResources: true, @@ -415,8 +413,8 @@ func buildSchedulingTestReservation(name, host string, allocations map[string]st Type: v1alpha1.ReservationTypeFailover, TargetHost: host, Resources: map[hv1.ResourceName]resource.Quantity{ - "memory": resource.MustParse("8Gi"), - "vcpus": resource.MustParse("4"), + hv1.ResourceMemory: resource.MustParse("8Gi"), + hv1.ResourceCPU: resource.MustParse("4"), }, }, Status: v1alpha1.ReservationStatus{ @@ -438,8 +436,8 @@ func buildSchedulingTestReservationNoStatus(name, host string) v1alpha1.Reservat Type: v1alpha1.ReservationTypeFailover, TargetHost: host, Resources: map[hv1.ResourceName]resource.Quantity{ - "memory": resource.MustParse("8Gi"), - "vcpus": resource.MustParse("4"), + hv1.ResourceMemory: resource.MustParse("8Gi"), + hv1.ResourceCPU: resource.MustParse("4"), }, }, Status: v1alpha1.ReservationStatus{ @@ -457,8 +455,8 @@ func buildSchedulingTestVM(uuid, hypervisor string) reservations.VM { //nolint:u ProjectID: "test-project", AvailabilityZone: "az1", Resources: map[string]resource.Quantity{ - "vcpus": resource.MustParse("4"), - "memory": resource.MustParse("8Gi"), + string(hv1.ResourceCPU): resource.MustParse("4"), + string(hv1.ResourceMemory): resource.MustParse("8Gi"), }, } } @@ -471,28 +469,46 @@ func buildSchedulingTestVMWithResources(uuid, hypervisor string, memoryMB, vcpus ProjectID: "test-project", AvailabilityZone: "az1", Resources: map[string]resource.Quantity{ - "vcpus": *resource.NewQuantity(vcpus, resource.DecimalSI), - "memory": *resource.NewQuantity(memoryMB*1024*1024, resource.BinarySI), + string(hv1.ResourceCPU): *resource.NewQuantity(vcpus, resource.DecimalSI), + string(hv1.ResourceMemory): *resource.NewQuantity(memoryMB*1024*1024, resource.BinarySI), }, } } -// captureSchedulerRequest spins up a test HTTP server that captures one scheduler request -// and returns a single host. The captured request is written into *out. -func captureSchedulerRequest(t *testing.T, host string, out *novaapi.ExternalSchedulerRequest) *httptest.Server { - t.Helper() - return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if err := json.NewDecoder(r.Body).Decode(out); err != nil { - http.Error(w, err.Error(), http.StatusBadRequest) - return - } - resp := novaapi.ExternalSchedulerResponse{Hosts: []string{host}} - _ = json.NewEncoder(w).Encode(resp) //nolint:errcheck - })) +// capturingPipeline records the last SchedulerRequest it ran and returns a +// single fixed host, so tests can assert the options the controller built. +type capturingPipeline struct { + host string + captured *sched.PlacementRequest +} + +func (p *capturingPipeline) Run(req sched.PlacementRequest) (sched.PlacementResult, error) { + *p.captured = req + return sched.PlacementResult{OrderedHosts: []string{p.host}}, nil +} + +// capturingProvider serves the same capturingPipeline for every pipeline name. +type capturingProvider struct{ pipeline *capturingPipeline } + +func (f capturingProvider) GetPipeline(string) (lib.FilterWeigherPipeline[sched.PlacementRequest], bool) { + return f.pipeline, true } -// buildMinimalController returns a FailoverReservationController wired to a scheduler at the given URL. -func buildMinimalController(t *testing.T, schedulerURL string) *FailoverReservationController { +// capturingGatherer is a no-op gatherer (the tests preselect their own hosts). +type capturingGatherer struct{} + +func (capturingGatherer) MutateWithAllCandidates(context.Context, *sched.PlacementRequest) error { + return nil +} + +// newCapturingScheduler builds an in-process scheduler whose pipeline returns +// the given host and records every request it ran into *out. +func newCapturingScheduler(host string, out *sched.PlacementRequest) *sched.Placement { + return sched.NewPlacement(capturingProvider{pipeline: &capturingPipeline{host: host, captured: out}}, capturingGatherer{}) +} + +// buildMinimalController returns a FailoverReservationController wired to the given scheduler. +func buildMinimalController(t *testing.T, scheduler *sched.Placement) *FailoverReservationController { t.Helper() scheme := runtime.NewScheme() if err := v1alpha1.AddToScheme(scheme); err != nil { @@ -502,10 +518,9 @@ func buildMinimalController(t *testing.T, schedulerURL string) *FailoverReservat t.Fatalf("add hv1 to scheme: %v", err) } k8sClient := fake.NewClientBuilder().WithScheme(scheme).Build() - schedulerClient := reservations.NewSchedulerClient(schedulerURL + "/scheduler/nova/external") config := FailoverConfig{} config.ApplyDefaults() - return NewFailoverReservationController(k8sClient, nil, config, schedulerClient, nil) + return NewFailoverReservationController(k8sClient, nil, config, scheduler, nil) } // ============================================================================ @@ -561,11 +576,10 @@ func TestFailoverSchedulerOptions(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - var capturedReq novaapi.ExternalSchedulerRequest - server := captureSchedulerRequest(t, "host-2", &capturedReq) - defer server.Close() + var capturedReq sched.PlacementRequest + scheduler := newCapturingScheduler("host-2", &capturedReq) - tt.call(buildMinimalController(t, server.URL), context.Background()) + tt.call(buildMinimalController(t, scheduler), context.Background()) if capturedReq.Options.ReadOnly != tt.wantReadOnly { t.Errorf("ReadOnly = %v, want %v", capturedReq.Options.ReadOnly, tt.wantReadOnly) @@ -578,61 +592,47 @@ func TestFailoverSchedulerOptions(t *testing.T) { } // ============================================================================ -// Test: imagePropertiesFromFlavor +// Test: trait and hypervisor_type constraints sent to the scheduler // ============================================================================ -func TestImagePropertiesFromFlavor(t *testing.T) { +// TestFailoverSchedulerFlavorConstraints verifies the scheduler request carries +// the flavor's traits and capabilities:hypervisor_type, so +// filter_has_requested_traits / filter_capabilities keep a workload off an +// incompatible host — including the qemu default when the flavor omits the +// hypervisor type. +func TestFailoverSchedulerFlavorConstraints(t *testing.T) { tests := []struct { name string extraSpecs map[string]string - want map[string]any + wantHVType string + wantTraits []string }{ + {name: "flavor without hypervisor_type defaults to canonical QEMU", extraSpecs: nil, wantHVType: "QEMU"}, + {name: "explicit CH preserved", extraSpecs: map[string]string{"capabilities:hypervisor_type": "CH"}, wantHVType: "CH"}, { - name: "qemu (lowercase) -> kvm img_hv_type", - extraSpecs: map[string]string{"capabilities:hypervisor_type": "qemu"}, - want: map[string]any{"img_hv_type": "kvm"}, - }, - { - name: "QEMU (uppercase) -> kvm img_hv_type", - extraSpecs: map[string]string{"capabilities:hypervisor_type": "QEMU"}, - want: map[string]any{"img_hv_type": "kvm"}, - }, - { - name: "CH -> kvm img_hv_type", - extraSpecs: map[string]string{"capabilities:hypervisor_type": "CH"}, - want: map[string]any{"img_hv_type": "kvm"}, - }, - { - name: "VMware -> hypervisor_type=vmware", - extraSpecs: map[string]string{"capabilities:hypervisor_type": "VMware vCenter Server"}, - want: map[string]any{"hypervisor_type": "vmware"}, - }, - { - name: "Ironic -> hypervisor_type=baremetal", - extraSpecs: map[string]string{"capabilities:hypervisor_type": "Ironic"}, - want: map[string]any{"hypervisor_type": "baremetal"}, - }, - { - name: "unspecified -> nil", - extraSpecs: map[string]string{}, - want: nil, - }, - { - name: "unknown value -> nil", - extraSpecs: map[string]string{"capabilities:hypervisor_type": "SomeFuturisticThing"}, - want: nil, + name: "lowercase qemu canonicalized and trait forwarded", + extraSpecs: map[string]string{"capabilities:hypervisor_type": "qemu", "trait:CUSTOM_HANA_EXCLUSIVE_HOST": "required"}, + wantHVType: "QEMU", + wantTraits: []string{"CUSTOM_HANA_EXCLUSIVE_HOST"}, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - got := imagePropertiesFromFlavor(tt.extraSpecs) - if len(got) != len(tt.want) { - t.Fatalf("length mismatch: got %v want %v", got, tt.want) + vm := buildSchedulingTestVM("vm-1", "host-1") + vm.FlavorExtraSpecs = tt.extraSpecs + resolved := resolveVMSpecForScheduling(context.Background(), vm, false, nil) + + var capturedReq sched.PlacementRequest + scheduler := newCapturingScheduler("host-2", &capturedReq) + c := buildMinimalController(t, scheduler) + _, _ = c.scheduleAndBuildNewFailoverReservation(context.Background(), vm, []string{"host-1", "host-2"}, nil, nil, resolved) //nolint:errcheck + + wantCap := []api.Capability{{Name: "hypervisor_type", Value: tt.wantHVType}} + if len(capturedReq.RequestedCapabilities) != 1 || capturedReq.RequestedCapabilities[0] != wantCap[0] { + t.Errorf("RequestedCapabilities = %v, want %v", capturedReq.RequestedCapabilities, wantCap) } - for k, v := range tt.want { - if got[k] != v { - t.Errorf("key %q: got %v want %v", k, got[k], v) - } + if !slices.Equal(capturedReq.RequiredTraits, tt.wantTraits) { + t.Errorf("RequiredTraits = %v, want %v", capturedReq.RequiredTraits, tt.wantTraits) } }) } diff --git a/internal/scheduling/reservations/failoverhost/controller.go b/internal/scheduling/reservations/failoverhost/controller.go index 8f001304f..def981f08 100644 --- a/internal/scheduling/reservations/failoverhost/controller.go +++ b/internal/scheduling/reservations/failoverhost/controller.go @@ -59,11 +59,11 @@ import ( "context" "errors" "maps" - "net/http" "sync" "time" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/event" @@ -107,10 +107,6 @@ type ControllerConfig struct { // GenerationInterval is the interval within which failover partitions are // recalculated and failover host reservations are updated accordingly. GenerationInterval metav1.Duration `json:"generationInterval"` - - // ExternalSchedulerURL is the URL of the external scheduler service that - // will be used to request failover host placements. - ExternalSchedulerURL string `json:"externalSchedulerURL"` } // Controller keeps the cluster's failover host reservations in sync with the @@ -129,6 +125,11 @@ type Controller struct { Config ControllerConfig + // Scheduler is the shared in-process scheduler the path finder probes for + // placement candidates. Required (the controller cannot compute failover + // partitions without it). + Scheduler *compute.Placement + // Monitor records Prometheus metrics for the calculation loop and the // reconciler. It may be nil (e.g. in tests); every call site nil-guards it. Monitor *Monitor @@ -336,15 +337,16 @@ func (c *Controller) SetupWithManager(mgr ctrl.Manager) error { // Default the generator. Tests may set it before calling Setup to substitute // a fake; here we wire the production pipeline stages. if c.generator == nil { + if c.Scheduler == nil { + return errors.New("scheduler cannot be nil") + } c.generator = &pipelineGenerator{ Client: c.Client, Monitor: c.Monitor, seeder: &seeder{k8s: c.Client}, pathFinder: &evacuationPathFinder{ - // Timeout so a hung scheduler can't stall the calculation loop. - httpClient: &http.Client{Timeout: 30 * time.Second}, - schedulerURL: c.Config.ExternalSchedulerURL, - monitor: c.Monitor, + scheduler: c.Scheduler, + monitor: c.Monitor, }, partitionBuilder: cliquePartitionBuilder{}, reservationPlanner: ratioReservationPlanner{ratio: c.Config.Ratio, minimum: c.Config.Minimum}, diff --git a/internal/scheduling/reservations/failoverhost/generation.go b/internal/scheduling/reservations/failoverhost/generation.go index fde3c7462..eb275f257 100644 --- a/internal/scheduling/reservations/failoverhost/generation.go +++ b/internal/scheduling/reservations/failoverhost/generation.go @@ -118,6 +118,7 @@ func (g *pipelineGenerator) Generate(ctx context.Context) (map[corev1.ObjectRefe g.Monitor.RecordCalculationAbort("seeding") return nil, topology{}, false } + log.Info("enumerated failover seeds to probe", "seeds", len(seeds)) paths, err := g.pathFinder.FindPaths(ctx, topo, seeds) if err != nil { @@ -125,12 +126,13 @@ func (g *pipelineGenerator) Generate(ctx context.Context) (map[corev1.ObjectRefe g.Monitor.RecordCalculationAbort("pathfinding") return nil, topology{}, false } + log.Info("probed seeds into failover paths", "seeds", len(seeds), "paths", len(paths)) // Cluster the failover edges into partitions. See partitioning.go for why a // partition is a maximal clique of mutually-evacuable hosts, and how empty // hosts are recruited into the partition that can reach them. partitions := g.partitionBuilder.BuildPartitions(paths) - log.V(1).Info("clustered failover paths into partitions", + log.Info("clustered failover paths into partitions", "paths", len(paths), "partitions", len(partitions)) // Maximal cliques overlap: a host with a mutual edge into two otherwise @@ -250,6 +252,8 @@ func (g *pipelineGenerator) collect(ctx context.Context) (topology, bool) { existing[corev1.ObjectReference{Name: reservation.Name}] = reservation } } + log.Info("collected cluster snapshot", + "hypervisors", len(hvsByName), "existingReservations", len(existing)) return topology{hypervisors: hvsByName, existing: existing}, true } diff --git a/internal/scheduling/reservations/failoverhost/generation_test.go b/internal/scheduling/reservations/failoverhost/generation_test.go index 074ea1a98..1d6191040 100644 --- a/internal/scheduling/reservations/failoverhost/generation_test.go +++ b/internal/scheduling/reservations/failoverhost/generation_test.go @@ -38,10 +38,9 @@ func TestGenerateProducesReservations(t *testing.T) { hvA := buildHV("hvA", 64, 32) hvB := buildHV("hvB", 64, 32) - finder, server := stubScheduler(t, map[string][]string{ + finder := stubScheduler(t, map[string][]string{ "flavor": {"hvA", "hvB"}, }, nil) - defer server.Close() g := &pipelineGenerator{ Client: newTestClient(t, hvA, hvB), @@ -84,10 +83,9 @@ func TestGenerateDeduplicatesSharedHost(t *testing.T) { // The path finder's edges are irrelevant here: the fake builder ignores them // and hands Generate the overlapping cliques directly. The stub still needs an // answer so FindPaths succeeds; one seed accepted on all three suffices. - finder, server := stubScheduler(t, map[string][]string{ + finder := stubScheduler(t, map[string][]string{ "flavor": {"hvA", "hvB", "hvC"}, }, nil) - defer server.Close() g := &pipelineGenerator{ Client: newTestClient(t, hvA, hvB, hvC), @@ -131,10 +129,9 @@ func TestGenerateSharedHostKeptInSmallestPartition(t *testing.T) { hvC := buildHV("hvC", 64, 32) hvD := buildHV("hvD", 64, 32) - finder, server := stubScheduler(t, map[string][]string{ + finder := stubScheduler(t, map[string][]string{ "flavor": {"hvA", "hvB", "hvC", "hvD"}, }, nil) - defer server.Close() g := &pipelineGenerator{ Client: newTestClient(t, hvA, hvB, hvC, hvD), diff --git a/internal/scheduling/reservations/failoverhost/integration_test.go b/internal/scheduling/reservations/failoverhost/integration_test.go index 013d8e269..c34dae82d 100644 --- a/internal/scheduling/reservations/failoverhost/integration_test.go +++ b/internal/scheduling/reservations/failoverhost/integration_test.go @@ -5,16 +5,13 @@ package failoverhost import ( "context" - "encoding/json" - "net/http" - "net/http/httptest" "slices" "sync" "testing" - novaapi "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/v1alpha1" "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + sched "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" "github.com/cobaltcore-dev/cortex/internal/scheduling/nova" "github.com/cobaltcore-dev/cortex/internal/scheduling/nova/plugins/filters" @@ -22,10 +19,18 @@ import ( hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/runtime" "sigs.k8s.io/controller-runtime/pkg/client" ) +// passthroughGatherer is a nova.CandidateGatherer that leaves the request's +// preselected hosts unchanged. The integration tests seed their own eligible +// hosts, so gathering is a no-op here. +type passthroughGatherer struct{} + +func (passthroughGatherer) MutateWithAllCandidates(context.Context, *sched.PlacementRequest) error { + return nil +} + // ============================================================================ // Integration Tests // @@ -199,7 +204,6 @@ func TestIntegration(t *testing.T) { for _, tc := range testCases { t.Run(tc.Name, func(t *testing.T) { env := newIntegrationTestEnv(t, tc.Hypervisors, tc.Flavors) - defer env.Close() ratio, minimum := tc.Ratio, tc.Minimum if ratio == 0 { @@ -234,19 +238,17 @@ func TestIntegration(t *testing.T) { // Test Environment // ============================================================================ -// integrationTestEnv wires a real Nova filter/weigher pipeline (served over -// HTTP) on top of a fake k8s client holding the fleet and the flavor-groups -// Knowledge CRD, so the generation pipeline can be driven end to end. +// integrationTestEnv wires a real Nova filter/weigher pipeline behind the shared +// in-process scheduler on top of a fake k8s client holding the fleet and the +// flavor-groups Knowledge CRD, so the generation pipeline can be driven end to +// end. type integrationTestEnv struct { - k8sClient client.Client - server *httptest.Server - schedulerURL string - flavors []compute.FlavorInGroup - hypervisors map[string]hv1.Hypervisor + k8sClient client.Client + scheduler *sched.Placement + flavors []compute.FlavorInGroup + hypervisors map[string]hv1.Hypervisor } -func (env *integrationTestEnv) Close() { env.server.Close() } - // newIntegrationTestEnv builds the fake cluster and the real scheduler pipeline. func newIntegrationTestEnv(t *testing.T, hypervisors []*hv1.Hypervisor, flavors []compute.FlavorInGroup) *integrationTestEnv { t.Helper() @@ -262,9 +264,9 @@ func newIntegrationTestEnv(t *testing.T, hypervisors []*hv1.Hypervisor, flavors k8sClient := newTestClient(t, objects...) controller := &nova.FilterWeigherPipelineController{ - BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[novaapi.ExternalSchedulerRequest]]{ + BasePipelineController: lib.BasePipelineController[lib.FilterWeigherPipeline[sched.PlacementRequest]]{ Client: k8sClient, - Pipelines: make(map[string]lib.FilterWeigherPipeline[novaapi.ExternalSchedulerRequest]), + Pipelines: make(map[string]lib.FilterWeigherPipeline[sched.PlacementRequest]), PipelineConfigs: make(map[string]v1alpha1.Pipeline), }, Monitor: sharedPipelineMonitor(), @@ -294,17 +296,15 @@ func newIntegrationTestEnv(t *testing.T, hypervisors []*hv1.Hypervisor, flavors controller.Pipelines[pipeline.Name] = result.Pipeline controller.PipelineConfigs[pipeline.Name] = pipeline - api := &integrationHTTPAPI{delegate: controller} - mux := http.NewServeMux() - mux.HandleFunc("/scheduler/nova/external", api.handle) - server := httptest.NewServer(mux) + // KVM requests trigger candidate gathering; the test seeds its own eligible + // hosts, so use a passthrough gatherer that preserves them. + scheduler := sched.NewPlacement(controller, passthroughGatherer{}) return &integrationTestEnv{ - k8sClient: k8sClient, - server: server, - schedulerURL: server.URL + "/scheduler/nova/external", - flavors: flavors, - hypervisors: hvsByName, + k8sClient: k8sClient, + scheduler: scheduler, + flavors: flavors, + hypervisors: hvsByName, } } @@ -319,7 +319,7 @@ func (env *integrationTestEnv) Generate(t *testing.T, ratio float64, minimum int g := &pipelineGenerator{ Client: env.k8sClient, seeder: &seeder{k8s: env.k8sClient}, - pathFinder: &evacuationPathFinder{httpClient: http.DefaultClient, schedulerURL: env.schedulerURL}, + pathFinder: &evacuationPathFinder{scheduler: env.scheduler}, partitionBuilder: cliquePartitionBuilder{}, reservationPlanner: ratioReservationPlanner{ratio: ratio, minimum: minimum}, } @@ -342,7 +342,7 @@ func (env *integrationTestEnv) AcceptsByHost(t *testing.T) map[string]map[string for _, hv := range env.hypervisors { zones[hv.Labels[corev1.LabelTopologyZone]] = true } - finder := &evacuationPathFinder{httpClient: http.DefaultClient, schedulerURL: env.schedulerURL} + finder := &evacuationPathFinder{scheduler: env.scheduler} accepts := map[string]map[string]bool{} for _, flavor := range env.flavors { for zone := range zones { @@ -429,49 +429,6 @@ func (env *integrationTestEnv) VerifySiblingSources(t *testing.T, desired map[co // probe exercises. const evacProbePipeline = "kvm-general-purpose-load-balancing" -// integrationHTTPAPI is a thin, deterministic HTTP front-end for the real Nova -// filter/weigher pipeline — the same shape the failover package's integration -// tests use. It does NOT shuffle, so the probe answers are stable. -type integrationHTTPAPI struct { - delegate nova.HTTPAPIDelegate -} - -func (a *integrationHTTPAPI) handle(w http.ResponseWriter, r *http.Request) { - defer r.Body.Close() - var reqData novaapi.ExternalSchedulerRequest - if err := json.NewDecoder(r.Body).Decode(&reqData); err != nil { - http.Error(w, "decode", http.StatusBadRequest) - return - } - rawBytes, err := json.Marshal(reqData) - if err != nil { - http.Error(w, "marshal", http.StatusInternalServerError) - return - } - decision := &v1alpha1.Decision{ - TypeMeta: metav1.TypeMeta{Kind: "Decision", APIVersion: "cortex.cloud/v1alpha1"}, - ObjectMeta: metav1.ObjectMeta{GenerateName: "nova-"}, - Spec: v1alpha1.DecisionSpec{ - SchedulingDomain: v1alpha1.SchedulingDomainNova, - PipelineRef: corev1.ObjectReference{Name: evacProbePipeline}, - ResourceID: reqData.Spec.Data.InstanceUUID, - NovaRaw: &runtime.RawExtension{Raw: rawBytes}, - }, - } - if err := a.delegate.ProcessNewDecisionFromAPI(r.Context(), decision); err != nil { - http.Error(w, "process", http.StatusInternalServerError) - return - } - if decision.Status.Result == nil { - http.Error(w, "no result", http.StatusInternalServerError) - return - } - w.Header().Set("Content-Type", "application/json") - if err := json.NewEncoder(w).Encode(novaapi.ExternalSchedulerResponse{Hosts: decision.Status.Result.OrderedHosts}); err != nil { - http.Error(w, "encode", http.StatusInternalServerError) - } -} - // ============================================================================ // Object Creation Functions // ============================================================================ diff --git a/internal/scheduling/reservations/failoverhost/partitioning.go b/internal/scheduling/reservations/failoverhost/partitioning.go index 3ce215ef5..8c158a52c 100644 --- a/internal/scheduling/reservations/failoverhost/partitioning.go +++ b/internal/scheduling/reservations/failoverhost/partitioning.go @@ -398,31 +398,21 @@ func characterizePartitions(partitions []Partition) []string { return labels } -// maxDescribedFlavors caps how many flavor names describePartition spells out, so -// a partition that runs hundreds of flavors doesn't bloat the reservation's -// PartitionDescription field. The rest are summarized as a "(+k more)" count; the -// full set is still logged per pass. -const maxDescribedFlavors = 5 - // describePartition renders the human-readable PartitionDescription stored on each // reservation: the distinguishing label (see characterizePartitions) followed by -// the flavors the partition can run (see Partition.Flavors), e.g. -// "spec.aggregates=[hana] | flavors: hana.4xl, hana.8xl (+3 more)". flavors is -// assumed sorted. Either half is omitted when empty, so a partition with no -// distinguishing label still reports its flavors and vice versa. +// every flavor the partition can run (see Partition.Flavors), e.g. +// "spec.aggregates=[hana] | flavors: hana.4xl, hana.8xl". The full flavor list is +// spelled out rather than truncated, so an individual flavor missing from one of +// the partitions supporting many flavors can be spotted. flavors is assumed sorted. +// Either half is omitted when empty, so a partition with no distinguishing label +// still reports its flavors and vice versa. func describePartition(label string, flavors []string) string { parts := []string{} if label != "" { parts = append(parts, label) } if len(flavors) > 0 { - shown := flavors - suffix := "" - if len(flavors) > maxDescribedFlavors { - shown = flavors[:maxDescribedFlavors] - suffix = fmt.Sprintf(" (+%d more)", len(flavors)-maxDescribedFlavors) - } - parts = append(parts, "flavors: "+strings.Join(shown, ", ")+suffix) + parts = append(parts, "flavors: "+strings.Join(flavors, ", ")) } return strings.Join(parts, " | ") } diff --git a/internal/scheduling/reservations/failoverhost/partitioning_test.go b/internal/scheduling/reservations/failoverhost/partitioning_test.go index 063ee4390..0ebe3c1da 100644 --- a/internal/scheduling/reservations/failoverhost/partitioning_test.go +++ b/internal/scheduling/reservations/failoverhost/partitioning_test.go @@ -315,10 +315,10 @@ func TestDescribePartition(t *testing.T) { {"flavors only", "", []string{"a", "b"}, "flavors: a, b"}, {"neither", "", nil, ""}, { - name: "flavor list is capped", + name: "flavor list is not truncated", label: "", flavors: []string{"f1", "f2", "f3", "f4", "f5", "f6", "f7"}, - want: "flavors: f1, f2, f3, f4, f5 (+2 more)", + want: "flavors: f1, f2, f3, f4, f5, f6, f7", }, } for _, tt := range tests { diff --git a/internal/scheduling/reservations/failoverhost/pathfinding.go b/internal/scheduling/reservations/failoverhost/pathfinding.go index 5ca983e97..195dff062 100644 --- a/internal/scheduling/reservations/failoverhost/pathfinding.go +++ b/internal/scheduling/reservations/failoverhost/pathfinding.go @@ -4,20 +4,17 @@ package failoverhost import ( - "bytes" "context" - "encoding/json" "fmt" "maps" - "net/http" "slices" "sort" - "strconv" "time" api "github.com/cobaltcore-dev/cortex/api/external/nova" "github.com/cobaltcore-dev/cortex/api/scheduling" "github.com/cobaltcore-dev/cortex/api/v1alpha1" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" "k8s.io/apimachinery/pkg/api/resource" logf "sigs.k8s.io/controller-runtime/pkg/log" @@ -71,21 +68,19 @@ type PathFinder interface { FindPaths(ctx context.Context, topo topology, seeds []Seed) ([]Path, error) } -// evacuationPathFinder finds failover paths by asking the external cortex -// scheduler, per seed, which hosts would accept that flavor, then pairing up the -// co-accepting hosts subject to gang-level fit. +// evacuationPathFinder finds failover paths by probing the shared in-process +// scheduler, per seed, for which hosts would accept that flavor, then pairing up +// the co-accepting hosts subject to gang-level fit. // // Asking the real scheduler (rather than reimplementing placement here) is // deliberate: it means failover paths are computed with exactly the same // filters, weighers and constraints that govern ordinary placement, so a // partition can never contain a target the scheduler would actually reject. type evacuationPathFinder struct { - // httpClient issues the scheduler requests. Give it a timeout so a hung - // scheduler cannot stall the whole calculation loop. Separately injected so - // tests can point it at a stub server. - httpClient *http.Client - // schedulerURL is the external cortex scheduler endpoint to POST to. - schedulerURL string + // scheduler is the shared in-process scheduler used to probe placement. The + // pathfinder issues read-only runs via RunWithoutLock. Separately injected so + // tests can point it at a fake pipeline. + scheduler *compute.Placement // monitor records scheduler request metrics. May be nil (e.g. in tests). monitor *Monitor } @@ -114,7 +109,23 @@ func (f *evacuationPathFinder) FindPaths(ctx context.Context, topo topology, see // edge, deduplicating across seeds. We materialize it into the []Path slice at // the end, carrying each edge's supporting flavors. edges := map[string]map[string]map[string]bool{} - for _, seed := range seeds { + // Log progress every ~10% of seeds probed: a large fleet crossed with its + // flavor catalogue yields thousands of seeds, each a scheduler round-trip, so a + // pass can take minutes. Without this the loop looks wedged. nextProgress is the + // next seed index at which to log; step is a tenth of the seed count (at least + // one, so tiny seed sets still report). + step := len(seeds) / 10 + if step == 0 { + step = 1 + } + nextProgress := step + for i, seed := range seeds { + if i+1 >= nextProgress { + log.Info("probing failover seeds", + "probed", i+1, "total", len(seeds), + "percent", (i+1)*100/len(seeds), "edgesSoFar", len(edges)) + nextProgress += step + } accepting, err := f.fetchSeedCandidates(ctx, seed, topo.hypervisors) if err != nil { // A lookup error is an infrastructure failure (the scheduler is @@ -219,10 +230,12 @@ func hvCapacity(hv hv1.Hypervisor) map[hv1.ResourceName]resource.Quantity { // set (scheduling.Options.Validate rejects a ReadOnly run otherwise, and the // scheduler calls Validate) so this probe neither creates in-flight // reservations nor writes CR allocations. -// - IgnoredReservationTypes = FailoverHost, so existing failover host -// reservations don't block the very hosts we are reasoning about — otherwise -// a reserved host would be hidden from its own partition and the clustering -// would fragment. +// - IgnoredReservationTypes = every reservation type, so no existing +// reservation blocks the hosts we are reasoning about. Failover host +// reservations would otherwise hide a reserved host from its own partition +// and fragment the clustering; committed-resource, failover and in-flight +// reservations would understate capacity, which is wrong under the +// empty-host reasoning below. // - AssumeEmptyHosts, so candidates are judged by their total capacity rather // than their current free capacity. A failover target is reserved *empty*, so // what matters for partitioning is whether a host could hold the workload once @@ -242,25 +255,62 @@ func hvCapacity(hv hv1.Hypervisor) map[hv1.ResourceName]resource.Quantity { func (f *evacuationPathFinder) fetchSeedCandidates(ctx context.Context, seed Seed, hvsByName map[string]hv1.Hypervisor) ([]hv1.Hypervisor, error) { log := logf.FromContext(ctx) - // Build valid image properties. We set the image hypervisor type explicitly so - // the filter that checks flavor/image compatibility can evaluate it. Since we - // only handle KVM virtual machines, we set it to KVM. - imageProps := map[string]any{ - "img_hv_type": string(api.NovaImageMetaHVTypeKVM), - } // Offer every known host as a candidate. The scheduler filters/weighs over // exactly this host set — it does not gather the cluster itself — so leaving // Hosts empty would make it return no candidates. Each host also needs a // weight entry, which the scheduler requires to accompany the host set. - hosts := make([]api.ExternalSchedulerHost, 0, len(hvsByName)) + hosts := make([]compute.Host, 0, len(hvsByName)) weights := make(map[string]float64, len(hvsByName)) for name := range hvsByName { - hosts = append(hosts, api.ExternalSchedulerHost{ComputeHost: name}) + hosts = append(hosts, compute.Host{ComputeHost: name}) weights[name] = 0 } - requestObj := api.ExternalSchedulerRequest{ - Hosts: hosts, - Weights: weights, + // Honor the flavor's trait and hypervisor-type constraints so the probe only + // considers compatible hosts (filter_has_requested_traits/filter_capabilities + // would otherwise see empty slices and become no-ops). The seeder only emits + // seeds whose flavor names a supported hypervisor type, so no default is needed. + requiredTraits, forbiddenTraits := api.SplitTraits(seed.Flavor.ExtraSpecs) + // Same HANA vs general-purpose split as regular failover placement, but with the + // load-balancing variant of the HANA pipeline used for pathfinding probes. + flavorType, err := api.FlavorTypeFromExtraSpecs(seed.Flavor.ExtraSpecs) + if err != nil { + return nil, err + } + pipeline := "kvm-general-purpose-load-balancing" + if flavorType == api.FlavorTypeHANA { + pipeline = "kvm-hana-load-balancing" + } + req := compute.PlacementRequest{ + Pipeline: pipeline, + // We offer the host set ourselves; the scheduler must not gather the + // cluster itself and overwrite our candidates. + GatherAllCandidates: false, + HypervisorFamily: api.HypervisorFamilyKVM, + // We only handle KVM virtual machines, so the flavor/image compatibility + // filter must see a KVM image hypervisor type. + ImageHypervisorType: api.ImageHypervisorTypeKVM, + Hosts: hosts, + Weights: weights, + // Probe with evacuation intent. A failover cold-migrates (evacuates) the + // workload, and evacuation does not run the live-migratability feature + // check, so no source host is needed. This is deliberately weaker than a + // live-migration probe — see the PathFinder doc. + Intent: api.EvacuateIntent, + // Probe a single flavor slot; NumInstances must be >= 1 or capacity/ + // bin-pack math treats the host as free. + NumInstances: 1, + AvailabilityZone: seed.AvailabilityZone, + RequiredTraits: requiredTraits, + ForbiddenTraits: forbiddenTraits, + // Match the flavor's hypervisor type (QEMU vs CH). The seeder only emits + // seeds whose flavor names a supported hypervisor type, so this is always + // set (no default needed). + RequestedCapabilities: api.HypervisorTypeCapability(seed.Flavor.ExtraSpecs, false), + AcceleratorsRequested: api.RequestsAccelerators(seed.Flavor.ExtraSpecs), + Flavor: api.FlavorSpec{ + VCPUs: seed.Flavor.VCPUs, + MemoryMB: seed.Flavor.MemoryMB, + }, Options: scheduling.Options{ // Read-only probe: don't write any resources. ReadOnly additionally // requires SkipInflight and SkipCommittedResourceTracking, else the @@ -273,87 +323,38 @@ func (f *evacuationPathFinder) fetchSeedCandidates(ctx context.Context, seed See // workload once cleared must still be a valid candidate. Without this, // occupied hosts never accept anyone and stay orphaned. AssumeEmptyHosts: true, - // Ignore existing failover host reservations, otherwise they would - // block these hosts, separate them from their failover partition and - // break the clustering. + // Ignore every existing reservation while probing. We reason about + // whether a host *could* hold the evacuated workload once it is cleared + // (AssumeEmptyHosts), not about its current free space, so any reservation + // — failover host, committed resource, failover or in-flight — that would + // otherwise block these hosts must be ignored. Failover host reservations + // in particular would hide the very hosts we are reasoning about and + // fragment the clustering; the others would understate capacity for the + // same empty-host reasoning. IgnoredReservationTypes: []v1alpha1.ReservationType{ v1alpha1.ReservationTypeFailoverHost, - }, - }, - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - NumInstances: 1, - AvailabilityZone: seed.AvailabilityZone, - // Probe with evacuation intent. A failover cold-migrates (evacuates) - // the workload, and evacuation does not run the live-migratability - // feature check, so no source_host is needed. The check type must be - // the literal "evacuate" nova uses, which maps to EvacuateIntent in - // GetIntent. See the PathFinder doc for why this is deliberately - // weaker than a live-migration probe. - SchedulerHints: map[string]any{ - "_nova_check_type": "evacuate", - }, - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - Name: seed.Flavor.Name, - ExtraSpecs: seed.Flavor.ExtraSpecs, - MemoryMB: seed.Flavor.MemoryMB, - VCPUs: seed.Flavor.VCPUs, - }, - }, - Image: api.NovaObject[api.NovaImageMeta]{ - Data: api.NovaImageMeta{ - Properties: api.NovaObject[map[string]any]{ - Data: imageProps, - }, - }, - }, + v1alpha1.ReservationTypeCommittedResource, + v1alpha1.ReservationTypeFailover, + v1alpha1.ReservationTypeInFlight, }, }, } - log.V(1).Info("external scheduler request prepared", "request", requestObj) - - reqBody, err := json.Marshal(requestObj) - if err != nil { - log.Error(err, "failed to marshal external scheduler request") - return nil, fmt.Errorf("failed to marshal external scheduler request: %w", err) - } - httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, - f.schedulerURL, bytes.NewReader(reqBody)) - if err != nil { - log.Error(err, "failed to create HTTP request") - return nil, fmt.Errorf("failed to create HTTP request: %w", err) - } - httpReq.Header.Set("Content-Type", "application/json") + log.V(1).Info("scheduler probe prepared", "request", req) startTime := time.Now() - response, err := f.httpClient.Do(httpReq) + result, err := f.scheduler.RunWithoutLock(ctx, req, nil, nil) duration := time.Since(startTime) if err != nil { f.monitor.RecordSchedulerRequest("error", duration) - log.Error(err, "failed to send external scheduler request") - return nil, fmt.Errorf("failed to send external scheduler request: %w", err) - } - defer response.Body.Close() - f.monitor.RecordSchedulerRequest(strconv.Itoa(response.StatusCode), duration) - log.V(1).Info("external scheduler request sent", - "duration", duration, "status", response.StatusCode) - - if response.StatusCode != http.StatusOK { - log.Error(nil, "external scheduler request failed", "status", response.StatusCode) - return nil, fmt.Errorf("external scheduler request failed with status: %d", response.StatusCode) - } - - var responseObj api.ExternalSchedulerResponse - if err := json.NewDecoder(response.Body).Decode(&responseObj); err != nil { - log.Error(err, "failed to decode external scheduler response") - return nil, fmt.Errorf("failed to decode external scheduler response: %w", err) + log.Error(err, "scheduler probe failed") + return nil, fmt.Errorf("scheduler probe failed: %w", err) } - log.V(1).Info("external scheduler response received", "response", responseObj) + f.monitor.RecordSchedulerRequest("200", duration) + log.V(1).Info("scheduler probe completed", "duration", duration, "hosts", result.OrderedHosts) candidates := []hv1.Hypervisor{} for _, hv := range hvsByName { - if slices.Contains(responseObj.Hosts, hv.Name) { + if slices.Contains(result.OrderedHosts, hv.Name) { candidates = append(candidates, hv) } } diff --git a/internal/scheduling/reservations/failoverhost/pathfinding_test.go b/internal/scheduling/reservations/failoverhost/pathfinding_test.go index e489e0ba4..e552a0661 100644 --- a/internal/scheduling/reservations/failoverhost/pathfinding_test.go +++ b/internal/scheduling/reservations/failoverhost/pathfinding_test.go @@ -5,66 +5,90 @@ package failoverhost import ( "context" - "encoding/json" - "net/http" - "net/http/httptest" + "fmt" "slices" "testing" api "github.com/cobaltcore-dev/cortex/api/external/nova" + "github.com/cobaltcore-dev/cortex/internal/scheduling/compute" + "github.com/cobaltcore-dev/cortex/internal/scheduling/lib" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" ) -// stubScheduler returns a path finder wired to an httptest server whose +// stubScheduler returns a path finder wired to an in-process scheduler whose // per-flavor answers come from candidatesByFlavor (keyed by the seed's flavor -// name); a flavor in errFlavors gets an HTTP 500. The handler asserts the request -// carries valid options, a non-empty host set, a kvm img_hv_type, and evacuation -// intent with no source_host — mirroring what the new seeded probe sends, so a -// regression that drops any of them is caught here. The caller closes the server. -func stubScheduler(t *testing.T, candidatesByFlavor map[string][]string, errFlavors map[string]bool) (*evacuationPathFinder, *httptest.Server) { +// name); a flavor in errFlavors makes the probe fail. The fake pipeline asserts +// the request carries valid options, a non-empty host set, AssumeEmptyHosts, a +// KVM image hypervisor type, and evacuation intent with no source host — +// mirroring what the seeded probe sends, so a regression that drops any of them +// is caught here. +func stubScheduler(t *testing.T, candidatesByFlavor map[string][]string, errFlavors map[string]bool) *evacuationPathFinder { t.Helper() - server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - var req api.ExternalSchedulerRequest - if err := json.NewDecoder(r.Body).Decode(&req); err != nil { - t.Errorf("decode request: %v", err) - } - if err := req.Options.Validate(); err != nil { - t.Errorf("invalid scheduling options: %v", err) - } - if !req.Options.AssumeEmptyHosts { - // A failover target is reserved empty, so candidates must be judged by - // total capacity. Dropping this would orphan every busy host. - t.Errorf("request for flavor %q must set AssumeEmptyHosts", req.Spec.Data.Flavor.Data.Name) - } - if len(req.Hosts) == 0 { - t.Errorf("request for flavor %q has no candidate hosts", req.Spec.Data.Flavor.Data.Name) - } - if got := req.Spec.Data.Image.Data.Properties.Data["img_hv_type"]; got != string(api.NovaImageMetaHVTypeKVM) { - // The hypervisor-type filter needs img_hv_type to judge flavor/image - // compatibility; dropping it would make that filter misbehave. - t.Errorf("request for flavor %q must set img_hv_type=%q, got %q", req.Spec.Data.Flavor.Data.Name, api.NovaImageMetaHVTypeKVM, got) - } - // The probe uses evacuation intent (cold migration) and sets no source_host: - // a failover evacuates rather than live-migrates, so the live-migratability - // feature check is deliberately skipped. Catch a regression to the stricter - // live-migration probe here. - if intent, err := req.GetIntent(); err != nil || intent != api.EvacuateIntent { - t.Errorf("request for flavor %q must use evacuate intent, got %v (err=%v)", req.Spec.Data.Flavor.Data.Name, intent, err) - } - if _, ok := req.Spec.Data.SchedulerHints["source_host"]; ok { - t.Errorf("request for flavor %q must not set source_host", req.Spec.Data.Flavor.Data.Name) - } - flavor := req.Spec.Data.Flavor.Data.Name - if errFlavors[flavor] { - w.WriteHeader(http.StatusInternalServerError) - return - } - if err := json.NewEncoder(w).Encode(api.ExternalSchedulerResponse{Hosts: candidatesByFlavor[flavor]}); err != nil { - t.Errorf("encode response: %v", err) - } - })) - finder := &evacuationPathFinder{httpClient: server.Client(), schedulerURL: server.URL} - return finder, server + provider := &stubPipelineProvider{pipeline: &stubPipeline{ + t: t, + candidatesByFlavor: candidatesByFlavor, + errFlavors: errFlavors, + }} + scheduler := compute.NewPlacement(provider, passthroughGatherer{}) + return &evacuationPathFinder{scheduler: scheduler} +} + +// stubPipelineProvider serves the same stub pipeline for every pipeline name the +// probe infers (always a KVM pipeline here). +type stubPipelineProvider struct { + pipeline *stubPipeline +} + +func (p *stubPipelineProvider) GetPipeline(string) (lib.FilterWeigherPipeline[compute.PlacementRequest], bool) { + return p.pipeline, true +} + +// stubPipeline is a fake filter/weigher pipeline that validates the probe's +// request shape and returns the preconfigured candidate hosts for the request's +// flavor, mirroring the old httptest stub. +type stubPipeline struct { + t *testing.T + candidatesByFlavor map[string][]string + errFlavors map[string]bool +} + +func (s *stubPipeline) Run(req compute.PlacementRequest) (compute.PlacementResult, error) { + // Identify which seed this probe is for by the trait the seed constructor + // puts on the flavor (see seed()), which the probe carries in RequiredTraits. + flavor := "" + if len(req.RequiredTraits) > 0 { + flavor = req.RequiredTraits[0] + } + if err := req.Options.Validate(); err != nil { + s.t.Errorf("invalid scheduling options: %v", err) + } + if !req.Options.AssumeEmptyHosts { + // A failover target is reserved empty, so candidates must be judged by + // total capacity. Dropping this would orphan every busy host. + s.t.Errorf("request for flavor %q must set AssumeEmptyHosts", flavor) + } + if len(req.Hosts) == 0 { + s.t.Errorf("request for flavor %q has no candidate hosts", flavor) + } + if req.ImageHypervisorType != api.ImageHypervisorTypeKVM { + // The hypervisor-type filter needs the image hypervisor type to judge + // flavor/image compatibility; dropping it would make that filter misbehave. + s.t.Errorf("request for flavor %q must set ImageHypervisorType=%q, got %q", flavor, api.ImageHypervisorTypeKVM, req.ImageHypervisorType) + } + // The probe uses evacuation intent (cold migration) and sets no source host: + // a failover evacuates rather than live-migrates, so the live-migratability + // feature check is deliberately skipped. Catch a regression to the stricter + // live-migration probe here. + if req.Intent != api.EvacuateIntent { + s.t.Errorf("request for flavor %q must use evacuate intent, got %v", flavor, req.Intent) + } + if req.SourceHost != "" { + s.t.Errorf("request for flavor %q must not set a source host", flavor) + } + if s.errFlavors[flavor] { + return compute.PlacementResult{}, fmt.Errorf("probe failed for flavor %q", flavor) + } + return compute.PlacementResult{OrderedHosts: s.candidatesByFlavor[flavor]}, nil } // pathNames collapses failover edges into sorted "source->target" strings, for @@ -89,9 +113,14 @@ func hostTopo(hostMemGi map[string]int64) topology { return topology{hypervisors: hypervisors} } -// seed is a tiny constructor for a flavor seed named name in az1. +// seed is a tiny constructor for a flavor seed named name in az1. The name is +// also set as a required trait so the stub pipeline can tell which seed a probe +// request belongs to (the probe carries the flavor's traits in RequiredTraits). func seed(name string) Seed { - return Seed{Flavor: Flavor{Name: name}, AvailabilityZone: "az1"} + return Seed{ + Flavor: Flavor{Name: name, ExtraSpecs: map[string]string{"trait:" + name: "required"}}, + AvailabilityZone: "az1", + } } func TestFindPathsErrorsOnLookupFailure(t *testing.T) { @@ -100,11 +129,10 @@ func TestFindPathsErrorsOnLookupFailure(t *testing.T) { // return an error to abort the pass rather than silently dropping paths — // otherwise the caller would publish an empty desired state and prune valid // reservations. - finder, server := stubScheduler(t, + finder := stubScheduler(t, map[string][]string{"good": {"a", "b"}}, map[string]bool{"bad": true}, ) - defer server.Close() // "bad" is probed and errors; the pass must abort regardless of "good". topo := hostTopo(map[string]int64{"a": 32, "b": 32}) @@ -116,10 +144,9 @@ func TestFindPathsErrorsOnLookupFailure(t *testing.T) { func TestFindPathsPairsCoAcceptingHosts(t *testing.T) { // A single seed accepted on a, b and c makes all three mutually compatible. // Equal-sized hosts yield edges both ways between every pair. - finder, server := stubScheduler(t, map[string][]string{ + finder := stubScheduler(t, map[string][]string{ "flavor": {"a", "b", "c"}, }, nil) - defer server.Close() topo := hostTopo(map[string]int64{"a": 32, "b": 32, "c": 32}) got, err := finder.FindPaths(context.Background(), topo, []Seed{seed("flavor")}) @@ -136,10 +163,9 @@ func TestFindPathsDropsTargetSmallerThanSource(t *testing.T) { // A seed is accepted on "big" (64Gi), "small" (16Gi) and "equal" (64Gi). The // gang-fit constraint keeps an edge only toward a target at least as large as // the source, so no edge points at "small", and big<->equal are mutual. - finder, server := stubScheduler(t, map[string][]string{ + finder := stubScheduler(t, map[string][]string{ "flavor": {"big", "small", "equal"}, }, nil) - defer server.Close() topo := hostTopo(map[string]int64{"big": 64, "small": 16, "equal": 64}) got, err := finder.FindPaths(context.Background(), topo, []Seed{seed("flavor")}) @@ -157,11 +183,10 @@ func TestFindPathsDropsTargetSmallerThanSource(t *testing.T) { func TestFindPathsDeduplicatesAcrossSeeds(t *testing.T) { // Two seeds both accepted on the same pair of equal hosts. Each directed edge // must appear once, not once per seed. - finder, server := stubScheduler(t, map[string][]string{ + finder := stubScheduler(t, map[string][]string{ "flavorA": {"hvA", "hvB"}, "flavorB": {"hvA", "hvB"}, }, nil) - defer server.Close() topo := hostTopo(map[string]int64{"hvA": 64, "hvB": 64}) got, err := finder.FindPaths(context.Background(), topo, []Seed{seed("flavorA"), seed("flavorB")}) @@ -175,10 +200,9 @@ func TestFindPathsDeduplicatesAcrossSeeds(t *testing.T) { func TestFindPathsSingleCandidateYieldsNoEdge(t *testing.T) { // A seed accepted on only one host can form no pair, so contributes no edge. - finder, server := stubScheduler(t, map[string][]string{ + finder := stubScheduler(t, map[string][]string{ "flavor": {"only"}, }, nil) - defer server.Close() topo := hostTopo(map[string]int64{"only": 64, "other": 64}) got, err := finder.FindPaths(context.Background(), topo, []Seed{seed("flavor")}) @@ -194,11 +218,10 @@ func TestFindPathsRecordsSupportingFlavors(t *testing.T) { // flavorA is accepted on both hosts, flavorB only on hvA. The edge hvA<->hvB is // produced solely by flavorA, so each directed edge must carry exactly // ["flavorA"] — flavorB co-accepts on no pair and leaves no trace. - finder, server := stubScheduler(t, map[string][]string{ + finder := stubScheduler(t, map[string][]string{ "flavorA": {"hvA", "hvB"}, "flavorB": {"hvA"}, }, nil) - defer server.Close() topo := hostTopo(map[string]int64{"hvA": 64, "hvB": 64}) got, err := finder.FindPaths(context.Background(), topo, []Seed{seed("flavorA"), seed("flavorB")}) diff --git a/internal/scheduling/reservations/failoverhost/seeding.go b/internal/scheduling/reservations/failoverhost/seeding.go index 1af4c98fc..4dd9d7508 100644 --- a/internal/scheduling/reservations/failoverhost/seeding.go +++ b/internal/scheduling/reservations/failoverhost/seeding.go @@ -25,8 +25,8 @@ import ( // fields fed into api.NovaFlavor, so the rest of the flavor-group shape never // leaks into the pathfinder. type Flavor struct { - // Name is the flavor name, forwarded as the probe's flavor name so - // flavor-name-sensitive filters behave as in a real placement. + // Name is the flavor name, used for pipeline inference, trait/capability + // extraction and to label the failover edges a seed supports. Name string // MemoryMB and VCPUs size the probe. RAM is in MiB (despite the "MB" label), // matching what the scheduler expects on NovaFlavor.MemoryMB. diff --git a/internal/scheduling/reservations/flavor_groups.go b/internal/scheduling/reservations/flavor_groups.go deleted file mode 100644 index b2ce7d859..000000000 --- a/internal/scheduling/reservations/flavor_groups.go +++ /dev/null @@ -1,95 +0,0 @@ -// Copyright SAP SE -// SPDX-License-Identifier: Apache-2.0 - -package reservations - -import ( - "context" - "errors" - "fmt" - - "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" - "k8s.io/apimachinery/pkg/api/meta" - "k8s.io/apimachinery/pkg/types" - ctrl "sigs.k8s.io/controller-runtime" - "sigs.k8s.io/controller-runtime/pkg/client" -) - -var flavorGroupsLog = ctrl.Log.WithName("flavor_groups") - -// FindFlavorInGroups searches all flavor groups for a flavor by name. -// Returns the flavor group name and flavor details, or an error if the flavor -// is not found in any group. -func FindFlavorInGroups(flavorName string, flavorGroups map[string]compute.FlavorGroupFeature) (groupName string, flavor *compute.FlavorInGroup, err error) { - for gName, fg := range flavorGroups { - for i, f := range fg.Flavors { - if f.Name == flavorName { - return gName, &fg.Flavors[i], nil - } - } - } - return "", nil, fmt.Errorf("flavor %q not found in any flavor group", flavorName) -} - -// FlavorGroupKnowledgeClient accesses flavor group data from Knowledge CRDs. -type FlavorGroupKnowledgeClient struct { - client.Client -} - -// Get retrieves the flavor groups Knowledge CRD and returns it if ready. -// Returns nil, nil if not ready yet. -func (c *FlavorGroupKnowledgeClient) Get(ctx context.Context) (*v1alpha1.Knowledge, error) { - knowledge := &v1alpha1.Knowledge{} - err := c.Client.Get(ctx, types.NamespacedName{ - Name: "flavor-groups", - // Namespace is empty as Knowledge is cluster-scoped - }, knowledge) - - if err != nil { - return nil, fmt.Errorf("failed to get flavor groups knowledge: %w", err) - } - - if meta.IsStatusConditionTrue(knowledge.Status.Conditions, v1alpha1.KnowledgeConditionReady) { - return knowledge, nil - } - - // Found but not ready yet - return nil, nil -} - -// GetAllFlavorGroups returns all flavor groups as a map. -// If knowledgeCRD is provided, uses it directly. Otherwise fetches the Knowledge CRD. -func (c *FlavorGroupKnowledgeClient) GetAllFlavorGroups(ctx context.Context, knowledgeCRD *v1alpha1.Knowledge) (map[string]compute.FlavorGroupFeature, error) { - // If no CRD provided, fetch it - if knowledgeCRD == nil { - var err error - knowledgeCRD, err = c.Get(ctx) - if err != nil { - return nil, err - } - if knowledgeCRD == nil { - return nil, errors.New("flavor groups knowledge is not ready") - } - } - - // Unbox the features from the raw extension - features, err := v1alpha1.UnboxFeatureList[compute.FlavorGroupFeature]( - knowledgeCRD.Status.Raw, - ) - if err != nil { - return nil, fmt.Errorf("failed to unbox flavor group features: %w", err) - } - - // Build map for efficient lookups, skipping any groups that fail validation. - flavorGroupMap := make(map[string]compute.FlavorGroupFeature, len(features)) - for _, feature := range features { - if err := feature.Validate(); err != nil { - flavorGroupsLog.Error(err, "skipping invalid flavor group from Knowledge CRD") - continue - } - flavorGroupMap[feature.Name] = feature - } - - return flavorGroupMap, nil -} diff --git a/internal/scheduling/reservations/inflight/controller_test.go b/internal/scheduling/reservations/inflight/controller_test.go index cf6751227..220048e87 100644 --- a/internal/scheduling/reservations/inflight/controller_test.go +++ b/internal/scheduling/reservations/inflight/controller_test.go @@ -297,12 +297,12 @@ func TestReconcile_ResizeSizeMismatchRequeues(t *testing.T) { // Ready condition to VMSizeMismatch. scheme := newTestScheme(t) reserved := map[hv1.ResourceName]resource.Quantity{ - "cpu": resource.MustParse("4"), - "memory": resource.MustParse("8Gi"), + hv1.ResourceCPU: resource.MustParse("4"), + hv1.ResourceMemory: resource.MustParse("8Gi"), } current := map[hv1.ResourceName]resource.Quantity{ - "cpu": resource.MustParse("2"), - "memory": resource.MustParse("4Gi"), + hv1.ResourceCPU: resource.MustParse("2"), + hv1.ResourceMemory: resource.MustParse("4Gi"), } res := newResizeReservation("res-1", "vm-uuid-1", "host-1", novaapi.ResizeIntent, reserved) hv := newHypervisor("host-1", "vm-uuid-1") @@ -332,8 +332,8 @@ func TestReconcile_ResizeSizeMatchesDeletesReservation(t *testing.T) { // reservation can be freed like a normal in-flight reservation. scheme := newTestScheme(t) reserved := map[hv1.ResourceName]resource.Quantity{ - "cpu": resource.MustParse("4"), - "memory": resource.MustParse("8Gi"), + hv1.ResourceCPU: resource.MustParse("4"), + hv1.ResourceMemory: resource.MustParse("8Gi"), } res := newResizeReservation("res-1", "vm-uuid-1", "host-1", novaapi.ResizeIntent, reserved) hv := newHypervisor("host-1", "vm-uuid-1") @@ -362,13 +362,13 @@ func TestReconcile_ResizeSizeMatchesSemanticallyDeletesReservation(t *testing.T) scheme := newTestScheme(t) // Reservation spec side: built like the API server would (MustParse). reserved := map[hv1.ResourceName]resource.Quantity{ - "cpu": resource.MustParse("4"), - "memory": resource.MustParse("8Gi"), + hv1.ResourceCPU: resource.MustParse("4"), + hv1.ResourceMemory: resource.MustParse("8Gi"), } // VM client side: built like the nova vm client does (NewQuantity). current := map[hv1.ResourceName]resource.Quantity{ - "cpu": *resource.NewQuantity(4, resource.DecimalSI), - "memory": *resource.NewQuantity(8*1024*1024*1024, resource.BinarySI), + hv1.ResourceCPU: *resource.NewQuantity(4, resource.DecimalSI), + hv1.ResourceMemory: *resource.NewQuantity(8*1024*1024*1024, resource.BinarySI), } res := newResizeReservation("res-1", "vm-uuid-1", "host-1", novaapi.ResizeIntent, reserved) hv := newHypervisor("host-1", "vm-uuid-1") @@ -395,7 +395,7 @@ func TestReconcile_ResizeSizeMatchesSemanticallyDeletesReservation(t *testing.T) func TestReconcile_ResizeStaleReservationSkipsVMClient(t *testing.T) { scheme := newTestScheme(t) reserved := map[hv1.ResourceName]resource.Quantity{ - "cpu": resource.MustParse("4"), + hv1.ResourceCPU: resource.MustParse("4"), } res := newResizeReservation("res-1", "vm-uuid-1", "host-1", novaapi.ResizeIntent, reserved) // Instance ended up on host-2, not the target host-1. @@ -431,10 +431,10 @@ func TestReconcile_RebuildSizeMismatchRequeues(t *testing.T) { // both intents actually trip the vmClient check. scheme := newTestScheme(t) reserved := map[hv1.ResourceName]resource.Quantity{ - "cpu": resource.MustParse("4"), + hv1.ResourceCPU: resource.MustParse("4"), } current := map[hv1.ResourceName]resource.Quantity{ - "cpu": resource.MustParse("2"), + hv1.ResourceCPU: resource.MustParse("2"), } res := newResizeReservation("res-1", "vm-uuid-1", "host-1", novaapi.RebuildIntent, reserved) hv := newHypervisor("host-1", "vm-uuid-1") @@ -459,7 +459,7 @@ func TestReconcile_ResizeVMClientErrorReturnsError(t *testing.T) { // reservation here would be a resource-accounting bug. scheme := newTestScheme(t) reserved := map[hv1.ResourceName]resource.Quantity{ - "cpu": resource.MustParse("4"), + hv1.ResourceCPU: resource.MustParse("4"), } res := newResizeReservation("res-1", "vm-uuid-1", "host-1", novaapi.ResizeIntent, reserved) hv := newHypervisor("host-1", "vm-uuid-1") @@ -486,7 +486,7 @@ func TestReconcile_NonResizeIntentSkipsVMClient(t *testing.T) { // all — we leave it nil to make an accidental call panic loudly. scheme := newTestScheme(t) res := newResizeReservation("res-1", "vm-uuid-1", "host-1", novaapi.LiveMigrationIntent, - map[hv1.ResourceName]resource.Quantity{"cpu": resource.MustParse("4")}) + map[hv1.ResourceName]resource.Quantity{hv1.ResourceCPU: resource.MustParse("4")}) hv := newHypervisor("host-1", "vm-uuid-1") k8sClient := newTestClient(scheme, res, hv) c := &Controller{Client: k8sClient, VMClient: nil} diff --git a/internal/scheduling/reservations/quota/config.go b/internal/scheduling/reservations/quota/config.go index 03e9378cd..02d6bfd08 100644 --- a/internal/scheduling/reservations/quota/config.go +++ b/internal/scheduling/reservations/quota/config.go @@ -18,7 +18,9 @@ type QuotaControllerConfig struct { FullReconcileInterval metav1.Duration `json:"fullReconcileInterval"` // CRStateFilter defines which CommittedResource states to include - // when summing cr_actual_usage. Default: ["confirmed", "guaranteed"] + // when summing cr_actual_usage. Empty means no states count; the + // deployment supplies the default (confirmed) via Helm. It MUST mirror + // the filter_quota_enforcement step's crStateFilter. CRStateFilter []v1alpha1.CommitmentStatus `json:"crStateFilter"` // EnableHVDiff enables incremental TotalUsage updates via HV instance diffs. @@ -31,15 +33,14 @@ type QuotaControllerConfig struct { FlavorGroupResourceConfig map[string]commitments.FlavorGroupResourcesConfig `json:"flavorGroupResourceConfig,omitempty"` } -// ApplyDefaults fills in any unset values with defaults. +// ApplyDefaults fills in any unset values with defaults. CRStateFilter is +// intentionally not defaulted here: an empty filter means no states count, and +// the deployment supplies the intended set (confirmed) via Helm. func (c *QuotaControllerConfig) ApplyDefaults() { defaults := DefaultQuotaControllerConfig() if c.FullReconcileInterval.Duration == 0 { c.FullReconcileInterval = defaults.FullReconcileInterval } - if len(c.CRStateFilter) == 0 { - c.CRStateFilter = defaults.CRStateFilter - } if c.EnableHVDiff == nil { c.EnableHVDiff = defaults.EnableHVDiff } @@ -53,15 +54,12 @@ func (c *QuotaControllerConfig) IsHVDiffEnabled() bool { return *c.EnableHVDiff } -// DefaultQuotaControllerConfig returns a default configuration. +// DefaultQuotaControllerConfig returns a default configuration. CRStateFilter +// has no code-level default; the deployment supplies it via Helm. func DefaultQuotaControllerConfig() QuotaControllerConfig { enableHVDiff := true return QuotaControllerConfig{ FullReconcileInterval: metav1.Duration{Duration: 5 * time.Minute}, - CRStateFilter: []v1alpha1.CommitmentStatus{ - v1alpha1.CommitmentStatusConfirmed, - v1alpha1.CommitmentStatusGuaranteed, - }, - EnableHVDiff: &enableHVDiff, + EnableHVDiff: &enableHVDiff, } } diff --git a/internal/scheduling/reservations/quota/controller.go b/internal/scheduling/reservations/quota/controller.go index a64ae07a5..24b3c2931 100644 --- a/internal/scheduling/reservations/quota/controller.go +++ b/internal/scheduling/reservations/quota/controller.go @@ -11,7 +11,7 @@ import ( "time" "github.com/cobaltcore-dev/cortex/api/v1alpha1" - "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" + knowledge "github.com/cobaltcore-dev/cortex/internal/knowledge/extractor/plugins/compute" "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations" commitments "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/commitments" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" @@ -74,8 +74,7 @@ func (c *QuotaController) ReconcilePeriodic(ctx context.Context) error { logger.Info("starting full quota reconcile") // Fetch flavor groups from Knowledge CRD - flavorGroupClient := &reservations.FlavorGroupKnowledgeClient{Client: c.Client} - flavorGroups, err := flavorGroupClient.GetAllFlavorGroups(ctx, nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, c.Client, nil) if err != nil { logger.Error(err, "failed to get flavor groups") c.Metrics.RecordReconcileResult(false) @@ -228,8 +227,7 @@ func (c *QuotaController) Reconcile(ctx context.Context, req ctrl.Request) (ctrl } // Fetch flavor groups for CRUsage computation - flavorGroupClient := &reservations.FlavorGroupKnowledgeClient{Client: c.Client} - flavorGroups, err := flavorGroupClient.GetAllFlavorGroups(ctx, nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, c.Client, nil) if err != nil { logger.Error(err, "failed to get flavor groups") return ctrl.Result{}, err @@ -267,8 +265,7 @@ func (c *QuotaController) Reconcile(ctx context.Context, req ctrl.Request) (ctrl // a ProjectQuota is first created and has no persisted TotalUsage yet. func (c *QuotaController) computeTotalUsageForProject(ctx context.Context, projectID string) (map[string]map[string]int64, error) { // Fetch flavor groups from Knowledge CRD - flavorGroupClient := &reservations.FlavorGroupKnowledgeClient{Client: c.Client} - flavorGroups, err := flavorGroupClient.GetAllFlavorGroups(ctx, nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, c.Client, nil) if err != nil { return nil, fmt.Errorf("failed to get flavor groups: %w", err) } @@ -362,8 +359,7 @@ func (c *QuotaController) ReconcileHVDiff(ctx context.Context, oldHV, newHV *hv1 logger.V(1).Info("HV instance diff detected", "added", len(added), "removed", len(removed)) // Get flavor groups for mapping - flavorGroupClient := &reservations.FlavorGroupKnowledgeClient{Client: c.Client} - flavorGroups, err := flavorGroupClient.GetAllFlavorGroups(ctx, nil) + flavorGroups, err := knowledge.GetAllFlavorGroups(ctx, c.Client, nil) if err != nil { logger.Error(err, "failed to get flavor groups for incremental update") return err @@ -408,7 +404,7 @@ func (c *QuotaController) accumulateAddedVM( ctx context.Context, vmUUID string, flavorToGroup map[string]string, - flavorGroups map[string]compute.FlavorGroupFeature, + flavorGroups map[string]knowledge.FlavorGroupFeature, projectDeltas map[string]*usageDelta, ) { @@ -502,7 +498,7 @@ func (c *QuotaController) accumulateRemovedVM( ctx context.Context, vmUUID string, flavorToGroup map[string]string, - flavorGroups map[string]compute.FlavorGroupFeature, + flavorGroups map[string]knowledge.FlavorGroupFeature, projectDeltas map[string]*usageDelta, ) { @@ -562,7 +558,7 @@ func (c *QuotaController) applyDeltaAndUpdateStatus( projectID string, delta *usageDelta, projectCRs []v1alpha1.CommittedResource, - flavorGroups map[string]compute.FlavorGroupFeature, + flavorGroups map[string]knowledge.FlavorGroupFeature, ) error { // Collect all AZs affected by this delta @@ -741,7 +737,7 @@ func (c *QuotaController) Start(ctx context.Context) error { func (c *QuotaController) computeTotalUsage( vms []reservations.VM, flavorToGroup map[string]string, - flavorGroups map[string]compute.FlavorGroupFeature, + flavorGroups map[string]knowledge.FlavorGroupFeature, ) map[string]map[string]map[string]int64 { // result[projectID][resourceName] = ResourceQuotaUsage{PerAZ: {az: amount}} result := make(map[string]map[string]map[string]int64) @@ -805,7 +801,7 @@ func groupCRsByProject(crs []v1alpha1.CommittedResource) map[string][]v1alpha1.C // computeCRUsage computes the committed resource usage from a pre-filtered slice of CRs for one project. // It reads UsedResources from each CR's status and converts to commitment units (multiples for RAM, raw for cores). -func (c *QuotaController) computeCRUsage(crs []v1alpha1.CommittedResource, flavorGroups map[string]compute.FlavorGroupFeature) map[string]map[string]int64 { +func (c *QuotaController) computeCRUsage(crs []v1alpha1.CommittedResource, flavorGroups map[string]knowledge.FlavorGroupFeature) map[string]map[string]int64 { result := make(map[string]map[string]int64) for i := range crs { @@ -823,8 +819,8 @@ func (c *QuotaController) computeCRUsage(crs []v1alpha1.CommittedResource, flavo continue } - // Get used amount from UsedResources map - if len(cr.Status.UsedResources) == 0 { + // Get used amount from Billing.UsedResources map + if cr.Status.Billing == nil || len(cr.Status.Billing.UsedResources) == 0 { continue } @@ -834,7 +830,7 @@ func (c *QuotaController) computeCRUsage(crs []v1alpha1.CommittedResource, flavo switch spec.ResourceType { case v1alpha1.CommittedResourceTypeMemory: resourceName = commitments.ResourceNameRAM(spec.FlavorGroupName) - memQty, ok := cr.Status.UsedResources["memory"] + memQty, ok := cr.Status.Billing.UsedResources[string(hv1.ResourceMemory)] if !ok { continue } @@ -850,7 +846,7 @@ func (c *QuotaController) computeCRUsage(crs []v1alpha1.CommittedResource, flavo usedAmount = usedBytes / (1024 * 1024 * 1024) case v1alpha1.CommittedResourceTypeCores: resourceName = commitments.ResourceNameCores(spec.FlavorGroupName) - cpuQty, ok := cr.Status.UsedResources["cpu"] + cpuQty, ok := cr.Status.Billing.UsedResources[string(hv1.ResourceCPU)] if !ok { continue } @@ -948,7 +944,7 @@ func (c *QuotaController) updateProjectQuotaStatusWithRetry( totalUsage map[string]map[string]int64, paygUsage map[string]map[string]int64, fullReconcile bool, - flavorGroups map[string]compute.FlavorGroupFeature, + flavorGroups map[string]knowledge.FlavorGroupFeature, ) error { return retry.RetryOnConflict(retry.DefaultRetry, func() error { @@ -1070,7 +1066,7 @@ func buildUsageSummary(usage map[string]int64) string { // For RAM resources in variable-ratio groups: value is already in GiB = declared units (1:1). // For cores and instances: value is 1:1 (no conversion). // Uses FlavorGroupFeature from the Knowledge CRD for RAM unit conversion (consistent with the usage API). -func buildLimesSummary(values map[string]int64, flavorGroups map[string]compute.FlavorGroupFeature) string { +func buildLimesSummary(values map[string]int64, flavorGroups map[string]knowledge.FlavorGroupFeature) string { if len(values) == 0 { return "" } @@ -1094,16 +1090,16 @@ func buildLimesSummary(values map[string]int64, flavorGroups map[string]compute. // vmResourceUnits computes RAM commitment units (GiB) and cores from a VM's resources. func vmResourceUnits(resources map[string]resource.Quantity) (ramGiB, cores int64) { - memQty := resources["memory"] + memQty := resources[string(hv1.ResourceMemory)] serverRAMMiB := memQty.Value() / (1024 * 1024) // bytes to MiB ramGiB = serverRAMMiB / 1024 // MiB to GiB (1 GiB per unit) - vcpuQty := resources["vcpus"] + vcpuQty := resources[string(hv1.ResourceCPU)] cores = vcpuQty.Value() return ramGiB, cores } // buildFlavorToGroupMap builds a flavorName → flavorGroupName lookup from flavor groups. -func buildFlavorToGroupMap(flavorGroups map[string]compute.FlavorGroupFeature) map[string]string { +func buildFlavorToGroupMap(flavorGroups map[string]knowledge.FlavorGroupFeature) map[string]string { result := make(map[string]string) for groupName, group := range flavorGroups { for _, flavor := range group.Flavors { @@ -1186,12 +1182,20 @@ func crUsedAmountChangePredicate() predicate.Predicate { if !ok1 || !ok2 { return false } - // Trigger if UsedResources changed - if len(oldCR.Status.UsedResources) != len(newCR.Status.UsedResources) { + // Trigger if Billing.UsedResources changed + oldUsed := map[string]resource.Quantity{} + newUsed := map[string]resource.Quantity{} + if oldCR.Status.Billing != nil { + oldUsed = oldCR.Status.Billing.UsedResources + } + if newCR.Status.Billing != nil { + newUsed = newCR.Status.Billing.UsedResources + } + if len(oldUsed) != len(newUsed) { return true } - for key, oldQty := range oldCR.Status.UsedResources { - newQty, ok := newCR.Status.UsedResources[key] + for key, oldQty := range oldUsed { + newQty, ok := newUsed[key] if !ok || oldQty.Cmp(newQty) != 0 { return true } diff --git a/internal/scheduling/reservations/quota/controller_test.go b/internal/scheduling/reservations/quota/controller_test.go index a5e5c937b..ac7eb4f6d 100644 --- a/internal/scheduling/reservations/quota/controller_test.go +++ b/internal/scheduling/reservations/quota/controller_test.go @@ -18,8 +18,16 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client/fake" ) +// testQuotaConfig returns a controller config mirroring the production +// deployment, where Helm supplies the CR state filter (confirmed). +func testQuotaConfig() QuotaControllerConfig { + c := DefaultQuotaControllerConfig() + c.CRStateFilter = []v1alpha1.CommitmentStatus{v1alpha1.CommitmentStatusConfirmed} + return c +} + func TestComputeTotalUsage(t *testing.T) { - ctrl := &QuotaController{Config: DefaultQuotaControllerConfig()} + ctrl := &QuotaController{Config: testQuotaConfig()} flavorGroups := map[string]compute.FlavorGroupFeature{ "hana_v2": { @@ -45,8 +53,8 @@ func TestComputeTotalUsage(t *testing.T) { ProjectID: "project-a", AvailabilityZone: "az-1", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("34359738368"), // 32768 MiB in bytes - "vcpus": resource.MustParse("8"), + string(hv1.ResourceMemory): resource.MustParse("34359738368"), // 32768 MiB in bytes + string(hv1.ResourceCPU): resource.MustParse("8"), }, }, { @@ -55,8 +63,8 @@ func TestComputeTotalUsage(t *testing.T) { ProjectID: "project-a", AvailabilityZone: "az-1", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("68719476736"), // 65536 MiB in bytes - "vcpus": resource.MustParse("16"), + string(hv1.ResourceMemory): resource.MustParse("68719476736"), // 65536 MiB in bytes + string(hv1.ResourceCPU): resource.MustParse("16"), }, }, { @@ -65,8 +73,8 @@ func TestComputeTotalUsage(t *testing.T) { ProjectID: "project-a", AvailabilityZone: "az-2", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("34359738368"), - "vcpus": resource.MustParse("8"), + string(hv1.ResourceMemory): resource.MustParse("34359738368"), + string(hv1.ResourceCPU): resource.MustParse("8"), }, }, { @@ -75,8 +83,8 @@ func TestComputeTotalUsage(t *testing.T) { ProjectID: "project-b", AvailabilityZone: "az-1", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("4294967296"), // 4096 MiB in bytes - "vcpus": resource.MustParse("2"), + string(hv1.ResourceMemory): resource.MustParse("4294967296"), // 4096 MiB in bytes + string(hv1.ResourceCPU): resource.MustParse("2"), }, }, { @@ -85,8 +93,8 @@ func TestComputeTotalUsage(t *testing.T) { ProjectID: "project-c", AvailabilityZone: "az-1", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("4294967296"), - "vcpus": resource.MustParse("2"), + string(hv1.ResourceMemory): resource.MustParse("4294967296"), + string(hv1.ResourceCPU): resource.MustParse("2"), }, }, } @@ -146,7 +154,7 @@ func TestComputeTotalUsage(t *testing.T) { } func TestComputeCRUsage(t *testing.T) { - ctrl := &QuotaController{Config: DefaultQuotaControllerConfig()} + ctrl := &QuotaController{Config: testQuotaConfig()} // Flavor groups — MemoryMB value is no longer used for unit conversion (now fixed at 1 GiB). testFlavorGroups := map[string]compute.FlavorGroupFeature{ @@ -166,7 +174,7 @@ func TestComputeCRUsage(t *testing.T) { State: v1alpha1.CommitmentStatusConfirmed, }, Status: v1alpha1.CommittedResourceStatus{ - UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("5Gi")}, + Billing: &v1alpha1.BillingStatus{UsedResources: map[string]resource.Quantity{string(hv1.ResourceMemory): resource.MustParse("5Gi")}}, }, }, { @@ -175,10 +183,10 @@ func TestComputeCRUsage(t *testing.T) { FlavorGroupName: "hana_v2", AvailabilityZone: "az-1", ResourceType: v1alpha1.CommittedResourceTypeMemory, - State: v1alpha1.CommitmentStatusGuaranteed, + State: v1alpha1.CommitmentStatusConfirmed, }, Status: v1alpha1.CommittedResourceStatus{ - UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("3Gi")}, + Billing: &v1alpha1.BillingStatus{UsedResources: map[string]resource.Quantity{string(hv1.ResourceMemory): resource.MustParse("3Gi")}}, }, }, { @@ -190,7 +198,7 @@ func TestComputeCRUsage(t *testing.T) { State: v1alpha1.CommitmentStatusConfirmed, }, Status: v1alpha1.CommittedResourceStatus{ - UsedResources: map[string]resource.Quantity{"cpu": resource.MustParse("2")}, + Billing: &v1alpha1.BillingStatus{UsedResources: map[string]resource.Quantity{string(hv1.ResourceCPU): resource.MustParse("2")}}, }, }, // Different project — should be excluded by groupCRsByProject @@ -203,7 +211,7 @@ func TestComputeCRUsage(t *testing.T) { State: v1alpha1.CommitmentStatusConfirmed, }, Status: v1alpha1.CommittedResourceStatus{ - UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("5Gi")}, + Billing: &v1alpha1.BillingStatus{UsedResources: map[string]resource.Quantity{string(hv1.ResourceMemory): resource.MustParse("5Gi")}}, }, }, // Pending state — should be excluded by state filter @@ -216,7 +224,7 @@ func TestComputeCRUsage(t *testing.T) { State: v1alpha1.CommitmentStatusPending, }, Status: v1alpha1.CommittedResourceStatus{ - UsedResources: map[string]resource.Quantity{"memory": resource.MustParse("2Gi")}, + Billing: &v1alpha1.BillingStatus{UsedResources: map[string]resource.Quantity{string(hv1.ResourceMemory): resource.MustParse("2Gi")}}, }, }, } @@ -225,7 +233,7 @@ func TestComputeCRUsage(t *testing.T) { crsByProject := groupCRsByProject(allCRs) result := ctrl.computeCRUsage(crsByProject["project-a"], testFlavorGroups) - // Should include confirmed + guaranteed for project-a only + // Should include confirmed CRs for project-a only ramUsage := result["hw_version_hana_v2_ram"] if ramUsage["az-1"] != 8 { // 5 + 3 t.Errorf("expected cr ram usage az-1 = 8, got %d", ramUsage["az-1"]) @@ -390,13 +398,13 @@ func TestIncrementDecrementUsage(t *testing.T) { } func TestIsCRStateIncluded(t *testing.T) { - ctrl := &QuotaController{Config: DefaultQuotaControllerConfig()} + ctrl := &QuotaController{Config: testQuotaConfig()} if !ctrl.isCRStateIncluded(v1alpha1.CommitmentStatusConfirmed) { t.Error("expected confirmed to be included") } - if !ctrl.isCRStateIncluded(v1alpha1.CommitmentStatusGuaranteed) { - t.Error("expected guaranteed to be included") + if ctrl.isCRStateIncluded(v1alpha1.CommitmentStatusGuaranteed) { + t.Error("expected guaranteed to NOT be included") } if ctrl.isCRStateIncluded(v1alpha1.CommitmentStatusPending) { t.Error("expected pending to NOT be included") @@ -449,7 +457,7 @@ func TestUsageDelta(t *testing.T) { func TestReconcile_NilTotalUsage(t *testing.T) { // When TotalUsage is nil, Reconcile should skip and return no error. // This validates the early-return branch logic used in Reconcile(). - ctrl := &QuotaController{Config: DefaultQuotaControllerConfig()} + ctrl := &QuotaController{Config: testQuotaConfig()} // computeCRUsage on nil slice should return empty map (no panic) result := ctrl.computeCRUsage(nil, nil) @@ -466,7 +474,7 @@ func TestReconcile_NilTotalUsage(t *testing.T) { func TestAccumulateAddedVM_UnknownFlavor(t *testing.T) { // Verifies that accumulateAddedVM gracefully handles a VM with an unknown flavor - ctrl := &QuotaController{Config: DefaultQuotaControllerConfig()} + ctrl := &QuotaController{Config: testQuotaConfig()} flavorGroups := map[string]compute.FlavorGroupFeature{ "hana_v2": { @@ -486,8 +494,8 @@ func TestAccumulateAddedVM_UnknownFlavor(t *testing.T) { ProjectID: "project-a", AvailabilityZone: "az-1", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("4294967296"), - "vcpus": resource.MustParse("2"), + string(hv1.ResourceMemory): resource.MustParse("4294967296"), + string(hv1.ResourceCPU): resource.MustParse("2"), }, }, nil }, @@ -529,7 +537,7 @@ func TestAccumulateAddedVM_KnownFlavor(t *testing.T) { qc := &QuotaController{ Client: k8sClient, - Config: DefaultQuotaControllerConfig(), + Config: testQuotaConfig(), } flavorGroups := map[string]compute.FlavorGroupFeature{ @@ -550,8 +558,8 @@ func TestAccumulateAddedVM_KnownFlavor(t *testing.T) { AvailabilityZone: "az-1", CreatedAt: vmCreatedAt, Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("34359738368"), // 32768 MiB - "vcpus": resource.MustParse("8"), + string(hv1.ResourceMemory): resource.MustParse("34359738368"), // 32768 MiB + string(hv1.ResourceCPU): resource.MustParse("8"), }, }, nil }, diff --git a/internal/scheduling/reservations/quota/integration_test.go b/internal/scheduling/reservations/quota/integration_test.go index 9f55a25f6..e1c0e259a 100644 --- a/internal/scheduling/reservations/quota/integration_test.go +++ b/internal/scheduling/reservations/quota/integration_test.go @@ -166,8 +166,8 @@ func TestIntegration(t *testing.T) { ProjectID: "project-a", AvailabilityZone: "az-1", CreatedAt: "2099-01-01T00:00:00Z", // far future, always AFTER last reconcile Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("34359738368"), // 32768 MiB = 32 GiB - "vcpus": resource.MustParse("8"), + string(hv1.ResourceMemory): resource.MustParse("34359738368"), // 32768 MiB = 32 GiB + string(hv1.ResourceCPU): resource.MustParse("8"), }, }, ), @@ -397,8 +397,8 @@ func TestIntegration(t *testing.T) { UUID: "vm-unknown", FlavorName: "nonexistent-flavor", ProjectID: "project-x", AvailabilityZone: "az-1", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("4294967296"), - "vcpus": resource.MustParse("2"), + string(hv1.ResourceMemory): resource.MustParse("4294967296"), + string(hv1.ResourceCPU): resource.MustParse("2"), }, }, }, @@ -540,8 +540,8 @@ func TestIntegration(t *testing.T) { ProjectID: "project-a", AvailabilityZone: "az-1", CreatedAt: "2099-01-01T00:00:00Z", // after last reconcile Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("34359738368"), // 32768 MiB = 32 GiB - "vcpus": resource.MustParse("8"), + string(hv1.ResourceMemory): resource.MustParse("34359738368"), // 32768 MiB = 32 GiB + string(hv1.ResourceCPU): resource.MustParse("8"), }, }, ), @@ -638,8 +638,8 @@ func TestIntegration(t *testing.T) { ProjectID: "project-a", AvailabilityZone: "az-1", CreatedAt: "2099-01-01T00:00:00Z", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("34359738368"), // 32768 MiB = 32 GiB - "vcpus": resource.MustParse("8"), + string(hv1.ResourceMemory): resource.MustParse("34359738368"), // 32768 MiB = 32 GiB + string(hv1.ResourceCPU): resource.MustParse("8"), }, }, ), @@ -671,8 +671,8 @@ func TestIntegration(t *testing.T) { ProjectID: "project-b", AvailabilityZone: "az-1", CreatedAt: "2099-01-01T00:00:00Z", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("4294967296"), // 4096 MiB = 4 GiB - "vcpus": resource.MustParse("2"), + string(hv1.ResourceMemory): resource.MustParse("4294967296"), // 4096 MiB = 4 GiB + string(hv1.ResourceCPU): resource.MustParse("2"), }, }, ), @@ -706,8 +706,8 @@ func TestIntegration(t *testing.T) { ProjectID: "project-a", AvailabilityZone: "az-1", CreatedAt: "2099-01-01T00:00:00Z", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("34359738368"), - "vcpus": resource.MustParse("8"), + string(hv1.ResourceMemory): resource.MustParse("34359738368"), + string(hv1.ResourceCPU): resource.MustParse("8"), }, }, ), @@ -737,8 +737,8 @@ func TestIntegration(t *testing.T) { ProjectID: "project-a", AvailabilityZone: "az-1", CreatedAt: "2099-01-01T00:00:00Z", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("34359738368"), - "vcpus": resource.MustParse("8"), + string(hv1.ResourceMemory): resource.MustParse("34359738368"), + string(hv1.ResourceCPU): resource.MustParse("8"), }, }, }, @@ -777,8 +777,8 @@ func TestIntegration(t *testing.T) { ProjectID: "project-a", AvailabilityZone: "az-1", CreatedAt: "2099-01-01T00:00:00Z", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("34359738368"), - "vcpus": resource.MustParse("8"), + string(hv1.ResourceMemory): resource.MustParse("34359738368"), + string(hv1.ResourceCPU): resource.MustParse("8"), }, }, ), @@ -960,8 +960,8 @@ var testVMs = []reservations.VM{ ProjectID: "project-a", AvailabilityZone: "az-1", CreatedAt: "2025-12-01T00:00:00Z", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("34359738368"), // 32768 MiB - "vcpus": resource.MustParse("8"), + string(hv1.ResourceMemory): resource.MustParse("34359738368"), // 32768 MiB + string(hv1.ResourceCPU): resource.MustParse("8"), }, }, // vm-2: hana_v2, 64 GiB RAM (65536/1024), 16 cores @@ -970,8 +970,8 @@ var testVMs = []reservations.VM{ ProjectID: "project-a", AvailabilityZone: "az-1", CreatedAt: "2025-12-01T00:00:00Z", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("68719476736"), // 65536 MiB - "vcpus": resource.MustParse("16"), + string(hv1.ResourceMemory): resource.MustParse("68719476736"), // 65536 MiB + string(hv1.ResourceCPU): resource.MustParse("16"), }, }, // vm-3: hana_v2, 32 GiB RAM (32768/1024), 8 cores @@ -980,8 +980,8 @@ var testVMs = []reservations.VM{ ProjectID: "project-a", AvailabilityZone: "az-2", CreatedAt: "2025-12-01T00:00:00Z", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("34359738368"), // 32768 MiB - "vcpus": resource.MustParse("8"), + string(hv1.ResourceMemory): resource.MustParse("34359738368"), // 32768 MiB + string(hv1.ResourceCPU): resource.MustParse("8"), }, }, // vm-4: general, 4 GiB RAM (4096/1024), 2 cores @@ -990,8 +990,8 @@ var testVMs = []reservations.VM{ ProjectID: "project-a", AvailabilityZone: "az-1", CreatedAt: "2025-12-01T00:00:00Z", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("4294967296"), // 4096 MiB - "vcpus": resource.MustParse("2"), + string(hv1.ResourceMemory): resource.MustParse("4294967296"), // 4096 MiB + string(hv1.ResourceCPU): resource.MustParse("2"), }, }, // vm-5: general, 4 GiB RAM (4096/1024), 2 cores @@ -1000,8 +1000,8 @@ var testVMs = []reservations.VM{ ProjectID: "project-b", AvailabilityZone: "az-1", CreatedAt: "2025-12-01T00:00:00Z", Resources: map[string]resource.Quantity{ - "memory": resource.MustParse("4294967296"), // 4096 MiB - "vcpus": resource.MustParse("2"), + string(hv1.ResourceMemory): resource.MustParse("4294967296"), // 4096 MiB + string(hv1.ResourceCPU): resource.MustParse("2"), }, }, } @@ -1135,7 +1135,7 @@ func newIntegrationTestEnv(t *testing.T, tc IntegrationTestCase) *integrationTes controller := &QuotaController{ Client: k8sClient, VMSource: vmSrc, - Config: DefaultQuotaControllerConfig(), + Config: testQuotaConfig(), Metrics: NewQuotaMetrics(nil), // no-op metrics } @@ -1305,7 +1305,7 @@ func (env *integrationTestEnv) executeAction(action TestAction) { if err := env.client.Get(ctx, client.ObjectKey{Name: action.CRName}, &cr); err != nil { env.t.Fatalf("failed to get CR %s: %v", action.CRName, err) } - cr.Status.UsedResources = usedResourcesFromMultiples(cr.Spec.ResourceType, cr.Spec.FlavorGroupName, action.UsedAmount) + cr.Status.Billing = &v1alpha1.BillingStatus{UsedResources: usedResourcesFromMultiples(cr.Spec.ResourceType, cr.Spec.FlavorGroupName, action.UsedAmount)} if err := env.client.Status().Update(ctx, &cr); err != nil { env.t.Fatalf("failed to update CR %s status: %v", action.CRName, err) } @@ -1430,7 +1430,7 @@ func makeCR(name, projectID, flavorGroup, az string, resourceType v1alpha1.Commi }, } if usedAmount != nil { - cr.Status.UsedResources = usedResourcesFromMultiples(resourceType, flavorGroup, *usedAmount) + cr.Status.Billing = &v1alpha1.BillingStatus{UsedResources: usedResourcesFromMultiples(resourceType, flavorGroup, *usedAmount)} } return cr } @@ -1446,11 +1446,11 @@ func usedResourcesFromMultiples(resourceType v1alpha1.CommittedResourceType, fla } bytesVal := multiples * 1024 * 1024 * 1024 return map[string]resource.Quantity{ - "memory": *resource.NewQuantity(bytesVal, resource.BinarySI), + string(hv1.ResourceMemory): *resource.NewQuantity(bytesVal, resource.BinarySI), } case v1alpha1.CommittedResourceTypeCores: return map[string]resource.Quantity{ - "cpu": *resource.NewQuantity(multiples, resource.DecimalSI), + string(hv1.ResourceCPU): *resource.NewQuantity(multiples, resource.DecimalSI), } default: return nil diff --git a/internal/scheduling/reservations/scheduler_client.go b/internal/scheduling/reservations/scheduler_client.go deleted file mode 100644 index e5ee2c3ee..000000000 --- a/internal/scheduling/reservations/scheduler_client.go +++ /dev/null @@ -1,231 +0,0 @@ -// Copyright SAP SE -// SPDX-License-Identifier: Apache-2.0 - -package reservations - -import ( - "bytes" - "context" - "encoding/json" - "fmt" - "io" - "net/http" - "time" - - api "github.com/cobaltcore-dev/cortex/api/external/nova" - "github.com/cobaltcore-dev/cortex/api/scheduling" - "github.com/go-logr/logr" - logf "sigs.k8s.io/controller-runtime/pkg/log" -) - -// DefaultHTTPTimeout is the default timeout for HTTP requests to the scheduler API. -const DefaultHTTPTimeout = 30 * time.Second - -var log = logf.Log.WithName("scheduler-client").WithValues("module", "reservations") - -// loggerFromContext returns a logger with greq and req values from the context. -func loggerFromContext(ctx context.Context) logr.Logger { - return log.WithValues( - "greq", GlobalRequestIDFromContext(ctx), - "req", RequestIDFromContext(ctx), - ) -} - -// NOTE+FIXME: we should not send ourselves REST API calls. This needs to be replaced by direct Go call (if possible) or communication via CRDs - -// SchedulerClient is a client for the external scheduler API. -// It can be used by both the ReservationReconciler and FailoverReservationController. -type SchedulerClient struct { - // URL of the external scheduler API. - URL string - // HTTP client to use for requests. - HTTPClient *http.Client -} - -// NewSchedulerClient creates a new SchedulerClient with a default timeout. -func NewSchedulerClient(url string) *SchedulerClient { - return &SchedulerClient{ - URL: url, - HTTPClient: &http.Client{ - Timeout: DefaultHTTPTimeout, - }, - } -} - -// ScheduleReservationRequest contains the parameters for scheduling a reservation. -type ScheduleReservationRequest struct { - // InstanceUUID is the unique identifier for the reservation (usually the reservation name). - InstanceUUID string - // ProjectID is the OpenStack project ID. - ProjectID string - // FlavorName is the name of the flavor. - FlavorName string - // FlavorExtraSpecs are extra specifications for the flavor. - FlavorExtraSpecs map[string]string - // MemoryMB is the memory in MB. - MemoryMB uint64 - // VCPUs is the number of virtual CPUs. - VCPUs uint64 - // EligibleHosts is the list of hosts that can be considered for placement. - EligibleHosts []api.ExternalSchedulerHost - // IgnoreHosts is a list of hosts to ignore during scheduling. - // This is used for failover reservations to avoid placing on the same host as the VMs. - IgnoreHosts []string - // Pipeline is the name of the pipeline to execute. - // If empty, the default pipeline will be used. - Pipeline string - // AvailabilityZone is the availability zone to schedule in. - // This is used by the filter_correct_az filter to ensure hosts are in the correct AZ. - AvailabilityZone string - // SchedulerHints are hints passed to the scheduler pipeline. - // Used to set _nova_check_type for evacuation intent detection. - SchedulerHints map[string]any - // ImageProperties are Glance-style image properties (e.g., "img_hv_type", "hypervisor_type") - // forwarded on Spec.Data.Image.Data.Properties.Data. Optional; when empty the outgoing - // request leaves the image field at its zero value. Used by FilterImagePropertiesStep. - ImageProperties map[string]any -} - -// ScheduleReservationResponse contains the result of scheduling a reservation. -type ScheduleReservationResponse struct { - // Hosts is the ordered list of hosts that the reservation can be placed on. - // The first host is the best choice. - Hosts []string -} - -// ScheduleReservation calls the external scheduler API to find a host for a reservation. -// The context should contain GlobalRequestID and RequestID for logging (use WithGlobalRequestID/WithRequestID). -func (c *SchedulerClient) ScheduleReservation(ctx context.Context, req ScheduleReservationRequest, opts scheduling.Options) (*ScheduleReservationResponse, error) { - logger := loggerFromContext(ctx) - - if err := opts.Validate(); err != nil { - return nil, fmt.Errorf("invalid scheduling options: %w", err) - } - - // Build weights map (all zero for reservations) - weights := make(map[string]float64, len(req.EligibleHosts)) - for _, host := range req.EligibleHosts { - weights[host.ComputeHost] = 0.0 - } - - // Build ignore hosts pointer - var ignoreHosts *[]string - if len(req.IgnoreHosts) > 0 { - ignoreHosts = &req.IgnoreHosts - } - - // Build the context with request IDs - var globalReqID *string - if greq := GlobalRequestIDFromContext(ctx); greq != "" { - globalReqID = &greq - } - - // Build the external scheduler request - externalSchedulerRequest := api.ExternalSchedulerRequest{ - Pipeline: req.Pipeline, - Hosts: req.EligibleHosts, - Weights: weights, - Options: opts, - Context: api.NovaRequestContext{ - RequestID: RequestIDFromContext(ctx), - GlobalRequestID: globalReqID, - ProjectID: req.ProjectID, - }, - Spec: api.NovaObject[api.NovaSpec]{ - Data: api.NovaSpec{ - InstanceUUID: req.InstanceUUID, - NumInstances: 1, // One for each reservation. - ProjectID: req.ProjectID, - AvailabilityZone: req.AvailabilityZone, - IgnoreHosts: ignoreHosts, - SchedulerHints: req.getSchedulerHints(), - Flavor: api.NovaObject[api.NovaFlavor]{ - Data: api.NovaFlavor{ - Name: req.FlavorName, - ExtraSpecs: req.FlavorExtraSpecs, - MemoryMB: req.MemoryMB, - VCPUs: req.VCPUs, - // Disk is currently not considered. - }, - }, - Image: api.NovaObject[api.NovaImageMeta]{ - Data: api.NovaImageMeta{ - Properties: api.NovaObject[map[string]any]{ - Data: req.ImageProperties, - }, - }, - }, - }, - }, - } - - logger.V(1).Info("sending external scheduler request", - "url", c.URL, - "pipeline", req.Pipeline, - "instanceUUID", req.InstanceUUID, - "flavorName", req.FlavorName, - "flavorExtraSpecs", req.FlavorExtraSpecs, - "eligibleHostsCount", len(req.EligibleHosts), - "ignoreHostsCount", len(req.IgnoreHosts), - "hasSchedulerHints", len(req.SchedulerHints) > 0, - "availabilityZone", req.AvailabilityZone) - - // Marshal the request - reqBody, err := json.Marshal(externalSchedulerRequest) - if err != nil { - logger.Error(err, "failed to marshal external scheduler request") - return nil, fmt.Errorf("failed to marshal external scheduler request: %w", err) - } - - // Create HTTP request with context - httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, c.URL, bytes.NewReader(reqBody)) - if err != nil { - logger.Error(err, "failed to create HTTP request") - return nil, fmt.Errorf("failed to create HTTP request: %w", err) - } - httpReq.Header.Set("Content-Type", "application/json") - - // Send the request and measure duration - startTime := time.Now() - response, err := c.HTTPClient.Do(httpReq) - duration := time.Since(startTime) - if err != nil { - logger.Error(err, "failed to send external scheduler request") - return nil, fmt.Errorf("failed to send external scheduler request: %w", err) - } - defer response.Body.Close() - - // Check response status - if response.StatusCode != http.StatusOK { - body, err := io.ReadAll(io.LimitReader(response.Body, 1024)) - if err != nil { - logger.Error(err, "failed to read error response body", "statusCode", response.StatusCode) - return nil, fmt.Errorf("external scheduler returned status %d", response.StatusCode) - } - logger.Error(nil, "external scheduler returned non-OK status", "statusCode", response.StatusCode, "body", string(body)) - return nil, fmt.Errorf("external scheduler returned status %d: %s", response.StatusCode, string(body)) - } - - // Decode the response - var externalSchedulerResponse api.ExternalSchedulerResponse - if err := json.NewDecoder(response.Body).Decode(&externalSchedulerResponse); err != nil { - logger.Error(err, "failed to decode external scheduler response") - return nil, fmt.Errorf("failed to decode external scheduler response: %w", err) - } - - logger.V(1).Info("received external scheduler response", - "hostsCount", len(externalSchedulerResponse.Hosts), - "duration", duration.Round(time.Millisecond)) - - return &ScheduleReservationResponse{ - Hosts: externalSchedulerResponse.Hosts, - }, nil -} - -// getSchedulerHints returns the scheduler hints, or an empty map if nil. -func (req ScheduleReservationRequest) getSchedulerHints() map[string]any { - if req.SchedulerHints == nil { - return make(map[string]any) - } - return req.SchedulerHints -} diff --git a/internal/scheduling/reservations/vm_source.go b/internal/scheduling/reservations/vm_source.go index ab5998b62..a32235417 100644 --- a/internal/scheduling/reservations/vm_source.go +++ b/internal/scheduling/reservations/vm_source.go @@ -38,7 +38,7 @@ type VM struct { // CreatedAt is the ISO 8601 timestamp when the VM was created in Nova. // Used by the quota controller to distinguish new VMs from migrations. CreatedAt string - // Resources contains the VM's resource allocations (e.g., "memory", "vcpus"). + // Resources contains the VM's resource allocations (e.g., "memory", "cpu"). Resources map[string]resource.Quantity // FlavorExtraSpecs contains the flavor's extra specifications (e.g., traits, capabilities). // This is used by filters like filter_has_requested_traits and filter_capabilities. @@ -130,8 +130,8 @@ func (s *DBVMSource) ListVMs(ctx context.Context) ([]VM, error) { continue } resources := map[string]resource.Quantity{ - "vcpus": *resource.NewQuantity(int64(flavor.VCPUs), resource.DecimalSI), //nolint:gosec - "memory": *resource.NewQuantity(int64(flavor.RAM)*1024*1024, resource.BinarySI), //nolint:gosec + string(hv1.ResourceCPU): *resource.NewQuantity(int64(flavor.VCPUs), resource.DecimalSI), //nolint:gosec + string(hv1.ResourceMemory): *resource.NewQuantity(int64(flavor.RAM)*1024*1024, resource.BinarySI), //nolint:gosec } vms = append(vms, VM{ UUID: server.ID, @@ -194,8 +194,8 @@ func (s *DBVMSource) ListVMsByProject(ctx context.Context, projectID string) ([] continue } resources := map[string]resource.Quantity{ - "vcpus": *resource.NewQuantity(int64(flavor.VCPUs), resource.DecimalSI), //nolint:gosec - "memory": *resource.NewQuantity(int64(flavor.RAM)*1024*1024, resource.BinarySI), //nolint:gosec + string(hv1.ResourceCPU): *resource.NewQuantity(int64(flavor.VCPUs), resource.DecimalSI), //nolint:gosec + string(hv1.ResourceMemory): *resource.NewQuantity(int64(flavor.RAM)*1024*1024, resource.BinarySI), //nolint:gosec } vms = append(vms, VM{ UUID: server.ID, @@ -235,8 +235,8 @@ func (s *DBVMSource) GetVM(ctx context.Context, vmUUID string) (*VM, error) { } resources := map[string]resource.Quantity{ - "vcpus": *resource.NewQuantity(int64(flavor.VCPUs), resource.DecimalSI), //nolint:gosec - "memory": *resource.NewQuantity(int64(flavor.RAM)*1024*1024, resource.BinarySI), //nolint:gosec + string(hv1.ResourceCPU): *resource.NewQuantity(int64(flavor.VCPUs), resource.DecimalSI), //nolint:gosec + string(hv1.ResourceMemory): *resource.NewQuantity(int64(flavor.RAM)*1024*1024, resource.BinarySI), //nolint:gosec } return &VM{ diff --git a/pkg/multicluster/client_test.go b/pkg/multicluster/client_test.go index f31b5e6d3..2d0b7b06a 100644 --- a/pkg/multicluster/client_test.go +++ b/pkg/multicluster/client_test.go @@ -563,8 +563,8 @@ func TestGVKFromHomeScheme_Success(t *testing.T) { }{ {"ConfigMap", &corev1.ConfigMap{}, configMapGVK}, {"ConfigMapList", &corev1.ConfigMapList{}, configMapListGVK}, - {"Decision", &v1alpha1.Decision{}, schema.GroupVersionKind{Group: "cortex.cloud", Version: "v1alpha1", Kind: "Decision"}}, - {"DecisionList", &v1alpha1.DecisionList{}, schema.GroupVersionKind{Group: "cortex.cloud", Version: "v1alpha1", Kind: "DecisionList"}}, + {"Pipeline", &v1alpha1.Pipeline{}, schema.GroupVersionKind{Group: "cortex.cloud", Version: "v1alpha1", Kind: "Pipeline"}}, + {"PipelineList", &v1alpha1.PipelineList{}, schema.GroupVersionKind{Group: "cortex.cloud", Version: "v1alpha1", Kind: "PipelineList"}}, } for _, tt := range tests { @@ -1778,7 +1778,7 @@ func TestClient_InitFromConf_GVKFormatting(t *testing.T) { expectedStr string }{ {"core ConfigMap", configMapGVK, "v1/ConfigMap"}, - {"cortex Decision", schema.GroupVersionKind{Group: "cortex.cloud", Version: "v1alpha1", Kind: "Decision"}, "cortex.cloud/v1alpha1/Decision"}, + {"cortex Pipeline", schema.GroupVersionKind{Group: "cortex.cloud", Version: "v1alpha1", Kind: "Pipeline"}, "cortex.cloud/v1alpha1/Pipeline"}, } for _, tt := range tests { diff --git a/postgres/Dockerfile b/postgres/Dockerfile index 4e9880a27..223438de8 100644 --- a/postgres/Dockerfile +++ b/postgres/Dockerfile @@ -1,4 +1,4 @@ -FROM debian:trixie-slim@sha256:f6b08b09a05725f9431502d3b5714f5c4576e82c1a84fa1debe0c24a7e0c3ec4 +FROM debian:trixie-slim@sha256:a72b4433de1cf7d6bae0c4cee74d0547f58601a643b3375f070c51600d63b6b2 # explicitly set user/group IDs RUN set -eux; \ diff --git a/tools/visualize-committed-resources/main.go b/tools/visualize-committed-resources/main.go deleted file mode 100644 index 4cb99ce7c..000000000 --- a/tools/visualize-committed-resources/main.go +++ /dev/null @@ -1,659 +0,0 @@ -// Copyright SAP SE -// SPDX-License-Identifier: Apache-2.0 - -// Tool to visualize CommittedResource CRDs and their child Reservation slots. -// -// Usage: -// -// go run tools/visualize-committed-resources/main.go [flags] -// -// Flags: -// -// --contexts=ctx1,ctx2 Kubernetes contexts to query (default: current context) -// --filter-project=id Show only CRs for this project ID (substring match) -// --filter-az=az Show only CRs in this availability zone (substring match) -// --filter-group=name Show only CRs for this flavor group (substring match) -// --filter-state=state Show only CRs in this state (e.g. confirmed, reserving) -// --active Shorthand: show only confirmed/guaranteed CRs -// --views=v1,v2,... Views to show (default: all). Available: summary, commitments, reservations, allocations, usage -// --hide=v1,v2,... Views to hide (applied after --views) -// --watch=interval Refresh interval (e.g. 2s, 5s). Clears screen between refreshes. -package main - -import ( - "context" - "flag" - "fmt" - "os" - "sort" - "strconv" - "strings" - "time" - - "github.com/cobaltcore-dev/cortex/api/v1alpha1" - apimeta "k8s.io/apimachinery/pkg/api/meta" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/runtime" - utilruntime "k8s.io/apimachinery/pkg/util/runtime" - "k8s.io/client-go/tools/clientcmd" - "sigs.k8s.io/controller-runtime/pkg/client" -) - -var scheme = runtime.NewScheme() - -func init() { - utilruntime.Must(v1alpha1.AddToScheme(scheme)) -} - -// ── ANSI colours ────────────────────────────────────────────────────────────── - -const ( - colReset = "\033[0m" - colBold = "\033[1m" - colGreen = "\033[32m" - colYellow = "\033[33m" - colRed = "\033[31m" - colCyan = "\033[36m" - colBlue = "\033[34m" - colGray = "\033[90m" -) - -func green(s string) string { return colGreen + s + colReset } -func yellow(s string) string { return colYellow + s + colReset } -func red(s string) string { return colRed + s + colReset } -func cyan(s string) string { return colCyan + s + colReset } -func blue(s string) string { return colBlue + s + colReset } -func gray(s string) string { return colGray + s + colReset } -func bold(s string) string { return colBold + s + colReset } - -func resourceTypeBadge(rt v1alpha1.CommittedResourceType) string { - switch rt { - case v1alpha1.CommittedResourceTypeCores: - return yellow("[CPU]") - case v1alpha1.CommittedResourceTypeMemory: - return blue("[RAM]") - default: - return gray("[?]") - } -} - -// ── Views ───────────────────────────────────────────────────────────────────── - -const ( - viewSummary = "summary" - viewCommitments = "commitments" - viewReservations = "reservations" - viewAllocations = "allocations" - viewUsage = "usage" -) - -var allViews = []string{viewSummary, viewCommitments, viewReservations, viewAllocations, viewUsage} - -type viewSet map[string]bool - -func parseViews(s string) viewSet { - vs := make(viewSet) - if s == "all" || s == "" { - for _, v := range allViews { - vs[v] = true - } - return vs - } - for _, v := range strings.Split(s, ",") { - vs[strings.TrimSpace(v)] = true - } - return vs -} - -func (vs viewSet) hide(s string) { - if s == "" { - return - } - for _, v := range strings.Split(s, ",") { - delete(vs, strings.TrimSpace(v)) - } -} - -func (vs viewSet) has(v string) bool { return vs[v] } - -// ── k8s client ──────────────────────────────────────────────────────────────── - -func getClientForContext(contextName string) (client.Client, error) { - kubeConfig := clientcmd.NewNonInteractiveDeferredLoadingClientConfig( - clientcmd.NewDefaultClientConfigLoadingRules(), - &clientcmd.ConfigOverrides{CurrentContext: contextName}, - ) - c, err := kubeConfig.ClientConfig() - if err != nil { - return nil, fmt.Errorf("getting kubeconfig for context %q: %w", contextName, err) - } - return client.New(c, client.Options{Scheme: scheme}) -} - -type contextClient struct { - name string - client client.Client -} - -func contextDisplayName(ctx string) string { - if ctx == "" { - return "(current)" - } - return ctx -} - -// ── helpers ─────────────────────────────────────────────────────────────────── - -func printHeader(title string) { - line := strings.Repeat("─", 80) - fmt.Println() - fmt.Println(bold(line)) - fmt.Printf("%s %s\n", bold("▶"), bold(title)) - fmt.Println(bold(line)) -} - -func truncate(s string, n int) string { - if len(s) <= n { - return s - } - return s[:n-1] + "…" -} - -func age(t *metav1.Time) string { - if t == nil { - return gray("—") - } - d := time.Since(t.Time).Round(time.Second) - switch { - case d < time.Minute: - return fmt.Sprintf("%ds", int(d.Seconds())) - case d < time.Hour: - return fmt.Sprintf("%dm", int(d.Minutes())) - case d < 24*time.Hour: - return fmt.Sprintf("%dh", int(d.Hours())) - default: - return fmt.Sprintf("%dd", int(d.Hours()/24)) - } -} - -func crReadyStatus(cr v1alpha1.CommittedResource) string { - cond := apimeta.FindStatusCondition(cr.Status.Conditions, v1alpha1.CommittedResourceConditionReady) - if cond == nil { - return gray("unknown") - } - switch cond.Reason { - case v1alpha1.CommittedResourceReasonAccepted: - return green("Accepted") - case v1alpha1.CommittedResourceReasonRejected: - return red("Rejected") - case v1alpha1.CommittedResourceReasonReserving: - return yellow("Reserving") - case v1alpha1.CommittedResourceReasonPlanned: - return gray("Planned") - default: - return yellow(cond.Reason) - } -} - -func resReadyStatus(res v1alpha1.Reservation) string { - cond := apimeta.FindStatusCondition(res.Status.Conditions, v1alpha1.ReservationConditionReady) - if cond == nil { - return gray("pending") - } - if cond.Status == metav1.ConditionTrue { - return green("Ready") - } - return red("NotReady: " + truncate(cond.Message, 40)) -} - -func stateColour(state v1alpha1.CommitmentStatus) string { - switch state { - case v1alpha1.CommitmentStatusConfirmed, v1alpha1.CommitmentStatusGuaranteed: - return green(string(state)) - case v1alpha1.CommitmentStatusPlanned, v1alpha1.CommitmentStatusPending: - return yellow(string(state)) - case v1alpha1.CommitmentStatusExpired, v1alpha1.CommitmentStatusSuperseded: - return gray(string(state)) - default: - return string(state) - } -} - -// ── filters ─────────────────────────────────────────────────────────────────── - -type filters struct { - project string - az string - group string - state string - active bool -} - -func (f filters) match(cr v1alpha1.CommittedResource) bool { - if f.project != "" && !strings.Contains(cr.Spec.ProjectID, f.project) { - return false - } - if f.az != "" && !strings.Contains(cr.Spec.AvailabilityZone, f.az) { - return false - } - if f.group != "" && !strings.Contains(cr.Spec.FlavorGroupName, f.group) { - return false - } - if f.state != "" && !strings.EqualFold(string(cr.Spec.State), f.state) { - return false - } - if f.active { - s := cr.Spec.State - if s != v1alpha1.CommitmentStatusConfirmed && s != v1alpha1.CommitmentStatusGuaranteed { - return false - } - } - return true -} - -// ── views ───────────────────────────────────────────────────────────────────── - -func printSummary(crs []v1alpha1.CommittedResource, reservations []v1alpha1.Reservation) { - printHeader("Summary") - - byState := make(map[v1alpha1.CommitmentStatus]int) - byReady := map[string]int{"Accepted": 0, "Reserving": 0, "Rejected": 0, "Planned": 0, "Unknown": 0} - for _, cr := range crs { - byState[cr.Spec.State]++ - cond := apimeta.FindStatusCondition(cr.Status.Conditions, v1alpha1.CommittedResourceConditionReady) - if cond == nil { - byReady["Unknown"]++ - } else { - byReady[cond.Reason]++ - } - } - - resReady, resNotReady, resPending := 0, 0, 0 - for _, res := range reservations { - cond := apimeta.FindStatusCondition(res.Status.Conditions, v1alpha1.ReservationConditionReady) - switch { - case cond == nil: - resPending++ - case cond.Status == metav1.ConditionTrue: - resReady++ - default: - resNotReady++ - } - } - - fmt.Printf(" CommittedResources : %s\n", bold(fmt.Sprintf("%d total", len(crs)))) - for _, s := range []v1alpha1.CommitmentStatus{ - v1alpha1.CommitmentStatusConfirmed, - v1alpha1.CommitmentStatusGuaranteed, - v1alpha1.CommitmentStatusPending, - v1alpha1.CommitmentStatusPlanned, - v1alpha1.CommitmentStatusExpired, - v1alpha1.CommitmentStatusSuperseded, - } { - if n := byState[s]; n > 0 { - fmt.Printf(" %-14s %d\n", string(s)+":", n) - } - } - fmt.Println() - fmt.Printf(" Ready conditions : %s accepted, %s reserving, %s rejected\n", - green(strconv.Itoa(byReady["Accepted"])), - yellow(strconv.Itoa(byReady["Reserving"])), - red(strconv.Itoa(byReady["Rejected"])), - ) - fmt.Println() - fmt.Printf(" Reservation slots : %s total — %s ready, %s not-ready, %s pending\n", - bold(strconv.Itoa(len(reservations))), - green(strconv.Itoa(resReady)), - red(strconv.Itoa(resNotReady)), - yellow(strconv.Itoa(resPending)), - ) -} - -func printCommitments(crs []v1alpha1.CommittedResource, showUsage bool) { - printHeader(fmt.Sprintf("CommittedResources (%d)", len(crs))) - - if len(crs) == 0 { - fmt.Println(gray(" (none)")) - return - } - - for _, cr := range crs { - accepted := gray("—") - if cr.Status.AcceptedSpec != nil { - accepted = cr.Status.AcceptedSpec.Amount.String() - } - - endStr := gray("no expiry") - if cr.Spec.EndTime != nil { - remaining := time.Until(cr.Spec.EndTime.Time).Round(time.Minute) - if remaining < 0 { - endStr = red(fmt.Sprintf("expired %s ago", age(cr.Spec.EndTime))) - } else { - endStr = gray(fmt.Sprintf("exp in %s", remaining)) - } - } - - // Line 1: identity + type + ready + state + group + az - fmt.Printf("\n %s %s %s %s group=%s az=%s age=%s\n", - bold(cyan(cr.Spec.CommitmentUUID)), - resourceTypeBadge(cr.Spec.ResourceType), - crReadyStatus(cr), - stateColour(cr.Spec.State), - cr.Spec.FlavorGroupName, - cr.Spec.AvailabilityZone, - age(&cr.CreationTimestamp), - ) - // Line 2: project + amount + expiry - fmt.Printf(" project=%s amount=%s accepted=%s %s\n", - cr.Spec.ProjectID, - cr.Spec.Amount.String(), - accepted, - endStr, - ) - - // Line 3 (optional): usage - if mem, ok := cr.Status.UsedResources["memory"]; ok { - cpu := cr.Status.UsedResources["cpu"] - usageAgeStr := gray("—") - if cr.Status.LastUsageReconcileAt != nil { - usageAgeStr = age(cr.Status.LastUsageReconcileAt) - } - fmt.Printf(" used=%s cpu=%s instances=%d usage-age=%s\n", - mem.String(), cpu.String(), len(cr.Status.AssignedInstances), usageAgeStr) - } - - if showUsage && len(cr.Status.AssignedInstances) > 0 { - fmt.Printf(" assigned instances (%d):\n", len(cr.Status.AssignedInstances)) - for _, inst := range cr.Status.AssignedInstances { - fmt.Printf(" %s\n", gray(inst)) - } - } - } -} - -func printReservations(crs []v1alpha1.CommittedResource, reservations []v1alpha1.Reservation, showAllocations bool) { - // Index reservations by CommitmentUUID for display under each CR. - byUUID := make(map[string][]v1alpha1.Reservation) - for _, res := range reservations { - if res.Spec.CommittedResourceReservation == nil { - continue - } - uuid := res.Spec.CommittedResourceReservation.CommitmentUUID - byUUID[uuid] = append(byUUID[uuid], res) - } - - printHeader("Reservation Slots") - - if len(reservations) == 0 { - fmt.Println(gray(" (none)")) - return - } - - for _, cr := range crs { - slots := byUUID[cr.Spec.CommitmentUUID] - if len(slots) == 0 { - continue - } - fmt.Printf("\n %s %s %s\n", - bold(cyan(cr.Spec.CommitmentUUID)), - gray(cr.Spec.FlavorGroupName), - gray(fmt.Sprintf("%d slot(s)", len(slots))), - ) - - sort.Slice(slots, func(i, j int) bool { - return slots[i].Name < slots[j].Name - }) - - for _, res := range slots { - targetHost := res.Spec.TargetHost - statusHost := res.Status.Host - var hostStr string - switch { - case statusHost == "": - hostStr = yellow(targetHost) + gray(" (not yet placed)") - case statusHost != targetHost: - hostStr = red(fmt.Sprintf("target=%s status=%s (migrating?)", targetHost, statusHost)) - default: - hostStr = green(targetHost) - } - - genOK := "" - if s := res.Status.CommittedResourceReservation; s != nil { - spec := res.Spec.CommittedResourceReservation - if spec != nil && s.ObservedParentGeneration != spec.ParentGeneration { - genOK = yellow(fmt.Sprintf(" [gen: spec=%d observed=%d]", - spec.ParentGeneration, s.ObservedParentGeneration)) - } - } - - resources := "" - var resourcesSb391 strings.Builder - for rname, qty := range res.Spec.Resources { - fmt.Fprintf(&resourcesSb391, "%s=%s ", rname, qty.String()) - } - resources += resourcesSb391.String() - - fmt.Printf(" %s host=%s %s %s%s\n", - truncate(res.Name, 40), - hostStr, - resReadyStatus(res), - gray(strings.TrimSpace(resources)), - genOK, - ) - - if showAllocations { - specAllocs := 0 - statusAllocs := 0 - if res.Spec.CommittedResourceReservation != nil { - specAllocs = len(res.Spec.CommittedResourceReservation.Allocations) - } - if res.Status.CommittedResourceReservation != nil { - statusAllocs = len(res.Status.CommittedResourceReservation.Allocations) - } - - if specAllocs > 0 || statusAllocs > 0 { - fmt.Printf(" allocations: spec=%d confirmed=%d\n", specAllocs, statusAllocs) - if res.Spec.CommittedResourceReservation != nil { - statusAlloc := map[string]string{} - if res.Status.CommittedResourceReservation != nil { - statusAlloc = res.Status.CommittedResourceReservation.Allocations - } - for vmUUID, alloc := range res.Spec.CommittedResourceReservation.Allocations { - resources := "" - var resourcesSb422 strings.Builder - for rname, qty := range alloc.Resources { - fmt.Fprintf(&resourcesSb422, "%s=%s ", rname, qty.String()) - } - resources += resourcesSb422.String() - confirmedHost, confirmed := statusAlloc[vmUUID] - state := "" - if confirmed { - state = green("confirmed on " + confirmedHost) - } else { - state = yellow(fmt.Sprintf("spec-only (grace since %s)", age(&alloc.CreationTimestamp))) - } - fmt.Printf(" vm=%s %s %s\n", - truncate(vmUUID, 36), - gray(strings.TrimSpace(resources)), - state, - ) - } - } - } - } - } - } -} - -// ── main ────────────────────────────────────────────────────────────────────── - -func main() { - contextsFlag := flag.String("contexts", "", "Comma-separated Kubernetes contexts to query (default: current context)") - filterProject := flag.String("filter-project", "", "Show only CRs for this project ID (substring match)") - filterAZ := flag.String("filter-az", "", "Show only CRs in this availability zone (substring match)") - filterGroup := flag.String("filter-group", "", "Show only CRs for this flavor group (substring match)") - filterState := flag.String("filter-state", "", "Show only CRs in this state") - activeOnly := flag.Bool("active", false, "Show only confirmed/guaranteed CRs") - viewsFlag := flag.String("views", "all", "Views: all, summary, commitments, reservations, allocations") - hideFlag := flag.String("hide", "", "Views to hide (applied after --views)") - watchInterval := flag.Duration("watch", 0, "Refresh interval (e.g. 2s, 5s). 0 = run once.") - limitFlag := flag.Int("limit", 200, "Max CRs to fetch (0 = unlimited)") - flag.Parse() - - views := parseViews(*viewsFlag) - views.hide(*hideFlag) - - f := filters{ - project: *filterProject, - az: *filterAZ, - group: *filterGroup, - state: *filterState, - active: *activeOnly, - } - - contextNames := []string{""} - if *contextsFlag != "" { - contextNames = strings.FieldsFunc(*contextsFlag, func(r rune) bool { return r == ',' }) - for i := range contextNames { - contextNames[i] = strings.TrimSpace(contextNames[i]) - } - } - var clients []contextClient - for _, name := range contextNames { - cl, err := getClientForContext(name) - if err != nil { - fmt.Fprintf(os.Stderr, "error creating client for context %q: %v\n", contextDisplayName(name), err) - os.Exit(1) - } - clients = append(clients, contextClient{name: name, client: cl}) - } - - ctx := context.Background() - var prevDigest string - first := true - for { - crs, reservations := fetchSnapshot(ctx, clients, f, *limitFlag) - if d := snapshotDigest(crs, reservations); first || d != prevDigest { - if !first { - fmt.Printf("\n%s %s %s\n", - bold("━━━ changed at"), - bold(time.Now().Format(time.RFC3339)), - bold("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"), - ) - } - printSnapshot(crs, reservations, f, views) - prevDigest = d - first = false - } - if *watchInterval == 0 { - break - } - time.Sleep(*watchInterval) - } -} - -// snapshotDigest returns a string that changes whenever any CRD is added, removed, or updated. -func snapshotDigest(crs []v1alpha1.CommittedResource, reservations []v1alpha1.Reservation) string { - var b strings.Builder - for _, cr := range crs { - fmt.Fprintf(&b, "%s:%s ", cr.Name, cr.ResourceVersion) - } - for _, res := range reservations { - fmt.Fprintf(&b, "%s:%s ", res.Name, res.ResourceVersion) - } - return b.String() -} - -func fetchSnapshot(ctx context.Context, clients []contextClient, f filters, limit int) ([]v1alpha1.CommittedResource, []v1alpha1.Reservation) { - multiContext := len(clients) > 1 - - var allCRs []v1alpha1.CommittedResource - var allReservations []v1alpha1.Reservation - - for _, cc := range clients { - var listOpts []client.ListOption - if limit > 0 { - listOpts = append(listOpts, client.Limit(int64(limit))) - } - - var crList v1alpha1.CommittedResourceList - if err := cc.client.List(ctx, &crList, listOpts...); err != nil { - fmt.Fprintf(os.Stderr, "warning: error listing CommittedResources in context %q: %v\n", contextDisplayName(cc.name), err) - continue - } - var resList v1alpha1.ReservationList - if err := cc.client.List(ctx, &resList, append(listOpts, client.MatchingLabels{ - v1alpha1.LabelReservationType: v1alpha1.ReservationTypeLabelCommittedResource, - })...); err != nil { - fmt.Fprintf(os.Stderr, "warning: error listing Reservations in context %q: %v\n", contextDisplayName(cc.name), err) - continue - } - - if crList.Continue != "" { - fmt.Fprintf(os.Stderr, yellow("warning: CR list truncated at %d in context %q — use --limit=0 or a higher value\n"), limit, contextDisplayName(cc.name)) - } - if resList.Continue != "" { - fmt.Fprintf(os.Stderr, yellow("warning: Reservation list truncated at %d in context %q — use --limit=0 or a higher value\n"), limit, contextDisplayName(cc.name)) - } - - for _, cr := range crList.Items { - if f.match(cr) { - if multiContext { - cr.Name = cr.Name + "@" + contextDisplayName(cc.name) - } - allCRs = append(allCRs, cr) - } - } - for _, res := range resList.Items { - if res.Spec.CommittedResourceReservation == nil { - continue - } - if multiContext { - res.Name = res.Name + "@" + contextDisplayName(cc.name) - } - allReservations = append(allReservations, res) - } - } - - sort.Slice(allCRs, func(i, j int) bool { - if allCRs[i].Spec.FlavorGroupName != allCRs[j].Spec.FlavorGroupName { - return allCRs[i].Spec.FlavorGroupName < allCRs[j].Spec.FlavorGroupName - } - return allCRs[i].Spec.CommitmentUUID < allCRs[j].Spec.CommitmentUUID - }) - - matchedUUIDs := make(map[string]bool, len(allCRs)) - for _, cr := range allCRs { - matchedUUIDs[cr.Spec.CommitmentUUID] = true - } - var reservations []v1alpha1.Reservation - for _, res := range allReservations { - if matchedUUIDs[res.Spec.CommittedResourceReservation.CommitmentUUID] { - reservations = append(reservations, res) - } - } - return allCRs, reservations -} - -func printSnapshot(crs []v1alpha1.CommittedResource, reservations []v1alpha1.Reservation, f filters, views viewSet) { - fmt.Printf("\n%s — %s\n", - bold("visualize-committed-resources"), - gray(time.Now().Format(time.RFC3339)), - ) - if f.project != "" || f.az != "" || f.group != "" || f.state != "" || f.active { - fmt.Printf("%s project=%q az=%q group=%q state=%q active=%v\n", - gray("filters:"), f.project, f.az, f.group, f.state, f.active) - } - - if views.has(viewSummary) { - printSummary(crs, reservations) - } - if views.has(viewCommitments) { - printCommitments(crs, views.has(viewUsage)) - } - if views.has(viewReservations) { - printReservations(crs, reservations, views.has(viewAllocations)) - } - - fmt.Println() -} diff --git a/tools/visualize-reservations/main.go b/tools/visualize-reservations/main.go index c99ff2eb1..8f4055103 100644 --- a/tools/visualize-reservations/main.go +++ b/tools/visualize-reservations/main.go @@ -822,10 +822,10 @@ func main() { var sizeStr string if len(res.Spec.Resources) > 0 { var parts []string - if vcpus, ok := res.Spec.Resources["vcpus"]; ok { + if vcpus, ok := res.Spec.Resources[hv1.ResourceCPU]; ok { parts = append(parts, fmt.Sprintf("%dvcpu", vcpus.Value())) } - if mem, ok := res.Spec.Resources["memory"]; ok { + if mem, ok := res.Spec.Resources[hv1.ResourceMemory]; ok { memGB := mem.Value() / (1024 * 1024 * 1024) parts = append(parts, fmt.Sprintf("%dGB", memGB)) } @@ -1156,7 +1156,7 @@ func main() { for name, qty := range res.Spec.Resources { // Convert to appropriate units switch name { - case "memory": + case hv1.ResourceMemory: // Memory is in bytes (resource.Quantity.Value() returns bytes) // 4080Mi = 4080 * 1024 * 1024 bytes memBytes := qty.Value() @@ -1168,7 +1168,7 @@ func main() { memMB := memBytes / (1024 * 1024) sizeInfo = append(sizeInfo, fmt.Sprintf("%s: %dMB", name, memMB)) } - case "cpu": + case hv1.ResourceCPU: sizeInfo = append(sizeInfo, fmt.Sprintf("%s: %d", name, qty.Value())) default: sizeInfo = append(sizeInfo, fmt.Sprintf("%s: %s", name, qty.String())) @@ -1427,10 +1427,10 @@ func main() { // Calculate total reservation resources for _, res := range failoverReservations { if len(res.Spec.Resources) > 0 { - if vcpus, ok := res.Spec.Resources["vcpus"]; ok { + if vcpus, ok := res.Spec.Resources[hv1.ResourceCPU]; ok { totalResCPU += vcpus.Value() } - if mem, ok := res.Spec.Resources["memory"]; ok { + if mem, ok := res.Spec.Resources[hv1.ResourceMemory]; ok { totalResRAMGB += mem.Value() / (1024 * 1024 * 1024) } } @@ -1513,42 +1513,40 @@ func printHypervisorSummary(hypervisors []hv1.Hypervisor, reservations []v1alpha } // Get capacity from hypervisor status - if cpu, ok := hv.Status.Capacity["cpu"]; ok { + if cpu, ok := hv.Status.Capacity[hv1.ResourceCPU]; ok { summary.CapacityCPU = cpu.Value() } - if mem, ok := hv.Status.Capacity["memory"]; ok { + if mem, ok := hv.Status.Capacity[hv1.ResourceMemory]; ok { // Memory is in bytes, convert to GB summary.CapacityMemoryGB = mem.Value() / (1024 * 1024 * 1024) } // Get allocation (used by VMs) from hypervisor status - if cpu, ok := hv.Status.Allocation["cpu"]; ok { + if cpu, ok := hv.Status.Allocation[hv1.ResourceCPU]; ok { summary.UsedByVMsCPU = cpu.Value() } - if mem, ok := hv.Status.Allocation["memory"]; ok { + if mem, ok := hv.Status.Allocation[hv1.ResourceMemory]; ok { // Memory is in bytes, convert to GB summary.UsedByVMsMemoryGB = mem.Value() / (1024 * 1024 * 1024) } // Get failover reservation resources - // Note: Reservations use "vcpus" key, not "cpu" if failoverRes, ok := failoverResPerHost[hv.Name]; ok { - if cpu, ok := failoverRes["vcpus"]; ok { + if cpu, ok := failoverRes[string(hv1.ResourceCPU)]; ok { summary.FailoverResCPU = cpu } - if mem, ok := failoverRes["memory"]; ok { + if mem, ok := failoverRes[string(hv1.ResourceMemory)]; ok { // Memory from reservations is in bytes summary.FailoverResMemGB = mem / (1024 * 1024 * 1024) } } // Get committed reservation resources - // Note: Reservations use "vcpus" key, not "cpu" if committedRes, ok := committedResPerHost[hv.Name]; ok { - if cpu, ok := committedRes["vcpus"]; ok { + if cpu, ok := committedRes[string(hv1.ResourceCPU)]; ok { summary.CommittedResCPU = cpu } - if mem, ok := committedRes["memory"]; ok { + if mem, ok := committedRes[string(hv1.ResourceMemory)]; ok { // Memory from reservations is in bytes summary.CommittedResMemGB = mem / (1024 * 1024 * 1024) }