Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions api/v1alpha1/reservation_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -178,17 +178,41 @@ type FailoverReservationSpec struct {
// in buildup.
type InFlightReservationSpec struct {
// VMID is the OpenStack server uuid from Nova assigned to the virtual
// machine expected to land on this reservation slot.
// machine expected to land on this reservation slot. It identifies the
// subject of the reservation (what is being placed), which is distinct
// from the reservation's owner (what controls its lifecycle, expressed via
// ownerReferences).
//
// TODO: generalize this into a domain-agnostic subject identifier once a
// non-Nova use case needs in-flight reservations. For now (KISS) we always
// have a VM uuid, so we keep this Nova-specific field.
VMID string `json:"vmID,omitempty"`
// UserID is the identifier of the user who owns the virtual machine.
UserID string `json:"userID,omitempty"`
// ProjectID is the identifier of the project/tenant that owns
// the virtual machine.
ProjectID string `json:"projectID,omitempty"`
// Intent defines which kind of virtual machine lifecycle operation
// triggered the placement of this in-flight reservation.
// triggered the placement of this in-flight reservation. The reconciler
// dispatches on this to decide which terminal signal releases the
// reservation.
// +kubebuilder:validation:Optional
Intent SchedulingIntent `json:"intent"`
// OperationID groups all in-flight reservations created within a single
// scheduling call (e.g. one reservation per candidate host for an initial
// placement, or target + source host for a live migration). It is a
// Cortex-generated, time-sortable uuid (UUIDv7), independent of any
// external caller's request id, so uniqueness does not depend on callers.
//
// When one operation reaches a terminal state, the reconciler releases
// exactly the reservations sharing this OperationID, leaving concurrent
// operations on the same VM untouched. For VM-owned in-flight reservations,
// when multiple operations exist for the same VM the newest one wins:
// operations are ordered by (metadata.creationTimestamp, OperationID), with
// the UUIDv7 OperationID breaking ties within the timestamp's one-second
// resolution, and older operations' reservations are released.
// +kubebuilder:validation:Optional
OperationID string `json:"operationID,omitempty"`
}

// FailoverHostReservationSpec defines the spec fields specific to failover
Expand Down
5 changes: 2 additions & 3 deletions cmd/manager/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -494,9 +494,8 @@ func main() {
if slices.Contains(mainConfig.EnabledControllers, "inflight-reservation-controller") {
setupLog.Info("enabling controller",
"controller", "inflight-reservation-controller")
config := conf.GetConfigOrDie[inflight.NovaVMClientConfig]()
vmClient := inflight.NewNovaVMClient(config)
controller := &inflight.Controller{Client: multiclusterClient, VMClient: vmClient}
monitor := inflight.NewMonitor(metrics.Registry)
controller := &inflight.Controller{Client: multiclusterClient, Monitor: monitor}
if err := controller.SetupWithManager(ctx, mgr); err != nil {
setupLog.Error(err, "unable to create controller",
"controller", "inflight-reservation-controller")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,14 +66,16 @@ confirmed anything:
## Learning what the platform actually did

In-flight reservations are only half the story: they express Cortex's *guess*, and the guess has to be
reconciled with reality. Two feedback channels close that loop:

- **Exclusion hints on the request.** When the platform retries a placement, it tells Cortex which hosts
it has already ruled out (an `ignore_hosts`-style list). Cortex feeds those forward so the next scoring
round does not re-offer a host the platform has already rejected.
- **Watching the reservation resources.** Cortex watches its own reservation resources and reconciles
in-flight state against confirmed placements as they arrive — promoting a guess to a confirmed slot when
the platform lands where expected, and dropping the other per-candidate guesses for that workload.
reconciled with reality. The feedback loop is closed through the **VirtualMachine CRD**: every in-flight
reservation is owned by the VM CRD of the workload it reserves for, and Cortex watches that CRD to learn
when the owning operation has reached a terminal state (placed and running, resized, migrated, or failed).
Once the VM CRD confirms the outcome, Cortex releases the whole group of per-candidate guesses for that
operation. A VM CRD is therefore required: without it there is no trustworthy terminal signal, and the
hold is kept until the operation confirms or the TTL backstop expires it.

Exclusion hints also flow back on the request itself: when the platform retries a placement, it tells
Cortex which hosts it has already ruled out (an `ignore_hosts`-style list). Cortex feeds those forward so
the next scoring round does not re-offer a host the platform has already rejected.

Because the platform's retry behaviour is not uniform, the number of candidates Cortex offers is matched
to how the caller retries:
Expand Down
28 changes: 26 additions & 2 deletions helm/library/cortex/files/crds/cortex.cloud_reservations.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,25 @@ spec:
intent:
description: |-
Intent defines which kind of virtual machine lifecycle operation
triggered the placement of this in-flight reservation.
triggered the placement of this in-flight reservation. The reconciler
dispatches on this to decide which terminal signal releases the
reservation.
type: string
operationID:
description: |-
OperationID groups all in-flight reservations created within a single
scheduling call (e.g. one reservation per candidate host for an initial
placement, or target + source host for a live migration). It is a
Cortex-generated, time-sortable uuid (UUIDv7), independent of any
external caller's request id, so uniqueness does not depend on callers.

When one operation reaches a terminal state, the reconciler releases
exactly the reservations sharing this OperationID, leaving concurrent
operations on the same VM untouched. For VM-owned in-flight reservations,
when multiple operations exist for the same VM the newest one wins:
operations are ordered by (metadata.creationTimestamp, OperationID), with
the UUIDv7 OperationID breaking ties within the timestamp's one-second
resolution, and older operations' reservations are released.
type: string
projectID:
description: |-
Expand All @@ -296,7 +314,13 @@ spec:
vmID:
description: |-
VMID is the OpenStack server uuid from Nova assigned to the virtual
machine expected to land on this reservation slot.
machine expected to land on this reservation slot. It identifies the
subject of the reservation (what is being placed), which is distinct
from the reservation's owner (what controls its lifecycle, expressed via
ownerReferences).

non-Nova use case needs in-flight reservations. For now (KISS) we always
have a VM uuid, so we keep this Nova-specific field.
type: string
type: object
resources:
Expand Down
Loading
Loading