-
Notifications
You must be signed in to change notification settings - Fork 11
feat(openstack-sync-operator): manage ironic runbooks using openstack-sync-operator #2314
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,7 +11,6 @@ resources: | |
| # less than ideal addition but necessary so that we can have the ironic.conf.d loading | ||
| # working due to the way the chart hardcodes the config-file parameter which then | ||
| # takes precedence over the directory | ||
| - ./runbook-crd | ||
| - ./runbook-operator | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We're dropping That's not just redundant, the two fight each other:
Can we remove Generated by Claude Code |
||
| # Alerting | ||
| - pr-clean-failed-servers.yaml | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,218 @@ | ||
| --- | ||
| apiVersion: apiextensions.k8s.io/v1 | ||
| kind: CustomResourceDefinition | ||
| metadata: | ||
| name: ironicrunbooks.baremetal.ironicproject.org | ||
| spec: | ||
| group: baremetal.ironicproject.org | ||
| names: | ||
| kind: IronicRunbook | ||
| listKind: IronicRunbookList | ||
| plural: ironicrunbooks | ||
| shortNames: | ||
| - rb | ||
| singular: ironicrunbook | ||
| scope: Namespaced | ||
| versions: | ||
| - name: v1alpha1 | ||
| served: true | ||
| storage: true | ||
| additionalPrinterColumns: | ||
| - name: Runbook | ||
| type: string | ||
| jsonPath: .spec.runbookName | ||
| - name: Description | ||
| type: string | ||
| jsonPath: .spec.description | ||
| priority: 1 | ||
| - name: Public | ||
| type: boolean | ||
| jsonPath: .spec.public | ||
| - name: SyncStatus | ||
| type: string | ||
| jsonPath: .status.syncStatus | ||
| - name: Age | ||
| type: date | ||
| jsonPath: .metadata.creationTimestamp | ||
| schema: | ||
| openAPIV3Schema: | ||
| description: >- | ||
| IronicRunbook defines one Ironic runbook. The operator-owned API | ||
| contract keeps OpenStack credentials on every CR so reconciliation can | ||
| be grouped by cloud, matching the other openstack-sync plugins. | ||
| type: object | ||
| required: | ||
| - spec | ||
| properties: | ||
| apiVersion: | ||
| type: string | ||
| kind: | ||
| type: string | ||
| metadata: | ||
| type: object | ||
| spec: | ||
| description: IronicRunbookSpec defines the desired runbook data. | ||
| type: object | ||
| required: | ||
| - cloudCredentialsRef | ||
| - runbookName | ||
| - steps | ||
| properties: | ||
| cloudCredentialsRef: | ||
| description: >- | ||
| cloudCredentialsRef points to a Kubernetes Secret containing | ||
| an OpenStack clouds.yaml file. The operator reads this secret | ||
| directly at reconcile time; no volume mount is required. | ||
| type: object | ||
| required: | ||
| - secretName | ||
| - cloudName | ||
| properties: | ||
| secretName: | ||
| description: >- | ||
| Name of a Secret in the same namespace as this resource. | ||
| The Secret must contain a key named clouds.yaml holding | ||
| an OpenStack clouds.yaml file. | ||
| type: string | ||
| minLength: 1 | ||
| maxLength: 253 | ||
| cloudName: | ||
| description: >- | ||
| Name of the cloud entry within the clouds.yaml to | ||
| authenticate as. | ||
| type: string | ||
| minLength: 1 | ||
| maxLength: 256 | ||
| runbookName: | ||
| description: >- | ||
| Runbook name, and the identity the operator syncs by. Renaming | ||
| creates a new runbook rather than renaming the existing one. | ||
| type: string | ||
| minLength: 1 | ||
| maxLength: 255 | ||
| pattern: ^[A-Za-z0-9._~-]+$ | ||
| description: | ||
| description: Human-readable runbook description. | ||
| type: string | ||
| maxLength: 255 | ||
| traits: | ||
| description: >- | ||
| Traits deciding which nodes this runbook may act on. A node | ||
| must carry at least one; a runbook with no traits matches no nodes. | ||
| type: array | ||
| default: [] | ||
| items: | ||
| type: string | ||
| minLength: 1 | ||
| maxLength: 255 | ||
| pattern: ^CUSTOM_[A-Z0-9_]+$ | ||
| steps: | ||
| description: Ordered runbook steps. | ||
| type: array | ||
| minItems: 1 | ||
| items: | ||
| type: object | ||
| required: | ||
| - interface | ||
| - step | ||
| - order | ||
| properties: | ||
| interface: | ||
| description: Interface that owns this cleaning step. | ||
| type: string | ||
| enum: | ||
| - bios | ||
| - deploy | ||
| - firmware | ||
| - management | ||
| - power | ||
| - raid | ||
| - vendor | ||
| step: | ||
| description: Step name for the selected interface. | ||
| type: string | ||
| minLength: 1 | ||
| maxLength: 255 | ||
| args: | ||
| description: Step-specific arguments. | ||
| type: object | ||
| x-kubernetes-preserve-unknown-fields: true | ||
| order: | ||
| description: Execution order. Lower numbers run first. | ||
| type: integer | ||
| minimum: 0 | ||
| disableRamdisk: | ||
| description: Whether to run without booting the cleaning ramdisk. | ||
| type: boolean | ||
| default: false | ||
| public: | ||
| description: >- | ||
| Whether the runbook is available to all projects. A public | ||
| runbook cannot have an owner. | ||
| type: boolean | ||
| default: false | ||
| owner: | ||
| description: >- | ||
| Project that owns this runbook. Leave unset to let Ironic | ||
| assign the credentials' own project. | ||
| type: string | ||
| maxLength: 255 | ||
| extra: | ||
| description: >- | ||
| Additional runbook metadata. The operator also keeps its | ||
| ownership markers here, under _understack_runbook_ keys. | ||
| type: object | ||
| x-kubernetes-preserve-unknown-fields: true | ||
| status: | ||
| description: IronicRunbookStatus defines the observed sync state. | ||
| type: object | ||
| properties: | ||
| ironicUUID: | ||
| description: Ironic UUID of this runbook. | ||
| type: string | ||
|
Comment on lines
+170
to
+172
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This is the field an operator would reach for to correlate a CR with Generated by Claude Code |
||
| syncStatus: | ||
| description: SyncStatus indicates the synchronization state with Ironic. | ||
| type: string | ||
| enum: | ||
| - Synced | ||
| - Failed | ||
| - Unknown | ||
| lastSyncTime: | ||
| description: LastSyncTime is the last time the operator attempted to sync the runbook. | ||
| type: string | ||
| format: date-time | ||
| observedGeneration: | ||
| description: ObservedGeneration is the metadata generation last processed by the operator. | ||
| type: integer | ||
| format: int64 | ||
| message: | ||
| description: Message provides details about the last sync attempt. | ||
| type: string | ||
| maxLength: 2048 | ||
| conditions: | ||
| description: Conditions describe current observed state. | ||
| type: array | ||
| items: | ||
| type: object | ||
| required: | ||
| - type | ||
| - status | ||
| properties: | ||
| type: | ||
| type: string | ||
| status: | ||
| type: string | ||
| enum: | ||
| - "True" | ||
| - "False" | ||
| - Unknown | ||
| reason: | ||
| type: string | ||
| message: | ||
| type: string | ||
| maxLength: 2048 | ||
| lastTransitionTime: | ||
| type: string | ||
| format: date-time | ||
| subresources: | ||
| status: {} | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| # IronicRunbook Examples | ||
|
|
||
| Reference CRs for the `ironicRunbooks` openstack-sync hook. **Nothing here is | ||
| applied.** The parent `kustomization.yaml` lists only the shared runbooks, and | ||
| this directory is not one of its `resources`. | ||
|
|
||
| ## Using one | ||
|
|
||
| Copy the file to where the CRs for your site live, usually | ||
| `<deploy-repo>/<site>/openstack-sync-plugins/`, add it to that directory's | ||
| `kustomization.yaml`, then adjust three things: | ||
|
|
||
| 1. `metadata.namespace`: must be the namespace the operator watches | ||
| (`POD_NAMESPACE`, commonly `openstack`). The samples use | ||
| `baremetal-system` and `default`, which the hook will not see. | ||
| 2. `spec.cloudCredentialsRef`: the Secret holding `clouds.yaml` and the cloud | ||
| entry to authenticate with. | ||
| 3. `spec.traits`: Ironic only runs a runbook on a node carrying at least one of | ||
| them, so a runbook with no traits matches no nodes. | ||
|
|
||
| Step names and arguments in these files are illustrative. Check that the | ||
| `interface` and `step` you want exist on the target hardware before relying on | ||
| them, and replace the firmware URLs and checksums with real ones. | ||
|
|
||
| ## Removing one | ||
|
|
||
| Deleting the CR does not delete the Ironic runbook. The hook only prunes when | ||
| `PRUNE` is enabled for it, and the chart default is `false`, so a removed CR | ||
| leaves the runbook in Ironic with nothing reconciling it. Delete both, or turn | ||
| pruning on deliberately — see | ||
| `docs/operator-guide/server-firmware-update.md#removing-a-runbook`. | ||
|
|
||
| ## The examples | ||
|
|
||
| | File | Purpose | | ||
| |------|---------| | ||
| | `runbook_v1alpha1_minimal.yaml` | Smallest valid CR: required fields only | | ||
| | `runbook_v1alpha1_complete.yaml` | Every field, with each one annotated | | ||
| | `runbook_bios_config.yaml` | BIOS settings for virtualization on compute nodes | | ||
| | `runbook_raid_config.yaml` | RAID setup, OS volume plus data volume | | ||
| | `runbook_firmware_update.yaml` | BIOS, BMC and NIC firmware updates | | ||
| | `runbook_disk_cleaning.yaml` | Disk erasure for node reuse | | ||
| | `runbook_gpu_node_setup.yaml` | BIOS and firmware for GPU nodes | | ||
|
|
||
| ## Validation | ||
|
|
||
| Editors pick up the published spec schema from the `yaml-language-server` line at | ||
| the top of `../bmc_maintenance.yaml`; add the same line to a copied example to | ||
| get completion and checking. Kubernetes validates the full CR against the CRD in | ||
| `components/openstack-sync-operator/crds/` when ArgoCD applies it. | ||
|
|
||
| Running a synced firmware runbook against a node is covered in | ||
| `docs/operator-guide/server-firmware-update.md`. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| # yaml-language-server: $schema=https://rackerlabs.github.io/understack/schema/openstack-sync/ironic-runbook.schema.json | ||
| # BMC Maintenance Runbook | ||
| # | ||
| # Clears the BMC job queue and resynchronizes the BMC clock on Dell iDRAC nodes. | ||
| # Runs without booting the cleaning ramdisk, so it is safe for out-of-band only work. | ||
| # | ||
| # Only nodes carrying the traits below are eligible. | ||
|
|
||
| apiVersion: baremetal.ironicproject.org/v1alpha1 | ||
| kind: IronicRunbook | ||
| metadata: | ||
| name: bmc-maintenance | ||
| namespace: openstack | ||
| labels: | ||
| app.kubernetes.io/name: openstack-sync-plugins | ||
| app.kubernetes.io/component: ironic-runbooks | ||
| app.kubernetes.io/part-of: openstack-sync | ||
| use-case: bmc-maintenance | ||
| hardware-type: general | ||
| spec: | ||
| cloudCredentialsRef: | ||
| # System-scoped credential: Ironic requires system_scope:all to publish a | ||
| # runbook, which the project-scoped infrasetup credential cannot satisfy. | ||
| secretName: infrasetup-system | ||
| cloudName: understack | ||
| runbookName: bmc-maintenance | ||
| description: "Performs BMC maintenance operations including clearing the job queue and synchronizing the BMC clock." | ||
| public: true | ||
| disableRamdisk: true | ||
| traits: | ||
| - CUSTOM_DELL_IDRAC | ||
| steps: | ||
| - interface: management | ||
| step: clear_job_queue | ||
| order: 1 | ||
| - interface: management | ||
| step: set_bmc_clock | ||
| order: 2 | ||
| extra: | ||
| version: "1.0.0" | ||
| use_case: "BMC housekeeping and clock synchronization" | ||
| warnings: | ||
| - "Clearing the job queue discards pending BMC jobs, including scheduled firmware updates" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the one that worries me most, because it's data loss on upgrade rather than a bug we can patch later.
components/ironic/runbook-crd/bases/baremetal.ironicproject.org_runbooks.yamldefinesironicrunbooks.baremetal.ironicproject.org— the same CRD name andkind: IronicRunbookthatcomponents/openstack-sync-operator/crds/now ships. Dropping this line doesn't delete the directory, it just stops rendering it, so the CRD moves from the ironic Application to the openstack-sync-operator Application. Both run withprune: true.ArgoCD gives us no ordering guarantee between two Applications, so the ironic app is free to prune the CRD before (or after) the sync-operator app adopts it. A pruned CRD cascade-deletes every
IronicRunbookin the cluster, includingbmc-maintenance, which this PR moves toopenstack-sync-plugins/ironic-runbooks/examples/— i.e. somewhere that is never applied. So the CR we ship today just disappears with nothing recreating it.Two more things on top of that:
spec.cloudCredentialsRefrequired. Any site manifest written against the old schema gets rejected on apply, not migrated.upgrade-impactlabel and nochangelog.d/fragment, so operators get no warning that they need to re-apply their runbooks.Options I'd be happy with: keep the CRD rendered from
components/ironicfor one release and have the sync-operator app not own it yet, or add explicit sync-wave ordering plus anupgrade-impactnote documenting that runbook CRs must be re-applied. Either way I think we needbmc-maintenanceto stay in an applied path.Generated by Claude Code
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if we moved the new CRDs to the understack.rackspace.net namespace that we've discussed about all of these?