Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion components/ironic/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ resources:
# less than ideal addition but necessary so that we can have the ironic.conf.d loading
# working due to the way the chart hardcodes the config-file parameter which then
# takes precedence over the directory
- ./runbook-crd

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the one that worries me most, because it's data loss on upgrade rather than a bug we can patch later.

components/ironic/runbook-crd/bases/baremetal.ironicproject.org_runbooks.yaml defines ironicrunbooks.baremetal.ironicproject.org — the same CRD name and kind: IronicRunbook that components/openstack-sync-operator/crds/ now ships. Dropping this line doesn't delete the directory, it just stops rendering it, so the CRD moves from the ironic Application to the openstack-sync-operator Application. Both run with prune: true.

ArgoCD gives us no ordering guarantee between two Applications, so the ironic app is free to prune the CRD before (or after) the sync-operator app adopts it. A pruned CRD cascade-deletes every IronicRunbook in the cluster, including bmc-maintenance, which this PR moves to openstack-sync-plugins/ironic-runbooks/examples/ — i.e. somewhere that is never applied. So the CR we ship today just disappears with nothing recreating it.

Two more things on top of that:

  • The new CRD makes spec.cloudCredentialsRef required. Any site manifest written against the old schema gets rejected on apply, not migrated.
  • No upgrade-impact label and no changelog.d/ fragment, so operators get no warning that they need to re-apply their runbooks.

Options I'd be happy with: keep the CRD rendered from components/ironic for one release and have the sync-operator app not own it yet, or add explicit sync-wave ordering plus an upgrade-impact note documenting that runbook CRs must be re-applied. Either way I think we need bmc-maintenance to stay in an applied path.


Generated by Claude Code

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we moved the new CRDs to the understack.rackspace.net namespace that we've discussed about all of these?

- ./runbook-operator

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're dropping ./runbook-crd but leaving ./runbook-operator (shell-operator-ironic) deployed, and its RBAC in runbook-operator/role.yaml still grants it ironicrunbooks. So with plugins.ironicRunbooks: true we have two operators reconciling the same IronicRunbook objects against the same Ironic API.

That's not just redundant, the two fight each other:

  • The legacy hook's status patch doesn't set observedGeneration, so the new operator's _status_is_current never sees a current status and re-syncs. Its own patch then trips the legacy hook. That's a permanent 30s reconcile ping-pong hitting Ironic, with no CR change driving it.
  • The legacy hook rewrites extra without preserving the _understack_runbook_* markers, so it strips exactly the ownership markers is_managed_runbook needs. Once they're gone, prune stops recognizing its own runbooks and silently keeps orphans forever.

Can we remove ./runbook-operator in this PR too, or gate it so only one of the two is ever active?


Generated by Claude Code

# Alerting
- pr-clean-failed-servers.yaml
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: ironicrunbooks.baremetal.ironicproject.org
spec:
group: baremetal.ironicproject.org
names:
kind: IronicRunbook
listKind: IronicRunbookList
plural: ironicrunbooks
shortNames:
- rb
singular: ironicrunbook
scope: Namespaced
versions:
- name: v1alpha1
served: true
storage: true
additionalPrinterColumns:
- name: Runbook
type: string
jsonPath: .spec.runbookName
- name: Description
type: string
jsonPath: .spec.description
priority: 1
- name: Public
type: boolean
jsonPath: .spec.public
- name: SyncStatus
type: string
jsonPath: .status.syncStatus
- name: Age
type: date
jsonPath: .metadata.creationTimestamp
schema:
openAPIV3Schema:
description: >-
IronicRunbook defines one Ironic runbook. The operator-owned API
contract keeps OpenStack credentials on every CR so reconciliation can
be grouped by cloud, matching the other openstack-sync plugins.
type: object
required:
- spec
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
description: IronicRunbookSpec defines the desired runbook data.
type: object
required:
- cloudCredentialsRef
- runbookName
- steps
properties:
cloudCredentialsRef:
description: >-
cloudCredentialsRef points to a Kubernetes Secret containing
an OpenStack clouds.yaml file. The operator reads this secret
directly at reconcile time; no volume mount is required.
type: object
required:
- secretName
- cloudName
properties:
secretName:
description: >-
Name of a Secret in the same namespace as this resource.
The Secret must contain a key named clouds.yaml holding
an OpenStack clouds.yaml file.
type: string
minLength: 1
maxLength: 253
cloudName:
description: >-
Name of the cloud entry within the clouds.yaml to
authenticate as.
type: string
minLength: 1
maxLength: 256
runbookName:
description: >-
Runbook name, and the identity the operator syncs by. Renaming
creates a new runbook rather than renaming the existing one.
type: string
minLength: 1
maxLength: 255
pattern: ^[A-Za-z0-9._~-]+$
description:
description: Human-readable runbook description.
type: string
maxLength: 255
traits:
description: >-
Traits deciding which nodes this runbook may act on. A node
must carry at least one; a runbook with no traits matches no nodes.
type: array
default: []
items:
type: string
minLength: 1
maxLength: 255
pattern: ^CUSTOM_[A-Z0-9_]+$
steps:
description: Ordered runbook steps.
type: array
minItems: 1
items:
type: object
required:
- interface
- step
- order
properties:
interface:
description: Interface that owns this cleaning step.
type: string
enum:
- bios
- deploy
- firmware
- management
- power
- raid
- vendor
step:
description: Step name for the selected interface.
type: string
minLength: 1
maxLength: 255
args:
description: Step-specific arguments.
type: object
x-kubernetes-preserve-unknown-fields: true
order:
description: Execution order. Lower numbers run first.
type: integer
minimum: 0
disableRamdisk:
description: Whether to run without booting the cleaning ramdisk.
type: boolean
default: false
public:
description: >-
Whether the runbook is available to all projects. A public
runbook cannot have an owner.
type: boolean
default: false
owner:
description: >-
Project that owns this runbook. Leave unset to let Ironic
assign the credentials' own project.
type: string
maxLength: 255
extra:
description: >-
Additional runbook metadata. The operator also keeps its
ownership markers here, under _understack_runbook_ keys.
type: object
x-kubernetes-preserve-unknown-fields: true
status:
description: IronicRunbookStatus defines the observed sync state.
type: object
properties:
ironicUUID:
description: Ironic UUID of this runbook.
type: string
Comment on lines +170 to +172

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ironicUUID is declared here but nothing ever writes it — patch_resource_status has no parameter for it, and sync_runbook only logs the UUID.

This is the field an operator would reach for to correlate a CR with openstack baremetal runbook show, so it's worth wiring up rather than leaving as a permanently-empty field. Either thread it through patch_resource_status or drop it from the schema until we do.


Generated by Claude Code

syncStatus:
description: SyncStatus indicates the synchronization state with Ironic.
type: string
enum:
- Synced
- Failed
- Unknown
lastSyncTime:
description: LastSyncTime is the last time the operator attempted to sync the runbook.
type: string
format: date-time
observedGeneration:
description: ObservedGeneration is the metadata generation last processed by the operator.
type: integer
format: int64
message:
description: Message provides details about the last sync attempt.
type: string
maxLength: 2048
conditions:
description: Conditions describe current observed state.
type: array
items:
type: object
required:
- type
- status
properties:
type:
type: string
status:
type: string
enum:
- "True"
- "False"
- Unknown
reason:
type: string
message:
type: string
maxLength: 2048
lastTransitionTime:
type: string
format: date-time
subresources:
status: {}
16 changes: 16 additions & 0 deletions components/openstack-sync-operator/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ rbac:
plugins:
openstackPlaceholder: false
neutronRouterFlavors: false
ironicRunbooks: false

pluginData:
openstackPlaceholder:
Expand All @@ -55,3 +56,18 @@ pluginData:
# When true, removing a NeutronRouterFlavor CR also deletes its unused
# operator-managed OpenStack flavor. Enable this before removing the CR.
PRUNE: false

ironicRunbooks:
hook:
path: /hooks/ironic_runbooks.py
crd: crds/baremetal.ironicproject.org_ironicrunbooks.yaml
envPrefix: IRONIC_RUNBOOK
env:
SYNC_CRONTAB: "0 * * * *"
# Ironic readiness wait before a runbook reconcile fails.
# Total wait is READY_RETRIES * READY_DELAY seconds.
READY_RETRIES: 30
READY_DELAY: 10
# When true, removing an IronicRunbook CR also deletes its
# operator-owned Ironic runbook. Enable this before removing the CR.
PRUNE: false
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# IronicRunbook Examples

Reference CRs for the `ironicRunbooks` openstack-sync hook. **Nothing here is
applied.** The parent `kustomization.yaml` lists only the shared runbooks, and
this directory is not one of its `resources`.

## Using one

Copy the file to where the CRs for your site live, usually
`<deploy-repo>/<site>/openstack-sync-plugins/`, add it to that directory's
`kustomization.yaml`, then adjust three things:

1. `metadata.namespace`: must be the namespace the operator watches
(`POD_NAMESPACE`, commonly `openstack`). The samples use
`baremetal-system` and `default`, which the hook will not see.
2. `spec.cloudCredentialsRef`: the Secret holding `clouds.yaml` and the cloud
entry to authenticate with.
3. `spec.traits`: Ironic only runs a runbook on a node carrying at least one of
them, so a runbook with no traits matches no nodes.

Step names and arguments in these files are illustrative. Check that the
`interface` and `step` you want exist on the target hardware before relying on
them, and replace the firmware URLs and checksums with real ones.

## Removing one

Deleting the CR does not delete the Ironic runbook. The hook only prunes when
`PRUNE` is enabled for it, and the chart default is `false`, so a removed CR
leaves the runbook in Ironic with nothing reconciling it. Delete both, or turn
pruning on deliberately — see
`docs/operator-guide/server-firmware-update.md#removing-a-runbook`.

## The examples

| File | Purpose |
|------|---------|
| `runbook_v1alpha1_minimal.yaml` | Smallest valid CR: required fields only |
| `runbook_v1alpha1_complete.yaml` | Every field, with each one annotated |
| `runbook_bios_config.yaml` | BIOS settings for virtualization on compute nodes |
| `runbook_raid_config.yaml` | RAID setup, OS volume plus data volume |
| `runbook_firmware_update.yaml` | BIOS, BMC and NIC firmware updates |
| `runbook_disk_cleaning.yaml` | Disk erasure for node reuse |
| `runbook_gpu_node_setup.yaml` | BIOS and firmware for GPU nodes |

## Validation

Editors pick up the published spec schema from the `yaml-language-server` line at
the top of `../bmc_maintenance.yaml`; add the same line to a copied example to
get completion and checking. Kubernetes validates the full CR against the CRD in
`components/openstack-sync-operator/crds/` when ArgoCD applies it.

Running a synced firmware runbook against a node is covered in
`docs/operator-guide/server-firmware-update.md`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# yaml-language-server: $schema=https://rackerlabs.github.io/understack/schema/openstack-sync/ironic-runbook.schema.json
# BMC Maintenance Runbook
#
# Clears the BMC job queue and resynchronizes the BMC clock on Dell iDRAC nodes.
# Runs without booting the cleaning ramdisk, so it is safe for out-of-band only work.
#
# Only nodes carrying the traits below are eligible.

apiVersion: baremetal.ironicproject.org/v1alpha1
kind: IronicRunbook
metadata:
name: bmc-maintenance
namespace: openstack
labels:
app.kubernetes.io/name: openstack-sync-plugins
app.kubernetes.io/component: ironic-runbooks
app.kubernetes.io/part-of: openstack-sync
use-case: bmc-maintenance
hardware-type: general
spec:
cloudCredentialsRef:
# System-scoped credential: Ironic requires system_scope:all to publish a
# runbook, which the project-scoped infrasetup credential cannot satisfy.
secretName: infrasetup-system
cloudName: understack
runbookName: bmc-maintenance
description: "Performs BMC maintenance operations including clearing the job queue and synchronizing the BMC clock."
public: true
disableRamdisk: true
traits:
- CUSTOM_DELL_IDRAC
steps:
- interface: management
step: clear_job_queue
order: 1
- interface: management
step: set_bmc_clock
order: 2
extra:
version: "1.0.0"
use_case: "BMC housekeeping and clock synchronization"
warnings:
- "Clearing the job queue discards pending BMC jobs, including scheduled firmware updates"
Loading
Loading