fix(migration): release NVMe target paths a migration never cuts over to

[noctarius]

EnsureMigrationPaths connected a migration's target paths on every consumer node and nothing ever disconnected them. A migration that did not cut over left them behind, and they did not go away: the control plane answers CreateMigration with ctrl-loss-tmo=3600, and the kernel resets its reconnect counter whenever a reconnect gets far enough, so an abandoned controller retried a target that had stopped answering for it for as long as the node was up. Once one of them was live without serving a namespace, Inspect reported it as controller-not-contributing, VerifyMigrationPaths turned that into a failure, and the failure aborted the very migration whose release would have cleaned it up — so the leak did not merely accumulate, it stopped every later migration of the subsystem from ever passing.

Run fio-mig-1787159565 is the shape of it: 16 of 23 migrations failed on "NVMe path validation failed", and 2.5h after the run the three hosts still carried 32 lvol controllers where ~18 were expected, 18 of them stuck connecting or live with zero namespaces, with the target logging ~146 rejected CONNECTs a minute.

Added, in operator/internal/volumemigration/release.go:

• ReleaseMigrationPaths, the counterpart EnsureMigrationPaths lacked. It releases a target path only when no namespace is accessible over it, which is what makes it safe without a record of which paths a run created: a parked target path is inaccessible by definition pre-cutover, while a path that was already there is a live HA path and is serving.
• ReapDeadControllers, which clears the husks a lost path leaves behind. The selection is derived from nvmeof.Inspect rather than from its own reading of sysfs, so "the reap clears what validation rejects" holds by construction. It is narrower than atlas's own auto-repair set on purpose: Attach repairs controller-not-contributing to force a re-enumeration and reconnects afterwards, and this caller has no such half.

Wired in so that every route out of Validating gives the paths back:

• the validation Job reaps before connecting and releases what it established when validation fails, on the node that has the paths, before the exit code cancels the migration;
• simplyblock-rebalancer gains --mode=release-migration-paths, which the operator runs from cancelAndFail and reconcileAbort for the nodes whose own Job passed and therefore never learns the migration was cancelled anyway.

Also in this change:

• ctrl_loss_tmo for migration paths is now vmigration.CtrlLossTmoSec (60s), matching the DefaultCtrlLossTmo the CSI driver connects every other path with; the control plane's hour was never a deliberate choice for a probe path. Applied where the response is ingested, so status.connections records the connect that will be made.
• Connection.target() sends ctrl_loss_tmo and fast_io_fail_tmo whenever they are set to anything, including the negative "retry forever" and the zero "fail I/O at once"; a > 0 guard silently turned either into the kernel default.
• reconcileNonOptimizedPaths matches an expected endpoint against an attached one by address and port, extracted as missingEndpoints. A storage node serves one subsystem on several ports, so keying by address alone let a stale controller at 10.0.0.112:4426 stand in for the published 10.0.0.112:4428 and leave the volume a path short with a reconcile that found nothing to do every tick. The comment there promising a disconnect that never happened now says why there is none: an endpoint missing from the control plane's answer looks exactly like a node in restart, which is why atlas refuses to repair DefectStaleEndpoint unattended.

Cutover drives every path of the subsystem inaccessible for about two seconds by design, so both halves account for it. ReleaseMigrationPaths declines while no path on the subsystem is accessible — in that window every controller looks like it carries nothing, including the one about to become the data path — and the reap is unaffected, reading namespace legs rather than ANA states.

fio_migration_test.py bounds that window at CUTOVER_PAUSE_CRIT_S instead of the 60s it allowed before. The measurement was already right; the threshold was an order of magnitude looser than anything an application survives, which is why the run reported OK for the two migrations that killed eight of its ten pods at 6s and 8s of every path inaccessible. On that run's data the new bound fails those four and passes the five healthy migrations that measured 3s.

[Copilot]

Pull request overview

This PR prevents leaked NVMe-oF migration target controllers from persisting on consumer nodes when a migration never reaches cutover, by adding explicit host-side cleanup (release + reap) and wiring it into failure/abort paths. It also tightens fio migration validation around the expected cutover pause window and fixes endpoint matching in the CSI initiator logic to account for multiple ports on the same storage node.

Changes:

• Add host-side cleanup primitives for migrations (ReleaseMigrationPaths, ReapDeadControllers) and invoke them from validation failures and operator abort/fail paths via a new rebalancer mode.
• Standardize migration-path ctrl_loss_tmo to vmigration.CtrlLossTmoSec and improve endpoint matching in the CSI driver by keying on ip:port.
• Tighten fio migration test validation to bound the “all paths inaccessible” cutover pause to a much smaller default threshold.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
operator/test/fio_migration_test.py	Introduces cutover pause constants and tightens the default ANA stall threshold and messaging.
operator/internal/volumemigration/validate.go	Adds CtrlLossTmoSec constant and modifies timeout propagation behavior in Connection.target().
operator/internal/volumemigration/release.go	New implementation of migration-path release and dead-controller reaping based on nvmeof.Inspect.
operator/internal/volumemigration/release_test.go	New unit tests covering release/reap safety rules and edge cases.
operator/internal/controller/volumemigration_migration_paths_test.go	Updates expectations to reflect the ctrl-loss-tmo override at ingestion.
operator/internal/controller/volumemigration_helpers_test.go	Adds tests ensuring release jobs are created and correctly configured on abort/fail paths.
operator/internal/controller/volumemigration_controller.go	Wires ctrl-loss-tmo override and adds creation of per-node release Jobs; factors validation/release Jobs via shared helper.
operator/dist/install.yaml	Updates generated CRD docs to describe connections being reused for release and ctrlLossTmo override semantics.
operator/config/crd/bases/storage.simplyblock.io_volumemigrations.yaml	Updates CRD base docs matching the install manifest change.
operator/cmd/simplyblock-rebalancer/validate_migration.go	Adds reap-before-validate, release-on-failure, and a new release-migration-paths mode.
operator/cmd/simplyblock-rebalancer/validate_migration_test.go	Adds tests for reap/release ordering and ensuring cleanup doesn’t mask validation outcomes.
operator/cmd/simplyblock-rebalancer/main.go	Expands CLI modes and updates usage text accordingly.
operator/api/v1alpha1/volumemigration_types.go	Updates Go type docs for status.connections to reflect release usage and ctrlLossTmo override.
helm-charts/charts/simplyblock-operator/crds/storage.simplyblock.io_volumemigrations.yaml	Updates Helm-packaged CRD docs mirroring the operator CRD base changes.
csi-driver/pkg/util/initiator.go	Fixes non-optimized endpoint reconciliation to match expected endpoints by ip:port and adds parsing helpers.
csi-driver/pkg/util/initiator_test.go	Adds tests for endpoint parsing and missing-endpoint detection behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.