[2.16] plan pause and cancellation

[jiaqiluo]

Issues:

• [Day 2 Ops] Plan Pausing rancher#54265
• [Day 2 Ops] Plan Cancellation rancher#54255

Summary

This PR adds support for plan pause and cancellation in system-agent's remote mode (pkg/k8splan).

Operators can now suspend or cancel executing plans directly via Kubernetes Secret annotations. This enables safe plan resumption from checkpoints and grants granular control over process termination behavior.

Usage

Pause a Plan

Set plan.cattle.io/paused = "true" on the plan Secret.

• Behavior: The currently running instruction completes, but subsequent instructions are blocked.
• State: A checkpoint is saved.
• Resume: Remove the annotation or set it to "false".

Cancel a Plan

Set plan.cattle.io/canceled = "true" on the plan Secret.

• Behavior: Immediately terminates the in-flight instruction (SIGTERM/SIGKILL on Unix; Job Object termination on Windows).
• State: Cancellation is terminal—canceled plans cannot be resumed directly.
• Restart: The orchestrator must reset plan-state to pending or recreate the Plan secret.

Resume a Paused Plan

Remove the plan.cattle.io/paused annotation or set its value to "false".

Feature Details

Annotation Handling

• Valid values: "true" and "false" (absent defaults to "false").
• Invalid values: Handled as configuration errors. The plan halts, error status writes are skipped (preventing write-loop storms), and the workqueue retries using exponential backoff/rate limiting.
• Precedence: canceled = "true" always takes precedence over paused (even if paused contains an invalid value). This allows operators to quickly stop runaway processes without needing to fix annotation typos first.

Execution vs. Observation

• Execution: Suppressed while either annotation is active, blocking new instructions from firing.
• Observation: Remains active. Probes continue running and status fields persist during pause/cancellation. This ensures MachineHealthCheck receives accurate, up-to-date node health data even when a plan is held.

Pause vs. Cancel Comparison

Feature Aspect	Pause	Cancel
Termination Latency	Natural (waits for current instruction to finish)	Immediate (SIGTERM immediately, SIGKILL after 10s)
Process Handling	Lets process finish naturally; writes checkpoint	Signals entire process group (Unix) or Job Object (Windows)
Resumability	Yes (checkpoint preserves progress)	No (terminal; requires resetting plan-state to pending)
Annotation Removal	Unpauses and resumes execution from checkpoint	Does not un-cancel; requires orchestrator state reset

Backward Compatibility (Checksum Flow)

For legacy orchestrators operating without plan-state support:

• Annotations generate warning logs but trigger no operational changes.
• Plan reconciliation proceeds normally via checksum comparison.
• Checkpoints, progress persistence, and interrupt watchers remain inactive.

Tests Added

Unit Tests

• interrupt_test.go: Annotation parsing, interrupt state resolution, precedence rules, polling logic, and edge cases.
• plan_progress_test.go: Checkpoint parsing/marshaling, resume state resolution, and sanitization.
• suppression_test.go: Matrix of annotation combinations and their impacts on reconciliation.
• reconcile_test.go (expanded): plan-state transitions, checkpoint updates, and interrupt handling during application.
• applyinator_test.go (expanded): Interruption channels, resume-from behaviors, and probe execution while on hold.
• process_unix_test.go: Process group termination, signal handling, and zombie reaping.
• process_windows_test.go: Job Object termination boundaries and edge cases.

Integration & E2E Tests

• cancellation_test.go: Full-lifecycle cancellation workflows.
• pause_test.go: Pause, resume, and edge-case execution sequences.
• helpers_test.go: Framework helpers for annotation manipulation.

Note that E2E Tests for Windows will be added in the rancher/wins repo. See rancher/wins#565

Known Limitations

1. Local Mode Unsupported (pkg/localplan): Pause and cancel capabilities require the remote mode plan-state flow. Local plans remain checksum-only without annotation support.
2. Process Tree Boundaries: Unix child processes calling setsid() escape process group signals and confirmation checks (cgroups are required to fully bridge this gap). A TerminationIncomplete flag is populated when this condition is detected to alert operators.

---

Verification & Testing Instructions

Run the full test suite:

make test      # Unit tests
make test-e2e  # E2E tests (requires Kind cluster)

[Copilot]

🟡 Changes recommended

Interrupt races, lost periodic progress, reversible cancellation, and incomplete Windows termination reporting remain.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds remote-plan pausing, resumable checkpoints, terminal cancellation, and cross-platform process-tree termination.

Changes:

• Adds pause/cancel annotation handling and checkpoint persistence.
• Extends Applyinator with interruption-aware execution and process cleanup.
• Adds comprehensive unit and end-to-end coverage.

File summaries

File	Description
go.mod	Updates plan dependencies and Windows support.
go.sum	Refreshes dependency checksums.
pkg/applyinator/applyinator.go	Implements interruptible plan execution.
pkg/applyinator/applyinator_test.go	Tests interruption and process handling.
pkg/applyinator/process_unix.go	Manages Unix process groups.
pkg/applyinator/process_unix_test.go	Tests Unix process-tree termination.
pkg/applyinator/process_windows.go	Manages Windows Job Objects.
pkg/k8splan/interrupt.go	Handles pause and cancellation annotations.
pkg/k8splan/interrupt_test.go	Tests interrupt parsing and persistence.
pkg/k8splan/plan_decision_test.go	Updates canceled-state coverage.
pkg/k8splan/plan_progress.go	Implements resumable checkpoints.
pkg/k8splan/plan_progress_test.go	Tests checkpoint behavior.
pkg/k8splan/reconcile.go	Integrates interrupts into reconciliation.
pkg/k8splan/secret_outcome.go	Clears consumed checkpoints.
pkg/k8splan/secret_outcome_test.go	Tests checkpoint cleanup.
pkg/k8splan/watcher.go	Preserves checkpoints across conflicts.
pkg/k8splan/watcher_test.go	Tests conflict merging.
test/e2e/suites/remote-plan/cancellation_test.go	Adds cancellation scenarios.
test/e2e/suites/remote-plan/failure_handling_test.go	Tests terminal failed plans.
test/e2e/suites/remote-plan/helpers_test.go	Adds interruption test helpers.
test/e2e/suites/remote-plan/pause_test.go	Adds pause/resume scenarios.
test/framework/secret.go	Adds annotation and checkpoint helpers.
test/go.mod	Updates test-module dependencies.
test/go.sum	Refreshes test dependency checksums.
test/README.md	Documents restart coverage.

Review details

Suppressed comments (1)

pkg/applyinator/applyinator.go:458

• A pause that arrives while the current instruction exits non-zero is assigned to result.Interruption here. reconcileSecret handles interruptions before OneTimeApplySucceeded, so the real instruction failure is recorded as a pause instead of plan-state: failed, and its failure counters are lost. Pause should only stop at a successful boundary; preserve the failure when only pause is pending (while still allowing cancellation to take precedence), including the later post-periodic interruption check in Apply.

			// A canceled instruction may fail because its context was killed. Re-check the interrupt
			// state so cancellation is reported as an interruption rather than a plan failure.
			// Pause does not interrupt a running instruction, so a failure observed with pause pending
			// is still a genuine instruction failure.
			result.Interruption = checkInterruption(cancel, pause)

• Files reviewed: 25/27 changed files
• Comments generated: 5
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Pause can suppress genuine failures, terminal probe conflicts can terminate the agent, and Windows termination lacks behavioral tests.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

pkg/applyinator/applyinator.go:469

• A pause that arrives while the current instruction is running is incorrectly treated as the cause of any later non-zero exit. checkInterruption(cancel, pause) returns InterruptionPaused, so reconciliation bypasses normal failure bookkeeping, records an in-progress resume checkpoint, and unpausing reruns the failed instruction without the planner resetting the failed plan. Pause is boundary-only, so a genuine instruction failure must remain a failure; only cancellation should reclassify this failed execution as an interruption.
  pkg/applyinator/process_windows.go:120
• The Windows cancellation path has no executable coverage: all process-tree tests skip Windows, while only the Unix implementation has a platform-specific test (pkg/applyinator/process_unix_test.go:19). Please add a Windows test that assigns a command which spawns a descendant, cancels it, and verifies the Job Object terminates the whole tree (plus the direct-child fallback); otherwise this critical platform-specific behavior can regress while the Windows CI remains green.

• Files reviewed: 25/27 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Resume writes can overwrite concurrent Secret updates, and pauses can incorrectly mask genuine instruction failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 25/27 changed files
• Comments generated: 3
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Interrupt timing can allow canceled or paused work to start, and partial cancellations may omit required warnings.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

pkg/applyinator/applyinator.go:783

• Using exec.Command detaches process startup from ctx, while the cancellation watchdog is not armed until after cmd.Start(). If cancellation arrives during image staging or command setup, ctx is already canceled but the command is still launched and only killed afterward. Preserve a context-aware pre-start check (without bypassing the process-tree watchdog) so canceled instructions cannot start new work.
  pkg/applyinator/process_windows.go:119
• The new Windows Job Object path has no Windows-specific tests, while the process-tree cancellation tests explicitly skip Windows and only the Unix implementation gets a platform test. Add Windows coverage for successful assignment/termination, assignment fallback, and ActiveProcesses confirmation; otherwise the platform-specific cancellation implementation can compile yet fail at runtime unnoticed.

pkg/k8splan/reconcile.go:519

• This suppresses the required partial-application warning when cancellation kills the first instruction: Completed counts only successfully finished instructions, so zero does not mean the running instruction made no changes. It also suppresses warnings for a canceled periodic instruction after all one-time instructions completed. Distinguish cancellation before execution from cancellation during an apply, and emit the inconsistency warning for every in-flight cancellation that may have produced side effects.

func partialCancellationLogs(completed, total int) {
	if completed <= 0 || completed >= total {
		return

• Files reviewed: 25/27 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

The change substantially alters execution, persistence, concurrency, and cross-platform process termination semantics and requires final human validation.

Review details

• Files reviewed: 25/27 changed files
• Comments generated: 0 new
• Review effort level: Balanced