Fix graceful engine shutdown on SIGTERM and SIGINT

[cooktheryan]

podman container stop fetchit currently waits for the stop timeout and escalates to SIGKILL because Bash runs the engine as a child without forwarding signals. FetchIt also has no termination handler. This change makes the entry script exec FetchIt and handles SIGTERM/SIGINT before engine initialization, so normal shutdown exits promptly with status 0.

An engine lifetime context survives configuration reloads, cancels jobs and the Podman client, and prevents canceled reloads or queued methods from starting new work. Shutdown waits for the scheduler without holding the reload mutex and allows at most five seconds for startup or running jobs to finish. If work does not honor cancellation, the engine logs grace-period expiry and exits; this does not guarantee completion or rollback of arbitrary host operations. Stopping the engine leaves deployed workloads running.

Adds unit coverage for running jobs, uncooperative jobs, canceled methods/reloads, initialization cancellation, and a blocked startup/reload lock. The image workflow now checks SIGTERM/SIGINT against the built image, and the unit workflow includes shutdown tests in its race checks. Running guidance and unreleased notes describe the behavior.

Validation:

• Full Go unit suite and vet passed with the documented build tags.
• Shutdown, retirement, and configuration reload tests passed five repeated runs under the race detector.
• Shell syntax checks, git diff --check, and Sphinx HTML build with warnings as errors passed.
• Built the production Dockerfile on a disposable AWS Fedora 44 x86_64 host with Podman 5.8.7. SIGTERM exited 0 in about one second; SIGINT exited 0 in under one second.
• A deliberately blocked Git clone during startup exited 0 after 5.199 seconds, with the grace-expiry log and no SIGKILL.
• A real workload deployed from a temporary local smart HTTP Git repository remained running after the engine exited 0 in 0.374 seconds, even with cleanupOnRemoval enabled.
• Temporary AWS resources were removed.

The original shutdown hang reproduces on both published latest and current main. On Podman 5.8.7 the stop command itself already returns 0; the forced container exit was 137. Podman sends SIGTERM first and escalates to SIGKILL at the timeout.

Fixes #287.

Summary by Sourcery

Enable FetchIt to terminate gracefully on SIGTERM and SIGINT while bounding shutdown time and preventing new work from starting.

New Features:

• Handle SIGTERM and SIGINT for prompt, graceful engine shutdown with a successful exit status.
• Preserve deployed workloads when the FetchIt engine stops.

Bug Fixes:

• Prevent container shutdown from timing out and escalating to SIGKILL when the engine runs under the entry script.
• Prevent canceled startup, reloads, and queued methods from initializing or starting new work.

Enhancements:

• Add bounded shutdown coordination that cancels engine activity, waits for schedulers, and reports grace-period expiry without blocking configuration reloads.

CI:

• Verify graceful SIGTERM and SIGINT shutdown against the built container image and include shutdown tests in race checks.

Documentation:

• Document engine shutdown behavior and add unreleased release notes.

Tests:

• Add unit coverage for graceful shutdown, uncooperative jobs, canceled work, startup cancellation, and reload-lock contention.

[sourcery-ai]

Sorry @cooktheryan, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 5 days and 22 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[sourcery-ai]

Reviewer's Guide

The PR changes FetchIt from a signal-isolated Bash child into a directly signaled process, then propagates a cancelable engine lifetime through initialization, scheduling, and reloads. Shutdown cancels new and active work, waits for the scheduler without blocking reload coordination, exits successfully after cooperative completion or a bounded five-second grace period, and is covered by unit, race, and built-image integration checks.

Sequence diagram for graceful FetchIt shutdown

sequenceDiagram
    participant Podman
    participant Entry as entry.sh
    participant FetchIt
    participant Scheduler
    participant Jobs

    Podman->>Entry: SIGTERM or SIGINT
    Entry->>FetchIt: forward signal via exec
    FetchIt->>FetchIt: signal.NotifyContext()
    FetchIt->>FetchIt: cancel lifetime context
    FetchIt->>Scheduler: Stop()
    Scheduler->>Jobs: cancel and stop scheduling
    alt jobs finish within 5 seconds
        Jobs-->>Scheduler: completed
        Scheduler-->>FetchIt: shutdown complete
        FetchIt-->>Podman: exit status 0
    else work remains after grace period
        FetchIt-->>Podman: log grace-period expiry
        FetchIt-->>Podman: exit status 0
    end

Loading

Flow diagram for cancellation-aware engine startup and reload

flowchart TD
    Signal["SIGTERM or SIGINT"] --> Cancel["Cancel engine lifetime context"]
    Cancel --> BlockNew["Prevent queued methods and reloads from starting"]
    BlockNew --> StopScheduler["scheduler.Stop()"]
    StopScheduler --> Wait["Wait without holding configRestartMu"]
    Wait --> Complete["Exit 0 after cooperative completion"]
    Wait --> Expire["After five-second grace period"]
    Expire --> Exit["Log expiry and exit 0"]
    Cancel --> Startup["Cancel initialization and startup Git operations"]
    Startup --> Exit

Loading

File-Level Changes

Change	Details	Files
Make the container and engine respond directly to termination signals with prompt, successful shutdown.	Exec FetchIt from the entry script so it becomes PID 1 and receives signals. Register SIGTERM/SIGINT handling around the Cobra command context. Run initialization and the main engine lifetime under a cancelable context. Cancel active work and stop the scheduler during shutdown, with a five-second grace-period bound and expiry logging.	scripts/entry.sh pkg/engine/start.go pkg/engine/shutdown.go pkg/engine/types.go
Make reloads, startup, and queued work honor engine cancellation across the engine lifetime.	Preserve the lifetime context through configuration reloads. Prevent canceled methods, reloads, and startup phases from creating new work. Avoid holding the reload mutex while waiting for scheduler jobs to finish. Handle initialization cancellation without creating the scheduler or Podman connection.	pkg/engine/fetchit.go pkg/engine/shutdown.go
Add coverage for shutdown concurrency and cancellation edge cases.	Test waiting for cooperative running jobs and bounding uncooperative jobs. Test canceled queued methods, reloads, initialization, and lock-blocked shutdown.	pkg/engine/shutdown_test.go
Validate graceful shutdown in CI and document the operational behavior.	Run shutdown tests under the race detector. Exercise SIGTERM and SIGINT against the built container image. Document shutdown, grace-period, and workload-retention behavior. Add unreleased release notes.	.github/workflows/docker-image.yml .github/workflows/unit-tests.yml scripts/test-shutdown.sh docs/running.rst docs/release_notes.rst

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#287	Ensure FetchIt receives SIGTERM and SIGINT directly when running in a container, rather than having signals stop at the shell wrapper.	✅
#287	Implement graceful engine shutdown that promptly cancels or stops scheduled work, waits briefly for running work, and exits successfully without requiring SIGKILL.	✅
#287	Prevent shutdown-time configuration reloads, startup operations, or queued methods from initializing or starting new work, with tests and container-level validation covering the behavior.	✅

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.