Skip to content

Handle SIGTERM for a graceful shutdown that drains in-flight ingest ticks and HTTP requests #271

Closed
#375
Closed
Handle SIGTERM for a graceful shutdown that drains in-flight ingest ticks and HTTP requests#271
#375
Assignees
graceuvala-collab
Labels
Stellar WaveIssues in the Stellar wave programarea/backendBackend crates: api, store, ingest, webhooks, bin/serverdifficulty/mediumMedium difficultyenhancementNew feature or request

Description

@Emmyt24
Emmyt24
opened on Sep 23, 2026

Description

bin/server/src/main.rs starts the axum HTTP server and the ingest supervisor. Audit whether the process currently handles SIGTERM gracefully (draining in-flight requests and letting an in-progress ingest tick finish its current page before the process exits) or exits immediately, which in a rolling-deploy environment (Kubernetes, ECS) can abort an in-flight ingest page mid-processing or drop an in-flight client request.

Requirements and Context

  • If graceful shutdown isn't already implemented, add it using axum's with_graceful_shutdown and a signal-handling future.
  • The ingest supervisor's current tick should be allowed to finish (up to a bounded timeout) rather than being aborted mid-page.
  • Log the shutdown sequence clearly (signal received, draining, drained, exiting) so it's diagnosable in production logs.

Suggested Execution

Branch: feat/server/graceful-shutdown

Implement Changes

  • Add a tokio::signal::unix::signal(SignalKind::terminate()) listener in bin/server/src/main.rs.
  • Wire it into axum's .with_graceful_shutdown(...) for the HTTP server.
  • Add a bounded-wait mechanism (e.g. a CancellationToken from tokio_util) that the ingest supervisor's loop checks between ticks, with a maximum drain timeout before forcing exit.

Test and Commit

  • server_stops_accepting_new_connections_on_sigterm_but_drains_in_flight_ones (integration test using a slow test route).
  • ingest_supervisor_finishes_its_current_tick_before_honoring_a_shutdown_signal.
  • server_force_exits_after_the_drain_timeout_if_a_tick_hangs.
  • Run cargo test --workspace locally before committing.

Example Commit Message

feat(server): graceful shutdown on SIGTERM draining in-flight work

The process previously exited immediately on SIGTERM, which could abort an in-flight
ingest page mid-processing or drop an in-flight HTTP request during a rolling deploy.
Adds graceful draining with a bounded timeout.

Guidelines

  • Document the drain timeout value and how an operator would tune it via env var, in README.md's deployment section.
  • Reference this issue with Closes #<issue-number> in the PR description.
  • Open your pull request against the dev-branch branch — PRs targeting main will not be reviewed.

Activity

  1. Emmyt24
    added
    enhancementNew feature or request
    area/backendBackend crates: api, store, ingest, webhooks, bin/server
    difficulty/mediumMedium difficulty
    on Sep 23, 2026
  2. drips-wave
    added
    Stellar WaveIssues in the Stellar wave program
    on Sep 24, 2026

  3. graceuvala-collab commented on Sep 24, 2026

    @graceuvala-collab
    graceuvala-collab
    on Sep 24, 2026 – with Drips Wave

    @graceuvala-collab has applied to work on this issue as part of the Stellar Wave Program's 9th wave.

    Hi! I’m a full-stack developer with a strong blockchain background. I’d love to handle this issue and can deliver a clean, reliable solution. Kindly assign it to me.

    ℹ️ Repo Maintainers: To accept this application, review their application or assign @graceuvala-collab to this issue.

  4. Emmyt24
    assigned
    graceuvala-collab
    on Sep 24, 2026

  5. drips-wave commented on Sep 24, 2026

    @drips-wave
    drips-wavebot
    on Sep 24, 2026 – with Drips Wave

    Congratulations, @graceuvala-collab! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on September 30, 2026.

    🧑‍💻 @graceuvala-collab: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

    Warning

    When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

    🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

    🌊 Happy Wave 🌊

  6. Emmyt24
    closed this as completedin #375on Sep 28, 2026
  7. Emmyt24
    added a commit that references this issue on Sep 28, 2026

    fix: graceful shutdown, resumable key backfill, constant-time OTP com…

    23a95b0

  8. grantfox-oss commented on Sep 28, 2026

    @grantfox-oss
    grantfox-ossbot
    on Sep 28, 2026 – with GrantFox OSS

    🎉 This issue has been marked as completed on GrantFox!

    @graceuvala-collab's PR #375 was approved and merged by @Emmyt24.

    🏆 @graceuvala-collab: You earned 35 FoxPoints for this contribution! Your current tier: Explorer (188 total points). Track your full progress on GrantFox.

    👏 Great work, @graceuvala-collab! Keep contributing to Octo-Protocol-org.

  9. drips-wave commented on Sep 28, 2026

    @drips-wave
    drips-wavebot
    on Sep 28, 2026 – with Drips Wave

    This issue has been completed by @graceuvala-collab as part of the Stellar Wave Program's 9th Wave 🥳

    😎 @graceuvala-collab: You earned 200 Points for completing this issue! After the current Wave ends, you'll be eligible for a percentage of the Wave's reward pool based on the percentage of total points you've earned. Learn more here. You can also Leave a review to share your experience working on this issue.

    🧑‍💻 Repo maintainers: How'd the contributor do? Leave a review to share your experience working with them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Stellar WaveIssues in the Stellar wave programarea/backendBackend crates: api, store, ingest, webhooks, bin/serverdifficulty/mediumMedium difficultyenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions