Skip to content

feat: workflow staging - #140

Merged
vtnphan merged 11 commits into
devfrom
feat/workflow-staging
Aug 21, 2026
Merged

feat: workflow staging#140
vtnphan merged 11 commits into
devfrom
feat/workflow-staging

Conversation

@vtnphan

@vtnphan vtnphan commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Pull Request

Summary

SBP-487 Gadi compute nodes have no network access, so Nextflow can't clone a workflow's pipeline repo from GitHub itself at run time. This PR adds a workflow-repo-staging service that clones a workflow's GitHub repo as a bare git repo (plus a plain checkout for pipeline-bundled assets) and stages both onto Gadi via Globus, caching the result per commit sha so every run of the same workflow reuses it for free.

Changes

  • app/services/workflow_repo_staging.py (new): resolves a workflow's (repo_url, default_revision) to its current commit sha via the GitHub API, clones it as a bare repo (git init --bare + git fetch --depth 1 + git gc with ref-packing disabled so the branch ref survives as an uploadable file), extracts a plain checkout of the same commit via git archive for pipeline-bundled assets (e.g. bindcraft's default settings JSON), and uploads both to Gadi in one Globus transfer. Exposes ensure_repo_staging_requested (cheap, synchronous - called at launch time) and sync_workflow_repo_staging (the actual clone/upload, run by the scheduler).
  • app/db/models/core.py: adds repo_staged_commit_sha, repo_staging_status, repo_gadi_path, repo_staging_transfer_id, repo_staging_error_message, repo_staging_updated_at to Workflow, caching the most recently staged commit per workflow.
  • alembic/versions/20260818_033139_add_workflow_repo_staging_cache_b1f4c9a7e021.py (new): migration for the columns above.
  • app/scheduler/jobs.py, app/run_scheduler.py: new sync_workflow_repo_staging scheduled job (2 min interval), mirroring the existing sync_data_transfers job.
  • app/routes/workflows.py: launch requests now call ensure_repo_staging_requested and point Seqera's pipeline param at the staged bare repo (file:<gadi_path>.git) instead of GitHub directly; bindflow's settings_filters/settings_advanced samplesheet columns are filled in with the staged repo's bundled default JSON paths when left unset by the frontend.
  • app/services/bindflow_executor.py: resolve_bindflow_asset_path fills in the known default asset path (under the staged repo's assets checkout) when a settings field is empty, and passes through any custom value unchanged.
  • app/services/globus_transfer.py: minor changes supporting the repo-staging transfer (shared with existing S3-input staging).
  • app/db/admin.py, app/config.py: adds an AdminSettings.title field (fixes .env's DB_ADMIN_TITLE breaking settings validation) and threads settings through _mount_starlette_admin explicitly instead of re-resolving it internally.
  • Dockerfile: installs git, required by the new bare-repo-clone staging step.

How to Test

  1. Apply the new alembic migration (alembic upgrade head).
  2. Launch any workflow (e.g. bindcraft/de-novo-design) via the API - confirm a Workflow.repo_staging_status row moves pendingin_progresscompleted as the scheduler's sync_workflow_repo_staging job runs, and that repo_gadi_path is populated.
  3. Confirm the Seqera launch payload's pipeline field is file:<repo_gadi_path>.git, and that the run picks up bindflow's default settings JSON from the staged repo when settings_filters/settings_advanced are left unset.
  4. pytest -q (819 passing), ruff check app, mypy app.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Checklist

  • I have added tests that prove my fix is effective or that my feature works
  • I have added or updated documentation where necessary
  • I have run linting and unit tests locally
  • The code follows the project's style guidelines

@vtnphan
vtnphan marked this pull request as ready for review August 20, 2026 04:12
@vtnphan vtnphan changed the title Feat/workflow staging feat: workflow staging Aug 20, 2026

@marius-mather marius-mather left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try to do a full review later, but the first thing that stood out to me is we are doing unauthenticated requests to GitHub and risk getting blocked/rate limited. I think we should do this properly by authenticating with a secret/token.

My preference is usually to use an established library to deal with an API, like https://pygithub.readthedocs.io/en/stable/examples/Repository.html, but I guess we'd need to look at whether it can do some of the specific git operations we want to know if it's worth it.

Comment thread app/services/workflow_repo_staging.py Outdated

@marius-mather marius-mather left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good apart from the previous comments about GitHub API use

Comment thread app/routes/workflows.py
@marius-mather
marius-mather self-requested a review August 21, 2026 03:48

@marius-mather marius-mather left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good to go. remember to add the required secrets into infra before deploying/merging

@vtnphan
vtnphan merged commit c586418 into dev Aug 21, 2026
3 checks passed
@vtnphan
vtnphan deleted the feat/workflow-staging branch August 21, 2026 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants