feat: workflow staging - #140
Conversation
marius-mather
left a comment
There was a problem hiding this comment.
I will try to do a full review later, but the first thing that stood out to me is we are doing unauthenticated requests to GitHub and risk getting blocked/rate limited. I think we should do this properly by authenticating with a secret/token.
My preference is usually to use an established library to deal with an API, like https://pygithub.readthedocs.io/en/stable/examples/Repository.html, but I guess we'd need to look at whether it can do some of the specific git operations we want to know if it's worth it.
marius-mather
left a comment
There was a problem hiding this comment.
looking good apart from the previous comments about GitHub API use
marius-mather
left a comment
There was a problem hiding this comment.
good to go. remember to add the required secrets into infra before deploying/merging
Pull Request
Summary
SBP-487 Gadi compute nodes have no network access, so Nextflow can't clone a workflow's pipeline repo from GitHub itself at run time. This PR adds a workflow-repo-staging service that clones a workflow's GitHub repo as a bare git repo (plus a plain checkout for pipeline-bundled assets) and stages both onto Gadi via Globus, caching the result per commit sha so every run of the same workflow reuses it for free.
Changes
app/services/workflow_repo_staging.py(new): resolves a workflow's(repo_url, default_revision)to its current commit sha via the GitHub API, clones it as a bare repo (git init --bare+git fetch --depth 1+git gcwith ref-packing disabled so the branch ref survives as an uploadable file), extracts a plain checkout of the same commit viagit archivefor pipeline-bundled assets (e.g. bindcraft's default settings JSON), and uploads both to Gadi in one Globus transfer. Exposesensure_repo_staging_requested(cheap, synchronous - called at launch time) andsync_workflow_repo_staging(the actual clone/upload, run by the scheduler).app/db/models/core.py: addsrepo_staged_commit_sha,repo_staging_status,repo_gadi_path,repo_staging_transfer_id,repo_staging_error_message,repo_staging_updated_attoWorkflow, caching the most recently staged commit per workflow.alembic/versions/20260818_033139_add_workflow_repo_staging_cache_b1f4c9a7e021.py(new): migration for the columns above.app/scheduler/jobs.py,app/run_scheduler.py: newsync_workflow_repo_stagingscheduled job (2 min interval), mirroring the existingsync_data_transfersjob.app/routes/workflows.py: launch requests now callensure_repo_staging_requestedand point Seqera'spipelineparam at the staged bare repo (file:<gadi_path>.git) instead of GitHub directly; bindflow'ssettings_filters/settings_advancedsamplesheet columns are filled in with the staged repo's bundled default JSON paths when left unset by the frontend.app/services/bindflow_executor.py:resolve_bindflow_asset_pathfills in the known default asset path (under the staged repo's assets checkout) when a settings field is empty, and passes through any custom value unchanged.app/services/globus_transfer.py: minor changes supporting the repo-staging transfer (shared with existing S3-input staging).app/db/admin.py,app/config.py: adds anAdminSettings.titlefield (fixes.env'sDB_ADMIN_TITLEbreaking settings validation) and threadssettingsthrough_mount_starlette_adminexplicitly instead of re-resolving it internally.Dockerfile: installsgit, required by the new bare-repo-clone staging step.How to Test
alembic upgrade head).Workflow.repo_staging_statusrow movespending→in_progress→completedas the scheduler'ssync_workflow_repo_stagingjob runs, and thatrepo_gadi_pathis populated.pipelinefield isfile:<repo_gadi_path>.git, and that the run picks up bindflow's default settings JSON from the staged repo whensettings_filters/settings_advancedare left unset.pytest -q(819 passing),ruff check app,mypy app.Type of change
Checklist