Repository navigation
fix(mlflow): nest the logged runs under the pipeline's own run - #27
Merged
hallelx2 merged 1 commit intoOct 25, 2026
Merged
Conversation
`cancerag.pipeline` opens an MLflow run for the pipeline as a whole. The
`advanced` step then calls `log_all_runs`, which opened its own top-level run
per CV fold, and MLflow refuses a second active top-level run:
Run with UUID ... is already active. To start a new run, first end the
current run with mlflow.end_run(). To start a nested run, call start_run
with nested=True
So the step died on its final phase and advanced_summary.json was never
written -- while optuna_trials.csv and the tuned models sat on disk, which
made it look like the step had simply not run yet rather than failed. It is
the second terminal failure in the chain, independent of the figures one.
All five call sites now go through a `_record` helper that nests when a run
is active and stays top-level when none is. Nesting is the right
relationship rather than a workaround: these per-fold and per-experiment
records genuinely are children of the pipeline run. Run standalone, nothing
is active and the behaviour is unchanged.
The tracking URI and experiment are no longer set under an active run
either. They are already fixed by whoever opened it, and a nested run
belongs to its parent's experiment regardless, so setting them only
misreported where the records land.
The tests were checked against the unfixed code: reverting the helper
reproduces the exact production exception.
hallelx2
deleted the
halleluyaholudele/hal-4159-advanced-step-dies-on-mlflow
branch
October 25, 2026 07:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by reading
advanced_traceback.txtbefore the rebuild reaches this step, rather than waiting for it to fail again.cancerag.pipelineopens an MLflow run for the pipeline as a whole (MlflowTelemetry, on unless--no-mlflow).log_all_runsthen opened its own top-level run per CV fold, and MLflow refuses a second active top-level run — so the first row raised and took the step with it.This is a second terminal failure, independent of the figures one. It explains why
--listreportedadvanced nowhileoptuna_trials.csv,lightgbm_tuned_final.joblibandstacking_meta.jsonwere all on disk: Optuna finished, thenrun_allfell over on its final MLflow phase, soadvanced_summary.jsonwas never written. The step looked un-run rather than failed.Change
All five call sites go through a
_recordhelper that nests when a run is active and stays top-level when none is. Nesting is the right relationship, not a workaround — these per-fold and per-experiment records genuinely are children of the pipeline run. Run standalone (python -m cancerag.ml.mlflow_logging), behaviour is unchanged.The tracking URI and experiment are also no longer set under an active run: they are already fixed by whoever opened it, and a nested run belongs to its parent's experiment regardless, so setting them only misreported where the records land.
Verification
start_runreproduces the exact production exception:pytest tests/unit— 474 passed, 31 skipped (was 470; +4 here)ruff checkclean on both touched files (also dropped two pre-existing unused imports)test_every_call_site_goes_through_the_helpergreps the module, so a sixthmlflow.start_runadded later reintroduces nothing silentlyrun_id—log_all_runsmust not close the pipeline's runThe rebuild is in
model_trainingand reachesadvancedin roughly an hour; this goes to the host before then.Closes HAL-4159