Skip to content

fix(mlflow): nest the logged runs under the pipeline's own run - #27

Merged
hallelx2 merged 1 commit into
mainfrom
halleluyaholudele/hal-4159-advanced-step-dies-on-mlflow
Oct 25, 2026
Merged

hallelx2 merged 1 commit into
mainfrom
halleluyaholudele/hal-4159-advanced-step-dies-on-mlflow

Conversation

@hallelx2

Copy link
Copy Markdown
Collaborator

Found by reading advanced_traceback.txt before the rebuild reaches this step, rather than waiting for it to fail again.

Exception: Run with UUID 8ec8e4919f6d403bb6bc8ae0bb9a1acb is already active.
To start a new run, first end the current run with mlflow.end_run().
To start a nested run, call start_run with nested=True

  cancerag/ml/advanced.py:73        summary["phases"]["mlflow"] = log_all_runs()
  cancerag/ml/mlflow_logging.py:58  with mlflow.start_run(run_name=...)

cancerag.pipeline opens an MLflow run for the pipeline as a whole (MlflowTelemetry, on unless --no-mlflow). log_all_runs then opened its own top-level run per CV fold, and MLflow refuses a second active top-level run — so the first row raised and took the step with it.

This is a second terminal failure, independent of the figures one. It explains why --list reported advanced no while optuna_trials.csv, lightgbm_tuned_final.joblib and stacking_meta.json were all on disk: Optuna finished, then run_all fell over on its final MLflow phase, so advanced_summary.json was never written. The step looked un-run rather than failed.

Change

All five call sites go through a _record helper that nests when a run is active and stays top-level when none is. Nesting is the right relationship, not a workaround — these per-fold and per-experiment records genuinely are children of the pipeline run. Run standalone (python -m cancerag.ml.mlflow_logging), behaviour is unchanged.

The tracking URI and experiment are also no longer set under an active run: they are already fixed by whoever opened it, and a nested run belongs to its parent's experiment regardless, so setting them only misreported where the records land.

Verification

  • The tests were checked against the unfixed code. Reverting the helper to a plain start_run reproduces the exact production exception:
    FAILED test_logs_runs_inside_an_active_run - Exception: Run with UUID … is already active
    FAILED test_nested_flag_follows_whether_a_run_is_active - KeyError: 'nested'
    
    A test that passes against the bug would not be worth having.
  • pytest tests/unit — 474 passed, 31 skipped (was 470; +4 here)
  • ruff check clean on both touched files (also dropped two pre-existing unused imports)
  • test_every_call_site_goes_through_the_helper greps the module, so a sixth mlflow.start_run added later reintroduces nothing silently
  • Asserts the outer run survives with the same run_id — log_all_runs must not close the pipeline's run

The rebuild is in model_training and reaches advanced in roughly an hour; this goes to the host before then.

Closes HAL-4159

`cancerag.pipeline` opens an MLflow run for the pipeline as a whole. The
`advanced` step then calls `log_all_runs`, which opened its own top-level run
per CV fold, and MLflow refuses a second active top-level run:

    Run with UUID ... is already active. To start a new run, first end the
    current run with mlflow.end_run(). To start a nested run, call start_run
    with nested=True

So the step died on its final phase and advanced_summary.json was never
written -- while optuna_trials.csv and the tuned models sat on disk, which
made it look like the step had simply not run yet rather than failed. It is
the second terminal failure in the chain, independent of the figures one.

All five call sites now go through a `_record` helper that nests when a run
is active and stays top-level when none is. Nesting is the right
relationship rather than a workaround: these per-fold and per-experiment
records genuinely are children of the pipeline run. Run standalone, nothing
is active and the behaviour is unchanged.

The tracking URI and experiment are no longer set under an active run
either. They are already fixed by whoever opened it, and a nested run
belongs to its parent's experiment regardless, so setting them only
misreported where the records land.

The tests were checked against the unfixed code: reverting the helper
reproduces the exact production exception.
@hallelx2
hallelx2 merged commit a817151 into main Oct 25, 2026
2 checks passed
@hallelx2
hallelx2 deleted the halleluyaholudele/hal-4159-advanced-step-dies-on-mlflow branch October 25, 2026 07:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant