Skip to content

fix(mlflow): verify or reclaim the tracking port instead of adopting it - #41

Merged
HackedRico merged 3 commits into
mitre:mainfrom
HackedRico:fix/mlflow-srv-ops
Sep 3, 2026
Merged

HackedRico merged 3 commits into
mitre:mainfrom
HackedRico:fix/mlflow-srv-ops

Conversation

@HackedRico

Copy link
Copy Markdown
Collaborator

Description

CALDERA decided MLflow was already running from a bare TCP connect, which only
proves something answers on the port. Whatever held it became the tracking
server for every run. Two ways that broke runs:

  • A listener that is not MLflow, such as AirPlay Receiver on macOS :5000, was
    adopted, so every create_run failed with a 403 and the run died at birth.
  • Another CALDERA tree's server was adopted, so this tree's runs landed in that
    tree's store, and the boot reconcile then marked that tree's live runs KILLED.

The bootstrap now fingerprints the listener first. The tracking API says whether
it is MLflow, and the artifact root says whether it serves this tree's store.

Situation at boot Behaviour
Port free Start MLflow against plugins/mcp/mlruns.db
Our own MLflow already there Adopt it, spawn nothing, kill nothing
A different MLflow on the port Reclaim it, server plus uvicorn workers, and restart against this tree's store
Something that is not MLflow Leave it alone, log the port and the mlflow.port override
Shutdown Stop only the server this process started

An unidentified process is never killed, only reported. The reclaim finds its
target by cmdline, since macOS denies a port to pid lookup without root.

Two smaller changes: MLflow is spawned via the interpreter running CALDERA
rather than a bare mlflow, because restarting is now a normal path and PATH
may not carry the right venv. And the create_run in mcp_svc.execute is
wrapped, since it was the only MLflow write in a run that was not best effort,
so a broken tracking server reached the chat UI as a raw 403 naming neither
MLflow nor the port.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Both failures were reproduced first, then fixed.

  • 12 new unit tests, full suite green, pyflakes clean.
  • Mutation checked. Reinstating the original port check turns four tests
    red. Two refusal tests originally could not fail against the bug they
    described, and now assert that the operator is told.
  • Lifecycle scenarios with real processes and signals on a scratch port:
    clean start and stop, foreign squatter reclaimed, second boot adopts,
    non-MLflow squatter refused and left alive, kill -9 leak then adopted.
  • Real server.py on :5000 with a foreign MLflow squatting on it: logged
    the mismatch, killed the squatter, came up on its own store, minted a run,
    and stopped MLflow with SIGTERM. Clean restart and stop after that.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation (nothing documents
    the MLflow bootstrap; README and PLUGIN_MCP.md mention MLflow only as the
    observability layer)
  • I have added tests that prove my fix is effective or that my feature works

The bootstrap decided MLflow was already running from a bare TCP connect,
which only proves something answers. Two ways that broke runs:

A listener that is not MLflow, such as AirPlay Receiver on macOS :5000,
was adopted as the tracking server, so every create_run failed with a 403
and the operator saw a run that died at birth.

Another Caldera tree's server was adopted too, so this tree's runs landed
in that tree's store, and the boot reconcile then found that tree's live
runs with an empty live cache and marked them KILLED.

Fingerprint the listener instead: the tracking API says whether it is
MLflow at all, and the artifact root says whether it serves this tree's
store. Adopt only our own, reclaim a foreign MLflow by cmdline since
macOS denies a port to pid lookup without root, and leave an
unidentified process alone with a log naming the port and mlflow.port.

Restarting is a normal path now, so spawn via the interpreter running
Caldera rather than a bare "mlflow" that PATH may not carry.
Every MLflow write in a run is best effort except the create_run that
mints it, so a tracking server that is down surfaced only there, as a raw
provider exception. /execute returned "API request to endpoint
/api/2.0/mlflow/experiments/get-by-name failed with error code 403" to
the chat UI, naming neither MLflow nor the port it came from.

Wrap it so the operator is told the tracking server is unavailable, at
which URI, and where to look for the cause.
Both refusal tests only asserted that nothing was spawned and nothing was
killed, which the original bare port check also satisfied, so neither
could fail against the bug they describe. Reinstating the old adopt
anything behaviour left them green.

Assert the log instead: silent adoption was the defect, so not spawning
is not the property that matters, being told why runs are about to fail
is.
@HackedRico
HackedRico merged commit ed33833 into mitre:main Sep 3, 2026
3 checks passed
@HackedRico
HackedRico deleted the fix/mlflow-srv-ops branch September 3, 2026 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant