Skip to content

fix: retry provider discovery periodically instead of once at startup - #19

Open
AditiSah05 wants to merge 1 commit into
mofa-org:mainfrom
AditiSah05:feat/discovery-retry-ticker
Open

AditiSah05 wants to merge 1 commit into
mofa-org:mainfrom
AditiSah05:feat/discovery-retry-ticker

Conversation

@AditiSah05

Copy link
Copy Markdown

Problem

Provider discovery (discover_all()) ran exactly once, at engine startup (engine.rs, in the constructor). If a provider — e.g. Ollama — wasn't reachable at that exact moment, the engine marked it unavailable and never checked again.

Consequence: start Ollama after the engine (slow boot, forgot to start it, container startup-order race), and /v1/capabilities stays empty forever. The only way out is a manual POST /v1/discovery/refresh call or a full engine restart — neither is discoverable to a normal user, and both contradict the project's own "zero-config, auto-detects, auto-recovers" pitch in the README.

The engine already runs two other background loops that repeat forever — spawn_idle_eviction() (unloads idle models) and spawn_artifact_sweep() (cleans old files). Discovery had no equivalent.

Fix

  • mofa-engine-core/src/engine.rs

    • New spawn_discovery_ticker() — same pattern as the two existing tickers: a tokio::spawn loop on a tokio::time::interval, calling refresh_resources() (discovery + health) every tick.
    • Called at the end of the constructor, alongside the other two ticker spawns.
    • New fields discovery_task (task handle) and discovery_interval; both cleaned up in Drop (task aborted on engine shutdown, matching existing tasks).
    • New test discovery_ticker_picks_up_a_provider_that_was_down_at_startup: a fake provider (FlakyDiscoveryProvider) fails discovery until flipped "available" mid-test; the test asserts the model appears in capabilities() on its own, with no manual refresh_resources() call after the initial startup check.
    • Mechanical: build_engine_with_pricing test helper gained a discovery_secs parameter (3 call sites updated); a few explicit EngineConfig { ... } test literals needed the new field.
  • mofa-engine-core/src/config.rs

    • New DiscoveryConfig struct: refresh_secs: u64, default 30, 0 disables the background refresh (same "0 = off" convention as idle_timeout_secs and artifacts.retention_secs).
    • Wired into EngineConfig as a new discovery field.
  • mofa-engine-sdk/src/client.rs, mofa-engine-sdk/src/server.rs

    • No behavior change — test-only EngineConfig { ... } literals updated to include the new field so the crates still compile.
  • config.example.toml

    • New [discovery] section documenting refresh_secs.

Testing

  • cargo test -p mofa-engine-core — 179 passed. (9 pre-existing failures are unrelated to this change: local TTS/ASR/image-gen fixture tests spawn sh, which isn't on PATH on this Windows dev machine.)
  • cargo test -p mofa-engine-sdk — 17 passed, including the existing auth_gate_rejects_missing_token_but_allows_public_routes test (unaffected).
  • cargo build --workspace — clean.

Live-verified end to end, not just unit-tested:

  1. Ran the built engine with Ollama genuinely down. Log showed the discovery warning repeating every refresh_secs (5s in this run), not a single one-shot warning as before.
  2. Brought a fake Ollama HTTP responder online on port 11434 mid-run, with the engine still running (no restart).
  3. Within one tick interval, the log showed discovered 1 models from 'ollama' and /v1/capabilities returned the model — confirmed in the dashboard UI too.

Non-goals

  • No UI/dashboard code changed — the dashboard already polls /v1/capabilities; this PR just makes the data behind that endpoint stay fresh.
  • No change to the existing manual POST /v1/discovery/refresh endpoint — still works, still useful for an immediate on-demand check.

@AditiSah05
AditiSah05 force-pushed the feat/discovery-retry-ticker branch from 7794440 to 448ddf6 Compare September 2, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant