Repository navigation
Insights: treat the model's cold start as a wait, not a failure - #52
Conversation
The Modal endpoint scales to zero and answers 503 "no upstreams available" while a container boots; the first refused request is what starts it. Both the chat and the takeaways run reported that as "the analysis did not finish". Now opening the page probes /health to wake the model, status reports modelState, a chat during the boot is a 503 with starting=true that the page waits out (polling status, up to eight minutes) before resending, and the takeaways thread waits for ready before it runs.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| def chat(self, database_id, org_name, history): | ||
| """`history` is the validated (role, content) list. Returns the JSON | ||
| the page renders: the answer plus the queries that produced it.""" | ||
| if self.model_state() != "ready": |
There was a problem hiding this comment.
Blocking. Remove this pre-check, or treat only a probe 503 as "starting". The chat must still send the real request when /health fails for any other reason.
_probe returns False on every failure: a 404, a 401, a DNS error, a timeout, or a 5xx other than 503. chat then raises ModelStarting and never calls the model. The comment at server/insights.py:40 says HOTOTEL_INSIGHTS_BASE_URL accepts "any OpenAI-compatible endpoint". Most such endpoints (OpenAI, OpenRouter, a vLLM behind a path-rewriting proxy) have no /health beside /v1.
Failure scenario: an operator points HOTOTEL_INSIGHTS_BASE_URL at https://api.openai.com/v1. /health answers 404, so modelState is always starting. Every chat returns 503 starting: true. The page waits 8 minutes, then shows "the model did not start". The takeaways thread also waits 8 minutes in wait_ready and caches an error. Before this PR, the same configuration worked.
Suggested fix:
- In
chat, drop the gate._runalready turns a client 503 intoModelStartingthrough_is_cold_start, and that 503 is the request that wakes Modal. - In
_probe, return"starting"only forHTTPErrorwithcode == 503, or for a connection error. Treat 404 asready, so endpoints without/healthare not blocked. - Add a test where
/healthreturns 404 andmodel_state()returnsready.
There was a problem hiding this comment.
Fixed in 6de0ce0: the gate in chat is gone (_run's 503 -> ModelStarting is the only path), _probe returns starting only for HTTPError 503 or a connection failure, and test_the_engine_tells_a_cold_start_from_a_failure now runs a local /health that answers 404 (-> ready) and 503 (-> starting).
| if self.insights: | ||
| # opening the page is the moment to wake a scaled-to-zero | ||
| # model, so it is booting while the person reads or types | ||
| self.insights.warm() |
There was a problem hiding this comment.
nit: Remove the warm() call (not blocking). The next line calls self.insights.status(), which calls model_state() synchronously in the same request. When the cache has expired, warm() and status() each send a separate /health probe at the same time. model_state does not deduplicate a probe that is already running. The status request still blocks up to the 10-second probe timeout either way.
There was a problem hiding this comment.
Removed warm() and its call in 6de0ce0; status() probes synchronously and that probe wakes the endpoint.
| # -- the endpoint's cold start ------------------------------------------- | ||
|
|
||
| def model_state(self): | ||
| """'ready', 'starting' or 'ready' for a hosted API that is always up. |
There was a problem hiding this comment.
super nit: Fix the docstring wording (not blocking). "'ready', 'starting' or 'ready'" lists ready twice. Write "'ready' or 'starting'; always 'ready' for a hosted API".
| # a background thread can afford to wait out a cold start; the | ||
| # page shows "starting the model" meanwhile | ||
| if not self.wait_ready(): | ||
| raise ModelStarting() |
There was a problem hiding this comment.
nit: Give this exception a message (not blocking). ModelStarting() has no message, so the except block at line 498 logs error: takeaways for <id>: with no cause. An operator cannot tell an 8-minute cold-start timeout from other failures. Use raise ModelStarting(f"model not ready after {MODEL_START_TIMEOUT}s").
There was a problem hiding this comment.
Done in 6de0ce0: ModelStarting carries the timeout message, and the in-run case carries the client error text.
There was a problem hiding this comment.
Review
Blocking Issues
server/insights.py:453:chatrefuses to run whenever the/healthprobe fails, for any reason._probereturnsFalseon 404, 401, DNS errors, timeouts and every 5xx. The code documentsHOTOTEL_INSIGHTS_BASE_URLas "any OpenAI-compatible endpoint" (server/insights.py:40). An endpoint with no/healthbeside/v1reportsstartingforever. On such an endpoint, every chat returns 503starting: true, and the takeaways thread waits 8 minutes before it caches an error.
Action Required
- Remove the
model_state()gate inchat._runalready turns a client 503 intoModelStartingthrough_is_cold_start. - In
_probe, return not-ready only for HTTP 503 or a connection error. Treat other responses, such as 404, as ready. - Add a test where
/healthanswers 404 andmodel_state()returnsready.
…ute wait A cold boot read the 57GB of weights off the volume at ~60MB/s (a minute per shard, 16 shards) and the 15-minute startup timeout killed it, so the page only ever saw 503s. The class now allows 30 minutes and loads the shards with vLLM's parallel prefetch strategy, which the loader only turns on by itself for NFS/Lustre, not the volume's 9P mount. The server and the page wait up to 20 minutes for "ready" to match.
| if (d.status === 'pending') { | ||
| meta.textContent = 'analysing the last 30 days, usually one to three minutes'; | ||
| meta.textContent = d.modelState === 'starting' | ||
| ? 'starting the model first, two to three minutes' |
There was a problem hiding this comment.
super nit: Change "two to three minutes" to "usually a few minutes" (not blocking). This commit raised the wait to 20 minutes and changed STARTING_NOTE to match, but the takeaways card still promises two to three minutes.
There was a problem hiding this comment.
Changed to 'usually a few minutes' in 6de0ce0.
There was a problem hiding this comment.
Review
Blocking Issues
server/insights.py:453andserver/insights.py:355: The blocking issue from the previous review (thread #4301117480) is still open. Commit b6d096c did not change the code.chatstill refuses to send the request whenmodel_state()is notready._probestill returnsFalseon every failure, including a 404 from an endpoint that has no/health. WhenHOTOTEL_INSIGHTS_BASE_URLpoints at an OpenAI-compatible API without/health, every chat returns 503starting: true. The page then waits 20 minutes, which is longer than the 8-minute wait in the previous commit, before it reports a failure. The takeaways thread waits the same 20 minutes inwait_readyand then caches an error.
Action Required
- Remove the
model_state()gate inchat._runalready turns a client 503 intoModelStarting. - In
_probe, return not-ready only for anHTTPErrorwith code 503 or for a connection error. Treat other HTTP codes, such as 404, as ready. - Add a test where
/healthreturns 404 andmodel_state()returnsready.
…ys asks Review fixes. The chat no longer gates on the probe: _run turns the endpoint's 503 into ModelStarting, and that request is what wakes Modal. The probe reports "starting" only for a 503 or a connection failure, so an OpenAI-compatible host without /health (404) is treated as up. The separate warm() call went: status() probes synchronously already. ModelStarting carries a message so the log says why a run gave up.
The model is deliberately not kept on around the clock. A successful sign-in starts its cold boot in the background, so it is minutes along by the time the person opens Insights, and it still scales to zero ten minutes after the last request. A failed sign-in wakes nothing.
1 participant
On hototel.com both the chat and the takeaways run failed with "the analysis did not finish". Server logs:
Error code: 503 - {'error': 'no upstreams available'}from the Modal endpoint. The model container had scaled to zero; Modal's endpoint rejects with 503 while a new one boots, and that first request is what starts it (per Modal's Servers docs). The boot itself was also reading weights off the volume at ~60MB/s and would have been killed by the 15-minute startup timeout.The model stays scale-to-zero on purpose: it runs only after someone signs in, for as long as they keep using it.
/api/insights/statusprobes/health(which also wakes it) and reportsmodelState(ready/starting), cached a few seconds. Only a 503 or no connection means starting; an OpenAI-compatible host without/health(404) counts as up.starting: true; the page shows "Starting the model…", polls status every 10s (up to 20 min) and resends the question.readybefore running; the card says "starting the model first".startup_timeout30 min and--safetensors-load-strategy prefetchfor the volume's 9P mount. Redeployed; a cold start now measures 184s.Tests: three new cases in
test_insights.py; all suites pass.