No response show in the page

[linuaries]

Local LLMs is ollama(llama3.1:8b), query:"Can you give the 5 biggest cities n population size in order". After several steps, there is no response in page.

Here is the log of flask:
"127.0.0.1 - - [21/Sep/2024 10:11:30] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [21/Sep/2024 10:11:51] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [21/Sep/2024 10:14:47] "GET /query?query=Can%20you%20give%20the%205%20biggest%20cities%20in%20population%20size%20in%20order%3F HTTP/1.1" 200 -
Final answer requested but not enough steps provided. Continuing...
Step 2 content exceeded 700 characters. Retrying...
Final answer requested but not enough steps provided. Continuing...
Building index...
Index built and saved
127.0.0.1 - - [21/Sep/2024 10:17:54] "GET /query?query=Can%20you%20give%20the%205%20biggest%20cities%20in%20population%20size%20in%20order%3F HTTP/1.1" 200 -
Final answer requested but not enough steps provided. Continuing...
Step 3 content exceeded 700 characters. Retrying...
Step 4 content exceeded 700 characters. Retrying...
Step 7 content exceeded 700 characters. Retrying...
Building index...
Index built and saved
"

[punnerud]

Does it work with any other questions?
I see that answers is created "Step 3 content exceeded 700 characters. Retrying..." but they are considered too long for a good step-by-step reasoning process after my tests. Thats why its restarts and do another query if the answer is over 700 characters.

[punnerud]

The 700-character retry you spotted was the cause, and it was worse than slow — it never terminated.

app.py:258-261 did continue without incrementing step_count, so while step_count < max_steps could not advance. Nothing was yielded to the browser during any of it, so the page stayed blank while the terminal repeated the same line forever. Reproduced against current main with a model that always answers long:

Step 1 content exceeded 700 characters. Retrying...
Step 1 content exceeded 700 characters. Retrying...
... (indefinitely)
  STILL RUNNING after 15 seconds, having yielded 0 events

There was a second copy of the same mistake at app.py:330-336: when the model asked to finish before reaching 5 steps, that branch also did continue without incrementing, so a model that wanted to answer immediately hung in exactly the same way.

Fixed in #4. Both branches now retry at most three times and then use what the model gave, truncated, with a marker in the UI saying the step was shortened. The stream also sends a heartbeat while the model is thinking, so a slow first token no longer looks like a hang.

Two related things that were making long answers more likely in the first place:

• max_tokens was sent as a top-level key, where Ollama ignores it — it belongs under options as num_predict. So the 300-token limit was never actually applied. Same for temperature.
• Steps are now requested with a JSON response schema, so the model cannot answer with prose that fails to parse. That also removes the Step 5: Parsing Error node visible in the project's own example.png.

Tests now feed in a model that always answers too long, and one that always wants to stop early, and require the endpoint to terminate in both cases.

[punnerud]

Fixed and released in mpe-lkg 0.3.0.

Both retry branches now stop after three attempts and use what the model gave, truncated, with a marker in the page saying the step was shortened. The stream also sends a heartbeat while the model is thinking, so a slow first token no longer looks like a hang.

Two things that were making long answers likelier in the first place are fixed too: max_tokens was being sent as a top-level key where Ollama ignores it (it belongs under options as num_predict, so the 300-token limit never applied), and steps are now requested with a JSON response schema so the model cannot answer with prose that fails to parse — which also removes the Step 5: Parsing Error node visible in this project's own example.png.

Regression tests feed in a model that always answers too long, and one that always wants to stop early, and require the endpoint to terminate in both cases. They run on Linux, macOS and Windows on every push.

pip install --upgrade mpe-lkg
mpe-lkg