[OMEGA-350] Add unit tests for LLM token-budget handling - #362
Conversation
| def make_base(create, name="ASICloud"): | ||
| provider = llm.AIProvider(name, "ASI_API_KEY", "minimax/minimax-m3", "https://example.invalid/v1") | ||
| provider._client = NS(chat=NS(completions=NS(create=create))) | ||
| return provider |
There was a problem hiding this comment.
make_base is defined but never used.
There was a problem hiding this comment.
@paul-v-snet thanks for catching this, pushed the fix in 7f5bb68.
| def test_empty_reply_out_of_budget_is_explained(): | ||
| create = FakeCreate(chat_response("", "length")) | ||
| assert sent_text(make_openrouter(create).chat(PROMPT)) == llm.LLM_EMPTY_RESPONSE_MESSAGE | ||
|
|
||
|
|
||
| def test_empty_reply_with_stop_is_not_blamed_on_the_budget(): | ||
| create = FakeCreate(chat_response("", "stop", completion_tokens=0, reasoning_tokens=0)) | ||
| assert make_openrouter(create).chat(PROMPT) == "" | ||
|
|
||
|
|
||
| def test_openai_empty_reply_out_of_budget_is_explained(): | ||
| create = FakeCreate(responses_response("", "incomplete", "max_output_tokens")) | ||
| assert sent_text(make_openai(create).chat(PROMPT, max_tokens=120)) == llm.LLM_EMPTY_RESPONSE_MESSAGE | ||
|
|
||
|
|
||
| def test_openai_empty_reply_without_incomplete_reason_returns_empty(): | ||
| create = FakeCreate(responses_response("", "completed", output_tokens=0, reasoning_tokens=0)) | ||
| assert make_openai(create).chat(PROMPT) == "" | ||
|
|
||
|
|
||
| def test_asione_empty_reply_out_of_budget_is_explained(): | ||
| create = FakeCreate(chat_response("", "length")) | ||
| assert sent_text(make_asione(create).chat(PROMPT)) == llm.LLM_EMPTY_RESPONSE_MESSAGE |
There was a problem hiding this comment.
All test_*_empty_reply_* tests also pass when chat() crashes, since each provider's chat() returns "" when it catches an exception. Is this the expected behavior?
There was a problem hiding this comment.
@paul-v-snet thanks for the catch, it holds for two of them.
These three expect the notice, so a crash inside chat() already made them fail.
- test_empty_reply_out_of_budget_is_explained
- test_openai_empty_reply_out_of_budget_is_explained
- test_asione_empty_reply_out_of_budget_is_explained
These two expect "" and did pass on a crash, because chat() catches the exception and returns "" in lib_llm_ext.py, openai.py and asione.py.
- test_empty_reply_with_stop_is_not_blamed_on_the_budget
- test_openai_empty_reply_without_incomplete_reason_returns_empty
They now also check through swallowed_errors that chat() logged no exception. With an exception raised inside chat() all five tests fail.
Pushed the fix in 7f5bb68.
paul-v-snet
left a comment
paul-v-snet
left a comment
There was a problem hiding this comment.
Looks good to me now, thank you @TossSky, approved.
2 participants
Description
Follow-up to #337, based on its branch. Unit tests only, no behaviour change.
Autotests/unit/test_llm_budget.py(15 tests), registered inrun_mandatory, covers what #337 added: thefinish_reasonandincomplete_reasonchecks before the notice, the notice itself on an empty reply that ran out of budget for OpenRouter, OpenAI and ASI:One, the ASI:One reasoning budget mapping, the OpenRouter reasoning body,[LLM_USAGE]at INFO on both APIs, and an API error returning an empty string. The provider modules are loaded by file path withopenaiandconfigstubbed, so the tests need no container, network or token.Earlier versions of this PR also changed provider behaviour: a retry without reasoning, one notice per streak, and keeping a truncated reply out of the loop. All three are dropped after the discussion here and in #337.
How Has This Been Tested?
CI:
tests/pytest.sh65 passed, Phase 1 144 passed (129 + 15 new), Phase 2 6 passed, MeTTa tests green.Checklist