Repository navigation
fix(detection): expect a tool task 302's memory server actually exposes - #201
Open
rudycelekli wants to merge 1 commit into
Open
rudycelekli wants to merge 1 commit into
rudycelekli wants to merge 1 commit into
Conversation
Task 302 lists create_entities as an expected tool, but none of its servers (enterprise_memory_manager, database_server, http_server) expose it. Only registry capabilities are passed to the agent as allowed tools, so the tools_called validation for this task could never pass. The memory step maps to enterprise_memory_manager's update_long_term_objectives. Add a test that every task's servers exist in the registry and every expected tool is a capability of one of the task's servers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this? (check all applicable)
Related issue: N/A (found by reading the code; reproduced on
mainat 6b33dff)What changed?
Replace
create_entitieswithupdate_long_term_objectivesin task 302. Add tests/test_tasks_registry_consistency.py: every task server exists in the registry, and every expected tool is a capability of one of the task's servers.Why?
Detection/tasks.json task 302 has
create_entitiesin expected_tools, but none of its servers (enterprise_memory_manager, database_server, http_server) expose it in mcp_servers_registry.json. main_benchmark only passes registry capabilities as allowed tools, so_validate_expected_toolsfor task 302 can never reach coverage 1.0. The memory step maps to enterprise_memory_manager'supdate_long_term_objectives, which task 111 already uses.How did you test it?
pytest tests/test_tasks_registry_consistency.py. On main: 1 failed ({302: ['create_entities']}). On the branch: 2 passed.pytest tests/ -qwith detcore-venv: 258 passed (main: 256).Potential risks
The change only affects the input described above; other paths go through the same code as before, and the component's existing tests pass.