To test each of the 5 models, I used a fixed set of 5 specific tasks (instances) from the sympy/sympy repository.
The average message length is 0. The model doesn't spend any tokens on reasoning at all. It works like a bare-bones code generator: if it receives an error, it immediately issues a new command. This leads to a loop in problem sympy-14248 (209 steps).
This model consistently has an average message length of over 1,300–1,400 characters, which is the longest of all the models analyzed. Despite the enormous argument lengths, Gemini produces fewer tool errors (tool_errors) on average than its competitors. For example, in the challenging problem 14248, where all models have argument lengths of 150+, Gemini produces only 3 errors compared to 15 for Codex and 18 for Claude 4.6.
A counter-intuitive trend is observed: the newer Claude 4.6 Opus increased its trajectory to 135 steps compared to 98.7 in version 4.5. However, it is interesting that the percentage of errors in the new model has increased.
This model writes short explanations (~9–10 characters), almost like Codex, but manages to avoid getting too bogged down. However, her error rate (more graphs/steps_vs_error.png) on the graph is quite high.
Conclusion: The amount of reasoning does not always directly correlate with effectiveness, and newer versions of models (Claude 4.6) may exhibit redundancy compared to their predecessors.
