Repository navigation
fix(pilot): retry opening the VLM stream once - #302
Merged
Merged
Conversation
#280 dropped the single retry when opening the VLM stream timed out or failed, so a turn now ends after the first 30 s wait. Three Webots scenarios (explore_smoke, scene_object_fixture, object_navigation) give the model 35-45 s before the first plan to leave time for exploration and perception; they relied on that retry and have failed on dev since. Opening the stream dispatches no RTDL node, so retrying it once cannot repeat an action. A stream that stalls after it opened still ends the turn as before. The idle timeout stays 30 s.
github-actions
Bot
added
type:bug
enkerewpo
had a problem deploying
to
gpu-testing
October 27, 2026 06:27 — with 
GitHub Actions
Failure
GitHub Actions
Failure
enkerewpo
had a problem deploying
to
gpu-testing
October 27, 2026 06:44 — with
GitHub Actions
Failure
GitHub Actions
Failure
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Since #280 (RTDL Streaming Execution) the Webots integration suite on dev fails the same three scenarios on both scene backends:
cap/explore_smoke,cap/scene_object_fixtureandflow/object_navigation. In each,rbnx askends withpilot stream error: deadline has elapsed; Pilot sends the round's prompt, gets no response within the 30 s VLM idle timeout and ends the turn, and the fake VLM later hitsBrokenPipeErrorwriting its reply.These scenarios deliberately hold the first plan for 35-45 s so exploration and perception have time to run. Before #280, Pilot retried opening the VLM stream once after the 30 s timeout, and the retry received the reply a few seconds later. #280 replaced that with
.await??, so the first timeout now ends the turn.Fix
Restore one bounded retry when opening the VLM stream fails or times out. Opening the stream dispatches no RTDL node, so the retry cannot repeat an action. A stream that stalls after it opened still ends the turn and cancels its plan as before, and the idle timeout stays 30 s.
Testing
scripts/test_execute_stream.pygains three cases: an opening delayed past the timeout is retried once with the same prompt and finishes; a model that never answers is requested twice and the turn fails; a stream that stalls after dispatching an action is requested once and the action runs once. The delayed case fails on the unfixed binary.bb66fabf, time out at38ead76b, and pass with this fix.cargo fmt --check,cargo clippy -p robonix-pilot --all-targets -D warnings,cargo test -p robonix-pilot(111 passed),testing/simulate_ci.py(16 scenarios), testing unit tests (24 passed).dev-ci-testbefore this lands on dev.