[KouweiLee]

Enhance the VLM verifier service to capture and verify both before and after images during capability execution. The service now requires a "before" image to be captured before executing the action, and an "after" image to validate the result. Update the verification logic to handle these phases, ensuring that visual evidence is clearly defined and processed. Adjust tests to validate the new functionality, including checks for image capture and verification outcomes.

• Update README to reflect changes in service behavior.
• Modify service logic to store and evaluate images based on execution phase.
• Enhance tests to cover new before/after verification scenarios.
• Ensure backward compatibility with existing verification rules.

[enkerewpo]

Thanks for improving the VLM verifier. Comparing the state before and after a call is a reasonable idea, but the current implementation has a design problem: it assumes that the two frames actually show the target and the effect, and nothing guarantees that.

• Both phases read the single camera_provider_id from the rule. The before frame is taken before the skill starts, and a skill such as pick first moves the base and the arm, so with a wrist camera the before frame usually shows something else.
• The after frame is not guaranteed to show the effect either: the arm may have retracted, the base may have moved away, or the gripper and the object occlude the contact at the decisive moment.
• The verdict is binary, so "not visible" is reported as either passed or failed. A false failure can make Pilot redo an action that actually succeeded; a false pass hides a real failure.
• The prompt changes make this worse: the negative cases (wrong object, occlusion, ambiguous evidence, nearby target without two identifiable jaws), the contract ID and target_args are no longer given to the model, and temperature: 0 was removed. VLM judges are already biased toward "success" when evidence is ambiguous (see FailBench below).

A simple example: a mobile manipulator with a wrist camera runs pick on a bottle on a table. When the before frame is taken, the robot has just arrived and the arm is folded, so the wrist camera sees the floor. During pick the arm reaches the table, grasps and lifts the bottle, then folds back to its carry pose. In the after frame the camera again faces the floor or the side of the robot. Neither frame shows the bottle in the gripper above the table, so the model has nothing to judge the grasp from.

Verifying the outcome of an executed action when the camera may not see it has been studied; some relevant work:

• FailBench (2026): benchmarks VLMs as success judges on 2,197 manipulation attempts from 14 sources. The best model reaches only 0.77 mean balanced accuracy, none exceeds 0.60 on contact-rich assembly, and models show a systematic bias toward predicting success under ambiguous evidence.
• SuccessVQA (Du et al., 2023): success detection as VQA over short video clips (8 frames at 1 FPS) rather than a single image. Moving the camera from the front to the back drops a dedicated success detector by 10–50 points; the fine-tuned VLM drops by less than 10.
• Guardian (2025): the closest to this PR, judging from the subtask start and end frames. Keeping the start and end images separate beats concatenating them (0.72 vs 0.69 single-view), and adding views encoded separately raises execution accuracy from 0.74 (all views stitched into one grid) to 0.83.
• Code-as-Monitor (CVPR 2025): monitors constraints continuously during execution from front and top RGB-D views. Using only the front view drops success from 65% to 40%, due to occlusion and visual-programming errors.
• Gemini Robotics 1.5 (2025): treats multi-view success detection as a core capability and gives tasks where no single camera has enough information; success is judged continuously during execution (evaluated at 5 Hz), not once at the end.
• VAP-TAMP (2026): checks PDDL preconditions before and effects after each action with a VLM, asking 5 paraphrases and taking a majority vote. It also asks the VLM whether the current view is sufficient; if the answers disagree or the view is insufficient, the VLM suggests a new viewpoint (left, right, closer, above), up to a viewpoint budget.
• REFLECT (CoRL 2023): selects keyframes when the scene graph changes, an audio event starts or ends, or a subgoal ends, and accumulates a 3D scene across frames so objects that are temporarily out of view are kept rather than dropped.
• ProTracer (2026): picks keyframes at change points of gripper and end-effector motion; removing this proprioceptive signal drops failure-onset localization (acc@0.5s) from 71.18% to 30.79%.
• A Physical Agentic Loop for Language-Guided Grasping (2026): a Watchdog layer decides grasp outcome from gripper effort and closure (SUCCESS, EMPTY, SLIP, WEAK, STALL, TIMEOUT) over a short settle window plus a micro-lift; an optional vision verifier only checks that the grasped object is the intended one.
• Embodied-R1.5 (2026): the Corrector reads a FIFO buffer of frames sampled at a fixed rate, runs asynchronously, and answers SUCCESS, PROCESS (still executing, keep waiting) or FAIL.
• Chong et al., Robust Task Planning via Failure Detection Using Scene Graph From Multi-View Images (RA-L 2026): merges per-view scene graphs into one with a GNN and compares it with the expected scene graph predicted by the LLM; the difference is the failure cause fed back for replanning.

Common to these: more than one view, evidence taken at the right moment (during execution or at gripper/subgoal events) rather than one frame after the call, non-visual signals for grasping, and an explicit "cannot tell yet" outcome instead of forcing pass/fail.

Changes that fit this PR without much work:

1. Let the model say it cannot tell. Ask it first whether the target and the effect are visible in the given frames (as VAP-TAMP does); if not, fail the verify call with an error such as "target not visible". Executor already maps a verifier error to "result verification unavailable" and keeps the original result, so no contract change is needed.
2. Accept a list of cameras (camera_provider_ids, e.g. head and wrist) and send each view as a separate labelled image, not stitched (Guardian).
3. Take the before/after frames from the call's time window instead of a separate blocking before round trip: keep a short timestamped frame buffer per subscribed camera in CameraPool, have Executor pass the call's start and end times in the envelope, and pick the frames nearest to them. This also removes the per-call _before_images cache, which today has no bound or expiry and leaks when the target fails or the plan is canceled; the new tests for duplicates, bounds and expiry currently fail and are not run by CI.
4. Keep the stricter prompt from dev (negative cases, contract ID and target_args, temperature: 0), add "if the target is not visible in any frame, report that you cannot tell", and state that the before frame may show a different view and is context only. Prompt loosening, if wanted, should come as a separate PR with results on real grasps.

Gripper signals for grasp success and keyframes at gripper events are worth a follow-up issue, but they need the arm primitives to expose those signals, so they are out of scope here.

[KouweiLee]

Thanks for improving the VLM verifier. Comparing the state before and after a call is a reasonable idea, but the current implementation has a design problem: it assumes that the two frames actually show the target and the effect, and nothing guarantees that.

• Both phases read the single camera_provider_id from the rule. The before frame is taken before the skill starts, and a skill such as pick first moves the base and the arm, so with a wrist camera the before frame usually shows something else.
• The after frame is not guaranteed to show the effect either: the arm may have retracted, the base may have moved away, or the gripper and the object occlude the contact at the decisive moment.
• The verdict is binary, so "not visible" is reported as either passed or failed. A false failure can make Pilot redo an action that actually succeeded; a false pass hides a real failure.
• The prompt changes make this worse: the negative cases (wrong object, occlusion, ambiguous evidence, nearby target without two identifiable jaws), the contract ID and target_args are no longer given to the model, and temperature: 0 was removed. VLM judges are already biased toward "success" when evidence is ambiguous (see FailBench below).

A simple example: a mobile manipulator with a wrist camera runs pick on a bottle on a table. When the before frame is taken, the robot has just arrived and the arm is folded, so the wrist camera sees the floor. During pick the arm reaches the table, grasps and lifts the bottle, then folds back to its carry pose. In the after frame the camera again faces the floor or the side of the robot. Neither frame shows the bottle in the gripper above the table, so the model has nothing to judge the grasp from.

Verifying the outcome of an executed action when the camera may not see it has been studied; some relevant work:

• FailBench (2026): benchmarks VLMs as success judges on 2,197 manipulation attempts from 14 sources. The best model reaches only 0.77 mean balanced accuracy, none exceeds 0.60 on contact-rich assembly, and models show a systematic bias toward predicting success under ambiguous evidence.
• SuccessVQA (Du et al., 2023): success detection as VQA over short video clips (8 frames at 1 FPS) rather than a single image. Moving the camera from the front to the back drops a dedicated success detector by 10–50 points; the fine-tuned VLM drops by less than 10.
• Guardian (2025): the closest to this PR, judging from the subtask start and end frames. Keeping the start and end images separate beats concatenating them (0.72 vs 0.69 single-view), and adding views encoded separately raises execution accuracy from 0.74 (all views stitched into one grid) to 0.83.
• Code-as-Monitor (CVPR 2025): monitors constraints continuously during execution from front and top RGB-D views. Using only the front view drops success from 65% to 40%, due to occlusion and visual-programming errors.
• Gemini Robotics 1.5 (2025): treats multi-view success detection as a core capability and gives tasks where no single camera has enough information; success is judged continuously during execution (evaluated at 5 Hz), not once at the end.
• VAP-TAMP (2026): checks PDDL preconditions before and effects after each action with a VLM, asking 5 paraphrases and taking a majority vote. It also asks the VLM whether the current view is sufficient; if the answers disagree or the view is insufficient, the VLM suggests a new viewpoint (left, right, closer, above), up to a viewpoint budget.
• REFLECT (CoRL 2023): selects keyframes when the scene graph changes, an audio event starts or ends, or a subgoal ends, and accumulates a 3D scene across frames so objects that are temporarily out of view are kept rather than dropped.
• ProTracer (2026): picks keyframes at change points of gripper and end-effector motion; removing this proprioceptive signal drops failure-onset localization (acc@0.5s) from 71.18% to 30.79%.
• A Physical Agentic Loop for Language-Guided Grasping (2026): a Watchdog layer decides grasp outcome from gripper effort and closure (SUCCESS, EMPTY, SLIP, WEAK, STALL, TIMEOUT) over a short settle window plus a micro-lift; an optional vision verifier only checks that the grasped object is the intended one.
• Embodied-R1.5 (2026): the Corrector reads a FIFO buffer of frames sampled at a fixed rate, runs asynchronously, and answers SUCCESS, PROCESS (still executing, keep waiting) or FAIL.
• Chong et al., Robust Task Planning via Failure Detection Using Scene Graph From Multi-View Images (RA-L 2026): merges per-view scene graphs into one with a GNN and compares it with the expected scene graph predicted by the LLM; the difference is the failure cause fed back for replanning.

Common to these: more than one view, evidence taken at the right moment (during execution or at gripper/subgoal events) rather than one frame after the call, non-visual signals for grasping, and an explicit "cannot tell yet" outcome instead of forcing pass/fail.

Changes that fit this PR without much work:

1. Let the model say it cannot tell. Ask it first whether the target and the effect are visible in the given frames (as VAP-TAMP does); if not, fail the verify call with an error such as "target not visible". Executor already maps a verifier error to "result verification unavailable" and keeps the original result, so no contract change is needed.
2. Accept a list of cameras (camera_provider_ids, e.g. head and wrist) and send each view as a separate labelled image, not stitched (Guardian).
3. Take the before/after frames from the call's time window instead of a separate blocking before round trip: keep a short timestamped frame buffer per subscribed camera in CameraPool, have Executor pass the call's start and end times in the envelope, and pick the frames nearest to them. This also removes the per-call _before_images cache, which today has no bound or expiry and leaks when the target fails or the plan is canceled; the new tests for duplicates, bounds and expiry currently fail and are not run by CI.
4. Keep the stricter prompt from dev (negative cases, contract ID and target_args, temperature: 0), add "if the target is not visible in any frame, report that you cannot tell", and state that the before frame may show a different view and is context only. Prompt loosening, if wanted, should come as a separate PR with results on real grasps.

Gripper signals for grasp success and keyframes at gripper events are worth a follow-up issue, but they need the arm primitives to expose those signals, so they are out of scope here.

Thanks for the detailed review. I have made the following changes:

1. Added an explicit observable judgment. Insufficient evidence is reported as verification unavailable through the existing error path, without changing the Verify contract.
2. Added camera_provider_ids support. Each camera’s BEFORE and AFTER images are sent separately with camera and phase labels. Existing single-camera configurations remain supported.
3. Restored the stricter prompt, target_contract_id, target_args, and temperature: 0. BEFORE images are explicitly treated as context only, and action outputs are not considered visual proof of success.
   Regarding timestamped frame buffering, I currently prefer retaining the latest-frame approach. Assuming cameras publish continuously and the pipeline operates normally, I expect the latest available frame to be reasonably close to the verification trigger. Therefore, I would keep periodic frame sampling and timestamp-window retrieval outside this PR for now. This does not guarantee strict temporal alignment, and cleanup after failed or cancelled calls remains a separate concern.

In local BEHAVIOR tests with action assistance, the multi-view configuration completed all six actions and passed the official BDDL goal check. An independent no-op placement was correctly rejected, and subsequent actions were blocked. For the same placement sample, head-camera-only verification reported insufficient evidence, while adding two table views produced an observable, successful verdict. Cameras were configured per action to avoid confusion from unrelated views.

[enkerewpo]

Ran head 91cbcd0b locally: the verifier unit tests, the executor tests, the smoke script, and the real service.verify() path.

What needs fixing before merge (details inline):

• the pending before-image cache has no bound or expiry, and the tests written for it fail;
• a duplicate before for the same call_id is accepted and overwrites the stored frames;
• smoke_mcp.py fails in teardown before it reaches the overlap checks;
• none of the verifier Python tests run in CI, which is how these failures got in. Please add python3 -m unittest discover -s tests -p test_verifier.py (in services/verifiers/vlm_verifier) to ci.yml.

Current state of that command: 26 pass, 2 fail, 1 error. cargo test -p robonix-executor passes (55/55).

[enkerewpo]

After the 10 synchronous checks (both camera sets) pass, the next entry.CAMERAS.close() here fails, so the overlap checks never run. The pool still holds channels opened during the previous check(). That call has already terminated its Atlas and left the ROBONIX_ATLAS patch, so directory.close() sends the disconnect to the default endpoint:

File "tests/smoke_mcp.py", line 209, in main
File "vlm_verifier/camera.py", line 202, in close
File "vlm_verifier/directory.py", line 42, in close
grpc._channel._InactiveRpcError: status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; ... ipv4:127.0.0.1:50051: ... Connection refused"

When I ignored only this error, all 20 checks (1/2 cameras × sync/overlap × pass/reject/unobservable/malformed/missing camera) passed, so the logic looks fine and this is a cleanup-order bug. Close the camera pool inside check() before Atlas is torn down, or make close() tolerate an unreachable directory.

[KouweiLee]

Ran head 91cbcd0b locally: the verifier unit tests, the executor tests, the smoke script, and the real service.verify() path.

What needs fixing before merge (details inline):

• the pending before-image cache has no bound or expiry, and the tests written for it fail;
• a duplicate before for the same call_id is accepted and overwrites the stored frames;
• smoke_mcp.py fails in teardown before it reaches the overlap checks;
• none of the verifier Python tests run in CI, which is how these failures got in. Please add python3 -m unittest discover -s tests -p test_verifier.py (in services/verifiers/vlm_verifier) to ci.yml.

Current state of that command: 26 pass, 2 fail, 1 error. cargo test -p robonix-executor passes (55/55).

Sorry，I thought some changes are needless, so I delete some codes and 3 tests failed. I've changed back and fixed them. I think it is needless to include vlm verifier test in ci, so I didn't do.

[enkerewpo]

LGTM！