Repository navigation
Add openarm-mjlab-eval: measure a policy's success rate - #10
Merged
Merged
Conversation
Contributor
|
@Manas-arumalla Before adding eval or a new feature, I am planning to refactor the codebase a bit and update the README for more usability. Is it ok? |
Contributor
Author
Contributor
Author
|
@k1000dai If any help is needed in making the README please do let me know |
Contributor
|
@Manas-arumalla Thanks! I will let you know after I create the PR for the README. Please review and tell me your thoughts! |
Every task here reports a success rate and there was no shared way to reproduce one. train and play exist; nothing answers what fraction of episodes actually succeed, so each number had to be taken on trust or re-derived with a throwaway script. This adds one command that works for any registered task with no task-specific code, and encodes four things that are easy to get wrong when writing that script by hand: Success is read from a named termination term rather than `done & !time_out`. On any task with a failure termination -- a dropped cube, a puck knocked off the table -- the coarser test counts those failures as successes. Each environment is scored once. After it terminates it stops updating, so an environment that succeeds, resets and succeeds again inside one rollout cannot be counted twice. The episode budget is derived from the config as episode_length_s / (decimation * timestep). A hardcoded step count silently truncates or over-runs whenever a task's episode length changes. Observation corruption is disabled, since training noise is a regularizer and leaving it on measures the policy plus the noise. Choosing the success term is the one step whose failure is silent: a wrong choice prints a plausible number that means something else. So resolve_success_term prefers an explicit --success-term, then a term named success, then the sole remaining term once clock and failure terms are set aside, and raises rather than guessing when that is ambiguous. Seven tests pin those rules, including that puck_fell is never inferred as success. For the same reason the returned rates are nested under "terms" rather than flattened: a task may have a termination term literally named success, and a flat layout would collide with it. Verified on three tasks. Reach reports 1.000 with the success term inferred correctly. Door reports 1.000 for the policy trained after the handle geometry fix and 0.000 for the one trained before it, which is the mismatch this is meant to surface.
k1000dai
force-pushed
the
feat/eval-harness
branch
from
October 2, 2026 07:58
eab9239 to
f4550f4
Compare
Contributor
|
@Manas-arumalla works well on my side. |
OpenArm-Lift's block_fell is a failure termination, so without --success-term both it and lifted_target were candidates and the tool refused to guess. Add it to the failure terms, and test that every registered task resolves to exactly one success term on its own.
Contributor
Author
|
@k1000dai did the changes please check i forgot to let you know |
Contributor
|
(Thank you for letting me know) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every task in this repo reports a success rate, and there's currently no shared way to reproduce one. train and play exist, but nothing answers what fraction of episodes actually succeed, so each number has to be taken on trust or re-derived with a script. This adds one command that does it for any registered task, with no task-specific code.
It reports the fraction of episodes reaching the task's success termination, alongside the rate of every other termination term and the episode-length distribution. Defaults to the latest checkpoint and 256 episodes.
Things it gets right that are easy to get wrong by hand:
Success is read from a named termination term, not done & !time_out. On any task with a failure termination a dropped cube, a puck knocked off the table the coarser test counts those failures as successes.
Each environment is scored once. After it terminates it stops updating, so an environment that succeeds, resets and succeeds again inside one rollout can't be counted twice.
The episode budget comes from the config as episode_length_s / (decimation * timestep), rather than a hardcoded step count that silently truncates whenever a task's episode length changes.
Choosing which termination term means success is the one step whose failure is silent a wrong choice prints a plausible number that means something else. So resolve_success_term prefers an explicit --success-term, then a term named success, then the sole remaining term once clock and failure terms are set aside, and raises rather than guessing when that's ambiguous.