GRPOTrainer tool discovery evaluates every property on an environment

[Rome-1]

Reproduction

GRPOTrainer discovers an environment's tools with inspect.getmembers(env, predicate=inspect.ismethod), at init and again for every batch (trl/trainer/grpo_trainer.py, and the same lookup in trl/experimental/async_grpo/async_rollout_worker.py).
inspect.getmembers calls getattr on every attribute before it applies the predicate, so every property on the environment runs during tool discovery.

import inspect


class Env:
    def reset(self, **kwargs) -> str:
        return "ready"

    def move(self, x: str) -> str:
        """Make a move.

        Args:
            x: The move.
        """
        return x

    @property
    def reward(self) -> float:
        raise RuntimeError("reward was evaluated during tool discovery")


env = Env()
# The expression GRPOTrainer uses to discover an environment's tools:
tools = [m for name, m in inspect.getmembers(env, predicate=inspect.ismethod)]

Traceback (most recent call last):
  File "mre.py", line 23, in <module>
    tools = [m for name, m in inspect.getmembers(env, predicate=inspect.ismethod)]
  ...
  File "mre.py", line 18, in reward
    raise RuntimeError("reward was evaluated during tool discovery")
RuntimeError: reward was evaluated during tool discovery

In the trainer this matters for trl/experimental/harbor/_env.py, whose HarborEnv.reward property runs the Harbor task's verifier.
A pooled instance that was reset in an earlier batch and whose reward was never read has its verifier run by tool discovery at the start of the next step.
With a custom reward_funcs that does not read environments, the second training step fails before generating; in our run the verifier raised because the task's verifier environment names OPENAI_API_KEY and the key was not set, since nothing was supposed to grade at that point.
With the key set, the run would not fail: it would grade every pooled environment's previous rollout on every step, pay for any LLM-judge calls, and discard the result at reset().

The full trainer traceback and a two-step trainer test that reproduces it are in #7360.

Proposed fix

List the tools on the class and bind them by name, so a property is seen as a descriptor and never evaluated:

for member_name, _ in inspect.getmembers(type(instance), predicate=inspect.isfunction):
    ...
    methods.append(getattr(instance, member_name))

#7360 implements this at all four call sites with tests, and lists the four corner cases of the tool contract that move (public staticmethod, classmethod, a bound method assigned on the instance, functools.partialmethod), none of which any environment in the repository uses.
inspect.getmembers_static would preserve the contract more closely but needs Python 3.11, and CI still runs 3.10.

System Info

• trl 1.13.0, transformers 5.17.0, vllm 0.28.0, one L40S, GRPOTrainer with use_vllm=True, vllm_mode="colocate", and a custom reward_funcs.
• The lookup is unchanged on main as of 42757b5.
• The reproduction above is plain inspect and runs on any Python version.

This report and #7360 were written with an AI coding agent; the reproduction was run as shown.
#7360 is marked AI-generated, and I understand that means it may not be reviewed; filing the issue so the bug is on record either way.

[avinash-git-24]

Hey team, inspect.getmembers(environment, predicate=inspect.ismethod) in grpo_trainer.py and async_rollout_worker.py evaluates every @property on the environment. I can switch that to check inspect.getattr_static first if you'd like to assign me!