Skip to content

[CI][Tests] Stop leaking lazy vllm.envs values between tests - #50

Merged
Leonccaa merged 1 commit into
mainfrom
f/0.1.5-test-isolation-flash-next
Oct 28, 2026
Merged

Leonccaa merged 1 commit into
mainfrom
f/0.1.5-test-isolation-flash-next

Conversation

@Leonccaa

Copy link
Copy Markdown
Owner

Change

tests/config/test_sm70_acceleration.py::test_flash_next_batch_memory_and_explicit_off passed on its own but failed when it ran after tests/v1/worker/test_gdn_spec_state_replay.py or tests/v1/worker/test_gdn_replay_review_cpu.py in the same pytest process (for example pytest tests/v1/worker/test_gdn_spec_state_replay.py tests/v1/worker/test_gdn_replay_review_cpu.py tests/config/test_gdn_replay_policy.py tests/config/test_sm70_acceleration.py → 1 failed on main).

Cause: those two files used monkeypatch.setattr(envs, NAME, value) on names that vllm.envs resolves lazily through its module __getattr__. pytest records the resolved value as the old one, and its undo writes that value back as a concrete module attribute where none existed. The attribute then shadows the environment for every later test, so the acceleration test's setenv("VLLM_SM70_FUSED_SIGMOID_MIXED_QKV", "0") had no effect and that control kept reason None instead of user_override.

Fix, tests only:

  • The two leaking files now patch the environment (setenv, after dropping any existing shadow), which pytest restores exactly. All the names involved are plain boolean environment getters read at call time.
  • As a guard, the acceleration test fixture drops concrete shadows of every declared env name, not only the DFlash2 verifier ones.
  • The assertion now names the offending control when it fails; its meaning is unchanged.

No runtime change; no effect on the support matrix.

Validation

CPU, in the ReVolta build image, each case a fresh pytest process:

  • Before: the failing test alone passes; after either leaking file it fails; the four-file combination gives 1 failed, 135 passed.
  • After: the minimal failing orders pass, each of the four files passes alone (50 / 24 / 19 / 43), and the four-file combination passes (136).
  • Each half was also checked on its own. With only the leak fix, the combination passes and a leak probe plugin (diffing vars(vllm.envs) and os.environ around every test) finds no attribute left behind. With only the fixture guard, the combination passes although the leak is still there.
  • pre-commit passes on the changed files.

Attribution

Root cause and fix are the work of a local model: Qwen3.8-Flash-Next (Uncensored AWQ g32) served by ReVolta 0.1.4 on our 4 × V100 server, working through the pi coding agent. It bisected the order, wrote a leak probe, traced the mechanism to pytest's undo of attributes resolved through a module __getattr__, fixed the tests and verified each half of the fix separately. Claude Code found the failing order while running the 0.1.5 changes together, reviewed the fix and opened this PR. Commits are signed off under the DCO.

🤖 Generated with Claude Code

test_flash_next_batch_memory_and_explicit_off passed alone but failed after tests/v1/worker/test_gdn_spec_state_replay.py or test_gdn_replay_review_cpu.py in the same process. Those tests used monkeypatch.setattr(envs, NAME, value) on names that vllm.envs resolves lazily through module __getattr__: pytest records the resolved value as the old one and its undo writes it back as a concrete module attribute, which then shadows the environment for every later test. The acceleration test's setenv(NAME, "0") had no effect and VLLM_SM70_FUSED_SIGMOID_MIXED_QKV kept reason None.

The two files now patch the environment (setenv, after dropping any existing shadow), which pytest restores exactly. As a guard, the acceleration test fixture drops concrete shadows of every declared env name, not only the DFlash2 verifier ones, and the assertion names the offending control when it fails. Tests only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
@Leonccaa
Leonccaa merged commit b7a6fe8 into main Oct 28, 2026
4 checks passed
@Leonccaa
Leonccaa deleted the f/0.1.5-test-isolation-flash-next branch October 28, 2026 14:30
@Leonccaa Leonccaa mentioned this pull request Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant