Skip to content

Validation discards output from runners that report on stderr (jest, unittest) #163

Description

@k21993

Description

Validation never sees the output of a test runner that reports on stderr, so every candidate from such a repo is dropped with "no fail-to-pass tests after validation" (pr_runtime_validate.py L301).

The pieces line up like this:

  • _build_stage_script echoes R2E_START_TEST_OUTPUT / R2E_END_TEST_OUTPUT to stdout and runs the test commands with no redirect (pr_runtime_validate.py L82-84).
  • ExecResult.truncated() returns stdout first, then \n--- stderr ---\n, then stderr (bootstrap/docker.py L36-44).
  • _slice_test_output cuts between the two markers (pr_runtime_validate.py L176-189), so anything the runner wrote to stderr sits after the end marker and is discarded.

Which runners this hits

I ran each one and captured the streams to separate files:

Runner Result lines on stdout on stderr
jest 30.5.2 (--verbose) 0 26
unittest / Django (-v) 0 every line
mocha 12.0.2 4 0
vitest 5.0.1 (--reporter=verbose) 2 0
pytest, go test, cargo test all 0

So jest repos have never produced tasks through pr_runtime, commit_runtime or cve_patches, no matter what the parser does. Worth noting jest_parser.py is exercised by unit tests only, which is why this stayed hidden.

(A caveat for anyone re-measuring: cmd 2>&1 >/dev/null does not isolate stderr under zsh, because MULTIOS duplicates the stream. Redirect to two files instead.)

Reproduction

Running the generated script shape in bash, with the runner's output fed through truncated() and _slice_test_output:

main:        sliced 0 chars, parsed 0 tests
with a redirect on the test block: sliced 252 chars, parsed 2 tests

Proposed fix

Fold stderr into stdout for the test block only:

echo R2E_START_TEST_OUTPUT
set +x
{ <test_cmds joined with && >; } 2>&1
set -x
echo R2E_END_TEST_OUTPUT

set +x matters: with set -uxo pipefail at the top, the shell's own trace also goes to stderr, and without it the trace would now land inside the sliced region.

The graded path needs no change, since build_eval_script already captures both streams into /logs/verifier/test_output.log.

I have a PR ready and will link it here. It is also a prerequisite for unittest and Django support, which I will send separately once this lands.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions