Skip to content

open_source_prompt.md §3.1.6 reads an unrunnable toolchain as a failing test suite #2515

Closed

Description

@argszero

What happens

emrg/server/open_source_prompt.md §3.1.6 (Test) tells a cycle to run the target project's suite with a bare command — npm test, pytest tests/ -v, cargo test — and the paragraph after it reads any non-zero outcome as the state of that project's tests: "Failing → fix, re-test, until green; a failure you cannot fix does not get hidden — say so in the PR description."

That is false whenever the toolchain is not on the subprocess's PATH. A runtime installed through a version manager (asdf, nvm, pyenv) or exported by a shell startup file is resolved for an interactive shell and frequently not for the process that runs the suite. The command then exits 127 (command not found) or fails to exec, which is not a test result at all — no test was collected, no test ran.

Why it matters

Two opposite misreports come out of one unreadable signal, and both are expensive:

  • A false red about the project. A cycle reports "the project's suite fails" in a closing summary or a PR description when the suite never started. The next cycle, or a maintainer, then acts on a defect that does not exist.
  • A false green. A cycle that checks off "existing tests pass" from an exit status it did not inspect claims a run that never happened.

The template already treats "could not measure" and "measured clean" as different things everywhere else — gh pr checks reporting zero runs is explicitly "a dropped push event, never a pass". §3.1.6 is the one place that conflates them.

What the fix must satisfy

  1. §3.1.6 states that a tool a version manager or a shell startup file exports may be invisible to the subprocess that runs the suite, and says how to resolve it: locate the interpreter, then put its directory on PATH in the same block (export PATH="<dir>:$PATH") or call it by absolute path.
  2. It distinguishes the two outcomes: command not found / exec failure is a recorded skip — the command and the exact error are named, the suite is stated not to have run, and no verdict about the tree is offered — while a suite that ran and failed is fixed or named in the PR description, as today.
  3. The rule lives in §3.1.6, the section that owns running the suite, and does not appear twice in the file.
  4. A test pins the rule in tests/test_scheduler.py, in the shape the neighbouring template tests use: render the template with Jinja2, assert the distinctive sentences are present, and assert the rule sits inside §3.1.6 rather than at the end of the file.

Scope

Prompt template only. No runtime behaviour, no version declaration and no packaging change. The template ships inside the daemon, so the change reaches installed hosts with the next release rather than with this merge.

Activity

  1. argszero commented on Oct 23, 2026

    @argszero
    OwnerAuthor

    Handled by #2516

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions