Skip to content

Provenance live-probe guard checks no R packages while its consumer requires four, so the testset errors on any bare checkout #43

Closed

Description

@hyperpolymath

Measured

The full Julia suite errors locally with:

live probes of what is actually installed   |  0 pass  1 error   3.8s
provenance: could not prove 'r': package 'dada2' is not installed in the R library in use

Zero passes before the error means it threw before the first @test.

The cause: the guard asks a different question than its consumer

test/unit/test_provenance.jl:439-448

r_ready = try
    PV.probe_r(; packages = String[], timeout = 30)   # <- probes NO packages
    true
catch err
    @info "Provenance: skipping live R probe" reason=sprint(showerror, err)
    false
end

if r_ready
    record = PV.probe_r(; timeout = 120)              # <- probes FOUR packages

src/core/provenance.jl:292

const R_PACKAGES = ["dada2", "Biostrings", "ShortRead", "vegan"]

The guard asks "can R be reached at all?". The consumer asks "can R be reached
and are dada2, Biostrings, ShortRead and vegan all installed?"
. On any checkout
where R exists but the packages do not — which is every bare checkout, and this
development box — the guard returns true and the consumer throws outside any
try
, erroring the testset.

The @info "skipping live R probe" branch therefore cannot fire for the case it was
written for. The file's own header states the intent it fails to meet:

Nothing here shells out to a tool or evaluates R unless the tool or the package is
genuinely present, so the suite runs on a bare checkout.

Measured context

Acceptance criteria

  1. The guard probes exactly what the consumer will probe — pass the same package
    list to both, or probe once and reuse the record. A test asserting the two call
    sites agree is preferable to a comment saying they should.
  2. On a checkout where R is present but R_PACKAGES are not, the testset skips
    loudly and passes
    — it must not error.
  3. A skip must name which packages were missing, not merely that a probe failed.
    Per the standing rule that a skip is not a pass, the skip must be visible in the
    run summary rather than only in @info output that nobody reads.
  4. Add the negative control: a case where R is reachable and the packages are absent
    must be distinguishable in the test output from a case where R is unreachable.

🤖 Generated with Claude Code

https://claude.ai/code/session_01X3hgXxWm6umMgZkjYyHnnm

Activity

  1. hyperpolymath commented on Sep 22, 2026

    @hyperpolymath
    OwnerAuthor

    Fix is up in #47.

    Reproduced the reported error before fixing it. This box has R 4.5.0 and none of dada2, Biostrings, ShortRead, vegan — the world the issue describes — so the two sides of the split could be run against the real module:

    • the guard's old call (packages = String[]) returns a record, i.e. reports ready;
    • the consumer's call throws, outside any try — provenance: could not prove 'r': R is reachable but these required packages are not installed: dada2, Biostrings, ShortRead, vegan.

    After the fix there is one probe, r_probe_status(), and its record is what the suite asserts on. Against the same box: status :packages_missing, missing == R_PACKAGES (all four named), the testset titled r — not probed (packages_missing): ... dada2, Biostrings, ShortRead, vegan and recorded Broken rather than passed — 20 pass / 1 broken / 0 fail, with the live probe path exercised rather than skipped.

    Two things worth flagging beyond the four criteria:

    1. ProbeFailure now carries the absent package names as a field, so classification is a field read rather than a text match on reason; probe_r collects every absent package instead of dying on the first, so a checkout missing four reports four rather than one.
    2. On CI the skip block is a hard assertion instead. ci.yml installs R and restores all 79 packages from renv.lock, so a skip there is a provisioning regression rather than a bare checkout — the same reading the tool loop above already takes, and the failure names the missing packages through the testset title.

    The source-check testset in the PR earned its place immediately: it caught this PR's own prose quoting the forbidden empty-list literal in a comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions