Local conformance test data harvested from reference implementations

[andrew]

The upstream package-url/vers-spec conformance suite is thin and uneven: 491 npm native-parse cases but 1 npm containment case, 977 maven comparison cases but 0 maven native-parse cases, 1 gem case total, and zero data for cargo, composer, cpan, dart/pub, deb, golang, hackage, hex, rpm, semver. Passing it doesn't prove much beyond "can parse the URI format".

Rather than wait on upstream, add a testdata/local/tests/ directory in the same JSON schema, run it through the existing conformance runner, and fill it by harvesting fixtures from each ecosystem's own reference implementation (node-semver, pypa/packaging, composer/semver, pub_semver, rubygems, dpkg, rpm, dtolnay/semver, maven-artifact). The harvested JSON is committed so go test needs nothing but Go.

Infrastructure

• Change conformance_test.go to filepath.Glob("testdata/*/tests/*_<type>_test.json") instead of enumerating filenames, so both testdata/vers-spec/tests/ and testdata/local/tests/ are picked up and new upstream files run automatically after a submodule bump
• Wire up vers_canonical_parse_test.json (currently exists upstream but isn't run)
• Add testdata/local/skip.json mapping {filename: [{description, reason}]} for known intentional divergences, checked before each subtest with t.Skip(reason)

Seed local data from existing tests

• testdata/local/tests/composer_range_from_native_test.json from composer_pub_test.go tables
• testdata/local/tests/composer_version_cmp_test.json
• testdata/local/tests/pub_range_from_native_test.json
• testdata/local/tests/pub_version_cmp_test.json
• Trim composer_pub_test.go down to error-path cases once the JSON covers the happy paths

Harvester

• tools/harvest/main.go (invoked via go run ./tools/harvest, kept out of the library build) that shallow-clones reference repos, extracts assertion tables, and writes testdata/local/tests/*.json with source repo + commit SHA recorded in a top-level $comment field
• Makefile target harvest that runs it

Harvest: comparison tests (upstream has zero for all of these)

• npm from npm/node-semver/test/fixtures/{comparisons,equality}.js
• pypi from pypa/packaging/tests/test_version.py
• gem from rubygems/rubygems/test/rubygems/test_gem_version.rb
• composer from composer/semver/tests/ComparatorTest.php
• pub from dart-lang/tools/pkgs/pub_semver/test/version_test.dart
• cargo/semver from dtolnay/semver/tests/
• deb from dpkg/lib/dpkg/t/t-version.c
• rpm from rpm-software-management/rpm/tests/rpmvercmp.at
• go from golang.org/x/mod/semver test data

Harvest: native-range tests

• npm containment from node-semver/test/fixtures/{range-include,range-exclude}.js (upstream has 1 case)
• pypi from pypa/packaging/tests/test_specifiers.py (upstream has 3 cases)
• gem from rubygems/test_gem_requirement.rb (upstream has 1 case)
• composer from composer/semver/tests/VersionParserTest.php data providers
• pub from pub_semver/test/version_range_test.dart and version_constraint_test.dart
• cargo from dtolnay/semver/tests/test_version_req.rs
• maven from maven-artifact VersionRangeTest.java (upstream has 0 native cases)

Harvest: generated containment tests

• For each scheme, cross the harvested ranges against the harvested versions and shell out to the reference binary (npx semver, python -c 'import packaging.specifiers...', php -r, dart run, ruby -e 'Gem::Requirement...') to record the expected answer; skip a scheme with a warning if its runtime isn't on PATH

Follow-up

• Input length / constraint-count bounds on Parse and ParseNative (the Ruby vers gem caps at 2048 chars / 64 constraints; this library currently has no bound)
• Once testdata/local/ has meaningful coverage, open an upstream PR to package-url/vers-spec with the generated files and a link to the harvester; don't block on it landing

[andrew]

Two more things to fold in after looking at other implementations.

aboutcode-org/univers (Python, same maintainers as vers-spec) has already done a lot of the harvesting: tests/test_rubygems_gem_version.py, test_debian_version.py, test_rpm_vercmp.py, test_maven_version.py, test_gentoo_pkgcore.py, test_pacman_vercmp.py, test_bundler_version_ranges_spec.py, test_conan_*.py are all ports of upstream reference tests, each with an .ABOUT file recording source repo, commit SHA, and SPDX licence. Their tests/data/schema/*.json is where the vers-spec conformance data originated. The tests are Python code rather than JSON so they can't be consumed directly, but the assertion tables inside are regex-extractable and the attribution model (.ABOUT files) is worth copying. Their Composer support is a # TODO stub, and they have no Pub/Dart at all, so this repo is ahead on both. They also ship tests/data/npm_advisory.json (5311 lines of real GHSA advisory ranges) which is the production input shape.

vers-test schema 0.2 landed 2026-08-04. The submodule here is pinned at 470128d (pre-0.2). Bumping it will break TestConformance_RoundTrip: the file is now pypi_range_validate_test.json with test_type: "validate". test_group values changed base/advanced → required/recommended (harmless, not read). The 0.2 schema also renames input_scheme→input_type and scheme→type, but upstream's own test files still use the old names, so the runner needs to accept both and local JSON should match what upstream files actually contain (input_scheme, scheme) rather than what the schema says.

Additional items:

• Bump testdata/vers-spec submodule and fix conformance_test.go for 0.2: rename TestConformance_RoundTrip → TestConformance_Validate, match test_type "validate" and "roundtrip", glob *_validate_test.json and *_roundtrip_test.json
• Make versionCmpInput and fromNativeInput unmarshal both input_scheme/input_type and scheme/type so the runner survives the next upstream rename
• Add .ABOUT files (or a single testdata/local/PROVENANCE.md) alongside harvested JSON recording source repo, commit, and licence, following the univers pattern
• Import aboutcode-org/univers/tests/data/npm_advisory.json as a smoke test: parse every range, assert no error, no assertion on output
• Harvest assertion tables from univers's ported Python tests as a secondary source for deb, rpm, gentoo, alpine, pacman, conan (cheaper than going back to the C sources)
• tools/diff-univers/: shell out to python -c 'import univers...' with the same inputs the conformance suite uses, report disagreements; run manually, not in CI