Skip to content

Run the release half of the suite, and keep it running - #143

Merged
nnewson merged 1 commit into
mainfrom
release-suite-build
Aug 27, 2026
Merged

Run the release half of the suite, and keep it running#143
nnewson merged 1 commit into
mainfrom
release-suite-build

Conversation

@nnewson

@nnewson nnewson commented Aug 27, 2026

Copy link
Copy Markdown
Owner

Rejection tests guarded by #ifdef NDEBUG — the release behaviour of every writer that asserts in Dev and returns false under NDEBUG — have never run. Every preset here builds Dev, so those bodies compiled to nothing on every machine and in every job that has ever checked them: the cases ran, passed, and asserted precisely nothing. Death tests are not the fix; they prove the Dev assertion fires and say nothing about the half that ships.

Tag those cases [release-contract] (18 of them) and add a Linux-only CI job that builds a genuine Release tree and runs that selection. Linux alone because the behaviour does not vary by platform, and the tagged cases alone because the whole Release suite would drag in the optimisation-sensitive physics goldens.

Two mechanisms keep the job honest, since a job that checks nothing looks exactly like a job that finds no fault. Catch2 exits non-zero when a spec matches nothing, so a renamed tag fails rather than passing quietly; and tests/release_contract.cpp holds a hidden ([.]) sentinel case inside the same selection that fails unless NDEBUG is defined, so a Dev build cannot pass by mistake. Hidden is what keeps it out of ctest and tests-full, which are default runs.

And the tag itself is now enforced: release_contract_guard fails on a conditional case that is untagged (no check at all, and silent), on a tagged case with no conditional, and if the sentinel or the last tag disappears. Without it the convention lived only in prose, which is the same gap one layer up. Nine mutations tested.

The vcpkg-release preset builds into build-release/ while pointing VCPKG_INSTALLED_DIR back at build/vcpkg_installed, so the job reuses the ports the Dev tree and the CI cache already hold. The job sources tools/ci/ci-stages.sh rather than restating its commands, and run-local-ci.sh gains the stage.

Rejection tests guarded by `#ifdef NDEBUG` — the release behaviour of
every writer that asserts in Dev and returns `false` under NDEBUG — have
never run. Every preset here builds Dev, so those bodies compiled to
nothing on every machine and in every job that has ever checked them:
the cases ran, passed, and asserted precisely nothing. Death tests are
not the fix; they prove the Dev assertion fires and say nothing about
the half that ships.

Tag those cases `[release-contract]` (18 of them) and add a Linux-only
CI job that builds a genuine Release tree and runs that selection. Linux
alone because the behaviour does not vary by platform, and the tagged
cases alone because the whole Release suite would drag in the
optimisation-sensitive physics goldens.

Two mechanisms keep the job honest, since a job that checks nothing
looks exactly like a job that finds no fault. Catch2 exits non-zero when
a spec matches nothing, so a renamed tag fails rather than passing
quietly; and tests/release_contract.cpp holds a hidden ([.]) sentinel
case inside the same selection that fails unless NDEBUG is defined, so a
Dev build cannot pass by mistake. Hidden is what keeps it out of ctest
and tests-full, which are default runs.

And the tag itself is now enforced: release_contract_guard fails on a
conditional case that is untagged (no check at all, and silent), on a
tagged case with no conditional, and if the sentinel or the last tag
disappears. Without it the convention lived only in prose, which is the
same gap one layer up. Nine mutations tested.

The vcpkg-release preset builds into build-release/ while pointing
VCPKG_INSTALLED_DIR back at build/vcpkg_installed, so the job reuses the
ports the Dev tree and the CI cache already hold. The job sources
tools/ci/ci-stages.sh rather than restating its commands, and
run-local-ci.sh gains the stage.
@nnewson
nnewson merged commit 58f3cf5 into main Aug 27, 2026
13 of 15 checks passed
@nnewson
nnewson deleted the release-suite-build branch August 27, 2026 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant