Run the release half of the suite, and keep it running - #143
Merged
Conversation
Rejection tests guarded by `#ifdef NDEBUG` — the release behaviour of every writer that asserts in Dev and returns `false` under NDEBUG — have never run. Every preset here builds Dev, so those bodies compiled to nothing on every machine and in every job that has ever checked them: the cases ran, passed, and asserted precisely nothing. Death tests are not the fix; they prove the Dev assertion fires and say nothing about the half that ships. Tag those cases `[release-contract]` (18 of them) and add a Linux-only CI job that builds a genuine Release tree and runs that selection. Linux alone because the behaviour does not vary by platform, and the tagged cases alone because the whole Release suite would drag in the optimisation-sensitive physics goldens. Two mechanisms keep the job honest, since a job that checks nothing looks exactly like a job that finds no fault. Catch2 exits non-zero when a spec matches nothing, so a renamed tag fails rather than passing quietly; and tests/release_contract.cpp holds a hidden ([.]) sentinel case inside the same selection that fails unless NDEBUG is defined, so a Dev build cannot pass by mistake. Hidden is what keeps it out of ctest and tests-full, which are default runs. And the tag itself is now enforced: release_contract_guard fails on a conditional case that is untagged (no check at all, and silent), on a tagged case with no conditional, and if the sentinel or the last tag disappears. Without it the convention lived only in prose, which is the same gap one layer up. Nine mutations tested. The vcpkg-release preset builds into build-release/ while pointing VCPKG_INSTALLED_DIR back at build/vcpkg_installed, so the job reuses the ports the Dev tree and the CI cache already hold. The job sources tools/ci/ci-stages.sh rather than restating its commands, and run-local-ci.sh gains the stage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rejection tests guarded by
#ifdef NDEBUG— the release behaviour of every writer that asserts in Dev and returnsfalseunder NDEBUG — have never run. Every preset here builds Dev, so those bodies compiled to nothing on every machine and in every job that has ever checked them: the cases ran, passed, and asserted precisely nothing. Death tests are not the fix; they prove the Dev assertion fires and say nothing about the half that ships.Tag those cases
[release-contract](18 of them) and add a Linux-only CI job that builds a genuine Release tree and runs that selection. Linux alone because the behaviour does not vary by platform, and the tagged cases alone because the whole Release suite would drag in the optimisation-sensitive physics goldens.Two mechanisms keep the job honest, since a job that checks nothing looks exactly like a job that finds no fault. Catch2 exits non-zero when a spec matches nothing, so a renamed tag fails rather than passing quietly; and tests/release_contract.cpp holds a hidden ([.]) sentinel case inside the same selection that fails unless NDEBUG is defined, so a Dev build cannot pass by mistake. Hidden is what keeps it out of ctest and tests-full, which are default runs.
And the tag itself is now enforced: release_contract_guard fails on a conditional case that is untagged (no check at all, and silent), on a tagged case with no conditional, and if the sentinel or the last tag disappears. Without it the convention lived only in prose, which is the same gap one layer up. Nine mutations tested.
The vcpkg-release preset builds into build-release/ while pointing VCPKG_INSTALLED_DIR back at build/vcpkg_installed, so the job reuses the ports the Dev tree and the CI cache already hold. The job sources tools/ci/ci-stages.sh rather than restating its commands, and run-local-ci.sh gains the stage.