diff --git a/.github/workflows/mutation-testing.yml b/.github/workflows/mutation-testing.yml new file mode 100644 index 0000000..76ec690 --- /dev/null +++ b/.github/workflows/mutation-testing.yml @@ -0,0 +1,68 @@ +name: Mutation Testing + +on: + pull_request: + paths: + - "src/**" + - "tests/**" + - "composer.json" + - "phpunit.xml" + - ".github/workflows/mutation-testing.yml" + schedule: + - cron: "41 3 * * 0" + workflow_dispatch: + +permissions: + contents: read + +jobs: + mutation: + runs-on: ubuntu-latest + timeout-minutes: 90 + name: Pest mutation tests + + steps: + - name: Checkout code + uses: actions/checkout@v7 + with: + fetch-depth: 0 + + - name: Setup PHP + uses: shivammathur/setup-php@v2 + with: + php-version: "8.4" + extensions: dom, libxml, mbstring, simplexml, tokenizer, xmlwriter, zip + coverage: pcov + + - name: Install project dependencies + run: composer update --prefer-stable --prefer-dist --no-interaction + + - name: Run mutation tests + env: + EVENT_NAME: ${{ github.event_name }} + BASE_REF: ${{ github.base_ref }} + run: | + if [ "$EVENT_NAME" = "pull_request" ]; then + git fetch origin "$BASE_REF" + MUTATION_PATHS="$(git diff --name-only --diff-filter=AMR "origin/$BASE_REF...HEAD" -- 'src/*.php' 'src/**/*.php' | paste -sd, -)" + + if [ -z "$MUTATION_PATHS" ]; then + echo "No changed PHP source files to mutate." + exit 0 + fi + + vendor/bin/pest \ + --mutate \ + --parallel \ + --path="$MUTATION_PATHS" \ + --covered-only \ + --ignore-min-score-on-zero-mutations \ + --min=50 + else + vendor/bin/pest \ + --mutate \ + --parallel \ + --everything \ + --covered-only \ + --min=50 + fi diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 86aacd9..f2c8484 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -22,6 +22,7 @@ The supported PHP matrix is 8.0 through 8.5. CI must remain green on all support - Public API changes are checked automatically against `origin/main` with Roave Backward Compatibility Check. - Update README/API/cookbook documentation when public behavior changes. - Keep coverage at or above the configured 90% project and patch thresholds. +- Source/test changes are mutation-tested with Infection; new behavior should kill relevant mutants rather than only execute lines. - Use English for source code, comments, commit messages and pull-request descriptions. ## Compatibility diff --git a/README.md b/README.md index b9493fd..9aa1c19 100644 --- a/README.md +++ b/README.md @@ -402,6 +402,10 @@ The test suite enforces a minimum **90% project coverage** locally and in CI. Th composer test:coverage ``` +## Mutation testing + +In addition to the 90% line-coverage gate, source changes are mutation-tested with Pest. Pull requests mutate changed covered code, while a weekly/manual workflow runs the full covered source with a 50% minimum mutation score (initial full-run baseline: 53.66%). See [Mutation testing](docs/mutation-testing.md). + ## Compatibility regression testing The regular test suite includes a deterministic corpus for malformed XML, encoded and Unicode asset paths, malformed iDevice state and cyclic page hierarchies. diff --git a/docs/mutation-testing.md b/docs/mutation-testing.md new file mode 100644 index 0000000..8987c4b --- /dev/null +++ b/docs/mutation-testing.md @@ -0,0 +1,57 @@ +# Mutation testing + +Line coverage measures whether code executes. Mutation testing measures whether the test suite can detect meaningful behavioral changes. + +The project uses Pest's native mutation testing on PHP 8.4. + +Configured quality gate: + +- minimum mutation score: **50%**; +- only covered code is mutated, because line coverage is enforced separately at 90%. + +## Pull requests + +PRs calculate the changed PHP files under `src/` with Git and pass the resulting comma-separated list to Pest's supported `--path` filter: + +```bash +MUTATION_PATHS="$(git diff --name-only --diff-filter=AMR "origin/$BASE_REF...HEAD" -- 'src/*.php' 'src/**/*.php' | paste -sd, -)" + +vendor/bin/pest \ + --mutate \ + --parallel \ + --path="$MUTATION_PATHS" \ + --covered-only \ + --ignore-min-score-on-zero-mutations \ + --min=50 +``` + +When no PHP source file changed, the mutation job exits successfully without launching the mutation engine. + +## Scheduled and manual full runs + +A complete mutation run over all covered source code executes weekly and can also be started manually with `workflow_dispatch`: + +```bash +vendor/bin/pest \ + --mutate \ + --parallel \ + --everything \ + --covered-only \ + --min=50 +``` + +## Why PHP 8.4 only? + +Mutation tooling has a newer PHP requirement than the parser itself. Keeping mutation testing in a dedicated PHP 8.4 job avoids changing the library's PHP 8.0 runtime support. + + +## Baseline + +The first complete run after introducing mutation testing produced: + +- **53.66% mutation score**; +- 1,586 tested mutants; +- 1,373 untested mutants; +- 4 timed-out mutants. + +The initial CI floor is therefore set to **50%**, slightly below the measured baseline so existing code starts green while regressions are blocked. The intended maintenance strategy is to ratchet this threshold upward as escaped/untested mutants are addressed. diff --git a/mkdocs.yml b/mkdocs.yml index 9ed1e9e..8ffdfc0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -61,6 +61,7 @@ nav: - Security: security.md - Release Supply Chain: supply-chain.md - Backward Compatibility: backward-compatibility.md + - Mutation Testing: mutation-testing.md - iDevices: idevices.md - Assets and Package Entries: assets.md - Performance: performance.md