Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions .github/workflows/mutation-testing.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
name: Mutation Testing

on:
pull_request:
paths:
- "src/**"
- "tests/**"
- "composer.json"
- "phpunit.xml"
- ".github/workflows/mutation-testing.yml"
schedule:
- cron: "41 3 * * 0"
workflow_dispatch:

permissions:
contents: read

jobs:
mutation:
runs-on: ubuntu-latest
timeout-minutes: 90
name: Pest mutation tests

steps:
- name: Checkout code
uses: actions/checkout@v7
with:
fetch-depth: 0

- name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: "8.4"
extensions: dom, libxml, mbstring, simplexml, tokenizer, xmlwriter, zip
coverage: pcov

- name: Install project dependencies
run: composer update --prefer-stable --prefer-dist --no-interaction

- name: Run mutation tests
env:
EVENT_NAME: ${{ github.event_name }}
BASE_REF: ${{ github.base_ref }}
run: |
if [ "$EVENT_NAME" = "pull_request" ]; then
git fetch origin "$BASE_REF"
MUTATION_PATHS="$(git diff --name-only --diff-filter=AMR "origin/$BASE_REF...HEAD" -- 'src/*.php' 'src/**/*.php' | paste -sd, -)"

if [ -z "$MUTATION_PATHS" ]; then
echo "No changed PHP source files to mutate."
exit 0
fi

vendor/bin/pest \
--mutate \
--parallel \
--path="$MUTATION_PATHS" \
--covered-only \
--ignore-min-score-on-zero-mutations \
--min=50
else
vendor/bin/pest \
--mutate \
--parallel \
--everything \
--covered-only \
--min=50
fi
1 change: 1 addition & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ The supported PHP matrix is 8.0 through 8.5. CI must remain green on all support
- Public API changes are checked automatically against `origin/main` with Roave Backward Compatibility Check.
- Update README/API/cookbook documentation when public behavior changes.
- Keep coverage at or above the configured 90% project and patch thresholds.
- Source/test changes are mutation-tested with Infection; new behavior should kill relevant mutants rather than only execute lines.
- Use English for source code, comments, commit messages and pull-request descriptions.

## Compatibility
Expand Down
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,10 @@ The test suite enforces a minimum **90% project coverage** locally and in CI. Th
composer test:coverage
```

## Mutation testing

In addition to the 90% line-coverage gate, source changes are mutation-tested with Pest. Pull requests mutate changed covered code, while a weekly/manual workflow runs the full covered source with a 50% minimum mutation score (initial full-run baseline: 53.66%). See [Mutation testing](docs/mutation-testing.md).

## Compatibility regression testing

The regular test suite includes a deterministic corpus for malformed XML, encoded and Unicode asset paths, malformed iDevice state and cyclic page hierarchies.
Expand Down
57 changes: 57 additions & 0 deletions docs/mutation-testing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Mutation testing

Line coverage measures whether code executes. Mutation testing measures whether the test suite can detect meaningful behavioral changes.

The project uses Pest's native mutation testing on PHP 8.4.

Configured quality gate:

- minimum mutation score: **50%**;
- only covered code is mutated, because line coverage is enforced separately at 90%.

## Pull requests

PRs calculate the changed PHP files under `src/` with Git and pass the resulting comma-separated list to Pest's supported `--path` filter:

```bash
MUTATION_PATHS="$(git diff --name-only --diff-filter=AMR "origin/$BASE_REF...HEAD" -- 'src/*.php' 'src/**/*.php' | paste -sd, -)"

vendor/bin/pest \
--mutate \
--parallel \
--path="$MUTATION_PATHS" \
--covered-only \
--ignore-min-score-on-zero-mutations \
--min=50
```

When no PHP source file changed, the mutation job exits successfully without launching the mutation engine.

## Scheduled and manual full runs

A complete mutation run over all covered source code executes weekly and can also be started manually with `workflow_dispatch`:

```bash
vendor/bin/pest \
--mutate \
--parallel \
--everything \
--covered-only \
--min=50
```

## Why PHP 8.4 only?

Mutation tooling has a newer PHP requirement than the parser itself. Keeping mutation testing in a dedicated PHP 8.4 job avoids changing the library's PHP 8.0 runtime support.


## Baseline

The first complete run after introducing mutation testing produced:

- **53.66% mutation score**;
- 1,586 tested mutants;
- 1,373 untested mutants;
- 4 timed-out mutants.

The initial CI floor is therefore set to **50%**, slightly below the measured baseline so existing code starts green while regressions are blocked. The intended maintenance strategy is to ratchet this threshold upward as escaped/untested mutants are addressed.
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ nav:
- Security: security.md
- Release Supply Chain: supply-chain.md
- Backward Compatibility: backward-compatibility.md
- Mutation Testing: mutation-testing.md
- iDevices: idevices.md
- Assets and Package Entries: assets.md
- Performance: performance.md
Expand Down
Loading