Test/harden tests against mutation survivors

.claude/CLAUDE.md

@@ -3,15 +3,18 @@
 Testing rules and the import-stubbing pattern live in the root [CLAUDE.md](../CLAUDE.md); this file only records dev-team toolchain conventions.
 
 ## Stack
-- Python 3 (runs in an Alpine container; `transmission_rpc` is apk-only)
+- Python 3 (runs in an Alpine container; `transmission_rpc` comes from apk there, pinned from PyPI for tests)
 - Docker (`Dockerfile`, `entrypoint.sh`)
 - Tests: stdlib `unittest` (`tests/`)
 
 ## Commands
 - Test: `python -m unittest discover -s tests -v` (Windows: `py -m unittest discover -s tests -v`)
 - Lint/format: `ruff check .` / `ruff format .`
-- Typecheck: `mypy fetch_torrents.py torrent_scrape.py configure_transmission.py`
-- Mutation: `mutmut run` (mutmut<3)
+- Typecheck: `mypy` (files and settings in `pyproject.toml`)
+- CI's `test` job runs ruff check (defaults + bandit `S`), ruff format --check, mypy, shellcheck and hadolint before the tests; tool versions are pinned in `requirements-dev.txt`, the shellcheck/hadolint images in `.github/workflows/docker-image.yml` (commands in CONTRIBUTING.md)
+- Exec bit: a tracked file with a `#!` shebang must be `100755` in git, or ruff's EXE001 fails on CI. Ruff can't see the bit on Windows, so it passes locally; `tests/test_repo_hygiene.py` catches it instead. Fix with `git update-index --chmod=+x <file>` (`chmod` does nothing with `core.fileMode=false`), or drop the shebang from a module that is only imported
+- Coverage: `coverage run -m unittest discover -s tests` then `coverage report` (branch coverage; config in `pyproject.toml`, which also sets the `fail_under` ratchet CI enforces)
+- Mutation: `mutmut run` (mutmut<3; config in `pyproject.toml`, see CONTRIBUTING.md; run with `PYTHONDONTWRITEBYTECODE=1`)
 - Dev deps: `pip install -r requirements-dev.txt` (includes `requirements-test.txt`)
 
 ## Active agent templates

.claude/skills/pr/SKILL.md

@@ -9,6 +9,6 @@ description: Verify and open a pull request for linux-iso-seeder — runs lint,
 2. Run, in order, and stop on the first failure (report its output; do not open the PR):
    - `ruff check .`
    - `ruff format --check .`
-   - `mypy fetch_torrents.py torrent_scrape.py configure_transmission.py` — advisory; report findings but do not block.
+   - `mypy` (files and settings come from `pyproject.toml`)
    - `python -m unittest discover -s tests -v` (Windows: `py -m unittest discover -s tests -v`) — must pass; CI gates the Docker build on this.
 3. Push the branch and run `gh pr create --base main` with a summary of the change and the test result.

.gitattributes

@@ -1 +1,3 @@
 *.sh text eol=lf
+# Recorded upstream pages: keep byte-for-byte as captured.
+tests/fixtures/pages/** -text

.github/workflows/docker-image.yml

@@ -37,13 +37,48 @@ jobs:
     - name: Set up Python
       uses: actions/setup-python@v7
       with:
-        python-version: "3.x"
+        # alpine:latest's python3 minor; bump with requirements-test.txt.
+        python-version: "3.14"
 
-    - name: Install test dependencies
-      run: pip install -r requirements-test.txt
+    - name: Install dev dependencies
+      run: pip install -r requirements-dev.txt
+
+    - name: Lint
+      run: ruff check .
+
+    - name: Check formatting
+      run: ruff format --check .
+
+    - name: Type-check
+      run: mypy
+
+    - name: Lint shell scripts
+      run: docker run --rm -v "$PWD:/mnt" -w /mnt koalaman/shellcheck:v0.11.0 entrypoint.sh scripts/*.sh
+
+    - name: Lint Dockerfile
+      run: docker run --rm -i hadolint/hadolint:v2.15.1 < Dockerfile
 
     - name: Run tests
-      run: python -m unittest discover -s tests -v
+      run: coverage run -m unittest discover -s tests -v
+
+    # Fails below [tool.coverage.report] fail_under; that and the other
+    # settings (branch coverage, which modules) are in pyproject.toml.
+    - name: Coverage report
+      run: coverage report
+
+    # Same suite again with env vars exported that would flip outcomes if any
+    # test leaked the developer's shell environment into production code.
+    - name: Run tests with a hostile environment
+      run: python -m unittest discover -s tests
+      env:
+        SKIP_RATIO_CHECK: "true"
+        CLEANUP_MIN_RATIO: "5"
+        CLEANUP_SKIP_RATIO_CHECK: "true"
+        TRANSMISSION_RPC_USERNAME: x
+        TRANSMISSION_RPC_PASSWORD: y
+        FETCH_TORRENTS_DISTROS: arch
+        FETCH_TORRENTS_LOG_MAX_BYTES: "7"
+        FETCH_TORRENTS_ALWAYS_LOG: "false"
 
   build:
     needs: test
@@ -55,6 +90,19 @@ jobs:
     steps:
     - uses: actions/checkout@v7
 
+    # The test job uses pip packages on Ubuntu; this reruns the suite on the
+    # apk versions the container actually ships (Dockerfile `test` stage).
+    - name: Run unit tests inside the image
+      run: docker build --target test .
+
+    - name: Build image for smoke test
+      run: docker build -t linux-iso-seeder:smoke .
+
+    # Starts the image (root and RUN_AS_NON_ROOT) and checks it comes up;
+    # nothing below pushes unless this passes.
+    - name: Smoke-test the image
+      run: sh scripts/smoke_test.sh linux-iso-seeder:smoke
+
     - name: Log in to GitHub Container Registry
       uses: docker/login-action@v4
       with:

.github/workflows/drift.yml

@@ -0,0 +1,154 @@
+name: Live drift check
+
+# The merge check is deterministic by design (recorded pages, fakes, no
+# network). This weekly job checks the moving parts it can't: the live
+# distro sites, a live tracker, a real transmission-daemon, and the
+# published image as deployed (network + apk upgrade at start). It never
+# blocks anything; a failure opens (or comments on) one issue labelled
+# `drift`.
+
+on:
+  schedule:
+    - cron: "17 5 * * 1"  # Mondays 05:17 UTC
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+env:
+  IMAGE: ghcr.io/squazz/linux-iso-seeder:latest
+
+jobs:
+  scrapers:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v7
+
+    - uses: actions/setup-python@v7
+      with:
+        python-version: "3.x"
+
+    - run: pip install -r requirements-test.txt
+
+    # Every fetcher must return well-formed candidates, at least one of
+    # which answers 200.
+    - name: Run fetchers against the live sites
+      run: python scripts/live_check.py scrapers
+
+    # Fresh captures to diff against tests/fixtures/pages.
+    - name: Re-record fixtures
+      if: failure()
+      run: python scripts/record_fixtures.py
+
+    - name: Upload re-recorded fixtures
+      if: failure()
+      uses: actions/upload-artifact@v4
+      with:
+        name: rerecorded-fixtures
+        path: tests/fixtures/pages
+
+  tracker:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v7
+
+    - uses: actions/setup-python@v7
+      with:
+        python-version: "3.x"
+
+    - run: pip install -r requirements-test.txt
+
+    - name: Scrape a current Ubuntu torrent's tracker
+      run: python scripts/live_check.py tracker
+
+  transmission:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v7
+
+    # Just the daemon (no daily fetch loop), offline: the round trip uses a
+    # self-made tracker-less torrent, so nothing is downloaded.
+    - name: Start the published image's transmission-daemon
+      run: |
+        docker run -d --name seeder --network none --entrypoint sh "$IMAGE" -c \
+          'python3 /usr/local/bin/configure_transmission.py && exec transmission-daemon --foreground --config-dir /config --download-dir /downloads --watch-dir /watch'
+
+    # Inside the image, so it's the image's own apk py3-transmission-rpc
+    # and fetch_torrents.py talking to its own daemon.
+    - name: Add, read, log and remove a torrent through the RPC
+      run: |
+        docker cp scripts/transmission_roundtrip.py seeder:/tmp/
+        docker exec -w /usr/local/bin seeder python3 /tmp/transmission_roundtrip.py
+
+    - name: Daemon logs
+      if: always()
+      run: docker logs seeder
+
+  published-image:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v7
+
+    - name: Smoke-test the published image as deployed
+      run: sh scripts/smoke_test.sh "$IMAGE"
+      env:
+        SMOKE_NETWORK: bridge
+        SMOKE_SKIP_PACKAGE_UPDATES: "false"
+        SMOKE_TIMEOUT_SECONDS: "300"
+
+  # Trend only: results are uploaded, and a low score never fails the run.
+  mutation:
+    runs-on: ubuntu-latest
+    timeout-minutes: 180
+    continue-on-error: true
+    steps:
+    - uses: actions/checkout@v7
+
+    - uses: actions/setup-python@v7
+      with:
+        python-version: "3.x"
+
+    - run: pip install -r requirements-dev.txt
+
+    - name: Run mutmut
+      run: |
+        mutmut run --no-progress || true
+        mutmut results > mutmut-results.txt || true
+
+    - uses: actions/upload-artifact@v4
+      with:
+        name: mutmut-results
+        path: |
+          mutmut-results.txt
+          .mutmut-cache
+
+  report:
+    needs: [scrapers, tracker, transmission, published-image]
+    if: failure()
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+    steps:
+    - uses: actions/github-script@v7
+      with:
+        script: |
+          const label = "drift";
+          const run = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
+          const body = `The weekly live drift check failed: ${run}\n\n` +
+            "Check which job failed. For scraper drift, the run has a " +
+            "`rerecorded-fixtures` artifact to diff against `tests/fixtures/pages`.";
+          try {
+            await github.rest.issues.createLabel({ ...context.repo, name: label, color: "d93f0b" });
+          } catch (error) {
+            if (error.status !== 422) throw error;  // 422: already exists
+          }
+          const { data: open } = await github.rest.issues.listForRepo({
+            ...context.repo, labels: label, state: "open",
+          });
+          if (open.length > 0) {
+            await github.rest.issues.createComment({ ...context.repo, issue_number: open[0].number, body });
+          } else {
+            await github.rest.issues.create({
+              ...context.repo, title: "Live drift check failed", labels: [label], body,
+            });
+          }

.gitignore

@@ -219,3 +219,7 @@ plans/
 
 # CodeGraph local index (machine-local)
 .codegraph/
+
+# Mutation testing (mutmut 2.x)
+.mutmut-cache
+mutants/

CLAUDE.md

@@ -33,20 +33,52 @@ CI runs this same command on every push and PR (`.github/workflows/docker-image.
 
 ## Why the test setup looks the way it does
 
-`fetch_torrents.py` is written to run inside the container: it imports
-`transmission_rpc` (an apk-only package, not pip-installable) and opens log
-files under `/logs` at import time. `tests/test_fetch_torrents.py` stubs
-`transmission_rpc` via `sys.modules` and points `FETCH_TORRENTS_LOG_DIR` at a
-temp directory before importing the module, so tests run without a container
-or a live Transmission daemon. Follow that pattern for any new test file that
-needs to import `fetch_torrents`.
+`fetch_torrents.py` is written to run inside the container, which installs
+`transmission_rpc` from Alpine's `py3-transmission-rpc`. Tests use the same
+version from PyPI (`transmission-rpc`, pinned in `requirements-test.txt`;
+bump the pin when `alpine:latest` moves), so they run without a container
+or a live Transmission daemon. Never talk to a real `Client` in a test: use
+`tests/helpers.py` `FakeTransmission` and `FakeTorrent`, which are checked
+against the pinned library's `Client` signatures and `Torrent` fields (see
+`tests/test_helpers.py`), so they can't drift from it.
+
+Importing the module opens no files: log handlers are only attached by
+`configure_logging()`, which the `__main__` block calls. Tests that need it
+should restore the logger afterwards (see `configure_logging_for_test()`).
+The daily run itself is `run(config, client_factory=..., today=...)` with
+its I/O (Transmission client, clock, disk usage) injected, so it is tested
+directly with fakes from `tests/helpers.py` (`FakeTransmission`, `FakeDisk`);
+`tests/test_daily_run.py` keeps one runpy test for the `__main__` wiring.
+Env vars are read once by `load_config(env)`. Transmission's HTTP RPC (the
+ratio log) is faked with `helpers.FakeTransmissionRpc` as `requests.post`:
+409 plus session id first, like the daemon, recording each call's auth.
+The file locations are module globals read at import
+(`FETCH_TORRENTS_LOG_DIR`, `FETCH_TORRENTS_WATCH_DIR`); in-process tests
+point them at temp dirs with `helpers.use_watch_dir(self)` and
+`helpers.use_state_files(self)`, which return the paths, instead of
+patching them by hand.
+
+Every test module starts with `import helpers` (which also puts the repo
+root on `sys.path`) and a `setUpModule()` calling `helpers.isolate_module()`:
+that scrubs the developer's own `CLEANUP_*`, `FETCH_TORRENTS_*` etc. so they
+can't change test outcomes, and refuses any real network connection. HTTP is
+faked with `helpers.fake_response()` (a real `requests.Response`) or
+`helpers.RecordedPages` (pages captured by `scripts/record_fixtures.py`).
+`fetch_torrents` tests are split by concern: `test_ratio_gate.py`,
+`test_fetchers.py`, `test_scrapers.py`, `test_cleanup.py`,
+`test_persistence.py`, `test_logging.py`, `test_old_release_demand.py`,
+`test_config.py` (`load_config()`, `parse_bool()`, `make_client_factory()`,
+the run summary) and `test_daily_run.py` (`run()`, `__main__`).
+`test_helpers.py` tests the doubles in `tests/helpers.py` themselves;
+`test_repo_hygiene.py` checks the repository itself (shebang files are
+executable in git).
 
 Business logic that needs to be unit-testable should be kept as pure
 functions (taking plain data in, returning plain data out) separate from
 functions that perform I/O (RPC calls, network requests, file writes) — see
 `plan_cleanup()` vs. `cleanup_old_versions()` for the pattern.
 
-Test-only dependencies (currently just `requests` and `beautifulsoup4`,
-needed to import the module — `unittest` itself is stdlib) live in
+Test-only dependencies (`requests`, `beautifulsoup4` and `transmission-rpc`,
+needed to import the modules — `unittest` itself is stdlib) live in
 `requirements-test.txt`, separate from the apk packages the container
 actually runs with.