From 812f17995082ca19b523be01a19750abf11d147a Mon Sep 17 00:00:00 2001 From: Squazz Date: Sun, 18 Oct 2026 23:08:45 +0200 Subject: [PATCH] Adding tasks based on mutation testing --- tasks/10-ratio-gate-boundaries.md | 31 ++++++++++++++ tasks/11-get-previous-ratios-tests.md | 26 ++++++++++++ tasks/12-test-env-clock-isolation.md | 21 ++++++++++ tasks/13-bdecode-truncated-input.md | 29 ++++++++++++++ tasks/14-cleanup-boundaries.md | 25 ++++++++++++ tasks/15-ci-static-gates.md | 21 ++++++++++ tasks/16-dev-tooling-coverage-mutation.md | 20 +++++++++ tasks/17-daily-run-characterization.md | 39 ++++++++++++++++++ tasks/18-extract-run-and-logging.md | 22 ++++++++++ tasks/19-inject-cleanup-seams.md | 27 +++++++++++++ tasks/20-transmission-double-fidelity.md | 24 +++++++++++ tasks/21-scraper-contract-tests.md | 24 +++++++++++ tasks/22-container-smoke-test.md | 20 +++++++++ tasks/23-scheduled-live-drift-job.md | 20 +++++++++ tasks/24-parser-gaps.md | 16 ++++++++ tasks/25-configure-transmission-main.md | 18 +++++++++ tasks/26-assertions-and-fixtures.md | 22 ++++++++++ tasks/27-remeasure-baseline.md | 11 +++++ tasks/README.md | 49 +++++++++++++++++++++++ 19 files changed, 465 insertions(+) create mode 100644 tasks/10-ratio-gate-boundaries.md create mode 100644 tasks/11-get-previous-ratios-tests.md create mode 100644 tasks/12-test-env-clock-isolation.md create mode 100644 tasks/13-bdecode-truncated-input.md create mode 100644 tasks/14-cleanup-boundaries.md create mode 100644 tasks/15-ci-static-gates.md create mode 100644 tasks/16-dev-tooling-coverage-mutation.md create mode 100644 tasks/17-daily-run-characterization.md create mode 100644 tasks/18-extract-run-and-logging.md create mode 100644 tasks/19-inject-cleanup-seams.md create mode 100644 tasks/20-transmission-double-fidelity.md create mode 100644 tasks/21-scraper-contract-tests.md create mode 100644 tasks/22-container-smoke-test.md create mode 100644 tasks/23-scheduled-live-drift-job.md create mode 100644 tasks/24-parser-gaps.md create mode 100644 tasks/25-configure-transmission-main.md create mode 100644 tasks/26-assertions-and-fixtures.md create mode 100644 tasks/27-remeasure-baseline.md create mode 100644 tasks/README.md diff --git a/tasks/10-ratio-gate-boundaries.md b/tasks/10-ratio-gate-boundaries.md new file mode 100644 index 0000000..755c0ae --- /dev/null +++ b/tasks/10-ratio-gate-boundaries.md @@ -0,0 +1,31 @@ +# 10 — Ratio gate boundaries and branches + +**Priority**: P1 · **Class**: NO_REFACTOR · **Depends on**: — (do it alongside 12; the tests here must pin `SKIP_RATIO_CHECK`) + +## Problem +`should_fetch_torrent` / `_should_fetch_devuan` are the project's core business rule: don't fetch +a new release until the previous one has reached ratio 1.0. Every line of them runs in tests, but +mutation testing shows the tests don't pin the outcomes. Only 67% of `should_fetch_torrent` mutants +are killed, and one branch never runs at all. + +## Evidence (mutmut survivors, `fetch_torrents.py`) +- `:434` and `:374`: `>= 1.0` → `> 1.0` survives. No test has a previous ratio of exactly 1.0. +- `:432`: `return True # no previous version` → `False` survives, and coverage shows the line is never executed. +- `:430`: `v < version` → `v <= version` survives. +- `:364`: `n != name and DEVUAN_NAME_PATTERN.match(n)` → `or` survives. A non-Devuan ratio can count as the "prior Devuan" ratio. +- `:386–399`: the `SKIP_RATIO_CHECK`, unknown-distro and unparseable-name branches can all be flipped to `return False` without a test failing. +- `:416`: the documented rule that a live ratio beats the removed-history ratio for the same name is never asserted. + +## Acceptance criteria +- Prior version ratio exactly `1.0` → fetch (`True`); `0.999` → don't fetch (`False`). Do this for both a standard distro and Devuan. +- Only *newer* versions in `ratios` (no older one) → `True`. This must execute L432. +- `ratios` contains the *same* version as `name` only → behavior asserted explicitly. The `<` vs `<=` boundary at L430 must be killed. +- Devuan: `ratios` mixing a low-ratio non-Devuan torrent with a high-ratio Devuan one → the decision uses only the Devuan ratio. +- Name in both `ratios` (live) and `removed_history` with different ratios → the live value decides. +- `SKIP_RATIO_CHECK=true` → `True` even with a low prior ratio. Unknown distro → `True`. Unparseable name of a known distro → `True` (and logs an error). +- Re-running mutmut on `should_fetch_torrent` / `_should_fetch_devuan`: all the survivors listed above are killed. + +## Approach +1. Add the cases to the existing `ShouldFetchTorrent*` classes in `tests/test_fetch_torrents.py`. Wrap each class in `patch.dict(os.environ, {'SKIP_RATIO_CHECK': 'false'})` in `setUp`, so the developer's shell can't flip them (see task 12). +2. Each new test should fail when the corresponding mutant is applied by hand. Confirm this for at least the `>=` and L432 cases. +3. No production change is expected. If a test exposes a real bug, fix it test-first in this task. diff --git a/tasks/11-get-previous-ratios-tests.md b/tasks/11-get-previous-ratios-tests.md new file mode 100644 index 0000000..c92dc0c --- /dev/null +++ b/tasks/11-get-previous-ratios-tests.md @@ -0,0 +1,26 @@ +# 11 — Test `get_previous_ratios` in-process + +**Priority**: P1 · **Class**: NO_REFACTOR · **Depends on**: — + +## Problem +`get_previous_ratios(log_file)` (`fetch_torrents.py:163–195`) parses the previous run's ratio log. +Its output is the `ratios` input to the ratio gate (task 10). Only `RatioLogSurvivesRestartTests` +reaches it, and that runs in a subprocess, so coverage records L164–195 as unexecuted and only the +happy path is checked. The segment-selection logic is untested: last START wins, START without END, +and an END before START. + +## Acceptance criteria +In-process tests, each writing a temp log file and calling `ft.get_previous_ratios(path)`: +- Missing file → `{}`. +- No markers at all → every `[ratio] name → x.yyy` line in the file is parsed. +- Two complete START/END segments → only the **last** segment's ratios are returned. An entry that exists only in the first segment must be absent. +- Last START with no END after it (a crashed run) → everything after that START is parsed. +- An END that appears before the last START is ignored. +- Names with internal spaces/padding are stripped. A ratio without a decimal point (`→ 1`) is not matched, so the current regex behavior is pinned. +- Duplicate names inside a segment → the last one wins. +- Coverage of L164–195 is 100% in the normal `unittest` run. + +## Approach +Add a `GetPreviousRatiosTests` class next to `RatioLogSurvivesRestartTests`, and keep the subprocess +test as-is, since it guards the import-time truncation bug. Write the log lines in the real format +(`2026-08-01 00:00:00,000 INFO: [ratio] … → 0.664`). diff --git a/tasks/12-test-env-clock-isolation.md b/tasks/12-test-env-clock-isolation.md new file mode 100644 index 0000000..665b783 --- /dev/null +++ b/tasks/12-test-env-clock-isolation.md @@ -0,0 +1,21 @@ +# 12 — Isolate tests from env vars and the clock + +**Priority**: P1 · **Class**: NO_REFACTOR (test-side only; the production seams are task 19) · **Depends on**: — + +## Problem +Some tests pass only because the developer's shell and the wall clock happen to cooperate: +- `should_fetch_torrent` reads `SKIP_RATIO_CHECK` directly (`fetch_torrents.py:386`). With `SKIP_RATIO_CHECK=true` exported, the `ShouldFetchTorrent*` tests at `tests/test_fetch_torrents.py:241–385` fail. This is confirmed. +- `cleanup_*` reads `CLEANUP_*`. `get_rpc_credentials` reads `TRANSMISSION_RPC_*`, which affects the tests around `:1663–1712` and `:1777–1866`. +- `CleanupRemovesWatchFileTests` builds its history from `date.today()` (`:1855`) while production calls `date.today()` again. A run that crosses midnight shifts the 32-day offset. +- Temp dirs from `tempfile.mkdtemp` are never removed, in all three test files. + +## Acceptance criteria +- Every test class that runs code reading these env vars patches the environment with `patch.dict(os.environ, {...}, clear=False)` and explicitly sets **or deletes** the keys it depends on. Add a small helper, e.g. `isolated_env(**overrides)`, that removes `SKIP_RATIO_CHECK`, `CLEANUP_*`, `TRANSMISSION_RPC_*` and `FETCH_TORRENTS_*` (except `FETCH_TORRENTS_LOG_DIR`) unless they're overridden. +- Running the whole suite with `SKIP_RATIO_CHECK=true CLEANUP_MIN_RATIO=5 CLEANUP_SKIP_RATIO_CHECK=true TRANSMISSION_RPC_USERNAME=x TRANSMISSION_RPC_PASSWORD=y` exported is green. +- The stagnation integration test patches `ft.date` (or uses an injected `today` once task 19 lands) to a fixed date. No `date.today()` remains in `tests/`. +- Each `mkdtemp` is paired with `self.addCleanup(shutil.rmtree, path, ignore_errors=True)`, or uses `tempfile.TemporaryDirectory`. + +## Approach +1. First, run the suite with the hostile env above and record which tests fail. Those failures are the "red" for this task. +2. Add the helper and apply it, then re-run with the hostile env until green. +3. Optional: add a CI step that runs the suite a second time with that hostile env, so regressions are caught. diff --git a/tasks/13-bdecode-truncated-input.md b/tasks/13-bdecode-truncated-input.md new file mode 100644 index 0000000..d2adc79 --- /dev/null +++ b/tasks/13-bdecode-truncated-input.md @@ -0,0 +1,29 @@ +# 13 — Fix `bdecode` accepting/mis-reporting truncated input + +**Priority**: P1 (defect) · **Class**: NO_REFACTOR · **Depends on**: — + +## Problem +`torrent_scrape.bdecode` (`torrent_scrape.py:36–75`) is meant to raise `BencodeError` on malformed +data. Verified on 84761d3: + +| Input | Actual | Expected | +| --- | --- | --- | +| `b'4:ab'` (declares 4 bytes, has 2) | returns `(b'ab', 6)`, accepted silently | `BencodeError` | +| `b'i12'` (no terminating `e`) | `ValueError: subsection not found` (from `bytes.index`) | `BencodeError` | +| `b'iabce'` (non-numeric int) | `ValueError` from `int()` | `BencodeError` | +| `b'2x:ab'` (bad length) | `ValueError` from `bytes.index`/`int()` | `BencodeError` | + +The silent truncation matters most. A truncated `.torrent` download decodes into a wrong `info` dict, +so `compute_info_hash` produces the wrong hash, and the tracker scrape then reports "no demand" +rather than "bad file". `BencodeError` subclasses `ValueError`, so existing callers that catch +`ValueError` keep working. + +## Acceptance criteria +- Each input in the table raises `BencodeError` with a message naming the problem and its index. +- A string length that runs past the end of the data raises. +- Existing round-trip tests (`BdecodeBencodeTests`) stay green. +- Mutation: the new guard lines are killed. + +## Approach +Add the table above as tests in `BdecodeBencodeTests` and confirm each fails today. Then add a length +check in the byte-string branch, and wrap the `index`/`int` calls so they re-raise as `BencodeError`. diff --git a/tasks/14-cleanup-boundaries.md b/tasks/14-cleanup-boundaries.md new file mode 100644 index 0000000..d7617a6 --- /dev/null +++ b/tasks/14-cleanup-boundaries.md @@ -0,0 +1,25 @@ +# 14 — Cleanup planning boundary tests + +**Priority**: P2 · **Class**: NO_REFACTOR · **Depends on**: 12 + +## Problem +The cleanup planners decide what gets **deleted from disk**, and they're pure functions. Mutation +shows boundaries and loop control that no test pins down: `plan_stagnation_cleanup` 60%, +`_devuan_sort_key` 60%, `plan_cleanup` 67%, `_old_version_candidates` 68%. + +## Evidence (mutmut survivors, `fetch_torrents.py`) +- `:1140` `plan_cleanup`: `ratio <= min_ratio` boundary. No test has ratio exactly equal to `min_ratio`. +- `:1305` `plan_stagnation_cleanup`: the ratio-delta comparison isn't tested at exactly `min_ratio_delta`. +- `:1283` `_stagnation_anchor`: an anchor sample exactly `window_days` old isn't tested. +- `:1270` / `:1274` `update_ratio_history`: the sample-interval gate and the pruning age aren't tested at their edges. +- `:1096–1112` `_old_version_candidates` and `:1303`: `continue` → `break` survives. No input list puts an unparseable, Devuan or no-anchor torrent **before** valid candidates. +- `_devuan_sort_key`: ordering ties on `added_date` aren't exercised. + +## Acceptance criteria +- For each boundary, add a test at the edge value and one on each side of it, with the expected keep/remove asserted per torrent. Assert the actual lists, not only their length. +- For each `continue`, add a test whose list starts with the item that should be skipped, followed by valid candidates. All the valid candidates must still be planned. +- Re-running mutmut on these functions: ≥ 90% killed, with any remaining survivors documented as equivalent. + +## Approach +Extend `PlanCleanupTests`, `PlanCleanupDevuanGroupingTests`, `PlanStagnationCleanupTests` and +`UpdateRatioHistoryTests`. Each test needs one fixed `today`, not `date.today()`. diff --git a/tasks/15-ci-static-gates.md b/tasks/15-ci-static-gates.md new file mode 100644 index 0000000..4ed81f3 --- /dev/null +++ b/tasks/15-ci-static-gates.md @@ -0,0 +1,21 @@ +# 15 — Add ruff + mypy to CI + +**Priority**: P2 · **Class**: CI (no test-first) · **Depends on**: — + +## Problem +`.claude/CLAUDE.md` lists `ruff check .` and `mypy …` as project commands, but CI +(`.github/workflows/docker-image.yml`, `test` job) runs only unittest. Current state on 84761d3: +- `ruff check .`: **42 errors** (6 auto-fixable). +- `mypy fetch_torrents.py torrent_scrape.py configure_transmission.py`: **2 errors**, both in `fetch_torrents.py`. One is a missing-stubs note for an import at L14. The other is at L1048, where the `json=` argument to `requests.post` has type `dict[str, Collection[str]]`. + +## Acceptance criteria +- `ruff check .` and the mypy command both exit 0 locally. + - Fix the code where the finding is real. Where it's noise, use a targeted `# noqa` / `# type: ignore[code]` with a reason, or a rule exclusion in config. + - Missing stubs for `transmission_rpc`: use `ignore_missing_imports` scoped to that module only. +- Ruff and mypy settings live in config (`pyproject.toml` or `ruff.toml` + `mypy.ini`), so CI and local runs agree. +- The `test` job installs `requirements-dev.txt` and runs `ruff check .` and mypy before unittest. The build job is still gated on `test`. +- Don't run `ruff format` as part of this task, so the diff stays reviewable. Add a `ruff format --check` step in a separate PR if it's wanted. + +## Notes +`requirements-dev.txt` doesn't pin versions. Pin `ruff` and `mypy` so new upstream rules don't +break CI without warning. diff --git a/tasks/16-dev-tooling-coverage-mutation.md b/tasks/16-dev-tooling-coverage-mutation.md new file mode 100644 index 0000000..1a51a91 --- /dev/null +++ b/tasks/16-dev-tooling-coverage-mutation.md @@ -0,0 +1,20 @@ +# 16 — Dev tooling: coverage + mutmut in requirements-dev + +**Priority**: P2 · **Class**: CI/config · **Depends on**: — + +## Problem +The test-health audit had to install `coverage` and `mutmut` 2.5.1 ad hoc. `.claude/CLAUDE.md` +mentions `mutmut run` (mutmut<3), but neither tool is in `requirements-dev.txt` and there's no mutmut +config. That makes the baseline in `tasks/README.md` hard to reproduce. + +## Acceptance criteria +- `requirements-dev.txt` includes pinned `coverage` and `mutmut<3`. +- Coverage config (`[tool.coverage]` or `.coveragerc`): `branch = true`, `source` = the three modules, omit `tests/`. +- Mutmut config: `paths_to_mutate` = the three modules, `runner = python -m unittest discover -s tests`, `tests_dir = tests/`. +- `.gitignore` covers `.mutmut-cache`, `mutants/`, `.coverage`, `htmlcov/`. +- CI `test` job: run unittest under `coverage run --branch` and print `coverage report`. Report only; don't add a fail-under threshold yet (task 27 sets one). +- `.claude/CLAUDE.md` Commands section lists the coverage command. + +## Notes +Mutmut doesn't belong in the per-push merge check: the full run takes about 40 minutes on Windows. If +a scheduled mutation job is wanted, it fits in task 23's scheduled workflow. diff --git a/tasks/17-daily-run-characterization.md b/tasks/17-daily-run-characterization.md new file mode 100644 index 0000000..f1bd978 --- /dev/null +++ b/tasks/17-daily-run-characterization.md @@ -0,0 +1,39 @@ +# 17 — Lock in daily-run behavior with a `runpy` test + +**Priority**: P2 · **Class**: NO_REFACTOR (test only) · **Depends on**: 12 + +## Problem +The `if __name__ == "__main__":` block (`fetch_torrents.py:1351–1457`, about 100 lines) is the whole +daily run. It's untested, and it's where all the parts get wired together: +- the ratio log feeds `should_fetch_torrent` +- `removed_history` +- the per-distro fetch loop and its success/existing/not-found/failure counting +- the old-release demand check +- `log_seed_ratios_via_http`, then the cleanup branches selected by env vars +- `enforce_disk_usage_limit` +- the summary line + +Before task 18 can refactor this block, a behavior lock-in test has to pin down what it does now. + +## Acceptance criteria +A test class, e.g. `DailyRunCharacterizationTests`, runs the module as `__main__` via +`runpy.run_path(fetch_torrents.py, run_name='__main__')`. It runs: +- with `transmission_rpc.Client` stubbed (reusing the `sys.modules` stub) +- with `requests.get`/`requests.post` patched to canned responses for each distro +- with the watch dir, log dir and state files in a temp dir +- with `shutil.disk_usage` patched and the env pinned via the task-12 helper + +At minimum it asserts: +- Distros not in `FETCH_TORRENTS_DISTROS` are never fetched. +- A candidate whose predecessor has ratio < 1.0 is **not** downloaded, and one at ≥ 1.0 is. The ratio log is pre-written in the temp log dir. +- An exception in one distro's fetcher doesn't stop the others, and it's counted as a failure. +- Each cleanup function is called only when its env flag is set. +- Disk-usage enforcement runs after cleanup, and an exception there is logged, not raised. +- The summary line reports the right counts. + +The tests make no network calls and complete in under 2 s total. + +## Notes +Because the module runs again under `runpy`, its import-time logging setup runs again too. Point +`FETCH_TORRENTS_LOG_DIR` at the temp dir, and close the handlers in `addCleanup`. If this gets too +awkward, that's evidence for task 18. Don't change production code here. diff --git a/tasks/18-extract-run-and-logging.md b/tasks/18-extract-run-and-logging.md new file mode 100644 index 0000000..7475bb0 --- /dev/null +++ b/tasks/18-extract-run-and-logging.md @@ -0,0 +1,22 @@ +# 18 — Extract `run()` and `configure_logging()` + +**Priority**: P2 · **Class**: REFACTOR_REQUIRED · **Depends on**: 17 + +## Problem +There are two blockers to testing `fetch_torrents.py` directly: +1. **The `__main__` orchestration can't be called.** It reads env vars inline (`:1407–1410`, `:1446`, …) and calls `date.today()`, `time.time()` and the module-level fetchers directly. +2. **Logging is configured at import time.** It opens files under `log_dir` (`:19` onward), which is why every test file must set `FETCH_TORRENTS_LOG_DIR` *before* `import fetch_torrents` (see CLAUDE.md). + +## Acceptance criteria +- New `run(config, *, client_factory, today, now=time.time, fetchers=DEFAULT_FETCHERS) -> RunSummary`, where `config` is a small dataclass built by `load_config(os.environ)` from every env var `main` reads today. +- `__main__` becomes `configure_logging(log_dir); run(load_config(os.environ), ...)`. +- New `configure_logging(log_dir)`. Importing the module no longer opens files or adds handlers, and the `RatioLogSurvivesRestartTests` guarantee is preserved: ratios from the previous run are readable before the ratio handler truncates the file. +- The task-17 lock-in tests pass **unchanged** against the refactor, then are rewritten to call `run()` directly with fakes. +- `tests/test_fetch_torrents.py` no longer needs the env-before-import workaround for logging. The `transmission_rpc` stub stays until task 20. +- Update the CLAUDE.md section "Why the test setup looks the way it does" to match. +- `fetch_torrents.py` branch coverage ≥ 90%. + +## Approach +Do this in small steps, with the lock-in suite green after each one. Extract `load_config` first, +then `configure_logging`, then `run`. Follow the existing `plan_cleanup()` vs `cleanup_old_versions()` +pattern: keep the decisions pure and do the I/O at the edges. diff --git a/tasks/19-inject-cleanup-seams.md b/tasks/19-inject-cleanup-seams.md new file mode 100644 index 0000000..6169ac8 --- /dev/null +++ b/tasks/19-inject-cleanup-seams.md @@ -0,0 +1,27 @@ +# 19 — Inject `today` / client / config into the `cleanup_*` functions + +**Priority**: P2 · **Class**: REFACTOR_REQUIRED · **Depends on**: 17 (can go before or with 18) + +## Problem +The I/O wrappers `cleanup_old_versions` (`:1147`), `enforce_disk_usage_limit` (`:1196`), +`cleanup_stagnant_torrents` (`:1312`) and `should_fetch_torrent` (`:386`) each reach out for their +inputs, which makes their tests patch-heavy and fragile: +- each builds its own `Client(host='localhost', …)` +- they read `CLEANUP_*` / `SKIP_RATIO_CHECK` from `os.environ` +- they call `date.today()` + +The test-design review found: +- a spec-less `MagicMock` client verified by call arguments +- a `disk_usage` side-effect list that hard-codes how many times production calls it (`tests/test_fetch_torrents.py:1959`) +- an internal function patched out (`evaluate_old_release_demand`, `:1404`+) + +`check_old_releases_for_demand(today=…)` already shows the target pattern. + +## Acceptance criteria +- `should_fetch_torrent(name, ratios, removed_history=None, *, skip_ratio_check=False)`. The env read moves to the caller, i.e. `load_config` from task 18 or `__main__`. +- `cleanup_old_versions(client, *, today, skip_ratio_check, min_ratio)` +- `cleanup_stagnant_torrents(client, *, today, window_days, min_ratio_delta)` +- `enforce_disk_usage_limit(client_factory, threshold_percent, *, today, disk_usage=shutil.disk_usage, download_path=…)`, which still only connects once the threshold is exceeded. +- Tests pass a fake client with real `get_torrents()`/`remove_torrent()` behavior: an in-memory list that records removals. Assert the resulting state rather than `assert_called_once_with`. +- The `disk_usage` fake is driven by the fake client's removals (used space drops as torrents are removed), not by a fixed side-effect list. +- The task-17 lock-in tests stay green. diff --git a/tasks/20-transmission-double-fidelity.md b/tasks/20-transmission-double-fidelity.md new file mode 100644 index 0000000..babf7a5 --- /dev/null +++ b/tasks/20-transmission-double-fidelity.md @@ -0,0 +1,24 @@ +# 20 — Transmission client double that can't drift + +**Priority**: P3 · **Class**: REFACTOR_REQUIRED · **Depends on**: 19 + +## Problem +Every test replaces `transmission_rpc` with a `types.ModuleType` stub whose `Client` is a bare +`MagicMock`. Nothing checks that `Client(host=…, port=…, username=…, password=…)`, `get_torrents()`, +`remove_torrent(id, delete_data=True)` or the torrent attributes the code reads (`name`, `id`, +`ratio`, `added_date`, …) match the real library. The container installs Alpine's +`py3-transmission-rpc`, whose version moves with `alpine:latest` + `apk upgrade`. `log_seed_ratios_via_http` +also builds the raw RPC request by hand. + +CLAUDE.md says `transmission_rpc` is "apk-only, not pip-installable", but the cd-test-architecture +review reports that the PyPI package `transmission-rpc` exists. **Check this first.** + +## Acceptance criteria +- Verify: find which `transmission-rpc` version current Alpine ships (`apk info py3-transmission-rpc` in the image), and whether that version is on PyPI. +- If it is: + - pin it in `requirements-test.txt` + - build the client double with `create_autospec(transmission_rpc.Client, instance=True)`, and torrents from the real `Torrent` shape or a validated fake + - correct the CLAUDE.md claim +- If it isn't: introduce a thin `TransmissionAdapter` (`list_torrents() -> list[TorrentInfo]`, `remove(id)`) in production code. Then the only untested surface is the adapter itself, which task 23's scheduled job covers. +- Either way, a test fails if production calls a `Client` method or keyword that the pinned library doesn't have. +- Add a test for `log_seed_ratios_via_http` that asserts the request body matches the Transmission RPC spec fields it relies on. diff --git a/tasks/21-scraper-contract-tests.md b/tasks/21-scraper-contract-tests.md new file mode 100644 index 0000000..d7e9829 --- /dev/null +++ b/tasks/21-scraper-contract-tests.md @@ -0,0 +1,24 @@ +# 21 — Recorded-page scraper contract tests (+ `fetch_kali_latest`) + +**Priority**: P2 · **Class**: NO_REFACTOR · **Depends on**: — + +## Problem +The likeliest production failure is a distro site changing its page layout, but each `Fetch*NamingTests` +feeds a hand-written HTML/text snippet of about 5 lines into the scraper. Those snippets encode what the +author *thinks* the page looks like, so drift can't be detected. Also, `fetch_kali_latest` +(`fetch_torrents.py:576–601`) has **no test at all**, and the error paths of +`fetch_devuan_latest`, `fetch_linuxmint_cinnamon`, `fetch_fedora_workstation` and +`fetch_arch_latest` are uncovered (`:660–760`). + +## Acceptance criteria +- `tests/fixtures/pages//.html|txt`: one **real** captured response per upstream endpoint each fetcher hits. Capture them with a small script (`scripts/record_fixtures.py`, run by hand or by task 23's job) that writes the fixture plus a `.meta.json` with the URL and capture date. +- A shared `fake_response(fixture_path_or_text, status=200)` helper, replacing the ~17 ad-hoc `FakeResponse` classes (shared with task 26). +- For each fetcher, a test that replays the recorded pages and asserts the **exact** candidate names and URLs expected for that snapshot. +- `fetch_kali_latest`: happy path plus "page has no recognizable version" → empty result, no exception. +- Each fetcher's failure paths: HTTP 404/500, a connection error, and a page with the expected element missing. Each returns empty and logs. This covers the uncovered `except` branches. +- No network access in the merge check (a socket guard like the one cd-test-architecture used is a good addition). + +## Notes +VCR.py/`responses` aren't needed; `patch.object(ft.requests, 'get', side_effect=url_router)` reading fixtures +by URL is enough and adds no dependency. The fixtures are only snapshots, so task 23 refreshes them and +compares against the live sites on a schedule. diff --git a/tasks/22-container-smoke-test.md b/tasks/22-container-smoke-test.md new file mode 100644 index 0000000..70a18cd --- /dev/null +++ b/tasks/22-container-smoke-test.md @@ -0,0 +1,20 @@ +# 22 — Container smoke test in the build job + +**Priority**: P2 · **Class**: CI · **Depends on**: — + +## Problem +The image is built and pushed without ever being started: +- `entrypoint.sh` (apk upgrade, the `RUN_AS_NON_ROOT` user/chown dance, `configure_transmission.py`, the daemon start, the daily loop) has no test. +- `Dockerfile` installs Python deps from apk, while CI's `test` job uses pip versions on Ubuntu. Version skew between the two is never exercised. + +## Acceptance criteria +In `.github/workflows/docker-image.yml` `build` job, after building and **before** push, a step that: +- Runs the built image with `--network none -e SKIP_PACKAGE_UPDATES=true` and temp volumes, waits for "Starting linux-iso-seeder container", and checks that `transmission-daemon` is running and `configure_transmission.py` exited 0. +- Does the same with `RUN_AS_NON_ROOT=true PUID=1234 PGID=1234`, and checks that `/config` is owned by 1234. +- Runs `docker run --rm --entrypoint python3 -c "import fetch_torrents, torrent_scrape, transmission_rpc, requests, bs4"` with `FETCH_TORRENTS_LOG_DIR=/tmp`, which catches apk/pip import skew. +- Optional: runs the unit suite inside the image (`python3 -m unittest discover -s tests`) to test against the real apk package versions. This requires copying `tests/` into a test stage, not the final image. +- Fails the job (no push) on any of these failing. Stays under about 60 s. + +## Notes +`--network none` keeps the check deterministic. The daily fetch loop will log network errors, which +is expected; the smoke test only asserts startup. diff --git a/tasks/23-scheduled-live-drift-job.md b/tasks/23-scheduled-live-drift-job.md new file mode 100644 index 0000000..77a4c44 --- /dev/null +++ b/tasks/23-scheduled-live-drift-job.md @@ -0,0 +1,20 @@ +# 23 — Scheduled live-drift job + +**Priority**: P3 · **Class**: CI · **Depends on**: 21, 22 + +## Problem +Tests in the merge check are deterministic by design, but the outside world they depend on keeps moving: +- distro sites change layout +- trackers change behavior +- `alpine:latest` + `apk upgrade` at container start change the runtime **after** deploy + +Today nothing checks the published image or the live sites until a user notices missing torrents. + +## Acceptance criteria +New workflow `.github/workflows/drift.yml`, run on `schedule` (weekly) + `workflow_dispatch`, and **not** part of the PR check: +- **Live scrapers:** run each fetcher against the real site and assert it returns ≥ 1 well-formed candidate (name matches the distro pattern, URL ends in `.torrent`, HEAD returns 200). On failure, re-record fixtures with task 21's script and upload them as an artifact for comparison. +- **Live tracker scrape:** run `scrape_torrent` on one current Ubuntu torrent and assert it returns integer seeders/leechers. +- **Real Transmission:** start the published `ghcr.io/squazz/linux-iso-seeder:latest` (with network), wait for RPC, and run a short script that uses `transmission_rpc.Client` to add a known small torrent from `/watch`, list it, and remove it. +- **Published image smoke:** task 22's checks against `:latest`, without `SKIP_PACKAGE_UPDATES`. +- Optional: a mutation run (`mutmut run` per task 16) that uploads its results, for trend tracking. +- A failure opens or updates a single GitHub issue labelled `drift` (via `actions/github-script`) and doesn't block anything. diff --git a/tasks/24-parser-gaps.md b/tasks/24-parser-gaps.md new file mode 100644 index 0000000..f655297 --- /dev/null +++ b/tasks/24-parser-gaps.md @@ -0,0 +1,16 @@ +# 24 — Parser gaps: bencode str, UDP parsers, `parse_version_type` + +**Priority**: P3 · **Class**: NO_REFACTOR · **Depends on**: 13 + +## Evidence (mutmut survivors / coverage) +- `torrent_scrape.py:89–90`: `bencode(str)` is never exercised. `:95`: str dict keys (`key_bytes`) are never exercised. `bencode` is at 73% killed. +- `parse_udp_connect_response` 60%, `_check_for_udp_error_action` 67%. Short-packet and error-action payload branches aren't asserted precisely. +- `ScrapeUdpTrackerTests` patches `os.urandom` to a constant (`tests/test_torrent_scrape.py:315`), so the connect and scrape transaction IDs are identical. A mixed-up ID check would still pass. +- `fetch_torrents.py:320`, `:327–329`, `:333` `parse_version_type`: the returned `type_` is never asserted directly, and no test uses two architectures (amd64/arm64) of the same release, so they could be grouped together. + +## Acceptance criteria +- `bencode('abc') == b'3:abc'`. Dicts with str keys encode identically to bytes keys and are sorted by byte value. Unsupported types raise. +- UDP: a connect response shorter than 16 bytes raises. An error-action packet raises `UdpTrackerError` containing the tracker's message. A response with the wrong action raises. Scrape responses shorter than 8+12 bytes raise. +- `ScrapeUdpTrackerTests` gives `urandom` two **different** IDs (`side_effect=[b'\x00\x00\x00\x01', b'\x00\x00\x00\x02']`). Add a test where the scrape response echoes the *connect* ID, which must raise. +- `parse_version_type` table test: for each distro pattern, assert `(version, type_)` exactly, including amd64 vs arm64 Debian/Ubuntu names producing **different** `type_` values. +- Re-running mutmut on these functions: ≥ 90%, with any remaining survivors documented as equivalent. diff --git a/tasks/25-configure-transmission-main.md b/tasks/25-configure-transmission-main.md new file mode 100644 index 0000000..fea7941 --- /dev/null +++ b/tasks/25-configure-transmission-main.md @@ -0,0 +1,18 @@ +# 25 — `configure_transmission.main` end-to-end test + +**Priority**: P3 · **Class**: NO_REFACTOR · **Depends on**: — + +## Problem +`configure_transmission.main()` (`configure_transmission.py:131–152`) is the only uncovered part of +the module. It runs at every container start, before the daemon, and connects the env to the builders +and to `load_settings`/`save_settings`. If it breaks, the container fails at startup. Its builders are +well tested (89.8% mutation). + +## Acceptance criteria +With `TRANSMISSION_SETTINGS_FILE` pointing at a temp path, under `patch.dict(os.environ, …, clear=True)`: +- No settings file + `TRANSMISSION_RPC_USERNAME`/`PASSWORD` set → a file is created that contains exactly the expected overrides, and `port-forwarding-enabled` is `false` (the first-run default from commit 40f36cd). +- Existing settings file with unrelated keys → those keys are preserved and only the overridden keys change. `port-forwarding-enabled` is **not** forced when the file already exists and `TRANSMISSION_PORT_FORWARDING` is unset. +- `TRANSMISSION_RPC_WHITELIST` set with no username/password → `WARNING: …` goes to stderr. Capture it with `contextlib.redirect_stderr` and assert the message. +- Invalid `TRANSMISSION_PEER_LIMIT_GLOBAL=abc` / `0` → silently dropped (no `peer-limit-global` key), and the file is still written. Pin this current behavior; whether it *should* warn is a separate product decision. +- Unreadable or corrupt existing settings file → "starting fresh" on stderr, and a valid file is written. +- Coverage of `configure_transmission.py` is 100% lines. diff --git a/tasks/26-assertions-and-fixtures.md b/tasks/26-assertions-and-fixtures.md new file mode 100644 index 0000000..9b6853d --- /dev/null +++ b/tasks/26-assertions-and-fixtures.md @@ -0,0 +1,22 @@ +# 26 — Tighten weak assertions and consolidate fixtures + +**Priority**: P3 · **Class**: NO_REFACTOR · **Depends on**: 10, 14 (do it after the test-adding tasks to avoid merge pain) + +## Problem (from the test-design report, `.dev-team-reports/test-design-2026-10-18.md`) +- **Presence-only assertions** where the exact value is known: + - `tests/test_fetch_torrents.py:504` (`assertTrue(results)` + `assertIn`), `:1119`, `:228–238` + - `tests/test_configure_transmission.py:165` +- **Timeout test** (`:1729–1737`) checks that the `timeout` kwarg exists, not its value. +- **"Logs a warning" test** (`:1985`) never asserts the warning. +- **RPC auth tests** (`:1687`, `:1695`, `:1704`) and the not-called guards (`:890`, `:1931`, `:1939`) assert only on mock calls. Keep them (they cover real branches), but assert the observable outcome too. +- **Duplication:** + - about 17 copy-pasted `FakeResponse` classes (`:498` and others) + - the load/save/round-trip persistence trio, written 4 times (ratio history, removed history, old-release check state, settings) +- **One 2016-line test module.** + +## Acceptance criteria +- Each assertion above is replaced with an exact-value assertion (`assertEqual` on the full dict/list, or the specific kwarg value), or with `assertLogs(...)` checking the message. +- A `tests/helpers.py` with `fake_response()`, `FakeTorrent`, `isolated_env()` (from task 12) and the `transmission_rpc` stub/import bootstrap. No `class FakeResponse` remains in the test files. +- The persistence trio is a single parameterized mixin or `subTest` loop, run over the four stores. +- `tests/test_fetch_torrents.py` is split by concern: `test_ratio_gate.py`, `test_fetchers.py`, `test_cleanup.py`, `test_persistence.py`, `test_logging.py`, `test_old_release_demand.py`. Each imports via `tests/helpers.py`. +- The test count is the same or higher, the suite is green, and the Farley Score doesn't drop. Re-run `/test-design`. diff --git a/tasks/27-remeasure-baseline.md b/tasks/27-remeasure-baseline.md new file mode 100644 index 0000000..12f9524 --- /dev/null +++ b/tasks/27-remeasure-baseline.md @@ -0,0 +1,11 @@ +# 27 — Re-measure and record the new baseline + +**Priority**: P3 · **Class**: — · **Depends on**: the other tasks (can run after any milestone) + +## Acceptance criteria +- Re-run `/dev-team:test-health` and compare against the targets in `tasks/README.md`: + - critical-function mutation ≥ 90% for `fetch_torrents.py` and `torrent_scrape.py` + - `fetch_torrents.py` branch coverage ≥ 90% + - ruff, mypy and container smoke in CI +- Set `coverage report --fail-under=` in CI (task 16's step), so coverage can only ratchet upward. +- Update the table in `tasks/README.md` with the new numbers and remove the finished task files, or mark them done. diff --git a/tasks/README.md b/tasks/README.md new file mode 100644 index 0000000..0ab5732 --- /dev/null +++ b/tasks/README.md @@ -0,0 +1,49 @@ +# Test-health remediation tasks + +Source: `.dev-team-reports/test-health-2026-10-18.md` (local, gitignored), plus the +`cd-test-architecture` and `test-design` reports it links to. + +Baseline at 84761d3: 229 tests green, 80% branch coverage, Farley 8.5. Critical-logic +mutation scores: `fetch_torrents.py` 70.9%, `torrent_scrape.py` 82.6%, `configure_transmission.py` 89.8%. + +Every task with testable behavior follows the CLAUDE.md rule: write the failing test first, +confirm it fails for the expected reason, implement, then run +`python -m unittest discover -s tests -v` until it is green. Line numbers are as of 84761d3, +so search for the function name if they have drifted. + +## Targets ("a better place") + +| Metric | Now | Target | +| --- | --- | --- | +| Mutation score, critical `fetch_torrents.py` functions | 70.9% | ≥ 90% | +| Mutation score, critical `torrent_scrape.py` functions | 82.6% | ≥ 90% | +| Branch coverage, `fetch_torrents.py` | 75% | ≥ 90% (once `run()` exists) | +| CI gates | unittest only | unittest + ruff + mypy + container smoke | +| Scraper drift detection | none | recorded pages in the merge check + a scheduled live job | + +## Order + +| # | Task | Priority | Class | Depends on | +| --- | --- | --- | --- | --- | +| 10 | [Ratio gate boundaries and branches](10-ratio-gate-boundaries.md) | P1 | NO_REFACTOR | — | +| 11 | [Test `get_previous_ratios` in-process](11-get-previous-ratios-tests.md) | P1 | NO_REFACTOR | — | +| 12 | [Isolate tests from env vars and the clock](12-test-env-clock-isolation.md) | P1 | NO_REFACTOR | — | +| 13 | [Fix `bdecode` accepting/mis-reporting truncated input](13-bdecode-truncated-input.md) | P1 (bug) | NO_REFACTOR | — | +| 14 | [Cleanup planning boundary tests](14-cleanup-boundaries.md) | P2 | NO_REFACTOR | 12 | +| 15 | [Add ruff + mypy to CI](15-ci-static-gates.md) | P2 | CI | — | +| 16 | [Dev tooling: coverage + mutmut in requirements-dev](16-dev-tooling-coverage-mutation.md) | P2 | CI | — | +| 17 | [Lock in daily-run behavior with a `runpy` test](17-daily-run-characterization.md) | P2 | NO_REFACTOR | 12 | +| 18 | [Extract `run()` and `configure_logging()`](18-extract-run-and-logging.md) | P2 | REFACTOR_REQUIRED | 17 | +| 19 | [Inject `today` / client / config into the `cleanup_*` functions](19-inject-cleanup-seams.md) | P2 | REFACTOR_REQUIRED | 17 | +| 20 | [Transmission client double that can't drift](20-transmission-double-fidelity.md) | P3 | REFACTOR_REQUIRED | 19 | +| 21 | [Recorded-page scraper contract tests (+ `fetch_kali_latest`)](21-scraper-contract-tests.md) | P2 | NO_REFACTOR | — | +| 22 | [Container smoke test in the build job](22-container-smoke-test.md) | P2 | CI | — | +| 23 | [Scheduled live-drift job](23-scheduled-live-drift-job.md) | P3 | CI | 21, 22 | +| 24 | [Parser gaps: bencode str, UDP parsers, `parse_version_type`](24-parser-gaps.md) | P3 | NO_REFACTOR | 13 | +| 25 | [`configure_transmission.main` end-to-end test](25-configure-transmission-main.md) | P3 | NO_REFACTOR | — | +| 26 | [Tighten weak assertions and consolidate fixtures](26-assertions-and-fixtures.md) | P3 | NO_REFACTOR | 10, 14 | +| 27 | [Re-measure and record the new baseline](27-remeasure-baseline.md) | P3 | — | all | + +Tasks 10–13 are independent and can run in parallel. Any task that touches +`tests/test_fetch_torrents.py` can conflict with task 26's file split, so do 26 after the +test-adding tasks rather than before.