Traction consistency check, diff_results test-set marking, Python version matrix (6 issues)

.github/workflows/audits.yml

@@ -103,6 +103,38 @@ jobs:
           python "${{ matrix.audit }}/fair.py"
           echo "::endgroup::"
 
+  python-versions:
+    # The package declares requires-python >=3.10 (#848); this runs the fast suite on the oldest and
+    # newest supported interpreters so a change that only works on the version `profiler` uses is
+    # caught. A 3.9 run found real breakages, which is why the minimum is 3.10.
+    needs: changes
+    if: needs.changes.outputs.code_relevant == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.13"]
+    steps:
+      - uses: actions/checkout@v7
+
+      - uses: actions/setup-python@v7
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: 'pip'
+
+      - uses: actions/setup-node@v7
+        with:
+          node-version: '20'
+
+      - name: Install faircode + pytest
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[excel,parquet,proxy,mcp]" pytest
+
+      - name: Run the fast suite
+        run: python -m pytest tests/ -q -m "not slow"
+
   profiler:
     # Test + smoke-test the faircode dataset profiler (the diagnostic tool).
     # Not a required status check - safe to skip when nothing test-relevant

CHANGELOG.md

@@ -21,6 +21,10 @@ All notable changes to Fair Code are documented here, newest first.
 
 ### Added
 
+- `scripts/check_traction.py` (#1097), run by `make lint`: fails, naming the files and values, when the stars, forks, contributors, watching, social reach, countries or audits figures differ between the README, ROADMAP and METRICS, or when the README's "Show N completed items" differs from its `- [x]` lines. It compares the files with each other, so it works offline.
+- `scripts/diff_results.py` marks rows measured on different test sets (#1095): each row has a `sample` field (`same`, `different` or `unknown`, with `sample_sizes` when different), different rows are marked `~` in the text output and counted in `--summary`.
+- CONTRIBUTING.md lists the prerequisites for `make check` and a "Checks you may meet" table saying what each failing check means and how to fix it (#989).
+- CI runs the fast suite on Python 3.10 and 3.13 (#848).
 - Fourteen more explainers, each run against a real audit with the benchmark's current `results/` rows as the baseline where one exists: [Adversarial Debiasing](explainers/adversarial-debiasing.md), [Fairness Regularization](explainers/fairness-regularization.md) and [Fair Representation Learning](explainers/fair-representation-learning.md) (close #1083); [Measurement Bias](explainers/measurement-bias.md), [Aggregation Bias](explainers/aggregation-bias.md) and [Survivorship Bias](explainers/survivorship-bias.md) (close #1084); [Disaggregated Evaluation](explainers/disaggregated-evaluation.md), [Leave-One-Group-Out Evaluation](explainers/leave-one-group-out-evaluation.md) and [Choosing a Fairness Metric](explainers/choosing-a-fairness-metric.md) (close #1085); [Model Cards](explainers/model-cards.md), [Datasheets for Datasets](explainers/datasheets-for-datasets.md) and [Algorithmic Impact Assessments](explainers/algorithmic-impact-assessment.md) (close #1086); [Inferring Missing Protected Attributes](explainers/inferring-protected-attributes.md) and [Fairness Without Demographics](explainers/fairness-without-demographics.md) (close #1087). The explainer count goes from 68 to 82, with generated pages, OG images, sitemap entries, `llms.txt`, `llms-full.txt` and the MCP mirror.
 - `faircode.benchmark.load_audit(folder, strategy=None, random_state=None)` returns an `AuditData` prepared by the same `prepare_audit()` that `run_audit` uses, so explainers, notebooks and tests reproduce `results/` rows without copying the loader, encoding and split (closes #1078, SPEC section 11).
 - Browser tests (Chromium) that a 20,000-row file with about 18,000 distinct labels keeps the profile and compare flag lists at 51 items or fewer (closes #1080).
@@ -40,6 +44,7 @@ All notable changes to Fair Code are documented here, newest first.
 
 ### Changed
 
+- `requires-python` is now `>=3.10` and ruff targets `py310` (#848). A Python 3.9 CI run found real breakages (`X | None` evaluated at runtime, `Path.write_text(newline=)`), and the `mcp` extra needs 3.10 anyway.
 - The no-op `frozen-files.yml` workflow is removed; the repo ruleset no longer requires it (closes #1061).
 - The profile scaling test compares growth when the number of distinct labels quadruples instead of a wall-clock limit, so it no longer fails on a slow CI runner.
 - Per-group profile flags past 10 of the same kind in one dimension (or intersection) collapse into one summary flag, and the terminal, HTML and web views list at most 50 flags and then "and N more"; the full per-group detail stays in `--json` and the CSV (closes #1043, SPEC section 6).
@@ -50,7 +55,8 @@ All notable changes to Fair Code are documented here, newest first.
 
 ### Fixed
 
-- Profiling a file with a high-cardinality geography column was quadratic in `_build_flags` (a list membership test per group); it now uses a set, in Python and JavaScript. 60,000 rows with 45,000 zip codes profile in about 1.6 s instead of about 10 s (closes #1042).
+- `test_python_dash_m_faircode_runs_the_cli` only runs a `faircode` console script that sits next to the running interpreter, so a stray script on `PATH` installed for another Python no longer fails it (#1096).
+- `faircode/__main__.py` is marked `# pragma: no cover` with a pointer to the subprocess test, so the coverage figure no longer reports it as 0 percent 
- Profiling a file with a high-cardinality geography column was quadratic in `_build_flags` (a list membership test per group); it now uses a set, in Python and JavaScript. 60,000 rows with 45,000 zip codes profile in about 1.6 s instead of about 10 s (closes #1042).
 - The web engine labelled XLSX date cells with their Excel serial number (`43831`); they now read `2020-01-01 00:00:00` like pandas, computed from the serial so the label does not depend on the time zone, and time-only cells read `HH:MM:SS` (closes #1039).
 
 - `get_xlsx_sheet_info` now closes the workbook it opens to list sheet names, rather than leaking the file handle. The unclosed handle raised a `ResourceWarning` on garbage collection and, on Windows, kept the `.xlsx` locked against deletion or overwrite ([#1051](https://github.com/yakew7/Fair-Code/issues/1051)).

CONTRIBUTING.md

@@ -74,9 +74,26 @@ make build-explainers  # regenerate explainer pages, sitemap, and OG images (dar
 make fix-explainer-count  # correct any stale "N explainers" mention in README/CONTRIBUTORS/METRICS/ROADMAP after adding/removing an explainer
 make key-fold         # regenerate the JS case-folding table (assets/profiler-engine.js KEY_FOLD) after a Unicode upgrade
 make favicons          # regenerate favicon.ico, apple-touch-icon.png, icon-{192,512}.png after editing logo.svg
-make lint              # em-dash-free check + broken internal doc link check + ruff (unused imports, undefined names)
+make lint              # em-dash-free check + broken internal doc link check + issue-number check + explainer counts + traction figures + ruff
 ```
 
+Prerequisites for the full `make check`: Python 3.10 or newer, Node 18 or newer (`key-fold-check` and the JavaScript parity tests run the browser engines under `node`), and the GitHub CLI `gh` signed in (`scripts/check_issue_refs.py` asks GitHub whether an issue or PR number cited in the docs exists; it is skipped offline, and the lint workflow runs it as informational only).
+
+### Checks you may meet
+
+| If this fails | What it means | How to fix it |
+|---|---|---|
+| `check_em_dash.py` | A tracked file contains an em dash | Replace it with a hyphen, comma or colon |
+| `check_broken_links.py` | A relative link in the docs points at a missing file or heading | Fix the path or heading |
+| `check_issue_refs.py` | A `#NNN` in ROADMAP, CHANGELOG, CONTRIBUTORS or METRICS is not a real issue or PR | Correct the number |
+| `check_explainer_count.py` | An "N explainers" figure, or an explainer listing in the README or `llms.txt`, is stale | `make fix-explainer-count` (a missing listing has to be added by hand) |
+| `check_traction.py` | Stars, forks, contributors, watching, social reach, countries or audits differ between README, ROADMAP and METRICS, or the README's "Show N completed items" differs from its `- [x]` lines | Edit the figure in the file(s) the message names so they agree; the README's "External Contributors" is one below the total |
+| `key-fold-check` | The JavaScript case-folding table `KEY_FOLD` no longer matches Python | `make key-fold`, then commit `assets/profiler-engine.js` |
+| A test that runs a README command | Every `faircode ...` line in the README, CONTRIBUTING and SPEC code blocks is checked against the CLI's argument parser; one of them uses a flag or value that no longer exists | Fix the example or the CLI |
+| `check_generated_files_current.py` (CI) | Generated explainer pages, sitemap, OG images or `llms-full.txt` are out of date | `make build-explainers`, then commit the result |
+
+The tests run in any pytest import mode, including `--import-mode=importlib`: `pyproject.toml` puts the repo root and `tests/` on `pythonpath`. The helpers shared by the JavaScript parity tests (`tests/test_js_parity_*.py`) live in `tests/js_helpers.py`.
+
 Optionally install the git hooks so the checks run automatically:
 
 ```bash

Makefile

@@ -45,11 +45,12 @@ favicons:  ## Regenerate favicon.ico/PNGs and apple-touch-icon.png from logo.svg
 fix-explainer-count:  ## Correct stale "N explainers" mentions in README/CONTRIBUTORS/METRICS/ROADMAP
 	$(PY) scripts/check_explainer_count.py --fix
 
-lint:  ## Enforce the em-dash-free rule + check for broken doc links + issue numbers in docs (needs gh, skipped offline) + explainer counts and listings + ruff (mirrors the lint workflow)
+lint:  ## Enforce the em-dash-free rule + check for broken doc links + issue numbers in docs (needs gh, skipped offline) + explainer counts and listings + traction figures agreeing across README/ROADMAP/METRICS + ruff (mirrors the lint workflow)
 	$(PY) scripts/check_em_dash.py
 	$(PY) scripts/check_broken_links.py
 	$(PY) scripts/check_issue_refs.py
 	$(PY) scripts/check_explainer_count.py
+	$(PY) scripts/check_traction.py
 	ruff check faircode scripts tests
 
 audits:  ## Run every audit's unfair.py/fair.py + the CLI smoke tests (mirrors audits.yml's required run-audits job, plus profiler/benchmark-harness's CLI steps)

README.md

@@ -1022,6 +1022,9 @@ orientation" error pointing you at `split`, which round-trips
 unambiguously. `.xlsx` needs the optional `excel` extra, `.parquet` needs
 the optional `parquet` extra.
 
+Python 3.10 or newer is required (the code uses 3.10 features such as `X | None` annotations and
+`Path.write_text(newline=...)`, and the `mcp` extra needs 3.10 too). CI runs the suite on 3.10, 3.12 and 3.13.
+
 ```bash
 pip install -e .                                   # installs the faircode console script
 pip install -e ".[excel]"                          # + .xlsx support (openpyxl)
@@ -1255,7 +1258,7 @@ pip install -r requirements-lock.txt
 
 **For new audits and reruns:** Update `results/` as usual. Do not regenerate `paper/results-frozen/` or move the `v1.0-paper` tag as part of an ordinary audit. Run `scripts/freeze_paper_results.py` only when deliberately creating a new reference snapshot; it replaces the existing frozen snapshot and mirrors its CSV files into `faircode/_results_frozen/`.
 
-To see how far `results/` has moved from that snapshot, run `python3 scripts/diff_results.py`. It joins the two tables per audit, strategy, model, protected attribute and metric and prints the per-row change, flagging rows that moved by more than `--threshold` (default 0.01) or exist on one side only (`--json`, `--all`, `--kind` and `--fail-on-change` are also available; `--audit NAME` limits it to one or more audits and `--summary` prints one line per audit; it only reads).
+To see how far `results/` has moved from that snapshot, run `python3 scripts/diff_results.py`. It joins the two tables per audit, strategy, model, protected attribute and metric and prints the per-row change, flagging rows that moved by more than `--threshold` (default 0.01) or exist on one side only (`--json`, `--all`, `--kind` and `--fail-on-change` are also available; `--audit NAME` limits it to one or more audits and `--summary` prints one line per audit; each row has a `sample` field, `same`, `different` or `unknown`, and rows whose group sizes differ between the two files (so they were scored on different test rows) are marked `~` and counted; it only reads).
 
 To deliberately create a new reference snapshot, run:
 

faircode/SPEC.md

@@ -606,7 +606,11 @@ value empty on one side only), `unchanged`, `only_current` or `only_reference`.
 side only are always flagged. `--json` prints the structured diff, `--all` lists unchanged rows, and
 `--fail-on-change` exits 1 when anything is flagged. `--audit NAME` (repeatable) restricts the
 join to those audits, and `--summary` prints one line per audit and kind instead (rows compared,
-rows flagged, largest absolute delta); both work with `--json`. It only reads both directories.
+rows flagged, rows on different test sets, largest absolute delta); both work with `--json`. Each row also
+has a `sample` field: `same` when both files report the same group sizes (fairness: `n_disadvantaged` and
+`n_advantaged`; performance: `n`), `different` when they differ (the two runs were scored on different test
+rows, so the delta mixes a model change with a change in who was tested; `sample_sizes` holds both), and
+`unknown` when a size is missing or the row is on one side only. It only reads both directories.
 
 ### Loading an audit in code
 

faircode/__main__.py

@@ -2,5 +2,7 @@
 
 from .cli import main
 
-if __name__ == "__main__":
+# Covered by test_python_dash_m_faircode_runs_the_cli (#966), which runs it in a subprocess that
+# coverage does not measure, hence the pragma (#986).
+if __name__ == "__main__":  # pragma: no cover
     raise SystemExit(main())

pyproject.toml

@@ -7,7 +7,7 @@ name = "faircode"
 version = "2.3.0"
 description = "Diagnostic profiler that audits a tabular dataset for demographic representation."
 readme = "README.md"
-requires-python = ">=3.8"
+requires-python = ">=3.10"
 license = { text = "MIT" }
 authors = [{ name = "Yash Kewlani" }]
 keywords = ["fairness", "bias", "dataset", "profiling", "demographics", "audit"]
@@ -54,7 +54,7 @@ markers = [
 ]
 
 [tool.ruff]
-target-version = "py38"
+target-version = "py310"
 
 [tool.ruff.lint]
 # Real bugs only (unused imports, undefined names, redefinitions) - Pyflakes'