Skip to content

test: Add pytest infrastructure and regression tests for the sklearn Ridge decoder - #6

Open
KenyaOtsuka wants to merge 2 commits into
KamitaniLab:devfrom
KenyaOtsuka:test/sklearn-ridge-decoder-regression-tests
Open

test: Add pytest infrastructure and regression tests for the sklearn Ridge decoder#6
KenyaOtsuka wants to merge 2 commits into
KamitaniLab:devfrom
KenyaOtsuka:test/sklearn-ridge-decoder-regression-tests

Conversation

@KenyaOtsuka

Copy link
Copy Markdown

Why

The repository had no automated tests, making changes to the sklearn Ridge
decoder difficult to validate.

This PR adds a regression test suite that pins the current behavior before a
follow-up refactor of the decoder. The decoding implementation itself is
unchanged.

What

  • Add the missing scikit-learn dependency and pytest development setup.
  • Add GitHub Actions to run the test suite on PRs and pushes.
  • Add synthetic fixtures covering repeated stimuli, multi-file subjects,
    multiple ROIs, and 2-D / 3-D feature layers.
  • Add golden outputs generated from the current implementation.
  • Add 32 tests covering training, prediction, normalization, label alignment,
    chunking / Fortran-order handling, output layout, and edge cases.
Details for reviewers
  • pytest uses pythonpath = ["."] so the repository's top-level analysis
    modules are importable without modifying sys.path.
  • Synthetic data uses unbalanced repeat counts so per-trial and per-stimulus
    statistics are distinguishable.
  • Training data is split across two .h5 files to exercise the existing
    multi-BData paths.
  • The reference decoder independently reproduces the current numpy /
    scikit-learn Ridge semantics, including normalization, label expansion,
    chunking, order='F' reshaping, and float32 behavior.
  • Golden comparisons use a tolerance because float32 Ridge results can differ
    slightly across BLAS implementations.
  • The chunk-order test uses position-dependent perturbations so an incorrect
    C-vs-Fortran reshape cannot pass accidentally.
  • The current failure on zero-variance feature units is explicitly pinned.
  • The decoder artifact layout is tested because evaluation.py reads
    normalization files from that layout.
  • Regenerate goldens only with python -m tests.generate_golden, and only when
    the expected numerical output intentionally changes.
  • Each test runs in its own working directory because the scripts create a
    relative DistComp SQLite database under ./tmp/.

Test

uv sync --group dev
uv run pytest  # 32 passed

…Ridge decoder

The repository had no automated tests, which made it unsafe to change the
decoding implementation. This adds the first test suite, pinning the current
behavior of `train_decoder_sklearn_ridge.py` and `predict_feature.py`.

- Add `scikit-learn` to the project dependencies. It is imported by
  `train_decoder_sklearn_ridge.py` but was never declared, so `uv sync`
  produced an environment that could not run the script.
- Add a `dev` dependency group with pytest and `[tool.pytest.ini_options]`,
  including `pythonpath = ["."]` so the top-level analysis modules are
  importable from the tests without any `sys.path` manipulation.
- `tests/helpers/synthetic.py`: build tiny `BData` / `Features` datasets on
  disk, with repeated stimuli (unbalanced repeats), two ROIs, a subject split
  over two `.h5` files, and both a 2-D `fc`-like and a 3-D `conv`-like feature
  layer so the chunking / `order='F'` path is exercised.
- `tests/helpers/legacy_ridge.py`: a plain numpy + scikit-learn reference
  implementation of the direct brain -> feature Ridge decoder, mirroring bdpy's
  `ModelTraining` / `ModelTest` semantics exactly.
- `tests/generate_golden.py` and `tests/data/golden/`: regression fixtures
  recording the pipeline output on a fixed-seed synthetic dataset. Run it as
  `python -m tests.generate_golden`.
- Tests covering the artifact contract, decoded-feature output format,
  normalization parameters, label alignment, chunking, and the behavior of
  feature units with zero training variance.
- Add a GitHub Actions workflow running the suite on push and pull requests.

No change to the decoding implementation.
@KenyaOtsuka
KenyaOtsuka changed the base branch from main to dev September 9, 2026 08:31
@KenyaOtsuka
KenyaOtsuka marked this pull request as ready for review September 9, 2026 14:24
@KenyaOtsuka
KenyaOtsuka requested review from HirokiYasuda03 and ganow and a lite review from Copilot September 9, 2026 14:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A couple of tests load .npz goldens via np.load(...) without closing the returned NpzFile, which can leak file descriptors across the test run.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a first automated test/CI baseline for the scikit-learn Ridge decoding pipeline, including synthetic fixtures and golden regression outputs, so future refactors can be validated without changing the current decoder behavior.

Changes:

  • Add pytest + scikit-learn dependencies and configure pytest import behavior via pythonpath.
  • Introduce a synthetic on-disk dataset generator plus a reference (legacy) Ridge implementation used to pin current semantics.
  • Add GitHub Actions workflow to run the test suite on pushes and pull requests.
File summaries
File Description
uv.lock Locks new runtime/dev dependencies (scikit-learn, pytest and transitive deps).
pyproject.toml Adds scikit-learn, defines dev dependency group, and sets pytest configuration.
.github/workflows/test.yml Runs uv sync --group dev and pytest on CI (Python 3.11).
README.md Documents how to run tests and how to regenerate golden fixtures.
.gitignore Ignores Python bytecode caches and pytest cache.
tests/conftest.py Adds shared fixtures and ensures each test runs in an isolated working directory.
tests/generate_golden.py Script to regenerate the committed golden .npz outputs.
tests/helpers/synthetic.py Generates small synthetic BData + Features trees to exercise decoder edge cases.
tests/helpers/pipeline.py Calls the real training/prediction scripts and provides helpers to read outputs.
tests/helpers/legacy_ridge.py Reference numpy/sklearn implementation to match current scripts and support regression tests.
tests/test_pipeline_golden.py End-to-end golden regression test for predictions + normalization artifacts.
tests/test_legacy_reference.py Validates the reference implementation against scripts and goldens.
tests/test_predict_feature.py Pins decoded-feature output layout and behavior for predict_feature.
tests/test_train_decoder_sklearn_ridge.py Pins decoder artifact layout, normalization definitions, and rerun-skipping behavior.
tests/test_chunking.py Pins chunking behavior and Fortran-order reshape semantics.
tests/test_label_alignment.py Pins label alignment and feature expansion semantics used during training.
tests/test_zero_variance_features.py Pins current failure mode for zero-variance feature units.
Review details
  • Files reviewed: 15/20 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/test_legacy_reference.py Outdated
Comment thread tests/test_pipeline_golden.py Outdated
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants