Reproducibility: measured REPRODUCING.md, repeatable seeds, Docker on 3.13, README benchmark in every simulator

.github/dependabot.yml

@@ -61,3 +61,12 @@ updates:
       day: "monday"
     labels:
       - "dependencies"
+    ignore:
+      # The image's Python must be one pyproject.toml supports (requires-python);
+      # a minor bump such as 3.13 -> 3.14 broke that once (#810). Move it by hand
+      # together with requires-python and the CI matrix.
+      # test/repro/test_packaging_consistency.py checks the three agree.
+      - dependency-name: "python"
+        update-types:
+          - "version-update:semver-major"
+          - "version-update:semver-minor"

.github/workflows/docker.yml

@@ -0,0 +1,37 @@
+# Builds the Dockerfile and runs the reproducibility tests inside the image, so
+# the container is checked, not only described. Runs when the image's inputs change.
+name: Docker image
+
+on:
+  push:
+    branches: [ master ]
+    paths: [ "Dockerfile", "pyproject.toml", "poetry.lock", ".github/workflows/docker.yml" ]
+  pull_request:
+    branches: [ master ]
+    paths: [ "Dockerfile", "pyproject.toml", "poetry.lock", ".github/workflows/docker.yml" ]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  docker:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      # The image is about 7 GB (torch bundles its CUDA libraries); make room.
+      - name: Free runner disk space
+        run: sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
+      - name: Build image
+        run: docker build -t bindsnet .
+      # Runners have no GPU, and the image's CUDA torch build can segfault there while
+      # preloading CUDA (see .github/scripts/use_cpu_torch.sh), so the test container
+      # swaps in the CPU build of the same version; everything else is the image as built.
+      - name: Run reproducibility tests in the image
+        run: >-
+          docker run --rm bindsnet bash -c
+          ".github/scripts/use_cpu_torch.sh python && python -m pytest -q test/repro"

.github/workflows/publish.yml

@@ -24,7 +24,7 @@ jobs:
     - name: Check release tag matches pyproject version
       if: github.event_name == 'release'
       run: |
-        version=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['tool']['poetry']['version'])")
+        version=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
         tag="${GITHUB_REF_NAME#v}"
         if [ "$version" != "$tag" ]; then
           echo "Release tag $GITHUB_REF_NAME does not match pyproject version $version"

CHANGELOG.md

@@ -7,6 +7,41 @@ see the [GitHub releases / tags](https://github.com/BindsNET/bindsnet/releases).
 ## [Unreleased]
 
 ### Fixed
+- `--seed` now makes GPU runs repeatable in `eth_mnist.py`, `batch_eth_mnist.py`,
+  `supervised_mnist.py`, `conv_mnist.py`, `conv1d_MNIST.py`, `conv3d_MNIST.py`,
+  `SOM_LM-SNNs.py` and `breakout/random_network_baseline.py`: they call
+  `torch.manual_seed(seed)` unconditionally, which seeds the CPU generator (weight
+  initialisation, Poisson encoding, data shuffling) and every GPU. Before, the GPU path
+  called only `torch.cuda.manual_seed_all`; two runs of
+  `batch_eth_mnist.py --seed 0 --n_train 640 --n_test 320` gave 0.37 and 0.47.
+  `dot_tracing.py` now uses its `--seed` for torch and the dot simulator (default
+  seed 1, since the simulator reads 0 as "seed from the clock").
+- `supervised_mnist.py` plots input-to-excitatory weights through
+  `feature_index["weight"]` (`MulticompartmentConnection` has no `.w`), so it runs to
+  the end.
+- `reservoir.py` scores its readout on the MNIST test set (it reused the training
+  loader before) and reports the number of images it actually scored.
+- `conv_mnist.py` now tests the network: it labels output neurons from training spikes
+  (`assign_labels`) and reports all-activity and proportion-weighting test accuracy,
+  like `eth_mnist.py`.
+- `--plot` works as a switch in `eth_mnist.py`, `supervised_mnist.py`, `conv_mnist.py`,
+  `reservoir.py`, `conv1d_MNIST.py`, `conv3d_MNIST.py` and `SOM_LM-SNNs.py`: plotting is
+  off unless `--plot` is given (it was `store_true` with a default of `True`).
+  `conv_mnist.py` no longer pauses 1 s per sample.
+- Training and test loops in the MNIST examples run exactly `--n_train` / `--n_test`
+  samples (`>=` instead of `>`, which ran one more).
+- The Breakout examples run with the locked `ale-py` 0.12.1: `GymEnvironment` maps
+  `BreakoutDeterministic-v4` to its ale-py 0.8.1 definition, `Breakout-v4` with
+  `frameskip=4` (newer ale-py no longer register that id).
+- `play_breakout_from_ANN.py` loads `trained_shallow_ANN.pt` under PyTorch 2.6+: the
+  file is now a `state_dict` with identical tensors, loaded with `weights_only=True`
+  (it was a pickled `Net` object).
+- `publish.yml` reads the version from `[project]`, where it has lived since 0.3.4, so
+  a published GitHub release passes its tag check (it read `[tool.poetry]`; the last
+  PyPI upload was started by hand, which skips the check).
+- Dockerfile on `python:3.13-slim`, inside `requires-python = ">=3.11,<3.14"` (#810 had
+  moved it to 3.14); Dependabot no longer proposes minor or major Python bumps for the
+  image.
 - API reference on Read the Docs was empty: the build never installed `bindsnet`, so
   every `automodule` failed to import (`No module named 'matplotlib'`), and
   `docs/pyproject.toml` downgraded Sphinx to 7.2.6. `.readthedocs.yaml` now installs a
@@ -22,6 +57,26 @@ see the [GitHub releases / tags](https://github.com/BindsNET/bindsnet/releases).
   `environment.cue_reward`, `environment.dot_simulator` and
   `analysis.dotTrace_plotter`, which were missing.
 
+### Added
+- `test/repro/test_packaging_consistency.py`: checks that the Dockerfile's Python is in
+  `requires-python`, that CI tests every supported Python, that `CITATION.cff` and
+  `pyproject.toml` carry the same version, that `publish.yml` reads that version, and
+  that every example with `--seed` seeds torch unconditionally.
+- `.github/workflows/docker.yml`: builds the image and runs `test/repro` inside it when
+  the Dockerfile, `pyproject.toml` or `poetry.lock` change.
+- `examples/benchmark/bindsnet_scaling.py`: the README scaling benchmark for BindsNET
+  alone (no other simulator needed).
+- `examples/benchmark/benchmark.py` rewritten: every simulator (BindsNET CPU/GPU,
+  BRIAN2, brian2genn, PyNEST, ANNarchy CPU/GPU) gets the network the README describes,
+  each in its own process, and a simulator that is not installed is skipped. It
+  replaces the 2018 script, which imported an `experiments` helper package that was not
+  released and used its own input settings per simulator: BindsNET 0-1 Hz with U(0, 1)
+  weights, BRIAN2 15 Hz with U(0, 0.01) weights and no `on_pre` (so its synapses did not
+  drive the membrane), PyNEST 60 Hz connected one-to-one, ANNarchy 50 Hz with U(0, 1)
+  weights. `annarchy.py` and `gpu_annarchy.py` merged into it; `plot_benchmark.py` reads
+  its CSV.
+- `test/environment/test_gym_environment.py`.
+
 ### Changed
 - CI: one test workflow (`python-app.yml`: job `build` on Python 3.13 plus a 3.11/3.12
   matrix, Poetry 2.4.3 with dependency caching, superseded runs cancelled);

DATA.md

@@ -31,7 +31,9 @@ the library provides.
 
 ### Atari — Breakout (and Space Invaders)
 - **Loader:** `bindsnet.environment.GymEnvironment("BreakoutDeterministic-v4")`
-  (see `examples/breakout/*.py`).
+  (see `examples/breakout/*.py`). `GymEnvironment` maps this id to its definition,
+  `Breakout-v4` with `frameskip=4`, so it works with ale-py 0.9 and later, which
+  register Breakout under `Breakout-v4`.
 - **Upstream source:** Arcade Learning Environment via `gymnasium[atari]` + `ale-py`
   (declared in `pyproject.toml`). ROMs are provided through the ALE/AutoROM tooling.
 - **Obtained by:** the Gymnasium/ALE runtime; not stored in this repo.
@@ -49,14 +51,18 @@ the library provides.
 ## 2. Synthetic stimuli (no external dataset)
 
 ### Scaling-benchmark Poisson drive
-Used by `examples/benchmark/benchmark.py` and reported in the README "Benchmarking"
-section and Hazan et al. 2018:
+Described in the README "Benchmarking" section and Hazan et al. 2018:
 - Population of **n** Poisson input neurons, firing rates drawn from **U(0, 100) Hz**.
 - Connected all-to-all to an equally sized population of LIF neurons; connection
   weights sampled from **N(0, 1)**.
 - **n** varied 250 → 10,000 in steps of 250; each run simulated **1,000 ms** at
   **dt = 1.0 ms**.
 
+`examples/benchmark/benchmark.py` (every simulator) and
+`examples/benchmark/bindsnet_scaling.py` (BindsNET only) implement exactly this, since
+2026-10; the 2018 script used its own input settings for each simulator (see
+[REPRODUCING.md](REPRODUCING.md#scaling-benchmark)).
+
 This stimulus is generated programmatically; there is no dataset to download.
 
 ---

Dockerfile

@@ -4,7 +4,7 @@
 #   docker run --rm -it --gpus all bindsnet bash    # --gpus needs the NVIDIA container toolkit
 # The torch wheels from the cu130 index bundle their CUDA libraries, so no CUDA base
 # image is needed; the host only needs an NVIDIA driver.
-FROM python:3.14-slim
+FROM python:3.13-slim
 
 ARG DEPS=dev
 ARG POETRY_VERSION=2.4.3

README.md

@@ -114,10 +114,16 @@ preprocessing — is declared in [DATA.md](https://github.com/BindsNET/bindsnet/
 ## Reproducing results
 
 [REPRODUCING.md](https://github.com/BindsNET/bindsnet/blob/master/REPRODUCING.md) maps each shipped model and published claim to its
-model class, example script, exact command, seed, and expected output (e.g. the
+model class, example script, exact command, seed, and measured output (e.g. the
 Diehl & Cook 2015 MNIST replication via `examples/mnist/eth_mnist.py`, and the
 Hazan et al. 2018 scaling benchmark).
 
+Each BindsNET release is an archived snapshot. BindsNET is a research package that
+keeps gaining features and fixes, and results can also depend on the hardware, so
+results may differ between releases. A published result is reproduced with the
+release it was produced on; please name that release in the publication, ideally by
+its version DOI (see [Citing the software](#citing-the-software)).
+
 ## Background
 
 The simulation of biologically plausible spiking neuron dynamics can be challenging. It is typically done by solving ordinary differential equations (ODEs) which describe said dynamics. PyTorch does not explicitly support the solution of differential equations (as opposed to [`brian2`](https://github.com/brian-team/brian2), for example), but we can convert the ODEs defining the dynamics into difference equations and solve them at regular, short intervals (a `dt` on the order of 1 millisecond) as an approximation. Of course, under the hood, packages like `brian2` are doing the same thing. Doing this in [`PyTorch`](https://pytorch.org/) is exciting for a few reasons:
@@ -137,6 +143,8 @@ We simulated a network with a population of n Poisson input neurons with firing
 
 Several packages, including BRIAN and PyNEST, allow the setting of certain global preferences; e.g., the number of CPU threads, the number of OpenMP processes, etc. We chose these settings for our benchmark study in an attempt to maximize each library's speed, but note that BindsNET requires no setting of such options. Our approach, inheriting the computational model of PyTorch, appears to make the best use of the available hardware, and therefore makes it simple for practitioners to get the best performance from their system with the least effort.
 
+The figure above is the 2018 result. `examples/benchmark/benchmark.py` runs exactly the setup described above for BindsNET, BRIAN2, PyNEST and ANNarchy in one command (BindsNET alone: `examples/benchmark/bindsnet_scaling.py`), and [REPRODUCING.md](REPRODUCING.md#scaling-benchmark) lists the times measured with it. It replaces the script written in 2018, which used its own input settings for each simulator.
+
 <p align="middle">
 <img src="https://raw.githubusercontent.com/BindsNET/bindsnet/master/docs/BindsNET%20benchmark.png" alt="BindsNET%20Benchmark"  width="503" height="403">
 </p>