Repository navigation
Reproducibility: measured REPRODUCING.md, repeatable seeds, Docker on 3.13, README benchmark in every simulator - #811
Open
Hananel-Hazan wants to merge 8 commits into
Open
Hananel-Hazan wants to merge 8 commits into
Hananel-Hazan wants to merge 8 commits into
Conversation
…ython 3.13 - Examples seeded only the GPU generator on GPU runs, so --seed did not make runs repeatable; torch.manual_seed is now unconditional (checked by a test). - supervised_mnist crash, reservoir scored on training images, conv_mnist had no test phase, --plot could not be turned off, loops ran one extra sample. - Breakout: map removed BreakoutDeterministic-v4 id; ANN file re-saved as a state_dict and loaded with weights_only=True. - Dockerfile back on python:3.13-slim; Dependabot no longer bumps its minor version; CI builds the image and runs test/repro inside it. - Add BindsNET-only scaling benchmark script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… mismatch benchmark.py could not run (missing experiments package) and gave each simulator different input, none of them the README's U(0,100) Hz Poisson, all-to-all N(0,1) weights. Rewritten so BindsNET, BRIAN2, brian2genn, PyNEST and ANNarchy all get that network, one process per simulator (NEST 3.10 and ANNarchy 5.0 corrupt memory when loaded together). Run here for BindsNET, BRIAN2 2.9.0, NEST 3.10.0 and ANNarchy 5.0 CPU at n=250 and 500; brian2genn and ANNarchy GPU not run. DATA.md and README corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every Measured output cell was produced by its command at commit f392a65 (Threadripper 2990WX, RTX 2070): eth_mnist 0.75/0.76 (CPU), batch_eth_mnist 0.81/0.83 (GPU), supervised_mnist 0.64/0.64 (CPU), conv_mnist 0.12/0.14 (CPU, accuracy falls with training; adaptive threshold keeps rising), reservoir 53.00 % (CPU), Breakout mean reward 5.4 over 10 episodes, identical in two runs. Scaling benchmark times for BindsNET, BRIAN2, NEST and ANNarchy, with thread settings stated and not tuned. ANNarchy back on its default of one thread (all 64 threads made it slower here). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[tool.poetry] has no version since 0.3.4, so a release-triggered publish would fail its tag check with KeyError. Test added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Research package: results can change between releases and hardware, so a publication should name its release (version DOI). The 2018 benchmark script (6a1168d) did not implement the paper's described setup, so it cannot regenerate the figure; the long per-simulator table moves to CHANGELOG only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ignored folders (logs/) can hold git worktrees, whose duplicate test modules broke collection with 17 import-mismatch errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Same facts, each stated from what the code does now. Also corrects the benchmark.py docstring, which still said BRIAN2, NEST and ANNarchy had not been run, and two REPRODUCING.md phrases that went beyond the checks done. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The image's CUDA torch build segfaulted on the GPU-less runner while preloading CUDA (run 39191323281), the same cause as #807. The test container swaps in the CPU build of the same version first; the image is otherwise as built. Checked locally: 7 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the two items from the #765 re-score, and every REPRODUCING.md row now carries a measured result.
Docker
Dockerfileonpython:3.13-slim, insiderequires-python = ">=3.11,<3.14"(Bump python from 3.13-slim to 3.14-slim #810 had moved it to 3.14). Dependabot leaves the image's Python minor version to us.docker.ymlbuilds the image and runstest/reproinside it wheneverDockerfile,pyproject.tomlorpoetry.lockchange.Examples
--seednow makes GPU runs repeatable: 8 examples calltorch.manual_seedunconditionally, which also seeds the CPU generator used for weight init, Poisson encoding and shuffling (before, two runs of the same command gave 0.37 and 0.47). A test checks every example.supervised_mnist.pyruns to the end,reservoir.pyscores on the MNIST test set,conv_mnist.pyhas a test phase,--plotworks as a switch, loops run exactly--n_train/--n_testsamples,dot_tracing.pyuses its--seed.GymEnvironmentmapsBreakoutDeterministic-v4toBreakout-v4withframeskip=4), andtrained_shallow_ANN.ptis astate_dictwith identical tensors, loaded withweights_only=True.Benchmark
benchmark.pyruns the network the README describes in every simulator, one process each;bindsnet_scaling.pyruns BindsNET alone. Measured here with BindsNET, BRIAN2 2.9.0, NEST 3.10.0 and ANNarchy 5.0 (CPU); brian2genn and ANNarchy GPU are included and were not run. It replaces the 2018 script (6a1168d), which used its own input settings per simulator.Docs
f392a65(Threadripper 2990WX, RTX 2070), with device and time; raw benchmark CSVs inexamples/benchmark/results_2026-10/.CI
publish.ymlreads the version from[project], so a published GitHub release passes its tag check (tested).pyproject.toml:testpaths = ["test"].Tests: 199 passed locally; new:
test/repro/test_packaging_consistency.py,test/environment/test_gym_environment.py.Next: REPRODUCING.md traces
conv_mnist's declining accuracy to the adaptive threshold inDiehlAndCookNodes; a fix is on branchdc-one-spike-thetaand being measured for 0.3.5.🤖 Generated with Claude Code