Repository navigation
Conversation
There was a problem hiding this comment.
I didn't rerun anything. Everything below comes from the code, the CI logs and the two HF cards.
Are the bugs fixed?
Yes, all three from #271.
- Objective direction. v1 minimizes both gain error and relative ripple, as we agreed in #271.
v1.py, the docs, the dataset card and the shard manifest all use the same formulas. - Absolute value in v0. The docs now say the v0 labels are absolute values. v1 keeps the signed mean voltage and gain.
- Simulator provenance. Every v1 row records the commit, netlist, simulator, platform and container. I checked the recipe and netlist hashes on the card against
64e6107and they match. Only docs, tests and the ARM warning changed after that commit, so the published data comes from the code in this PR.
Of Mark's three asks on #271, the docs notice is done. The runtime warning only covers ARM, see 3 below. The v0 card on HF hasn't been touched and still has one line of text.
Changes I'd like before merging
A new user imports PowerElectronics, reads the docs page, and installs ngspice. That path is unchanged by this PR, and nothing on it mentions v1. Most of the points below follow from that.
-
The default import is still v0. Beams2D and Photonics2D both export v1 by default. Here the plain import and the problem registry both give v0, which maximizes ripple. So anything that walks the registry, EngiOpt included, optimizes the wrong way. Keeping v0 as the default protects existing v0 results. The cost is that every new user starts on the wrong objective without being told. The minimum fix is a one-time warning when v0 is created. It should say the objectives are historical and give the v1 import. The alternative is making v1 the default.
-
v1 doesn't check which ngspice it's running. Only the dataset generator does. Following the README's
apt-get install ngspiceon Ubuntu gives you ngspice 42. Your v1simulate()results will then differ from the v1 labels, with no warning. The docs line "Dataset generation rejects a different ngspice version..." also reads like it applies to the problem class. The canonical version and hashes are already constants in the generator. Could you move them intov1.pyand warn when the binary doesn't match? -
The ARM warning is written for v1 but also fires for v0. It tells ARM users to switch to the x86 runtime. For v0, your own comparison says ARM is probably closer to the published labels. And x86 users running v0 get no warning, even though the docs say their results probably won't match. The message should depend on the problem version.
-
v1 loads the dataset's main branch, not
v1.0.0. The card tells people to pin the tag, but the problem class loadsdataset_idwith no revision. So later pushes to main would change what v1 loads. The class should pin the tag itself. -
The warning at the top of the docs page is written for maintainers. It says to keep v0 immutable and record checksums. Someone new still won't know what to do. I'd change it to: use v1 for new work, here's the import, and v0 keeps its old objectives for reproducibility.
-
The Simulator section lost the install instructions. It's now about container digests, SBOMs and OCI referrers, which only matter if you're regenerating the dataset. Could you split it? One short part would say which ngspice to install and what to expect on each platform. A separate "Reproducing the v1 dataset" part would hold the container and generation details.
-
The status values aren't listed anywhere. The code can write
ok,invalid_measurements,undefined_relative_voltage_rippleandsimulation_error. The card only mentions two. Users need to know which rows to mask from a loss. A table with each status and which fields are null answers that. -
The v0 card on HF. Most people find the dataset through HF, so that card is where the provenance caveats need to be. Please add the notes from the docs page and a link to v1. This changes no labels.
Tests
-
v1 isn't in the shared problem tests.
test_problem_implementations.pyloops over the registry, which only has v0. So v1 never gets the contract checks the other problems get. Please add it to that test explicitly. -
Nothing runs the real v1 netlist through ngspice. The new v1 tests use a fake simulator, so nothing checks that the extra
print Vo_mean, Vpp, ...line parses from a real log. One test would do it. It should simulateVALID_DESIGNwith the CI ngspice and check every field is finite.
Scope
All ten are small and in code this PR touches. I'd leave the bigger #271 items for a follow-up issue: recording the backend in checkpoints, and a conformance set on the canonical backend in CI.
Description
This draft introduces an explicit PowerElectronics v1 path without changing the published v0 dataset or the package-level v0 default.
The intended v1 benchmark:
abs(dc_gain - 0.25)andVpp / abs(mean_voltage);The generator pins
IDEALLab/power_electronics_v0at revision5c4adb2ec5cfc71794988b1297a7ff8ffe59daa5, refuses dirty code, uncontainerized execution, noncanonical backend fingerprints, and existing output files by default, and writes each shard plus its manifest atomically. It contains no upload or overwrite path.The exact original v0 runtime remains unknown. The archived notebook is evidence for Windows/x86_64 and ngspice 36, while a small ngspice 44.2 reproduction on an ARM64 Mac is much closer to the published labels than the matching Linux x86_64 reproduction. Because the operating system, ngspice version, compiler, and build also differ, that comparison is evidence consistent with ARM but not proof of the original architecture. The documentation retains both facts rather than asserting either architecture as certain.
Frozen backend
40816f203b7e1c68ae37f4d9353bd302486d77988b98733c021f1ff71f48ae0211a4334ee90509f5edfdceef541711a34a1943d26a14cf0928ac8d5947b723745c4adb2ec5cfc71794988b1297a7ff8ffe59daa564e6107d44349e67e031cad36633c1af26817784The SIF, its dependency freeze, source bundle, pilot outputs, manifests, scripts, and logs are preserved on backed-up Euler group work storage rather than purgeable scratch.
Euler pilot results
The final pilot selected 100 evenly spaced source indices from each split and was run twice from isolated directories:
ok24e4eed443da4234bb576a722baa70ddd33acd339790f664926a2b9a16820909ok, 1invalid_measurements(source_index=223)bb677633e4548a1e9ea655f1b6b646ad14225603b79f19933149293af62239abok679c43afbf033a77add527613c6981ba57cc37a69c1642fb1d5b7fcc1a546482Both runs produced byte-identical JSONL hashes for all three splits. The same validation row was invalid in both runs, so the non-finite simulator measurement is preserved deterministically rather than dropped or replaced. A strict audit verified all selected indices, row counts, formulas, checksums, source revision, clean code commit, canonical image/binary fingerprints, statuses, and JSON finiteness.
Full Euler generation
After the repeatable pilot, the same frozen commit and image generated eight deterministic shards per split in Euler array job
14690255. All 24 tasks completed with exit code0:0. Each task used an isolated work/runtime directory and an offline copy of the pinned v0 cache; no Hugging Face token or upload path was present.The independent full-run audit passed with all 13,824 source indices present exactly once and all formulas, manifests, output hashes, source revisions, code commit, and backend fingerprints verified:
ok; 2invalid_measurements(indices 4,309 and 5,960)ok; 1invalid_measurements(index 223)okThe SHA-256 of the audit report is
6a489f82b2a99cadf0c1d3e4e8b53202bc2528ff2ac22a8653d06a24e153bf46. The full shards, manifests, audit report, Slurm/audit scripts, and logs are preserved with the SIF on backed-up Euler group work storage.Published dataset
The audited dataset is published at
IDEALLab/power_electronics_v1:v1.0.0eefad7d727ea1e5bef5e1b7088dea20a9b0cd67f68f3c8f9afa380490692f1a4417cb0773b6f0b1aed313ab138cc429b750291cfv1.0.0both reproduced the exact split counts, row order, statuses, invalid indices, and per-split semantic hashesThe three invalid simulator measurements remain as rows with status/validity fields and null raw/objective values. Consumers should retain those rows and mask their objective loss. The original
IDEALLab/power_electronics_v0repository and labels were not modified.Related to #271 and #276. This draft intentionally does not close either issue automatically; checkpoint/leaderboard integration remains out of scope.
Type of change
Checklist
pre-commitchecks withpre-commit run --all-filesruff check .andruff format --check .mypy engibenchReviewer Checklist