Skip to content

feat(power-electronics): add reproducible v1 backend - #279

Draft
SoheylM wants to merge 11 commits into
mainfrom
feat/power-electronics-v1-provenance
Draft

SoheylM wants to merge 11 commits into
mainfrom
feat/power-electronics-v1-provenance

Conversation

@SoheylM

@SoheylM SoheylM commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

This draft introduces an explicit PowerElectronics v1 path without changing the published v0 dataset or the package-level v0 default.

The intended v1 benchmark:

  • reuses the exact v0 input designs and train/validation/test membership;
  • does not claim to reproduce the v0 labels;
  • runs ngspice 44.2 in one frozen Linux x86_64 Apptainer image;
  • minimizes abs(dc_gain - 0.25) and Vpp / abs(mean_voltage);
  • retains signed mean voltage and signed gain, peak-to-peak voltage, validity/status fields, and exact source/code/simulator/container provenance.

The generator pins IDEALLab/power_electronics_v0 at revision 5c4adb2ec5cfc71794988b1297a7ff8ffe59daa5, refuses dirty code, uncontainerized execution, noncanonical backend fingerprints, and existing output files by default, and writes each shard plus its manifest atomically. It contains no upload or overwrite path.

The exact original v0 runtime remains unknown. The archived notebook is evidence for Windows/x86_64 and ngspice 36, while a small ngspice 44.2 reproduction on an ARM64 Mac is much closer to the published labels than the matching Linux x86_64 reproduction. Because the operating system, ngspice version, compiler, and build also differ, that comparison is evidence consistent with ARM but not proof of the original architecture. The documentation retains both facts rather than asserting either architecture as certain.

Frozen backend

  • image SHA-256: 40816f203b7e1c68ae37f4d9353bd302486d77988b98733c021f1ff71f48ae02
  • ngspice binary SHA-256: 11a4334ee90509f5edfdceef541711a34a1943d26a14cf0928ac8d5947b72374
  • ngspice version/platform: 44.2, Linux x86_64
  • source dataset revision: 5c4adb2ec5cfc71794988b1297a7ff8ffe59daa5
  • pilot EngiBench commit: 64e6107d44349e67e031cad36633c1af26817784

The SIF, its dependency freeze, source bundle, pilot outputs, manifests, scripts, and logs are preserved on backed-up Euler group work storage rather than purgeable scratch.

Euler pilot results

The final pilot selected 100 evenly spaced source indices from each split and was run twice from isolated directories:

Split Result Signed negative means JSONL SHA-256
train 100 ok 52 24e4eed443da4234bb576a722baa70ddd33acd339790f664926a2b9a16820909
val 99 ok, 1 invalid_measurements (source_index=223) 44 bb677633e4548a1e9ea655f1b6b646ad14225603b79f19933149293af62239ab
test 100 ok 47 679c43afbf033a77add527613c6981ba57cc37a69c1642fb1d5b7fcc1a546482

Both runs produced byte-identical JSONL hashes for all three splits. The same validation row was invalid in both runs, so the non-finite simulator measurement is preserved deterministically rather than dropped or replaced. A strict audit verified all selected indices, row counts, formulas, checksums, source revision, clean code commit, canonical image/binary fingerprints, statuses, and JSON finiteness.

Full Euler generation

After the repeatable pilot, the same frozen commit and image generated eight deterministic shards per split in Euler array job 14690255. All 24 tasks completed with exit code 0:0. Each task used an isolated work/runtime directory and an offline copy of the pinned v0 cache; no Hugging Face token or upload path was present.

The independent full-run audit passed with all 13,824 source indices present exactly once and all formulas, manifests, output hashes, source revisions, code commit, and backend fingerprints verified:

Split Rows Statuses Signed negative means
train 9,676 9,674 ok; 2 invalid_measurements (indices 4,309 and 5,960) 3,992
val 2,765 2,764 ok; 1 invalid_measurements (index 223) 1,168
test 1,383 1,383 ok 579

The SHA-256 of the audit report is 6a489f82b2a99cadf0c1d3e4e8b53202bc2528ff2ac22a8653d06a24e153bf46. The full shards, manifests, audit report, Slurm/audit scripts, and logs are preserved with the SIF on backed-up Euler group work storage.

Published dataset

The audited dataset is published at IDEALLab/power_electronics_v1:

  • immutable release tag: v1.0.0
  • publication commit: eefad7d727ea1e5bef5e1b7088dea20a9b0cd67f
  • 13,824 rows in the original split membership; no row was removed, clipped, or imputed
  • the published SHA-256 manifest validates all 56 staged content files; the manifest itself has SHA-256 68f3c8f9afa380490692f1a4417cb0773b6f0b1aed313ab138cc429b750291cf
  • an authenticated read-back at the publication commit and an unauthenticated clean-cache read-back through v1.0.0 both reproduced the exact split counts, row order, statuses, invalid indices, and per-split semantic hashes

The three invalid simulator measurements remain as rows with status/validity fields and null raw/objective values. Consumers should retain those rows and mask their objective loss. The original IDEALLab/power_electronics_v0 repository and labels were not modified.

Related to #271 and #276. This draft intentionally does not close either issue automatically; checkpoint/leaderboard integration remains out of scope.

Type of change

  • Documentation only change (no code changed)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist

  • I have run the pre-commit checks with pre-commit run --all-files
  • I have run ruff check . and ruff format --check .
  • I have run mypy engibench
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • The 37 focused PowerElectronics/ngspice tests pass locally
  • The documentation builds; its four warnings are pre-existing and outside this change
  • The full cross-problem suite passes locally. The sandbox run reached 183 passed and 12 skipped; 16 dataset-dependent tests could not access/write the external Hugging Face cache. CI is left to validate those tests in its normal environment.
  • GitHub CI passes on Linux Python 3.10-3.13, macOS Python 3.13, and Windows Python 3.13, along with build, docs, doctests, lint, mypy, and pre-commit.

Reviewer Checklist

  • The content of this PR brings value to the community. It is not too specific to a particular use case.
  • The tests and checks pass (linting, formatting, type checking).
  • The documentation is updated.
  • The code is understandable and commented.
  • There is no merge conflict.
  • Existing v0 results remain unchanged, and the changed semantics are isolated behind problem version v1.
  • The Euler pilot and frozen backend provide an adequate basis for the full dataset run.

@mkeeler43 mkeeler43 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't rerun anything. Everything below comes from the code, the CI logs and the two HF cards.

Are the bugs fixed?

Yes, all three from #271.

  • Objective direction. v1 minimizes both gain error and relative ripple, as we agreed in #271. v1.py, the docs, the dataset card and the shard manifest all use the same formulas.
  • Absolute value in v0. The docs now say the v0 labels are absolute values. v1 keeps the signed mean voltage and gain.
  • Simulator provenance. Every v1 row records the commit, netlist, simulator, platform and container. I checked the recipe and netlist hashes on the card against 64e6107 and they match. Only docs, tests and the ARM warning changed after that commit, so the published data comes from the code in this PR.

Of Mark's three asks on #271, the docs notice is done. The runtime warning only covers ARM, see 3 below. The v0 card on HF hasn't been touched and still has one line of text.

Changes I'd like before merging

A new user imports PowerElectronics, reads the docs page, and installs ngspice. That path is unchanged by this PR, and nothing on it mentions v1. Most of the points below follow from that.

  1. The default import is still v0. Beams2D and Photonics2D both export v1 by default. Here the plain import and the problem registry both give v0, which maximizes ripple. So anything that walks the registry, EngiOpt included, optimizes the wrong way. Keeping v0 as the default protects existing v0 results. The cost is that every new user starts on the wrong objective without being told. The minimum fix is a one-time warning when v0 is created. It should say the objectives are historical and give the v1 import. The alternative is making v1 the default.

  2. v1 doesn't check which ngspice it's running. Only the dataset generator does. Following the README's apt-get install ngspice on Ubuntu gives you ngspice 42. Your v1 simulate() results will then differ from the v1 labels, with no warning. The docs line "Dataset generation rejects a different ngspice version..." also reads like it applies to the problem class. The canonical version and hashes are already constants in the generator. Could you move them into v1.py and warn when the binary doesn't match?

  3. The ARM warning is written for v1 but also fires for v0. It tells ARM users to switch to the x86 runtime. For v0, your own comparison says ARM is probably closer to the published labels. And x86 users running v0 get no warning, even though the docs say their results probably won't match. The message should depend on the problem version.

  4. v1 loads the dataset's main branch, not v1.0.0. The card tells people to pin the tag, but the problem class loads dataset_id with no revision. So later pushes to main would change what v1 loads. The class should pin the tag itself.

  5. The warning at the top of the docs page is written for maintainers. It says to keep v0 immutable and record checksums. Someone new still won't know what to do. I'd change it to: use v1 for new work, here's the import, and v0 keeps its old objectives for reproducibility.

  6. The Simulator section lost the install instructions. It's now about container digests, SBOMs and OCI referrers, which only matter if you're regenerating the dataset. Could you split it? One short part would say which ngspice to install and what to expect on each platform. A separate "Reproducing the v1 dataset" part would hold the container and generation details.

  7. The status values aren't listed anywhere. The code can write ok, invalid_measurements, undefined_relative_voltage_ripple and simulation_error. The card only mentions two. Users need to know which rows to mask from a loss. A table with each status and which fields are null answers that.

  8. The v0 card on HF. Most people find the dataset through HF, so that card is where the provenance caveats need to be. Please add the notes from the docs page and a link to v1. This changes no labels.

Tests

  1. v1 isn't in the shared problem tests. test_problem_implementations.py loops over the registry, which only has v0. So v1 never gets the contract checks the other problems get. Please add it to that test explicitly.

  2. Nothing runs the real v1 netlist through ngspice. The new v1 tests use a fake simulator, so nothing checks that the extra print Vo_mean, Vpp, ... line parses from a real log. One test would do it. It should simulate VALID_DESIGN with the CI ngspice and check every field is finite.

Scope

All ten are small and in code this PR touches. I'd leave the bigger #271 items for a follow-up issue: recording the backend in checkpoints, and a conformance set on the canonical backend in CI.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants