Skip to content

Python core: split simulation.py and schema.py, one route assembly, shared identity, durability and validation helpers - #768

Draft
jtravs wants to merge 47 commits into
feature/two-backends-binary64from
refactor/python-core
Draft

Python core: split simulation.py and schema.py, one route assembly, shared identity, durability and validation helpers#768
jtravs wants to merge 47 commits into
feature/two-backends-binary64from
refactor/python-core

Conversation

@jtravs

@jtravs jtravs commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Restructures the Python core for organisation and fewer duplicated paths, with no loss of functionality. Stacked on #767.

What changes

Driver

  • driver/simulation.py (10.5k lines) is split by concern into settings, inputs, metadata, resume, runner and tdse_run, plus routes/{hollow,general,spatial,gnlse,materials,keywords}. simulation.py keeps the Simulation class and is now 3.7k lines. The split is a pure move, checked by comparing the abstract syntax trees with docstrings stripped: 253 definitions before and after.
  • Keyword routes:
    • _from_hollow and _from_general take one request dataclass.
    • One keyword_settings replaces three SimulationSettings( builds.
    • SimulationSettings.with_boundaries replaces the three _bounded(_watched(_with_boundary( chains.
    • Each prop_* keeps a typed signature equal to its from_* plus the run controls. Its body and Parameters section are derived from from_*, and test_route_keywords pins both.
  • from_radial/prop_radial are folded into from_free/prop_free, which dispatch on the grid type (breaking).
  • One defaults table in driver/defaults.py, which recipe/metadata.py now reads instead of re-implementing it.
  • driver/service.py is renamed driver/session_loop.py.

Output, service and statistics

  • output/schema.py is split into serialisation, provenance, records, metadata and store (pure move; breaking for anyone importing solprop.output.schema).
  • New solprop/_identity.py holds one canonical-digest helper for every stored identity, pinned by golden-digest tests.
  • New solprop/_durable.py holds one atomic file write, replacing three copies.
  • One Slurm batch_script and one table of thread variables.
  • service/_payload.py and _curves.py remove the duplicated helpers and the cross-module private imports.
  • StatDefinition now carries what about 15 parallel name tables held, and the dead implemented flag is removed.
  • RunMetadata JSON is field-driven; the stored format is unchanged.

Physics helpers

  • New solprop/_validation.py holds positive, finite, choice and shape with one wording; matching tests are updated to the same specificity.
  • tools.py becomes a tools/ package (pure move; public names unchanged).
  • Shared bases are verified bit-identical:
    • the rectangular and slab modes fold into _WallPairFill;
    • ZProfile gains default members;
    • DataPulse and TimeDataPulse share a private measured-pulse base.

Behaviour changes (each has a changelog fragment)

  • Resume: resume identities change once, because records name their new modules (ADR-077 already breaks resume identity in this release).
  • Radial runs: from_radial/prop_radial are removed; use from_free/prop_free with a RadialGrid.
  • Refusal wording: the refusal wording for positive, finite and choice arguments is uniform.
  • Slurm: Slurm scripts now also set the OpenBLAS idle default.

Not done

  • TDSE as a run extension. The tdse_binding threading is untouched and needs its own pass.
  • The log.py protocol reduction, which would remove documented public protocols.
  • pressure_zdw routed through solve_parameter: it was tried and reverted, because it was 8x slower.

Verification

  • Pure-move checks: AST multisets for simulation.py, schema.py and tools.py.
  • Bitwise value checks for the modes, profiles and pulses.
  • The five route configuration hashes are unchanged by the driver split.
  • Targeted test modules for every lane pass locally.
  • Two tutorials were regenerated, and the six radial ones were rebuilt for from_free.
  • No translate or backend hot path changed, so no A/B is needed.

🤖 Generated with Claude Code

jtravs and others added 30 commits November 12, 2026 05:41
@jtravs @claude
Merge the identical glass and Sellmeier omega checks into _optmat
e0d27ae 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Add shared argument refusals in solprop._validation
5b7c2a1 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Split output/schema.py into five modules by concern
5f391ea 
serialisation, provenance, records, metadata and store; a pure move (the
definitions' ASTs are unchanged). schema.py stays as a temporary re-export
for driver/simulation.py and driver/interface.py until their split lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Take every stored identity through one canonical digest
f65c328 
solprop/_identity.py holds canonical_json, text_digest and canonical_digest;
the resume key, scan records, coupling bases, TDSE bases and descriptors,
material populations and the TDSE consistency cache use them. The digests
keep their values, pinned by golden tests; a value JSON cannot hold now
raises instead of being recorded by str.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Split solprop.tools into a package of seven modules
5a325b6 
A pure move: the function and class bodies are unchanged; solprop.tools
re-exports every public name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Point the tools cross-references at their home modules
90f5546 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Drive RunMetadata's stored JSON from its field metadata
3fe5d7c 
The resume key's entries, the stored document and the reader come from
each field's metadata instead of three hand lists; the stored format is
unchanged (an older document reads back and rewrites to the same text).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Keep one process_stamp, in solprop._process
48a753b 
The Queue executor and the front-end service each had their own; the
service's also covered Windows. The service's implementation moves into
_process, which imports nothing from solprop and sits below both, and
the executor's copy goes. Its Linux read is _proc_stamp, which treats a
process that vanishes during the read as having no stamp and raises on
anything else. service.record re-exports the name for its own users and
the front end. On this Mac the old executor, the old service and the new
function give identical stamps for this process, a live child and a
missing pid; the Linux branch reads the same /proc field as both did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Split driver/simulation.py by concern
a44bf37 
A pure move: Simulation stays in simulation.py; its settings and their
admission go to settings.py, its input, launch and noise to inputs.py,
its run record to metadata.py, resume to resume.py, the many-input
runner to runner.py, the TDSE run glue to tdse_run.py, and the keyword
routes to routes/ (gnlse, hollow, general, spatial, with the material
selection and keyword attribution they share). The methods _from_spatial,
_from_general and _from_hollow, the resume methods and the TDSE run
methods become module functions taking the simulation (or its class).
Imports, test monkeypatches and documentation references follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Name the gas keyword table publicly for the recipe layer
d8db654 
recipe/provenance.py reads it across packages, so it is public API of
driver.routes.general rather than a private name imported from outside.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Replace small files through one atomic writer in solprop._durable
909382b 
The scan queue's state, the service's run records and the TDSE
consistency cache each had a temporary-file-and-rename copy; each now
encodes its JSON and calls write_atomic, which fsyncs, retries a rename a
reader blocks, and syncs the directory where the queue needs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Pass the hollow and general routes one request each
ae9b0de 
_from_hollow and _from_general take a frozen _HollowRequest or
_GeneralRequest built from the public constructor's locals by
keyword_request, instead of 64 and 41 keywords passed one by one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Document keyword_request as public API of driver.routes.keywords
8bb8251 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Build both Slurm batch scripts with one function in _process
0ccd007 
The scan executor's array script and the front end's run script share
batch_script (quoted directives, the site's lines, the BLAS idle
defaults) and one THREAD_VARIABLES table; each keeps its own thread
policy. A scan's script now also leaves OpenBLAS threads idle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Route pressure_zdw and pressure_rdw through solve_parameter
e39cea3 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Build keyword-route settings and defaults in one place
04e85e5 
keyword_settings builds the settings of the GNLSE, hollow and general
routes (controller, step defaults) in place of three SimulationSettings
constructions; SimulationSettings.with_boundaries applies a spatial
constructor's boundary keywords in place of three _bounded(_watched(
_with_boundary(...))) chains. The step-length fractions, the defaults-row
choice (route_row) and the described defaults (describe_default) live in
driver.defaults, which recipe/metadata.py now reads through one resolver
instead of re-implementing them; the test that only kept the two in sync
is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Share the service's payload reader and dispersion-curve helpers
6d441f7 
One read_recipe(payload) and Payload alias in service/_payload.py replace
three _recipe copies and eight aliases; the curve range, axis reach, zero
and dispersive-wave readings move from sheet.py to service/_curves.py, so
waveguide.py imports shared names instead of sheet's privates. The three
unrelated _plain functions become _plain, _framed and _point_value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Trim process_stamp's private docstrings to the private form
7927469 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Let a Marcatili subclass declare that dneff_dz does not apply
5302ca1 
Replaces the type(self) identity check with a class attribute the
structured-cladding and tabulated modes override.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Fold the rectangular and slab modes' shared members into their base
4987aab 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Describe each statistic once, in its StatDefinition
ffae363 
The parallel name tables of stats.py (general-basis admission, advice,
the field, metric and window evaluators, the weight families, profiles,
plasma families, apertures) become fields of StatDefinition, and the
public sets are derived from the catalogue with unchanged values. The
dead implemented flag and its refusal go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Note the StatDefinition change in the changelog
4388929 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Give ZProfile the defaults of a continuous monotone path
7b816fb 
Constant, PiecewiseLinear, Linear and PressureGradient subclass it
explicitly and drop their copies of jumps, derivative_is_analytic and
variation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Share the scaling of DataPulse and TimeDataPulse in a private base
874f67b 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Tighten the cross-package private-import ratchet to 85
5cec81d 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Rewrap the recipe docstrings the tools references lengthened
2d6ad57 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Record the resume-key change of the record split
74e9694 
The run record names its record types by module, so the split changes
every configuration hash once: the display test's record skeleton is
re-pinned, and the release note says old files are not resumed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Merge from_radial into from_free; compose prop_* from their constructors
9d425f0 
from_free and prop_free take a RadialGrid as well as a FreeGrid or
LineGrid and choose the transform from it; from_radial and prop_radial
are removed (changelog fragment), and recipes keep the radial route.

Each prop_* keeps its typed signature, now the constructor's followed by
the run controls, and its body is _propagate(Simulation.from_x, locals()).
@_propagates(Simulation.from_x) composes its Parameters section from the
constructor's and the shared run-controls entries, so each keyword is
documented once, on from_x (which takes prop_x's fuller entries where it
had shorter ones). test_route_keywords checks the signatures and runs
numpydoc's validate on the composed docstrings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs
Revert "Route pressure_zdw and pressure_rdw through solve_parameter"
38c87c4 
This reverts commit e39cea3.
@jtravs @claude
Merge the shared refusals, the tools package and the physics bases
2956986 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jtravs and others added 9 commits November 12, 2026 06:26
@jtravs
Merge the output split, canonical digests, durable writes and the sta…
53ccc1a 
…tistics registry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	docs/dev/architecture.md
#	docs/dev/figures/layers.svg
#	src/solprop/nonlinear/tdse.py
@jtravs @claude
Rename driver/service.py to driver/session_loop.py
1473e12 
It is SEP 47's session loop, not solprop.service. The private helpers it
imported from driver/run.py that only it uses move with it (_announce,
_earlier_execution, _method_history, _record_abandoned_step,
_record_execution, _refuse_unread_collar, _resume_position, _retain,
_FailedObservation, _MarchKeeper, _Status), and with _MarchKeeper the
MARCH_SNAPSHOT_AUTO_BYTES limit it reads; the two run.py also uses
become public there: history_of and open_run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Record the session loop's rename in the changelog
ccaf283 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Regenerate the gallery pages of the scripts that now call from_free
c4d6edf 
The six radial tutorials and lessons call Simulation.from_free and
prop_free; their committed outputs are rebuilt (SOLPROP_RUN_EXAMPLES=stale).
Their numbers match the parent commit's code run the same way: the
differences from the previous pages predate this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs
Merge the updated two-backends branch
ac4f144 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	docs/dev/architecture.md
#	docs/dev/figures/layers.svg
#	src/solprop/driver/executors/_queue_state.py
#	src/solprop/output/schema.py
#	src/solprop/profiles.py
#	src/solprop/service/record.py
#	src/solprop/service/sheet.py
#	src/solprop/tools.py
#	src/solprop/translate/envelope_plasma.py
#	src/solprop/translate/tdse.py
#	tests/test_process.py
#	tools/size_budget.toml
@jtravs @claude
Give simulation.py's size entry its headroom
34f5a88 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Merge refactor/python-core: PR-3 fixes, the OUT and PHYS lanes
699ca62 
Their changes to driver/simulation.py, run.py and service.py are ported
into the modules the split moved them to (inputs, settings, resume,
routes, session_loop); an AST comparison by name finds every definition
they changed equal to theirs in the merge, and every docstring they
changed equal to theirs or explained by this branch's own edits.

After the merge nothing imports solprop.output.schema, so it is removed:
the driver and solprop_native (backend.py, session.py) imports are
rewritten to output.serialisation/provenance/records/metadata/store,
as are two tutorial sources (their gallery pages are not rebuilt here)
and a pending changelog fragment; figure 1 drops it, and modes/material.py's
comment names the output records instead.

The radial benchmark cases keep their prop_radial label, which their
tracked definition digests include.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Point the output layer's settings references at driver.settings
25a6b07 
SimulationSettings moved out of driver.simulation; the three references
in output/callback.py and output/store.py would fail the nitpicky docs
build, and slab.py's literal named default_radial_order's old home.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Rebuild the analysing-a-run and far-field tutorials for the output split
175df91 
The sources now import from solprop.output.records and friends. Run times on the
Mac: 78 s and 42 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs jtravs added the refactor Internal restructuring with no change in behaviour label Nov 12, 2026
@read-the-docs-community

read-the-docs-community Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Documentation build overview

📚 solprop | 🛠️ Build #35834450 | 📁 Comparing d87f5fe against latest (f247c60)

  🔍 Preview build  

476 files changed · + 52 added · ± 403 modified · - 21 deleted

+ Added

± Modified

- Deleted

jtravs and others added 8 commits November 12, 2026 07:05
@jtravs @claude
Skip prop_* docstring composition when docstrings are stripped
85433c7 
Under python -OO every __doc__ is None and _propagates raised on import;
it now leaves the docstring None. A test imports the interface with -OO.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Call from_free in the gemm-shapes benchmark
9506a2a 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Write NumPy scalars as their values in canonical identities
acc9cc3 
canonical_json no longer stringifies unknown values, so a record holding
np.bool_ (a comparison of arrays) or an np.integer raised TypeError. Its
default= now writes a NumPy bool, integer or floating scalar as the Python
value it holds and still refuses anything else; the stored golden
digests are unchanged. A material manifest's entries take the same rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Keep one MARCH_SNAPSHOT_AUTO_BYTES, in translate.run_request
82c177c 
The session loop defined its own copy; it now imports the request's, and
the tests keep patching the name where the loop reads it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Call open_run where open_run_record wrapped it
6a51389 
The wrapper only discarded open_run's return; the refused-run case it
documented is now described on open_run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Document the wall-pair index methods as the public API they are
200e1e1 
neff, neff_minus_one and dneff_domega live on the private _WallPairFill
base; they take full numpydoc sections again (frequencies, position,
units, shapes), and RectMode and SlabMode name them so that the API
reference documents them on the public classes. The private-sections
cap rises by those three, public API defined in a private class.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs @claude
Call lambda_rdw_all once in shortest_wave
80007f9 
Its range defaults to the tool's own when None, so both branches were
one call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs
Merge remote-tracking branch 'origin/feature/two-backends-binary64' i…
d87f5fe 
…nto refactor/python-core
@codecov

codecov Bot commented Nov 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.51836% with 16 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/solprop/driver/interface.py 89.83% 3 Missing and 3 partials ⚠️
src/solprop/driver/settings.py 98.15% 3 Missing and 2 partials ⚠️
src/solprop/driver/resume.py 98.08% 2 Missing and 1 partial ⚠️
src/solprop/output/metadata.py 99.35% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

refactor Internal restructuring with no change in behaviour

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jtravs