Skip to content

Run a loop's iterations in parallel - #40

Merged
seamm merged 6 commits into
mainfrom
dev
Oct 5, 2026
Merged

seamm merged 6 commits into
mainfrom
dev

Conversation

@paulsaxe

@paulsaxe paulsaxe commented Oct 5, 2026

Copy link
Copy Markdown
Contributor
  • New option "Run iterations in parallel" (off by default). Each iteration then
    runs in an evaluator of its own, at the same time as others, on this machine or
    bundled into queue jobs, and what it did -- structures, properties, table rows
    and cells, files written in the job's directory -- is merged back in iteration
    order. The iterations must be independent: each sees the job's database as it
    was when the loop started (a job-level file it reads is read as it is at that
    moment), and variables set in the loop are not visible after it. break,
    continue and skipping work as before; a failed iteration is never merged.
    Two iterations changing the same table cell or structure are an error unless
    "the later iteration wins" is chosen, which keeps the later change and writes a
    warning to job.out.
  • Its settings, shown only for a parallel loop: iterations at once (by default
    as many as cores and memory allow), cores, memory and time per iteration, what
    each iteration gets (the structures it works on, or the whole database -- every
    iteration's copy is written as the loop starts, which needs disk space), where
    calculations run (inline, or as separate tasks -- experimental) and same-cell
    writes. Starting an evaluator takes about five seconds, so short iterations are
    better run serially.
  • A job stopped part way through a parallel loop resumes it: merged iterations are
    not run again and running ones resume where they were; an iteration whose
    evaluator was killed is run again, up to twice.
  • A user guide for the Loop step.
  • Requires molsystem, seamm and seamm-exec 2026.10.5.

🤖 Generated with Claude Code

paulsaxe and others added 6 commits October 4, 2026 17:03
New parameters (parallel, iterations at once, cores, memory and time per
iteration, snapshot, placement, same-cell writes), shown in the dialog only
for a parallel loop. Each iteration runs in an evaluator of its own, resumed
into just that iteration; the parent merges them in order, one transaction
and checkpoint each. The serial loop is refactored around a set-up per
iteration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An iteration whose evaluator stopped before it ended (its checkpoint still
'running') runs again, resuming, up to twice; one that finished its iteration
is merged even if it was killed as it exited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A loop over systems makes the iteration's configuration current before
  merging it (M2).
- Tables are re-exported and loop_entry.db removed after a break or exit too
  (S6); tables made in the body are variables after the merge (S7); merge
  warnings go to job.out; each iteration gets the chain of directories to read
  job-level files from.
- tests/test_parallel.py runs a loop serially and in parallel with a fake step
  plug-in and compares the databases (S8).
- Requires molsystem and seamm-exec; the user guide updated (conflicts on
  structures, appends, separate placement experimental, costs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The end-to-end parallel test builds and runs its flowcharts with this
interpreter and the package's source first on PYTHONPATH (inherited by the
iterations), not the installed console scripts; failures show their output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@paulsaxe paulsaxe added the enhancement New feature or request label Oct 5, 2026
@paulsaxe
paulsaxe requested a review from seamm October 5, 2026 13:57

@seamm seamm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@seamm
seamm merged commit a1a390d into main Oct 5, 2026
10 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants