engine: measure and hill-climb generated diagram quality - #1065
Conversation
The layout eval harness scored diagrams with a metric an optimizer could game. Sprawl only ever rewarded shrinking, so crowded layouts beat their spacious hand-drawn references. Lines through names, stacked flow pipes, and a valve drawn larger than its scored box went uncharged, and a summed label term made one obscured label cost the same in a 300-variable model as in a 10-variable one. The corpus also carried a duplicate model, two one-aux fixtures, and a model with nothing to lay out. The metric now scores the drawn scene with rate-normalized terms, adds label strike-through, crowding (clearance deficits and links too short to show their arrow), and long connectors, and reports where each defect is. Weights were checked against judged pairs -- metamorphic taste checks (crowd, spread, jitter, shuffle, exile, stack) over every reference and production layout, plus visual reference-vs-production judgments -- and a log-space fit moved the priors under 15%. The declutter pass sees the same obstacles, pipes included, and no longer excuses label collisions between a flow and the stock it attaches to (that excuse had let compaction stack their names). The harness is split into modules: a graded corpus (curated, imported, multi-view, none; size tiers; ad hoc extras), the timed production layout, defect overlays and view JSON per render, the taste battery, and comparison against any earlier run re-scored under the current weights. The aggregate verdict is now a paired Wilcoxon signed-rank test; the unpaired test over per-model medians could essentially never reach significance across models whose costs span orders of magnitude. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
Agents and notebook users rarely get a fresh layout of a finished model: MCP edit_model and pysimlin's patch sync lay out the first edit fresh and every later edit incrementally, preserving what is already placed. The harness only measured the fresh path, so the diagrams those users actually see went unevaluated. The replay builds each model from empty over a few edits (whole chains first, then other variables nearest the backbone), syncs the diagram after each through the same production calls, and scores, renders, and times the result. The first replay across the small and medium tiers shows the incremental path far behind the fresh one: chains added by later edits land on top of each other (reliability's two side chains, delays' three smooth chains) and labels pile up, at up to six times the fresh layout's cost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
A stock face is 35-45px long and a valve is drawn 18px across, so two cloud flows sharing a face at the 1/3 and 2/3 offsets drew their valves, clouds, and names on top of each other. The layout eval renders showed this as the worst defect in cross_element, bathtub, catastrophe, and mortgage_econ (sweep medians -36%, -35%, -9%, -7% once fixed; no other model moved). classify_flow_sides now seats side flows one per face in preference order (outflows right, bottom, top; inflows left, top, bottom) and never on a face a chain flow holds, sharing a face only when every allowed face is taken. On the incremental path it receives the faces the stock's side flows are already drawn on, and each keeps its face unless it sits on a chain face or its preferred face has come free, so adding a sibling never moves a flow the patch did not touch; a flow moved to the opposite face is rebuilt with its label side kept. The three copies of the side-flow geometry (layout_chain, attachment_based_flow_position, create_flow_view_element) now share one valve-position and one pipe-points helper covering all four faces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
compute_metadata drew a variable's links from the heads of its reads, but the parse turns `SMTH1(input, delay)` into a synthesized module instance, hoists a non-identifier argument into a helper, and captures a PREVIOUS argument, so the heads name those helpers rather than what the modeler typed; and a `LOOKUP(table, x)` table is recorded as a referenced table, not a read (#650). Every such input lost its arrow, and a variable reading only through a builtin call was classified as a constant. The layout eval corpus showed it as rows of "isolated" parameters in bathtub, catastrophe, beer_game, mortgage_econ, and wonderland that are in fact wired in. drawn_reads now replaces a read of a helper with what the helper reads, recursively, and adds every table the variable or its helpers call. Sweep costs rise on those models, as they must: the previously parked variables cost nothing without links, and now carry their real ones. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
The declutter pass chose each label's side by the area other shapes and labels covered, so it could not see a link through a name -- the metric's most common defect in every generated corpus layout (a mean strike of 0.25-0.6 per label) -- nor a name jammed against a neighbor. It now charges each candidate side through metrics::LabelScene exactly as the metric charges the label there: covered by shapes and other labels, struck by links and pipes, and crowding its neighbors. On the layout eval corpus (small and medium tiers, 8 seeds) the sweep medians fall 27% in aggregate, significantly on 20 of 21 models; renders show names moving off their links (logistic_growth now reads as a loop) and stock names off crowded faces. The metric charged a pipe through a name by the area of its 4px band, a fifth of what a hairline link through the same name cost, so a stock's name drawn over an inflow entering that face barely registered. A pipe now strikes a name the way a link does (a flow's own pipe excepted, including for the stock it enters) and no longer counts as coverage. The pair clearance crowding measures moves into footprint_gap so the chooser and the term cannot drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
layout/mod.rs had grown to 5925 lines, 75 under the project's file-length lint, and the incremental path is about to grow. incremental_layout and the helpers only it uses (new-element placement and settling, flow-face reclassification, connector and cloud diffing) move verbatim into layout/incremental.rs; the public ones are re-exported from layout so every caller's path is unchanged. No behavior change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
incremental_layout placed new elements with a force pass that treats them as points and chose their label sides by connector angles, with no declutter at all, so a diagram an agent builds over several edit_model calls ended up far worse than a fresh layout of the same model (2-8x the metric cost in the layout eval replay): names on circles, names struck by links, parameters jammed together. Its polish step now runs declutter::declutter_part over the elements the patch created: their label sides are chosen by what the metric charges (the same LabelScene the fresh declutter uses) and the new free-floating elements are pushed off whatever they landed on. Everything that was already drawn is an obstacle only -- no zoom, no compaction -- so the incremental contract holds: untouched elements keep their positions and label sides byte for byte. On the corpus replay (small and medium tiers) the incremental cost falls on 18 of 21 models and rises on none. The angle-based label pass only serves the fresh layout now, so its per-element filter goes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
When an edit added whole stock-flow chains to a drawn diagram -- the usual agent edit, "add the capital sector" -- incremental_layout seeded every new stock at one periphery point and settled each chain as a rigid group, so the chains landed on each other and on the existing diagram (the delays replay drew four chains in one pile; reliability stacked Developers onto Product Comprehensiveness). place_new_chains now lays each chain whose stocks are all new out with layout_chain, exactly as a fresh layout draws a chain, and sets it down below the diagram or to its right: nearer the drawn variables it reads from or feeds, or, with none, whichever keeps the diagram nearer square, preferring below. The placed chain is held still while the new parameters settle around it. On the corpus replay the incremental cost falls on delays (6.71 -> 2.16), reliability (2.71 -> 1.31), fishbanks, workforce, and lotka_volterra, and moves under 3% elsewhere. A new stock hung off an existing chain still takes the generic placement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
A stock an edit hangs off a drawn chain ("infected now drains into
recovered") took the generic new-element placement: seeded near variables it
reads, which for a fresh stock is nothing, so it went to the diagram's
periphery and its pipe bent to reach it.
place_chain_extensions sets each such stock one chain step past its drawn
neighbor, in the neighbor's row -- right of the stock it drains, left of the
stock it feeds -- fanning vertically only past a stock or parameter already
drawn there, and places stocks reached only through other new stocks from
them in turn. The flow between the two, now between two drawn stocks, takes
the stock-pair valve position the existing path already gives such flows.
The layout eval replay adds chains whole and never exercises this case; the
new incremental test pins it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
The SVG renderer sizes its viewBox from each node's drawn extent, but module_bounds returned only the rounded rect, so a module's name hanging past the diagram's edge was cut off -- visible in the layout eval renders of hares_and_foxes, where both module names were clipped. The TS Canvas's moduleBounds merges the label, as the aux and stock bounds already do on both sides. module_bounds now includes the label, and the label-free rect becomes module_shape_bounds, which the layout metric keeps using as the module's shape (it charges labels separately). The new render test rows every labeled node kind the renderer bounds: aux, flow, stock, and module. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
The metric-aware label chooser charged each candidate side by scanning every node and connector in the diagram, so on large models it doubled production layout time (wrld3_03 1.5s -> 3.5s, covid19 1.2s -> 2.3s on the layout eval large tier). LabelScene now buckets nodes by the region they can reach (the shape grown by its label's size and the crowding clearance) and connectors by their bounding boxes, and a query visits only the candidates whose cells meet the label or the owner's own shape -- every pair the crowding term can charge involves one of the two. Skipped items add exact zeros and the candidates are visited in index order, so every sum is bit-identical to a full scan: the new metrics test checks that on every side of every exemplar element, and the large tier's layouts come out identical while production time falls about 35% (wrld3_03 2.2s, covid19 1.3s). The rest of the large-model cost is the declutter relaxation's all-pairs overlap scan, which predates the chooser. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
remove_overlaps tested every pair of footprints on every iteration -- up to 400 iterations per relax, several relaxes per declutter, four seeds per production layout -- and a profile of wrld3_03 put a quarter of its layout time there. Each iteration now buckets the items' translated boxes, grown by the separation margin, into a grid and tests only the pairs that share a cell: any pair separation_mtv would push apart meets in grown boxes. Candidates are visited in the full scan's (i, j) order and a skipped pair pushes nothing, so the accumulated displacements are bit-identical; the new declutter test checks that against an all-pairs oracle on jammed random scenes, and the layout eval large tier reproduces its layouts exactly while production time falls another third (wrld3_03 2.2s -> 1.5s, covid19 1.3s -> 0.9s). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
The declutter separated footprints to a 6px margin and compacted the diagram down to exactly that margin, while the metric charges crowding for any two footprints closer than 8px -- so the tightest arrangement the declutter reached was one the metric charged by construction. The margin is now the metric's COMFORTABLE_CLEARANCE, and the label sides are chosen once more after compaction, which moves everything closer without re-siding. On the layout eval corpus the effect is small and mostly within noise (aggregate -1.6%, n.s.); where it moves a picture it removes visible jamming, e.g. ai_modules_arrays' input no longer touches the module box. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
The committed baseline predated every change to the metric's label terms (pipes now strike names instead of covering area) and to the layouts themselves -- side-flow faces, builtin-input and lookup-table links, the metric-aware label chooser, the declutter margin -- so every run's diff against it mixed metric drift with layout change. Re-seeded over the whole corpus at 8 seeds, as its README describes. The engine map now points at layout/incremental.rs, where the incremental path lives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
Hand-drawn corpus references almost never cross links (12 of 17 have none), while production layouts crossed on up to two fifths of their connectors -- typically a parameter drawn on the far side of a chain or cluster from its consumer. Instrumenting the fresh pipeline showed the crossings are all set by the force pass: its annealing counts crossings over straight chords between point nodes, and the declutter afterwards neither adds nor removes them. layout::polish visits each free node (auxiliary, module, ghost) sitting on a crossing, right after the connectors are built, and tries it at a ring of spots around its neighbors' centroid, keeping a spot only where its own connectors cross strictly less and its shape lands clear of every other shape and pipe. A node's move changes only its own connectors' crossings, so the diagram's count never rises. On the layout eval corpus the sweep medians fall 7% in aggregate on the small and medium tiers (groupon -26%, delays -20%, bathtub -17%, all significant, none significantly worse) and 16% on the large tier, for about 0.1s more on wrld3_03. build_view_segments now builds one connector's segments at a time so the polish can recharge just those. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
The polish charged a spot only by its connectors' crossings and checked only that the node's shape landed clear, so a node could step off a crossing onto a spot where its links struck other names, or its own name took a link, or its label jammed a neighbor. The fresh layout's declutter ran afterwards and repaired much of that; the incremental layout, which polishes what an edit adds, cannot re-side the labels already drawn, and there the moves traded crossings for strikes (fishbanks' replay +0.21 on strikes). A spot is now charged what the metric charges the node locally -- crossings and struck names in the metric's per-connector and per-label units -- and the node's shape and label must both land clear of every shape, label, and pipe. In the fresh layout the polish runs on the settled geometry after the declutter, then the label sides are chosen again around the moved nodes. The incremental layout polishes the free nodes an edit creates, before its own declutter. The scene the spots are charged against is cached per pass and refreshed only for what a move changes. Against the crossings-only polish on the layout eval corpus: fresh medians -0.4% (catastrophe -6.7%, mortgage_econ -5.8%, significant; none significantly worse), large tier -6.5% (covid19 -29%, wrld3_03 -15%), incremental geomean -1.3%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1065 +/- ##
==========================================
+ Coverage 93.24% 93.27% +0.03%
==========================================
Files 254 257 +3
Lines 153508 154305 +797
==========================================
+ Hits 143131 143926 +795
- Misses 10377 10379 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 96d14bd077
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| .zip(&scene.labels) | ||
| .enumerate() | ||
| .filter(|(j, _)| *j != node) | ||
| .flat_map(|(_, (shapes, label))| shapes.iter().chain(label.iter())) |
There was a problem hiding this comment.
Block candidates that cover nonincident links
When a ring candidate places the moved node over a nonincident causal link, this obstacle scan still accepts it because it checks only scene.shapes and scene.labels; link geometry lives separately in scene.segments. If the move reduces an incident crossing or label strike, it can therefore be committed while increasing node_connector_overlap, creating the false visual connection that the calibrated metric is meant to prevent. Include nonincident link segments in the candidate-clearance check or charge this metric term before accepting the move.
AGENTS.md reference: src/simlin-engine/AGENTS.md:L190-L190
Useful? React with 👍 / 👎.
| ## The eval harness | ||
|
|
||
| ``` | ||
| cargo run --release -p simlin-engine --features png_render,file_io --example layout_eval |
There was a problem hiding this comment.
Enable the required layout_eval feature
The documented command cannot launch this example because src/simlin-engine/Cargo.toml declares the target's required-features as png_render, file_io, and layout_eval, while this command activates only the first two. Cargo's run --help confirms that --features selects the features to activate, so users following the new harness documentation will be told the target is unavailable; add layout_eval to the feature list here and in the duplicate usage text.
AGENTS.md reference: AGENTS.md:L122-L122
Useful? React with 👍 / 👎.
| polish::polish_crossings(&mut state.elements); | ||
| declutter::declutter_part(&mut state.elements, |_| true, |_| false); |
There was a problem hiding this comment.
Route connectors before running geometry-aware polish
When stock endpoints are diagonally offset or a dependency belongs to a feedback loop, this fresh-layout path optimizes labels and crossings against the pre-routing geometry, then orthogonalize_flow_pipes rewrites the pipe and apply_loop_curvature rewrites the link afterward. Those final routes can newly cross nodes or labels, so the pipe-aware chooser and crossing polish never inspect the geometry users actually receive; the incremental path has the same ordering. Route connectors first, or rerun the geometry-aware passes after both routing steps.
AGENTS.md reference: src/simlin-engine/AGENTS.md:L190-L191
Useful? React with 👍 / 👎.
| let side = match elem { | ||
| ViewElement::Aux(a) => Some(a.label_side), | ||
| ViewElement::Stock(s) => Some(s.label_side), | ||
| ViewElement::Flow(f) => Some(f.label_side), | ||
| ViewElement::Module(m) => Some(m.label_side), |
There was a problem hiding this comment.
Include groups and alias labels in chain placement bounds
When incrementally adding a whole chain to an imported/grouped diagram or beside an alias with a long source name, footprint_bounds understates the existing diagram: node_shape_box returns no rectangle for groups, and this label match excludes aliases. place_new_chains can therefore choose a below/right placement that overlaps the visible group box or alias label; because the new stock-flow backbone is fixed during declutter_part, the collision remains in the returned view. Include group rectangles and resolved alias-label bounds in this advertised union of drawn extents.
AGENTS.md reference: src/simlin-engine/AGENTS.md:L190-L190
Useful? React with 👍 / 👎.
| /// One view authored in Vensim. Its arrangement is a trustworthy exemplar, | ||
| /// but Vensim draws a variable AS its wrapped name (no circle, no side | ||
| /// label), so the label geometry our renderer imposes on it is not what the | ||
| /// author saw; label-dependent terms over it are not comparable. |
There was a problem hiding this comment.
Cite or qualify the Vensim rendering claim
This classification uses the assertion that Vensim renders variables as wrapped names without circles or side labels to decide that imported reference labels are not comparable, but no source is cited and the claim is not marked unverified. Because that premise controls how the new calibration harness interprets its reference corpus, attach verified Vensim evidence next to it or explicitly qualify the claim as unverified rather than letting it carry the grading decision.
AGENTS.md reference: AGENTS.md:L111-L111
Useful? React with 👍 / 👎.
Simlin draws diagrams for models that have none: an agent building a model over MCP
edit_model, a notebook user patching from pysimlin, an imported equation file. This branch first makes the layout-quality measurement trustworthy, then uses it -- together with looking at every render -- to improve what users actually see, for fresh layouts and for diagrams built edit by edit.Measuring taste
layout/metrics.rs,layout/taste.rs): every term is scored over the geometry the renderer draws and is a rate, so one defect costs the same in a 10-variable model as in a 300-variable one.analyze_layoutreports each defect's location, and the harness draws them over the render. A taste battery checks that the metric charges edits every modeler calls regressions (cramp, inflate, jitter, shuffle, exile a parameter, stack a node).examples/layout_eval/): graded corpus, production timing, defect overlays, run-to-run comparison (per-model Mann-Whitney, paired Wilcoxon aggregate), and a replay that builds each model from empty over four edits, syncing the diagram throughincremental_layoutthe wayedit_modeldoes.docs/design/layout-quality.md.Fresh layouts
SMTH1,DELAY3,TREND, ...) andLOOKUPtables get their links (layout: dependency-graph extraction misses lookup-table references, so lookup tables appear as isolated nodes with no connectors #650). They were parked in rows as "isolated".layout/polish.rs) moves free nodes off link crossings on the settled geometry. A spot is charged the metric's local crossings and strikes and must keep the node's shape and label clear. Hand-drawn references almost never cross links; generated ones crossed on up to two fifths of their connectors.Diagrams built edit by edit
The incremental path moves to
layout/incremental.rs.declutter::declutter_part). The existing contract holds: untouched elements keep position and label side byte for byte.Rendering and speed
module_boundsincludes the label, so rendered module names are no longer clipped at the diagram edge.Evidence
Harness runs compared against the code at c42b5c9 (metric and harness only, no layout change):
declutter_partonly moves and re-sides what it mayWhat this does not establish:
LayoutState::from_existing_viewwould mis-seed uids for a project whose variable uids collide with its view's cloud or link uids; MCP and libsimlin never produce that, and it is not fixed hereFixes #650
🤖 Generated with Claude Code
https://claude.ai/code/session_01QB8VqnbKj6FUNZKRm5tteu