diff --git a/docs/archive-format.md b/docs/archive-format.md index 48e45cf..4af7884 100644 --- a/docs/archive-format.md +++ b/docs/archive-format.md @@ -60,7 +60,7 @@ outputs: solve: {by: {ymf: ..., proteus_git: ..., numpy: ..., platform: ...}, created: ...} storage: poisson_2365d177bb1f.h5 # or: inline verification: {l2_errors: {u: 0.0105...}} - approximation: {time_collections: [...]} # the solver's domain tree + approximation: {times: [...]} # the solver's domain tree reproduced: [{created: ..., by: {...}}] # each rerun that matched, bitwise ``` @@ -160,9 +160,9 @@ formats has to keep two representations consistent, and eventually won't. A solver's document has three top-level keys: ```yaml -ymf_archive_version: 1 # refused on read if it isn't the version this ymf knows -domain: # the mesh/time/field tree -- see below - time_collections: [...] +ymf_archive_version: 2 # refused on read if it isn't a version this ymf knows +domain: # the time/space/field tree -- see below + times: [...] extra: # optional: anything with no XDMF equivalent Problem: {...} # typically the problem spec that produced the run solution_paths: {...} @@ -179,34 +179,34 @@ Here is one time step of `heat.ymf` from [`examples/write_archive.py`](../examples/write_archive.py): ```yaml -ymf_archive_version: 1 +ymf_archive_version: 2 domain: - time_collections: - - name: Mesh Spatial_Domain - data: - - time: 0.0 - topology: - type: Triangle - number_of_elements: 512 - data_item: {format: HDF, data_type: Int, precision: 4, - dimensions: [512, 3], data: 'heat.h5:/elements'} - geometry: - type: XYZ - data_item: {format: HDF, data_type: Float, precision: 8, - dimensions: [289, 3], data: 'heat.h5:/nodes'} - attributes: - - name: T - attribute_type: Scalar - center: Node - data_item: {format: HDF, data_type: Float, precision: 8, - dimensions: [289], data: 'heat.h5:/T_t0'} - - name: q - attribute_type: Vector - center: Node - data_item: {format: HDF, data_type: Float, precision: 8, - dimensions: [289, 3], data: 'heat.h5:/q_t0'} - - time: 10.0 - ... + times: + - time: 0.0 + spaces: + Mesh Spatial_Domain: + topology: + type: Triangle + number_of_elements: 512 + data_item: {format: HDF, data_type: Int, precision: 4, + dimensions: [512, 3], data: 'heat.h5:/elements'} + geometry: + type: XYZ + data_item: {format: HDF, data_type: Float, precision: 8, + dimensions: [289, 3], data: 'heat.h5:/nodes'} + attributes: + - name: T + attribute_type: Scalar + center: Node + data_item: {format: HDF, data_type: Float, precision: 8, + dimensions: [289], data: 'heat.h5:/T_t0'} + - name: q + attribute_type: Vector + center: Node + data_item: {format: HDF, data_type: Float, precision: 8, + dimensions: [289, 3], data: 'heat.h5:/q_t0'} + - time: 10.0 + ... extra: Problem: name: Transient heat conduction, unit square @@ -225,7 +225,7 @@ The structure follows XDMF, so the conversion is a direct mapping, but the *names* are YMF's own: keys are `lower_case_with_underscores` like everything else in the format, and `ymf2xmf` supplies the capitalization XDMF wants. (The correspondence was never one-to-one anyway — -`time_collections`, `spatial_collection` and `attributes` as a list are +`times`, `spaces`, `spatial_collection` and `attributes` as a list are YMF's, with no XDMF element behind them.) Values that name a type — `Triangle`, `Node`, `Scalar`, `Float`, `XYZ` — keep their capitals, which is both the naming rule and what XDMF needs. An archive written with the @@ -233,11 +233,12 @@ old capitalized keys still reads: they are folded on load. ```text domain -└── time_collections: [collection, ...] one per finite-element space - └── collection: {name, data: [step, ...]} - └── step: one instant, in one of two forms - ├── uniform: {time, topology, geometry, attributes} - └── spatial: {time, spatial_collection: [grid, ...]} one grid per subdomain +└── times: [instant, ...] one per archived instant, in time order + └── instant: {time, spaces: {name: space, ...}} + └── space: the state of one finite-element space at that instant, + in one of two forms + ├── uniform: {topology, geometry, attributes} + └── spatial: {spatial_collection: [grid, ...]} one grid per subdomain grid = {name?, topology, geometry, attributes} topology {type, number_of_elements, nodes_per_element?, data_item} @@ -248,12 +249,33 @@ data_item {format: HDF, data_type, precision, dimensions, data: "file.h5:/datas | {format: XML, data_type, precision, dimensions, values: [1.0, 2.5, ...]} ``` -**Why a list of time collections.** A solver writes one collection per +**Why spaces inside each instant.** A solver writes one mesh per finite-element space: a Proteus run with quadratic velocity holds both `Mesh Spatial_Domain` (the linear base mesh) and `Mesh_c0p2_Lagrange` (same elements, more nodes). 195 of the 814 Proteus archives surveyed when this -model was designed had two. `{"TimeCollection": {...}}` (singular) is -accepted as input shorthand and normalized to the list. +model was designed had two. The instants are the archive's own sampling of +the run, and every space is archived at every instant it exists at, so the +instant is the outer level: "which instants does this archive hold?" is +`[i["time"] for i in domain["times"]]`, and an appended record is one whole +instant. A space that is first archived late is simply absent from the +earlier instants. The key order of `spaces` is the order the spaces were +first archived in, and it is part of the format (a hot start depends on it). + +XDMF nests the other way round (a temporal collection per grid type), so +`ymf2xmf` writes one temporal collection per space, in that order, and +`read_xdmf` joins them back into instants on time. For a space-by-space view +of a domain, `space_names(domain)` and `series(domain, name)`; +`time_collections(domain)` is the format 1 shape as a view. + +**Format 1.** Until `ymf_archive_version` 2 the domain was a list of +`time_collections`, each `{name, data: [step, ...]}` with a step +`{time, }`. It is still read, in memory, as the transposed domain +(`transpose_collections`): steps of different spaces are joined on exact +time, a repeated time pairs up by occurrence, and a space missing at a time +is omitted from that instant. Nothing is rewritten on disk. Code that built +a domain with `new_domain(name)` / `add_collection` / `add_uniform_step(..., +collection=)` keeps working and is upgraded when the domain is validated or +written; `{"time_collection": {...}}` (singular) is accepted as before. **Closed sets.** These fields take a fixed set of values, and anything else is rejected when the archive is written: @@ -282,19 +304,19 @@ disagree with what's in the file. ```python import h5py, numpy as np -from ymf.archive import (new_domain, add_uniform_step, topology, geometry, +from ymf.archive import (new_domain, add_instant, uniform_grid, topology, geometry, attribute, data_item_for, write_ymf) -domain = new_domain("Mesh Spatial_Domain") +domain = new_domain() with h5py.File("heat.h5", "w") as h5: h5["nodes"], h5["elements"] = nodes, elements # mesh written once topo = topology("Triangle", len(elements), data_item_for(elements, "heat.h5:/elements")) geom = geometry(data_item_for(nodes, "heat.h5:/nodes")) for k, t in enumerate(times): h5["T_t%d" % k] = T = solve_to(t) - add_uniform_step(domain, t, topo, geom, [ + add_instant(domain, t, {"Mesh Spatial_Domain": uniform_grid(topo, geom, [ attribute("T", data_item_for(T, "heat.h5:/T_t%d" % k)), - ]) + ])}) write_ymf(domain, "heat.ymf", extra={"Problem": problem}) ``` @@ -388,7 +410,7 @@ genuine mismatch is not. Errors name the offending field: ```text data_item_for().dimensions: declared [81] (81 values) but the array has shape [25] (25 values) -time_collections[0].data[0].topology.type: 'Triangel' is not a known XDMF topology type. ... +times[0].spaces[Mesh].topology.type: 'Triangel' is not a known XDMF topology type. ... ``` The first of these is a real bug this check found in Proteus: a field @@ -468,66 +490,29 @@ XDMF XML with this model: rather than from an `.xmf`; only a ≤ 1.9 archive, which never recorded the order, still needs the `.xmf` written beside it. -## Open: the nesting of time and space may be inverted - -Recorded 2026-10-23, from cekees. The document today has **one temporal -collection per function space**, each holding its own list of instants: - -```yaml -time_collections: -- name: Mesh_c0p1_Lagrange - data: [{time: 0.0, ...}, {time: 0.5, ...}] -- name: Mesh_c0p2_Lagrange - data: [{time: 0.0, ...}, {time: 0.5, ...}] -``` - -That shape is inherited from XDMF, where a `Temporal` `GridCollection` -belongs to a grid type. The likely better model is the transpose — **one -time collection, with the spaces nested inside each instant**: - -```yaml -times: -- time: 0.0 - spaces: - Mesh_c0p1_Lagrange: {topology: ..., geometry: ..., attributes: ...} - Mesh_c0p2_Lagrange: {...} -``` - -The objection to it is operator splitting: sub-models advance on different -time steps, so there is no single instant at which every space is current. -It does not hold, for two reasons: - -- an operator-split or fractional-step calculation **has synchronization - points** where every space is known, and those are the logical places to - archive anyway; -- where it does not, there are rational ways to write every space at the - finest-grained step. - -More to the point, **the archiving step is a separate choice from the -numerical step**. Writing every space at every step it was approximated at -is usually not affordable, so the archive already samples; and once it -samples, it samples at instants, which is exactly what the transposed shape -describes. +## Time outside, space inside -What it would buy: "which instants does this archive hold?" becomes a -direct question rather than a walk over every collection; and a writer -appending one record per instant needs no lazily-created collections, so -the records would not have to name a collection at all. The append support -added for proteus had to name them precisely because collections appear as -models first archive — see :func:`ymf.archive.append_step`. +Decided 2026-10-25 (cekees), after being recorded as open on 2026-10-23. The +domain used to hold one temporal collection per function space, each with +its own list of instants, because XDMF does; it now holds one list of +instants with the spaces inside each. The reasons, the migration and the +Proteus changes are in +[design-time-outside-space.md](design-time-outside-space.md). In short: +the Proteus writer already refused a space missing at an instant; an +instant is now the unit of an appended record, so a killed run loses a whole +instant and never one space of it; and `append_step` no longer names +collections. -What it would cost: `ymf2xmf` transposes on the way out, both ends of the -reader and writer change, and it is a format version. +Proteus's HDF5 metadata version (`ymf_archive_metadata_version`, 2) is a +different number and did not change. ## Limits - Fields carry no units. The archive says what shape and type a field is, not what physical quantity it is. Today that information lives in the problem spec in `extra`. -- One level of spatial nesting: a step is a single grid or a flat - collection of subdomain grids. -- Time is nested inside space rather than the other way round; see the - section above. +- One level of spatial nesting: a space at an instant is a single grid or a + flat collection of subdomain grids. - `extra` is stored as-is and not validated. -- Format version 1 is the only one. A reader refuses any other version - rather than guess. +- Format version 2 is written. Version 1 is read (and upgraded in memory); a + reader refuses any other version rather than guess. diff --git a/docs/design-time-outside-space.md b/docs/design-time-outside-space.md new file mode 100644 index 0000000..4fc681f --- /dev/null +++ b/docs/design-time-outside-space.md @@ -0,0 +1,363 @@ +# Design note: time outside, space inside + +**Status: approved by cekees 2026-10-25 (all six recommendations); implemented in the ymf PR that follows this note. See § Implementation notes for what differs from the plan.** +Closes the open question at the end of [archive-format.md](archive-format.md) +(recorded 2026-10-23). Drafted 2026-10-25. + +The Proteus writer is a separate PR, after the ymf one (§ 7). + +## 1. The proposal + +Today a domain is a list of **time collections, one per finite-element space**, +each holding its own list of instants. The proposal transposes it: **one list +of instants, with the spaces inside each instant.** + +```yaml +# format 1 (today) # format 2 (proposed) +time_collections: times: +- name: Mesh_c0p1_Lagrange - time: 0.0 + data: [{time: 0.0, ...}, spaces: + {time: 0.5, ...}] Mesh_c0p1_Lagrange: {topology: ..., geometry: ..., attributes: ...} +- name: Mesh_c0p2_Lagrange Mesh_c0p2_Lagrange: {...} + data: [{time: 0.0, ...}, - time: 0.5 + {time: 0.5, ...}] spaces: {...} +``` + +A space's value is exactly what a step is today, minus `time`: a uniform grid +`{topology, geometry, attributes}` or a spatial one `{spatial_collection: [...]}`. +Everything below the space level (grids, topology, geometry, attributes, +data items, inline values, canonicalization, validation) is **unchanged**. + +## 2. Why it is worth a format version + +Measured against the code on `main` (ymf `ef44cbe`, proteus `9a3f1160`): + +1. **The writer already requires it.** `allGatherIncremental` raises + `RuntimeError` if any collection lacks a grid for the instant being + archived ("every collection is written at every archived instant"). + The only thing that varies is *which* spaces exist: a model that first + archives at step 40 adds its space then. The proposal encodes exactly that + (§ 4.3). Format 1 spends a whole second level of indexing on a + redundancy the writer enforces anyway. +2. **An instant becomes the unit of atomicity.** Proteus's live append + (`_appendLiveStep`) writes one record per collection per step. A run killed + between two of those leaves collection A at step N and collection B at + step N-1, and the reader keeps both. With one record per instant a kill + loses a whole instant or none of it. This is a correctness gain, not + tidiness. +3. **"Which instants does this archive hold?"** is `[t["time"] for t in + domain["times"]]`. Today `Archiver.archived_times` walks the first + collection and has a comment that "every collection covers the same + instants, so one is enough", a fact the format does not state. +4. **`append_step` stops needing collection names** (§ 4.2), which is the + thing the lazily created collections forced. + +The objection (operator splitting: no instant at which every space is +current) is answered in archive-format.md: the archive samples at +synchronization points, and the archiving step is a choice separate from the +numerical step. It also never applied to what Proteus does today, per item 1. +A space that is genuinely absent at an instant is simply absent from `spaces`. + +## 3. Two version numbers, and which of them moves + +"v1/v2" is ambiguous in this codebase. There are two independent numbers: + +| number | where | values today | meaning | +|---|---|---|---| +| `ymf_archive_version` | top of a `.ymf` document (`ymf.archive.ARCHIVE_FORMAT_VERSION`) | 1 | shape of the `domain` tree | +| `METADATA_FORMAT_VERSION` | HDF5 root attribute `ymf_archive_metadata_version` (`proteus.Archiver.AR_base`) | 1 = XDMF `` fragments (Proteus <= 1.9.x), 2 = YAML grid dicts | how per-step, per-rank grid metadata is stashed in the `.h5` | + +**Proposal: `ymf_archive_version` goes 1 -> 2; `METADATA_FORMAT_VERSION` +stays at 2.** The HDF5 stash is datasets named `_` and the +collection names live in `ymf_archive_collections`. That is a storage layout +for the *writer's* working state, and the transposition happens when the +document is assembled from it (`assemble_domain`) and when live records are +appended. Leaving it alone means: + +- no change to the collective MPI path (`allGatherIncremental`, the + `allreduce(MAX)` width agreement, the per-rank gather); +- hot start (`_collection_names`, `_seed_domain`) is untouched; +- every existing Proteus 2.0-line archive's `.h5` assembles into a format 2 + document with no migration, and `scripts/gatherTimes` can regenerate the + `.ymf` from any `.h5` (it already does). + +The cost is that the `.h5` keeps `n_spaces x n_steps` small metadata +datasets. That is the existing cost and can be revisited separately. + +The closure document (`ymf: 0.2.x`, an output's `approximation`) carries the +domain too. It has no archive-version key; it is guarded by the package +version in `ymf:` (`check_version` refuses a newer writer). A format 2 +`approximation` is written by a newer ymf, so an old reader refuses it +instead of misreading it. See § 7 for the version number this implies. + +## 4. Changes, by component + +### 4.1 Data model (`ymf.archive`) + +New canonical in-memory shape: `domain = {"times": [instant, ...]}` with +`instant = {"time": float, "spaces": {name: grid_or_spatial}}`. + +- `new_domain()` -> `{"times": []}`. The `name` argument goes (a domain no + longer starts with a named collection). Callers that passed a name keep + working for one release with a `DeprecationWarning` (§ 7). +- `add_instant(domain, time, spaces)`, with `spaces` a mapping built from + `uniform_grid(...)` / `spatial_grid(...)` (thin wrappers over the existing + `grid()` and `{"spatial_collection": ...}`). +- `add_uniform_step` / `add_spatial_step` / `add_collection` keep their + signatures and map onto the new shape: they find the instant whose time + equals `time` (or append one) and set `spaces[collection]`. They exist so + the Proteus writer, `examples/write_archive.py` and the tests can migrate in + steps. `collection=` as an integer index is only defined while a domain was + built through these shims; a name is the supported form. +- **Space order is significant** and is the order of the first instant that + contains the space, then insertion order. `pyyaml`'s dumper already runs + `sort_keys=False`. Hot start needs the creation order (`EBMesh` would sort + ahead of `Mesh`), so the format *states* it: the key order of `spaces` is + document order. +- Two views for consumers that think per space (the `MeshTools` mesh reader, + `scripts/clearh5.py`, `ymf2xmf --show`): + `space_names(domain)` and `series(domain, name)` -> `[(time, grid), ...]`. + `findArchiveGrid(domain, MeshTag, id_in_collection)` becomes a three-line + use of `series`. +- `canonicalize_domain` / `validate_domain` gain: `times` is a list, + `time` is a float, instants are not required to be strictly increasing (a + hot-started run may repeat a time; see § 8), `spaces` is a non-empty + mapping, and the existing grid checks run per space with the path + `times[i].spaces[name]...` in errors. +- `select_domain(domain, collections=, fields=, steps=)`: `collections` + filters `spaces`; `fields` unchanged; **`steps` now indexes instants, not + per-collection steps**. This is a behavior change only when collections have + different step counts, which the writer forbids except for lazily added + spaces. Keep the keyword name `collections=` (it is the CLI's `-c`). + +### 4.2 `append_step` without collection names + +```python +append_step(path, instant, *, fsync=False) # instant = {"time": t, "spaces": {...}} +``` + +One YAML document per instant, same `---` ... `...` framing, same +truncation rule (an instant is complete or absent). The record is +`{"step": instant}`; the `"collection"` key disappears. `read_ymf` folds a +record by appending it to `times`, creating nothing by name because the +spaces are *in* the record. A space that first appears at step 40 is just a +key that first appears in the 41st record. + +The old three-argument form `append_step(path, collection, step)` is kept for +format 1 writers during the deprecation window and writes a **format 1** +record; a file is never mixed (§ 4.4 `read_ymf` refuses it). + +### 4.3 Writer (Proteus `Archiver.py`) — after approval + +Territory note: this is the only proteus change, and it is contained. + +- `allGatherIncremental` keeps its loop over collections and its HDF5 stash + unchanged. It builds `live_steps` as today, then `_appendLiveStep` groups + them into **one** `{"time": t, "spaces": {name: grid_or_spatial}}` record + and makes **one** `append_step` call per instant instead of one per + collection. Its header writes `new_domain()` (empty `times`). +- `assemble_domain` iterates steps outermost and collections innermost, + building each instant from the datasets that exist at that step. Its + `n_steps is None` end-detection ("count up until a step is missing") becomes + "until no collection has the step". The v1-metadata path (`_v1_grids`) + feeds the same builder. +- `_inlineStep` / `_inlineDomain` build instants directly. +- `ArchiveCollection` / `ArchiveDomain` (the *running*, one-step-deep objects + that the ~25 `write*Xdmf_*` writers use) are **not changed**. They are a + writer-side convenience and never see the document's shape. +- `archived_times` becomes `[i["time"] for i in domain["times"]]`. +- Readers in proteus that index `time_collections`: `MeshTools.findArchiveGrid` + and `readArchiveDomain` callers (`MeshTools.py:7003, 7138`), + `scripts/clearh5.py:44`, `scripts/gatherTimes:43`, + `proteus/cli/ymfrun.py` (via `domain_arrays`, which walks the tree and + needs only the new walk), and the tests listed in § 6. + +### 4.4 Reader (`read_ymf`) + +- `ymf_archive_version` 1 **or** 2 is accepted; anything else is refused as + today. A version 1 document is **upgraded in memory** by + `transpose_collections(domain)` (§ 5) and returned in the format 2 shape. + Callers never see `time_collections` again. +- Appended records: a stream whose header is format 1 may contain + `{"collection", "step"}` records (an in-flight run from before the + upgrade); they are folded into the header's collections *then* the whole + domain is transposed. A format 2 header with a format 1 record, or the + reverse, is an error naming the line. +- The truncation rule is unchanged. + +### 4.5 `ymf2xmf` and `ymf.xdmf` + +XDMF's model is the old one (a Temporal `GridCollection` of grids of one +type), and every viewer expects it. The conversion therefore **transposes +back**: + +- `build_xdmf_tree`: for each space name (in document order), one + `Grid CollectionType=Temporal` holding that space's instants, exactly the + tree emitted today. The XML is byte-identical for an archive that round + trips, which makes the change testable: convert a format 1 archive and its + format 2 upgrade, diff the `.xmf`. +- `parse_xdmf_domain`: reads per-space collections, then + `transpose_collections` joins them on time (§ 5). `read_xdmf` and + `round_trip_equal` keep their contracts. +- `ymf.xdmf.collection_names` (used by Proteus hot start for <= 1.9 + archives) is about the `.xmf` and does not change. +- CLI: `--show` prints one line per space as now (`series()` provides it) + plus an instants line (`6 instants, t=0..50, 2 spaces`). `-s/--steps` + selects instants. `-c/--collection` selects spaces. The `-i` prompt is + unchanged in appearance. No flag is renamed. + +I considered emitting XDMF as a Temporal collection of Spatial collections +(one per instant, spaces as the members). Viewers would then treat the +linear and quadratic meshes at an instant as one multi-block dataset, which is +what ParaView does with per-rank subdomain grids, but the two meshes overlap +cell for cell and are not pieces of one mesh. Keeping them as separate +datasets is the current behavior and the right one. + +## 5. Transposing format 1: the migration + +One function, used by `read_ymf`, `parse_xdmf_domain`, and the Proteus +v1-metadata path: + +``` +transpose_collections(collections) -> times +``` + +1. Collections are taken in list order; that order becomes the order of + `spaces`. +2. Each collection's steps are keyed by `(time, occurrence)`, where + `occurrence` is how many earlier steps of the **same collection** had that + exact `time`. This pairs repeats correctly if a time legitimately recurs. +3. Instants are the union of those keys, ordered by first appearance when the + collections are walked step-major (step 0 of every collection, then + step 1, ...). When all collections have equal-length, equal-time series + (the writer's normal case) this is just "zip". +4. A space missing at a key is omitted from that instant's `spaces`. +5. Times are compared exactly. Proteus stores them with `repr(float)` and + YAML round trips a float exactly, so equality is the right test; a + tolerance would merge instants that are really distinct. + +A collection with a single step (a Uniform or Spatial top-level grid read +from foreign XDMF) is one key at its time (0.0 if it has none), as +`parse_xdmf_domain` does today. + +**Archives on disk need no rewrite.** Format 1 files stay readable +indefinitely. `ymf2xmf --upgrade run.ymf` (new, optional) rewrites a format 1 +`.ymf` as format 2, and `scripts/gatherTimes` regenerates the document from +the `.h5` in whatever format the installed ymf writes. The `.h5` is +format-agnostic (§ 3). + +Cases and where each is covered: + +| archive | handled by | +|---|---| +| Proteus <= 1.9.x: metadata v1 (XDMF fragments), no `.ymf` | unchanged read path; `assemble_domain` builds instants | +| Proteus 2.0-line, metadata v2, `.ymf` format 1 | `read_ymf` upgrades in memory | +| In-flight / killed run, format 1 header + per-collection records | `read_ymf` folds then transposes (§ 4.4) | +| Closure `approximation: {time_collections: [...]}` | the same `transpose_collections`, applied on load; written as `times` | +| Foreign `.xmf` | `parse_xdmf_domain` | + +## 6. Test plan + +New tests, then the existing suites that mention the shape. + +- `transpose_collections`: zip case, lazily added space, repeated time, + single-step collection, empty collection, order preservation. +- Round trip: format 1 fixture -> read -> write (format 2) -> read equals; + format 1 -> `ymf2xmf` and format 2 -> `ymf2xmf` give identical `.xmf`. +- `append_step`: a killed-mid-instant file drops the whole instant; a space + appearing at record 41; format mismatch errors. +- `select_domain` with `steps=-1` and unequal step counts. +- The shipped examples and `tests/fixtures` in both formats. +- Proteus (after approval): `test_archive_dtypes`, `test_archive_no_xml`, + `test_adr_problem` archive checks, `test/ci/test_systemStep.py` + (`archived_times`), a hot-start test from a format 1 archive, and a + two-space run (c0p1 + c0p2) at 1 and 2 ranks, global and per-subdomain, + comparing `.xmf` from `ymf2xmf` to the pre-change output. +- Because proteus CI installs ymf from `@main`, the ymf PR merges first; + until the Proteus PR lands the new ymf must still serve the old Proteus + writer, which is what the shims in § 4.1 are for. + +## 7. Compatibility and release order + +- ymf: the new reader accepts both formats; the new writer writes format 2. + `ymf.__version__` goes to the next minor (0.3.0 proposed) since the closure + `approximation` shape changes and `check_version` is what protects old + readers. +- Shims (`new_domain(name)`, `add_collection`, `add_uniform_step(..., + collection=)`, three-argument `append_step`) keep the current Proteus + writer working against the new ymf unchanged, but with the format 1 record + for `append_step`. One release later they warn; one after that they go. +- Order: (1) ymf PR implementing § 4.1, 4.2, 4.4, 4.5 and § 5, merged; + (2) Proteus PR changing `Archiver.py` (§ 4.3) and its readers, merged; + (3) shims deprecated. Between 1 and 2 nothing in Proteus changes behavior. +- External: any script that does `domain["time_collections"]` directly breaks + on format 2. In this tree that is `scripts/clearh5.py`, `scripts/gatherTimes`, + `MeshTools.py`. I have not searched user code outside these repositories; + `air-water-vv` is the one worth checking before release. + +## 8. Things I could not verify + +- **Hot start repeating a time.** Whether a hot-started run can write an + instant at a time already in the archive. The occurrence index in § 5 + handles it; whether the writer should *replace* or *append* is a Proteus + question this note does not answer. +- **Lazily added spaces in a real run.** The writer's own check forbids a + collection with no grid at an instant, but I found no case in the test suite + that adds a space mid-run. The docstring of `append_step` says it happens. + The design handles it; no fixture exercises it yet. +- **`approximation` consumers outside these repos.** +- **Cost.** I have not measured parse time of a transposed document against + the current one. The record count drops from `n_spaces x n_steps` to + `n_steps`, so the append path should get cheaper; the one-pass transpose on + load is O(total steps). + +## Decisions needed + +1. **Approve the transposition** and `ymf_archive_version` 1 -> 2. +2. **Key name**: `times` / `spaces` as drawn, or something else + (`instants`)? Rename is free until code exists. +3. **`spaces` as a mapping** (lookup by name; order is document order, which + the format must state) **or a list** of `{name, ...}` like `attributes` + (order is structural; one extra key per entry). Recommended: mapping. +4. **Leave `METADATA_FORMAT_VERSION` at 2** and transpose only at assembly + (§ 3). Recommended: yes. +5. **Release number** for ymf (0.3.0?) and whether the deprecation window is + one release or two. +6. **Optional `ymf2xmf --upgrade`**: wanted, or is "format 1 reads forever" + enough? + +## Implementation notes + +Differences from the plan above, and what the ymf PR does not do. + +- **`ymf2xmf --upgrade` is not implemented.** Format 1 reads indefinitely, + and `scripts/gatherTimes` regenerates a document from the `.h5`. +- **Format 1 records in a format 2 header are accepted.** The current Proteus + writer writes `write_ymf(new_domain(name))` (now a format 2 header) and then + `append_step(path, name, step)`. `read_ymf` joins those records into + instants, so a run of today's Proteus against the new ymf still reads. Only + an instant record inside a format **1** header is an error. +- **A format 1 header's empty collections are dropped.** A space exists once + an instant holds it, so `new_domain("Mesh")` followed by nothing reads as + no spaces. The `_times` assertion of a test that expected the name was + changed accordingly. +- **`add_uniform_step(..., collection=)` defaults to `None`.** On a format 2 + domain that means the space `"TimeCollection"`, which is what + `new_domain()` with no name used to create; an integer indexes the spaces + seen so far. `add_collection` on a format 2 domain is a no-op. +- **`new_domain(name)` still returns a format 1 domain**, which every + function upgrades; `new_domain()` returns `{"times": []}`. +- **`time_collections(domain)`** returns a format 1 domain's list as it is. +- **The compatibility claim in § 7 was too strong.** What the shims keep + working is *writing*. *Reading* changed shape: `read_ymf(...)[0] + ["time_collections"]` is gone, and Proteus has consumers of exactly that + (`scripts/clearh5.py`, which would silently loop over nothing, and tests + in `test_archive_parallel_metadata`, `test_archive_v1_compat`, + `test_hdf5_api_migration`, `test_archive_dtypes`). Proteus CI installs ymf + from `@main`. So the order is: (1) a Proteus PR whose readers accept either + shape, with no dependence on the new ymf; (2) this ymf PR; (3) the Proteus + writer PR (§ 4.3). The ymf PR should not merge before (1). +- Verified: `.xmf` output of `write_archive.py` (both archives) and of a + two-space, three-instant domain is byte-identical to what the pre-change + code wrote, from a format 1 domain, from a format 2 domain, and from a + format 1 `.ymf` read back (`tests/fixtures/format1_two_spaces.*`). diff --git a/examples/read_archive.py b/examples/read_archive.py index 49aef40..a85473e 100644 --- a/examples/read_archive.py +++ b/examples/read_archive.py @@ -33,7 +33,7 @@ import numpy as np -from ymf.archive import canonicalize_domain, read_ymf +from ymf.archive import canonicalize_domain, read_ymf, series, space_names from ymf.xdmf import read_xdmf @@ -83,17 +83,17 @@ def main(argv=None) -> int: print("kappa in the spec: %g %s" % (kappa_spec, problem["strong_form"]["coefficients"]["κ"]["units"])) - collection = domain["time_collections"][0] + (space,) = space_names(domain) times, peaks = [], [] print("\n time subdomains max T [K]") - for step in collection["data"]: + for t, step in series(domain, space): grids = grids_of(step) # The peak over the whole domain is the peak over its subdomains. peak = max(load_array(field(g, "T")["data_item"], base_dir).max() for g in grids) - times.append(step["time"]) + times.append(t) peaks.append(peak) - print(" %5.1f %10d %9.4f" % (step["time"], len(grids), peak)) + print(" %5.1f %10d %9.4f" % (t, len(grids), peak)) # ln(T_max - T_wall) = ln(A) - 2 pi^2 kappa t, so the slope gives kappa. slope = np.polyfit(times, np.log(np.array(peaks) - t_wall), 1)[0] diff --git a/examples/write_archive.py b/examples/write_archive.py index e17fe19..26deb4f 100644 --- a/examples/write_archive.py +++ b/examples/write_archive.py @@ -41,8 +41,9 @@ import yaml from ymf.archive import ( - add_spatial_step, - add_uniform_step, + add_instant, + spatial_grid, + uniform_grid, attribute, data_item_for, geometry, @@ -135,7 +136,7 @@ def open_store(outdir, stem, inline): def write_serial(outdir, nodes, elements, times, problem, inline=False): """One grid per time step. Returns the path of the .ymf written.""" store = open_store(outdir, "heat", inline) - domain = new_domain("Mesh Spatial_Domain") + domain = new_domain() # The mesh does not move, so it is stored once and every step refers # to the same arrays. topo = topology("Triangle", len(elements), store.item("elements", elements)) @@ -149,11 +150,11 @@ def write_serial(outdir, nodes, elements, times, problem, inline=False): for k, t in enumerate(times): T = exact_temperature(nodes, t) q = exact_heat_flux(nodes, t) - add_uniform_step(domain, t, topo, geom, [ + add_instant(domain, t, {"Mesh Spatial_Domain": uniform_grid(topo, geom, [ attribute("T", store.item("T_t%d" % k, T)), attribute("q", store.item("q_t%d" % k, q), attribute_type="Vector"), side_attr, - ]) + ])}) store.close() ymf_path = outdir / "heat.ymf" @@ -183,7 +184,7 @@ def write_split(outdir, nodes, elements, times, problem, inline=False): """Each step is a SpatialCollection of per-subdomain grids.""" store = open_store(outdir, "heat_split", inline) parts = split_mesh(nodes, elements) - domain = new_domain("Mesh Spatial_Domain") + domain = new_domain() meshes = [ (topology("Triangle", len(local_elements), store.item("elements_p%d" % rank, local_elements)), @@ -198,7 +199,7 @@ def write_split(outdir, nodes, elements, times, problem, inline=False): grids.append(grid(topo, geom, [ attribute("T", store.item("T_p%d_t%d" % (rank, k), T)), ], name="subdomain_%d" % rank)) - add_spatial_step(domain, t, grids) + add_instant(domain, t, {"Mesh Spatial_Domain": spatial_grid(grids)}) store.close() ymf_path = outdir / "heat_split.ymf" diff --git a/src/ymf/archive.py b/src/ymf/archive.py index d80dbab..12c0eea 100644 --- a/src/ymf/archive.py +++ b/src/ymf/archive.py @@ -53,28 +53,32 @@ A *domain* is a plain dict -- no classes, no schema objects, so it costs nothing to build, gathers across MPI natively, and serializes directly:: - {"time_collections": [{"name": str, "data": [step, ...]}, ...]} + {"times": [{"time": float, "spaces": {name: space, ...}}, ...]} -There is a *list* of time collections, not one, because a solver writes one -per finite-element space: a real Proteus archive commonly holds both -``Mesh Spatial_Domain`` (the linear base mesh) and ``Mesh_c0p2_Lagrange`` -(the quadratic space, on the same elements with more nodes). 195 of the 814 -archives surveyed when this model was designed had two. A single-collection -document may be written as ``{"time_collection": {...}}`` for convenience; -:func:`canonicalize_domain` normalizes it to the list form, which is the -only form written to disk. +There is a *mapping* of spaces inside each instant, not one grid, because a +solver writes one mesh per finite-element space: a real Proteus archive +commonly holds both ``Mesh Spatial_Domain`` (the linear base mesh) and +``Mesh_c0p2_Lagrange`` (the quadratic space, on the same elements with more +nodes). 195 of the 814 archives surveyed when this model was designed had +two. A space first archived late is absent from the earlier instants. The +order of the names is the order they were first archived in. -Each ``step`` is one instant in time and takes exactly one of two forms: +Format 1 nested the other way, ``{"time_collections": [{"name": str, "data": +[step, ...]}, ...]}``; it is still accepted everywhere and +:func:`canonicalize_domain` upgrades it. See +docs/design-time-outside-space.md. + +A space at an instant takes exactly one of two forms: *uniform* -- a single grid, which is what a serial run or a globally synchronized parallel write produces:: - {"time": float, "topology": T, "geometry": G, "attributes": [A, ...]} + {"topology": T, "geometry": G, "attributes": [A, ...]} *spatial* -- a collection of grids, one per MPI rank, which is what a subdomain-per-rank write produces:: - {"time": float, "spatial_collection": [grid, ...]} + {"spatial_collection": [grid, ...]} and the pieces are:: @@ -141,6 +145,13 @@ "geometry", "grid", "new_domain", + "add_instant", + "uniform_grid", + "spatial_grid", + "space_names", + "series", + "time_collections", + "transpose_collections", "add_collection", "add_uniform_step", "add_spatial_step", @@ -181,7 +192,14 @@ class YmfArchiveError(ValueError): #: into every document by :func:`write_ymf` and checked by :func:`read_ymf`, #: so a version mismatch produces a clear error instead of a confusing #: parse failure somewhere downstream. -ARCHIVE_FORMAT_VERSION = 1 +ARCHIVE_FORMAT_VERSION = 2 + +#: Document formats :func:`read_ymf` accepts. Format 1 nests time inside +#: space (``time_collections``, one per space); format 2 nests space inside +#: time (``times``, each instant holding its ``spaces``). A format 1 +#: document is upgraded in memory on read and never written; see +#: docs/design-time-outside-space.md. +_READABLE_FORMAT_VERSIONS = (1, 2) #: Separators framing an appended step. The end marker is YAML's explicit #: document-end token, and it is what lets a reader tell a complete record @@ -526,23 +544,61 @@ def grid( return g -def new_domain(name: str = _DEFAULT_TIME_COLLECTION_NAME) -> Dict[str, Any]: - """Create a domain with one empty, named time collection. +def new_domain(name: Optional[str] = None) -> Dict[str, Any]: + """Create an empty domain. - Add further collections with :func:`add_collection` -- one per - finite-element space that needs its own mesh. + The domain is ``{"times": []}``: add instants with :func:`add_instant`. + ``name`` is the format 1 spelling, which started a domain with one named + time collection; given, it still does, and :func:`add_uniform_step` and + friends then keep building format 1 (which :func:`canonicalize_domain` + upgrades). New code should not pass it. """ + if name is None: + return {"times": []} return {"time_collections": [{"name": str(name), "data": []}]} +def uniform_grid( + topo: Dict[str, Any], + geom: Dict[str, Any], + attributes: Optional[Iterable[Dict[str, Any]]] = None, +) -> Dict[str, Any]: + """The value of a space at an instant, as one grid: ``{topology, geometry, attributes}``.""" + return {"topology": topo, "geometry": geom, + "attributes": list(attributes) if attributes is not None else []} + + +def spatial_grid(grids: Iterable[Dict[str, Any]]) -> Dict[str, Any]: + """The value of a space at an instant, as one grid per subdomain.""" + return {"spatial_collection": list(grids)} + + +def add_instant( + domain: Dict[str, Any], time: float, spaces: Dict[str, Dict[str, Any]] +) -> Dict[str, Any]: + """Append one instant holding ``spaces`` (name -> :func:`uniform_grid` or + :func:`spatial_grid`). Returns the new instant. + + The order of ``spaces`` is the order the archive records its spaces in, + which a hot start depends on. + """ + if "time_collections" in domain: + raise YmfArchiveError( + "domain is in format 1 (time_collections); add_instant needs a " + "format 2 domain, from new_domain() with no name") + instant = {"time": float(time), "spaces": dict(spaces)} + domain.setdefault("times", []).append(instant) + return instant + + def add_collection(domain: Dict[str, Any], name: str) -> Dict[str, Any]: - """Append a new named time collection. Returns it. + """Format 1 spelling: append a new named time collection. Returns it. - Use one collection per finite-element space: the nodes of a quadratic - space are not the nodes of the linear mesh, so they cannot share a - grid, and a viewer needs to see them as separate meshes over the same - elements. + On a format 2 domain this does nothing, and returns ``None``: a space + exists once an instant holds it, so there is nothing to declare. """ + if "times" in domain: + return None collections = domain.setdefault("time_collections", []) if any(c.get("name") == name for c in collections): raise YmfArchiveError( @@ -559,7 +615,7 @@ def _resolve_collection(domain: Dict[str, Any], collection: Any) -> Dict[str, An collections = domain.get("time_collections") if not collections: raise YmfArchiveError( - "domain has no time collections; build it with new_domain()" + "domain has no time collections; build it with new_domain(name)" ) if isinstance(collection, int): try: @@ -578,6 +634,33 @@ def _resolve_collection(domain: Dict[str, Any], collection: Any) -> Dict[str, An ) +def _add_to_space(domain: Dict[str, Any], time: float, collection: Any, + grid_like: Dict[str, Any]) -> Dict[str, Any]: + """Set ``collection``'s value at ``time`` in a format 2 domain. + + Joins the last instant if it is at that time and lacks the space, else + starts one, which is how a writer that goes space by space within a + step builds the instant. Returns the instant. + """ + if collection is None: + collection = _DEFAULT_TIME_COLLECTION_NAME + elif isinstance(collection, int): + names = space_names(domain) + if not -len(names) <= collection < len(names): + raise YmfArchiveError( + "domain has %d space(s); no index %d. On a format 2 domain " + "a space is named." % (len(names), collection)) + collection = names[collection] + t = float(time) + times = domain.setdefault("times", []) + # A writer fills an instant space by space, so the instant to join is the + # last one; a space already present there means this is a new instant. + if times and times[-1]["time"] == t and collection not in times[-1]["spaces"]: + times[-1]["spaces"][collection] = grid_like + return times[-1] + return add_instant(domain, t, {collection: grid_like}) + + def add_uniform_step( domain: Dict[str, Any], time: float, @@ -585,17 +668,21 @@ def add_uniform_step( geom: Dict[str, Any], attributes: Optional[Iterable[Dict[str, Any]]] = None, *, - collection: Any = 0, + collection: Any = None, ) -> Dict[str, Any]: """Append a timestep holding a single grid. Returns the new step. ``collection`` selects which time collection to append to, by index or by name; it defaults to the first, which is the only one in a - single-mesh archive. + single-mesh archive. On a format 2 domain it names the space, and the + step joins the instant at ``time``. """ - step: Dict[str, Any] = {"time": float(time), "topology": topo, "geometry": geom} - step["attributes"] = list(attributes) if attributes is not None else [] - _resolve_collection(domain, collection)["data"].append(step) + grid_like = uniform_grid(topo, geom, attributes) + if "times" in domain: + _add_to_space(domain, time, collection, grid_like) + return grid_like + step: Dict[str, Any] = {"time": float(time), **grid_like} + _resolve_collection(domain, 0 if collection is None else collection)["data"].append(step) return step @@ -604,14 +691,108 @@ def add_spatial_step( time: float, grids: Iterable[Dict[str, Any]], *, - collection: Any = 0, + collection: Any = None, ) -> Dict[str, Any]: """Append a timestep holding one grid per subdomain. Returns the new step.""" - step = {"time": float(time), "spatial_collection": list(grids)} - _resolve_collection(domain, collection)["data"].append(step) + grid_like = spatial_grid(grids) + if "times" in domain: + _add_to_space(domain, time, collection, grid_like) + return grid_like + step = {"time": float(time), **grid_like} + _resolve_collection(domain, 0 if collection is None else collection)["data"].append(step) return step +def space_names(domain: Dict[str, Any]) -> List[str]: + """The spaces of a format 2 domain, in the order they first appear.""" + names: List[str] = [] + seen = set() + for instant in domain.get("times", []): + for name in instant["spaces"]: + if name not in seen: + seen.add(name) + names.append(name) + return names + + +def series(domain: Dict[str, Any], name: str) -> List[Tuple[float, Dict[str, Any]]]: + """``[(time, grid), ...]`` for one space, in instant order. + + The grid is the space's own dict (not a copy), so editing it edits the + domain. + """ + return [(i["time"], i["spaces"][name]) + for i in domain.get("times", []) if name in i["spaces"]] + + +def time_collections(domain: Dict[str, Any]) -> List[Dict[str, Any]]: + """The format 1 view of a format 2 domain: ``[{name, data: [step, ...]}]``. + + For code that thinks one space at a time. A copy of the structure; the + grids inside are shared. A domain still in format 1 is returned as it is. + """ + if "times" not in domain and "time_collections" in domain: + return domain["time_collections"] + return [{"name": name, + "data": [{"time": t, **grid_like} for t, grid_like in series(domain, name)]} + for name in space_names(domain)] + + +class _Joiner: + """Builds instants out of per-space steps, joining on exact time. + + A step is keyed by ``(time, occurrence)``, where ``occurrence`` counts + the earlier steps of the same space at that same time, so a time that + legitimately recurs pairs up correctly. Times are compared exactly: + they are written with ``repr(float)`` and YAML returns them unchanged, + and a tolerance would merge instants that are really distinct. + """ + + def __init__(self) -> None: + self.times: List[Dict[str, Any]] = [] + self._by_key: Dict[Tuple[float, int], Dict[str, Any]] = {} + self._seen: Dict[Tuple[str, float], int] = {} + + def add(self, space: str, step: Dict[str, Any]) -> None: + t = float(step["time"]) + occurrence = self._seen.get((space, t), 0) + self._seen[(space, t)] = occurrence + 1 + instant = self._by_key.get((t, occurrence)) + if instant is None: + instant = {"time": t, "spaces": {}} + self._by_key[(t, occurrence)] = instant + self.times.append(instant) + instant["spaces"][space] = {k: v for k, v in step.items() if k != "time"} + + +def transpose_collections(collections: Iterable[Dict[str, Any]]) -> List[Dict[str, Any]]: + """Turn format 1 time collections into format 2 instants. + + Instants are ordered by time when every collection's own steps are in + non-decreasing time order (a space first archived late then lands among + the instants it belongs to, and equal times keep first-appearance order); + otherwise by first appearance, walking step 0 of every collection, then + step 1, and so on. Equal-length series zip either way. A space missing at + an instant is omitted from it. Collections keep their list order inside + each instant. + """ + collections = [c for c in collections] + joiner = _Joiner() + longest = max((len(c.get("data", [])) for c in collections), default=0) + for i in range(longest): + for c in collections: + data = c.get("data", []) + if i < len(data): + joiner.add(c.get("name", _DEFAULT_TIME_COLLECTION_NAME), data[i]) + order = [c.get("name", _DEFAULT_TIME_COLLECTION_NAME) for c in collections] + for instant in joiner.times: + instant["spaces"] = {n: instant["spaces"][n] for n in order if n in instant["spaces"]} + if all(all(a["time"] <= b["time"] for a, b in zip(c.get("data", []), c.get("data", [])[1:])) + for c in collections): + return sorted(joiner.times, key=lambda instant: instant["time"]) # stable + return joiner.times + + # --------------------------------------------------------------------------- # canonicalization # --------------------------------------------------------------------------- @@ -783,52 +964,75 @@ def fold_archive_keys(obj: Any) -> Any: def canonicalize_domain(domain: Dict[str, Any]) -> Dict[str, Any]: - """Return ``domain`` with every defaultable field filled in explicitly. - - Use this before comparing a hand-written sparse domain against one read - back from a file -- see :func:`_canonicalize_data_item` for why a direct - ``==`` can otherwise fail without anything having been lost. - - The defaults applied are exactly: ``DataItem.Format`` (inferred from - ``Data`` vs ``Include``), ``DataItem.DataType`` (``Float``), - ``DataItem.Precision`` (4, XDMF's default), - ``Attribute.AttributeType`` (``Scalar``), - ``Attribute.Center`` (``Node``), ``Geometry.Type`` (``XYZ``), - ``Attributes`` (empty list), and ``TimeCollections[i].Name``. A singular - ``TimeCollection`` is normalized into a one-element ``TimeCollections``. - Every other - field is required, and ``Topology.NodesPerElement`` is optional-absent + """Return ``domain`` in format 2, with every defaultable field filled in. + + Accepts both shapes and always returns ``{"times": [...]}``: a format 1 + domain (``time_collections``, or the singular ``time_collection``) is + upgraded with :func:`transpose_collections`. Use this before comparing a + hand-written sparse domain against one read back from a file -- see + :func:`_canonicalize_data_item` for why a direct ``==`` can otherwise + fail without anything having been lost. + + The defaults applied are exactly: ``DataItem.format`` (inferred from + ``data`` vs ``include``), ``DataItem.data_type`` (``Float``), + ``DataItem.precision`` (4, XDMF's default), ``attribute_type`` + (``Scalar``), ``center`` (``Node``), ``geometry.type`` (``XYZ``), + ``attributes`` (empty list), and a format 1 collection's ``name``. Every + other field is required, and ``nodes_per_element`` is optional-absent rather than defaulted. """ domain = fold_archive_keys(domain) - collections = domain.get("time_collections") - if collections is None: - single = domain.get("time_collection") - if single is None: - return {} - # Singular form is input sugar; the canonical output is always the list. - collections = [single] - result_collections = [] - seen: Dict[str, int] = {} - for ci, collection in enumerate(collections): - name = collection.get("name", _DEFAULT_TIME_COLLECTION_NAME) - if name in seen: + if "times" in domain: + if "time_collections" in domain or "time_collection" in domain: raise YmfArchiveError( - f"time_collections[{ci}].name: duplicate collection name {name!r} " - f"(also at TimeCollections[{seen[name]}]); names are how a reader " - "tells the meshes apart, so they must be unique" + "domain has both 'times' and 'time_collections'; a domain is " + "in one format or the other") + instants = domain["times"] + else: + collections = domain.get("time_collections") + if collections is None: + single = domain.get("time_collection") + if single is None: + return {} + # Singular form is input sugar; the canonical output is always the list. + collections = [single] + seen: Dict[str, int] = {} + canonical_collections = [] + for ci, collection in enumerate(collections): + name = collection.get("name", _DEFAULT_TIME_COLLECTION_NAME) + if name in seen: + raise YmfArchiveError( + f"time_collections[{ci}].name: duplicate collection name {name!r} " + f"(also at time_collections[{seen[name]}]); names are how a reader " + "tells the meshes apart, so they must be unique" + ) + seen[name] = ci + canonical_collections.append( + { + "name": name, + "data": [ + _canonicalize_step(step, f"time_collections[{ci}].data[{i}]") + for i, step in enumerate(collection.get("data", [])) + ], + } ) - seen[name] = ci - result_collections.append( - { - "name": name, - "data": [ - _canonicalize_step(step, f"time_collections[{ci}].data[{i}]") - for i, step in enumerate(collection.get("data", [])) - ], - } - ) - return {"time_collections": result_collections} + return {"times": transpose_collections(canonical_collections)} + result = [] + for i, instant in enumerate(instants): + where = f"times[{i}]" + if "time" not in instant: + raise YmfArchiveError(f"{where}: instant has no 'time'") + spaces = instant.get("spaces") + if not isinstance(spaces, dict) or not spaces: + raise YmfArchiveError(f"{where}.spaces: must name at least one space") + canonical_spaces = {} + for name, space in spaces.items(): + step = _canonicalize_step(dict(space, time=instant["time"]), + f"{where}.spaces[{name}]") + del step["time"] + canonical_spaces[str(name)] = step + result.append({"time": float(instant["time"]), "spaces": canonical_spaces}) + return {"times": result} # --------------------------------------------------------------------------- @@ -921,16 +1125,16 @@ def validate_domain(domain: Dict[str, Any]) -> Dict[str, Any]: Raises :class:`YmfArchiveError` naming the offending path. """ canonical = canonicalize_domain(domain) - for ci, collection in enumerate(canonical.get("time_collections", [])): - for i, step in enumerate(collection["data"]): - where = f"time_collections[{ci}].data[{i}]" - if "spatial_collection" in step: - if not step["spatial_collection"]: + for i, instant in enumerate(canonical.get("times", [])): + for name, space in instant["spaces"].items(): + where = f"times[{i}].spaces[{name}]" + if "spatial_collection" in space: + if not space["spatial_collection"]: raise YmfArchiveError(f"{where}.spatial_collection: must not be empty") - for j, g in enumerate(step["spatial_collection"]): + for j, g in enumerate(space["spatial_collection"]): _validate_grid_like(g, f"{where}.spatial_collection[{j}]") else: - _validate_grid_like(step, where) + _validate_grid_like(space, where) return canonical @@ -1079,43 +1283,49 @@ def write_ymf( def append_step( path: str | Path, - collection: str, - step: Dict[str, Any], + instant_or_collection: Any, + step: Optional[Dict[str, Any]] = None, *, fsync: bool = False, ) -> None: - """Append one timestep to a ``.ymf`` file, without rewriting it. + """Append one instant to a ``.ymf`` file, without rewriting it. A writer that produces a timestep at a time should not have to rewrite the whole document to record it. Rewriting is O(steps) per step, which is O(steps^2) over a run -- an inheritance from XML, where a document has to be closed to be well formed. YAML has no such requirement: a stream of documents separated by ``---`` is valid after any complete - document, so a step can simply be appended and flushed. + document, so an instant can simply be appended and flushed. The file therefore has two shapes, and :func:`read_ymf` accepts both: - * **one document**, what :func:`write_ymf` produces, with every step - nested under its collection. This is the canonical form and what a - completed run leaves behind. - * **a stream**: that same document first, then one document per step. - This is what a run in progress leaves, and what a run that was killed - leaves for good. + * **one document**, what :func:`write_ymf` produces, with every instant + in ``times``. This is the canonical form and what a completed run + leaves behind. + * **a stream**: that same document first, then one document per + instant. This is what a run in progress leaves, and what a run that + was killed leaves for good. - Each appended document names its collection, rather than indexing into - the header's list, because collections are created lazily -- a model - that first archives at step 40 adds one then -- so an index agreed at - step 0 would be wrong by step 40. Naming them also means the header - need not know them all in advance. + ``instant`` is ``{"time": t, "spaces": {name: grid_or_spatial}}``. It + holds every space known at that instant, so a record names no collection + and a space first archived at step 40 is just a key that first appears + in the 41st record. An instant is also the unit of atomicity: a run + killed mid-write loses a whole instant or none of it, never one space of + it. - ``step`` is the dict :func:`add_uniform_step` and :func:`add_spatial_step` - put in a collection's ``data``. + The format 1 call, ``append_step(path, collection, step)``, still works + and appends a record for one space at one time; :func:`read_ymf` joins + such records into instants on read. ``fsync`` forces the write to disk rather than to the page cache. It is off by default: a flush already survives the process dying, which is the case this exists for, and only a machine losing power needs more. """ - text = _dump({"collection": collection, "step": step}) + if step is None: + record = {"step": instant_or_collection} + else: + record = {"collection": instant_or_collection, "step": step} + text = _dump(record) with open(path, "a", encoding="utf-8") as stream: # One write, so a concurrent reader sees a whole record or none. The # trailing "..." is YAML's explicit document-end marker, and it is @@ -1132,15 +1342,16 @@ def append_step( def read_ymf(path: str | Path) -> Tuple[Dict[str, Any], Optional[Any]]: """Read a ``.ymf`` document written by :func:`write_ymf`. - Returns a ``(domain, extra)`` tuple. Raises :class:`YmfArchiveError` on + Returns a ``(domain, extra)`` tuple, the domain always in format 2 + (a format 1 document is upgraded; see :func:`transpose_collections`). Raises :class:`YmfArchiveError` on a format-version mismatch, so an archive from an incompatible writer fails with a clear message rather than a confusing downstream ``KeyError``. Reads both shapes described in :func:`append_step`: a single document, - or that document followed by one per appended step. Appended steps are - folded into their named collection, which is created if the header did - not have it. + or that document followed by one per appended step. An appended instant + is added to ``times``; a format 1 record, which names its collection, is + joined into the instant at its time. A trailing partial document -- a run killed mid-write -- is dropped, with every complete step before it kept. That is the whole point of @@ -1154,19 +1365,27 @@ def read_ymf(path: str | Path) -> Tuple[Dict[str, Any], Optional[Any]]: if not isinstance(document, dict): raise YmfArchiveError(f"{path}: expected a YAML mapping at the top level") version = document.get("ymf_archive_version") - if version != ARCHIVE_FORMAT_VERSION: + if version not in _READABLE_FORMAT_VERSIONS: raise YmfArchiveError( f"{path}: archive format version {version!r}, but this ymf writes " - f"and reads version {ARCHIVE_FORMAT_VERSION}" + f"version {ARCHIVE_FORMAT_VERSION} and reads " + f"{', '.join(str(v) for v in _READABLE_FORMAT_VERSIONS)}" ) - domain = document.get("domain", {}) + domain = _upgrade_domain(document.get("domain", {})) for record in documents[1:]: if not isinstance(record, dict) or "step" not in record: raise YmfArchiveError( f"{path}: appended document is not a step record; got " f"{sorted(record) if isinstance(record, dict) else type(record).__name__}" ) - _fold_step(domain, record["collection"], record["step"]) + if "collection" in record: + _fold_step(domain, record["collection"], record["step"]) + elif version == 1: + raise YmfArchiveError( + f"{path}: a format 1 document holds records that name a " + "collection; this one holds an instant") + else: + domain.setdefault("times", []).append(fold_archive_keys(record["step"])) if truncated: warnings.warn( f"{path}: the last appended step is incomplete and was dropped " @@ -1178,14 +1397,36 @@ def read_ymf(path: str | Path) -> Tuple[Dict[str, Any], Optional[Any]]: return domain, document.get("extra") +def _upgrade_domain(domain: Any) -> Dict[str, Any]: + """A domain in format 2 shape, without canonicalizing it. + + Cheaper than :func:`canonicalize_domain`, which :func:`read_ymf` leaves + to the caller as it always has: this only moves format 1's steps into + instants. + """ + domain = fold_archive_keys(domain) + if not isinstance(domain, dict) or "times" in domain: + return domain + collections = domain.get("time_collections") + if collections is None and domain.get("time_collection") is not None: + collections = [domain["time_collection"]] + if collections is None: + return domain + return dict({k: v for k, v in domain.items() + if k not in ("time_collections", "time_collection")}, + times=transpose_collections(collections)) + + def _fold_step(domain: Dict[str, Any], collection: str, step: Dict[str, Any]) -> None: - """Put an appended step into its collection, creating it if need be.""" - collections = domain.setdefault("time_collections", []) - for existing in collections: - if existing.get("name") == collection: - existing.setdefault("data", []).append(step) - return - collections.append({"name": collection, "data": [step]}) + """Put a record that names its collection into the instant at its time. + + The format 1 record: one space at one time. It joins the last instant if + that is at the same time and lacks the space, as a writer that goes + space by space within a step produces them, and otherwise starts one. + """ + step = fold_archive_keys(step) + grid_like = {k: v for k, v in step.items() if k != "time"} + _add_to_space(domain, step["time"], collection, grid_like) def _load_stream(text: str) -> Tuple[List[Any], bool]: @@ -1231,11 +1472,15 @@ def load_array(item: Dict[str, Any], base_dir: str | Path = "."): def _data_items(domain: Dict[str, Any]): - """Every DataItem dict of a canonical domain, with a readable path.""" - for collection in domain.get("time_collections", []): - for i, step in enumerate(collection["data"]): - for j, g in enumerate(step.get("spatial_collection", [step])): - where = "%s/%d%s" % (collection["name"], i, "" if "topology" in step else "/%d" % j) + """Every DataItem dict of a canonical domain, with a readable path. + + Paths are ``//...`` where ``n`` counts that space's own + steps, so they do not depend on which other spaces an instant holds. + """ + for name in space_names(domain): + for i, (_, space) in enumerate(series(domain, name)): + for j, g in enumerate(space.get("spatial_collection", [space])): + where = "%s/%d%s" % (name, i, "" if "topology" in space else "/%d" % j) yield where + "/Topology", g["topology"] yield where + "/Geometry", g["geometry"] for a in g["attributes"]: @@ -1305,31 +1550,32 @@ def select_domain( """ import copy - all_collections = domain.get("time_collections", []) - names = [c.get("name", "") for c in all_collections] + domain = canonicalize_domain(domain) + names = space_names(domain) + chosen_names = names if collections is not None: - keep = [] + chosen_names = [] for want in collections: - matched = [c for c in all_collections - if c.get("name", "") == want or want in c.get("name", "")] + matched = [n for n in names if n == want or want in n] if not matched: raise YmfArchiveError( "no collection matches %r; the archive holds %s" % (want, names)) - for c in matched: - if c not in keep: - keep.append(c) - all_collections = keep + for n in matched: + if n not in chosen_names: + chosen_names.append(n) + # keep the archive's own order of spaces, not the order asked for + chosen_names = [n for n in names if n in chosen_names] - def grids_of(step: Dict[str, Any]): - if "spatial_collection" in step: - return step["spatial_collection"] - return [step] + def grids_of(space: Dict[str, Any]): + if "spatial_collection" in space: + return space["spatial_collection"] + return [space] if fields is not None: present = set() - for c in all_collections: - for step in c.get("data", []): - for g in grids_of(step): + for instant in domain.get("times", []): + for n in chosen_names: + for g in grids_of(instant["spaces"].get(n, {"attributes": []})): present.update(a["name"] for a in g.get("attributes", [])) missing = [f for f in fields if f not in present] if missing: @@ -1337,21 +1583,27 @@ def grids_of(step: Dict[str, Any]): "no field named %s; the archive holds %s" % (", ".join(repr(f) for f in missing), sorted(present))) - chosen = [] - for c in all_collections: - data = c.get("data", []) - indices = (step_indices(steps, len(data)) if steps is not None - else list(range(len(data)))) - new_steps = [] - for i in indices: - step = copy.deepcopy(data[i]) + # An instant that holds none of the chosen spaces is not in the selection, + # and `steps` then counts the instants that remain. + kept = [i for i in domain.get("times", []) + if any(n in i["spaces"] for n in chosen_names)] + indices = (step_indices(steps, len(kept)) if steps is not None + else list(range(len(kept)))) + new_times = [] + for k in indices: + instant = kept[k] + spaces = {} + for n in chosen_names: + if n not in instant["spaces"]: + continue + space = copy.deepcopy(instant["spaces"][n]) if fields is not None: - for g in grids_of(step): + for g in grids_of(space): g["attributes"] = [a for a in g.get("attributes", []) if a["name"] in fields] - new_steps.append(step) - chosen.append(dict(c, data=new_steps)) - return dict(domain, time_collections=chosen) + spaces[n] = space + new_times.append({"time": instant["time"], "spaces": spaces}) + return dict(domain, times=new_times) def inline_domain(domain: Dict[str, Any], base_dir: str | Path = ".") -> Dict[str, Any]: diff --git a/src/ymf/cli.py b/src/ymf/cli.py index bec8de7..f01748f 100644 --- a/src/ymf/cli.py +++ b/src/ymf/cli.py @@ -22,6 +22,7 @@ from typing import List, Optional, Sequence from ymf.archive import (YmfArchiveError, canonicalize_domain, read_ymf, select_domain, + space_names, time_collections, validate_domain) from ymf.xdmf import write_xdmf @@ -150,7 +151,7 @@ def describe(domain) -> List[str]: Each collection is numbered so the interactive prompt can refer to it. """ lines = [] - for i, collection in enumerate(domain.get("time_collections", []), start=1): + for i, collection in enumerate(time_collections(canonicalize_domain(domain)), start=1): steps = collection["data"] times = [step["time"] for step in steps] span = ("no steps" if not times @@ -203,7 +204,7 @@ def interactive_selection(domain, ask=None): pass their own. """ ask = ask or input - collections = domain.get("time_collections", []) + collections = time_collections(canonicalize_domain(domain)) names = [c["name"] for c in collections] print("Collections (blank for all):") chosen_names = _pick("collections> ", names, ask) @@ -247,12 +248,12 @@ def ymf2xmf_main(argv: Optional[Sequence[str]] = None) -> int: parser.add_argument("--show", action="store_true", help="describe what the archive holds and stop; convert nothing") parser.add_argument("-c", "--collection", action="append", metavar="NAME", - help="convert only this time collection, by name or part of one " + help="convert only this space (time collection), by name or part of one " "(repeatable; default: all)") parser.add_argument("-f", "--field", action="append", metavar="NAME", help="convert only this field (repeatable; default: all)") parser.add_argument("-s", "--steps", metavar="SPEC", - help="convert only these steps: indices and slices, e.g. -1, " + help="convert only these instants: indices and slices, e.g. -1, " "0:10, ::5 (default: all)") parser.add_argument("-i", "--interactive", action="store_true", help="ask which collections, fields and steps to convert") @@ -274,6 +275,12 @@ def ymf2xmf_main(argv: Optional[Sequence[str]] = None) -> int: domain = canonicalize_domain(domain) print("%s" % (source,)) if not args.interactive: + times = [i["time"] for i in domain.get("times", [])] + if times: + print(" %d instant%s, t=%g..%g, %d space%s" + % (len(times), "" if len(times) == 1 else "s", + times[0], times[-1], len(space_names(domain)), + "" if len(space_names(domain)) == 1 else "s")) for line in describe(domain): print(line) if args.show: diff --git a/src/ymf/xdmf.py b/src/ymf/xdmf.py index d439973..5ab3fac 100644 --- a/src/ymf/xdmf.py +++ b/src/ymf/xdmf.py @@ -69,7 +69,8 @@ from xml.etree.ElementTree import (Element, ElementTree, ParseError, SubElement, fromstring, parse as et_parse) -from ymf.archive import YmfArchiveError, canonicalize_domain +from ymf.archive import (YmfArchiveError, canonicalize_domain, + time_collections, transpose_collections) XDMF_HEADER = b'\n\n' @@ -516,7 +517,9 @@ def build_xdmf_tree( # Canonicalize up front so this function reads one well-defined shape # rather than re-deriving defaults inline at every use. canonical = canonicalize_domain(domain) - for time_collection in canonical.get("time_collections", []): + # XDMF nests the other way round -- a temporal collection per grid type -- + # so the spaces inside each instant come back out as one collection each. + for time_collection in time_collections(canonical): collection_elem = SubElement( domain_elem, "Grid", @@ -661,7 +664,7 @@ def parse_xdmf_domain(root: Element) -> Dict[str, Any]: "data": steps, }) - return {"time_collections": collections} if collections else {} + return {"times": transpose_collections(collections)} if collections else {} def parse_xdmf_extra(root: Element) -> Optional[Any]: diff --git a/tests/fixtures/format1_two_spaces.xmf b/tests/fixtures/format1_two_spaces.xmf new file mode 100644 index 0000000..340e891 --- /dev/null +++ b/tests/fixtures/format1_two_spaces.xmf @@ -0,0 +1,82 @@ + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/tests/fixtures/format1_two_spaces.ymf b/tests/fixtures/format1_two_spaces.ymf new file mode 100644 index 0000000..b52977c --- /dev/null +++ b/tests/fixtures/format1_two_spaces.ymf @@ -0,0 +1,175 @@ +ymf_archive_version: 1 +domain: + time_collections: + - name: Mesh + data: + - time: 0.0 + topology: + type: Triangle + number_of_elements: 1 + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [1, 3] + data: f.h5:/e + geometry: + type: XYZ + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [3, 2] + data: f.h5:/n + attributes: + - name: u + attribute_type: Scalar + center: Node + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [3] + data: f.h5:/u0 + - time: 0.5 + topology: + type: Triangle + number_of_elements: 1 + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [1, 3] + data: f.h5:/e + geometry: + type: XYZ + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [3, 2] + data: f.h5:/n + attributes: + - name: u + attribute_type: Scalar + center: Node + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [3] + data: f.h5:/u0.5 + - time: 1.0 + topology: + type: Triangle + number_of_elements: 1 + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [1, 3] + data: f.h5:/e + geometry: + type: XYZ + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [3, 2] + data: f.h5:/n + attributes: + - name: u + attribute_type: Scalar + center: Node + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [3] + data: f.h5:/u1 + - name: Mesh_c0p2 + data: + - time: 0.0 + topology: + type: Triangle_6 + number_of_elements: 1 + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [1, 6] + data: f.h5:/e6 + geometry: + type: XYZ + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [6, 2] + data: f.h5:/n6 + attributes: + - name: u + attribute_type: Scalar + center: Node + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [6] + data: f.h5:/v0 + - time: 0.5 + topology: + type: Triangle_6 + number_of_elements: 1 + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [1, 6] + data: f.h5:/e6 + geometry: + type: XYZ + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [6, 2] + data: f.h5:/n6 + attributes: + - name: u + attribute_type: Scalar + center: Node + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [6] + data: f.h5:/v0.5 + - time: 1.0 + topology: + type: Triangle_6 + number_of_elements: 1 + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [1, 6] + data: f.h5:/e6 + geometry: + type: XYZ + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [6, 2] + data: f.h5:/n6 + attributes: + - name: u + attribute_type: Scalar + center: Node + data_item: + format: HDF + data_type: Float + precision: 4 + dimensions: [6] + data: f.h5:/v1 diff --git a/tests/test_archive.py b/tests/test_archive.py index 26755f7..6159c74 100644 --- a/tests/test_archive.py +++ b/tests/test_archive.py @@ -5,6 +5,7 @@ import pytest +from ymf.archive import time_collections # noqa: E402 from ymf.archive import ( domain_arrays, inline_domain, @@ -231,10 +232,10 @@ def test_canonicalize_fills_every_documented_default(): } } c = canonicalize_domain(sparse) - # the singular input form normalizes into the canonical list form - assert list(c) == ["time_collections"] - assert len(c["time_collections"]) == 1 - tc = c["time_collections"][0] + # the singular format 1 input form is upgraded to format 2 + assert list(c) == ["times"] + assert len(time_collections(c)) == 1 + tc = time_collections(c)[0] assert tc["name"] == "TimeCollection" step = tc["data"][0] assert step["geometry"]["type"] == "XYZ" @@ -253,7 +254,7 @@ def test_canonicalize_supplies_an_empty_attributes_list(): topology("Triangle", 1, data_item([1, 3], "f.h5:/e")), geometry(data_item([3, 2], "f.h5:/n")), ) - assert canonicalize_domain(domain)["time_collections"][0]["data"][0]["attributes"] == [] + assert time_collections(canonicalize_domain(domain))[0]["data"][0]["attributes"] == [] def test_canonicalize_of_an_empty_domain_is_empty(): @@ -312,7 +313,7 @@ def test_validate_returns_the_canonicalized_document(): def test_validate_rejects_an_unknown_topology_type(): domain = make_uniform_domain() - domain["time_collections"][0]["data"][0]["topology"]["type"] = "Dodecahedron" + time_collections(domain)[0]["data"][0]["topology"]["type"] = "Dodecahedron" with pytest.raises(YmfArchiveError, match="not a known XDMF topology type"): validate_domain(domain) @@ -332,21 +333,21 @@ def test_validate_accepts_every_topology_type_proteus_emits(): def test_validate_rejects_a_bad_centering(): domain = make_uniform_domain() - domain["time_collections"][0]["data"][0]["attributes"][0]["center"] = "Vertex" + time_collections(domain)[0]["data"][0]["attributes"][0]["center"] = "Vertex" with pytest.raises(YmfArchiveError, match=r"attributes\[0\]\.center"): validate_domain(domain) def test_validate_rejects_a_bad_attribute_type(): domain = make_uniform_domain() - domain["time_collections"][0]["data"][0]["attributes"][0]["attribute_type"] = "Scalarr" + time_collections(domain)[0]["data"][0]["attributes"][0]["attribute_type"] = "Scalarr" with pytest.raises(YmfArchiveError, match=r"attributes\[0\]\.attribute_type"): validate_domain(domain) def test_validate_rejects_a_bad_data_type(): domain = make_uniform_domain() - domain["time_collections"][0]["data"][0]["attributes"][0]["data_item"]["data_type"] = "Double" + time_collections(domain)[0]["data"][0]["attributes"][0]["data_item"]["data_type"] = "Double" with pytest.raises(YmfArchiveError, match="data_type"): validate_domain(domain) @@ -355,7 +356,7 @@ def test_validate_rejects_duplicate_attribute_names_within_one_grid(): # Viewers key fields by name within a grid, so a duplicate silently # shadows rather than erroring at read time -- worth catching on write. domain = make_uniform_domain() - attrs = domain["time_collections"][0]["data"][0]["attributes"] + attrs = time_collections(domain)[0]["data"][0]["attributes"] attrs[1]["name"] = attrs[0]["name"] attrs[1]["center"] = attrs[0]["center"] with pytest.raises(YmfArchiveError, match="duplicate .*-centred attribute name"): @@ -366,7 +367,7 @@ def test_validate_allows_the_same_name_on_different_centerings(): # Point data and cell data are separate namespaces in a viewer; an # exporter writing T both cell- and node-centred is legitimate. domain = make_uniform_domain() - attrs = domain["time_collections"][0]["data"][0]["attributes"] + attrs = time_collections(domain)[0]["data"][0]["attributes"] attrs[1]["name"] = attrs[0]["name"] attrs[0]["center"], attrs[1]["center"] = "Cell", "Node" validate_domain(domain) @@ -386,14 +387,14 @@ def test_validate_rejects_an_empty_spatial_collection(): def test_validate_rejects_empty_dimensions(): domain = make_uniform_domain() - domain["time_collections"][0]["data"][0]["attributes"][0]["data_item"]["dimensions"] = [] + time_collections(domain)[0]["data"][0]["attributes"][0]["data_item"]["dimensions"] = [] with pytest.raises(YmfArchiveError, match="must not be empty"): validate_domain(domain) def test_validate_rejects_a_negative_dimension(): domain = make_uniform_domain() - domain["time_collections"][0]["data"][0]["attributes"][0]["data_item"]["dimensions"] = [-1] + time_collections(domain)[0]["data"][0]["attributes"][0]["data_item"]["dimensions"] = [-1] with pytest.raises(YmfArchiveError, match="non-negative"): validate_domain(domain) @@ -436,7 +437,7 @@ def test_text_include_data_items_round_trip_through_yaml(tmp_path): write_ymf(domain, path) read_domain, _ = read_ymf(path) assert read_domain == canonicalize_domain(domain) - assert read_domain["time_collections"][0]["data"][0]["attributes"][0]["data_item"]["include"] == "./d/u.txt" + assert time_collections(read_domain)[0]["data"][0]["attributes"][0]["data_item"]["include"] == "./d/u.txt" def test_extra_payload_round_trips_and_is_stored_readably(tmp_path): @@ -485,17 +486,17 @@ def test_read_ymf_rejects_a_non_mapping_document(tmp_path): def test_write_ymf_validates_by_default(tmp_path): domain = make_uniform_domain() - domain["time_collections"][0]["data"][0]["attributes"][0]["center"] = "Nowhere" + time_collections(domain)[0]["data"][0]["attributes"][0]["center"] = "Nowhere" with pytest.raises(YmfArchiveError, match="center"): write_ymf(domain, tmp_path / "archive.ymf") def test_write_ymf_can_skip_validation_deliberately(tmp_path): domain = make_uniform_domain() - domain["time_collections"][0]["data"][0]["attributes"][0]["center"] = "Nowhere" + time_collections(domain)[0]["data"][0]["attributes"][0]["center"] = "Nowhere" path = tmp_path / "archive.ymf" write_ymf(domain, path, validate=False) - assert read_ymf(path)[0]["time_collections"][0]["data"][0]["attributes"][0]["center"] == "Nowhere" + assert time_collections(read_ymf(path)[0])[0]["data"][0]["attributes"][0]["center"] == "Nowhere" # -------------------------------------------------------------------------- @@ -504,7 +505,7 @@ def test_write_ymf_can_skip_validation_deliberately(tmp_path): def test_a_single_grid_fragment_round_trips_as_a_string(): - g = make_spatial_domain()["time_collections"][0]["data"][0]["spatial_collection"][0] + g = time_collections(make_spatial_domain())[0]["data"][0]["spatial_collection"][0] assert load_grid(dump_grid(g)) == g @@ -513,13 +514,13 @@ def test_grid_fragments_reassemble_into_a_spatial_step(): # to a string, the strings are collected, and the master reassembles. original = make_spatial_domain(n_ranks=4) fragments = [ - dump_grid(g) for g in original["time_collections"][0]["data"][0]["spatial_collection"] + dump_grid(g) for g in time_collections(original)[0]["data"][0]["spatial_collection"] ] - rebuilt = new_domain(original["time_collections"][0]["name"]) + rebuilt = new_domain(time_collections(original)[0]["name"]) add_spatial_step(rebuilt, 0.0, [load_grid(f) for f in fragments]) assert ( - canonicalize_domain(rebuilt)["time_collections"][0]["data"][0] - == canonicalize_domain(original)["time_collections"][0]["data"][0] + time_collections(canonicalize_domain(rebuilt))[0]["data"][0] + == time_collections(canonicalize_domain(original))[0]["data"][0] ) @@ -567,14 +568,14 @@ def _two_space_domain(): def test_new_domain_starts_with_one_named_collection(): domain = new_domain("Mesh Spatial_Domain") - assert [c["name"] for c in domain["time_collections"]] == ["Mesh Spatial_Domain"] + assert [c["name"] for c in time_collections(domain)] == ["Mesh Spatial_Domain"] def test_add_collection_appends_a_second_mesh(): domain = _two_space_domain() - names = [c["name"] for c in domain["time_collections"]] + names = [c["name"] for c in time_collections(domain)] assert names == ["Mesh Spatial_Domain", "Mesh_c0p2_Lagrange"] - assert [len(c["data"]) for c in domain["time_collections"]] == [2, 2] + assert [len(c["data"]) for c in time_collections(domain)] == [2, 2] def test_add_collection_rejects_a_duplicate_name(): @@ -585,12 +586,12 @@ def test_add_collection_rejects_a_duplicate_name(): def test_steps_can_target_a_collection_by_name_or_index(): domain = _two_space_domain() - by_name = domain["time_collections"][1]["data"] + by_name = time_collections(domain)[1]["data"] assert all(step["topology"]["type"] == "Tri_6" for step in by_name) # index 0 is the default target assert all( step["topology"]["type"] == "Triangle" - for step in domain["time_collections"][0]["data"] + for step in time_collections(domain)[0]["data"] ) @@ -694,7 +695,7 @@ def test_validate_refuses_inline_values_under_another_format(): topology("Polyvertex", 1, data_item([1], values=[0], data_type="Int"), nodes_per_element=1), geometry(data_item([1, 3], values=[0.0, 0.0, 0.0]))) - domain["time_collections"][0]["data"][0]["geometry"]["data_item"]["format"] = "HDF" + time_collections(domain)[0]["data"][0]["geometry"]["data_item"]["format"] = "HDF" with pytest.raises(YmfArchiveError, match=r"geometry\.data_item: inline 'values' need format='XML'"): validate_domain(domain) @@ -743,7 +744,7 @@ def test_inline_domain_copies_hdf5_arrays_in_exactly(tmp_path): [attribute("u", data_item_for(u, "m.h5:/u"))]) referenced = domain_arrays(domain, tmp_path) inline = inline_domain(domain, tmp_path) - assert all("values" in a["data_item"] for a in inline["time_collections"][0]["data"][0]["attributes"]) + assert all("values" in a["data_item"] for a in time_collections(inline)[0]["data"][0]["attributes"]) copied = domain_arrays(inline) # no base directory needed any more assert set(copied) == set(referenced) for key in referenced: @@ -765,7 +766,7 @@ def _one_step(time): "data_item": data_item([4, 3], "o.h5:/e", data_type="Int")}, {"type": "XYZ", "data_item": data_item([6, 3], "o.h5:/n")}, [], collection=0) - return domain["time_collections"][0]["data"][0] + return time_collections(domain)[0]["data"][0] def _archive(tmp_path, name="a.ymf"): @@ -776,7 +777,7 @@ def _archive(tmp_path, name="a.ymf"): def _times(domain): return {c["name"]: [s["time"] for s in c["data"]] - for c in domain["time_collections"]} + for c in time_collections(domain)} def test_appended_steps_are_read_back(tmp_path): @@ -815,7 +816,7 @@ def test_appending_matches_writing_the_whole_document(tmp_path): for t in (0.0, 0.5): step = _one_step(t) append_step(appended, "Mesh Spatial_Domain", step) - whole["time_collections"][0]["data"].append(step) + time_collections(whole)[0]["data"].append(step) written = tmp_path / "written.ymf" write_ymf(whole, written) assert read_ymf(appended)[0] == read_ymf(written)[0] @@ -858,7 +859,8 @@ def test_a_kill_during_the_very_first_step_leaves_the_header(tmp_path): stream.write("---\ncollection: X\nstep:\n time: 0.0\n topo") with pytest.warns(RuntimeWarning): domain, _ = read_ymf(path) - assert _times(domain) == {"Mesh Spatial_Domain": []} + # a space exists once an instant holds it; the header held none + assert _times(domain) == {} def test_a_complete_archive_does_not_warn(tmp_path): diff --git a/tests/test_cli.py b/tests/test_cli.py index 7abf609..0ab2fdd 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -6,6 +6,7 @@ import pytest +from ymf.archive import time_collections # noqa: E402 from ymf.archive import ( YmfArchiveError, add_spatial_step, @@ -91,7 +92,7 @@ def test_the_conversion_preserves_every_dataset_reference(archive): """ domain, _ = read_ymf(archive) expected = [] - for collection in domain["time_collections"]: + for collection in time_collections(domain): for step in collection["data"]: for g in step.get("spatial_collection", [step]): expected.append(g["topology"]["data_item"]["data"]) @@ -139,7 +140,7 @@ def test_an_empty_archive_is_refused(tmp_path): def test_a_malformed_archive_is_refused_by_default(tmp_path): domain = make_domain() - domain["time_collections"][0]["data"][0]["attributes"][0]["center"] = "Nowhere" + time_collections(domain)[0]["data"][0]["attributes"][0]["center"] = "Nowhere" path = tmp_path / "bad.ymf" write_ymf(domain, path, validate=False) with pytest.raises(YmfArchiveError, match="center"): @@ -148,7 +149,7 @@ def test_a_malformed_archive_is_refused_by_default(tmp_path): def test_validation_can_be_skipped_deliberately(tmp_path): domain = make_domain() - domain["time_collections"][0]["data"][0]["attributes"][0]["center"] = "Nowhere" + time_collections(domain)[0]["data"][0]["attributes"][0]["center"] = "Nowhere" path = tmp_path / "bad.ymf" write_ymf(domain, path, validate=False) assert ymf2xmf(path, validate=False).exists() @@ -188,7 +189,7 @@ def test_a_missing_file_is_a_usage_error(tmp_path): def test_a_malformed_archive_exits_nonzero_without_a_traceback(tmp_path, capsys): """A bad input file is the user's problem to see, not a stack trace.""" domain = make_domain() - domain["time_collections"][0]["data"][0]["attributes"][0]["center"] = "Nowhere" + time_collections(domain)[0]["data"][0]["attributes"][0]["center"] = "Nowhere" path = tmp_path / "bad.ymf" write_ymf(domain, path, validate=False) assert ymf2xmf_main([str(path)]) == 1 @@ -197,7 +198,7 @@ def test_a_malformed_archive_exits_nonzero_without_a_traceback(tmp_path, capsys) def test_the_cli_skips_validation_when_asked(tmp_path): domain = make_domain() - domain["time_collections"][0]["data"][0]["attributes"][0]["center"] = "Nowhere" + time_collections(domain)[0]["data"][0]["attributes"][0]["center"] = "Nowhere" path = tmp_path / "bad.ymf" write_ymf(domain, path, validate=False) assert ymf2xmf_main([str(path), "--no-validate"]) == 0 diff --git a/tests/test_closure.py b/tests/test_closure.py index 1c7182b..576597a 100644 --- a/tests/test_closure.py +++ b/tests/test_closure.py @@ -5,6 +5,7 @@ import pytest from ymf import closure +from ymf.archive import time_collections # noqa: E402 from ymf.archive import (YmfArchiveError, add_uniform_step, data_item, geometry, new_domain, read_document, topology) @@ -155,14 +156,14 @@ def test_inherited_outputs_still_point_at_their_files(tmp_path): (key, _, r), = closure.planned_outputs(spec) output = fake_output(r) domain = canonicalize_domain(output["approximation"]) - domain["time_collections"][0]["data"][0]["geometry"]["data_item"] = data_item( + time_collections(domain)[0]["data"][0]["geometry"]["data_item"] = data_item( [2, 2], "poisson_x.h5:/nodes") output["approximation"] = domain closure.write(base / "poisson.archive.ymf", spec, {key: output}) child = tmp_path / "child.ymf" child.write_text("extends: base/poisson.archive.ymf\n", encoding="utf-8") _, kept, _ = closure.load(child) - step = kept[key]["approximation"]["time_collections"][0]["data"][0] + step = time_collections(kept[key]["approximation"])[0]["data"][0] assert step["geometry"]["data_item"]["data"] == "base/poisson_x.h5:/nodes" diff --git a/tests/test_nesting.py b/tests/test_nesting.py new file mode 100644 index 0000000..38e7b16 --- /dev/null +++ b/tests/test_nesting.py @@ -0,0 +1,266 @@ +"""Time outside, space inside: format 2 of the archive domain. + +See docs/design-time-outside-space.md. The format 1 fixture was written by +ymf before the change (two spaces, three instants), with the .xmf it gave. +""" + +from pathlib import Path + +import pytest + +from ymf.archive import ( + ARCHIVE_FORMAT_VERSION, YmfArchiveError, add_collection, add_instant, + add_uniform_step, append_step, attribute, canonicalize_domain, data_item, + geometry, new_domain, read_ymf, select_domain, series, space_names, + spatial_grid, time_collections, topology, transpose_collections, + uniform_grid, validate_domain, write_ymf, +) +from ymf.xdmf import build_xdmf_tree, parse_xdmf_domain, write_xdmf + +FIXTURES = Path(__file__).parent / "fixtures" + + +def grid_at(name, t): + return uniform_grid( + topology("Triangle", 1, data_item([1, 3], "f.h5:/e")), + geometry(data_item([3, 2], "f.h5:/n")), + [attribute("u", data_item([3], "f.h5:/%s_%g" % (name, t)))]) + + +def collection(name, times): + return {"name": name, "data": [dict({"time": t}, **grid_at(name, t)) for t in times]} + + +def test_format_is_2(): + assert ARCHIVE_FORMAT_VERSION == 2 + + +def test_equal_series_zip(): + times = transpose_collections([collection("A", [0.0, 0.5]), collection("B", [0.0, 0.5])]) + assert [i["time"] for i in times] == [0.0, 0.5] + assert [list(i["spaces"]) for i in times] == [["A", "B"], ["A", "B"]] + + +def test_a_space_first_archived_late_is_absent_from_early_instants(): + times = transpose_collections([collection("A", [0.0, 0.5, 1.0]), collection("B", [1.0])]) + assert [list(i["spaces"]) for i in times] == [["A"], ["A"], ["A", "B"]] + + +def test_a_repeated_time_pairs_by_occurrence(): + times = transpose_collections([collection("A", [0.0, 1.0, 1.0]), collection("B", [0.0, 1.0, 1.0])]) + assert [i["time"] for i in times] == [0.0, 1.0, 1.0] + assert all(list(i["spaces"]) == ["A", "B"] for i in times) + + +def test_close_but_unequal_times_are_distinct_instants(): + times = transpose_collections([collection("A", [0.1 + 0.2]), collection("B", [0.3])]) + assert len(times) == 2 + + +def test_empty_and_single_step_collections(): + assert transpose_collections([collection("A", [])]) == [] + assert len(transpose_collections([collection("A", [2.0])])) == 1 + + +def test_space_order_is_the_order_given(): + times = transpose_collections([collection("Z", [0.0]), collection("A", [0.0])]) + assert list(times[0]["spaces"]) == ["Z", "A"] + + +def test_the_format_1_fixture_reads_as_format_2(): + domain, _ = read_ymf(FIXTURES / "format1_two_spaces.ymf") + assert list(domain) == ["times"] + assert [i["time"] for i in domain["times"]] == [0.0, 0.5, 1.0] + assert space_names(domain) == ["Mesh", "Mesh_c0p2"] + + +def test_format_1_converts_to_the_same_xmf_it_always_did(tmp_path): + domain, _ = read_ymf(FIXTURES / "format1_two_spaces.ymf") + write_xdmf(domain, tmp_path / "out.xmf") + assert (tmp_path / "out.xmf").read_text() == (FIXTURES / "format1_two_spaces.xmf").read_text() + + +def test_xmf_round_trips_through_the_transpose(tmp_path): + domain, _ = read_ymf(FIXTURES / "format1_two_spaces.ymf") + write_xdmf(domain, tmp_path / "a.xmf") + from ymf.xdmf import read_xdmf + again, _ = read_xdmf(tmp_path / "a.xmf") + assert again == canonicalize_domain(domain) + + +def test_write_then_read_is_format_2_and_equal(tmp_path): + domain, _ = read_ymf(FIXTURES / "format1_two_spaces.ymf") + write_ymf(domain, tmp_path / "x.ymf") + assert (tmp_path / "x.ymf").read_text().startswith("ymf_archive_version: 2\n") + again, _ = read_ymf(tmp_path / "x.ymf") + assert again == canonicalize_domain(domain) + + +def test_the_format_1_api_still_builds_something_that_upgrades(): + d = new_domain("A") + add_collection(d, "B") + for t in (0.0, 1.0): + for name in ("A", "B"): + add_uniform_step(d, t, **{"topo": grid_at(name, t)["topology"], "geom": grid_at(name, t)["geometry"]}, + collection=name) + assert "time_collections" in d + c = canonicalize_domain(d) + assert [list(i["spaces"]) for i in c["times"]] == [["A", "B"], ["A", "B"]] + + +def test_format_2_steps_join_the_instant_at_their_time(): + d = new_domain() + for t in (0.0, 1.0): + for name in ("A", "B"): + add_uniform_step(d, t, grid_at(name, t)["topology"], grid_at(name, t)["geometry"], + collection=name) + assert [list(i["spaces"]) for i in d["times"]] == [["A", "B"], ["A", "B"]] + + +def test_an_instant_with_no_spaces_is_refused(): + with pytest.raises(YmfArchiveError, match=r"times\[0\]\.spaces"): + canonicalize_domain({"times": [{"time": 0.0, "spaces": {}}]}) + + +def test_errors_name_the_space_and_instant(): + bad = grid_at("A", 0.0) + bad["topology"]["type"] = "Dodecahedron" + d = new_domain() + add_instant(d, 0.0, {"A": bad}) + with pytest.raises(YmfArchiveError, match=r"times\[0\]\.spaces\[A\]\.topology"): + validate_domain(d) + + +def test_both_formats_in_one_domain_is_refused(): + with pytest.raises(YmfArchiveError, match="both"): + canonicalize_domain({"times": [], "time_collections": []}) + + +# --- append_step --------------------------------------------------------- + + +def header(tmp_path): + path = tmp_path / "run.ymf" + write_ymf(new_domain(), path) + return path + + +def instant(t, *names): + return {"time": t, "spaces": {n: grid_at(n, t) for n in names}} + + +def test_instants_append_and_read_back(tmp_path): + path = header(tmp_path) + for t in (0.0, 0.5): + append_step(path, instant(t, "A", "B")) + domain, _ = read_ymf(path) + assert [i["time"] for i in domain["times"]] == [0.0, 0.5] + assert space_names(domain) == ["A", "B"] + + +def test_a_space_may_first_appear_at_a_later_record(tmp_path): + path = header(tmp_path) + append_step(path, instant(0.0, "A")) + append_step(path, instant(1.0, "A", "B")) + domain, _ = read_ymf(path) + assert [len(series(domain, n)) for n in ("A", "B")] == [2, 1] + + +def test_a_kill_loses_a_whole_instant_not_one_space_of_it(tmp_path): + path = header(tmp_path) + append_step(path, instant(0.0, "A", "B")) + append_step(path, instant(1.0, "A", "B")) + text = path.read_text() + path.write_text(text[: text.rindex("B:")]) # cut inside the last instant + with pytest.warns(RuntimeWarning): + domain, _ = read_ymf(path) + assert [i["time"] for i in domain["times"]] == [0.0] + assert list(domain["times"][0]["spaces"]) == ["A", "B"] + + +def test_format_1_records_are_joined_into_instants(tmp_path): + """What the Proteus writer does today: one record per collection per step.""" + path = header(tmp_path) + for t in (0.0, 1.0): + for name in ("A", "B"): + append_step(path, name, dict({"time": t}, **grid_at(name, t))) + domain, _ = read_ymf(path) + assert [list(i["spaces"]) for i in domain["times"]] == [["A", "B"], ["A", "B"]] + + +def test_an_instant_record_in_a_format_1_document_is_refused(tmp_path): + path = tmp_path / "old.ymf" + path.write_text((FIXTURES / "format1_two_spaces.ymf").read_text()) + append_step(path, instant(2.0, "Mesh")) + with pytest.raises(YmfArchiveError, match="format 1"): + read_ymf(path) + + +def test_a_format_1_stream_in_flight_reads(tmp_path): + path = tmp_path / "old.ymf" + path.write_text("ymf_archive_version: 1\ndomain:\n time_collections:\n - name: A\n data: []\n") + for t in (0.0, 1.0): + append_step(path, "A", dict({"time": t}, **grid_at("A", t))) + domain, _ = read_ymf(path) + assert [i["time"] for i in domain["times"]] == [0.0, 1.0] + + +def test_a_future_format_is_refused(tmp_path): + path = tmp_path / "f.ymf" + path.write_text("ymf_archive_version: 3\ndomain: {}\n") + with pytest.raises(YmfArchiveError, match="version 3"): + read_ymf(path) + + +# --- select and the views -------------------------------------------------- + + +def three_by_two(): + d = new_domain() + for t in (0.0, 1.0, 2.0): + add_instant(d, t, {"A": grid_at("A", t), "B": grid_at("B", t)}) + return d + + +def test_steps_index_instants_across_spaces(): + picked = select_domain(three_by_two(), steps="-1") + assert [i["time"] for i in picked["times"]] == [2.0] + assert list(picked["times"][0]["spaces"]) == ["A", "B"] + + +def test_collections_filter_spaces_and_keep_archive_order(): + picked = select_domain(three_by_two(), collections=["B", "A"]) + assert space_names(picked) == ["A", "B"] + only = select_domain(three_by_two(), collections=["B"]) + assert space_names(only) == ["B"] and len(only["times"]) == 3 + + +def test_instants_without_a_chosen_space_are_dropped_before_steps_count(): + d = new_domain() + add_instant(d, 0.0, {"A": grid_at("A", 0.0)}) + add_instant(d, 1.0, {"A": grid_at("A", 1.0), "B": grid_at("B", 1.0)}) + picked = select_domain(d, collections=["B"], steps="0") + assert [i["time"] for i in picked["times"]] == [1.0] + + +def test_series_and_the_format_1_view_agree(): + d = three_by_two() + assert [t for t, _ in series(d, "A")] == [0.0, 1.0, 2.0] + view = time_collections(d) + assert [c["name"] for c in view] == ["A", "B"] + assert [s["time"] for s in view[1]["data"]] == [0.0, 1.0, 2.0] + + +def test_a_spatial_space_survives(): + d = new_domain() + g = grid_at("A", 0.0) + add_instant(d, 0.0, {"A": spatial_grid([dict(g, name="r0"), dict(g, name="r1")])}) + c = validate_domain(d) + assert len(c["times"][0]["spaces"]["A"]["spatial_collection"]) == 2 + + +def test_parse_xdmf_domain_returns_format_2(): + from xml.etree.ElementTree import ElementTree + tree = build_xdmf_tree(three_by_two()) + domain = parse_xdmf_domain(tree.getroot()) + assert list(domain) == ["times"] and space_names(domain) == ["A", "B"] diff --git a/tests/test_proteus_archive.py b/tests/test_proteus_archive.py index 17e2d35..e84e234 100644 --- a/tests/test_proteus_archive.py +++ b/tests/test_proteus_archive.py @@ -38,6 +38,7 @@ import pytest +from ymf.archive import time_collections # noqa: E402 from ymf.archive import ( canonicalize_domain, read_ymf, @@ -74,7 +75,7 @@ def test_fixtures_are_present(): def test_fixtures_cover_every_observed_topology_type(): found = set() for path in FIXTURES: - for collection in _parse(path)["time_collections"]: + for collection in time_collections(_parse(path)): for step in collection["data"]: found.add(step["topology"]["type"]) assert found == EXPECTED_TOPOLOGY_TYPES @@ -83,7 +84,7 @@ def test_fixtures_cover_every_observed_topology_type(): @pytest.mark.parametrize("path", FIXTURES, ids=lambda p: p.stem) def test_real_proteus_archive_parses_into_the_domain_model(path): domain = _parse(path) - collections = domain["time_collections"] + collections = time_collections(domain) assert collections, "archive has no time collections" for tc in collections: assert tc["name"], "time collection has no name" @@ -139,7 +140,7 @@ def test_reemitted_xmf_preserves_every_hdf5_dataset_reference(path, tmp_path): def references(domain): refs = [] - for collection in domain["time_collections"]: + for collection in time_collections(domain): refs.append(collection["name"]) for step in collection["data"]: grids = step.get("spatial_collection", [step]) diff --git a/tests/test_select.py b/tests/test_select.py index 607391f..cb8c417 100644 --- a/tests/test_select.py +++ b/tests/test_select.py @@ -3,6 +3,7 @@ import pytest +from ymf.archive import canonicalize_domain, time_collections # noqa: E402 from ymf.archive import ( YmfArchiveError, add_spatial_step, @@ -50,7 +51,7 @@ def fields(names): def names_of(domain): - return [c["name"] for c in domain["time_collections"]] + return [c["name"] for c in time_collections(domain)] def fields_of(step): @@ -84,7 +85,7 @@ def test_a_step_the_archive_does_not_have_is_refused(): def test_everything_is_kept_by_default(): domain = two_spaces() - assert select_domain(domain) == domain + assert select_domain(domain) == canonicalize_domain(domain) def test_a_collection_is_picked_by_name_or_part_of_it(): @@ -97,7 +98,7 @@ def test_a_collection_is_picked_by_name_or_part_of_it(): def test_fields_are_picked_on_every_grid(): picked = select_domain(two_spaces(), fields=["p"]) - base, quadratic = picked["time_collections"] + base, quadratic = time_collections(picked) assert all(fields_of(step) == ["p"] for step in base["data"]) # the quadratic space has no p: its grids keep their mesh and lose their fields assert all(fields_of(step) == [] for step in quadratic["data"]) @@ -106,7 +107,7 @@ def test_fields_are_picked_on_every_grid(): def test_fields_are_picked_inside_spatial_collections(): picked = select_domain(two_spaces(spatial=True), fields=["u"], collections=["Spatial"]) - for step in picked["time_collections"][0]["data"]: + for step in time_collections(picked)[0]["data"]: assert [fields_of(g) for g in step["spatial_collection"]] == [["u"], ["u"]] @@ -117,17 +118,17 @@ def test_an_unknown_field_is_refused_with_what_there_is(): def test_steps_apply_to_every_collection(): picked = select_domain(two_spaces(), steps="-1") - for c in picked["time_collections"]: + for c in time_collections(picked): assert [s["time"] for s in c["data"]] == [pytest.approx(0.3)] picked = select_domain(two_spaces(), steps="::2") - assert [s["time"] for s in picked["time_collections"][0]["data"]] == [0.0, pytest.approx(0.2)] + assert [s["time"] for s in time_collections(picked)[0]["data"]] == [0.0, pytest.approx(0.2)] def test_the_source_is_not_modified(): domain = two_spaces() select_domain(domain, collections=["c0p2"], fields=["u2"], steps="0") assert names_of(domain) == ["Mesh Spatial_Domain", "Mesh_c0p2_Lagrange"] - assert fields_of(domain["time_collections"][0]["data"][3]) == ["u", "p", "vof"] + assert fields_of(time_collections(domain)[0]["data"][3]) == ["u", "p", "vof"] # -------------------------------------------------------------------------- @@ -157,7 +158,7 @@ def test_the_conversion_can_cherry_pick(archive, tmp_path): assert [a.attrib["Name"] for a in steps[0].findall("Attribute")] == ["vof"] # the archive itself is untouched domain, _ = read_ymf(archive) - assert len(domain["time_collections"]) == 2 + assert len(time_collections(domain)) == 2 def test_the_flags_reach_the_conversion(archive, tmp_path, capsys): @@ -184,7 +185,7 @@ def test_show_describes_and_converts_nothing(archive, capsys): def test_describe_lists_every_field_any_step_carries(): domain = two_spaces(n_steps=2) # a field that appears only at the second step is still listed - domain["time_collections"][0]["data"][1]["attributes"].append( + time_collections(domain)[0]["data"][1]["attributes"].append( attribute("late", data_item([9], "out.h5:/late", precision=8))) (base, _) = describe(domain) assert base.endswith("fields: u, p, vof, late") diff --git a/tests/test_xdmf.py b/tests/test_xdmf.py index 756f9a2..f25aa05 100644 --- a/tests/test_xdmf.py +++ b/tests/test_xdmf.py @@ -108,7 +108,7 @@ def test_canonicalize_domain_is_idempotent(): def test_canonicalize_domain_fills_precision_default(): # SAMPLE_DOMAIN's Topology DataItem has no explicit Precision. canonical = canonicalize_domain(SAMPLE_DOMAIN) - topo_data_item = canonical["time_collections"][0]["data"][0]["topology"]["data_item"] + topo_data_item = time_collections(canonical)[0]["data"][0]["topology"]["data_item"] assert topo_data_item["precision"] == 4 # XDMF default for Int @@ -275,6 +275,7 @@ def test_full_notebook_example_round_trips(tmp_path): # original TimeCollection/Topology/Geometry model did not cover. # ========================================================================== +from ymf.archive import time_collections # noqa: E402 from ymf.archive import ( # noqa: E402 (grouped with the tests that use it) add_spatial_step, add_uniform_step, @@ -397,7 +398,7 @@ def test_absent_nodes_per_element_stays_absent(): root = build_xdmf_tree(domain).getroot() topo_elem = root.find("Domain").find("Grid").find("Grid").find("Topology") assert "nodes_per_element" not in topo_elem.attrib - assert "nodes_per_element" not in parse_xdmf_domain(root)["time_collections"][0]["data"][0]["topology"] + assert "nodes_per_element" not in time_collections(parse_xdmf_domain(root))[0]["data"][0]["topology"] def _text_include_domain(): @@ -515,7 +516,7 @@ def test_a_bare_uniform_grid_reads_as_a_one_step_collection(tmp_path): path.write_text(XDMF3_UNIFORM) domain, extra = read_xdmf(path) assert extra is None - (collection,) = domain["time_collections"] + (collection,) = time_collections(domain) assert collection["name"] == "plate" (step,) = collection["data"] assert step["time"] == 0.0 @@ -531,14 +532,14 @@ def test_a_bare_uniform_grid_reads_as_a_one_step_collection(tmp_path): def test_a_bare_uniform_grid_keeps_its_time(): domain = _read(_uniform('