Proposed HDF5 input format and draft implementation

[The9Cat]

Summary of changes

• Proposed and implement and HDF5 schema for body input data
• Provide a converter to read ascii and emit compressed HDF5 and vice versa
• Implement a body reader for initializing EXP

History and intent

This started as a initial body file compressor. Using HDF5 to exploit the intrinsic stuctural similarities in a body file seemed like a good way to go. After completing the standalone routine, I realized that I was more than half way to an implementation of a reader for the Component class.

This is low priority and does not fix any bugs. But the ugly and inefficient ASCII tables should probably be replaced by HDF5 now that most of EXP is reading/writing using HDF5.

Implementation details

• The HDF5 schema takes the following form:
  /
  ├── Attributes
  │ ├── num_particles (int)
  │ ├── num_aux_ints (int)
  │ └── num_aux_floats (int)
  └── particles/ (Group)
  ├── index (Dataset, unsigned long)
  ├── m (Dataset, float/double)
  ├── x, y, z (Datasets, float/double)
  ├── u, v, w (Datasets, float/double)
  ├── aux_int_0, aux_int_1, ... (Datasets, int)
  └── aux_float_0, aux_float_1, ... (Datasets, float/double)
• The design goal is keeping like data together and using chunking, bit shuffling, and compression at the HDF5 level to achieve up to a factor of 5 savings in file size for float32 and a factor of 2.5 savings for float64.
• The index and aux_int_*, aux_float_* fields are optional.
• This schema is a precise mapping of the current body file data fields, except for the inverse ordering. One could make this more Gadget-like, but that would muddy the clarity here.
• I separated pos into x, y, z (and vel into u, v, w) to simply the std::variant coding to allow both float and double arrays. It is possible that consolidating into 2d arrays could be more efficient. But this is easier to code.
• The precision of the float data type is deduced by reading the m dataset. We do more sanity checks, e.g. make sure that every dataset has the same datatype but this seems like overkill.
• The converter routine hdf5bods was initially designed to compress body files and it achieves that goal. It is implemented with OpenMP which gives some modest performance gains (but not MPI). A $10^6$ particle body files requires 10 seconds for conversion on a laptop.
• The Component class checks whether the specified body file is HDF5 and reverts to the original ascii method if not; so all of the prior behavior is retained for compatibility.
• If the optional index field (unsigned long) exists, it will populate the EXP Particle.indx field. Similarly, the stand-along conversion routine has can read and write the index field for consistency with the original ASCII body table format.
• The stand-alone hdf5bods converter has support for multiple HDF5 compression filters for testing. The default GZip has the highest compression ratio but it also takes the most CPU time.

Tests so far

• Code compiles
• The converter code, hdf5bods, has built in round-trip verification which passes
• Checked that exp can read the new HDF5 files and produce the same run for a simple case
• Review HDF5 schema
• Test initial condition body file with int and float auxiliary attributes
• Test initial condition body file with index values

[Copilot]

Pull request overview

This PR introduces an HDF5-based format for EXP particle “body” input data, along with a converter utility and initial integration into Component so EXP can initialize from either ASCII or HDF5 while preserving backward compatibility.

Changes:

• Added hdf5bods utility to convert EXP ASCII body files to compressed HDF5 (and back), with optional round-trip verification.
• Updated Component initialization to detect HDF5 body files and read/distribute particles from HDF5 when applicable (fallback to existing ASCII reader otherwise).
• Bumped project/documentation version to 7.11.0.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
utils/PhaseSpace/hdf5bods.cc	New ASCII↔HDF5 body-file converter using HighFive + OpenMP; writes proposed schema attributes/datasets.
utils/PhaseSpace/CMakeLists.txt	Builds/installs the new hdf5bods executable.
src/Component.H	Adds declarations for HDF5 initialization/read helpers.
src/Component.cc	Adds HDF5 detection and HDF5-based body ingest + MPI distribution.
include/ScopeTimer.H	New simple scope-based timing helper used by the converter.
doc/exp.cfg.breathe	Version bump for generated docs.
doc/exp.cfg	Version bump for generated docs.
CMakeLists.txt	Project version bump to 7.11.0.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.