EOS26

Description

All scripts to run the EOS26 simulation with 21cmFASTv4.2. EOS26 is run phase by phase i.e. ICs first, then PFs, then PHFs, and finally coevals.

How to run

All scripts accept an optional --test flag to run a small test box (HII_DIM=200).

Where the data go

All simulation caches (EOS26_cache/, EOS26_test_HIIDIM200/, EOS26_test_HIIDIM200_compare/) and the scaling-test caches (EOS26_scaling_sims/) are written below one data root (run_scripts/settings.py, scaling/config.sh):

• on Gadi: /g/data/qp00/$USER, because scratch is too small and I/O on gdata is as fast. Every PBS job must request -l storage=gdata/qp00.
• elsewhere: the repository root.
• EOS26_DATA=/some/path overrides both.

Code, logs and the scaling results (scaling/results/) stay in the repository. logs/ holds the production run logs: the 4.3.1 run and the cache copy/migration at the top level, the old 4.2 run (ICs, PFs, halo catalogs, coeval attempts, plots) in logs/EOS26_v4.2/, and the HII_DIM=200 test reference in logs/HII_DIM200_reference/. Scaling and benchmark logs are in scaling/. PBS's own <jobname>.o<jobid>/.e<jobid> files are git-ignored: before committing, python pbs_scripts/fold_pbs_output.py appends what only they contain (PBS resource summary, compare_* results, tracebacks) to the job's log and deletes them.

21cmFAST version

The PBS scripts in pbs_scripts/ (and postprocess/plots_job.sh) activate the environment with source envs/use.sh v4.3: 21cmFAST 4.3.1 (built by envs/create_venv_v4.3.sh gadi) and the modules it needs. 4.3.1 fixes the coeval memory growth of 4.3.0 (#809, fixed by #810; bug_reports/5_v4.3_sfrd_history_reloaded.md). Submit with qsub -v EOS26_VERSION=v4.2 … to use the old v4.2 environment. EOS26_VERSION=v4.3_patch (4.3.0 + envs/patches/v4.3.0_sfr_allzero.patch, the same fix) is kept for the scaling record and is not needed with 4.3.1. The scaling jobs use the same venvs (scaling/config.sh).

21cmFAST 4.3 cannot find boxes in a cache written by 4.2, because the cache hashes changed. Its ICs, PFs and halo catalogs can be converted, and are then the same as what 4.3.0 computes (ICs and PFs bit-identical, halo masses to ~1e-5). 4.3.1 finds and reads a 4.3.0 cache (same hashes), and for the EOS26 template its CLASS tables, sigma(M) and ICs are bit-identical to 4.3.0's (#808 only changes the transfer function beyond CLASS's k range, k > 1119/Mpc here). Coevals depend on physics that changed in 4.3, so they have to be recomputed:

21cmfast migrate /g/data/qp00/$USER/EOS26_cache --kind InitialConditions PerturbedField HaloCatalog

On Gadi, bash pbs_scripts/submit_copy_and_migrate.sh [--after <running copy job>] does the whole move. It submits chained copy rounds on copyq (pbs_scripts/copy_cache_job.sh, under 10 h each, each continuing where the last stopped), then the migration on hugemem (pbs_scripts/migrate_cache_job.sh). The migration only runs once a copy round has verified the copy and written /g/data/qp00/$USER/EOS26_cache.copy_complete. Follow the copy with tail -f /g/data/qp00/$USER/rsync_EOS26_cache.log.

This writes converted copies next to the originals, so it needs as much free space again as those boxes take. It holds one array at a time in memory; the largest is a high-resolution IC field, ~360 GB at HII_DIM=1500, so run it on hugemem. run_N_coevals.py stops if the ICs, PFs or halo catalogs for its inputs are missing, instead of computing them inside a coeval job.

Comparison workflow

• Generate the reference once with --test and no --compare. It is written to EOS26_test_HIIDIM200/.
• Run the production workflow with --compare to check each output against that reference.
• For a local workflow check, use both flags: --test --compare. Candidate files then go to EOS26_test_HIIDIM200_compare/, so the reference is never overwritten. Both test modes use the same input parameters and random seed (42).

End-to-end test

• sbatch sbatch_scripts/full_test_job.sh — runs all three pipeline steps sequentially in test mode.
• qsub pbs_scripts/full_test_job.sh — PBS equivalent of the end-to-end test.

Scaling Measurements

Peak memory (and time) of every phase as a function of HII_DIM and N_THREADS, for 21cmFAST v4.2 and v4.3, to find the largest HII_DIM that fits safely on a 3 TB megamem node. Full description: scaling/README.md.

• bash scaling/campaign_v4.2.sh / bash scaling/campaign_v4.3.sh / bash scaling/campaign_v4.3_patch.sh submit one PBS job per (HII_DIM, N_THREADS). Re-running a campaign script resumes unfinished configurations from the beginning of their unfinished phase.
• python3 scaling/summarize.py prints the status/results table and writes scaling/results/<version>/summary_*.csv.
• The v4.3 environment is created with envs/create_venv_v4.3.sh (Mac or Gadi). Switch versions with source envs/use.sh v4.2|v4.3|v4.3_patch. v4.3 is 21cmFAST 4.3.1; scaling/results/v4.3/ was measured with 4.3.0. v4.3_patch (4.3.0 + envs/patches/v4.3.0_sfr_allzero.patch, envs/create_venv_v4.3_patch.sh) writes its scaling results to scaling/results/v4.3_patch/; it has the same fix as 4.3.1.
• The previous harness and its v4.2 results are in scaling_old/. They are superseded; see scaling_old/ASSESSMENT.md.

Production run (in order)

1. Initial conditions (ICs):
   • sbatch sbatch_scripts/ICs_job.sh [--test]

• qsub pbs_scripts/ICs_job.sh [--test] on PBS.
• Writes EOS26.toml (full template with embedded node redshifts and random seed) and EOS26_minimal.toml, then runs run_scripts/run_ICs.py.

2. Perturbed fields (PFs):
   • bash sbatch_scripts/submit_PF_jobs.sh [--test] — submits one job per PF (indices 0–91) (not used for production).
   • sbatch sbatch_scripts/N_PF_job.sh <z_idx> [N] [--test] — runs a batch of N PFs (default N=10) starting from redshift index z_idx.

• bash pbs_scripts/submit_PF_jobs.sh [--test] and qsub pbs_scripts/N_PF_job.sh <z_idx> [N] [--test] are the PBS equivalents.

3. Perturbed halo fields (PHFs):
   • sbatch sbatch_scripts/PHFs_job.sh [--test] — runs run_scripts/run_PHFs.py.

• qsub pbs_scripts/PHFs_job.sh [--test] on PBS.

4. Coevals:
   • sbatch sbatch_scripts/N_coeval_job.sh [N] [--test] — runs a batch of N coevals (default N=10) with run_scripts/run_N_coevals.py.

• qsub pbs_scripts/N_coeval_job.sh [N] [--test] on PBS.

5. Plots (any time):

• qsub postprocess/plots_job.sh — slices of the ICs and PFs, and the lightcones of what is cached so far (postprocess/plot_lightcone.py --quantities …; --list shows which quantities and redshifts are there). Resubmit it after coeval jobs to extend the lightcones; only new boxes are read (<cache>/lightcone_planes/).

Table

EOS26 simulation step	Computation time [hrs]		Memory [TB]		Storage [TB]		SUs
	Estimated	Actual	Estimated	Actual	Estimated	Actual	Estimated	Actual
Initial conditions	17+~3 write	30	2.63	2.41	1.42	1.57
One perturbed field	1.42	0.2	0.545	0.505	--Gb x 92 = 2.3Tb	2.4Tb
Perturbed halo fields	22 hrs		0.71	0.77	~330 G	33 G	720 EM	617 EM
Evolving astrophysics for one coeval	7		3.65		2.6Tb x 92 = 240 Tb 0.215 x 92 = 20Tb without XRS		672 EM x 92 = 62k

Scaling test results

Note (2026-09-30): the values below come from the superseded v4.2 measurements in scaling_old/ (the HII_DIM=500 coeval point covers only 5/92 coevals; see scaling_old/ASSESSMENT.md). They will be replaced by the results of the new scaling/ campaigns.

When updated with --update-readme, each extrapolated value is the mean ± 1σ from all available scaling points. Each extrapolated quantity shows both its current model (affine overhead + coefficient * HII_DIM^3 for peak RSS; free-exponent power law for time and storage) and a fixed-cubic a=3 fit. The regression uncertainty is combined in quadrature with a fixed 10% relative measurement-uncertainty floor. EOS-1: HII_DIM = 1500 (1.5 cMpc/cell, 2250 Mpc). EOS-2: HII_DIM = 1200 (1.667 cMpc/cell, 2000 Mpc). Storage for PFs and coevals is the total across all 92 node redshifts; coeval storage includes retained IonizedBox and excludes transient XraySourceBox. The scaling-test fits are trained only on HII_DIM<=500 and, for initial conditions, that range's fixed per-job overhead makes the fitted compute-time noticeably underestimate the production values now measured at both EOS target sizes (elapsed_seconds fit for HII_DIM=1500: 17h vs. 29.8h measured).

EOS26 simulation step	Computation time [hrs]		Memory		Storage
	N=100	N=500	N=100	N=500	N=100	N=500
Scaling tests (measured)
Initial conditions	0.0374	0.624	0.939 GB	97.2 GB	0.464 GB	58 GB
One perturbed field	0.0345	0.0447	0.334 GB	20.2 GB	0.016 GB	2 GB
Perturbed halo fields	0.0498	0.659	0.564 GB	51 GB	0.207 GB	25.4 GB
Evolving astrophysics for one coeval	0.00199	0.423	1.16 GB	97.2 GB	0.0601 GB x 92 = 5.53 GB	n/a (partial run: 5/92 coevals)
Extrapolated to EOS (HII_DIM = 1500, 1.5 cMpc/cell, 2250 Mpc)
Initial conditions	a=1.72: 3.04 ± 2.21	a=3: 17 ± 1.83	affine (overhead + a=3): 2.62 ± 0.262 TB	a=3: 2.63 ± 0.263 TB	a=3: 1.57 ± 0.149 TB	a=3: 1.57 ± 0.157 TB
One perturbed field	a=0.152: 0.05 ± 0.0074	a=3: 1.42 ± 0.832	affine (overhead + a=3): 0.54 ± 0.054 TB	a=3: 0.546 ± 0.0547 TB	a=3: 0.0539 ± 0.00515 TB	a=3: 0.054 ± 0.0054 TB
Perturbed halo fields	a=1.6: 2.84 ± 2.21	a=3: 18.1 ± 2.07	affine (overhead + a=3): 1.37 ± 0.137 TB	a=3: 1.38 ± 0.138 TB	a=2.99: 0.675 ± 0.0648 TB	a=3: 0.686 ± 0.0686 TB
Evolving astrophysics for one coeval	a=3.26: 11 ± 7.78	a=3: 11.1 ± 1.36	affine (overhead + a=3): 2.59 ± 0.267 TB	a=3: 2.65 ± 0.272 TB	a=3: 0.202 ± 0.0193 TB x 92 = 18.6 ± 1.77 TB	a=3: 0.203 ± 0.0203 TB x 92 = 18.6 ± 1.86 TB

EOS Fixed-Cubic Runtime Plan

Planning values use fixed a=3 central estimates at HII_DIM=1500, a 1.5 h IC-read allowance per dependent job, and a 13 h IC-write time (measured production elapsed time minus the small-scale a=3 cubic fit's compute-only prediction at the same HII_DIM; see estimate_ic_write_hours) to infer output-write throughput. Coeval output includes retained IonizedBox and excludes transient XraySourceBox. The table uses the 48 h maximum job walltime; coeval jobs are deliberately limited to 3 outputs for margin. Peak RSS is the per-phase process estimate; full-phase output totals all 92 PFs or coevals as applicable.

Phase	Work unit	Units/job	Compute/unit [h]	IC read/job [h]	Write/unit [h]	Estimated job walltime [h]	Jobs for full phase	Serial phase walltime [h]	Peak RSS [TB]	Output, full phase [TB]
Initial conditions	all ICs	1	16.98	0.00	12.85	29.83	1	29.8	2.63	1.42
Perturbed fields	one PF	24	1.42	1.50	0.44	46.19	4	184.8	0.55	4.52
Perturbed halo fields	all halo fields	1	18.08	1.50	5.63	25.21	1	25.2	1.38	0.62
Coevals	one coeval	3	11.10	1.50	1.66	39.80	31	1233.7	2.65	16.95