Skip to content

New features: chunking and compression #93

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
khou2020 wants to merge 207 commits into
base: master
Choose a base branch
from
Open

New features: chunking and compression #93

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
207 commits
Select commit Hold shift + click to select a range
a95e7d5
pncio, a new I/O driver
wkliao Feb 17, 2025
b456c6b
update RELEASE_NOTES for version 1.15.0-alpha
wkliao Oct 26, 2025
d410d6f
add note and URL of 1.15.0-alpha release
wkliao Oct 27, 2025
95f18ae
Bug fix: check write buffer size consumed after increment
wkliao Nov 10, 2025
f187818
new error core NC_EMULTIDEFINE_HINTS for inconsistent I/O hints set b…
wkliao Nov 8, 2025
4c22ef5
rename hint variables used internally
wkliao Nov 8, 2025
38ebb40
One Bcast and Allreduce to check consistency of all I/O hints
wkliao Nov 8, 2025
f091228
move setting is_open inside of open call
wkliao Nov 9, 2025
3d080c9
parallel_run.sh run ncmpidiff based on driver mode
wkliao Nov 9, 2025
a275d3c
make ncmpi_close() returns the same error code
wkliao Nov 15, 2025
173680a
duplicate MPI info for non-INA aggregators in case later switch to in…
wkliao Nov 15, 2025
b92a710
pncio_write_str_naive.c assert disp and offset are 0s
wkliao Nov 15, 2025
6d51197
benchmarks/C/aggregation.c add indep I/O mode option
wkliao Nov 15, 2025
6be87b9
benchmarks/C/pnetcdf_put_vara.c add indep I/O mode option
wkliao Nov 15, 2025
74f724c
benchmarks/C/write_block_read_column.c add indep I/O mode option
wkliao Nov 15, 2025
897f355
benchmarks/C make ptest problem size larger
wkliao Nov 15, 2025
50e639e
FLASH-IO/parallel_run.sh separate cases for running ncmpidiff
wkliao Nov 15, 2025
6c671a6
test/C add an option to run in indep I/O mode
wkliao Nov 15, 2025
0700185
test/nonblocking/bput_varn.m4 add an option to put multiple variables
wkliao Nov 15, 2025
0077899
column_wise.m4 add testing for indep I/O mode
wkliao Nov 15, 2025
acf38f9
flexible_bput.c add testing for indep I/O mode
wkliao Nov 15, 2025
f3a6ab1
testcases/alignment_test.c increases problem size
wkliao Nov 15, 2025
c3ad583
tst_multi_redefine.c tests entering define mode multiple times
wkliao Nov 15, 2025
fd66711
PNETCDF_PROFILING: print profiling timings by world rank 0 only
wkliao Nov 15, 2025
53537df
Github actions: disable profiling and add MIMIC_LUSTRE
wkliao Nov 17, 2025
7be3994
FLASH-IO/parallel_run.sh test independent data mode
wkliao Nov 19, 2025
495a21b
alignment_test.c test indep data mode before collective
wkliao Nov 19, 2025
8b66e65
buftype_free.c test independent data mode and check read buffer contents
wkliao Nov 19, 2025
c80ac4a
tst_multi_redefine.c checks variable contents after growing file header
wkliao Nov 23, 2025
b376b1a
wrf_io.c add option of using blocking APIs
wkliao Nov 27, 2025
c06d4d0
wrf_io.c add a check of file contents after write when in debug mode
wkliao Dec 4, 2025
191f196
testcases/flexible.c add test of independent data mode
wkliao Dec 5, 2025
10b53d5
buftype_free.c define INDEP_MODE and COLL_MODE for better readability
wkliao Dec 6, 2025
810d4d7
flexible.c increase I/O tests size
wkliao Dec 6, 2025
fb83bcd
flexible2.c add testing of independent data mode
wkliao Dec 6, 2025
b76f453
ncmpidiff.c print command-line options when first countering a differ…
wkliao Dec 8, 2025
38898af
bug fix: buf_view.off[] is constructed based on the user buffer, buf
wkliao Dec 11, 2025
417edef
print an extra INA profile timer
wkliao Dec 11, 2025
ae782b0
ncmpio_intra_node.c copying noncontiguous write buffer to contiguous …
wkliao Dec 11, 2025
840ef98
testcases/varn_int.c fix varid used in testing
wkliao Dec 11, 2025
a143f20
testcase/varn_int.c add tests independent data mode and set nc_ibuf_s…
wkliao Dec 12, 2025
1c8bad1
testcase/flexible_var.c increase problem size and test set nc_ibuf_si…
wkliao Dec 12, 2025
3ce0091
testcase/flexible_varm.c add tests independent data mode and set nc_i…
wkliao Dec 12, 2025
56e61b6
testcase/iput_all_kinds.m4 add tests independent data mode and set nc…
wkliao Dec 12, 2025
ea64110
ncadios: Fix typo causing compilation error
sbelsk Dec 12, 2025
1821bd7
bug fix: ensure str_factor <= overstriping_ratio * total_num_OSTs
wkliao Dec 14, 2025
4c520e1
pncio_lustre_open.c printf to stdout when in DEBUG mode
wkliao Dec 14, 2025
3c3523d
Bug fix: pncio_lustre_open.c ensure OST ID is within range of numOSTs
wkliao Dec 14, 2025
dd4e47b
INA reuse recv_buf when possible to reduce memory footprint
wkliao Dec 15, 2025
298c2e7
INA report memory footprint high-water mark instead
wkliao Dec 15, 2025
ee45c5f
valgrind warn: to avoid unnecessary increment of fd->flat_file.idx
wkliao Dec 16, 2025
67a87f1
ncmpio_getput.m4 check zero-sized request as buf pointer may not be i…
wkliao Dec 16, 2025
f67469d
Bug fix: move data after adding new variables
wkliao Dec 17, 2025
cca757d
Ensure hint nc_var_align_size supersedes nc_header_align_size
wkliao Dec 19, 2025
79c415b
add fix_end, end offset of last fix-sized variable
wkliao Dec 19, 2025
de263a3
Revisit of aligment hints for setting the offset of header and data s…
wkliao Dec 19, 2025
9ecfb68
testcases/file_create_open.c test with adding an MPI info object
wkliao Dec 19, 2025
101c27d
tst_redefine.c update hint alignment calculation
wkliao Dec 19, 2025
63b6575
RELEASE_NOTES rewording
wkliao Dec 19, 2025
22fbd57
Add CI job for ADIOS backend
sbelsk Dec 17, 2025
4a261fb
Fix syntax error in test script
sbelsk Dec 18, 2025
70afcca
Enable ncmpi_inq_header_size() to be called in define mode.
wkliao Dec 16, 2025
1bf7e2c
WRF-IO: fix run script when NetCDF4 is enabled
wkliao Dec 19, 2025
c2a948e
test/CXX programs require at least 2 command-line arguments
wkliao Dec 19, 2025
733cce2
testcases/null_args.m4 skip calls to ncmpi_fill_var_rec when NetCDF4 …
wkliao Dec 19, 2025
ec7d8f3
skip tst_grow_data.c when NetCDF4 is enabled
wkliao Dec 19, 2025
b1687d9
tst_inq_header_size.c skip tests for NetCDF4 files
wkliao Dec 19, 2025
96d9488
fix warning: declaration of 'i' shadows a previous local [-Wshadow]
wkliao Dec 19, 2025
d5aaf5a
tst_data_move.c checks whether the data movement subroutines are called
wkliao Dec 20, 2025
2229a51
tst_grow_data.c rename NY/NX to LON/LAT
wkliao Dec 20, 2025
c18ce7b
Obtain the total number of OSTs, given a folder
wkliao Dec 20, 2025
01864a1
ncmpidiff.c - add stat checks for input files
wkliao Dec 21, 2025
935d843
Move all calls of getenv() to file create/open
wkliao Dec 23, 2025
d969a90
ncoffsets.c add command-line option '-d' to replace getenv PNETCDF_VE…
wkliao Dec 23, 2025
3a2b78f
Add note on how environment variable PNETCDF_VERBOSE_DEBUG_MODE works
wkliao Dec 23, 2025
bcb4124
pass environment variables, env_mode, from dispatcher to all drivers
wkliao Dec 23, 2025
a87d014
make safe mode a part of ncp->flags
wkliao Dec 23, 2025
8bb93cd
Add functions to check env variable PNETCDF_RELAX_COORD_BOUND
wkliao Dec 23, 2025
36a7a4c
Call is_relax_coord_bound() instead of check macro RELAX_COORD_BOUND
wkliao Dec 23, 2025
14da611
fix a missing printf argument
wkliao Dec 23, 2025
3c09e2a
testcases/varn_int.c check NULL before free()
wkliao Dec 24, 2025
8580cfa
When MIMIC_LUSTRE is not enabled, set hint cb_nodes=2 for all testings
wkliao Dec 24, 2025
bf829f2
Github action: use OpenMPI 5.0.8
wkliao Dec 24, 2025
3155af7
fix bash script error of "test: =: unary operator expected"
wkliao Dec 25, 2025
e7f9985
align print line
wkliao Dec 25, 2025
33478fd
PNCIO_Calc_my_req() remove argument fd_start as it is not used
wkliao Dec 24, 2025
9d29687
pncio_aggregate.c - add assert(fd->flat_file.count > 0);
wkliao Dec 29, 2025
b8133de
bug fix: special care for when fd->flat_file.count == 0, entire file …
wkliao Dec 24, 2025
65b694e
Check file system type at rank 0 only and then bcast result
wkliao Dec 27, 2025
5c96a6e
WRF-IO add a benchmark option of moving data section when header grows
wkliao Dec 28, 2025
e97e07b
ptest - add tests of romio_no_indep_rw = true and false
wkliao Dec 29, 2025
c98ff3e
ncmpio_enddef.c - participate collective I/O even if zero-sized requests
wkliao Dec 29, 2025
6b0d6a2
ncmpio_file_io.c returns immediately when file handler is NULL
wkliao Dec 29, 2025
293f2a1
Set buf_view.type = MPI_BYTE for zero-sized requests
wkliao Dec 29, 2025
8456526
check hint romio_no_indep_rw and set the flag accordingly
wkliao Dec 29, 2025
e040d51
Bug fix: collectively write number of records when in collective data…
wkliao Dec 30, 2025
d39934d
ncmpio_fill.c fill buffer is always contiguous
wkliao Dec 30, 2025
eb5c919
Try issend/irecv to replace MPI_Alltoall
wkliao Dec 24, 2025
faefcd7
add an option of MPI_Allreduce besides MPI_Allgather
wkliao Dec 25, 2025
217fa25
Github actions: change timeout to 90 minutes
wkliao Dec 31, 2025
8d73bbc
hints.c add print prefix 'info'
wkliao Jan 1, 2026
63fa6a8
New hint nc_data_move_chunk_size
wkliao Jan 1, 2026
be24c5b
wrf_io.c print hint nc_data_move_chunk_size
wkliao Jan 1, 2026
23d3338
test/parallel_run.sh - set nc_data_move_chunk_size to 100
wkliao Jan 1, 2026
3c9761a
Github actions: separate tests of MIMIC_LUSTRE
wkliao Jan 1, 2026
4954b6a
wrf_io.c print hint cb_node_list
wkliao Jan 2, 2026
bab027d
move setting hint data_chunk to after file open
wkliao Jan 2, 2026
882c25a
Improve data section movement
wkliao Jan 3, 2026
8ffc131
parallel_run.sh setenv MIMIC_STRIPE_SIZE to 1048576
wkliao Jan 3, 2026
0b251e9
Github actions: increase timeout to 120 minutes
wkliao Jan 3, 2026
2a976ba
Separate ncmpidiff core subroutines into a file and build it as a lib…
wkliao Jan 4, 2026
9b80824
Pin GHA dependencies by hash
williamjallen Jan 6, 2026
5569d26
Configure Dependabot updates for GitHub Actions
williamjallen Jan 6, 2026
384e7ec
Bump actions/checkout from 4.3.1 to 6.0.1
dependabot[bot] Jan 8, 2026
a0c1ab8
Bug fix: indivisible case when nprocs_my_node % naggrs_my_node > 0
wkliao Jan 12, 2026
194f125
Github action: try /tmp and $RUNNER_TEMP
wkliao Jan 10, 2026
e94f32c
Github action: make -s LIBTOOLFLAGS=--silent V=1
wkliao Jan 17, 2026
b006e89
dtype_decode.c warning uninitialized variable
wkliao Jan 19, 2026
20a3445
ncmpio_enddef.c -Wshorten-64-to-3 warning
wkliao Jan 6, 2026
a8356fe
free ncp->mpiinfo before getting a new one
wkliao Jan 23, 2026
e9b8ad4
bug fix: pncio - condition on info object duplication
wkliao Jan 23, 2026
9ccd5a2
ncbbio_log.c warning integer_sign_change
wkliao Jan 25, 2026
e6815e3
ncmpidiff.c - ncmpidiff_core() return data type
wkliao Jan 6, 2026
c80c5fc
ncoffsets.c use uint32_t and uint64_t in byte swap
wkliao Jan 19, 2026
bdc065a
ncvalidator.c use uint32_t and uint64_t in byte swap
wkliao Jan 19, 2026
bde76b0
github actions permission fix, recommended by code scanning alerts
wkliao Jan 26, 2026
0c573ad
m4/utils.m4 add CDF5_EXTRA_ITYPE_LIST and _all option
wkliao Jan 16, 2026
d342480
test: define TESTOUTDIR in config.h
wkliao Jan 5, 2026
a77e439
test/testcases -Wshorten-64-to-3 warning
wkliao Jan 6, 2026
212f030
test/common: add C struc loop_opts and tst_main()
wkliao Jan 5, 2026
4da1426
test/C - merge pres_temp_4D_wr.c and pres_temp_4D_rd.c
wkliao Jan 14, 2026
4ecee6f
test/cdf_format make use of tst_main()
wkliao Jan 14, 2026
ae041fc
test/cdl make use of tst_main()
wkliao Jan 14, 2026
e80da8b
test/header make use of tst_main()
wkliao Jan 9, 2026
cc81c7d
test/testcases make use of tst_main()
wkliao Jan 14, 2026
7c77312
test/nonblocking make use of tst_main()
wkliao Jan 17, 2026
b7256c7
test/F90 make use of get_args()
wkliao Jan 18, 2026
87b4093
test/fandc Makefile.am remove use of variable TESTPROGRAMS
wkliao Jan 18, 2026
80bd14f
test/CXX update parallel_run.sh and seq_runs.sh
wkliao Jan 18, 2026
65af3ec
test/adios clean files all have tail name nc or bb
wkliao Jan 18, 2026
a1aceab
test/burst_buffer - make use of tst_main()
wkliao Jan 18, 2026
2e87ddb
test/nc_test clean files all have tail name nc or bb
wkliao Jan 18, 2026
add3460
test/nc4 clean files have tail name nc or bb. remove unused variables
wkliao Jan 18, 2026
4c88b10
test/nf90_test clean files all have tail name nc or bb
wkliao Jan 18, 2026
37562b9
test/nf_test clean files all have tail name nc or bb
wkliao Jan 18, 2026
201a118
test/subfile Makefile.am remove use of variable TESTPROGRAMS
wkliao Jan 18, 2026
5d1b6d0
nc_test/test_write.m4 -Wstringop-truncation
wkliao Jan 19, 2026
ac0ab19
testcases/put_all_kinds.m4/iput_all_kinds.m4 remove need of calling r…
wkliao Jan 20, 2026
565a327
tst_grow_data.c use nonblocking put
wkliao Jan 20, 2026
8377bc1
test/largefile make use of tst_main()
wkliao Jan 18, 2026
91b9634
benchmarks/C skip test no_indep_rw_hint and ptests on 4 and 6 processes
wkliao Jan 19, 2026
4ae375e
benchmarks/FLASH-IO skip test no_indep_rw_hint and ptests on 4 and 6 …
wkliao Jan 19, 2026
cfebccf
benchmarks/WRF-IO skip test no_indep_rw_hint and ptests on 4 and 6 pr…
wkliao Jan 19, 2026
fdd2e85
examples make ptests run 4 and 6 processes
wkliao Jan 19, 2026
4f14fbd
ncbbio_log.c use NULL in calls to realpath()
wkliao Jan 26, 2026
913206d
Github action: combinations of Ubuntu/MacOS, MPICH/OpenMPI, UFS/Lustre
wkliao Jan 27, 2026
bb5f1df
Github actions: delete yaml files that have been moved to main.yml
wkliao Jan 28, 2026
c72521b
Github actions: test NetCDF4 and ADIOS
wkliao Jan 28, 2026
23628d1
fix compilation warnings found by github code scanning
wkliao Jan 28, 2026
23c2df8
update github CI status badge
wkliao Jan 29, 2026
210a42c
Github actions: fix OpenMPI network issue when running make ptest
wkliao Jan 29, 2026
9a30756
Change CI permissions to read-only
williamjallen Jan 31, 2026
20cb396
testutils.c: print timing result for each test when in verbose mode
wkliao Feb 1, 2026
411ed39
UFS: obtain file system block size as striping size
wkliao Feb 1, 2026
d418df6
tst_redefine.c - uses nonblocking APIs
wkliao Feb 1, 2026
336c740
Bump actions/checkout from 6.0.1 to 6.0.2
dependabot[bot] Feb 1, 2026
5028855
testutils.c fix logic of skip_diff
wkliao Feb 2, 2026
d9bb93f
testcases/flexible.c return error immediately found
wkliao Feb 2, 2026
4a1efcc
increase test array sizes
wkliao Feb 4, 2026
cac0edb
fix compilation warnings
wkliao Feb 4, 2026
c5fd5eb
add a new hint nc_striping
wkliao Feb 7, 2026
1ce736f
spell check.
wkliao Feb 9, 2026
14901ce
rename internal variable striping to nc_striping to be consistent
wkliao Feb 11, 2026
e8ef124
add node IDs of all processes as an attribute cached in MPI communicator
wkliao Feb 11, 2026
aabcd01
receive node IDs of all processes from dispatcher's file create/open
wkliao Feb 12, 2026
bc3ca06
Update data structure for storing node IDs and number of nodes
wkliao Feb 12, 2026
4d0cc55
Update file create/open APIs for receive node IDs of all processes
wkliao Feb 12, 2026
60ba8b0
add config option to enable chunking feature
khou2020 Dec 4, 2022
f89b83f
add chunk drivere
khou2020 Dec 4, 2022
842283c
add ncmpi_var_get/set_chunk and ncmpi_var_get/set_filter
khou2020 Dec 4, 2022
9b7e90c
enable chunk driver when chunking hint is set
khou2020 Dec 4, 2022
1279cd8
rename default filter hint to nc_chunk_default_filter
khou2020 Dec 5, 2022
5d89f43
add documentation about chunked I/O driver
khou2020 Dec 5, 2022
809a408
bugfix:chunk driver build condition is ENABLE_CHUNKING
khou2020 Dec 5, 2022
b4084d0
always build chunk related APIs
khou2020 Dec 5, 2022
69650bc
bug fix: add ncchunkio to DIST_DIR
khou2020 Dec 5, 2022
bda1f75
bugfix: add new m4 scripts to extra dist
khou2020 Dec 5, 2022
5331435
bugfix: add ncchkioi_profile_timers.m4 to extra_dist
khou2020 Dec 5, 2022
349f61a
bugfix: add nchunk io to m4 include
khou2020 Dec 5, 2022
0c508de
define ENABLE_COMPRESSION if --enable-zlib or --enable-sz is set
wkliao May 8, 2025
e245b4f
add example of using chunking and compression
wkliao May 8, 2025
13bc324
run chunk_compress.c only when compression is enabled
wkliao May 8, 2025
dcc706e
Remove unused variable, initialize err to NC_NOERR
wkliao May 24, 2025
e3dbcf3
add chunking example, examples/C/chunk_io.c
wkliao Jun 19, 2025
bf0766c
Add verbose to print which processes perform compression/decompression
wkliao Jun 19, 2025
b3cc689
comment out unused printf
wkliao Jun 19, 2025
b78c8a7
test: add examples/C/chunk_2D.c which uses 2D checkerboard partitioni…
wkliao Jun 20, 2025
2bcd2d9
examples/C/chunk_2D.c one wait_all for all nonblocking reads
wkliao Jun 20, 2025
246835c
chunk_2D.c: fix use of NX by mistake
wkliao Jul 1, 2025
e129914
use MPI_COMM_SELF to call MPI_Pack and MPI_Unpack
wkliao Jul 1, 2025
852673d
bug fix: sbuf index, array of size nsend
wkliao Jul 1, 2025
8f09513
chunk: fix printing expected number of records
wkliao Jul 9, 2025
5144445
chunk_2D.c add partitioning along time for decompression
wkliao Jul 9, 2025
a1f27c2
bug fix: all ranks participate wait call whenin collective data mode
wkliao Sep 9, 2025
5ee20a3
add test program for chunking nonblocking API call
wkliao Sep 9, 2025
be25547
new example program for chunking/compression feature: chunk_compress_…
wkliao Feb 16, 2026
4324dca
rebase master branch
wkliao Feb 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
8 changes: 8 additions & 0 deletions .github/dependabot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "monthly"
cooldown:
default-days: 7
206 changes: 0 additions & 206 deletions .github/workflows/mac_mpich.yml
View file
Open in desktop

This file was deleted.

Loading