Repository navigation
Add defragmentation and fix/test read/write/read/write cycle of maps - #183
Conversation
04891be to
6f5095d
Compare
| sparse_map_size = (defrag_n_cov + 1) * nfine_per_cov | ||
|
|
||
| if self.is_bit_packed_map: | ||
| defrag_sparse_map = _PackedBoolArray(size=sparse_map_size) |
There was a problem hiding this comment.
this could be very big for bit packed footprints of a survey
To avoid the high memory usage you can use rustfits to create the output file. Then rustfits supports writing into the file using the same slice notation you use for numpy arrays, so the algorithm below can I think be unchanged. https://rustfits.readthedocs.io/en/latest/tutorial/images.html#writing-pixels-back
There was a problem hiding this comment.
This is a temporary up to 2x memory usage which I don't think is the worst thing... I was thinking this wasn't a common operation anyway, and you should have some memory overhead when making a giant footprint (vs using it). But I do see the point of having a minimal-memory-usage version and can consider that in the future.
There were a couple of bad bugs related to memory ownership and reserved fits keywords that meant that it was difficult to read in a map, modify it, and save it again.
This fixes those bugs and adds extensive tests.
This also adds defragmentation (which didn't work properly because of these serialization bugs!)