Store numpy objects to be compatible across numpy 2.0 #782
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes numpy 2.0 cross-compatibility by using
numpy.save
andnumpy.load
internally. As pickle stores also the function used to unpickle stored data this should be fully backwards compatible (i.e. reading old .nur files is unchanged).There is some performance impact to reading (writing is essentially unchanged), but at least on my laptop (SSD) the file IO still dominates. So possibly on superfast PCIE SSDs the performance impact is noticeable, but the
.nur
format is probably not optimal for very-high throughput in general.This replaces #720. My proposal is to merge this to master as a hotfix first, and then to develop after.
Medium to long term, other upgrades we can consider are safe(r) pickling (see https://docs.python.org/3/library/pickle.html#restricting-globals), and before I thought of this I got halfway to using
h5py
to store NuRadio events. I think it may still be worth finishing that up - there would be benefits to security, readability and potentially performance. But I think both of these can be discussed and implemented in future PRs.