Is it possible to load chunks from a dataarray in the background?

[GrowlingM1ke]

I've developed a GUI where I read in a video in .zarr format to xr.DataArray and play it in real time. The video playback works fine however there is a small but distinct lag when I reach the end of the current chunk I am on and I have to load in the next one.

I attempted to fix this by loading in the next chunk ahead of time whilst the video from the current chunk plays. However no matter what I attempt to do, be it loading the next chunk on a separate thread or trying to use dask.distributed to load it in the background, I am unable to get rid of the fact that my GUI slightly hangs whenever a .load() call is made.

Is there anything I can do to fix this?

Thank you in advance.

[GrowlingM1ke]

I believe I found a solution to my issue, threading is limited by GIL and since the load() function is significant the only solution was to create a new process, hence I used:

from concurrent.futures import ProcessPoolExecutor

Which resolved my issue.

If anyone is interested in the implementation, then I should have it up on my repo in the near future here