map is broken, includeing missing collect_similar for DiskGenerator

[felixcremer]

The DiskGenerator is missing an implementation of collect_similar which makes the filter function not work properly. See yeesian/ArchGDAL.jl#409 for the details.

[rafaqz]

map is actually broken in a lot of ways, not just collect_similar

I'm keen to try my very old idea of caching a whole column of chunks for iterate instead of a single chunk. Then iteration can follow the normal order and all of these problems go away. We can just delete DiskGenerator and DiskZip completely.

(As in your full band iteration - it works fine if we have the whole column)

[meggart]

I'm keen to try my very old idea of caching a whole column of chunks for iterate instead of a single chunk. Then iteration can follow the normal order and all of these problems go away. We can just delete DiskGenerator and DiskZip completely.

Won't this very easily lead to Out-Of-Memory errors for large multi-dimensional arrays where you just can not keep all chunks along the first dimension in memory? Very easy to construct examples where this fails.

Another option might just be to completely deprecate map, broadcast and reduce from DiskArrays and refer to DiskArrayEngine. Alternatively we could of course try to come up with more principled implementations, but I don't know if it would be worth the effort...

[rafaqz]

Does map work in DiskArrayEngine ?

It just seems to me that we are silently returning the wrong answer for map currently, which is worse than OOM errors. I am really keen to not silently return the wrong result ever.

We could always fall back to loading partial chunks if the whole column is too large for memory.

[meggart]

Is this resolve by #198 ?

[asinghvi17]

Yes, I just tried the example from ArchGDAL, and it seems like it is correct:

issetequal(filter(!(==(-9999)), collect(o)), filter(!(==(9999)), o)) # true

so this should be working now.