WIP PR for getting CI working as desired

[cshaley]

No description provided.

[cshaley]

Looks like the env file changes need to be merged in order to be pulled into the build

Note I skipped test_logical because it's failing. It looks like the cause is that logical.avro is not a valid avro file.

[cshaley]

@martindurant any idea where to look for resolving this error? It'll take me a good chunk of diving into this to understand what is the right way to fix this.

[martindurant]

The values seem of completely the wrong order, so it could be an int32->64 kind of thing. In pandas v2, more time types became allowed.
I don't have a huge amount of time right now, but I can move this repo to the intake org and make you collaborator if you like.

[cshaley]

sure, I'm fine with that.

This looks like a bug in fastparquet.

dask/fastparquet@f2f53b7

t should always be a dtype. I think it should just be t = str(t) + "[ns]" - then the following line (d = np.empty(0, dtype=t)) works as expected. Will raise a PR to fastparquet.

[martindurant]

Before going further, you might be interested by https://awkward-array.org/doc/main/reference/generated/ak.from_avro_file.html in awkward-array, which I was just made aware of. This respects the truly nested nature of avro and might be what you want, but you can also transform to arrow or pandas if the dataset is really tabular.

[cshaley]

Before going further, you might be interested by https://awkward-array.org/doc/main/reference/generated/ak.from_avro_file.html in awkward-array

Hmm, the reason I was updating uavro is to get intake-avro working. I wasn't familiar with awkward-array, but I bet I could drop it in to replace uavro in intake-avro. Thoughts?