Question - Smile encoding/scan queries

[EwanValentine]

Hey!

Just a quick one, I'm going to be using Druid + Go for some pretty heavy data fetching/processing. I was slightly concerned about having to do that via JSON+HTTP because we're talking about millions of rows. I noticed that Scan is supposedly more for this use case as it can stream results back to the client. I also noticed in the docs that it supports Smile encoding, which looks as though it could be a lot more efficient in terms of transferring data from a to b.

I noticed this library among a few others, which looks fantastic. Just wondered if you handled Scan queries by streaming at all? I had a glance through the code and it seemed like a regular HTTP request. Apologies if I'm being stupid or not understanding something here. I also wondered if you'd used Smile encoding/had any plans to support it at all?

I tried to get a response in Smile format using your library + https://github.com/zencoder/go-smile but seemed to get a few errors, which I raised here: apache/druid#10945

Appreciate any help/advice on this. Also, we'll probably use this library, so will be able to offer some help where/if needed :)

[jbguerraz]

Hello @EwanValentine :)

Thank you for that issue!

I'm fine to give Smile a try, or maybe better, to battle test JSON+GZIP. CF https://www.lucidchart.com/techblog/2019/12/06/json-compression-alternative-binary-formats-and-compression-methods/ TLDR:

If the data originated as JSON, and needs to be converted back to JSON in-order to be used, then JSON is probably going to be the most cost-effective persistence format as well. Just so-long as you choose the appropriate compression algorithm.

I'll anyway be happy if we make go-druid more performant! I'm available "anytime" on https://join.slack.com/t/grafadruid/shared_invite/zt-mt6vll3m-eDSAMiCHlbS6oPhTMImkfw so we can discuss that further.

[jbguerraz]

Hello @EwanValentine
I can see this apache/druid#10945 (comment) moved ahead.
Do you still want / need that Smile encoding ? is that something still relevant ?