An example on how to deserialize to a struct or hashmap?

[ultrabug]

Hello,

Thanks for this useful crate 👍

I'm a bit new to Rust and I must admit I'm confused on how to actually access some sort of Struct or Hashmap from your lib...

I read Avro messages from Kafka, I'd like a way to access the data inside it (key, value). It looks to me (so far) that I need a way to deserialize a DecodeResult into something workable like I would expect from a Struct.

The avro_rs crate exposes a nice Reader struct to deserialize the values from Avro but it doesn't look to work well with DecodeResult.

The gist of my question is, using this crate, how one is supposed to get a workable data object to play with?

PS: I'm using easy async.

loop {
        match consumer.recv().await {
            Err(e) => warn!("Kafka error: {}", e),
            Ok(m) => {
                let decoded = decoder.decode(m.payload()).await;
                match decoded {
                    Ok(v) => {
                        // this works tho I dunno how to access the actual data inside
                        println!("name {:?}", v.name.unwrap());
                        // what should I do to transform v.value into a struct deserialized by avro_rs?
                    },
                    Err(e) => println!("err {:?}", e),
                }
            }
        };
    }

Thanks for your help, I'll sure propose a documentation update once I nail it down 🥇

[gklijs]

It depends a lot on whether you know the schema of the data up front. If you don't you get back a Value::Record which contains the values, but can be tedious to get the information from. Like in

schema_registry_converter/src/async_impl/avro.rs

Line 593 in 95f48fb

async fn test_decoder_default() {

you then would need to match for the vector to contain some field with the expected name, and the expected kind of value, then read the value.

If you do know the schema you can do something similar to

schema_registry_converter/src/async_impl/avro.rs

Line 613 in 95f48fb

let item = match from_value::<Heartbeat>(&heartbeat) {

using the struct defined

schema_registry_converter/test_utils/src/lib.rs

Line 98 in 95f48fb

pub struct Heartbeat {

I fully agree this later option is kind of hidden, so I would very much appreciate a PR to make this use more clear.

[ultrabug]

Scratch that, so for now the best I have looks like :

                // m is a BorrowedMessage from rdkafka
                let decoded = decoder.decode(m.payload()).await.unwrap().value;
                println!("decoded = {:?}", decoded);

                let record = match decoded {
                    Value::Record(v) => v,
                    _ => vec![],
                };

                for (k, v) in record {
                    match k.as_str() {
                        "tag_id" => println!("tag_id = {:?}", from_value::<i32>(&v)),
                        _ => (),
                    };
                    println!("key = {:?} value = {:?}", k, from_value::<String>(&v));
                }

Although getting the Record sounds now fine, this sounds very cumbersome to access the data in it... which is my initial help request 🙏 please?

Example Record :

Record([("oid", String("asdfsdf")), ("uid", String("7873b372-497b-4953-8832-aaaa")), ("fd", Union(String("7873b372-497b-4953-8832-dddd"))), ("td", Union(Null)), ("cd", Union(String("7873b372-497b-4953-8832-sdsdsdsdsd"))), ("platform", String("room")), ("tag_id", Int(6310)), ("action_id", Int(6))])

[gklijs]

Indeed via this crate it's not easy to get the schema, but you have the data which is almost as good. I would expect if you create a struct with those 5,6? fields it would work. Else indeed you need some matching like you already did. That's kind of the price that's paid for type safety. Something I did before was having an function to extract the fields I wanted and put them in a struct, to prevent deep nested matches. It's niet that pretty, but readable and safe.

[ultrabug]

I see thank you, I'll continue a bit on this path and if I come up with something acceptable I'll allow myself a PR to demonstrate it.

[ultrabug]

Ok I'd like to add a not to this discussion because as my Rust skills progressed I finally understood what was going on and the problem I was facing come from the avro-rs lib which is actually missing some Union deserializing matches!

I will put up a PR there, this has been a rough ride I must say, and got me dropping the usage of this crate down in the rabbit hole :)

I'll open an issue on other findings related to this crate related to performance

[ultrabug]

ofc, if avro-rs accept my proposal, I'll add documentation here to demonstrate how it's done!

[ultrabug]

As a follow-up in case someone looks for the answer before a new release of apache avro, here is a sample code that demonstrates how to map Avro data to a struct.

#[derive(Default, Debug, Deserialize)]
struct Navigation {
  string_field: String,
  nullable_string_field: Option<String>,
}

// messages is rdkafka::message::OwnedMessage (or BorrowedMessage as you wish)
for msg in &messages {
        let decode_result = avro_decoder.decode(msg.payload()).await;
        match decode_result {
            Ok(decoded) => {
                let _navigation: Navigation = match &decoded.value {
                    Value::Record(_) => from_value::<Navigation>(&decoded.value).unwrap(),
                    _ => unimplemented!("from_value did not return a Value::Record"),
                };
            }
            Err(err) => {
                eprintln!("avro decode error {}", err);
                break;
            }
        }
    }

[e-osuna-g]

I've tried this but with a Partial Struct (just 1 field of many the Record has) an example in here,

#[derive(Debug, Deserialize, PartialEq, Eq)]
struct KafkaMessage {
    ClientMessageId: String,
    }
    match decoder.decode(msg.payload()).await {
        Ok(v) => {
            println!("value:{:?}", v.value);
            let item = match from_value::<KafkaMessage>(&v.value) {
                Ok(item) => {
                    println!("was decoded {:?}", item)
                },
                Err(e)=> println!("Error on decoding kafkaMessage: {:?}",e)
            };
            v
        },
        Err(e) => panic!("Error getting value: {}", e),
    };

i always get this error:

Error on decoding kafkaMessage: DeserializeValue("incorrect value of type: Enum")

is there a way to partially getting the record?

[gklijs]

You could use the 'raw' values from the record. It seems like the enum is not correct, maybe there is some additional nesting. Hard to say from the example. Maybe this help, it's old, but fetching values from the record, instead of using the from_value function.

[e-osuna-g]

I asked because it seems that it is possible, to create from structure with missing fields with the from_value fn, but i'll try the to_cmt method. thanks

[gklijs]

Yes, seems like from_value should work. Maybe it didn't have the expected enum options? Seems like it's actually better/easier to use the from_value fn.