FAQ:Basics

Why isn’t my KSQL query returning data?

Robin Moffatt has written an excellent blog post to help you figure out why. You can find it here.

Why does my Avro map get serialized by KSQL as an array of key-value pairs?

Let's say you have a source topic containing data serialized using Avro, and one of the fields in the value is an Avro map. If you were to simply duplicate the topic using ksqlDB, e.g.

CREATE STREAM DUP AS SELECT * FROM FOO;

Then, justifiably, you might expect the schema of DUP to also have the same Avro map type. However, as of ksqlDB 0.7 this is not the case. An Avro map type of, for example, `{"name":"entityMap","type":{"type":"map","values":"int"}, i.e. a map with string keys and int values, will be converted by KSQL into an array of key-value pairs, for example:

[
   "null",
   {
     "type":"array",
     "items": {
          "type":"record",
          "name":"some-name",
          "fields":[
             {
                "name":"key",
                "type":["null","string"],
                "default":null
             },
             {
                "name":"value",
                "type":["null","int"],
                "default":null
             }
           ]
       }
    }
  ]

Why is this?

This comes about due to the collision of two facts:

1. All columns in KSQL are currently NULLABLE. (At some point ksqlDb will support NOT NULL columns, tracking Github issue).
2. Avro maps don't support null keys.

So, while KSQL will happily deserialize an Avro map into a KSQL map, as the conversion from a non-null key to a nullable key is valid, it can not serialize a KSQL map into an Avro map, as it is possible for a KSQL map to have null keys, which Avro will reject. Instead, KSQL serializes the map as an array of key-value pairs.

Once KSQL supports NON NULL columns and fields it will be able to convert the Avro map type into a compatible KSQL Map type, i.e. one with a non-null key, allowing it to reserialize the data to an Avro map again.