Bug: avro schema with default value for date cannot be parsed

[StrikeW]

Describe the bug

customer reported cannot create a Table with kafka connector in DEBEZIUM AVRO format.

CREATE TABLE data_item  WITH ( CONNECTOR = 'kafka', TOPIC = 'data_item', PROPERTIES.BOOTSTRAP.SERVER = 'xxx', SCAN.STARTUP.MODE = 'earliest' )  FORMAT DEBEZIUM ENCODE AVRO (schema.registry='xxx');

Error message/log

SQL ERROR: ExecuteError: Protocol error: Avro schema parse error One union type Date must match the `default`'s value type Long

To Reproduce

The schema which is suspected to be causing this problem:

{
	"name": "first_insert_dt",
	"type": [
		{
			"type": "int",
			"connect.version": 1,
			"connect.default": 0,
			"connect.name": "org.apache.kafka.connect.data.Date",
			"logicalType": "date"
		},
		"null"
	],
	"default": 0
},

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

risingwave:nightly-20231102

Additional context

No response

[xiangjinwu]

Yet another upstream (apache_avro) bug: #12562 / #13030. I am working on migration from our fork to the official version. But may or may not have bandwidth this and next week.

It has been fixed in apache_avro v0.14.0

It was introduced in upstream since 2022-10-11 (after 0.14.0) and fixed 2023-09-07 (before 0.16.0).

Our fork is based on 2023-03-21, and there are some breaking API changes between 2023-03-21 and 2023-09-07.

    let s = apache_avro::Schema::parse_str(
        r#"{
            "name": "first_insert_dt",
            "type": [
                {
                        "type": "int",
                        "connect.version": 1,
                        "connect.default": 0,
                        "connect.name": "org.apache.kafka.connect.data.Date",
                        "logicalType": "date"
                },
                "null"
            ],
            "default": 0
        }"#,
    )
    .unwrap();

[tabVersion]

I am working on migration from our fork to the official version.

we introduce our crate to handle ref type in Avro, without handling cycle dependency. I am not sure the official sdk has fixed that.

[xiangjinwu]

we introduce our crate to handle ref type in Avro, without handling cycle dependency.

Yes. I have a prototype solution for both ref type and decimal, using the official version rather than our fork.

[xiangjinwu]

Update: The aforementioned schema definition works in upstream apache_avro 0.16.0, but a similar schema definition still fails:

{
    "name": "Root",
    "type": "record",
    "fields": [
        {
        	"name": "first_insert_dt",
	        "type": [
        		{
	        		"type": "int",
		        	"connect.version": 1,
			        "connect.default": 0,
        			"connect.name": "org.apache.kafka.connect.data.Date",
	        		"logicalType": "date"
		        },
        		"null"
	        ],
	        "default": 0
        }
    ]
}