Maxwell metadata corruption for Enum-datatype column

[Vrushyaa]

Issue - We observed in 'java.lang.ArrayIndexOutOfBoundsException:' error for column-type 'Enum' while building a configuration object (ColumnDef.build) for a database column definition inside 'ColumnDefDeserializer' class. This caused due to column metadata corruption inside 'column' table of Maxwell metabase.

Resolution - As suggested #1297 here, every time Metadata cleanup is not feasible here.

Error log -
02:44:39,474 INFO BinlogConnectorReplicator - Binlog connected. 02:44:39,658 INFO BinlogConnectorReplicator - Binlog disconnected. 02:44:39,667 INFO TaskManager - Stopping 5 tasks 02:44:39,667 ERROR TaskManager - cause: java.lang.ArrayIndexOutOfBoundsException: Index 9 out of bounds for length 9 at com.zendesk.maxwell.schema.columndef.EnumColumnDef.asString(EnumColumnDef.java:29) ~[pe-maxwell.jar:1.29.1] at com.zendesk.maxwell.schema.columndef.EnumColumnDef.asJSON(EnumColumnDef.java:17) ~[pe-maxwell.jar:1.29.1] at com.zendesk.maxwell.schema.columndef.EnumColumnDef.asJSON(EnumColumnDef.java:5) ~[pe-maxwell.jar:1.29.1] at com.zendesk.maxwell.replication.BinlogConnectorEvent.writeData(BinlogConnectorEvent.java:112) ~[pe-maxwell.jar:1.29.1] at com.zendesk.maxwell.replication.BinlogConnectorEvent.buildRowMap(BinlogConnectorEvent.java:162) ~[pe-maxwell.jar:1.29.1] at com.zendesk.maxwell.replication.BinlogConnectorEvent.jsonMaps(BinlogConnectorEvent.java:176) ~[pe-maxwell.jar:1.29.1] at com.zendesk.maxwell.replication.BinlogConnectorReplicator.getTransactionRows(BinlogConnectorReplicator.java:561) ~[pe-maxwell.jar:1.29.1] at com.zendesk.maxwell.replication.BinlogConnectorReplicator.getRow(BinlogConnectorReplicator.java:700) ~[pe-maxwell.jar:1.29.1] at com.zendesk.maxwell.replication.BinlogConnectorReplicator.work(BinlogConnectorReplicator.java:205) ~[pe-maxwell.jar:1.29.1] at com.zendesk.maxwell.util.RunLoopProcess.runLoop(RunLoopProcess.java:34) ~[pe-maxwell.jar:1.29.1] at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:317) ~[pe-maxwell.jar:1.29.1] at com.zendesk.maxwell.Maxwell.start(Maxwell.java:226) ~[pe-maxwell.jar:1.29.1] at com.zendesk.maxwell.Maxwell.main(Maxwell.java:356) ~[pe-maxwell.jar:1.29.1] 02:44:39,673 INFO TaskManager - Stopping: com.zendesk.maxwell.schema.PositionStoreThread@25d96b94 02:44:39,675 INFO TaskManager - Stopping: com.zendesk.maxwell.producer.PEKafkaProducerWorker@57e4bf5c 02:44:39,675 INFO KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 02:44:39,679 INFO TaskManager - Stopping: com.zendesk.maxwell.bootstrap.BootstrapController@1e5b555a 02:44:39,679 INFO TaskManager - Stopping: com.zendesk.maxwell.replication.BinlogConnectorReplicator@69d9b132 02:44:39,683 INFO TaskManager - Stopping: com.zendesk.maxwell.monitoring.MaxwellHTTPServerWorker@364854fe 02:44:39,683 ERROR - Exception occurred during shutdown:

Issue Query -
INSERT INTO columns (id, schema_id, table_id, name, charset, coltype, is_signed, enum_values, column_length) VALUES ('247561', '5', '21201', 'source_type', NULL, 'enum', '0', '["PICKER_TASK","SIDELINED_PICKER_TASK","INVENTORY","CLEANUP_TASK"]', NULL);

[osheroff]

can you dump the table schema? and also the corresponding row in maxwell.columns.

[Vrushyaa]

Additional - Mysql Version - 8.0.32, Database Name - th124, Table Name - verifier_task, Issue column - "source_type"

Column meta -

columns.csv

Table Schema -

`-- th124.verifier_task definition

CREATE TABLE verifier_task (
id bigint NOT NULL AUTO_INCREMENT,
tray_id varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
verifier_id varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
verifier_name varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
reference_id varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
reference_type enum('INVOICE_JIT','INVOICE','INVOICE_PROCUREMENT','SALE_RETURN','SALE_ORDER','PURCHASE_RETURN','AUDITOR_TASK','RECTIFICATION','JIT_BIN_RACKER_TASK','BIN_MOVEMENT_TASK','ITEM_FOUND_TASK','STOCK_TRANSFER','ALL_RETURN_ORDER','CONVERSION_TASK') COLLATE utf8mb4_unicode_ci DEFAULT NULL,
source_type enum('INVOICE','SALE_RETURN','RACKER_ISSUE','VERIFIER_ISSUE','VERIFIER_TASK','PICKER_TASK','AUDITOR_TASK','AUDITOR_ISSUE','RECTIFICATION','ALL_RETURN_ORDER','CONVERSION_TASK') COLLATE utf8mb4_unicode_ci DEFAULT NULL,
source_id bigint NOT NULL,
purchase_type enum('PROCUREMENT','JIT','StockTransfer','StockTransferReturn','RECTIFICATION','ICS','ICSReturn') CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
status enum('CREATED','ASSIGNED','IN_PROGRESS','PENDING_WITH_ISSUE','COMPLETED','CANCELLED') CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
created_by varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
updated_by_name varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
updated_by varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
created_on datetime(3) NOT NULL,
updated_on datetime(3) NOT NULL,
task_type enum('VERIFIER','BARCODER') CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'VERIFIER',
near_expiry tinyint(1) NOT NULL DEFAULT '0',
bin_allocation_id varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
bin_allocation_type varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
is_archived tinyint(1) NOT NULL DEFAULT '0',
dp_updated_at datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (id),
KEY idx_vt_ri_st (source_id,source_type),
KEY reference_id (reference_id,reference_type),
KEY task_type_index (task_type),
KEY idx_source_type_status_created_on (source_type,status,created_on)
) ENGINE=InnoDB AUTO_INCREMENT=131594 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;`

[osheroff]

do you know when the final enum value (CONVERSION_TASK ) was added to the table? is there a row in maxwell.schemas that corresponds to that schema change? (look in the deltas column).

[osheroff]

actually, please dump any row in maxwell.schemas that refers to that column or table.

[Vrushyaa]

do you know when the final enum value (CONVERSION_TASK ) was added to the table? --> Not sure on this.
deltas column is blank inside 'schemas' table

Please refer - table_id - 26595, schema_id - 5, database_id - 173, name - verifier_task

Meta Dump -
_databases__202412051050.csv
_schemas__202412051051.csv
tables_202412051051.csv

[osheroff]

you're not using any filters or blacklists are you?

There's only a few theories I can think of:

1. Maxwell failed to capture the schema properly, somehow only ending up with a view of 9 enum values instead of 10. I think there's a theoretical race condition where, if the table is altered during the initial capture, maxwell could possibly miss the alter. There also could be other reasons for the missing enum value, but I've never seen anything like that before.
2. Maxwell captured the schema properly, but the additional enum value was added later, and maxwell missed the table alter that added the new enum value. This could happen with blacklists, it could also happen if you manually move maxwell's binlog position around.

[Vrushyaa]

Yes, I am using database BLACKLIST_DB. But the blacklisted counts is around 166 in this deployment.