Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

a data loss problem about batch insert in mysql #2119

Open
jackjoesh opened this issue Oct 17, 2024 · 1 comment
Open

a data loss problem about batch insert in mysql #2119

jackjoesh opened this issue Oct 17, 2024 · 1 comment

Comments

@jackjoesh
Copy link

jackjoesh commented Oct 17, 2024

Hi @osheroff :
Excuse me, I find a data loss problem about batch insert in mysql.
For example:
If we insert 5 rows once by this sql pattern : insert into t1 values (data1),(data2),(data3),(data4),(data5);
Then we can find all rows's binlogs have the same position in mysql like the following picture:
image

Since Maxwell's Kafka callback function counting on mysql's raw position, Maxwell has data loss risk such as 2 rows(data1,data2) sending to Kafka fail and 3 rows (data3,data4,data5) sending to Kafka success, 3 rows'(data3,data4,data5) callback setting the same position as success and 2 rows(data1,data2) having no chance to re-send.
image

Then I find this open pr, it seems that you expand a sub offset in the same transaction, can this pr fix this problem? And I am curious that why you don't merge this pr to trunk? Does it need more testing?
#2035

And we issued an another data loss problem about a transaction are flowing to different Kafka partitions.
You fixed that by this pr: c20370c, but you reverted it due to this is a dump implemention.
And does this historical problem can also fixed by you new open pr?

These two problems have made our product environment losing some datas, so we are looking forward to getting help from you. Thank you very much!

@jackjoesh
Copy link
Author

jackjoesh commented Dec 9, 2024

is there anyone can help me confirm this problem? thanks @osheroff

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant