• notice
  • Congratulations on the launch of the Sought Tech site

Two lines of code fix the problem of parsing MySQL8.x binlog misalignment! !

write in front

MySQL is one of the most used relational databases in the Internet industry, and MySQL is open source. In-depth research on MySQL can deepen our understanding of database principles. Since the open source of mykit-data, many small partners have reported that mykit-data cannot correctly parse the MySQL8 binlog after trying it out. So I tested it, mykit-data has no problem when parsing the binlog of MySQL 5.x, and can correctly parse the result data. However, when parsing the binlog of MySQL 8.x, there is always a difference of 12 bits from the binlog log, which causes the parsing to fail.

Articles have been included in:



bug fix

It's too late today, I'm still studying the source code of MySQL 8.0.20, and I'll write a detailed article to share with my friends about the repair process. Here, I will directly say how I solved this problem.

The parsing results of MySQL5.x binlog and MySQL8.x binlog always have a digit deviation. The original code of the framework directly parses MySQL 5.x without any problem. When parsing MySQL 8.x, there is a digit error. .

During this period, I read almost all the official documents of MySQL, and rewrote the function of parsing binlog logs in mykit-data. There is no problem in parsing MySQL 5.x, but the parsing of MySQL 8.x is still misplaced.

Where is the problem? Just when I was at a loss to solve the problem, suddenly, I thought of an idea: Isn't it always misplaced when solving MySQL8.x binlog? Then I will read out the binlog data of the excess digits and ignore it directly, so that the subsequent binlog parsing operations are not aligned?

Hurry up and try it, so I added the following code to the source code of the mykit-data framework.

insert image description here

The above code is an encapsulation of the checksum reading for parsing MySQL binlog digits. When the read binlog digits do not reach the read limit digits, the binlog data will be read until the read binlog digits reach The position of the limit digits to read. For the specific internal logic, friends can read the source code of mykit-data.

After adding this logic, test and verify, parsing the binlog of the MySQL 8.x database turned out to be successful! A problem that has been plaguing me for days has been solved inadvertently! !

From the results of solving this problem, the binlog of MySQL8.x is essentially longer than the binlog of MySQL5.x, and the identifiers used to separate different event bits will be spliced in the middle. When we parse the binlog log of MySQL8.x , you can directly ignore these identifiers separating different event bits, the purpose is to align the parsing bits of binlog, so that the next event can be correctly parsed. And doing so will not affect the parsing result.

This is often the case. When you are struggling to solve a problem, and you are unable to find a solution for a long time, at an inadvertent moment, you will inadvertently solve this difficult problem, but the premise is that you need to deeply understand it. principle and try various ways and methods to solve it!

About mykit-data

mykit-data is a completely open source data heterogeneous middleware that supports plug-in and visualized data heterogeneous framework, and supports full, real-time/timed increments from MySQL to MySQL, MySQL to Oracle, Oracle to MySQL, and Oracle to Oracle data synchronization. Complete plug-in, visual operation. The log is used to avoid data loss during the synchronization process to the greatest extent. Support failure retry, manual intervention, support for viewing synchronized data and detailed log information.

Currently supports MySQL5.x, MySQL8.x, Oracle 11g and above. More heterogeneous data sources will be supported in the form of plug-ins in the future.

The open source address of mykit-data is as follows:

GitHub: https://github.com/sunshinelyz/mykit-data

Gitee: https://gitee.com/binghe001/mykit-data

Finally, friends, give a Star for this open source project! !


Technical otaku

Sought technology together

Related Topic


Leave a Reply