Hello there, my day job requires me to conduct similar projects as yours where I need to compare data from various sources. Recently I processed 30GB text file using Java with 2 mins runtime, parsing the text file into format readable by the target system.
Approach
There are 2 ways to approach this issue.
1. Download the data from MySQL database to a text file in your Linux system, and then run comparison using a Unix script between the two text files.
2. Upload the text file to the MySQL database and run the comparison in MySQL database using outer joins.
Next Step
Please feel free to contact me to discuss further on your file size and I can advise the best way to run the comparison. You can also let me know what constraint you have for example you need the comparison to run in X mins etc.