To reproduce the code, make sure to install all the dependencies and modules with respect to the python script you want to run. The libraries used in each python file are commented right beside the imports
It all started with scraping the data from the https://comelec.gov.ph/?r=2022NLE/ListsofCandidates/TentativeListsofCandidates containing PDF files for each local and national candidates
The first step is to use python script that imports beatifulsoup4 to parse the HTML source code from the website and capture the links associated to the anchor tags. These links are saved to a list in python and after collecting all of the links, it is iterated through a loop and download each file with a user-defined helper function "download(url: str, dest_folder: str, filename: str)"
This process then outputs a set of folders with respect to their location in the format of REGION_NUMBER/PROVINCE/MUNICIPALITY.PNG
The PDF Files are then fed to another python script that merges all the PDF files into 2 parts, since this is in preparation for converting the PDF file to and XLSX file. The limitation occurs in this conversion part where there is a limit of the number of pages that the conversion software can handle. This merging is just done by PyPDF2 module.
Step 4 (Converting XLSX Files to CSV Files)
The XLSX Files can be coverted to a CSV Files through Microsoft Excel by saving it as a CSV File. Since there are 2 XLSX files, then there will also be 2 CSV Files. At this point, these 2 CSV files can be merged to one, but following through the experience, they remained to be separated.
The 2 CSV Files are fed to a python script that uses "csv" module to simplify the workload of iterating through each row of the CSV files. The extraction of data are done based on the Entity Relationship Diagram which are also made as class models in a python file.
At this step, it already requires SQLAlchemy to handle the database. The database contains the following tables