Web Crawling

Brief for Data Extraction / Crawling Requirements We wish to extract BE and BTech Question papers for a select list of 72 subjects from Question paper bank sites. These are University sites as well as other independent sites that have collected these question papers for use in public domain. These question papers need to be from 2015 onwards only. A list of the Universities and the sites will be provided. Additional sources may need to be researched and added to this list to be able to complete the task required to 100% coverage The tasks will require the following output to be submitted: • Question papers by University / Subject in PDF format • Verification by a SME of the Question paper as to its veracity for the subject • Control sheet that displays at a glance the output vs the target Univs / Years and Subjects While this is a onetime activity for now, the same will need to be repeated twice a year to collect updated information as new papers will be released semester wise for the subjects. Response needed: • Proposed Methodology/s – tech-based scraping will be preferred with a manual intervention as needed for tasks that maybe best done via human intervention • Timelines – to be completed in a 60-day time period from commissioning • Cost – project cost linked to outcomes on percentage of completed data provided.