Python WEB site scraper and storing in MySQL database.

PLEASE DON"T BID I HAVE DEVELOPER SELECTED FOR THIS. Main Crawler - Add new companies, run daily. Get the cookie, add it to the session, If launched parameters are range it goes for Company number N to Company Number N. Else Recovery Crawl For each of failed CI in table do GET REQUEST from HKCRegistry If HTTP 200 Process HTML page through parser that inserts into the database. Remove from failed list Every day start from last succesful download CI number + 1 Get the cookie, add it to the session, do the GET request, If HTTP 200 Process HTML page through parser Check if the CI exists, If exits, updated else inserts into the database. Update the Last successful CI number If you get an error or a page with invalid data record it into the failed GET Record failed CI in failed table. Get session cookie again, and try again. If you get 5 consecute CI numbers fail stop crawl Run Recovery Crawl again before exiting. Refresh Active companies Crawler, Run constantly, should do aboujt 46600 companies per day. Triggers domain expiries (last_checked needs a value the first time it runs) Loop from last_checked company to Last_succesful CI downloaded. Select companies which have got active status HTTP GET CI number form HKCRegistry if Company status has changed update db. If the company is no longer active Write to CI Expire-Domain table. If company name has changed, update db. If Crawled CI has reached Last_succesfull downloaded Set last_checked to First CI of company active in registry. Unbankrupted firms Crawler Run constantly, should do aboujt 10600 companies per day. Triggers domain expiries Track status of last crawl position etc.... Sleect companies which are any status other than active HTTP GET CI number form HKCRegistry if Company status has changed update db. END loop ENDLOOP