This project is intended to be developed in pairs.  I require separate pre- and post- time estimates from each person.

GRADING CRITERION:

1. 50% scraper functionality
2. 25% web site functionality
3. 20% usability, coding practices, etc.
4. 3% - time pre-estimate
5. 2% - time post-estimate (should not be identical to pre-)

Using Microsoft Visual C#, recreate Pat's classtime finder.  This involves writing a Windows application which can scrape the registrar's website for either fall, spring or summer semester, and it also involves creating an ASP.NET website front end so users can query the scraped data:

RULES:

1. Configurable options for the scraper include at least:
   • fall vs. spring vs. summer
   • URL's of the three websites (must be persistent)
   • output only to files (default) vs. output to files and database
   • staging vs. production database (if database output selected)
2. The graphical user interface (GUI) of the scraper must remain responsive during the scraping process, and must report progress to the GUI, such as how many sections have been identified so far.
3. Scraper must be able to be stopped before completion (confirmation is advisable).
4. By default, your scraper must deposit all scraped information into .csv files on the local disk.  Once the user has determined the scraped .csv files are correct, the scraper can be re-run in a special mode which also deposits the scraped information into a database (either staging or production).
5. If the scraper is in database-update mode, you'll need to clear all pre-existing data out of the MeetingTimes table before writing new data to it.  The code to do this is:

       string theSQL = "delete from MeetingTimes";
       ConnectionSql.Open();
       System.Data.SqlClient.SqlCommand c1 =
          new System.Data.SqlClient.SqlCommand(theSQL, ConnectionSql);
       c1.ExecuteNonQuery();
       ConnectionSql.Close();

6. Your scraper should find at least as many sections as my scraper.  Deduction if less, extra credit if you find legitimate sections which I didn't.  Have a look at my readme file for the statistics.
7. The scraper GUI, when scraping has completed, should display the number of departments, courses and sections found, the URL which was scraped, and the location of the output files.
8. Please create an ASP.NET application which queries the database with an interface at least as good as the classtime finder website:
   
9. Please maintain two copies of the ASP.NET application; the first will pull from a staging database, and the second from the production database.  All changes should first be tested on the staging website and staging database before being copied (after testing) to the production site.  That is, your production site should never be allowed to appear "broken", either due to ASP.NET code or database issues.

RESOURCES and SAMPLE CODE:

1. The URL's to be scraped are:
   • FALL: http://www.upenn.edu/registrar/timetable/
   • SPRING: http://www.upenn.edu/registrar/roster/
   • SUMMER: http://www.upenn.edu/registrar/s-roster/
2. Here is my scraper database specification.  You may optionally use identical tables, which I already created for you in each student database on the http://wren.cis.upenn.edu/ server.
3. Here is Pat's scraper (v5.2); it has the database access part disabled, but you can run it locally and look in c:\reports for the results:
   
4. Here's a .zip file of all of my report files (from scraping all three semesters, as dumped into folder c:\reports).  It contains three files for each semester (departments, courses, and sections).
5. One technique for keeping the GUI responsive (RULE 2) is illustrated in the async delegate sample project that uses a background worker thread (here's its form code).  That is how my scraper does it.