Spring 2019 Project 3: Assess Learners - Revision history

Ssung40: /* Rubric */

2019-01-24T21:15:00Z

Rubric

Ssung40: /* Implement InsaneLearner (up to 10 point penalty) */

2019-01-20T05:22:33Z

Implement InsaneLearner (up to 10 point penalty)

Ssung40: /* Required, Allowed & Prohibited */

2019-01-19T21:50:54Z

Required, Allowed & Prohibited

Ssung40: /* Required, Allowed & Prohibited */

2019-01-19T21:40:44Z

Required, Allowed & Prohibited

Ssung40: /* Rubric */

2019-01-19T21:36:29Z

Rubric

Ssung40: /* Experiments and report (50 points) */

2019-01-19T21:25:46Z

Experiments and report (50 points)

Ssung40: /* Implement InsaneLearner (up to 10 point penalty) */

2019-01-19T21:17:02Z

Implement InsaneLearner (up to 10 point penalty)

Ssung40: /* Implement BagLearner (20 points) */

2019-01-19T21:13:31Z

Implement BagLearner (20 points)

Ssung40: /* What to turn in */

2019-01-05T20:53:52Z

What to turn in

Greg at 01:32, 4 January 2019

2019-01-04T01:32:47Z

@@ Line 236: / Line 236: @@
 * Is the report neat and well organized? (-5 points if not)
 * Is the experimental methodology well described (up to -5 points if not)
-* Does the chart properly reflect the intent of the assignment? (up to 10 points if not)
+* Does the chart properly reflect the intent of the assignment? (up to -10 points if not)
 * Overfitting / leaf_size question:

@@ Line 120: / Line 120: @@
 ==Implement InsaneLearner (up to 10 point penalty)==
-Your BagLearner should be able to accept any learner object so long as the learner obeys the API defined above.  We will test this in two ways: 1) By calling your BagLearner with an arbitrarily named class and 2) By having you implement InsaneLearner as described below.  If your code dies in either case, you will lose 10 points. We will also manually inspect your code for correct implementation and grade accordingly.
+Your BagLearner should be able to accept any learner object so long as the learner obeys the API defined above.  We will test this in two ways: 1) By calling your BagLearner with an arbitrarily named class and 2) By having you implement InsaneLearner as described below.  If your code dies in either case, you will lose 10 points. Note, grading script only does a rudimentary check thus we will also manually inspect your code for correct implementation and grade accordingly.
 Using your BagLearner class and the provided LinRegLearner class, implement InsaneLearner as follows: InsaneLearner should contain 20 BagLearner instances where each instance is composed of 20 LinRegLearner instances. We should be able to call your InsaneLearner using the following API:

@@ Line 265: / Line 265: @@
 * Your code must run in less than 10 seconds on one of the university-provided computers.
 * The code you submit should NOT include any data reading routines.  The provided testlearner.py code reads data for you.
-* The code you submit should NOT generate any output: No prints, no charts, etc.
+* The code you submit should NOT generate any output: No prints, no charts, etc. *Note that all charts you provide must be generated in Python (in testlearner.py)
 Allowed:

@@ Line 262: / Line 262: @@
 * Your code must implement a Random Tree learner.
 * Your project must be coded in Python 2.7.x.
-* Your code must run on one of the university-provided computers (e.g. buffet02.cc.gatech.edu).
+* Your code must run on one of the university-provided computers (e.g. buffet01.cc.gatech.edu).
 * Your code must run in less than 10 seconds on one of the university-provided computers.
 * The code you submit should NOT include any data reading routines.  The provided testlearner.py code reads data for you.

@@ Line 224: / Line 224: @@
 *** 1) For out of sample data is correlation with 1 bag lower than correlation for 20 bags?
 *** 2) Does the test complete in less than 10 seconds (i.e. 100 seconds for all 10 tests)?
-* Is the author() method correctly implemented for DTLearner, InsaneLearner, BagLearner and RTLearner? (-10 points for each if not)
+* Is the author() method correctly implemented for all files: DTLearner, InsaneLearner, BagLearner and RTLearner? (-10 points for each if not)
 * Is InsaneLearner correctly implemented in 20 lines or less (-10 points if not)
 * Does BagLearner work correctly with an arbitrarily named class (-10 points if not)
@@ Line 236: / Line 236: @@
 * Is the report neat and well organized? (-5 points if not)
 * Is the experimental methodology well described (up to -5 points if not)
 * Overfitting / leaf_size question:
@@ Line 249: / Line 250: @@
 * Comparison of DT and RT learning
 ** Is each quantitative experiment explained well enough that someone else could reproduce it (up to -5 points if not)
-** Are there at least two new quantitative properties that are compared (not correlation)? (-5 points if only one, -10 if none)
+** Are there at least two new quantitative properties that are compared (do not use correlation)? (-5 points if only one, -10 if none)
 ** Is each conclusion regarding each comparison supported well with charts? (up to -10 points if not)

@@ Line 160: / Line 160: @@
 Create a report that addresses the following questions.  Use 11pt font and single spaced lines. We expect that a complete report addressing all the criteria would be at least 3 pages. It should be no longer than 3000 words. To encourage conciseness we will deduct 10 points if the report is too long. The report should be submitted as <tt>report.pdf</tt> in PDF format.   Include charts (not tables) to support each of your answers.
-* Does overfitting occur with respect to leaf_size? Consider the dataset <tt>istanbul.csv</tt> with DTLearner.  For which values of leaf_size does overfitting occur? Use RMSE as your metric for assessing overfitting. Support your assertion with graphs/charts. (Don't use bagging).
+* Does overfitting occur with respect to leaf_size? Use the dataset <tt>istanbul.csv</tt> with DTLearner.  For which values of leaf_size does overfitting occur? Use RMSE as your metric for assessing overfitting. Support your assertion with graphs/charts. (Don't use bagging).
-* Can bagging reduce or eliminate overfitting with respect to leaf_size?  Again consider the dataset <tt>istanbul.csv</tt> with DTLearner.  To investigate this choose a fixed number of bags to use and vary leaf_size to evaluate. Provide charts to validate your conclusions. Use RMSE as your metric.
+* Can bagging reduce or eliminate overfitting with respect to leaf_size?  Again use the dataset <tt>istanbul.csv</tt> with DTLearner.  To investigate this choose a fixed number of bags to use and vary leaf_size to evaluate. Provide charts to validate your conclusions. Use RMSE as your metric.
-* Quantitatively compare "classic" decision trees (DTLearner) versus random trees (RTLearner).  In which ways is one method better than the other?  Provide at least two quantitative measures. Note that for this part of the report you must conduct new experiments, don't use the results of the experiments above for this.
+* Quantitatively compare "classic" decision trees (DTLearner) versus random trees (RTLearner).  In which ways is one method better than the other?  Provide at least two quantitative measures. Important, using two similar measures that illustrate the same broader metric does not count as two. (For example, do not use two measures for accuracy.) Note for this part of the report you must conduct new experiments, don't use the results of the experiments above for this.
 Note that all charts you provide must be generated in Python (in <tt>testlearner.py</tt>), and you must submit the python code you used to generate the plots.

@@ Line 86: / Line 86: @@
   Y = learner.query(Xtest)
-Where learner is the learning class to use with bagging. You should be able to support any learning class that obeys the API defined above for DTLearner and RTLearner. kwargs are keyword arguments to be passed on to the learner's constructor and they vary according to the learner (see example below). The  "bags" argument is the number of learners you should train using Bootstrap Aggregation.  If boost is true, then you should implement boosting (optional).  If verbose is True, your code can generate output.  Otherwise the code should should be silent.
+Where learner is the learning class to use with bagging. You should be able to support any learning class that obeys the API defined above for DTLearner and RTLearner. kwargs are keyword arguments to be passed on to the learner's constructor and they vary according to the learner (see example below). The  "bags" argument is the number of learners you should train using Bootstrap Aggregation.  If boost is true, then you should implement boosting (optional).  If verbose is True, your code can generate output.  Otherwise the code should be silent.
 As an example, if we wanted to make a random forest of 20 Decision Trees with leaf_size 1 we might call BagLearner as follows

@@ Line 196: / Line 196: @@
 Via Canvas, submit as individual files.
-* Your code as <tt>RTLearner.py</tt>, <TT>DTLearner.py</TT>, <tt>InsaneLearner.py</tt>, <tt>BagLearner.py</tt>, and <tt>testlearner.py</tt>.
+* Your code as <tt>RTLearner.py</tt>, <TT>DTLearner.py</TT>, <tt>InsaneLearner.py</tt>, <tt>BagLearner.py</tt>, and <tt>testlearner.py</tt>. Please submit the code (.py) into the assignment "Project 3: Assess Learners (Code)" only.
-* Your report as <tt>report.pdf</tt>
+* Your report as <tt>report.pdf</tt>. Please submit the report (.pdf) into the assignment "Project 3: Assess Learners (Report)" only.
 Unlimited resubmissions are allowed up to the deadline for the project.

@@ Line 19: / Line 19: @@
 Instructions:
-* Download the appropriate zip file [[File:18spring_assess_learners.zip]]
+* Download the appropriate zip file [[File:19spring_assess_learners.zip]]
 * You should see these files: