Assess learners: - Revision history

Tucker: Replaced content with "go here assess_learners"

2017-09-14T20:50:26Z

Replaced content with "go here assess_learners"

Show changes

Bhrolenok3: /* Rubric */

2017-09-14T19:50:53Z

Rubric

Bhrolenok3: /* Implement BagLearner (20 points) */

2017-09-14T19:45:44Z

Implement BagLearner (20 points)

Bhrolenok3: /* Experiments and report (50 points) */

2017-09-14T19:43:33Z

Experiments and report (50 points)

Bhrolenok3: /* Implement BagLearner (20 points) */

2017-09-14T19:39:33Z

Implement BagLearner (20 points)

Bhrolenok3: /* Template and Data */

2017-09-14T19:35:44Z

Template and Data

Bhrolenok3: /* Template and Data */

2017-09-14T19:35:22Z

Template and Data

Bhrolenok3: /* Template and Data */

2017-09-14T19:33:58Z

Template and Data

Tucker: /* Implement InsaneLearner (up to 10 point penalty) */

2017-09-14T19:29:41Z

Implement InsaneLearner (up to 10 point penalty)

Tucker: /* Extra Credit (0 points) */

2017-09-14T19:21:19Z

Extra Credit (0 points)

@@ Line 231: / Line 231: @@
 * Is the author() method correctly implemented for DTLearner, InsaneLearner, BagLearner and RTLearner? (-10 points for each if not)
 * Is InsaneLearner correctly implemented in 20 lines or less (-10 points if not)
 Report (50 points):

@@ Line 117: / Line 117: @@
 Boosting is an optional topic and not required.  There's a citation in the Resources section that outlines a method of implementing bagging. If the training set contains n data items, each bag should contain n items as well.  Note that because you should sample with replacement, some of the data items will be repeated.
-This code should not generate statistics or charts. If you want create charts and statistics.
+This code should not generate statistics or charts. If you want create charts and statistics, you can modify <tt>testlearner.py</tt>.
 You can use code like the below to instantiate several learners with the parameters listed in kwargs:

@@ Line 169: / Line 169: @@
 Create a report that addresses the following questions.  Use 11pt font and single spaced lines. We expect that a complete report addressing all the criteria would be at least 3 pages. It should be no longer than 3000 words. To encourage conciseness we will deduct 10 points if the report is too long. The report should be submitted as <tt>report.pdf</tt> in PDF format.  Do not submit word docs or latex files.  Include data as tables or charts to support each of your answers.
-* Does overfitting occur with respect to leaf_size? Consider the dataset <tt>istanbul.csv</tt> with RTLearner.  For which values of leaf_size does overfitting occur? Use RMSE as your metric for assessing overfitting. Support your assertion with graphs/charts. (Don't use bagging).
+* Does overfitting occur with respect to leaf_size? Consider the dataset <tt>istanbul.csv</tt> with both DTLearner and RTLearner.  For which values of leaf_size does overfitting occur? Use RMSE as your metric for assessing overfitting. Support your assertion with graphs/charts. (Don't use bagging).
 * Can bagging reduce or eliminate overfitting with respect to leaf_size?  To investigate this choose a fixed number of bags to use and vary leaf_size to evaluate. Do Random Trees provide any advantage in this respect? Provide charts and/or tables to validate your conclusions.
 * Quantitatively compare "classic" decision trees (DTLearner) versus random trees (RTLearner).  In which ways is one method better than the other?

@@ Line 99: / Line 99: @@
   Y = learner.query(Xtest)
-Where learner is the learning class to use with bagging. You should be able to support any learning class that obeys the API defined above for DTLearner and RTLearner. kwargs are keyword arguments to be passed on to the learner's constructor and they vary according to the learner (see example below).  "bags" is the number of learners you should train using Bootstrap Aggregation.  If boost is true, then you should implement boosting (optional).  If verbose is True, your code can generate output.  Otherwise the code should should be silent.
+Where learner is the learning class to use with bagging. You should be able to support any learning class that obeys the API defined above for DTLearner and RTLearner. kwargs are keyword arguments to be passed on to the learner's constructor and they vary according to the learner (see example below). The  "bags" argument is the number of learners you should train using Bootstrap Aggregation.  If boost is true, then you should implement boosting (optional).  If verbose is True, your code can generate output.  Otherwise the code should should be silent.
 As an example, if we wanted to make a random forest of 20 Decision Trees with leaf_size 1 we might call BagLearner as follows

@@ Line 50: / Line 50: @@
 * istanbul.csv
-We will mainly be working with the istanbul data.  This data includes the returns of multiple worldwide indexes for a number of days in history.  The overall objective is to predict what the return for the MSCI Emerging Markets (EM) index will be on the basis of the other index returns.  Y in this case is the last column to the right, and the X values are the remaining columns to the left (except the first column).  The first column of data in this file is the date, '''which you should ignore.'''
+We will mainly be working with the istanbul data.  This data includes the returns of multiple worldwide indexes for a number of days in history.  The overall objective is to predict what the return for the MSCI Emerging Markets (EM) index will be on the basis of the other index returns.  Y in this case is the last column to the right, and the X values are the remaining columns to the left (except the first column).  The first column of data in this file is the date, '''which you should ignore.'''  The grading script does this automatically for you, but you will have to handle it yourself when working on your report.
 When the auto grader tests your code we will randomly select 60% of the data to train on and use the other 40% for testing.

@@ Line 39: / Line 39: @@
 * <tt>LinRegLearner.py</tt>: An implementation of the LinRegLearner class.  You can use it as a template for implementing your learner classes.
 * <tt>testlearner.py</tt>: Simple testing scaffold that you can use to test your learners.
 In the Data/ directory you will find these files:

@@ Line 32: / Line 32: @@
 Instructions:
-* Update your copy of the class' github repo.  We will send separate instructions by email on how to do that.
+* Update your copy of the class' github repo. Follow the separate instructions on how to do that here: [[ML4T_Software_Setup#Updating_the_repository]].
 You will find these files in the assess_learners directory

@@ Line 130: / Line 130: @@
 Your BagLearner should be able to accept any learner object so long as the learner obeys the API defined above.  We will test this in two ways: 1) By calling your BagLearner with an arbitrarily named class and 2) By having you implement InsaneLearner as described below.  If your code dies in either case, you will lose 10 points.
-Using your BagLearner class and the provided LinRegLearner class, implement InsaneLearner as follows: InsaneLearner should contain 20 bagged learners where each of these learners is 20 bagged LinRegLearners. We should be able to call your InsaneLearner using the following API:
+Using your BagLearner class and the provided LinRegLearner class, implement InsaneLearner as follows: InsaneLearner should contain 20 BagLearner instances where each instance is composed of 20 LinRegLearner instances. We should be able to call your InsaneLearner using the following API:
   import InsaneLearner as it

← Older revision		Revision as of 19:21, 14 September 2017
Line 194:		Line 194:

	* Submit your report regarding boosting as report-boosting.pdf		* Submit your report regarding boosting as report-boosting.pdf
		+
		+	==What to turn in==
		+	Be sure to follow these instructions diligently!
		+
		+	Via T-Square, submit as attachment (no zip files; refer to schedule for deadline).
		+
		+	* Your code as <tt>RTLearner.py</tt>, <TT>DTLearner.py</TT>, <tt>InsaneLearner.py</tt> and <tt>BagLearner.py</tt>.
		+	* Your report as <tt>report.pdf</tt>
		+
		+	Unlimited resubmissions are allowed up to the deadline for the project.

	==Rubric==		==Rubric==