Difference between revisions of "MC3-Project-1"

From Quantitative Analysis Software Courses
Jump to navigation Jump to search
 
(44 intermediate revisions by 2 users not shown)
Line 1: Line 1:
==Draft==
 
  
This is a draft version of the project.  This notice will be removed once the project is finalized.
+
==Updates / FAQs==
 +
 
 +
* '''2017-02-10'''
 +
** Information regarding data source is updated in "Template and Data" section.
 +
** Details regarding author() method are fleshed out a bit.
  
==Updates / FAQs==
+
* '''2017-02-09'''
 +
** switch to new data source, istanbul.csv
 +
** author() method requirement added
 +
** rubric and experiment requirements updated
  
 
* Q: Can I use an ML library or do I have to write the code myself?  A: You must write the decision tree and bagging code yourself.  The LinRegLearner is provided to you. Do not use other libraries or your code will fail the auto grading test cases.
 
* Q: Can I use an ML library or do I have to write the code myself?  A: You must write the decision tree and bagging code yourself.  The LinRegLearner is provided to you. Do not use other libraries or your code will fail the auto grading test cases.
Line 18: Line 24:
 
You are to implement and evaluate three learning algorithms as Python classes: A Random Tree learner, a Linear Regression learner (provided for you) and a Bootstrap Aggregating learner.  The classes should be named RTLearner, LinRegLearner, and BagLearner respectively.  You can use the provided testlearner.py code as a framework for testing your code, we will use similar code in our autograder to test your learners.  Be sure that your solution does not depend on any code in testlearner.py
 
You are to implement and evaluate three learning algorithms as Python classes: A Random Tree learner, a Linear Regression learner (provided for you) and a Bootstrap Aggregating learner.  The classes should be named RTLearner, LinRegLearner, and BagLearner respectively.  You can use the provided testlearner.py code as a framework for testing your code, we will use similar code in our autograder to test your learners.  Be sure that your solution does not depend on any code in testlearner.py
  
We are considering this a <b>regression</b> problem (not classification).  So the goal is to return a continuous numerical result (not a discrete result).  In this project we are training & testing with static spatial data. In a later project we will make the transition to time series data.
+
We are considering this a <b>regression</b> problem (not classification).  So the goal is to return a continuous numerical result (not a discrete result).  In this project we are ignoring the time aspect of the data and treating it as if it is static data and time does not matter. In a later project we will make the transition to time series data.
  
 
You must write your own code for Random Tree learning and bagging. You are NOT allowed to use other peoples' code to implement Random Trees or bagging.
 
You must write your own code for Random Tree learning and bagging. You are NOT allowed to use other peoples' code to implement Random Trees or bagging.
Line 24: Line 30:
 
The project has two main components: The code for your learners, which will be auto graded, and your report, <tt>report.pdf</tt> that should include the components listed below.
 
The project has two main components: The code for your learners, which will be auto graded, and your report, <tt>report.pdf</tt> that should include the components listed below.
  
Your learner should be able to handle any dimension in X from 1 to N.
+
Your learner should be able to handle any dimension in X from 2 to N.
  
 
==Reference Material==
 
==Reference Material==
Line 53: Line 59:
  
 
==Template and Data==
 
==Template and Data==
 
'''note: we will probably be using a different dataset for Spring 2017''' You can still get started with this dataset, but be prepared for it to change.
 
  
 
Instructions:
 
Instructions:
Line 65: Line 69:
 
* <tt>testlearner.py</tt>: Helper code to test a learner class.
 
* <tt>testlearner.py</tt>: Helper code to test a learner class.
  
In the Data/ directory there are six files:
+
In the Data/ directory you will find these files:
 
* 3_groups.csv  
 
* 3_groups.csv  
 
* ripple_.csv
 
* ripple_.csv
Line 72: Line 76:
 
* winequality-white.csv
 
* winequality-white.csv
 
* winequality.names.txt
 
* winequality.names.txt
 +
* istanbul.csv
  
We will mainly be working with the wine-based data.  The other files are there as alternative sets for you to test your code on. Each data file contains N+1 columns: X1, X2, ... XN, and YWhen we test your code we will randomly select 60% of the data to train on and use the other 40% for testingHowever, as of this writing, the testlearner.py code uses the <b>first 60% of the data for training</b>, and the <b>remaining 40% for testing</b>That may be helpful because it will enable you to compare results with your friends on piazza.
+
We will mainly be working with the istanbul data.  This data includes the returns of multiple worldwide indexes for a number of days in historyThe overall objective is to predict what the return for the MSCI Emerging Markets (EM) index will be on the basis of the other index returnsY in this case is the last column to the right, and the X values are the remaining columns to the left (except the first column)The first column of data in this file is the date, '''which you should ignore.''' 
  
The wine data is also available here: [[Media:wine-data.zip]]
+
When we test your code we will randomly select 60% of the data to train on and use the other 40% for testing.  However, as of this writing, the testlearner.py code uses the <b>first 60% of the data for training</b>, and the <b>remaining 40% for testing</b>.  That may be helpful because it will enable you to compare results with your friends on piazza.
  
==Part 1: Implement RTLearner (30%)==
+
The other files, besides istanbul.csv are there as alternative sets for you to test your code on. Each data file contains N+1 columns: X1, X2, ... XN, and Y. 
 +
 
 +
The istanbul data is also available here: [[File:istanbul.csv]]
 +
 
 +
==Part 1: Implement RTLearner (40%)==
  
 
You should implement a Random Tree learner class in the file <tt>RTLearner.py</tt>.  You should consult the [http://www.interfacesymposia.org/I01/I2001Proceedings/ACutler/ACutler.pdf paper by Adele Cutler] as a reference.  Note that for this part of the project, your code should only build a single tree (not a forest).  We'll get to forests later in the project.  The primary differences between Cutler's Random Tree and the methodology originally proposed by [https://wwwold.cs.umd.edu/class/fall2009/cmsc828r/PAPERS/fulltext_Quilan_Ashwin_Kumar.pdf JR Quinlan] are:
 
You should implement a Random Tree learner class in the file <tt>RTLearner.py</tt>.  You should consult the [http://www.interfacesymposia.org/I01/I2001Proceedings/ACutler/ACutler.pdf paper by Adele Cutler] as a reference.  Note that for this part of the project, your code should only build a single tree (not a forest).  We'll get to forests later in the project.  The primary differences between Cutler's Random Tree and the methodology originally proposed by [https://wwwold.cs.umd.edu/class/fall2009/cmsc828r/PAPERS/fulltext_Quilan_Ashwin_Kumar.pdf JR Quinlan] are:
Line 101: Line 110:
 
==Part 2: Implement BagLearner (20%)==
 
==Part 2: Implement BagLearner (20%)==
  
Implement Bootstrap Aggregating as a Python class named BagLearner.  Your BagLearner class should be implemented in the file <tt>BagLearner.py</tt>.  It should support EXACTLY the API defined below.  You should implement the following functions/methods:
+
Implement Bootstrap Aggregating as a Python class named BagLearner.  Your BagLearner class should be implemented in the file <tt>BagLearner.py</tt>.  It should support EXACTLY the API defined below.  This API is designed so that BagLearner can accept any learner (e.g., RTLearner, LinRegLearner, even another BagLearner) as input and use it to generate a learner ensemble.  Your BagLearner should support the following function/method prototypes:
 
   
 
   
 
  import BagLearner as bl
 
  import BagLearner as bl
Line 116: Line 125:
 
This code should not generate statistics or charts. If you want create charts and statistics, modify testlearner.py for that purpose.
 
This code should not generate statistics or charts. If you want create charts and statistics, modify testlearner.py for that purpose.
  
==Part 3: Experiments and report (50%)==
+
==Part 3: Implement InsaneLearner (up to 10% penalty)==
 +
 
 +
Using your BagLearner class and the provided LinRegLearner class, implement InsaneLearner as follows: InsaneLearner should contain 20 bagged learners where each of these learners is composed of 20 bagged LinRegLearners. We should be able to call your InsaneLearner using the following API:
 +
 
 +
import InsaneLearner as it
 +
learner = it.InsaneLearner(verbose = False) # constructor
 +
learner.addEvidence(Xtrain, Ytrain) # training step
 +
Y = learner.query(Xtest) # query
 +
 
 +
The code for InsaneLearner should be less than 20 lines.  There is no credit for this, but a penalty if it is not implemented correctly.
 +
 
 +
==Part 4: Implement author() Method (up to 10% penalty)==
 +
 
 +
For BOTH BagLearner.py and RTLearner.py you should implement a method called <tt>author()</tt> that returns your Georgia Tech user ID as a string. This is the ID you use to log into t-square.  It is not your 9 digit student number.  Here is an example of how you might implement author() within a learner object:
 +
 
 +
<pre>
 +
class LinRegLearner(object):
 +
 
 +
    def __init__(self):
 +
        pass # move along, these aren't the drones you're looking for
 +
 
 +
    def author(self):
 +
        return 'tb34' # replace tb34 with your Georgia Tech username.
 +
</pre>
 +
 
 +
And here's an example of how it could be called from a testing program:
 +
 
 +
<pre>
 +
    # create a learner and train it
 +
    learner = lrl.LinRegLearner() # create a LinRegLearner
 +
    learner.addEvidence(trainX, trainY) # train it
 +
    print learner.author()
 +
</pre>
 +
 
 +
Check the template code for examples. We are adding those to the repo now, but it might not be there if you check right away.  Implementing this method correctly does not provide any points, but there will be a penalty for not implementing it.
 +
 
 +
==Part 5: Experiments and report (50%)==
  
 
Create a report that addresses the following questions.  Use 11pt font and single spaced lines. We expect that a complete report addressing all the criteria would be at least 3 pages. It should be no longer than 6 pages including charts, tables and text. To encourage conciseness we will deduct 2% for each page over 6 pages. The report should be submitted as <tt>report.pdf</tt> in PDF format.  Do not submit word docs or latex files.  Include data as tables or charts to support each of your answers.   
 
Create a report that addresses the following questions.  Use 11pt font and single spaced lines. We expect that a complete report addressing all the criteria would be at least 3 pages. It should be no longer than 6 pages including charts, tables and text. To encourage conciseness we will deduct 2% for each page over 6 pages. The report should be submitted as <tt>report.pdf</tt> in PDF format.  Do not submit word docs or latex files.  Include data as tables or charts to support each of your answers.   
  
* Consider the dataset <tt>winequality-white.csv</tt> with RTLearner.  For which values of leaf_size does overfitting occur? Use RMSE as your metric for assessing overfitting. Support your assertion with graphs/charts. (Don't use bagging).
+
* Does overfitting occur with respect to leaf_size? Consider the dataset <tt>istanbul.csv</tt> with RTLearner.  For which values of leaf_size does overfitting occur? Use RMSE as your metric for assessing overfitting. Support your assertion with graphs/charts. (Don't use bagging).
* Now use bagging in conjunction with RT with the <tt>winequality-red.csv</tt> dataset.  Choose some leaf_size and keep it fixed. How does performance (prediction quality) vary as you increase the number of bags?  Does overfitting occur with respect to the number of bags?
+
* Can bagging reduce or eliminate overfitting with respect to leaf_size?  Fix the number of bags and vary leaf_size to investigate. Provide charts and or tables to validate your conclusion.
* Can bagging reduce or eliminate overfitting with respect to leaf_size for the <tt>winequality_white</tt> dataset?  Fix the number of bags and vary leaf_size to investigate. Provide charts and or tables to validate your conclusion.
+
* Does overfitting occur with respect to number of bags? Choose some leaf_size and keep it fixed. How does RMSE vary as you increase the number of bags?  Does overfitting occur with respect to the number of bags? Support your assertion with graphs/charts.
 +
 
 +
==Hints & resources==
 +
 
 +
Some external resources that might be useful for this project:
 +
 
 +
* You may be interested to take a look at Andew Moore's slides on [http://www.autonlab.org/tutorials/mbl.html instance based learning].
 +
* A definition of [http://mathworld.wolfram.com/StatisticalCorrelation.html correlation] which we'll use to assess the quality of the learning.
 +
* [https://en.wikipedia.org/wiki/Bootstrap_aggregating Bootstrap Aggregating]
 +
* [https://en.wikipedia.org/wiki/AdaBoost AdaBoost]
 +
* [http://docs.scipy.org/doc/numpy/reference/generated/numpy.corrcoef.html numpy corrcoef]
 +
* [http://docs.scipy.org/doc/numpy/reference/generated/numpy.argsort.html numpy argsort]
 +
* [http://en.wikipedia.org/wiki/Root_mean_square RMS error]
 +
 
 +
You can use code like the below to instantiate several learners with the parameters listed in kwargs:
 +
 
 +
<pre>
 +
learners = []
 +
kwargs = {"k":10}
 +
for i in range(0,bags):
 +
    learners.append(learner(**kwargs))
 +
</pre>
  
 
==What to turn in==
 
==What to turn in==
 
Be sure to follow these instructions diligently!
 
Be sure to follow these instructions diligently!
  
Via T-Square, submit as attachment (no zip files; refer to schedule for deadline).  Note that this project appears as two different assignments on T-Square (one for the report and one for the code).
+
Via T-Square, submit as attachment (no zip files; refer to schedule for deadline).   
  
* Your code as <tt>RTLearner.py</tt> and <tt>BagLearner.py</tt>.
+
* Your code as <tt>RTLearner.py</tt> <tt>InsaneLearner.py</tt> and <tt>BagLearner.py</tt>.
 
* Your report as <tt>report.pdf</tt>
 
* Your report as <tt>report.pdf</tt>
  
Line 136: Line 202:
 
==Extra Credit (0%)==
 
==Extra Credit (0%)==
  
Implement boosting as part of BagLearner.  How does boosting affect performance for <tt>winequality_red</tt> data compared to not boosting?  
+
Implement boosting as part of BagLearner.  How does boosting affect performance compared to not boosting? Does overfitting occur as the number of bags with boosting increases? Create your own dataset for which overfitting occurs as the number of bags with boosting increases.
Does overfitting occur for either dataset as the number of bags with boosting increases?
 
Create your own dataset for which overfitting occurs as the number of bags with boosting increases.
 
  
* Your report regarding boosting as report-boosting.pdf
+
* Submit your report regarding boosting as report-boosting.pdf
  
 
==Rubric==
 
==Rubric==
  
 
Code (60 points):
 
Code (60 points):
* RTLearner in sample/out of sample test, auto grade 10 test cases (using winequality_white.csv), 4 points each: 40 points.
+
* RTLearner in sample/out of sample test, auto grade 10 test cases (8 using istanbul.csv, 2 using another data set), 4 points each: 40 points.
 
** For each test 60% of the data will be selected at random for training and 40% will be selected for testing.
 
** For each test 60% of the data will be selected at random for training and 40% will be selected for testing.
 
** Success criteria for each of the 10 tests:  
 
** Success criteria for each of the 10 tests:  
Line 152: Line 216:
 
*** 3) Is the correlation between predicted and actual results for '''in sample''' data  below 0.95 with leaf_size = 50?
 
*** 3) Is the correlation between predicted and actual results for '''in sample''' data  below 0.95 with leaf_size = 50?
 
*** 4) Does the test complete in less than 3 seconds (i.e. 30 seconds for all 10 tests)?
 
*** 4) Does the test complete in less than 3 seconds (i.e. 30 seconds for all 10 tests)?
* BagLearner, auto grade 10 test cases (using winequality_red.csv), 2 points each 20 points
+
* BagLearner, auto grade 10 test cases (8 using istanbul.csv, 2 using another data set), 2 points each 20 points
 
** For each test 60% of the data will be selected at random for training and 40% will be selected for testing.
 
** For each test 60% of the data will be selected at random for training and 40% will be selected for testing.
 
** leaf_size = 20
 
** leaf_size = 20
Line 158: Line 222:
 
*** 1) For out of sample data is correlation with 1 bag lower than correlation for 20 bags?
 
*** 1) For out of sample data is correlation with 1 bag lower than correlation for 20 bags?
 
*** 2) Does the test complete in less than 5 seconds (i.e. 50 seconds for all 10 tests)?
 
*** 2) Does the test complete in less than 5 seconds (i.e. 50 seconds for all 10 tests)?
 +
* Is the author() method correctly implemented for BagLearner and RTLearner? (-10% for each if not)
 +
* Is InsaneLearner correctly implemented and 20 lines or less (-10% if not)
  
 
Report (40 points):
 
Report (40 points):
 +
 +
* Is the report neat and well organized? (-5 points if not)
 +
* Is the experimental methodology well described (-5 points if not)
 +
 
* Overfitting / leaf_size question:
 
* Overfitting / leaf_size question:
** Is data (either a chart or table) provided to support the argument? (5 points)
+
** Is data (either a chart or table) provided to support the argument? (-5 points if not)
** Does the student state where the region of overfitting occurs (or state that there is no overfitting)? (5 points)
+
** Does the student state where the region of overfitting occurs (or state that there is no overfitting)? (-5 points if not)
** Are the starting point and direction of overfitting identified supported by the data (or if the student states that there is no overfitting, is that supported by the data)? (5 points)
+
** Are the starting point and direction of overfitting identified supported by the data (or if the student states that there is no overfitting, is that supported by the data)? (-5 points if not)
  
 
* Overfitting / leaf_size fixed, change number of bags:
 
* Overfitting / leaf_size fixed, change number of bags:
** Is data (either a chart or table) provided to support the argument? (5 points)
+
** Is data (either a chart or table) provided to support the argument? (-5 points if not)
** Does the student state where the region of overfitting occurs (or state that there is no overfitting)? (5 points)
+
** Does the student state where the region of overfitting occurs (or state that there is no overfitting)? (-5 points if not)
** Are the starting point and direction of overfitting identified supported by the data (or if the student states that there is no overfitting, is that supported by the data)? (5 points)
+
** Are the starting point and direction of overfitting identified supported by the data (or if the student states that there is no overfitting, is that supported by the data)? (-5 points if not)
  
 
* Overfitting / bagging fixed, change leaf_size:
 
* Overfitting / bagging fixed, change leaf_size:
** Is data (either a chart or table) provided to support the argument? (5 points)
+
** Is data (either a chart or table) provided to support the argument? (-5 points if not)
** Does the student state where the region of overfitting occurs (or state that there is no overfitting)? (2.5 points)
+
** Does the student state where the region of overfitting occurs (or state that there is no overfitting)? (-5 points if not)
** Are the starting point and direction of overfitting identified supported by the data (or if the student states that there is no overfitting, is that supported by the data)? (2.5 points)
+
** Are the starting point and direction of overfitting identified supported by the data (or if the student states that there is no overfitting, is that supported by the data)? (-5 points if not)
 +
 
 +
* Was the report exceptionally well done? (+1%)
  
 
==Required, Allowed & Prohibited==
 
==Required, Allowed & Prohibited==
Line 180: Line 252:
 
* Your code must implement a Random Tree learner.
 
* Your code must implement a Random Tree learner.
 
* Your project must be coded in Python 2.7.x.
 
* Your project must be coded in Python 2.7.x.
* Your code must run on one of the university-provided computers (e.g. buffet02.cc.gatech.edu), or on one of the provided virtual images.
+
* Your code must run on one of the university-provided computers (e.g. buffet02.cc.gatech.edu).
 
* Your code must run in less than 5 seconds on one of the university-provided computers.
 
* Your code must run in less than 5 seconds on one of the university-provided computers.
 
* The code you submit should NOT include any data reading routines.  The provided testlearner.py code reads data for you.
 
* The code you submit should NOT include any data reading routines.  The provided testlearner.py code reads data for you.
Line 203: Line 275:
 
==Acknowledgements and Citations==
 
==Acknowledgements and Citations==
  
The data used in this assignment was provided by [https://archive.ics.uci.edu/ml/datasets/Wine+Quality UCI's ML Datasets]
+
The data used in this assignment was provided by [http://archive.ics.uci.edu/ml/datasets/ISTANBUL+STOCK+EXCHANGE UCI's ML Datasets].
 
 
Data Set Information:
 
 
 
The two datasets are related to red and white variants of the Portuguese "Vinho Verde" wine. For more details, consult: [Web Link] or the reference [Cortez et al., 2009]. Due to privacy and logistic issues, only physicochemical (inputs) and sensory (the output) variables are available (e.g. there is no data about grape types, wine brand, wine selling price, etc.).
 
 
 
These datasets can be viewed as classification or regression tasks. The classes are ordered and not balanced (e.g. there are munch more normal wines than excellent or poor ones). Outlier detection algorithms could be used to detect the few excellent or poor wines. Also, we are not sure if all input variables are relevant. So it could be interesting to test feature selection methods.
 
 
 
Attribute Information:
 
 
 
For more information, read [Cortez et al., 2009].
 
Input variables (based on physicochemical tests):
 
# fixed acidity
 
# volatile acidity
 
# citric acid
 
# residual sugar
 
# chlorides
 
# free sulfur dioxide
 
# total sulfur dioxide
 
# density
 
# pH
 
# sulphates
 
# alcohol
 
 
 
Output variable (based on sensory data):
 
* quality (score between 0 and 10)
 
 
 
Please include this citation if you plan to use this database:
 
 
 
P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis.
 
Modeling wine preferences by data mining from physicochemical properties. In Decision Support Systems, Elsevier, 47(4):547-553, 2009.
 
  
 
==Legacy==
 
==Legacy==
  
 
[[MC3-Project-1-legacy]]
 
[[MC3-Project-1-legacy]]

Latest revision as of 17:47, 14 June 2017

Updates / FAQs

  • 2017-02-10
    • Information regarding data source is updated in "Template and Data" section.
    • Details regarding author() method are fleshed out a bit.
  • 2017-02-09
    • switch to new data source, istanbul.csv
    • author() method requirement added
    • rubric and experiment requirements updated
  • Q: Can I use an ML library or do I have to write the code myself? A: You must write the decision tree and bagging code yourself. The LinRegLearner is provided to you. Do not use other libraries or your code will fail the auto grading test cases.
  • Q: Which libraries am I allowed to use? Which library calls are prohibited? A: The use of classes that create and maintain their own data structures are prohibited. So for instance, use of scipy.spatial.KDTree is not allowed because it builds a tree and keeps that data structure around for reference later. The intent for this project is that YOU should be building and maintaining the data structures necessary. You can, however, use methods that return immediate results and do not retain data structures
    • Examples of things that are allowed: sqrt(), sort(), argsort() -- note that these methods return an immediate value and do not retain data structures for later use.
    • Examples of things that are prohibited: any scikit add on library, scipy.spatial.KDTree, importing things from libraries other than pandas, numpy or scipy.
  • Q: How should I read in the data? A: Your code does not need to read in data, that is handled for you in the testlearner.py code. For testing your code you can modify testlearner.py to read in different datasets, but your solution should NOT depend on any special code in testlearner.py
  • Q: How many data items should be in each bag? A: If the training set is of size N, each bag should contain N items. Note that since sampling is with replacement some of the data items will be repeated.

Overview

You are to implement and evaluate three learning algorithms as Python classes: A Random Tree learner, a Linear Regression learner (provided for you) and a Bootstrap Aggregating learner. The classes should be named RTLearner, LinRegLearner, and BagLearner respectively. You can use the provided testlearner.py code as a framework for testing your code, we will use similar code in our autograder to test your learners. Be sure that your solution does not depend on any code in testlearner.py

We are considering this a regression problem (not classification). So the goal is to return a continuous numerical result (not a discrete result). In this project we are ignoring the time aspect of the data and treating it as if it is static data and time does not matter. In a later project we will make the transition to time series data.

You must write your own code for Random Tree learning and bagging. You are NOT allowed to use other peoples' code to implement Random Trees or bagging.

The project has two main components: The code for your learners, which will be auto graded, and your report, report.pdf that should include the components listed below.

Your learner should be able to handle any dimension in X from 2 to N.

Reference Material

"Official" course-based materials:

Additional supporting materials:

You can use code like the below to instantiate several learners with the parameters listed in kwargs:

learners = []
kwargs = {"k":10}
for i in range(0,bags):
    learners.append(learner(**kwargs))

Template and Data

Instructions:

  • Update your copy of the class' github repo. We will send separate instructions by email on how to do that.

You will find these files in the mc3_p1 directory

  • Data/: Contains data for you to test your learning code on.
  • LinRegLearner.py: An implementation of the LinRegLearner class. You can use it as a template for implementing your learner classes.
  • testlearner.py: Helper code to test a learner class.

In the Data/ directory you will find these files:

  • 3_groups.csv
  • ripple_.csv
  • simple.csv
  • winequality-red.csv
  • winequality-white.csv
  • winequality.names.txt
  • istanbul.csv

We will mainly be working with the istanbul data. This data includes the returns of multiple worldwide indexes for a number of days in history. The overall objective is to predict what the return for the MSCI Emerging Markets (EM) index will be on the basis of the other index returns. Y in this case is the last column to the right, and the X values are the remaining columns to the left (except the first column). The first column of data in this file is the date, which you should ignore.

When we test your code we will randomly select 60% of the data to train on and use the other 40% for testing. However, as of this writing, the testlearner.py code uses the first 60% of the data for training, and the remaining 40% for testing. That may be helpful because it will enable you to compare results with your friends on piazza.

The other files, besides istanbul.csv are there as alternative sets for you to test your code on. Each data file contains N+1 columns: X1, X2, ... XN, and Y.

The istanbul data is also available here: File:Istanbul.csv

Part 1: Implement RTLearner (40%)

You should implement a Random Tree learner class in the file RTLearner.py. You should consult the paper by Adele Cutler as a reference. Note that for this part of the project, your code should only build a single tree (not a forest). We'll get to forests later in the project. The primary differences between Cutler's Random Tree and the methodology originally proposed by JR Quinlan are:

  1. The feature i to split on at each level is determined randomly. It is not determined using information gain or correlation, etc.
  2. The split value for each node is determined by: Randomly selecting two samples of data and taking the mean of their Xi values.

Your code should support exactly the API defined below. DO NOT import any modules besides those listed in the prohibited/allowed section below. You should implement the following functions/methods:

import RTLearner as rt
learner = rt.RTLearner(leaf_size = 1, verbose = False) # constructor
learner.addEvidence(Xtrain, Ytrain) # training step
Y = learner.query(Xtest) # query

Where "leaf_size" is the maximum number of samples to be aggregated at a leaf. While the tree is being constructed recursively, if there are leaf_size or fewer elements at the time of the recursive call, the data should be aggregated into a leaf.

Xtrain and Xtest should be ndarrays (numpy objects) where each row represents an X1, X2, X3... XN set of feature values. The columns are the features and the rows are the individual example instances. Y and Ytrain are single dimension ndarrays that indicate the value we are attempting to predict with X.

If "verbose" is True, your code can print out information for debugging. If verbose = False your code should not generate ANY output. When we test your code, verbose will be False.

This code should not generate statistics or charts. You may modify testlearner.py to generate statistics and charts.

Part 2: Implement BagLearner (20%)

Implement Bootstrap Aggregating as a Python class named BagLearner. Your BagLearner class should be implemented in the file BagLearner.py. It should support EXACTLY the API defined below. This API is designed so that BagLearner can accept any learner (e.g., RTLearner, LinRegLearner, even another BagLearner) as input and use it to generate a learner ensemble. Your BagLearner should support the following function/method prototypes:

import BagLearner as bl
learner = bl.BagLearner(learner = rt.RTLearner, kwargs = {"leaf_size":1}, bags = 20, boost = False, verbose = False)
learner.addEvidence(Xtrain, Ytrain)
Y = learner.query(Xtest)

Where learner is the learning class to use with bagging. kwargs are keyword arguments to be passed on to the learner's constructor and they vary according to the learner (see hints above). "bags" is the number of learners you should train using Bootstrap Aggregation. If boost is true, then you should implement boosting.

If verbose is True, your code can generate output. Otherwise it should be silent.

Notes: See hints section above for example code you might use to instantiate your learners. Boosting is an optional topic and not required. There's a citation below in the Resources section that outlines a method of implementing bagging. If the training set contains n data items, each bag should contain n items as well. Note that because you should sample with replacement, some of the data items will be repeated.

This code should not generate statistics or charts. If you want create charts and statistics, modify testlearner.py for that purpose.

Part 3: Implement InsaneLearner (up to 10% penalty)

Using your BagLearner class and the provided LinRegLearner class, implement InsaneLearner as follows: InsaneLearner should contain 20 bagged learners where each of these learners is composed of 20 bagged LinRegLearners. We should be able to call your InsaneLearner using the following API:

import InsaneLearner as it
learner = it.InsaneLearner(verbose = False) # constructor
learner.addEvidence(Xtrain, Ytrain) # training step
Y = learner.query(Xtest) # query

The code for InsaneLearner should be less than 20 lines. There is no credit for this, but a penalty if it is not implemented correctly.

Part 4: Implement author() Method (up to 10% penalty)

For BOTH BagLearner.py and RTLearner.py you should implement a method called author() that returns your Georgia Tech user ID as a string. This is the ID you use to log into t-square. It is not your 9 digit student number. Here is an example of how you might implement author() within a learner object:

class LinRegLearner(object):

    def __init__(self):
        pass # move along, these aren't the drones you're looking for

    def author(self):
        return 'tb34' # replace tb34 with your Georgia Tech username.

And here's an example of how it could be called from a testing program:

    # create a learner and train it
    learner = lrl.LinRegLearner() # create a LinRegLearner
    learner.addEvidence(trainX, trainY) # train it
    print learner.author()

Check the template code for examples. We are adding those to the repo now, but it might not be there if you check right away. Implementing this method correctly does not provide any points, but there will be a penalty for not implementing it.

Part 5: Experiments and report (50%)

Create a report that addresses the following questions. Use 11pt font and single spaced lines. We expect that a complete report addressing all the criteria would be at least 3 pages. It should be no longer than 6 pages including charts, tables and text. To encourage conciseness we will deduct 2% for each page over 6 pages. The report should be submitted as report.pdf in PDF format. Do not submit word docs or latex files. Include data as tables or charts to support each of your answers.

  • Does overfitting occur with respect to leaf_size? Consider the dataset istanbul.csv with RTLearner. For which values of leaf_size does overfitting occur? Use RMSE as your metric for assessing overfitting. Support your assertion with graphs/charts. (Don't use bagging).
  • Can bagging reduce or eliminate overfitting with respect to leaf_size? Fix the number of bags and vary leaf_size to investigate. Provide charts and or tables to validate your conclusion.
  • Does overfitting occur with respect to number of bags? Choose some leaf_size and keep it fixed. How does RMSE vary as you increase the number of bags? Does overfitting occur with respect to the number of bags? Support your assertion with graphs/charts.

Hints & resources

Some external resources that might be useful for this project:

You can use code like the below to instantiate several learners with the parameters listed in kwargs:

learners = []
kwargs = {"k":10}
for i in range(0,bags):
    learners.append(learner(**kwargs))

What to turn in

Be sure to follow these instructions diligently!

Via T-Square, submit as attachment (no zip files; refer to schedule for deadline).

  • Your code as RTLearner.py InsaneLearner.py and BagLearner.py.
  • Your report as report.pdf

Unlimited resubmissions are allowed up to the deadline for the project.

Extra Credit (0%)

Implement boosting as part of BagLearner. How does boosting affect performance compared to not boosting? Does overfitting occur as the number of bags with boosting increases? Create your own dataset for which overfitting occurs as the number of bags with boosting increases.

  • Submit your report regarding boosting as report-boosting.pdf

Rubric

Code (60 points):

  • RTLearner in sample/out of sample test, auto grade 10 test cases (8 using istanbul.csv, 2 using another data set), 4 points each: 40 points.
    • For each test 60% of the data will be selected at random for training and 40% will be selected for testing.
    • Success criteria for each of the 10 tests:
      • 1) Does the correlation between predicted and actual results for in sample data exceed 0.95 with leaf_size = 1?
      • 2) Does the correlation between predicted and actual results for out of sample data exceed 0.15 with leaf_size=1?
      • 3) Is the correlation between predicted and actual results for in sample data below 0.95 with leaf_size = 50?
      • 4) Does the test complete in less than 3 seconds (i.e. 30 seconds for all 10 tests)?
  • BagLearner, auto grade 10 test cases (8 using istanbul.csv, 2 using another data set), 2 points each 20 points
    • For each test 60% of the data will be selected at random for training and 40% will be selected for testing.
    • leaf_size = 20
    • Success criteria for each run of the 10 tests:
      • 1) For out of sample data is correlation with 1 bag lower than correlation for 20 bags?
      • 2) Does the test complete in less than 5 seconds (i.e. 50 seconds for all 10 tests)?
  • Is the author() method correctly implemented for BagLearner and RTLearner? (-10% for each if not)
  • Is InsaneLearner correctly implemented and 20 lines or less (-10% if not)

Report (40 points):

  • Is the report neat and well organized? (-5 points if not)
  • Is the experimental methodology well described (-5 points if not)
  • Overfitting / leaf_size question:
    • Is data (either a chart or table) provided to support the argument? (-5 points if not)
    • Does the student state where the region of overfitting occurs (or state that there is no overfitting)? (-5 points if not)
    • Are the starting point and direction of overfitting identified supported by the data (or if the student states that there is no overfitting, is that supported by the data)? (-5 points if not)
  • Overfitting / leaf_size fixed, change number of bags:
    • Is data (either a chart or table) provided to support the argument? (-5 points if not)
    • Does the student state where the region of overfitting occurs (or state that there is no overfitting)? (-5 points if not)
    • Are the starting point and direction of overfitting identified supported by the data (or if the student states that there is no overfitting, is that supported by the data)? (-5 points if not)
  • Overfitting / bagging fixed, change leaf_size:
    • Is data (either a chart or table) provided to support the argument? (-5 points if not)
    • Does the student state where the region of overfitting occurs (or state that there is no overfitting)? (-5 points if not)
    • Are the starting point and direction of overfitting identified supported by the data (or if the student states that there is no overfitting, is that supported by the data)? (-5 points if not)
  • Was the report exceptionally well done? (+1%)

Required, Allowed & Prohibited

Required:

  • Your code must implement a Random Tree learner.
  • Your project must be coded in Python 2.7.x.
  • Your code must run on one of the university-provided computers (e.g. buffet02.cc.gatech.edu).
  • Your code must run in less than 5 seconds on one of the university-provided computers.
  • The code you submit should NOT include any data reading routines. The provided testlearner.py code reads data for you.
  • The code you submit should NOT generate any output: No prints, no charts, etc.

Allowed:

  • You can develop your code on your personal machine, but it must also run successfully on one of the university provided machines or virtual images.
  • Your code may use standard Python libraries.
  • You may use the NumPy, SciPy, matplotlib and Pandas libraries. Be sure you are using the correct versions.
  • You may reuse sections of code (up to 5 lines) that you collected from other students or the internet.
  • Code provided by the instructor, or allowed by the instructor to be shared.
  • Cheese.

Prohibited:

  • Any other method of reading data besides testlearner.py
  • Any libraries not listed in the "allowed" section above.
  • Any code you did not write yourself (except for the 5 line rule in the "allowed" section).
  • Any Classes (other than Random) that create their own instance variables for later use (e.g., learners like kdtree).
  • Code that includes any data reading routines. The provided testlearner.py code reads data for you.
  • Code that generates any output when verbose = False: No prints, no charts, etc.

Acknowledgements and Citations

The data used in this assignment was provided by UCI's ML Datasets.

Legacy

MC3-Project-1-legacy