MC3-Project-3

Updates / FAQs

• 2016-10-31 Project finalized.

• Q: In a previous project there was a constraint of holding a single position until exit. Does that apply to this project? Yes, hold one position til exit.

• Q: Is that 10 calendar days, or 10 trading days (i.e., days when SPY was traded)? A: Always use trading days.

• Q: Are there constraints for Python modules allowed for this project? Can we experiment with modules for optimization or technical analysis and cite or are we expected to write everything from scratch for this project as well? A: The constraints are the same as for the first learning project. You've already written the learners you need.

• Q: I want to read some other values from the data besides just adjusted close, how can I do that? A: Please modify an old version of util.py to do that, include that new util.py with your submission.

• Q: Are we required to trade in only 500 share blocks? (and have no more than 500 shares long or short at a time as in some of the previous assignments) A: Yes. This will enable comparison between results more easily.

• Q: Are we limited to leverage of 2.0 on the portfolio? A: There is no limit on leverage.

• Q: Are we only allowed one position at a time? A: You can be in one of three states: -500 shares, +500 shares, 0 shares.

Overview

In this project you will develop trading strategies using Technical Analysis, and test them using your market simulator. You will then utilize your Random Tree learner to train and test a learning trading algorithm.

• Part 1: Develop and describe a set of at least 3 technical indicators. At least one of these indicators must be substantially different from the indicators whose code was presented in class.
• Part 2: Devise and test a rule-based trading strategy using your indicators from Part 1. Test its performance in sample using your market simulator.
• Part 3: Use you decision tree learner to create a classifier that decides when to trade. Test its performance in sample using your market simulator.
• Part 4: Comparative analysis.

In this project we shift from an auto graded format to a report format. For this project your grade will be based on the PDF report you submit, not your code. However, you will also submit your code that will be checked visually to ensure it appropriately matches the report you submit.

Data Details, Dates and Rules

Use the following parameters for Part 2, 3 and 4:

• Use only the data provided for this course. You are not allowed to import external data.
• Trade only the symbol IBM (however, you may, if you like, use data from other symbols to inform your strategy).
• The in sample/training period is January 1, 2006 to December 31 2009.
• The out of sample/testing period is January 1, 2010 to December 31 2010.
• Starting cash is $100,000.
• Allowable positions are: 500 shares long, 500 shares short, 0 shares.
• There is no limit on leverage.

Part 1: Technical Indicators (20%)

Develop and describe at least 3 and at most 5 technical indicators. You may find our lecture on time series processing to be helpful. For each indicator you should create a single chart that shows the price history of the stock during the in-sample period, "helper data" and the value of the indicator itself. As an example, if you were using price/SMA as an indicator you would want to create a chart with 3 lines: Price, SMA, Price/SMA

Your report description of each indicator should enable someone to reproduce it just by reading the description. We want a written description here, not code, however, it is OK to augment your written description with a pseudocode figure.

At least one of the indicators you use should be completely different from the ones presented in our lectures. (i.e. something other than SMA, Bollinger Bands, RSI)

Deliverables:

• Descriptive text (2 to 3 pages with figures).
• 3 to 5 charts (one for each indicator)
• Code: indicators.py

Part 2: Manual Rule-Based Trader (30%)

Devise a set of rules using the indicators you created in Part 1 above. Your rules should be designed to trigger a "long" or "short" entry for a 10 trading day hold. In other words, once an entry is initiated, you must remain in the position for 10 trading days. In your report you must describe your trading rules so that another person could implement them based only on your description. We want a written description here, not code, however, it is OK to augment your written description with a pseudocode figure.

You should tweak your rules as best you can to get the best performance possible from during the in sample period (do not peek at out of sample performance). Use your rule-based strategy to generate an orders file over the in sample period, then run that file through your market simulator to create a chart that includes the following components over the in sample period:

• Price of IBM (normalized to 1.0 at the start): Black line
• Value of the rule-based portfolio (normalized to 1.0 at the start): Blue line
• Vertical green lines indicating LONG entry points.
• Vertical red lines indicating SHORT entry points.
• Vertical black lines indicating exits (long or short).

Note that each red or green vertical line should be followed by a black line before another entry occurs. We will check for that. We expect that your rule-based strategy should outperform the stock IBM over the in sample period.

Deliverables:

• Descriptive text (1 or 2 pages with chart) that provides a compelling justification for rule-based system developed.
• Text must describe rule based system in sufficient detail that another person could implement it.
• 1 chart.
• Code: rule_based.py (generates an orders file)

Part 3: ML Trader (30%)

Convert your decision tree regression learner into a classification learner. The classifications should be:

• +1: BUY
• 0: DO NOTHING
• -1: SELL

The X data for each sample (day) are simply the values of your indicators for the stock -- you should have 3 to 5 of them. The Y data (or classifications) will be based on 10 day return. You should classify the example as a +1 or "BUY" if the 10 day return exceeds a certain value, let's call it YBUY for the moment. You should classify the example as a -1 or "SELL" if the 10 day return is below a certain value we'll call YSELL. In all other cases the sample should be classified as a 0 or "DO NOTHING."

Note that your X values are calculated each day from the current day's (and earlier) data, but the Y value is calculated using data from the future. You may tweak various parameters of your learner to maximize return (more on that below). Train and test your learning strategy over the in sample period. Whenever a BUY or SELL is encountered, you must enter the corresponding position and hold it for 10 days. That means, for instance, that if you encounter a BUY on day 1, then a SELL on day 2, you must keep the stock still until the 10 days expire, even though you received this conflicting information. The reason for this is that we're trying to provide a way to directly compare the manual strategy versus the ML strategy.

Use your ML-based strategy to generate an orders file over the in sample period, then run that file through your market simulator to create a chart that includes the following components over the in sample period:

• Price of IBM (normalized to 1.0 at the start): Black line.
• Value of the rule-based portfolio (normalized to 1.0 at the start): Blue line.
• Value of the ML-based portfolio (normalized to 1.0 at the start): Green line.
• Vertical green lines indicating LONG entry points.
• Vertical red lines indicating SHORT entry points.
• Vertical black lines indicating exits (long or short).

Note that each red or green vertical line should be followed by a black line before another entry occurs. We will check for that. We expect that the ML-based strategy will outperform the manual strategy, however it is possible that it does not. If it is the case that your manual strategy does better, you should try to explain why in your report.

You should tweak the parameters of your learner to maximize performance during the in sample period. Here is a partial list of things you can tweak:

• Adjust YSELL and YBUY.
• Adjust leaf_size.
• Utilize bagging and adjust the number of bags.

Deliverables:

• Descriptive text (1 or 2 pages with chart) that describes your ML approach.
• Text must describe ML based system in sufficient detail that another person could implement it.
• 1 chart
• Code: ML_based.py (generates an orders file)
• Additional code files as necessary to support ML_based.py (e.g. RTLearner.py and so on).

Part 4: Comparative Analysis (20%)

Evaluate the performance of both of your strategies in the out of sample period. Note that you should not train or tweak your learner on this data. You should use the classification learned using the training data only. Create a chart that shows, out of sample:

• Performance of the stock: Black line
• Performance of manual strategy: Blue line
• Performance of the ML strategy: Green line
• All three should be normalized to 1.0 at the start.

Create a table that summarizes the performance of the stock, the manual strategy and the ML strategy for both in sample and out of sample periods. Utilize your experience in this class to determine which factors are best to use for comparing these strategies. If performance out of sample is worse than in sample, do your best to explain why. Also if the manual and ML strategies perform substantially differently, explain why. Is one method or the other more or less susceptible to the same underlying flaw? Why or why not?

Deliverables:

• Descriptive text (1 or 2 pages including figures)
• 1 chart

Hints

Overall, I recommend the following steps in the creation of your strategies:

• Indicator design hints:
  • For your X values: Identify and implement at least 3 technical features that you believe may be predictive of future return.
• Rule based design:
  • Use a cascade of if statements conditioned on the indicators to identify whether a BUY condition is met.
  • Use a cascade of if statements conditioned on the indicators to identify whether a SELL condition is met.
  • The conditions for BUY and SELL should be mutually exclusive.
  • If neither BUY or SELL is triggered, the result should be DO NOTHING.
  • For debugging purposes, you may find it helpful to plot the value of the rule-based output (-1, 0, 1) versus the stock price.
• Train a classification learner on in sample training data:
  • For your Y values: Use future 10 day return (not future price). Then classify that return as BUY, SELL or DO NOTHING. You're trying to predict a relative change that you can use to invest with.
  • For debugging purposes, you may find it helpful to plot the value of the training classification data (-1, 0, 1) versus the stock price in one color.
  • For debugging purposes, you may find it helpful to plot the value of the training classification output (-1, 0, 1) versus the stock price in another color. Ideally, these two lines should be very similar.

Template and Data

There is no github template for this project. You should create a directory for your code in ml4t/mc3-p3 and make a copy of util.py there. You should also copy into that directory your learner code and your market simulator code. You will have access to the data in the ML4T/Data directory but you should use ONLY the code in util.py to read it.

Choosing Technical Features -- Your X Values

You should have already successfully coded the Bollinger Band feature. Here's a suggestion of how to normalize that feature so that it will typically provide values between -1.0 and 1.0:

bb_value[t] = (price[t] - SMA[t])/(2 * stdev[t])

Two other good features worth considering are momentum and volatility.

momentum[t] = (price[t]/price[t-N]) - 1

Volatility is just the stdev of daily returns.

Choosing Y

Your code should classify based on 10 day change in price. You need to build a new Y that reflects the 10 day change and aligns with the current date. Here's pseudo code for the calculation of Y

ret = (price[t+10]/price[t]) - 1.0
if ret > YBUY:
    Y[t] = +1 # BUY
else if ret < YSELL:
    Y[t] = -1 # SELL
else:
    Y[t] = 0

If you select Y in this manner and use it for training, your learner will classify 10 day returns.

Contents of Report

• Your report should be no more than 3000 words. Your report should contain no more than 12 charts. Penalties will apply if you violate these constraints.
• Include charts and text as identified in the sections above.

Expectations

• In-sample IBM backtests should perform very well -- The ML version should do better than the manual version.
• Out-of-sample IBM backtests should... (you should be able to complete this sentence).

What to turn in

Turn your project in via t-square.

• Your report as report.pdf
• All of your code, as necessary to run as .py files.
• Document how to run your code in readme.txt.
• No zip files please.

Rubric

Start with 100%, deductions as follows:

Indicators (up to 20% potential deductions):

• Is each indicator described in sufficient detail that someone else could reproduce it? (-5% for each if not)
• Is there a chart for each indicator that properly illustrates its operation? (-5% for each if not)
• Is at least one indicator different from those provided by the instructor's code (i.e., another indicator that is not SMA, Bollinger Bands or RSI) (-10% if not)
• Does the submitted code indicators.py properly reflect the indicators provided in the report (-20% if not)

Manual rule-based trader (up to 30% deductions):

• Is the trading strategy described with clarity and in sufficient detail that someone else could reproduce it? (-10%)
• Does the provided chart include:
  • Historic value of IBM normalized to 1.0 with black line (-5% if not)
  • Historic value of portfolio normalized to 1.0 with blue line (-10% if not)
  • Are the appropriate date ranges covered? (-5% if not)
  • Are vertical lines included to indicate entry, and exit (-10% if not)
• Does the submitted code rule_based.py properly reflect the strategy provided in the report? (-30% if not)
• Does the manual trading system provide higher cumulative return than IBM over the in-sample time period? (-5% if not)

ML-based trader (up to 30% deductions):

• Is the ML strategy described with clarity and in sufficient detail that someone else could reproduce it? (-10%)
• Are modifications/tweaks to the basic decision tree learner fully described (-10%)
• Does the methodology utilize a classification-based learner? (-10%)
• Does the provided chart include:
  • Historic value of IBM normalized to 1.0 with black line (-5% if not)
  • Historic value of rule-based portfolio normalized to 1.0 with blue line (-5% if not)
  • Historic value of ML-based portfolio normalized to 1.0 with green line (-10% if not)
  • Are the appropriate date ranges covered? (-5% if not)
  • Are vertical lines included to indicate entry, and exit (-10% if not)
• Does the submitted code ML_based.py properly reflect the strategy provided in the report? (-30% if not)
• Does the ML trading system provide 1.5x higher cumulative return or than IBM over the in-sample time period? (-5% if not)

Comparative analysis (up to 20% deductions):

• Is the appropriate chart provided (-5% for each missing element, up to a maximum of -10%)
• Is there a table that reports in-sample and out-of-sample data for the baseline (just the stock), rule-based, and ML-based strategies? (-5% for each missing element)
• Are differences between the in-sample and out-of-sample performances appropriately explained (-5%)

Required, Allowed & Prohibited

Required:

• Your project must be coded in Python 2.7.x.
• Your code must run on one of the university-provided computers (e.g. buffet02.cc.gatech.edu), or on one of the provided virtual images.
• Use only util.py to read data. If you want to read items other than adjusted close, modify util.py to do it, and submit your new version with your code.

Allowed:

• You can develop your code on your personal machine, but it must also run successfully on one of the university provided machines or virtual images.
• Your code may use standard Python libraries.
• You may use the NumPy, SciPy, matplotlib and Pandas libraries. Be sure you are using the correct versions.
• You may reuse sections of code (up to 5 lines) that you collected from other students or the internet.
• Code provided by the instructor, or allowed by the instructor to be shared.
• A herring.

Prohibited:

• Any other method of reading data besides util.py
• Any libraries not listed in the "allowed" section above.
• Any code you did not write yourself (except for the 5 line rule in the "allowed" section).

Legacy

• MC3-Project-2-Legacy-trader
• MC3-Project-2-Legacy
• MC3-Project-3-Legacy-Q
• MC3-Project-3-Legacy