MC3-Project-3

Updates / FAQs

• 2017-04-02
  • Clarified instructions regarding "best possible" to use your own market simulator with adjusted closing prices.

• 2017-03-16
  • Switch from IBM to AAPL. Position sizes changed. In sample and out of sample dates changed.
  • Added requirement for "best possible strategy".
  • Added requirement that indicators be standardized.
  • Changed from 10 day to 21 day holding. Chart requirements relaxed to just require a vertical line upon entry (no black vertical line on exit).
  • Added requirement for data visualization.

• Q: In a previous project there was a constraint of holding a single position until exit. Does that apply to this project? Yes, hold one position til exit.

• Q: Is that 21 calendar days, or 21 trading days (i.e., days when SPY was traded)? A: Always use trading days.

• Q: Are there constraints for Python modules allowed for this project? Can we experiment with modules for optimization or technical analysis and cite or are we expected to write everything from scratch for this project as well? A: The constraints are the same as for the first learning project. You've already written the learners you need.

• Q: I want to read some other values from the data besides just adjusted close, how can I do that? A: Please modify an old version of util.py to do that, include that new util.py with your submission.

• Q: Are we required to trade in only 200 share blocks? (and have no more than 200 shares long or short at a time as in some of the previous assignments) A: (update). You can trade up to 400 shares at a time as long as you maintain the requirement of 200, 0 or -200 shares. This will enable comparison between results more easily.

• Q: Are we limited to leverage of 2.0 on the portfolio? A: There is no limit on leverage.

• Q: Are we only allowed one position at a time? A: You can be in one of three states: -200 shares, +200 shares, 0 shares.

Overview

In this project you will develop trading strategies using Technical Analysis, and test them using your market simulator. You will then utilize your Random Tree learner to train and test a learning trading algorithm.

In this project we shift from an auto graded format to a report format. For this project your grade will be based on the PDF report you submit, not your code. However, you will also submit your code that will be checked visually to ensure it appropriately matches the report you submit.

Data Details, Dates and Rules

Use the following parameters for Part 2, 3 and 4:

• Use only the data provided for this course. You are not allowed to import external data.
• Trade only the symbol AAPL (however, you may, if you like, use data from other symbols to inform your strategy).
• The in sample/training period is January 1, 2008 to December 31 2009.
• The out of sample/testing period is January 1, 2010 to December 31 2011.
• Starting cash is $100,000.
• Allowable positions are: 200 shares long, 200 shares short, 0 shares.
• Benchmark: The performance of a portfolio starting with $100,000 cash, investing in 200 shares of AAPL and holding that position
• There is no limit on leverage.

Part 1: Technical Indicators (20%)

Develop and describe at least 3 and at most 5 technical indicators. You may find our lecture on time series processing to be helpful. For each indicator you should create a single chart that shows the price history of the stock during the in-sample period, "helper data" and the value of the indicator itself. As an example, if you were using price/SMA as an indicator you would want to create a chart with 3 lines: Price, SMA, Price/SMA. In order to facilitate visualization of the indicator you can normalize the data to 1.0 at the start of the date range (i.e. divide price[t] by price[0]).

You should "standardize" or "normalize" your indicators so that they have zero mean and standard deviation 1.0 One way to do this is the standard score transformation as described here: https://en.wikipedia.org/wiki/Standard_score . This transformation will help ensure that all of your indicators are considered with equal importance by your learner.

Your report description of each indicator should enable someone to reproduce it just by reading the description. We want a written description here, not code, however, it is OK to augment your written description with a pseudocode figure.

At least one of the indicators you use should be completely different from the ones presented in our lectures. (i.e. something other than SMA, Bollinger Bands, RSI)

Deliverables:

• Descriptive text (2 to 3 pages with figures).
• 3 to 5 charts (one for each indicator)
• Code: indicators.py

Part 2: Best Possible Strategy (5%)

Assume that you can see the future, but that you are constrained by the portfolio size and order limits as specified above. Create a set of trades that represents the best a strategy could possibly do during the in sample period. The holding time requirements described in the next sections do not apply to this exercise. The reason we're having you do this is so that you will have an idea of an upper bound on performance.

The intent is for you to use adjusted close prices with the market simulator that you wrote earlier in the course.

Provide a chart that reports:

• Benchmark (see definition above) normalized to 1.0 at the start: Black line
• Value of the best possible portfolio (normalized to 1.0 at the start): Blue line

You should also report in text:

• Cumulative return of the benchmark and portfolio
• Stdev of daily returns of benchmark and portfolio
• Mean of daily returns of benchmark and portfolio

Part 3: Manual Rule-Based Trader (20%)

Devise a set of rules using the indicators you created in Part 1 above. Your rules should be designed to trigger a "long" or "short" entry for a 21 trading day hold. In other words, once an entry is initiated, you must remain in the position for 21 trading days. In your report you must describe your trading rules so that another person could implement them based only on your description. We want a written description here, not code, however, it is OK to augment your written description with a pseudocode figure.

You should tweak your rules as best you can to get the best performance possible during the in sample period (do not peek at out of sample performance). Use your rule-based strategy to generate an orders file over the in sample period, then run that file through your market simulator to create a chart that includes the following components over the in sample period:

• Benchmark (see definition above) normalized to 1.0 at the start: Black line
• Value of the rule-based portfolio (normalized to 1.0 at the start): Blue line
• Vertical green lines indicating LONG entry points.
• Vertical red lines indicating SHORT entry points.

Note that each red or green vertical line should be at least 21 days from the preceding line. We will check for that. We expect that your rule-based strategy should outperform the benchmark over the in sample period.

Deliverables:

• Descriptive text (1 or 2 pages with chart) that provides a compelling justification for the rule-based system developed.
• Text must describe rule based system in sufficient detail that another person could implement it.
• 1 chart.
• Code: rule_based.py (generates an orders file)

Part 4: ML Trader (30%)

Convert your decision tree regression learner into a classification learner. The classifications should be:

• +1: LONG
• 0: DO NOTHING
• -1: SHORT

The X data for each sample (day) are simply the values of your indicators for the stock -- you should have 3 to 5 of them. The Y data (or classifications) will be based on 21 day return. You should classify the example as a +1 or "LONG" if the 21 day return exceeds a certain value, let's call it YBUY for the moment. You should classify the example as a -1 or "SHORT" if the 21 day return is below a certain value we'll call YSELL. In all other cases the sample should be classified as a 0 or "DO NOTHING." Note that it is very important that you train your learner with these classification values (not the 21 day returns). We will check for this.

Note that your X values are calculated each day from the current day's (and earlier) data, but the Y value (classification) is calculated using data from the future. You may tweak various parameters of your learner to maximize return (more on that below). Train and test your learning strategy over the in sample period. Whenever a LONG or SHORT is encountered, you must enter the corresponding position and hold it for 21 days. That means, for instance, that if you encounter a LONG on day 1, then a SHORT on day 2, you must keep the stock still until the 21 days expire, even though you received this conflicting information. The reason for this is that we're trying to provide a way to directly compare the manual strategy versus the ML strategy.

Important note: You must set the leaf_size parameter of your decision tree learner to 5 or larger. This requirement is intended to avoid a degenerate overfit solution to this problem.

Use your ML-based strategy to generate an orders file over the in sample period, then run that file through your market simulator to create a chart that includes the following components over the in sample period:

• Benchmark (see definition above) normalized to 1.0 at the start: Black line
• Value of the rule-based portfolio (normalized to 1.0 at the start): Blue line.
• Value of the ML-based portfolio (normalized to 1.0 at the start): Green line.
• Vertical green lines indicating LONG entry points.
• Vertical red lines indicating SHORT entry points.

We expect that the ML-based strategy will outperform the manual strategy, however it is possible that it does not. If it is the case that your manual strategy does better, you should try to explain why in your report.

You should tweak the parameters of your learner to maximize performance during the in sample period. Here is a partial list of things you can tweak:

• Adjust YSELL and YBUY.
• Adjust leaf_size.
• Utilize bagging and adjust the number of bags.

Deliverables:

• Descriptive text (1 or 2 pages with chart) that describes your ML approach.
• Text must describe ML based system in sufficient detail that another person could implement it.
• 1 chart
• Code: ML_based.py (generates an orders file)
• Additional code files as necessary to support ML_based.py (e.g. RTLearner.py and so on).

Part 5: Visualization of data (15%)

Choose two of your indicators, call them X1 and X2. Create 3 scatter plots where each point in each plot is located according to the indicator values on that day at X1, X2. Color each dot according to the following scheme:

• Green if the factors on that day satisfy "LONG" conditions.
• Red if the factors satisfy "SHORT" conditions.
• Black if neither "LONG" or "SHORT" are satisfied.

The scale for the scatter plot should be set to +-1.5 in both dimensions. This will help us check that you have standardized your indicators.

The 3 plots should be based on the in sample period (about 500 points):

1. Your rule-based strategy.
2. The training data for your ML strategy.
3. Response of your learner when queried with the same data (after training).

Part 6: Comparative Analysis (10%)

Evaluate the performance of both of your strategies in the out of sample period. Note that you should not train or tweak your learner on this data. You should use the classification learned using the training data only. Create a chart that shows, out of sample:

• Benchmark (see definition above) normalized to 1.0 at the start: Black line
• Performance of manual strategy: Blue line
• Performance of the ML strategy: Green line
• All three should be normalized to 1.0 at the start.

Create a table that summarizes the performance of the stock, the manual strategy and the ML strategy for both in sample and out of sample periods. Utilize your experience in this class to determine which factors are best to use for comparing these strategies. If performance out of sample is worse than in sample, do your best to explain why. Also if the manual and ML strategies perform substantially differently, explain why. Is one method or the other more or less susceptible to the same underlying flaw? Why or why not?

Deliverables:

• Descriptive text (1 or 2 pages including figures)
• 1 chart

Hints

Overall, I recommend the following steps in the creation of your strategies:

• Indicator design hints:
  • For your X values: Identify and implement at least 3 technical features that you believe may be predictive of future return.
• Rule based design:
  • Use a cascade of if statements conditioned on the indicators to identify whether a LONG condition is met.
  • Use a cascade of if statements conditioned on the indicators to identify whether a SHORT condition is met.
  • The conditions for LONG and SHORT should be mutually exclusive.
  • If neither LONG or SHORT is triggered, the result should be DO NOTHING.
  • For debugging purposes, you may find it helpful to plot the value of the rule-based output (-1, 0, 1) versus the stock price.
• Train a classification learner on in sample training data:
  • For your Y values: Use future 21 day return (not future price). Then classify that return as LONG, SHORT or DO NOTHING. You're trying to predict a relative change that you can use to invest with.
  • For debugging purposes, you may find it helpful to plot the value of the training classification data (-1, 0, 1) versus the stock price in one color.
  • For debugging purposes, you may find it helpful to plot the value of the training classification output (-1, 0, 1) versus the stock price in another color. Ideally, these two lines should be very similar.

Choosing Technical Features -- Your X Values

You should have already successfully coded the Bollinger Band feature:

bb_value[t] = (price[t] - SMA[t])/(stdev[t])

Two other good features worth considering are momentum and volatility.

momentum[t] = (price[t]/price[t-N]) - 1

Volatility is just the stdev of daily returns.

You still need to standardize the resulting values.

Choosing Y

Your code should classify based on 21 day change in price. You need to build a new Y that reflects the 21 day change and aligns with the current date. Here's pseudo code for the calculation of Y

ret = (price[t+21]/price[t]) - 1.0
if ret > YBUY:
    Y[t] = +1 # LONG
else if ret < YSELL:
    Y[t] = -1 # SHORT
else:
    Y[t] = 0

If you select Y in this manner and use it for training, your learner will classify 21 day returns.

Template and Data

There is no github template for this project. You should create a directory for your code in ml4t/mc3-p3 and make a copy of util.py there. You should also copy into that directory your learner code and your market simulator code. You will have access to the data in the ML4T/Data directory but you should use ONLY the code in util.py to read it.

Contents of Report

• Your report should be no more than 3000 words. Your report should contain no more than 14 charts. Penalties will apply if you violate these constraints.
• Include charts and text as identified in the sections above.

Expectations

• In-sample AAPL backtests should perform very well -- The ML version should do better than the manual version.
• Out-of-sample AAPL backtests should... (you should be able to complete this sentence).

What to turn in

Turn your project in via t-square.

• Your report as report.pdf
• All of your code, as necessary to run as .py files.
• Document how to run your code in readme.txt.
• No zip files please.

Rubric

Start with 100%, deductions as follows:

Indicators (up to 20% potential deductions):

• Is each indicator described in sufficient detail that someone else could reproduce it? (-5% for each if not)
• Is there a chart for each indicator that properly illustrates its operation? (-5% for each if not)
• Is at least one indicator different from those provided by the instructor's code (i.e., another indicator that is not SMA, Bollinger Bands or RSI) (-10% if not)
• Does the submitted code indicators.py properly reflect the indicators provided in the report (-20% if not)

Best possible (up to 5% potential deductions):

• Is the chart correct (dates and equity curve) (-5% for if not)
• Is the reported performance correct within 5% (-1% for each item if not)

Manual rule-based trader (up to 20% deductions):

• Is the trading strategy described with clarity and in sufficient detail that someone else could reproduce it? (-10%)
• Does the provided chart include:
  • Historic value of benchmark normalized to 1.0 with black line (-5% if not)
  • Historic value of portfolio normalized to 1.0 with blue line (-10% if not)
  • Are the appropriate date ranges covered? (-5% if not)
  • Are vertical lines included to indicate entries (-10% if not)
• Does the submitted code rule_based.py properly reflect the strategy provided in the report? (-20% if not)
• Does the manual trading system provide higher cumulative return than the benchmark over the in-sample time period? (-5% if not)

ML-based trader (up to 30% deductions):

• Is the ML strategy described with clarity and in sufficient detail that someone else could reproduce it? (-10%)
• Are modifications/tweaks to the basic decision tree learner fully described (-10%)
• Does the methodology utilize a classification-based learner? (-30%)
• Does the provided chart include:
  • Historic value of benchmark normalized to 1.0 with black line (-5% if not)
  • Historic value of rule-based portfolio normalized to 1.0 with blue line (-5% if not)
  • Historic value of ML-based portfolio normalized to 1.0 with green line (-10% if not)
  • Are the appropriate date ranges covered? (-5% if not)
  • Are vertical lines included to indicate entry (-10% if not)
• Does the submitted code ML_based.py properly reflect the strategy provided in the report? (-30% if not)
• Does the ML trading system provide 1.5x higher cumulative return or than the benchmark over the in-sample time period? (-5% if not)

Data visualization (up to 15% deductions):

• Is the X data reported in all three charts the same? (-5% if not)
• Is the X data standardized? (-5% if not)
• Is the Y data in the train and query plots similar (-5% if not)

Comparative analysis (up to 10% deductions):

• Is the appropriate chart provided (-5% for each missing element, up to a maximum of -10%)
• Is there a table that reports in-sample and out-of-sample data for the baseline (just the stock), rule-based, and ML-based strategies? (-5% for each missing element)
• Are differences between the in-sample and out-of-sample performances appropriately explained (-5%)

Required, Allowed & Prohibited

Required:

• Your project must be coded in Python 2.7.x.
• Your code must run on one of the university-provided computers (e.g. buffet02.cc.gatech.edu).
• Use only util.py to read data. If you want to read items other than adjusted close, modify util.py to do it, and submit your new version with your code.

Allowed:

• You can develop your code on your personal machine, but it must also run successfully on one of the university provided machines or virtual images.
• Your code may use standard Python libraries.
• You may use the NumPy, SciPy, matplotlib and Pandas libraries. Be sure you are using the correct versions.
• You may reuse sections of code (up to 5 lines) that you collected from other students or the internet.
• Code provided by the instructor, or allowed by the instructor to be shared.
• A herring.

Prohibited:

• Any other method of reading data besides util.py
• Any libraries not listed in the "allowed" section above.
• Any code you did not write yourself (except for the 5 line rule in the "allowed" section).

Legacy

• MC3-Project-2-Legacy-trader
• MC3-Project-2-Legacy
• MC3-Project-3-Legacy-Q
• MC3-Project-3-Legacy