Strategy learner

Updated November 27, 2018

• It is OK to read data "before" the start date in order to ramp up indicators before start of trading date. It is OK also NOT to do that, but in this case you can't start trading until after the indicators have enough lookback to work.
• You should use the same indicators and indicator meta parameters as you used in manual strategy. One exception: If your learning method is setting the values of the meta parameters of the indicators, this is OK.
• You should include at least 3 indicators in your learning strategy. *Caveat, if you used less than 3 indicators for manual strategy, please use the same for the machine learning strategy.
• The author() method is required in each python file.
• Plots are required to have proper plot components: labels on each axis, legend (if more than 1 variable), and title. Points will deducted if they do not.
• All plotting code and files necessary to produce the plots must be submitted.
• Charts must be provided for all results. Tables are OK when making a comparison between specific result quantities but they must be backed up by charts.

Updated July 25, 2018 to remove relation to manual strategy. Text that was removed from the earlier version of the assignment is indicated in strike through.

Updated July 26, 2018 additional requirements added relating to plotting and reporting on indicators. 9 Figures allowed.

Updated November 14 2018 updated number of charts and reference to manual strategy indicators.

Overview

In this project you will design a learning trading agent. You must draw on the learners you have created so far in the course. Your choices are:

1. Regression or classification-based learner: Create a strategy using your Random Forest learner. Suggestions if you follow this approach: Classification_Trader_Hints. Important note, if you choose this method, you must set the leaf_size for your learner to 5 or greater. This is to avoid degenerate overfitting in-sample.
2. Reinforcement Learner-based approach: Create a Q-learning-based strategy using your Q-Learner. Read the Classification_Trader_Hints first, because many of the ideas there are relevant for the Q trader, then see Q_Trader_Hints
3. Optimization-based learner: Create a scan-based strategy using an optimizer. Read the Classification_Trader_Hints first, because many of the ideas there are relevant for the Opto trader, then see Opto_Trader_Hints

Regardless of your choice above, your learner should work in the following way:

• In the training phase (e.g., addEvidence()) your learner will be provided with a stock symbol and a time period. It should use this data to learn a strategy. For instance, for a regression-based learner it will use this data to make predictions about future price changes.
• In the testing phase (e.g., testPolicy()) your learner will be provided a symbol and a date range. All learning should be turned OFF during this phase.
• You should use exactly the same indicators as in the manual strategy project so we can compare your results.

If the date range is the same as used for the training, it is an in-sample test. Otherwise it is an out-of-sample test. Your learner should return a trades dataframe like it did in the last project. Here are some important requirements: Your testPolicy() method should be much faster than your addEvidence() method. The timeout requirements (see rubric) will be set accordingly. Multiple calls to your testPolicy() method should return exactly the same result.

Overall, your tasks for this project include:

• Devise numerical/technical indicators to evaluate the state of a stock on each day.
• Build a strategy learner based on one of the learners described above that uses the indicators.
• Test/debug the strategy learner on specific symbol/time period problems.
• Write a report describing your learning strategy.

Scoring for the project will be based on trading strategy test cases and a report.

Template

Instructions:

• Download and install the files from this zip file File:18fall strategy learner.zip
• Place your existing Q-Learner or RTLearner or OptimizationLearner into the strategy_learner/ directory.
• Implement the StrategyLearner class in strategy_learner/StrategyLearner.py
• See "what to turn in" below for a list of files that should be submitted.
  • Your learning code: QLearner.py, RTLearner.py, BagLearner.py, and/or OptimizeLearner.py
• To test your strategy learner, follow the instructions on Running the grading scripts

Data Details, Dates and Rules

• Use only the data provided for this course. You are not allowed to import external data.
• For your report, trade only the symbol JPM. This will enable us to more easily compare results. We will test your learner with other symbols as well.
• You may use data from other symbols (such as SPY) to inform your strategy.
• The in sample/development period is January 1, 2008 to December 31 2009.
• The out of sample/testing period is January 1, 2010 to December 31 2011.
• Starting cash is $100,000.
• Allowable positions are: 1000 shares long, 1000 shares short, 0 shares.
• Benchmark: The performance of a portfolio starting with $100,000 cash, investing in 1000 shares of the symbol in use and holding that position. Include transaction costs.
• There is no limit on leverage.
• Transaction costs: Commission will always be $0.00, Impact may vary, and will be passed in as a parameter to the learner.
• Minimize use of herrings.

Implement Strategy Learner

For this part of the project you should develop a learner that can learn a trading policy using your learner. You should be able to use your Q-Learner or RTLearner from the earlier project directly, with no changes. If you want to use the optimization approach, you will need to create new code or that. You will need to write code in StrategyLearner.py to "wrap" your learner appropriately to frame the trading problem for it. Utilize the template provided in StrategyLearner.py.

Your StrategyLearner should implement the following API:

import StrategyLearner as sl
learner = sl.StrategyLearner(verbose = False, impact = 0.000) # constructor
learner.addEvidence(symbol = "AAPL", sd=dt.datetime(2008,1,1), ed=dt.datetime(2009,12,31), sv = 100000) # training phase
df_trades = learner.testPolicy(symbol = "AAPL", sd=dt.datetime(2010,1,1), ed=dt.datetime(2011,12,31), sv = 100000) # testing phase

The input parameters are:

• verbose: if False do not generate any output
• impact: The market impact of each transaction.
• symbol: the stock symbol to train on
• sd: A datetime object that represents the start date
• ed: A datetime object that represents the end date
• sv: Start value of the portfolio

The output result is:

• df_trades: A data frame whose values represent trades for each day. Legal values are +1000.0 indicating a BUY of 1000 shares, -1000.0 indicating a SELL of 1000 shares, and 0.0 indicating NOTHING. Values of +2000 and -2000 for trades are also legal when switching from long to short or short to long so long as net holdings are constrained to -1000, 0, and 1000.

Contents of Report

Write a report describing your system. The centerpiece of your report should be the description of how you utilized your learner to determine trades:

• Describe the steps you took to frame the trading problem as a learning problem for your learner. What are your indicators? Did you adjust the data in any way (dicretization, standardization)? Why or why not?

* NEW Describe at least 3 and at most 5 technical indicators that you used. You may find our lecture on time series processing to be helpful. For each indicator you should create a single, compelling chart that illustrates the indicator. As an example, you might create a chart that shows the price history of the stock, along with "helper data" (such as upper and lower bollinger bands) and the value of the indicator itself. Another example: If you were using price/SMA as an indicator you would want to create a chart with 3 lines: Price, SMA, Price/SMA. In order to facilitate visualization of the indicator you might normalize the data to 1.0 at the start of the date range (i.e. divide price[t] by price[0]). Your report description of each indicator should enable someone to reproduce it just by reading the description. We want a written description here, not code, however, it is OK to augment your written description with a pseudocode figure. At least one of the indicators you use should be completely different from the ones presented in our lectures. (i.e. something other than SMA, Bollinger Bands, RSI).

• NOTE (1) You are allowed to change your indicators from manual strategy assignment but only with the caveat that the same indicators must be used for both manual and machine learning strategies. So, yes, that means you will have to update your manual strategy and will have to describe the new indicators as well (see the strike out section above) . Remember, we want a fair comparison between the two for Experiment 1. (2) Employ the same indicators you used for manual strategy, even if you used less than 3 indicators.

*Experiment 1: Using the benchmark described above, plot the performance of the benchmark versus your strategy in sample. Trade only the symbol JPM for this evaluation. The code that implements this experiment and generates the relevant charts and data should be submitted as experiment1.py

• Experiment 1: Using exactly the same indicators that you used in manual_strategy, compare your manual strategy with your learning strategy in sample.
  • Describe your experiment in detail: Assumptions, parameter values and so on.
  • Describe the outcome of your experiment.
  • Would you expect this relative result every time with in-sample data? Explain why or why not.
• Experiment 2: Provide an hypothesis regarding how changing the value of impact should affect in sample trading behavior and results (provide at least two metrics). Conduct an experiment with JPM on the in sample period to test that hypothesis. Provide charts, graphs or tables that illustrate the results of your experiment. The code that implements this experiment and generates the relevant charts and data should be submitted as experiment2.py

In situations other than Experiment 1, you may choose to use other indicators than the ones you used in manual strategy. If this is the case, be sure to indicate that in your report. Your descriptions should be stated clearly enough that an informed reader could reproduce the results you report.

The report can be up to 2500 words long and contain up to NEW 9 6 figures (charts and/or tables).

What to turn in

Turn your project in via canvas. All submitted code must include comments with your name and User ID.

• Your learner.
• Your StrategyLearner as StrategyLearner.py
• Your report as report.pdf
• ALL of your code should be contained in the files:
  • indicators.py Your code that implements your indicators.
  • marketsimcode.py Optional if needed to support your strategy learner. An improved version of your marketsim code that accepts a "trades" data frame (instead of a file).
  • StrategyLearner.py
  • ManualStrategy.py
  • experiment1.py
  • experiment2.py
  • Your learning code: QLearner.py, or RTLearner and BagLearner.py or OptimizeLearner.py.
• Make sure that you submit all code that is necessary for your software to run including plotting. If your code crashes because of a missing file, you may lose signficant points on the code section.
• Do not submit any other files. Penalties will be assessed for additional files outside of the allowed ones as stated above.

Rubric

Code: 70 points

We will test StrategyLearner in the following situations:

• Training / in sample: January 1, 2008 to December 31 2009.
• Testing / out of sample: January 1, 2010 to December 31 2011.
• Symbols: ML4T-220, AAPL, UNH, SINE_FAST_NOISE
• Starting value: $100,000
• Benchmark: Buy 1000 shares on the first trading day, Sell 1000 shares on the last day.
• Commissions = $0.00, impact = 0.00

We expect the following outcomes in evaluating your system:

• For ML4T-220
  • addEvidence() completes without crashing within 25 seconds: 1 points
  • testPolicy() completes in-sample within 5 seconds: 2 points
  • testPolicy() returns same result when called in-sample twice: 2 points
  • testPolicy() returns an in-sample result with cumulative return greater than 100%: 5 points
  • testPolicy() returns an out-of-sample result with cumulative return greater than 100%: 5 points
• For AAPL
  • addEvidence() completes without crashing within 25 seconds: 1 points
  • testPolicy() completes in-sample within 5 seconds: 2 points
  • testPolicy() returns same result when called in-sample twice: 2 points
  • testPolicy() returns an in-sample result with cumulative return greater than benchmark: 5 points
  • testPolicy() returns an out-of-sample result within 5 seconds: 5 points
• For SINE_FAST_NOISE
  • addEvidence() completes without crashing within 25 seconds: 1 points
  • testPolicy() completes in-sample within 5 seconds: 2 points
  • testPolicy() returns same result when called in-sample twice: 2 points
  • testPolicy() returns an in-sample result with cumulative return greater than 200%: 5 points
  • testPolicy() returns an out-of-sample result within 5 seconds: 5 points
• For UNH
  • addEvidence() completes without crashing within 25 seconds: 1 points
  • testPolicy() completes in-sample within 5 seconds: 2 points
  • testPolicy() returns same result when called in-sample twice: 2 points
  • testPolicy() returns an in-sample result with cumulative return greater than benchmark: 5 points
  • testPolicy() returns an out-of-sample result within 5 seconds: 5 points
• Witheld test case 1: In sample test case for an unknown symbol.
  • If any part of code crashes: 0 points awarded.
  • testPolicy() returns an in-sample result with cumulative return greater than benchmark: 5 points
• Withheld test case 2: In sample test case to verify that strategy accounts for different values of impact
  • If any part of code crashes: 0 points awarded.
  • Learner returns different trades when impact value is significantly different: 5 points
• Is any submitted code lacking a comment with the student's name and User ID? -5 points for each occurance, up to a maximum of -20
• Does the submitted code and report reflect an understanding of the subject matter? (up to -50 if not)

We reserve the right to use different time periods if necessary to reduce auto grading time.

• IMPORTANT NOTES
  • For achieving the required cumulative return, recall that cr = (portval[-1]/portval[0]) - 1.0
  • The requirement that consecutive calls to testPolicy() produce the same output for the same input means that you cannot update, train, or tune your learner in this method. For example, a solution that uses Q-Learning should use querySetState() and not query() in testPolicy(). Updating, training, and tuning (query()) is fine inside addEvidence().
  • Your learner should not select different hyper-parameters based on the symbol. Hyper-parameters include (but are not limited to) things like features, discretization size, sub-learning methods (for ensemble learners). Tuning using cross-validation or otherwise pre-processing the data is OK, things like if symbol=="UNH" are not OK. There will be a withheld test case that checks your code on a valid symbol that is not one of the four listed above.
  • Presence of code like if symbol=="UNH" will result in a 20 point penalty.
  • When evaluating the trades generated by your learner, we will consider transaction costs (market impact).

Report: 30 points

NEW Indicators (up to 15 points potential deductions):

• Is there a compelling description why each indicator might work (-2 for each, up to a total of 6 off)
• Is each indicator described in sufficient detail that someone else could reproduce it? (-5 points for each if not)
• Is there a chart for each indicator that properly illustrates its operation? (-5 points for each if not)
• Is at least one indicator different from those provided by the instructor's code (i.e., another indicator that is not SMA, Bollinger Bands or RSI) (-10 points if not)

• Note, the above strike through points apply if you are using different indicators than what was originally used for manual strategy. Again, make sure you are using the same indicators for manual and machine learning strategies.
• Does the submitted code indicators.py properly reflect the indicators provided in the report (-20 points if not)

Learning:

• Is the method by which the learner is utilized to create a trading strategy described sufficiently clearly that an informed reader could reproduce the result? (up to 10 point deduction if not)
• Does report description match the code? (up to 10 point deduction if not)
• Are the two required experiments explained well? (up to 5 points deduction each if not)
• Are the two required experiments compellingly supported with tabular or graphical data? (up to 5 points deduction each if not)
• Does the report contain more than 2500 words? (10 point deduction if so)
• Does the report contain more than NEW 9 6 figures and/or tables? (10 point deduction if so)
• Does the submitted code and report reflect an understanding of the subject matter? (up to -20 if not)
• Is the report especially well written (up to 2 point bonus if so)
• Did the student choose to use optimization based learning, and did their learning strategy beat the manual strategy benchmark (+1 point bonus)

Required, Allowed & Prohibited

Required:

• Your project must be coded in Python 2.7.x.
• Your code must run on one of the university-provided computers (e.g. buffet02.cc.gatech.edu).
• All code must be your own.
• All charts must be created within Python and the code used to create them must be submitted.
• No external learning libraries allowed.

Allowed:

• You can develop your code on your personal machine, but it must also run successfully on one of the university provided machines or virtual images.
• Your code may use standard Python libraries.
• You may use the NumPy, SciPy, matplotlib and Pandas libraries. Be sure you are using the correct versions.
• Code provided by the instructor, or allowed by the instructor to be shared.
• Sardines.
• Use util.py (only) for reading data.

Prohibited:

• Any libraries not listed in the "allowed" section above.
• Any code you did not write yourself.
• Any Classes (other than Random) that create their own instance variables for later use (e.g., learners like kdtree).
• Print statements outside "verbose" checks (they significantly slow down auto grading).
• Any method for reading data besides util.py
• Excessive use of herrings.

Legacy

• MC3-Project-4-Legacy-Q-trader
• MC3-Project-2-Legacy-trader
• MC3-Project-2-Legacy
• 2018-Strategy-Learner-Legacy