Summer 2016 Project 5 - Revision history

Dave: /* Part 1: Implement QLearner */

2016-07-20T00:56:32Z

Part 1: Implement QLearner

Dave: /* Part 1: Implement QLearner */

2016-07-20T00:50:54Z

Part 1: Implement QLearner

Dave: /* Part 1: Implement QLearner */

2016-07-20T00:48:12Z

Part 1: Implement QLearner

Dave: /* Part 1: Implement QLearner */

2016-07-20T00:39:09Z

Part 1: Implement QLearner

Dave: /* Part 1: Implement QLearner */

2016-07-20T00:38:21Z

Part 1: Implement QLearner

Dave: /* Part 1: Implement QLearner */

2016-07-20T00:37:52Z

Part 1: Implement QLearner

Dave: /* Part 1: Implement QLearner */

2016-07-20T00:36:55Z

Part 1: Implement QLearner

Dave: /* Required, Allowed, & Prohibited */

2016-07-20T00:36:26Z

Required, Allowed, & Prohibited

Dave: /* Rubric */

2016-07-20T00:36:11Z

Rubric

Dave: /* What to turn in */

2016-07-20T00:35:58Z

What to turn in

← Older revision		Revision as of 00:56, 20 July 2016
Line 44:		Line 44:
	Here's an example of the API in use:		Here's an example of the API in use:

		+	<pre>
		+	import QLearner as ql
		+
		+	learner = ql.QLearner(num_states = 100, \
		+	num_actions = 4, \
		+	alpha = 0.2, \
		+	gamma = 0.9, \
		+	rar = 0.98, \
		+	radr = 0.999, \
		+	dyna = 0, \
		+	verbose = False)
		+
		+	s = 99 # our initial state
		+
		+	a = learner.querysetstate(s) # action for state s
		+
		+	s_prime = 5 # the new state we end up in after taking action a in state s
		+
		+	r = 0 # reward for taking action a in state s
		+
		+	next_action = learner.query(s_prime, r)
		+	</pre>

@@ Line 56: / Line 56: @@
 * <tt>dyna</tt> integer, conduct this number of dyna updates for each regular update.  When Dyna is used, 200 is a typical value.
 * <tt>verbose</tt> boolean, if True, your class is allowed to print debugging statements, if False, all printing is prohibited.
 ==Part 2: Navigation Problem Test Cases==

@@ Line 48: / Line 48: @@
 <b>The constructor QLearner()</b> should reserve space for keeping track of Q[s, a] for the number of states and actions.  It should initialize Q[] with uniform random values between -1.0 and 1.0.  Details on the input arguments to the constructor:
-* test
+* <tt>num_states</tt> integer, the number of states to consider
 ==Part 2: Navigation Problem Test Cases==

← Older revision		Revision as of 00:39, 20 July 2016
Line 47:		Line 47:

	<b>The constructor QLearner()</b> should reserve space for keeping track of Q[s, a] for the number of states and actions. It should initialize Q[] with uniform random values between -1.0 and 1.0. Details on the input arguments to the constructor:		<b>The constructor QLearner()</b> should reserve space for keeping track of Q[s, a] for the number of states and actions. It should initialize Q[] with uniform random values between -1.0 and 1.0. Details on the input arguments to the constructor:
		+
		+	* test

	==Part 2: Navigation Problem Test Cases==		==Part 2: Navigation Problem Test Cases==

@@ Line 44: / Line 44: @@
 Here's an example of the API in use:
-test
 ==Part 2: Navigation Problem Test Cases==

← Older revision		Revision as of 00:37, 20 July 2016
Line 43:		Line 43:

	Here's an example of the API in use:		Here's an example of the API in use:
		+
		+	test

	==Part 2: Navigation Problem Test Cases==		==Part 2: Navigation Problem Test Cases==

← Older revision		Revision as of 00:36, 20 July 2016
Line 34:		Line 34:

	==Part 1: Implement QLearner==		==Part 1: Implement QLearner==
		+
		+
		+	Your QLearner class should be implemented in the file <tt>QLearner.py</tt>. It should implement EXACTLY the API defined below. DO NOT import any modules besides those allowed below. Your class should implement the following methods:
		+
		+	* QLearner(...): Constructor, see argument details below.
		+	* query(s_prime, r): Update Q-table with <s, a, s_prime, r> and return new action for state s_prime, update rar.
		+	* querysetstate(s): Set state to s, return action for state s, but don't update Q-table or rar.
		+
		+	Here's an example of the API in use:

	==Part 2: Navigation Problem Test Cases==		==Part 2: Navigation Problem Test Cases==

← Older revision		Revision as of 00:36, 20 July 2016
Line 216:		Line 216:

	==Required, Allowed, & Prohibited==		==Required, Allowed, & Prohibited==
		+
		+
		+	Required:
		+	* Your project must be coded in Python 2.7.x.
		+	* Your code must run on one of the university-provided computers (e.g. buffet02.cc.gatech.edu), or on one of the provided virtual images.
		+
		+	Allowed:
		+	* You can develop your code on your personal machine, but it must also run successfully on one of the university provided machines or virtual images.
		+	* Your code may use standard Python libraries.
		+	* You may use the NumPy, SciPy, matplotlib and Pandas libraries. Be sure you are using the correct versions.
		+	* Code provided by the instructor, or allowed by the instructor to be shared.
		+	* Use util.py (only) for reading data.
		+
		+	Prohibited:
		+	* Any libraries not listed in the "allowed" section above.
		+	* Any code you did not write yourself
		+	* Any Classes (other than Random) that create their own instance variables for later use (e.g., learners like kdtree).
		+	* Print statements outside "verbose" checks (they significantly slow down auto grading).
		+	* Any method for reading data besides util.py

← Older revision		Revision as of 00:36, 20 July 2016
Line 173:		Line 173:

	==Rubric==		==Rubric==
		+
		+	Only your QLearner class will be tested.
		+
		+	* For basic Q-Learning (dyna = 0) we will test your learner against 10 test worlds with 500 iterations. Each test should complete in less than 2 seconds. For the test to be successful, your learner should find a path to the goal <= 1.5 x the number of steps our reference solution finds. We will check this by taking the min of all the 500 runs. Each test case is worth 8 points. We will initialize your learner with the following parameter values:
		+
		+	<Pre>
		+	learner = ql.QLearner(num_states=100,\
		+	num_actions = 4, \
		+	alpha = 0.2, \
		+	gamma = 0.9, \
		+	rar = 0.98, \
		+	radr = 0.999, \
		+	dyna = 0, \
		+	verbose=False) #initialize the learner
		+	</PRE>
		+
		+	* For Dyna-Q, we will set dyna = 200. We will test your learner against <tt>world03.csv</tt> with 50 iterations. The test should complete in less than 10 seconds. For the test to be successful, your learner should find a path to the goal <= 1.5 x the number of steps our reference solution finds. We will check this by taking the min of all 50 runs. The test case is worth 5 points. We will initialize your learner with the following parameter values:
		+
		+	<Pre>
		+	learner = ql.QLearner(num_states=100,\
		+	num_actions = 4, \
		+	alpha = 0.2, \
		+	gamma = 0.9, \
		+	rar = 0.5, \
		+	radr = 0.99, \
		+	dyna = 200, \
		+	verbose=False) #initialize the learner
		+	</PRE>
		+
		+	* We will test StrategyLearner in the following situations:
		+	** Training: Dec 31 2007 to Dec 31 2009
		+	** Testing: Dec 31 2009 to Dec 31 2011
		+	** Symbols: ML4T-220, IBM
		+	** Starting value: $10,000
		+	** Benchmark: Buy 100 shares on the first trading day, Sell 100 shares on the last day.
		+	* We expect the following outcomes in testing:
		+	** For ML4T-220, the trained policy should significantly outperform the benchmark in sample (7 points)
		+	** For ML4T-220, the trained policy should significantly outperform the benchmark out of sample (7 points)
		+	** For IBM, the trained policy should significantly outperform the benchmark in sample (7 points)
		+
		+	Training and testing for each situation should run in less than 30 seconds. We reserve the right to use different time periods if necessary to reduce auto grading time.

	==Required, Allowed, & Prohibited==		==Required, Allowed, & Prohibited==

@@ Line 163: / Line 163: @@
 ==What to turn in==
 ==Rubric==
 ==Required, Allowed, & Prohibited==