MC3-Project-2 - Revision history

Dave: /* DRAFT */

2017-07-03T19:37:25Z

DRAFT

Bhrolenok3: /* Rubric */ Note about negative reward in rubric

2017-07-02T18:16:47Z

Rubric: Note about negative reward in rubric

Bhrolenok3: /* Rubric */ iterations->epochs, clarified language, noted epoch step limit

2017-07-02T18:06:14Z

Rubric: iterations->epochs, clarified language, noted epoch step limit

Bhrolenok3: /* Part 3: Implement Dyna (5%) */

2017-07-02T17:57:12Z

Part 3: Implement Dyna (5%)

Bhrolenok3: /* Part 2: Navigation Problem Test Cases */ iterations->epochs

2017-07-02T17:55:22Z

Part 2: Navigation Problem Test Cases: iterations->epochs

Bhrolenok3: /* Template and Data */ updated paths for Summer 2017

2017-07-02T17:50:53Z

Template and Data: updated paths for Summer 2017

Tucker: /* Part 1: Implement Q-Learner (95%) */

2017-06-13T17:28:51Z

Part 1: Implement Q-Learner (95%)

Tucker: /* Updates / FAQs */

2017-05-19T23:03:42Z

Updates / FAQs

Tucker: /* Required, Allowed & Prohibited */

2017-04-02T15:06:03Z

Required, Allowed & Prohibited

Tucker: /* Rubric */

2017-03-30T16:57:07Z

Rubric

← Older revision		Revision as of 19:37, 3 July 2017
Line 1:		Line 1:
−	~~==DRAFT==~~
−
−	~~This assignment is under revision. This notice will be removed once it is final.~~
−
	==Updates / FAQs==		==Updates / FAQs==

@@ Line 196: / Line 196: @@
 * Benchmark: As a benchmark to compare your solution to, we will run our reference solution in the same world, with 500 epochs.  We will take the median reward of our reference across all of those 500 epochs.
 * Your score: For each world we will take the median cost your solution finds across all 500 epochs.
-* For a test to be successful, your learner should find a total reward >= 1.5 x the benchmark.
+* For a test to be successful, your learner should find a total reward >= 1.5 x the benchmark. Note that since reward for a single epoch is negative, your solution can be up to 50% worse than the reference solution and still pass.
 * There are 10 test cases, each test case is worth 9.5 points.
 * Here is how we will initialize your QLearner for these test cases:
@@ Line 211: / Line 211: @@
 </PRE>
-* For Dyna-Q, we will set dyna = 200.  We will test your learner against <tt>world01.csv</tt> and <tt>world02.csv</tt> with 50 epochs.  Scoring is similar to the non-dyna case: Each test should complete in less than 10 seconds. For the test to be successful, your learner should find solution with total reward to the goal >= 1.5 x the  median reward our reference solution across all 50 epochs.  We will check this by taking the median of all 50 runs. Each test case is worth 2.5 points.  We will initialize your learner with the following parameter values for these test cases:
+* For Dyna-Q, we will set dyna = 200.  We will test your learner against <tt>world01.csv</tt> and <tt>world02.csv</tt> with 50 epochs.  Scoring is similar to the non-dyna case: Each test should complete in less than 10 seconds. For the test to be successful, your learner should find solution with total reward to the goal >= 1.5 x the  median reward our reference solution across all 50 epochs.  Note that since reward for a single epoch is negative, your solution can be up to 50% worse than the reference solution and still pass.  We will check this by taking the median of all 50 runs. Each test case is worth 2.5 points.  We will initialize your learner with the following parameter values for these test cases:
 <Pre>

@@ Line 193: / Line 193: @@
 Only your QLearner class will be tested.
-* For basic Q-Learning (dyna = 0) we will test your learner against 10 test worlds with 500 iterations in each world.  One "iteration" means your robot reaches the goal one time.  Your QLearner retains its state (Q-table), and then we allow it to navigate to the goal again, over and over, 500 times.  Each test (500 iterations) should complete in less than 2 seconds.
+* For basic Q-Learning (dyna = 0) we will test your learner against 10 test worlds with 500 epochs in each world.  One "epoch" means your robot reaches the goal one time, or after 100000 steps, whichever comes first.  Your QLearner retains its state (Q-table), and then we allow it to navigate to the goal again, over and over, 500 times.  Each test (500 epochs) should complete in less than 2 seconds.  <b>NOTE</b>: an epoch where the robot fails to reach the goal will likely take <b>much</b> longer (in running time), than one that does reach the goal, and is a common reason for failing to complete test cases within the time limit.
-* Benchmark: As a benchmark to compare your solution to, we will run our reference solution in the same world, with 500 iterations.  We will take the median reward of our reference across all of those 500 iterations.
+* Benchmark: As a benchmark to compare your solution to, we will run our reference solution in the same world, with 500 epochs.  We will take the median reward of our reference across all of those 500 epochs.
-* Your score: For each world we will take the median cost your solution finds across all 500 iterations.
+* Your score: For each world we will take the median cost your solution finds across all 500 epochs.
 * For a test to be successful, your learner should find a total reward >= 1.5 x the benchmark.
 * There are 10 test cases, each test case is worth 9.5 points.
 <Pre>
@@ Line 210: / Line 211: @@
 </PRE>
-* For Dyna-Q, we will set dyna = 200.  We will test your learner against <tt>world01.csv</tt> and <tt>world02.csv</tt> with 50 iterations.  Scoring is similar to the non-dyna case: Each test should complete in less than 10 seconds. For the test to be successful, your learner should find solution with total reward to the goal >= 1.5 x the  median reward our reference solution across all iterations.  We will check this by taking the median of all 50 runs. Each test case is worth 2.5 points.  We will initialize your learner with the following parameter values:
+* For Dyna-Q, we will set dyna = 200.  We will test your learner against <tt>world01.csv</tt> and <tt>world02.csv</tt> with 50 epochs.  Scoring is similar to the non-dyna case: Each test should complete in less than 10 seconds. For the test to be successful, your learner should find solution with total reward to the goal >= 1.5 x the  median reward our reference solution across all 50 epochs.  We will check this by taking the median of all 50 runs. Each test case is worth 2.5 points.  We will initialize your learner with the following parameter values for these test cases:
 <Pre>

@@ Line 143: / Line 143: @@
 ==Part 3: Implement Dyna (5%)==
-Add additional components to your QLearner class so that multiple "hallucinated" experience tuples are used to update the Q-table for each "real" experience.  The addition of this component should speed convergence in terms of the number of calls to query().
+Add additional components to your QLearner class so that multiple "hallucinated" experience tuples are used to update the Q-table for each "real" experience.  The addition of this component should speed convergence in terms of the number of calls to query(), <b>not necessarily running time</b>.
-Note that it is not important that you implement Dyna exactly as described in the lecture.  The key requirement is that your code should somehow hallucinate additional experiences.  The precise method you use for discovering those experiences is flexible.  We will test your code on several test worlds with 50 iterations and with dyna = 200.  Our expectation is that with Dyna, the solution should be much better after 50 iterations than without.
+Note that it is not important that you implement Dyna exactly as described in the lecture.  The key requirement is that your code should somehow hallucinate additional experiences.  The precise method you use for discovering those experiences is flexible.  We will test your code on several test worlds with 50 epochs and with dyna = 200.  Our expectation is that with Dyna, the solution should be much better after 50 epochs than without.
 ==Part 4: Implement author() Method (0%)==

@@ Line 106: / Line 106: @@
 * 1 if the robot moves to the goal space
-Overall, we will assess the performance of a policy as the average reward it incurs to travel from the start to the goal (higher reward is better).  We assess a learner in terms of the reward it converges to over a given number of training iterations (trips from start to goal).  '''Important note:''' the problem includes random actions.  So, for example, if your learner responds with a "move north" action, there is some probability that the robot will actually move in a different direction.  For this reason, the "wise" learner develops policies that keep the robot well away from quicksand.  We map this problem to a reinforcement learning problem as follows:
+Overall, we will assess the performance of a policy as the median reward it incurs to travel from the start to the goal (higher reward is better).  We assess a learner in terms of the reward it converges to over a given number of training epochs (trips from start to goal).  '''Important note:''' the problem includes random actions.  So, for example, if your learner responds with a "move north" action, there is some probability that the robot will actually move in a different direction.  For this reason, the "wise" learner develops policies that keep the robot well away from quicksand.  We map this problem to a reinforcement learning problem as follows:
 * State: The state is the location of the robot, it is computed (discretized) as: column location * 10 + row location.
@@ Line 113: / Line 113: @@
 * T: The transition matrix can be inferred from the CSV map and the actions.
-Note that R and T are not known by or available to the learner.  The testing code <tt>testqlearner.py</tt> will test your code as follows (pseudo code):
+Note that R and T are not known by or available to the learner.  The code in <tt>testqlearner.py</tt> will test your code as follows (pseudo code):
 <pre>

@@ Line 24: / Line 24: @@
 ==Template and Data==
-* Update your local mc3_p2 directory using github.
+* Update your local mc3p2_qlearning_robot directory using github.
-* Implement the <tt>QLearner</tt> class in <tt>mc3_p2/QLearner.py</tt>.
+* Implement the <tt>QLearner</tt> class in <tt>mc3p2_qlearning_robot/QLearner.py</tt>.
-* To test your Q-learner, run <tt>'''python testqlearner.py'''</tt> from the <tt>mc3_p2/</tt> directory.
+* To debug your Q-learner, run <tt>'''python testqlearner.py'''</tt> from the <tt>mc3p2_qlearning_robot/</tt> directory. The grading script for this project is <tt>grade_robot_qlearning.py</tt>.
-* Note that example navigation problems are provided in the <tt>mc3_p2/testworlds</tt> directory
+* Note that example navigation problems are provided in the <tt>mc3p2_qlearning_robot/testworlds</tt> directory.
 ==Part 1: Implement Q-Learner (95%)==

@@ Line 33: / Line 33: @@
 Your QLearner class should be implemented in the file <tt>QLearner.py</tt>.  It should implement EXACTLY the API defined below.  DO NOT import any modules besides those allowed below.  Your class should implement the following methods:
-<b>The constructor QLearner()</b> should reserve space for keeping track of Q[s, a] for the number of states and actions.  It should initialize Q[] with uniform random values between -1.0 and 1.0.  Details on the input arguments to the constructor:
+<b>The constructor QLearner()</b> should reserve space for keeping track of Q[s, a] for the number of states and actions.  It should initialize Q[] with all zeros.  Details on the input arguments to the constructor:
 * <tt>num_states</tt> integer, the number of states to consider

← Older revision		Revision as of 23:03, 19 May 2017
Line 1:		Line 1:
		+	==DRAFT==
		+
		+	This assignment is under revision. This notice will be removed once it is final.
		+
	==Updates / FAQs==		==Updates / FAQs==

@@ Line 233: / Line 233: @@
 * You may reuse sections of code (up to 5 lines) that you collected from other students or the internet.
 * Code provided by the instructor, or allowed by the instructor to be shared.
 Prohibited: