CartPole DQN - Revision history

Petosa: /* Replay */

2018-02-18T06:41:37Z

Replay

Petosa: /* CartPole */

2018-02-18T01:12:04Z

CartPole

Petosa: /* CartPole */

2018-02-17T23:23:52Z

CartPole

Petosa: /* CartPole */

2018-02-17T23:21:33Z

CartPole

Petosa: /* CartPole */

2018-02-17T23:11:54Z

CartPole

Petosa: /* DQN Agent */

2018-02-17T23:05:18Z

DQN Agent

Petosa: /* CartPole */

2018-02-17T23:04:58Z

CartPole

Petosa: /* CartPole */

2018-02-17T22:50:43Z

CartPole

Petosa: /* Tutorial */

2018-02-17T22:47:30Z

Tutorial

Petosa: /* DQN Agent */

2018-02-17T22:35:51Z

DQN Agent

@@ Line 133: / Line 133: @@
      for i in range(len(predictions)):
          # Flag states as terminal (the last state before a epoch ended).
-         terminal_state = (next_states[i] == np.array([None, None, None, None])).all()
+         terminal_state = (next_states[i] == np.array([None]*self.input_dim)).all()
          # Update each state's Q-value prediction with our new estimate.
          # Terminal states have no future, so set their Q-value to their immediate reward.

@@ Line 198: / Line 198: @@
              state = next_state # Advance state
              agent.replay() # Train the network form replay samples.
 </PRE>

@@ Line 153: / Line 153: @@
 The first thing we want to do is create the CartPole gym environment and refresh the environment.
 <PRE>
-env = gym.make(env_name)
+env = gym.make("CartPole-v0")
 env.reset()
 </PRE>
@@ Line 186: / Line 186: @@
          if done: # Episode is completed due to failure or cap being reached.
-             print("Episode: {}, Total reward: {}, Explore P: {}".format(ep, total_reward, agent.explore_p)
+             print("Episode: {}, Total reward: {}, Explore P: {}".format(ep, total_reward, agent.explore_p))
              if total_reward == 999: # Simulation completed without failure. Save a copy of this network.
                  agent.model.save("cartpole.h5")

@@ Line 151: / Line 151: @@
 from DQNAgent import DQNAgent
 </PRE>
+The first thing we want to do is create the CartPole gym environment and refresh the environment.
 <PRE>
 env = gym.make(env_name)
 env.reset()
 </PRE>
 Gym caps episodes of CartPole to 200 steps. In other words, the epoch will be cut off after our cart takes 200 actions. We can disable this limit by setting it to <code>None</code> or by raising it to some constant. For this tutorial, we will set the limit to 1000.
 <PRE>
@@ Line 173: / Line 170: @@
 <ul>
 <li><code>state</code> is the state the environment is in after a step. For cartpole, this will be an numpy array of 4 values representing the position of the cart from the center, the carts velocity, the angle of the pole from the vertical, and the angular velocity of the pole.</li>
-<li><code>reward</code> is the immediate reward witnessed by the agent for taking this action.</li>
+<li><code>reward</code> is the immediate reward witnessed by the agent for taking this action. In CartPole, the reward is always 1 for staying alive.</li>
 <li><code>done</code> a boolean indicating whether the epoch is over.</li>
 </ul>
@@ Line 191: / Line 188: @@
              print("Episode: {}, Total reward: {}, Explore P: {}".format(ep, total_reward, agent.explore_p)
              timeseries.append(total_reward)
-            # Save
+             if total_reward == 999: # Simulation completed without failure. Save a copy of this network.
-             if save_thresh is not None and total_reward > save_thresh:
+                 agent.model.save("cartpole.h5")
-                 agent.model.save("modelout_" + str(int(total_reward)) + ".h5")
+             # Add experience to bucket (next_state is None since epoch is over).
              agent.remember(state, action, None, reward)
-             env.reset()
+             env.reset() # Reset environment
              break
-         else:
+         else: # Episode not over.
-             agent.remember(state, action, next_state, reward)
+             agent.remember(state, action, next_state, reward) # Store tuple.
-             state = next_state
+             state = next_state # Advance state
-             agent.replay()
+             agent.replay() # Train the network form replay samples.
 </PRE>

@@ Line 152: / Line 152: @@
 </PRE>
-The first thing we want to do is create the CartPole gym environment and refresh the environment.
 <PRE>
 env = gym.make(env_name)
@@ Line 161: / Line 162: @@
 <PRE>
 env._max_episode_steps = 1000
 </PRE>
 We now take the first step of our simulation and save its results to some variables:
@@ Line 171: / Line 176: @@
 <li><code>done</code> a boolean indicating whether the epoch is over.</li>
 </ul>

@@ Line 100: / Line 100: @@
 <PRE>
 def remember(self, state, action, next_state, reward):
-     # Create a blank state. Serves as next_state if this was the last experience tuple before the simulation ended.
+     # Create a blank state. Serves as next_state if this was the last experience tuple before the epoch ended.
      terminal_state = np.array([None]*self.input_dim)
      # Add experience tuple to bucket. Bucket is a deque, so older tuple falls out on overflow.
@@ Line 132: / Line 132: @@
      # Update the network's predictions with the new predictions we have.
      for i in range(len(predictions)):
-         # Flag states as terminal (the last state before a simulation ended).
+         # Flag states as terminal (the last state before a epoch ended).
          terminal_state = (next_states[i] == np.array([None, None, None, None])).all()
          # Update each state's Q-value prediction with our new estimate.

← Older revision		Revision as of 22:50, 17 February 2018
Line 162:		Line 162:
	env._max_episode_steps = 1000		env._max_episode_steps = 1000
	</PRE>		</PRE>
		+	We now take the first step of our simulation and save its results to some variables:
		+	<PRE>
		+	state, reward, done, _ = env.step(env.action_space.sample())
		+	</PRE>
		+	<ul>
		+	<li><code>state</code> is the state the environment is in after a step. For cartpole, this will be an numpy array of 4 values representing the position of the cart from the center, the carts velocity, the angle of the pole from the vertical, and the angular velocity of the pole.</li>
		+	</ul>

← Older revision		Revision as of 22:47, 17 February 2018
Line 142:		Line 142:
	</PRE>		</PRE>
	This completes our DQNAgent object. Now let's move on to the actual driver of the learning process, which interacts with CartPole to drive learning.		This completes our DQNAgent object. Now let's move on to the actual driver of the learning process, which interacts with CartPole to drive learning.
		+
		+	===CartPole===
		+	Now we will create the script that utilizes a DQNAgent to learn how to play CartPole. Start by creating a file <code>CartPole.py</code> and include the following imports:
		+
		+	<PRE>
		+	import gym
		+	import numpy as np
		+	from DQNAgent import DQNAgent
		+	</PRE>
		+
		+	The first thing we want to do is create the CartPole gym environment and refresh the environment.
		+	<PRE>
		+	env = gym.make(env_name)
		+	env.reset()
		+	</PRE>
		+
		+	Gym caps episodes of CartPole to 200 steps. In other words, the simulation will be cut off after our cart takes 200 actions. We can disable this limit by setting it to <code>None</code> or by raising it to some constant. For this tutorial, we will set the limit to 1000.
		+	<PRE>
		+	env._max_episode_steps = 1000
		+	</PRE>

@@ Line 25: / Line 25: @@
 Start by creating a file <code>DQNAgent.py</code> and include the following imports:
 <PRE>
 from keras.layers import Input, Dense
 from keras.optimizers import RMSprop
@@ Line 61: / Line 62: @@
      # Save instance variables.
-     self.learning_rate = learning_rate
+     self.input_dim = input_dim
      self.batch_size = batch_size
      self.gamma = gamma
@@ Line 103: / Line 105: @@
      self.memory.append((state, action, terminal_state if next_state is None else next_state, reward))
 </PRE>