HW2 (new posts)

couldn't connect to display "gate.tau.ac.il:0": Re: couldn't connect to display "gate.tau.ac.il:0"

Mon, 08 Apr 2019 09:33:59 +0000

The gate does not support graphics, you should either turn the graphics off or connect via VPN.

You can check out Vered's post for instructions:
http://rl-tau-2019.wikidot.com/forum/t-10700213#post-4190456

couldn't connect to display "gate.tau.ac.il:0": couldn't connect to display "gate.tau.ac.il:0"

Sun, 07 Apr 2019 15:31:47 +0000

I recieved this error when attempting to run the code on nova remotely.
When I ran the code from Schreiber's lab and from my computer everything worked fine, what is this error?

Programming - not making in nova: Re: Programming - not making in nova

Mon, 01 Apr 2019 20:13:22 +0000

Try to add env.close() at the end of the file and make sure you don't call plt.show() (which stays idle until you close the image).

Programming - not making in nova: Programming - not making in nova

Mon, 01 Apr 2019 17:36:02 +0000

Hi,
the line env = gym.make('FrozenLake-v0') does not work on the nova for us.
it runs and do not end.. (no error or something, all we know is that it gets to this line in the code and not further).

Any ideas to why it happens?

thanks

Programming Q1 - Duplicate states in mdp.P: Re: Programming Q1 - Duplicate states in mdp.P

Mon, 01 Apr 2019 08:45:20 +0000

If you iterate over all the values in the mdp.P[state][action] when you calculate the current value iteration it should be fine.

Programming Q1 - Duplicate states in mdp.P: Programming Q1 - Duplicate states in mdp.P

Sun, 31 Mar 2019 18:23:27 +0000

In mdp.P, the following holds according to the comments:
for every (state,action), mdp.P[state][action] is a list of (probability,next_state,reward) tuples.
is it okay for this list to contain duplicate state, for example:
P[0][0] = [(0.3333333333333333, 0, 0.0), (0.3333333333333333, 0, 0.0), (0.3333333333333333, 4, 0.0)]

if so - what should we do with these duplicates?

Thanks

Theoretical Q2.1: Re: Theoretical Q2.1

Tue, 26 Mar 2019 15:24:57 +0000

We always wish to minimize the number of states (that contains enough information for the optimal policy),
since the computation complexity of the Value Iteration depends on it.
You have to incorporate the randomness of the current new digit d in the states since the (optimal) policy should consider it.

Theoretical Q2.1: Theoretical Q2.1

Tue, 26 Mar 2019 11:03:24 +0000

We are asked to build an MDP.
On the one hand the hint suggests that we "may incorporate the random d" (include it as part of the state space?)
On the other hand the random d is not a part of the MDP i.e. the MDP should only care which slots are empty/occupied.
What should we do?

Theoretical part - Ex.2: value iteration

Tue, 26 Mar 2019 09:58:30 +0000

I suppose you mean Value Iteration.
Both definitions are equal- recall that

(1)

\begin{align} R(s,a):=E_{s'}[R(s,a,s')]=\sum_{s'}P(s,a,s')R(s,a,s') \end{align}

The rest are simply notional differences.

Theoretical part - Ex.2:

Tue, 26 Mar 2019 08:31:06 +0000

In the lecture we learned a function for the "value function". While in the HW in question 1 of the programming part we can see a different function representing the "value function".

Which function should we use in question 2 in the theoretical part?

Theoretical part - Ex.2:

Mon, 25 Mar 2019 22:01:29 +0000

Look at the definition of the transition probability matrix.
You should define it for every states s,s' and action a.
A probability p(s'|s,a) can't be 'invalid'- think what should be the value in these cases.

Theoretical part - Ex.2:

Mon, 25 Mar 2019 19:54:06 +0000

So what do we do with the transition probability function? Should we define the probability of the next state given some s, and an action that is invalid? or just ignore it?
Thanks

Theoretical part - Ex.2: Re: Theoretical part - Ex.2

Mon, 25 Mar 2019 09:31:04 +0000

A_t is not part of the MDP.
In order to define an MDP you need to define A, a set of all possible actions.

Theoretical part - Ex.2: Theoretical part - Ex.2

Sun, 24 Mar 2019 18:42:36 +0000

I'm having some problems defining an MDP for this problem, as the valid actions change depending on the current state (putting a number in a taken cell is forbidden, and the contents of the cells is represented in the current state)
Can you elaborate as to how we should define an MDP in this case? Should we ignore the possibility of an invalid action?

Issues with "from gym.spaces import prng": Issues with "from gym.spaces import prng"

Wed, 20 Mar 2019 20:11:13 +0000

tl;dr: pip install gym==0.10.3

For those of you who use the latest version of gym (0.12.0), the above import will not work.
That is because openai removed it in version 0.11.0: https://github.com/openai/gym
My advice for you is to downgrade gym version to match the version installed on nova, which is 0.10.3.
See tl;dr for suggested way to downgrade.