Theoretical part - Ex.2

Theoretical part - Ex.2

amitz (guest) 24 Mar 2019 18:42

I'm having some problems defining an MDP for this problem, as the valid actions change depending on the current state (putting a number in a taken cell is forbidden, and the contents of the cells is represented in the current state)
Can you elaborate as to how we should define an MDP in this case? Should we ignore the possibility of an invalid action?

Re: Theoretical part - Ex.2

Lee Cohen 25 Mar 2019 09:31

A_t is not part of the MDP.
In order to define an MDP you need to define A, a set of all possible actions.

Last edited on 25 Mar 2019 09:31 by Lee Cohen Show more

Amit Zohar (guest) 25 Mar 2019 19:54

So what do we do with the transition probability function? Should we define the probability of the next state given some s, and an action that is invalid? or just ignore it?
Thanks

Lee Cohen 25 Mar 2019 22:01

Look at the definition of the transition probability matrix.
You should define it for every states s,s' and action a.
A probability p(s'|s,a) can't be 'invalid'- think what should be the value in these cases.

Last edited on 25 Mar 2019 22:03 by Lee Cohen Show more

or wolkomir (guest) 26 Mar 2019 08:31

In the lecture we learned a function for the "value function". While in the HW in question 1 of the programming part we can see a different function representing the "value function".

Which function should we use in question 2 in the theoretical part?

value iteration

Lee Cohen 26 Mar 2019 09:58

I suppose you mean Value Iteration.
Both definitions are equal- recall that

(1)

\begin{align} R(s,a):=E_{s'}[R(s,a,s')]=\sum_{s'}P(s,a,s')R(s,a,s') \end{align}

The rest are simply notional differences.