Thanks

Can the Moed A exam and its solution be uploaded?

Thanks

In the LQR lecture we defined controllability as a sufficient condition for solving the ARE equations.
Then we defined stability which basically tell us if our system will explode or not depending on the eigenvalues of the proposed optimal solution.
Can someone explain how are the two related ?
We can reach every state but then cannot stay there? we will try to reach it but the system will be very unstable?
Also it says that a good system is a system where the eigenvalues are lower than 1 hence x_t goes to 0, why is it good?
We want x_t to be a specific state and not zero.

Thanks!

In the exams you published there are questions that provide traces and ask us to compute the V or Q function via some method.
My question is, how do we know if the traces were produced via on-policy or by off-policy?
This changes dramatically the computation of the estimated Q/V function.

Thanks

Thanks

When do we update the entry Q(5,5)?
Since it is the target (room 5), it seems it can be updated only when the episod is starting with that state?
If it would have been stayed zero, we would have never reached Q values greater than 100

In equation 8, shouldn't we write t instead of t square? indeed, in the last expression, the t square is replaced by t.

in the explanation before (16), delta(i) = mu1 - mu2

in equation 22, should we assume alpha is greather than 1?

Toda,
Rafi

Thanks

recitation 13, 13-1 part a - there is no 'return' action from state 1 to itself, index should start from 2.

recitation 11, page 11-4 , the decreasing curve is open-left and the increasing curve is open right, the x-axis is b(s(l)) state (tiger on left door).

page 11-5, the second equation : 0.15 + 0.7 * b(s(l))

Toda
Rafi

For those of us that did not attend the last class/recitation is there any special remarks we need to know?
Can we bring formula pages ?

Any information would be great,
Thanks!

in page 1-1, 1-2, 1-5 - mainly a matter of notation and being consistent with the class, the index of R/r should start from t and not from t + 1

in page 1-3, last equation at the bottom, in the second expression there is i believe a missing gamma

1-4, 1-5, perhaps it's worth mentioning as a footnote or something… V(lambda, t) is the same as G (lambda, t) (as presented in the first page) and V(n, t) is the same as G(n, t)

thanks
Rafi

in 5-6 , mu is a fuction of teta and not of sigma ; in the second gradient, it should be written deviation (sigma) and not average

thanks
Rafi

Thanks

Thanks!

1. Using google colab
You can copy my colab notebook: https://colab.research.google.com/drive/1JRS6xTvBKGL74mJP6wyj1RlBhU7-4A9O and transfer your code to the new notebook.
It requires adjusting the code and takes some babysitting (it only lasts 12H) but it's free available GPU to get you started quickly.

2. Using virtual environment on another server
Some of you may have access to more storage on the university's machine, this is for you (since it takes about 1.5gb and regular disk quota is 1gb).
Just follow the instructions below:

• ssh savant/rack-gamir-g04/5/6 (from nova)
• bash
• cd <directory for your env>
• # create the virtual environment once
• virtualenv -p /usr/local/lib/anaconda3-5.1.0/bin/python dqn_env
• # NOTE: you need to activate venv each time before running
• source dqn_env/bin/activate

• # install all of the requirements
• pip install "gym[atari]"==0.9.5
• pip install opencv-python
• pip install torch
• pip install matplotlib

• # run project from within the environment
• python main.py

• # sanity check from within the environment
• python
• import gym
• print(gym.version) # make sure it says 0.9.5
• gym.make('PongNoFrameskip-v4') # make sure this command succeeds

I tried unsuccessfully to install ffmpeg from some other places I found online.
is there an option not to use ffmpeg? what would you recommend to do in my case with windows?

Thanks