RL learning resources
See also the section Reinforcement Learning in my post of Learning resources.
Contents
Theory
Books
- RL Theory book (Available for free from the author)
- Puterman - MDPs
- Bandit Algorithms (Available for free from the author)
- Morimura RL book (not as theoretical, but in the middle between theory and application)
Papers
- Q-learning convergence:
- Jaakaal
- Sham Kakade PhD Thesis
Courses
- Nan Jiang RL course
- NOTE: since this is a running course check the links inside the provided website to previous semesters for slides/notes
- Nan Jiang Statistical RL course
- NOTE: since this is a running course check the links inside the provided website to previous semesters for slides/notes
- Wen Sun and Sham Kakade - Foundations of RL course
Workshops
- Theory of RL program
- Theory of Reinforcement Learning Boot Camp
- Mathematics of Online Decision Making
- NOTE: these talks are not necessarily on RL methods but on algorithms that process data in an online manner (instead of on batches of fixed data) which is useful for example when the data is continually being updated or when it is so large that the whole batch cannot fit in the memory of the computer.
- [Reinforcement Learning from Batch Data and Simulation](
- Deep Reinforcement Learning
YouTube
- 2021 RL Theory Course Website Youtube
- 2022 RL Theory Course Website Youtube
- NOTE: The 2022 course uses a flip classroom setting, which means the students watched the recordings from 2021 and attended class for discussions about those contents. Therefore, I recommend watching 2021 first (although maybe watching in parallel would be even better).
- RL Theory Seminars Website YouTube Channel
- YOUTUBE Brunskill - A tutorial on RL 1
- YOUTUBE Brunskill - A tutorial on RL 2
Suggested learning roadmap
- Prerrequisites
- Learn ML basics (regression; classification; NNs basics)
- Learn RL basics (value iteration; policy iteration; td; q-learning)
- Read Ch1-Ch4 of Sutton’s book (do not take too long on small details for the first read)
- Watch Silver’s RL lectures, Levine’s DRL lectures, Abbeel Foundations of RL lectures
- Implement Q-Learning in a simple environment (e.g. recycling robot in Sutton book; any gridworld from Sutton’s book). A better suggestion is to try to replicate an experiment from Sutton’s book and try to get the same discounted reward graph. You will notice you need to do a lot of debugging to make it work, and that the actual experiments have a lot of noise so that is why usually we average the results over many trials. In my opinion, only after implementing you can get a real understanding of the methods.
- For people with no mathematical background (my case when starting PhD), I suggest reading
- Real Analysis book (as much as needed; to be honest I still lack a lot of knowledge in this area). (e.g. Axler’s MIRA supplementary material book which discusses supremum/infimum/ and other concepts that are used a lot in theory papers)
- Linear Algebra, both the matrix setting (e.g. Gilbert Strange book/course) and vector space setting (e.g. Axler’s Linear Algebra Done Right)
- RL Theory (consult these resources is any order; try to complement what you do not understand from each other resource or cited papers)
- Watch RL Theory Course from Szepesvari; Read Badint Algorithms sections suggested;
Application
Books
- Sutton - RL book
- Morimura RL book (in the middle between theory and application)
Papers
- See the papers cited in these two repositories
- https://github.com/higgsfield/RL-Adventure
- https://github.com/henanmemeda/RL-Adventure-2
- See the papers cited in these website
- https://spinningup.openai.com/en/latest/spinningup/rl_intro.html
Websites
- https://spinningup.openai.com/en/latest/spinningup/rl_intro.html
- I think the library is outdated and it is better to use more modern libraries (dopamine; ray_rl; etc), although I might be wrong. The most important for me about this website is that it provides a good summary of the most important developments in Deep RL for value-based and policy gradient methods.
Workshops
Courses
- Silver RL course
- Abbeel Foundations of RL
- Levine Deep RL course