Reinforcement Learning

  1. Markov Decision Processes
    1. The Markov Property
      1. Definition and Significance
        1. Memoryless Property
          1. Examples of Markov Processes
            1. Non-Markov Processes and Solutions
            2. Formal Definition of MDPs
              1. State Space
                1. Finite vs Infinite States
                  1. State Space Structure
                    1. State Features and Representation
                    2. Action Space
                      1. Available Actions per State
                        1. Action Space Constraints
                          1. Action Dependencies
                          2. Transition Probability Function
                            1. Transition Matrix Representation
                              1. Stochastic Transitions
                                1. Deterministic Special Cases
                                2. Reward Function
                                  1. State-based Rewards
                                    1. Action-based Rewards
                                      1. State-Action-State Rewards
                                        1. Expected Reward Calculation
                                        2. Discount Factor
                                          1. Present Value Calculation
                                            1. Effects of Different Discount Rates
                                              1. Undiscounted vs Discounted Returns
                                            2. Returns and Value Functions
                                              1. Return Definition
                                                1. Cumulative Reward
                                                  1. Discounted Return
                                                    1. Average Return
                                                    2. State-Value Function
                                                      1. Expected Return from States
                                                        1. Value Function Properties
                                                        2. Action-Value Function
                                                          1. Expected Return from State-Action Pairs
                                                            1. Q-Function Properties
                                                            2. Relationship Between Value Functions
                                                            3. Policies in MDPs
                                                              1. Policy Definition
                                                                1. Deterministic Policies
                                                                  1. State-to-Action Mapping
                                                                  2. Stochastic Policies
                                                                    1. Probability Distributions over Actions
                                                                      1. Policy Parameterization
                                                                      2. Policy Evaluation
                                                                        1. Computing Value Functions for Given Policies
                                                                      3. Bellman Equations
                                                                        1. Bellman Expectation Equations
                                                                          1. For State-Value Functions
                                                                            1. For Action-Value Functions
                                                                              1. Recursive Structure
                                                                              2. Bellman Optimality Equations
                                                                                1. Optimal State-Value Function
                                                                                  1. Optimal Action-Value Function
                                                                                    1. Uniqueness Properties
                                                                                    2. System of Linear Equations
                                                                                      1. Matrix Form Representation
                                                                                        1. Solving Bellman Equations
                                                                                      2. Optimality in MDPs
                                                                                        1. Optimal Policies
                                                                                          1. Definition of Optimality
                                                                                            1. Existence and Uniqueness
                                                                                              1. Partial Ordering of Policies
                                                                                              2. Optimal Value Functions
                                                                                                1. Optimal State Values
                                                                                                  1. Optimal Action Values
                                                                                                    1. Relationship to Optimal Policies
                                                                                                    2. Finding Optimal Policies
                                                                                                      1. Policy Extraction from Value Functions
                                                                                                        1. Greedy Policy Construction

                                                                                                    Previous

                                                                                                    2. Mathematical Foundations

                                                                                                    Go to top

                                                                                                    Next

                                                                                                    4. Dynamic Programming