Reinforcement Learning

  1. Temporal-Difference Learning
    1. Introduction to TD Learning
      1. Bootstrapping Concept
        1. Sample-Based Updates
          1. Online Learning Capability
          2. TD Prediction
            1. TD(0) Algorithm
              1. Update Rule
                1. Learning Rate Effects
                  1. Convergence Properties
                  2. Bias-Variance Properties
                    1. Biased but Lower Variance
                      1. Comparison with Monte Carlo
                    2. TD Control Methods
                      1. SARSA (On-Policy TD Control)
                        1. Algorithm Description
                          1. Update Rule
                            1. Convergence Properties
                              1. Exploration Strategies
                              2. Q-Learning (Off-Policy TD Control)
                                1. Algorithm Description
                                  1. Off-Policy Nature
                                    1. Convergence Guarantees
                                      1. Exploration Independence
                                      2. Expected SARSA
                                        1. Algorithm Description
                                          1. Reduced Variance
                                            1. Performance Characteristics
                                          2. Multi-Step TD Methods
                                            1. n-Step TD Prediction
                                              1. n-Step Returns
                                                1. Bias-Variance Trade-off
                                                  1. Parameter Selection
                                                  2. n-Step SARSA
                                                    1. n-Step Q-Learning
                                                    2. TD(λ) Methods
                                                      1. Eligibility Traces
                                                        1. Trace Decay
                                                          1. Credit Assignment
                                                          2. TD(λ) Prediction
                                                            1. SARSA(λ)
                                                              1. Q(λ)
                                                              2. Comparing Learning Methods
                                                                1. Monte Carlo vs TD
                                                                  1. Bias-Variance Analysis
                                                                    1. Sample Efficiency
                                                                      1. Computational Requirements

                                                                    Previous

                                                                    5. Monte Carlo Methods

                                                                    Go to top

                                                                    Next

                                                                    7. Function Approximation