Machine Learning

9.

Reinforcement Learning

9.1.

Fundamental Concepts

9.1.1.

Agent-Environment Interaction

9.1.1.1.

Agent Definition

9.1.1.2.

Environment Definition

9.1.1.3.

Interaction Loop

9.1.2.

States and State Spaces

9.1.2.1.

State Representation

9.1.2.2.

Discrete vs. Continuous States

9.1.2.3.

Partially Observable States

9.1.2.4.

State Abstraction

9.1.3.

Actions and Action Spaces

9.1.3.1.

Discrete Actions

9.1.3.2.

Continuous Actions

9.1.3.3.

Action Selection

9.1.3.4.

Action Constraints

9.1.4.

Rewards and Reward Functions

9.1.4.1.

9.1.4.2.

9.1.4.3.

Sparse vs. Dense Rewards

9.1.4.4.

Delayed Rewards

9.1.5.

9.1.5.1.

Deterministic Policies

9.1.5.2.

Stochastic Policies

9.1.5.3.

Policy Representation

9.1.5.4.

Optimal Policies

9.1.6.

Value Functions

9.1.6.1.

State Value Function

9.1.6.2.

Action Value Function

9.1.6.3.

Advantage Function

9.1.6.4.

Bellman Equations

9.1.7.

Exploration vs. Exploitation

9.1.7.1.

Exploration Strategies

9.1.7.2.

9.1.7.3.

Upper Confidence Bound

9.1.7.4.

Thompson Sampling

9.2.

Markov Decision Processes

9.2.1.

Markov Property

9.2.1.1.

Memoryless Property

9.2.1.2.

State Transition Probabilities

9.2.2.

9.2.2.1.

9.2.2.2.

9.2.2.3.

Transition Function

9.2.2.4.

Reward Function

9.2.2.5.

Discount Factor

9.2.3.

Bellman Equations

9.2.3.1.

Bellman Expectation Equations

9.2.3.2.

Bellman Optimality Equations

9.2.3.3.

Value Iteration

9.2.3.4.

Policy Iteration

9.2.4.

Partially Observable MDPs

9.2.4.1.

9.2.4.2.

Observation Model

9.2.4.3.

POMDP Solving Methods

9.3.

Dynamic Programming

9.3.1.

Value Iteration

9.3.1.1.

Algorithm Steps

9.3.1.2.

Convergence Properties

9.3.1.3.

Computational Complexity

9.3.2.

Policy Iteration

9.3.2.1.

Policy Evaluation

9.3.2.2.

Policy Improvement

9.3.2.3.

Convergence Guarantees

9.3.3.

Generalized Policy Iteration

9.3.3.1.

Interleaving Evaluation and Improvement

9.3.3.2.

Asynchronous Updates

9.4.

Model-Free Methods

9.4.1.

Monte Carlo Methods

9.4.1.1.

9.4.1.2.

9.4.1.3.

9.4.1.4.

Exploring Starts

9.4.1.5.

On-Policy vs. Off-Policy

9.4.2.

Temporal Difference Learning

9.4.2.1.

9.4.2.2.

9.4.2.3.

TD(0) Algorithm

9.4.2.4.

TD(λ) and Eligibility Traces

9.4.3.

9.4.3.1.

Off-Policy Learning

9.4.3.2.

Q-Table Updates

9.4.3.3.

Convergence Properties

9.4.3.4.

Exploration Strategies

9.4.4.

9.4.4.1.

On-Policy Learning

9.4.4.2.

State-Action-Reward-State-Action

9.4.4.3.

Comparison with Q-Learning

9.4.5.

9.4.5.1.

Expected Value Updates

9.4.5.2.

Reduced Variance

9.5.

Function Approximation

9.5.1.

Linear Function Approximation

9.5.1.1.

Feature Vectors

9.5.1.2.

9.5.1.3.

Convergence Issues

9.5.2.

Non-Linear Function Approximation

9.5.2.1.

Neural Network Approximators

9.5.2.2.

Deep Q-Networks

9.5.2.3.

Experience Replay

9.5.2.4.

Target Networks

9.5.2.5.

9.5.2.6.

9.5.3.

Policy Gradient Methods

9.5.3.1.

REINFORCE Algorithm

9.5.3.2.

Policy Gradient Theorem

9.5.3.3.

Baseline Methods

9.5.3.4.

Actor-Critic Methods

9.5.3.4.1.

Advantage Actor-Critic

9.5.3.4.2.

Asynchronous Advantage Actor-Critic

9.5.3.4.3.

Proximal Policy Optimization

9.5.3.4.4.

Trust Region Policy Optimization

9.5.4.

Continuous Control

9.5.4.1.

Deep Deterministic Policy Gradient

9.5.4.2.

Twin Delayed DDPG

9.5.4.3.

Soft Actor-Critic

9.6.

Multi-Agent Reinforcement Learning

9.6.1.

Game Theory Basics

9.6.2.

Nash Equilibrium

9.6.3.

Cooperative vs. Competitive Settings

9.6.4.

Independent Learning

9.6.5.

Centralized Training with Decentralized Execution

9.7.

9.7.1.

9.7.1.1.

9.7.1.2.

9.7.1.3.

Real-Time Strategy

9.7.2.

9.7.2.1.

9.7.2.2.

Manipulation Tasks

9.7.2.3.

9.7.3.

Autonomous Systems

9.7.3.1.

Autonomous Driving

9.7.3.2.

9.7.4.

Resource Management

9.7.4.1.

Network Routing

9.7.4.2.

Energy Management

9.7.4.3.

Portfolio Management

Previous

8. Deep Learning and Neural Networks

Go to top

Next

10. Advanced Topics and Specialized Areas