Dynamic Programming

Complete Knowledge of MDP

Finite State and Action Spaces

Computational Requirements

Iterative Policy Evaluation Algorithm

Convergence Properties

Computational Complexity

Stopping Criteria

In-Place vs Synchronous Updates

Policy Improvement Theorem

Greedy Policy Construction

Policy Improvement Guarantees

Monotonic Improvement Property

Policy Iteration Algorithm

Convergence Analysis

Computational Complexity

Finite Convergence Property

Value Iteration Algorithm

Update Rules

Stopping Criteria

Convergence Properties

Relationship to Policy Iteration

Interleaving Evaluation and Improvement

Flexible Implementation

Convergence Guarantees

Modified Policy Iteration

Asynchronous Dynamic Programming

Prioritized Sweeping

Curse of Dimensionality

Model Requirements

Computational Scalability