Mathematics for Machine Learning and Data Science

Optimization Techniques for Machine Learning

6.1.

Fundamentals of Optimization

6.1.1.

Optimization Problem Formulation

6.1.1.1.

Objective Functions

6.1.1.2.

Decision Variables

6.1.1.3.

Constraints

6.1.2.

Types of Optimization Problems

6.1.2.1.

Unconstrained Optimization

6.1.2.2.

Constrained Optimization

6.1.2.3.

Linear Programming

6.1.2.4.

Nonlinear Programming

6.1.2.5.

6.1.3.

6.1.3.1.

6.1.3.2.

6.1.3.3.

Properties of Convex Problems

6.1.3.4.

Global vs Local Optima

6.1.4.

Loss Functions in Machine Learning

6.1.4.1.

6.1.4.2.

6.1.4.3.

6.1.4.4.

6.1.4.5.

6.2.

Unconstrained Optimization

6.2.1.

Optimality Conditions

6.2.1.1.

First-Order Necessary Conditions

6.2.1.2.

Second-Order Necessary Conditions

6.2.1.3.

Second-Order Sufficient Conditions

6.2.2.

6.2.2.1.

6.2.2.2.

6.2.2.3.

6.2.2.4.

6.2.3.

6.2.3.1.

Algorithm Description

6.2.3.2.

Convergence Analysis

6.2.3.3.

Step Size Selection

6.2.3.4.

Batch Gradient Descent

6.2.3.5.

Stochastic Gradient Descent

6.2.3.6.

Mini-Batch Gradient Descent

6.2.4.

Newton's Method

6.2.4.1.

Algorithm Description

6.2.4.2.

Convergence Properties

6.2.4.3.

Computational Considerations

6.2.5.

6.2.5.1.

6.2.5.2.

6.2.5.3.

6.2.6.

Conjugate Gradient Methods

6.2.6.1.

Linear Conjugate Gradient

6.2.6.2.

Nonlinear Conjugate Gradient

6.2.6.3.

Fletcher-Reeves Method

6.2.6.4.

Polak-Ribière Method

6.3.

Advanced Gradient-Based Methods

6.3.1.

Momentum Methods

6.3.1.1.

Classical Momentum

6.3.1.2.

Nesterov Accelerated Gradient

6.3.1.3.

Heavy Ball Method

6.3.2.

Adaptive Learning Rate Methods

6.3.2.1.

AdaGrad

6.3.2.1.1.

Algorithm Description

6.3.2.1.2.

Adaptive Learning Rates

6.3.2.1.3.

Convergence Properties

6.3.2.2.

RMSprop

6.3.2.2.1.

Exponential Moving Average

6.3.2.2.2.

Addressing AdaGrad Limitations

6.3.2.3.

Adam

6.3.2.3.1.

Adaptive Moment Estimation

6.3.2.3.2.

6.3.2.3.3.

6.3.2.4.

6.3.2.5.

6.3.3.

Learning Rate Scheduling

6.3.3.1.

6.3.3.2.

6.3.3.3.

6.3.3.4.

6.3.3.5.

6.3.4.

6.3.4.1.

Gradient Norm Clipping

6.3.4.2.

Gradient Value Clipping

6.3.4.3.

Applications to RNNs

6.4.

Constrained Optimization

6.4.1.

Equality Constrained Optimization

6.4.1.1.

Lagrange Multipliers

6.4.1.2.

Lagrangian Function

6.4.1.3.

First-Order Optimality Conditions

6.4.1.4.

Second-Order Conditions

6.4.2.

Inequality Constrained Optimization

6.4.2.1.

Karush-Kuhn-Tucker Conditions

6.4.2.2.

Complementary Slackness

6.4.2.3.

Constraint Qualification

6.4.3.

Penalty Methods

6.4.3.1.

Quadratic Penalty Method

6.4.3.2.

Exact Penalty Methods

6.4.3.3.

Augmented Lagrangian Method

6.4.4.

Barrier Methods

6.4.4.1.

Interior Point Methods

6.4.4.2.

Logarithmic Barrier Function

6.4.4.3.

Central Path

6.4.5.

Sequential Quadratic Programming

6.4.6.

Active Set Methods

6.5.

Specialized Optimization for Machine Learning

6.5.1.

Coordinate Descent

6.5.1.1.

Algorithm Description

6.5.1.2.

Block Coordinate Descent

6.5.1.3.

Applications to Lasso and Ridge Regression

6.5.2.

Proximal Methods

6.5.2.1.

Proximal Operators

6.5.2.2.

Proximal Gradient Method

6.5.2.3.

Accelerated Proximal Methods

6.5.2.4.

Applications to Sparse Optimization

6.5.3.

Subgradient Methods

6.5.3.1.

Subgradients and Subdifferentials

6.5.3.2.

6.5.3.3.

6.5.4.

6.5.4.1.

6.5.4.2.

6.5.4.3.

Applications to Online Learning

6.5.5.

Frank-Wolfe Algorithm

6.5.5.1.

Conditional Gradient Method

6.5.5.2.

Sparse Solutions

6.5.5.3.

Applications to Structured Sparsity

6.6.

Stochastic Optimization

6.6.1.

Stochastic Gradient Descent

6.6.1.1.

Algorithm Variants

6.6.1.2.

Convergence Analysis

6.6.1.3.

Learning Rate Schedules

6.6.2.

Variance Reduction Methods

6.6.2.1.

SVRG (Stochastic Variance Reduced Gradient)

6.6.2.2.

SAG (Stochastic Average Gradient)

6.6.2.3.

SAGA

6.6.3.

Online Optimization

6.6.3.1.

Online Gradient Descent

6.6.3.2.

Regret Analysis

6.6.3.3.

Follow-the-Regularized-Leader

6.6.4.

Evolutionary Algorithms

6.6.4.1.

Genetic Algorithms

6.6.4.2.

Particle Swarm Optimization

6.6.4.3.

Differential Evolution

6.7.

Optimization in Deep Learning

6.7.1.

Challenges in Deep Learning Optimization

6.7.1.1.

Non-Convexity

6.7.1.2.

High Dimensionality

6.7.1.3.

Saddle Points

6.7.1.4.

Vanishing and Exploding Gradients

6.7.2.

Batch Normalization

6.7.2.1.

Internal Covariate Shift

6.7.2.2.

Algorithm Description

6.7.2.3.

Effects on Optimization

6.7.3.

Dropout as Regularization

6.7.4.

Weight Initialization Strategies

6.7.4.1.

Xavier Initialization

6.7.4.2.

He Initialization

6.7.4.3.

Effects on Convergence

6.7.5.

Second-Order Methods for Deep Learning

6.7.5.1.

Natural Gradients

6.7.5.2.

K-FAC (Kronecker-Factored Approximate Curvature)

6.7.5.3.

Shampoo Algorithm

5. Statistics: From Data to Insights

Go to top

Back to Start

1. Foundations of Mathematical Notation and Concepts