Optimization Theory

  1. Mathematical Foundations
    1. Linear Algebra Prerequisites
      1. Vector Spaces
        1. Definition and Properties
          1. Subspaces
            1. Linear Independence
              1. Basis and Dimension
              2. Matrices and Matrix Operations
                1. Basic Operations
                  1. Matrix Multiplication
                    1. Transpose and Inverse
                      1. Rank and Nullspace
                      2. Vector and Matrix Norms
                        1. Vector Norms
                          1. L1 Norm (Manhattan Norm)
                            1. L2 Norm (Euclidean Norm)
                              1. Infinity Norm
                                1. p-Norms
                                2. Matrix Norms
                                  1. Induced Norms
                                    1. Frobenius Norm
                                      1. Spectral Norm
                                    2. Eigenvalues and Eigenvectors
                                      1. Definitions and Properties
                                        1. Characteristic Polynomial
                                          1. Diagonalization
                                            1. Spectral Decomposition
                                            2. Quadratic Forms
                                              1. Definition and Matrix Representation
                                                1. Positive Definite Matrices
                                                  1. Positive Semidefinite Matrices
                                                    1. Negative Definite and Semidefinite Matrices
                                                      1. Indefinite Matrices
                                                    2. Multivariable Calculus
                                                      1. Functions of Multiple Variables
                                                        1. Domain and Range
                                                          1. Level Sets and Contour Lines
                                                            1. Continuity and Limits
                                                            2. Partial Derivatives
                                                              1. Definition and Computation
                                                                1. Higher-Order Partial Derivatives
                                                                  1. Mixed Partial Derivatives
                                                                  2. Gradient Vector
                                                                    1. Definition and Geometric Interpretation
                                                                      1. Directional Derivatives
                                                                        1. Gradient as Direction of Steepest Ascent
                                                                        2. Jacobian Matrix
                                                                          1. Definition for Vector-Valued Functions
                                                                            1. Chain Rule Applications
                                                                            2. Hessian Matrix
                                                                              1. Definition and Computation
                                                                                1. Symmetry Properties
                                                                                  1. Relationship to Convexity
                                                                                  2. Taylor Series Expansions
                                                                                    1. First-Order Taylor Approximation
                                                                                      1. Second-Order Taylor Approximation
                                                                                        1. Remainder Terms
                                                                                          1. Applications in Optimization
                                                                                        2. Convex Analysis
                                                                                          1. Convex Sets
                                                                                            1. Definition and Basic Properties
                                                                                              1. Examples of Convex Sets
                                                                                                1. Hyperplanes
                                                                                                  1. Half-Spaces
                                                                                                    1. Polyhedra
                                                                                                      1. Ellipsoids
                                                                                                        1. Norm Balls
                                                                                                        2. Operations Preserving Convexity
                                                                                                          1. Intersection
                                                                                                            1. Affine Transformation
                                                                                                              1. Perspective Function
                                                                                                              2. Separation and Supporting Hyperplane Theorems
                                                                                                              3. Convex Functions
                                                                                                                1. Definition and Characterization
                                                                                                                  1. Examples of Convex Functions
                                                                                                                    1. Linear Functions
                                                                                                                      1. Quadratic Functions
                                                                                                                        1. Exponential Functions
                                                                                                                          1. Logarithmic Functions
                                                                                                                            1. Norms
                                                                                                                            2. Epigraph Characterization
                                                                                                                              1. Jensen's Inequality
                                                                                                                              2. Properties of Convex Functions
                                                                                                                                1. First-Order Conditions
                                                                                                                                  1. Second-Order Conditions
                                                                                                                                    1. Operations Preserving Convexity
                                                                                                                                      1. Nonnegative Weighted Sums
                                                                                                                                        1. Composition Rules
                                                                                                                                          1. Pointwise Maximum
                                                                                                                                        2. Convex Optimization Problems
                                                                                                                                          1. Standard Form
                                                                                                                                            1. Fundamental Properties
                                                                                                                                              1. Local vs Global Optima

                                                                                                                                          Previous

                                                                                                                                          1. Foundations of Optimization

                                                                                                                                          Go to top

                                                                                                                                          Next

                                                                                                                                          3. Unconstrained Optimization