Optimization algorithms and methods

Adaptive coordinate descent

Adaptive coordinate descent is an improvement of the coordinate descent algorithm to non-separable optimization by the use of . The adaptive coordinate descent approach gradually builds a transformation of the coordinate system such that the new coordinates are as decorrelated as possible with respect to the objective function. The adaptive coordinate descent was shown to be competitive to the state-of-the-art evolutionary algorithms and has the following invariance properties: 1. * Invariance with respect to monotonous transformations of the function (scaling) 2. * Invariance with respect to orthogonal transformations of the search space (rotation). CMA-like Adaptive Encoding Update (b) mostly based on principal component analysis (a) is used to extend the coordinate descent method (c) to the optimization of non-separable problems (d). The adaptation of an appropriate coordinate system allows adaptive coordinate descent to outperform coordinate descent on non-separable functions. The following figure illustrates the convergence of both algorithms on 2-dimensional Rosenbrock function up to a target function value , starting from the initial point . The adaptive coordinate descent method reaches the target value after only 325 function evaluations (about 70 times faster than coordinate descent), that is comparable to gradient-based methods. The algorithm has linear time complexity if update coordinate system every D iterations, it is also suitable for large-scale (D>>100) non-linear optimization. (Wikipedia).

Adaptive coordinate descent
Video thumbnail

Numerical Optimization Algorithms: Gradient Descent

In this video we discuss a general framework for numerical optimization algorithms. We will see that this involves choosing a direction and step size at each step of the algorithm. In this video, we investigate how to choose a direction using the gradient descent method. Future videos d

From playlist Optimization

Video thumbnail

Gradient descent

This video follows on from the discussion on linear regression as a shallow learner ( https://www.youtube.com/watch?v=cnnCrijAVlc ) and the video on derivatives in deep learning ( https://www.youtube.com/watch?v=wiiPVB9tkBY ). This is a deeper dive into gradient descent and the use of th

From playlist Introduction to deep learning for everyone

Video thumbnail

Gradient descent tutorial using Julia code

This is a beginner's tutorial explaining gradient descent. In this video I show you how to do implement gradient descent in Julia code. Our model is simple linear regression. I do assume some knowledge of matrix-vector multiplication and partial derivatives. If you want more info on th

From playlist Julia on Coursera

Video thumbnail

C34 Expanding this method to higher order linear differential equations

I this video I expand the method of the variation of parameters to higher-order (higher than two), linear ODE's.

From playlist Differential Equations

Video thumbnail

Information-Constrained Optimization: Can Adaptive Processing of Gradients Help?

A Google TechTalk, presented by Jayadev Acharya, Cornell University, at the 2021 Google Federated Learning and Analytics Workshop, Nov. 8-10, 2021. For more information about the workshop: https://events.withgoogle.com/2021-workshop-on-federated-learning-and-analytics/#content

From playlist 2021 Google Workshop on Federated Learning and Analytics

Video thumbnail

DDPS | "When and why physics-informed neural networks fail to train" by Paris Perdikaris

Physics-informed neural networks (PINNs) have lately received great attention thanks to their flexibility in tackling a wide range of forward and inverse problems involving partial differential equations. However, despite their noticeable empirical success, little is known about how such c

From playlist Data-driven Physical Simulations (DDPS) Seminar Series

Video thumbnail

Paris Perdikaris: "Overcoming gradient pathologies in constrained neural networks"

Machine Learning for Physics and the Physics of Learning 2019 Workshop III: Validation and Guarantees in Learning Physical Models: from Patterns to Governing Equations to Laws of Nature "Overcoming gradient pathologies in constrained neural networks" Paris Perdikaris - University of Penns

From playlist Machine Learning for Physics and the Physics of Learning 2019

Video thumbnail

Sebastian Bubeck: Chasing small sets

I will present an approach based on mirror descent (with a time-varying multiscale entropy functional) to chase small sets in arbitrary metric spaces. This could in particular resolve the randomized competitive ratio of the layered graph traversal problem introduced by Papadimitriou and Ya

From playlist Workshop: Continuous approaches to discrete optimization

Video thumbnail

Gradient Descent, Step-by-Step

Gradient Descent is the workhorse behind most of Machine Learning. When you fit a machine learning method to a training dataset, you're probably using Gradient Descent. It can optimize parameters in a wide variety of settings. Since it's so fundamental to Machine Learning, I decided to mak

From playlist Optimizers in Machine Learning

Video thumbnail

Adaptive Quadrature | Lecture 41 | Vector Calculus for Engineers

What is adaptive quadrature? Join me on Coursera: https://www.coursera.org/learn/numerical-methods-engineers Lecture notes at http://www.math.ust.hk/~machas/numerical-methods-for-engineers.pdf Subscribe to my channel: http://www.youtube.com/user/jchasnov?sub_confirmation=1

From playlist Numerical Methods for Engineers

Video thumbnail

Computational Differential Geometry, Optimization Algorithms by Mark Transtrum

26 December 2016 to 07 January 2017 VENUE: Madhava Lecture Hall, ICTS Bangalore Information theory and computational complexity have emerged as central concepts in the study of biological and physical systems, in both the classical and quantum realm. The low-energy landscape of classical

From playlist US-India Advanced Studies Institute: Classical and Quantum Information

Video thumbnail

Jorge Nocedal: "Tutorial on Optimization Methods for Machine Learning, Pt. 3"

Graduate Summer School 2012: Deep Learning, Feature Learning "Tutorial on Optimization Methods for Machine Learning, Pt. 3" Jorge Nocedal, Northwestern University Institute for Pure and Applied Mathematics, UCLA July 18, 2012 For more information: https://www.ipam.ucla.edu/programs/summ

From playlist GSS2012: Deep Learning, Feature Learning

Video thumbnail

Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates | TDLS

Toronto Deep Learning Series, 27 August 2018 For slides and more information, visit https://tdls.a-i.science/events/2018-08-27/ Paper Review: https://arxiv.org/abs/1611.01505 Speaker: https://www.linkedin.com/in/dave-macdonald-9934154/ Organizer: https://www.linkedin.com/in/amirfz/ Hos

From playlist Math and Foundations

Video thumbnail

Lecture 15 | Convex Optimization I (Stanford)

Professor Stephen Boyd, of the Stanford University Electrical Engineering department, lectures on how unconstrained minimization can be used in electrical engineering and convex optimization for the course, Convex Optimization I (EE 364A). Convex Optimization I concentrates on recognizi

From playlist Lecture Collection | Convex Optimization

Video thumbnail

5 Strange Cases of Animal Rain

Visit http://brilliant.org/scishow/ to get started learning STEM for free, and the first 200 people will get 20% off their annual premium subscription. You might want a really sturdy umbrella to dig into this video, because we’re discussing 5 animals that have a tendency to rain down from

From playlist Biology

Related pages

Mathematical optimization | Rosenbrock function | CMA-ES | Rosenbrock methods | Coordinate descent | Gradient descent | Algorithm | Principal component analysis