Theory of Convex Functions, Gradient Descent, Projected Gradient Descent, Newton’s Method, Quasi-Newton Methods, Coordinate Descent.

The Frank-Wolfe Algorithm, Subgradient Methods, Mirror Descent, Smoothing, Proximal Algorithms, Stochastic Optimization, Finite Sum Optimization, Min-Max Optimization, Accelerated Gradient, Gradient-free, adaptive methods

The overview of modern mathematical optimization methods, for applications in machine learning and data science.

Develop a strong theoretical foundation in convex functions and optimization principles.

Explore and analyze first order and second-order optimization methods, including gradient-based and Newton-type methods.

Understand and apply constrained optimization techniques such as Projected Gradient Descent and the Frank-Wolfe algorithm.

Investigate advanced and modern optimization algorithms including subgradient methods, mirror descent, and proximal techniques.

Mathematically define and identify convex sets, convex functions, and formulate convex optimization problems.

Apply and compare the performance of Gradient Descent, Newton's Method, and Quasi-Newton Methods for smooth optimization problems

Solve constrained optimization problems using algorithms such as Projected Gradient Descent, Frank-Wolfe, and Coordinate Descent.

Implement advanced algorithms like Mirror Descent, Proximal Methods, and understand their convergence properties.

Design and apply optimization algorithms in stochastic and non-smooth settings, including adaptive and gradient-free methods.

Boyd, Stephen and Lieven Vandenberghe, Convex optimization, Cambridge University Press , 2004

Nocedal, Jorge and Stephen Wright, Numerical optimization, Springer Science & Business Media , 2006

SIAM Journal on Numerical Analysis, Optimization and Control with Applications.