Optimization for Machine Learning and Data Science
Published:
This page gathers my projects and lecture notes on optimization, with a particular focus on concepts and algorithms that are essential in machine learning, data science, and operations research.
Optimization is one of the core mathematical pillars behind modern data-driven methods. From fitting regression models to training deep neural networks, optimization provides the framework for turning mathematical objectives into practical algorithms.
What this collection covers
This collection is being developed around the following themes:
- convex analysis,
- first-order optimization methods,
- second-order optimization methods,
- constrained optimization,
- numerical optimization algorithms,
- practical implementations for machine learning and operations research.
Main topics
Convex analysis
This section will cover the mathematical foundations needed to understand modern optimization, including:
- convex sets,
- convex functions,
- gradients and subgradients,
- optimality conditions,
- duality and geometric intuition.
Optimization algorithms
This section focuses on algorithmic methods used in practice.
Gradient methods
Topics in this section include:
- gradient descent,
- step-size selection,
- convergence intuition,
- variants used in large-scale learning.
Newton methods
Topics in this section include:
- Newton’s method,
- second-order approximations,
- Hessian-based updates,
- practical considerations in implementation.
Algorithms of interest
In this project, I plan to discuss a range of optimization methods commonly used in machine learning and operations research, including:
- Simplex Method,
- Gradient Descent,
- Newton-type methods,
- Alternating Direction Method of Multipliers (ADMM),
- and other foundational algorithms.
The aim is to connect theory, intuition, and implementation in a way that is both rigorous and practical.
Implementation
The implementation for this project is available here:
Project repository: Convex Optimization
Ongoing development
This page will continue to grow as I add:
- polished lecture notes,
- worked examples,
- derivations,
- coding tutorials,
- and project-based demonstrations of optimization methods in practice.