Controlled gradient descent: A control theoretical perspective for optimization
The Gradient Descent (GD) paradigm is a foundational principle of modern optimization algorithms. The GD algorithm and its variants, including accelerated optimization algorithms, geodesic optimization, natural gradient, and contraction-based optimization, to name a few, are used in machine learning...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2024-06-01
|
| Series: | Results in Control and Optimization |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S266672072400047X |
| _version_ | 1797242936766758912 |
|---|---|
| author | Revati Gunjal Syed Shadab Nayyer S.R. Wagh N.M. Singh |
| author_facet | Revati Gunjal Syed Shadab Nayyer S.R. Wagh N.M. Singh |
| author_sort | Revati Gunjal |
| collection | DOAJ |
| description | The Gradient Descent (GD) paradigm is a foundational principle of modern optimization algorithms. The GD algorithm and its variants, including accelerated optimization algorithms, geodesic optimization, natural gradient, and contraction-based optimization, to name a few, are used in machine learning and the system and control domain. Here, we proposed a new algorithm based on the control theoretical perspective, labeled as the Controlled Gradient Descent (CGD). Specifically, this approach overcomes the challenges of the abovementioned algorithms, which rely on the choice of a suitable geometric structure, particularly in machine learning. The proposed CGD approach visualizes the optimization as a Manifold Stabilization Problem (MSP) through the notion of an invariant manifold and its attractivity. The CGD approach leads to an exponential contraction of trajectories under the influence of a pseudo-Riemannian metric generated through the control procedure as an additional outcome. The efficacy of the CGD is demonstrated with various test objective functions like the benchmark Rosenbrock function, objective function with a lack of flatness, and semi-contracting objective functions often encountered in machine learning applications. |
| first_indexed | 2024-04-24T18:47:09Z |
| format | Article |
| id | doaj.art-28805938f551445ebb3089ef40e510c2 |
| institution | Directory Open Access Journal |
| issn | 2666-7207 |
| language | English |
| last_indexed | 2024-04-24T18:47:09Z |
| publishDate | 2024-06-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Results in Control and Optimization |
| spelling | doaj.art-28805938f551445ebb3089ef40e510c22024-03-27T04:53:13ZengElsevierResults in Control and Optimization2666-72072024-06-0115100417Controlled gradient descent: A control theoretical perspective for optimizationRevati Gunjal0Syed Shadab Nayyer1S.R. Wagh2N.M. Singh3Corresponding author.; Control and Decision Research Centre (CDRC), Electrical Engineering Department (EED), Veermata Jijabai Technological Institute (VJTI), Mumbai, IndiaControl and Decision Research Centre (CDRC), Electrical Engineering Department (EED), Veermata Jijabai Technological Institute (VJTI), Mumbai, IndiaControl and Decision Research Centre (CDRC), Electrical Engineering Department (EED), Veermata Jijabai Technological Institute (VJTI), Mumbai, IndiaControl and Decision Research Centre (CDRC), Electrical Engineering Department (EED), Veermata Jijabai Technological Institute (VJTI), Mumbai, IndiaThe Gradient Descent (GD) paradigm is a foundational principle of modern optimization algorithms. The GD algorithm and its variants, including accelerated optimization algorithms, geodesic optimization, natural gradient, and contraction-based optimization, to name a few, are used in machine learning and the system and control domain. Here, we proposed a new algorithm based on the control theoretical perspective, labeled as the Controlled Gradient Descent (CGD). Specifically, this approach overcomes the challenges of the abovementioned algorithms, which rely on the choice of a suitable geometric structure, particularly in machine learning. The proposed CGD approach visualizes the optimization as a Manifold Stabilization Problem (MSP) through the notion of an invariant manifold and its attractivity. The CGD approach leads to an exponential contraction of trajectories under the influence of a pseudo-Riemannian metric generated through the control procedure as an additional outcome. The efficacy of the CGD is demonstrated with various test objective functions like the benchmark Rosenbrock function, objective function with a lack of flatness, and semi-contracting objective functions often encountered in machine learning applications.http://www.sciencedirect.com/science/article/pii/S266672072400047XGradient descent (GD)Manifold StabilizationOptimizationOverparameterized NetworksPassivity and Immersion (P&I) |
| spellingShingle | Revati Gunjal Syed Shadab Nayyer S.R. Wagh N.M. Singh Controlled gradient descent: A control theoretical perspective for optimization Results in Control and Optimization Gradient descent (GD) Manifold Stabilization Optimization Overparameterized Networks Passivity and Immersion (P&I) |
| title | Controlled gradient descent: A control theoretical perspective for optimization |
| title_full | Controlled gradient descent: A control theoretical perspective for optimization |
| title_fullStr | Controlled gradient descent: A control theoretical perspective for optimization |
| title_full_unstemmed | Controlled gradient descent: A control theoretical perspective for optimization |
| title_short | Controlled gradient descent: A control theoretical perspective for optimization |
| title_sort | controlled gradient descent a control theoretical perspective for optimization |
| topic | Gradient descent (GD) Manifold Stabilization Optimization Overparameterized Networks Passivity and Immersion (P&I) |
| url | http://www.sciencedirect.com/science/article/pii/S266672072400047X |
| work_keys_str_mv | AT revatigunjal controlledgradientdescentacontroltheoreticalperspectiveforoptimization AT syedshadabnayyer controlledgradientdescentacontroltheoreticalperspectiveforoptimization AT srwagh controlledgradientdescentacontroltheoreticalperspectiveforoptimization AT nmsingh controlledgradientdescentacontroltheoreticalperspectiveforoptimization |