Low-Rank Gradient Descent by Romain Cosson, Ali Jadbabaie, Anuran Makur, Amirhossein Reisizadeh, Devavrat Shah

“…In this article, we leverage such low-rank structure to reduce the high computational cost of canonical gradient-based methods such as gradient descent (<monospace>GD</monospace>). Our proposed <italic>Low-Rank Gradient Descent</italic> (<monospace>LRGD</monospace>) algorithm finds an <inline-formula><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula>-approximate stationary point of a <inline-formula><tex-math notation="LaTeX">$p$</tex-math></inline-formula>-dimensional function by first identifying <inline-formula><tex-math notation="LaTeX">$r \leq p$</tex-math></inline-formula> significant directions, and then estimating the true <inline-formula><tex-math notation="LaTeX">$p$</tex-math></inline-formula>-dimensional gradient at every iteration by computing directional derivatives only along those <inline-formula><tex-math notation="LaTeX">$r$</tex-math></inline-formula> directions. …”
Get full text

On-manifold projected gradient descent by Aaron Mahler, Tyrus Berry, Tom Stephens, Harbir Antil, Michael Merritt, Jeanie Schreiber, Ioannis Kevrekidis

“…The tools are applied to the setting of neural network image classifiers, where we generate novel, on-manifold data samples and implement a projected gradient descent algorithm for on-manifold adversarial training. …”
Get full text

Correspondence between neuroevolution and gradient descent by Stephen Whitelam, Viktor Selin, Sang-Won Park, Isaac Tamblyn

“…The authors derive, and illustrate numerically, an analytic equivalence between the dynamics of neural network training under conditioned stochastic mutations, and under gradient descent.…”
Get full text

Semi-Stochastic Gradient Descent Methods by Jakub Konečný, Peter Richtárik

“…We propose a new method, S2GD (Semi-Stochastic Gradient Descent), which runs for one or several epochs in each of which a single full gradient and a random number of stochastic gradients is computed, following a geometric law. …”
Get full text

Gradient-Descent-like Ghost Imaging by Wen-Kai Yu, Chen-Xi Zhu, Ya-Xin Li, Shuo-Fei Wang, Chong Cao

Subjects: Get full text

Carathéodory sampling for stochastic gradient descent by Cosentino, F, Oberhauser, H, Abate, A

“…Many problems require to optimize empirical risk functions over large data sets. Gradient descent methods that calculate the full gradient in every descent step do not scale to such datasets. …”

Carathéodory sampling for stochastic gradient descent by Cosentino, F, Oberhauser, H, Abate, A

“…Many problems require to optimize empirical risk functions over large data sets. Gradient descent methods that calculate the full gradient in every descent step do not scale to such datasets. …”

Dual space preconditioning for gradient descent by Maddison, CJ, Paulin, D, Teh, YW, Doucet, A

“…Thus, in principle our method is capable of improving the conditioning of gradient descent on problems with a non-Lipschitz gradient or nonstrongly convex structure. …”

Pipelined Stochastic Gradient Descent with Taylor Expansion by Bongwon Jang, Inchul Yoo, Dongsuk Yook

Subjects: Get full text

Limited Gradient Descent: Learning With Noisy Labels by Yi Sun, Yan Tian, Yiping Xu, Jianxiang Li

“…To solve this problem, we propose a method that can estimate the optimal stopping timing without a clean validation set, called limited gradient descent. We modified the labels of a few samples in a noisy dataset to obtain false labels and to create a reverse pattern. …”
Get full text

Accelerated Gradient Descent Driven by Lévy Perturbations by Yuquan Chen, Zhenlong Wu, Yixiang Lu, Yangquan Chen, Yong Wang

Subjects: “…accelerated gradient descent…”
Get full text

Gradient Descent Batch Clustering for Image Classification by Jae-Sam Park

Subjects: Get full text

Stochastic gradient descent for optimization for nuclear systems by Austin Williams, Noah Walton, Austin Maryanski, Sandra Bogetic, Wes Hines, Vladimir Sobes

“…ADAM is a gradient descent method that accounts for gradients with a stochastic nature. …”
Get full text

Stochastic gradient descent for wind farm optimization by J. Quick, P.-E. Rethore, M. Mølgaard Pedersen, R. V. Rodrigues, M. Friis-Møller

“…This study presents stochastic gradient descent (SGD) for wind farm optimization, which is an approach that estimates the gradient of the AEP using Monte Carlo simulation, allowing for the consideration of an arbitrarily large number of atmospheric conditions. …”
Get full text

NOMA Codebook Optimization by Batch Gradient Descent by Zhongwei Si, Shaoguo Wen, Bing Dong

Subjects: “…Batch gradient descent…”
Get full text

Complexity control by gradient descent in deep networks by Tomaso Poggio, Qianli Liao, Andrzej Banburski

“…Here, the author demonstrates an implicit regularization in training deep networks, showing that the control of complexity in the training is hidden within the optimization technique of gradient descent.…”
Get full text

Gradient descent optimization in gene regulatory pathways. by Mouli Das, Subhasis Mukhopadhyay, Rajat K De

Get full text

Complexity control by gradient descent in deep networks by Poggio, Tomaso, Liao, Qianli, Banburski, Andrzej

“…For exponential-type loss functions, we solve this puzzle by showing an effective regularization effect of gradient descent in terms of the normalized weights that are relevant for classification.…”
Get full text

Complexity control by gradient descent in deep networks by Poggio, Tomaso A, Liao, Qianli, Banburski, Andrzej

“…For exponential-type loss functions, we solve this puzzle by showing an effective regularization effect of gradient descent in terms of the normalized weights that are relevant for classification.…”
Get full text

The complexity of gradient descent: CLS = PPAD ∩ PLS by Fearnley, J, Goldberg, P, Hollender, A, Savani, R