Counterexamples for Noise Models of Stochastic Gradients
Stochastic Gradient Descent (SGD) is a widely used, foundational algorithm in data science and machine learning. As a result, analyses of SGD abound making use of a variety of assumptions, especially on the noise behavior of the stochastic gradients. While recent works have achieved a high-degree of...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-12-01
|
| Series: | Examples and Counterexamples |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666657X23000253 |
| _version_ | 1797398977319010304 |
|---|---|
| author | Vivak Patel |
| author_facet | Vivak Patel |
| author_sort | Vivak Patel |
| collection | DOAJ |
| description | Stochastic Gradient Descent (SGD) is a widely used, foundational algorithm in data science and machine learning. As a result, analyses of SGD abound making use of a variety of assumptions, especially on the noise behavior of the stochastic gradients. While recent works have achieved a high-degree of generality on assumptions about the noise behavior of the stochastic gradients, it is unclear that such generality is necessary. In this work, we construct a simple example that shows that less general assumptions will be violated, while the most general assumptions will hold. |
| first_indexed | 2024-03-09T01:33:19Z |
| format | Article |
| id | doaj.art-87c63256cfa94959828c72045802d0b1 |
| institution | Directory Open Access Journal |
| issn | 2666-657X |
| language | English |
| last_indexed | 2024-03-09T01:33:19Z |
| publishDate | 2023-12-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Examples and Counterexamples |
| spelling | doaj.art-87c63256cfa94959828c72045802d0b12023-12-09T06:08:13ZengElsevierExamples and Counterexamples2666-657X2023-12-014100123Counterexamples for Noise Models of Stochastic GradientsVivak Patel0Department of Statistics, University of Wisconsin – Madison, 1300 University Ave, Madison, 53703, WI, USAStochastic Gradient Descent (SGD) is a widely used, foundational algorithm in data science and machine learning. As a result, analyses of SGD abound making use of a variety of assumptions, especially on the noise behavior of the stochastic gradients. While recent works have achieved a high-degree of generality on assumptions about the noise behavior of the stochastic gradients, it is unclear that such generality is necessary. In this work, we construct a simple example that shows that less general assumptions will be violated, while the most general assumptions will hold.http://www.sciencedirect.com/science/article/pii/S2666657X23000253Stochastic Gradient DescentNoise Models |
| spellingShingle | Vivak Patel Counterexamples for Noise Models of Stochastic Gradients Examples and Counterexamples Stochastic Gradient Descent Noise Models |
| title | Counterexamples for Noise Models of Stochastic Gradients |
| title_full | Counterexamples for Noise Models of Stochastic Gradients |
| title_fullStr | Counterexamples for Noise Models of Stochastic Gradients |
| title_full_unstemmed | Counterexamples for Noise Models of Stochastic Gradients |
| title_short | Counterexamples for Noise Models of Stochastic Gradients |
| title_sort | counterexamples for noise models of stochastic gradients |
| topic | Stochastic Gradient Descent Noise Models |
| url | http://www.sciencedirect.com/science/article/pii/S2666657X23000253 |
| work_keys_str_mv | AT vivakpatel counterexamplesfornoisemodelsofstochasticgradients |