From big data to smart data: a sample gradient descent approach for machine learning
Abstract This research paper presents an innovative approach to gradient descent known as ‘‘Sample Gradient Descent’’. This method is a modification of the conventional batch gradient descent algorithm, which is often associated with space and time complexity issues. The proposed approach involves t...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2023-10-01
|
| Series: | Journal of Big Data |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s40537-023-00839-9 |
| _version_ | 1827634006825369600 |
|---|---|
| author | Aadil Gani Ganie Samad Dadvandipour |
| author_facet | Aadil Gani Ganie Samad Dadvandipour |
| author_sort | Aadil Gani Ganie |
| collection | DOAJ |
| description | Abstract This research paper presents an innovative approach to gradient descent known as ‘‘Sample Gradient Descent’’. This method is a modification of the conventional batch gradient descent algorithm, which is often associated with space and time complexity issues. The proposed approach involves the selection of a representative sample of data, which is subsequently subjected to batch gradient descent. The selection of this sample is a crucial task, as it must accurately represent the entire dataset. To achieve this, the study employs the use of Principle Component Analysis (PCA), which is applied to the training data, with a condition that only those rows and columns of data that explain 90% of the overall variance are retained. This approach results in a convex loss function, where a global minimum can be readily attained. Our results indicate that the proposed method offers faster convergence rates, with reduced computation times, when compared to the conventional batch gradient descent algorithm. These findings demonstrate the potential utility of the ‘‘Sample Gradient Descent’’ technique in various domains, ranging from machine learning to optimization problems. In our experiments, both approaches were run for 30 epochs, with each epoch taking approximately 3.41 s. Notably, our ‘‘Sample Gradient Descent’’ approach exhibited remarkable performance, converging in just 8 epochs, while the conventional batch gradient descent algorithm required 20 epochs to achieve convergence. This substantial difference in convergence rates, along with reduced computation times, highlights the superior efficiency of our proposed method. These findings underscore the potential utility of the ‘‘Sample Gradient Descent’’ technique across diverse domains, ranging from machine learning to optimization problems. The significant improvements in convergence rates and computation times make our algorithm particularly appealing to practitioners and researchers seeking enhanced efficiency in gradient descent optimization. |
| first_indexed | 2024-03-09T15:07:27Z |
| format | Article |
| id | doaj.art-f41c807e3987460d836a049781eb74f2 |
| institution | Directory Open Access Journal |
| issn | 2196-1115 |
| language | English |
| last_indexed | 2024-03-09T15:07:27Z |
| publishDate | 2023-10-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Journal of Big Data |
| spelling | doaj.art-f41c807e3987460d836a049781eb74f22023-11-26T13:35:25ZengSpringerOpenJournal of Big Data2196-11152023-10-0110111310.1186/s40537-023-00839-9From big data to smart data: a sample gradient descent approach for machine learningAadil Gani Ganie0Samad Dadvandipour1University of MiskolcUniversity of MiskolcAbstract This research paper presents an innovative approach to gradient descent known as ‘‘Sample Gradient Descent’’. This method is a modification of the conventional batch gradient descent algorithm, which is often associated with space and time complexity issues. The proposed approach involves the selection of a representative sample of data, which is subsequently subjected to batch gradient descent. The selection of this sample is a crucial task, as it must accurately represent the entire dataset. To achieve this, the study employs the use of Principle Component Analysis (PCA), which is applied to the training data, with a condition that only those rows and columns of data that explain 90% of the overall variance are retained. This approach results in a convex loss function, where a global minimum can be readily attained. Our results indicate that the proposed method offers faster convergence rates, with reduced computation times, when compared to the conventional batch gradient descent algorithm. These findings demonstrate the potential utility of the ‘‘Sample Gradient Descent’’ technique in various domains, ranging from machine learning to optimization problems. In our experiments, both approaches were run for 30 epochs, with each epoch taking approximately 3.41 s. Notably, our ‘‘Sample Gradient Descent’’ approach exhibited remarkable performance, converging in just 8 epochs, while the conventional batch gradient descent algorithm required 20 epochs to achieve convergence. This substantial difference in convergence rates, along with reduced computation times, highlights the superior efficiency of our proposed method. These findings underscore the potential utility of the ‘‘Sample Gradient Descent’’ technique across diverse domains, ranging from machine learning to optimization problems. The significant improvements in convergence rates and computation times make our algorithm particularly appealing to practitioners and researchers seeking enhanced efficiency in gradient descent optimization.https://doi.org/10.1186/s40537-023-00839-9Big dataGradient decentMachine learningPCALoss function |
| spellingShingle | Aadil Gani Ganie Samad Dadvandipour From big data to smart data: a sample gradient descent approach for machine learning Journal of Big Data Big data Gradient decent Machine learning PCA Loss function |
| title | From big data to smart data: a sample gradient descent approach for machine learning |
| title_full | From big data to smart data: a sample gradient descent approach for machine learning |
| title_fullStr | From big data to smart data: a sample gradient descent approach for machine learning |
| title_full_unstemmed | From big data to smart data: a sample gradient descent approach for machine learning |
| title_short | From big data to smart data: a sample gradient descent approach for machine learning |
| title_sort | from big data to smart data a sample gradient descent approach for machine learning |
| topic | Big data Gradient decent Machine learning PCA Loss function |
| url | https://doi.org/10.1186/s40537-023-00839-9 |
| work_keys_str_mv | AT aadilganiganie frombigdatatosmartdataasamplegradientdescentapproachformachinelearning AT samaddadvandipour frombigdatatosmartdataasamplegradientdescentapproachformachinelearning |