A parallel and distributed stochastic gradient descent implementation using commodity clusters
Abstract Deep Learning is an increasingly important subdomain of artificial intelligence, which benefits from training on Big Data. The size and complexity of the model combined with the size of the training dataset makes the training process very computationally and temporally expensive. Accelerati...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2019-02-01
|
| Series: | Journal of Big Data |
| Subjects: | |
| Online Access: | http://link.springer.com/article/10.1186/s40537-019-0179-2 |
| _version_ | 1819265818914979840 |
|---|---|
| author | Robert K. L. Kennedy Taghi M. Khoshgoftaar Flavio Villanustre Timothy Humphrey |
| author_facet | Robert K. L. Kennedy Taghi M. Khoshgoftaar Flavio Villanustre Timothy Humphrey |
| author_sort | Robert K. L. Kennedy |
| collection | DOAJ |
| description | Abstract Deep Learning is an increasingly important subdomain of artificial intelligence, which benefits from training on Big Data. The size and complexity of the model combined with the size of the training dataset makes the training process very computationally and temporally expensive. Accelerating the training process of Deep Learning using cluster computers faces many challenges ranging from distributed optimizers to the large communication overhead specific to systems with off the shelf networking components. In this paper, we present a novel distributed and parallel implementation of stochastic gradient descent (SGD) on a distributed cluster of commodity computers. We use high-performance computing cluster (HPCC) systems as the underlying cluster environment for the implementation. We overview how the HPCC systems platform provides the environment for distributed and parallel Deep Learning, how it provides a facility to work with third party open source libraries such as TensorFlow, and detail our use of third-party libraries and HPCC functionality for implementation. We provide experimental results that validate our work and show that our implementation can scale with respect to both dataset size and the number of compute nodes in the cluster. |
| first_indexed | 2024-12-23T20:51:26Z |
| format | Article |
| id | doaj.art-e0f68ce3a7914c8b8832fe995e7ab6ac |
| institution | Directory Open Access Journal |
| issn | 2196-1115 |
| language | English |
| last_indexed | 2024-12-23T20:51:26Z |
| publishDate | 2019-02-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Journal of Big Data |
| spelling | doaj.art-e0f68ce3a7914c8b8832fe995e7ab6ac2022-12-21T17:31:40ZengSpringerOpenJournal of Big Data2196-11152019-02-016112310.1186/s40537-019-0179-2A parallel and distributed stochastic gradient descent implementation using commodity clustersRobert K. L. Kennedy0Taghi M. Khoshgoftaar1Flavio Villanustre2Timothy Humphrey3Florida Atlantic UniversityFlorida Atlantic UniversityLexisNexis Business Information SolutionsLexisNexis Business Information SolutionsAbstract Deep Learning is an increasingly important subdomain of artificial intelligence, which benefits from training on Big Data. The size and complexity of the model combined with the size of the training dataset makes the training process very computationally and temporally expensive. Accelerating the training process of Deep Learning using cluster computers faces many challenges ranging from distributed optimizers to the large communication overhead specific to systems with off the shelf networking components. In this paper, we present a novel distributed and parallel implementation of stochastic gradient descent (SGD) on a distributed cluster of commodity computers. We use high-performance computing cluster (HPCC) systems as the underlying cluster environment for the implementation. We overview how the HPCC systems platform provides the environment for distributed and parallel Deep Learning, how it provides a facility to work with third party open source libraries such as TensorFlow, and detail our use of third-party libraries and HPCC functionality for implementation. We provide experimental results that validate our work and show that our implementation can scale with respect to both dataset size and the number of compute nodes in the cluster.http://link.springer.com/article/10.1186/s40537-019-0179-2Parallel stochastic gradient descentParallel and distributed processingDeep learningBig dataNeural networkCluster computer |
| spellingShingle | Robert K. L. Kennedy Taghi M. Khoshgoftaar Flavio Villanustre Timothy Humphrey A parallel and distributed stochastic gradient descent implementation using commodity clusters Journal of Big Data Parallel stochastic gradient descent Parallel and distributed processing Deep learning Big data Neural network Cluster computer |
| title | A parallel and distributed stochastic gradient descent implementation using commodity clusters |
| title_full | A parallel and distributed stochastic gradient descent implementation using commodity clusters |
| title_fullStr | A parallel and distributed stochastic gradient descent implementation using commodity clusters |
| title_full_unstemmed | A parallel and distributed stochastic gradient descent implementation using commodity clusters |
| title_short | A parallel and distributed stochastic gradient descent implementation using commodity clusters |
| title_sort | parallel and distributed stochastic gradient descent implementation using commodity clusters |
| topic | Parallel stochastic gradient descent Parallel and distributed processing Deep learning Big data Neural network Cluster computer |
| url | http://link.springer.com/article/10.1186/s40537-019-0179-2 |
| work_keys_str_mv | AT robertklkennedy aparallelanddistributedstochasticgradientdescentimplementationusingcommodityclusters AT taghimkhoshgoftaar aparallelanddistributedstochasticgradientdescentimplementationusingcommodityclusters AT flaviovillanustre aparallelanddistributedstochasticgradientdescentimplementationusingcommodityclusters AT timothyhumphrey aparallelanddistributedstochasticgradientdescentimplementationusingcommodityclusters AT robertklkennedy parallelanddistributedstochasticgradientdescentimplementationusingcommodityclusters AT taghimkhoshgoftaar parallelanddistributedstochasticgradientdescentimplementationusingcommodityclusters AT flaviovillanustre parallelanddistributedstochasticgradientdescentimplementationusingcommodityclusters AT timothyhumphrey parallelanddistributedstochasticgradientdescentimplementationusingcommodityclusters |