Distributed Deep Learning: From Single-Node to Multi-Node Architecture
During the last years, deep learning (DL) models have been used in several applications with large datasets and complex models. These applications require methods to train models faster, such as distributed deep learning (DDL). This paper proposes an empirical approach aiming to measure the speedup...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-05-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/11/10/1525 |
| _version_ | 1797500327264518144 |
|---|---|
| author | Jean-Sébastien Lerat Sidi Ahmed Mahmoudi Saïd Mahmoudi |
| author_facet | Jean-Sébastien Lerat Sidi Ahmed Mahmoudi Saïd Mahmoudi |
| author_sort | Jean-Sébastien Lerat |
| collection | DOAJ |
| description | During the last years, deep learning (DL) models have been used in several applications with large datasets and complex models. These applications require methods to train models faster, such as distributed deep learning (DDL). This paper proposes an empirical approach aiming to measure the speedup of DDL achieved by using different parallelism strategies on the nodes. Local parallelism is considered quite important in the design of a time-performing multi-node architecture because DDL depends on the time required by all the nodes. The impact of computational resources (CPU and GPU) is also discussed since the GPU is known to speed up computations. Experimental results show that the local parallelism impacts the global speedup of the DDL depending on the neural model complexity and the size of the dataset. Moreover, our approach achieves a better speedup than Horovod. |
| first_indexed | 2024-03-10T03:00:19Z |
| format | Article |
| id | doaj.art-bd7c2ddf678040f7b007a5e6cdc781ef |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-10T03:00:19Z |
| publishDate | 2022-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-bd7c2ddf678040f7b007a5e6cdc781ef2023-11-23T10:46:25ZengMDPI AGElectronics2079-92922022-05-011110152510.3390/electronics11101525Distributed Deep Learning: From Single-Node to Multi-Node ArchitectureJean-Sébastien Lerat0Sidi Ahmed Mahmoudi1Saïd Mahmoudi2Science and Technology Department, Haute École en Hainaut, 7000 Mons, BelgiumComputer Science and Management Department, University of Mons, 7000 Mons, BelgiumComputer Science and Management Department, University of Mons, 7000 Mons, BelgiumDuring the last years, deep learning (DL) models have been used in several applications with large datasets and complex models. These applications require methods to train models faster, such as distributed deep learning (DDL). This paper proposes an empirical approach aiming to measure the speedup of DDL achieved by using different parallelism strategies on the nodes. Local parallelism is considered quite important in the design of a time-performing multi-node architecture because DDL depends on the time required by all the nodes. The impact of computational resources (CPU and GPU) is also discussed since the GPU is known to speed up computations. Experimental results show that the local parallelism impacts the global speedup of the DDL depending on the neural model complexity and the size of the dataset. Moreover, our approach achieves a better speedup than Horovod.https://www.mdpi.com/2079-9292/11/10/1525deep learningframeworksCPUGPUdistributed computing |
| spellingShingle | Jean-Sébastien Lerat Sidi Ahmed Mahmoudi Saïd Mahmoudi Distributed Deep Learning: From Single-Node to Multi-Node Architecture Electronics deep learning frameworks CPU GPU distributed computing |
| title | Distributed Deep Learning: From Single-Node to Multi-Node Architecture |
| title_full | Distributed Deep Learning: From Single-Node to Multi-Node Architecture |
| title_fullStr | Distributed Deep Learning: From Single-Node to Multi-Node Architecture |
| title_full_unstemmed | Distributed Deep Learning: From Single-Node to Multi-Node Architecture |
| title_short | Distributed Deep Learning: From Single-Node to Multi-Node Architecture |
| title_sort | distributed deep learning from single node to multi node architecture |
| topic | deep learning frameworks CPU GPU distributed computing |
| url | https://www.mdpi.com/2079-9292/11/10/1525 |
| work_keys_str_mv | AT jeansebastienlerat distributeddeeplearningfromsinglenodetomultinodearchitecture AT sidiahmedmahmoudi distributeddeeplearningfromsinglenodetomultinodearchitecture AT saidmahmoudi distributeddeeplearningfromsinglenodetomultinodearchitecture |