Understanding Communication Characteristics of Distributed Training
APNet 2024, August 03–04, 2024, Sydney, Australia
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
ACM|The 8th Asia-Pacific Workshop on Networking
2024
|
| Online Access: | https://hdl.handle.net/1721.1/156676 |
| _version_ | 1811073339567898624 |
|---|---|
| author | Li, Wenxue Liu, Xiangzhou Li, Yuxuan Jin, Yilun Tian, Han Zhong, Zhizhen Liu, Guyue Zhang, Ying Chen, Kai |
| author_facet | Li, Wenxue Liu, Xiangzhou Li, Yuxuan Jin, Yilun Tian, Han Zhong, Zhizhen Liu, Guyue Zhang, Ying Chen, Kai |
| author_sort | Li, Wenxue |
| collection | MIT |
| description | APNet 2024, August 03–04, 2024, Sydney, Australia |
| first_indexed | 2024-09-23T09:31:31Z |
| format | Article |
| id | mit-1721.1/156676 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T09:31:31Z |
| publishDate | 2024 |
| publisher | ACM|The 8th Asia-Pacific Workshop on Networking |
| record_format | dspace |
| spelling | mit-1721.1/1566762024-09-06T03:58:41Z Understanding Communication Characteristics of Distributed Training Li, Wenxue Liu, Xiangzhou Li, Yuxuan Jin, Yilun Tian, Han Zhong, Zhizhen Liu, Guyue Zhang, Ying Chen, Kai APNet 2024, August 03–04, 2024, Sydney, Australia Communication is pivotal in distributed training and a thorough understanding of its characteristics is essential for future optimizations. However, prior works are limited, either focusing on customized optimizations or conducting incomplete explorations on communication characteristics. In this work, we systematically analyze the communication characteristics of distributed training, considering two key aspects of communication: pattern and overhead, and assessing a broad spectrum of determinant factors. In particular, we extensively investigate the features of communication patterns, such as predictability, and comprehensively evaluate the impact of various factors on communication overhead. Additionally, we develop and validate an analytical formulation to estimate communication overhead, providing a mathematical understanding of models with predictability. 2024-09-05T13:55:09Z 2024-09-05T13:55:09Z 2024-08-03 2024-09-01T07:47:41Z Article http://purl.org/eprint/type/ConferencePaper 979-8-4007-1758-1 https://hdl.handle.net/1721.1/156676 Wenxue Li, Xiangzhou Liu, Yuxuan Li, Yilun Jin, Han Tian, Zhizhen Zhong, Guyue Liu, Ying Zhang, and Kai Chen. 2024. Understanding Communication Characteristics of Distributed Training. In Proceedings of the 8th Asia-Pacific Workshop on Networking (APNet '24). Association for Computing Machinery, New York, NY, USA, 1–8. PUBLISHER_POLICY en https://doi.org/10.1145/3663408.3663409 Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. The author(s) application/pdf ACM|The 8th Asia-Pacific Workshop on Networking Association for Computing Machinery |
| spellingShingle | Li, Wenxue Liu, Xiangzhou Li, Yuxuan Jin, Yilun Tian, Han Zhong, Zhizhen Liu, Guyue Zhang, Ying Chen, Kai Understanding Communication Characteristics of Distributed Training |
| title | Understanding Communication Characteristics of Distributed Training |
| title_full | Understanding Communication Characteristics of Distributed Training |
| title_fullStr | Understanding Communication Characteristics of Distributed Training |
| title_full_unstemmed | Understanding Communication Characteristics of Distributed Training |
| title_short | Understanding Communication Characteristics of Distributed Training |
| title_sort | understanding communication characteristics of distributed training |
| url | https://hdl.handle.net/1721.1/156676 |
| work_keys_str_mv | AT liwenxue understandingcommunicationcharacteristicsofdistributedtraining AT liuxiangzhou understandingcommunicationcharacteristicsofdistributedtraining AT liyuxuan understandingcommunicationcharacteristicsofdistributedtraining AT jinyilun understandingcommunicationcharacteristicsofdistributedtraining AT tianhan understandingcommunicationcharacteristicsofdistributedtraining AT zhongzhizhen understandingcommunicationcharacteristicsofdistributedtraining AT liuguyue understandingcommunicationcharacteristicsofdistributedtraining AT zhangying understandingcommunicationcharacteristicsofdistributedtraining AT chenkai understandingcommunicationcharacteristicsofdistributedtraining |