ConnectIt: a framework for static and incremental parallel graph connectivity algorithms
© VLDB Endowment. All rights reserved. Connected components is a fundamental kernel in graph applications. The fastest existing multicore algorithms for solving graph connectivity are based on some form of edge sampling and/or linking and compressing trees. However, many combinations of these design...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
VLDB Endowment
2021
|
| Online Access: | https://hdl.handle.net/1721.1/133950 |
| _version_ | 1811086609421959168 |
|---|---|
| author | Dhulipala, Laxman Hong, Changwan Shun, Julian |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Dhulipala, Laxman Hong, Changwan Shun, Julian |
| author_sort | Dhulipala, Laxman |
| collection | MIT |
| description | © VLDB Endowment. All rights reserved. Connected components is a fundamental kernel in graph applications. The fastest existing multicore algorithms for solving graph connectivity are based on some form of edge sampling and/or linking and compressing trees. However, many combinations of these design choices have been left unexplored. In this paper, we design the ConnectIt framework, which provides different sampling strategies as well as various tree linking and compression schemes. ConnectIt enables us to obtain several hundred new variants of connectivity algorithms, most of which extend to computing spanning forest. In addition to static graphs, we also extend ConnectIt to support mixes of insertions and connectivity queries in the concurrent setting. We present an experimental evaluation of ConnectIt on a 72-core machine, which we believe is the most comprehensive evaluation of parallel connectivity algorithms to date. Compared to a collection of state-of-the-art static multicore algorithms, we obtain an average speedup of 12.4x (2.36x average speedup over the fastest existing implementation for each graph). Using ConnectIt, we are able to compute connectivity on the largest publicly-available graph (with over 3.5 billion vertices and 128 billion edges) in under 10 seconds using a 72-core machine, providing a 3.1x speedup over the fastest existing connectivity result for this graph, in any computational setting. For our incremental algorithms, we show that our algorithms can ingest graph updates at up to several billion edges per second. To guide the user in selecting the best variants in ConnectIt for different situations, we provide a detailed analysis of the different strategies. Finally, we show how the techniques in ConnectIt can be used to speed up two important graph applications: approximate minimum spanning forest and SCAN clustering. |
| first_indexed | 2024-09-23T13:28:58Z |
| format | Article |
| id | mit-1721.1/133950 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T13:28:58Z |
| publishDate | 2021 |
| publisher | VLDB Endowment |
| record_format | dspace |
| spelling | mit-1721.1/1339502023-08-10T20:45:57Z ConnectIt: a framework for static and incremental parallel graph connectivity algorithms Dhulipala, Laxman Hong, Changwan Shun, Julian Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory © VLDB Endowment. All rights reserved. Connected components is a fundamental kernel in graph applications. The fastest existing multicore algorithms for solving graph connectivity are based on some form of edge sampling and/or linking and compressing trees. However, many combinations of these design choices have been left unexplored. In this paper, we design the ConnectIt framework, which provides different sampling strategies as well as various tree linking and compression schemes. ConnectIt enables us to obtain several hundred new variants of connectivity algorithms, most of which extend to computing spanning forest. In addition to static graphs, we also extend ConnectIt to support mixes of insertions and connectivity queries in the concurrent setting. We present an experimental evaluation of ConnectIt on a 72-core machine, which we believe is the most comprehensive evaluation of parallel connectivity algorithms to date. Compared to a collection of state-of-the-art static multicore algorithms, we obtain an average speedup of 12.4x (2.36x average speedup over the fastest existing implementation for each graph). Using ConnectIt, we are able to compute connectivity on the largest publicly-available graph (with over 3.5 billion vertices and 128 billion edges) in under 10 seconds using a 72-core machine, providing a 3.1x speedup over the fastest existing connectivity result for this graph, in any computational setting. For our incremental algorithms, we show that our algorithms can ingest graph updates at up to several billion edges per second. To guide the user in selecting the best variants in ConnectIt for different situations, we provide a detailed analysis of the different strategies. Finally, we show how the techniques in ConnectIt can be used to speed up two important graph applications: approximate minimum spanning forest and SCAN clustering. 2021-10-27T19:57:20Z 2021-10-27T19:57:20Z 2020 2021-09-24T13:50:26Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/133950 en 10.14778/3436905.3436923 Proceedings of the VLDB Endowment Creative Commons Attribution-NonCommercial-NoDerivs License http://creativecommons.org/licenses/by-nc-nd/4.0/ application/pdf VLDB Endowment VLDB Endowment |
| spellingShingle | Dhulipala, Laxman Hong, Changwan Shun, Julian ConnectIt: a framework for static and incremental parallel graph connectivity algorithms |
| title | ConnectIt: a framework for static and incremental parallel graph connectivity algorithms |
| title_full | ConnectIt: a framework for static and incremental parallel graph connectivity algorithms |
| title_fullStr | ConnectIt: a framework for static and incremental parallel graph connectivity algorithms |
| title_full_unstemmed | ConnectIt: a framework for static and incremental parallel graph connectivity algorithms |
| title_short | ConnectIt: a framework for static and incremental parallel graph connectivity algorithms |
| title_sort | connectit a framework for static and incremental parallel graph connectivity algorithms |
| url | https://hdl.handle.net/1721.1/133950 |
| work_keys_str_mv | AT dhulipalalaxman connectitaframeworkforstaticandincrementalparallelgraphconnectivityalgorithms AT hongchangwan connectitaframeworkforstaticandincrementalparallelgraphconnectivityalgorithms AT shunjulian connectitaframeworkforstaticandincrementalparallelgraphconnectivityalgorithms |