Optimizing Graph Neural Network Training on Large Graphs
Graphs can be used to represent many important classes of structured real-world data. For this reason, there has been an increase of research interest in various machine learning approaches to solve tasks such as link prediction and node property prediction. Graph Neural Network models demonstrate g...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2022
|
| Online Access: | https://hdl.handle.net/1721.1/138975 |
| _version_ | 1826189074272092160 |
|---|---|
| author | Stathas, Nickolas |
| author2 | Leiserson, Charles E. |
| author_facet | Leiserson, Charles E. Stathas, Nickolas |
| author_sort | Stathas, Nickolas |
| collection | MIT |
| description | Graphs can be used to represent many important classes of structured real-world data. For this reason, there has been an increase of research interest in various machine learning approaches to solve tasks such as link prediction and node property prediction. Graph Neural Network models demonstrate good performance on such tasks. However, the depth of the models and the size of the graphs they can be trained on is constrained either by the low processing throughput of CPUs or by the limited memory capacity of GPUs. Techniques such as neighborhood sampling are often used to create smaller mini-batch training examples that fit in GPU memory. In this thesis, I provide a systematic performance analysis of GNN training codes written using PyTorch Geometric, the most popular machine learning framework for GNNs. Through this performance analysis, I uncover significant performance bottlenecks related to neighborhood sampling and GPU data transfers. To address these issues, I create FastPyG: a performance-engineered fork of PyTorch Geometric, which achieves a 3-6× speedup over comparable PyTorch Geometric codes without impacting model accuracy. The core contribution included in FastPyG is fast_sampler, an efficient and parallel neighborhood sampling implementation in C++. |
| first_indexed | 2024-09-23T08:09:08Z |
| format | Thesis |
| id | mit-1721.1/138975 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T08:09:08Z |
| publishDate | 2022 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1389752022-01-15T04:09:09Z Optimizing Graph Neural Network Training on Large Graphs Stathas, Nickolas Leiserson, Charles E. Kaler, Tim Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Graphs can be used to represent many important classes of structured real-world data. For this reason, there has been an increase of research interest in various machine learning approaches to solve tasks such as link prediction and node property prediction. Graph Neural Network models demonstrate good performance on such tasks. However, the depth of the models and the size of the graphs they can be trained on is constrained either by the low processing throughput of CPUs or by the limited memory capacity of GPUs. Techniques such as neighborhood sampling are often used to create smaller mini-batch training examples that fit in GPU memory. In this thesis, I provide a systematic performance analysis of GNN training codes written using PyTorch Geometric, the most popular machine learning framework for GNNs. Through this performance analysis, I uncover significant performance bottlenecks related to neighborhood sampling and GPU data transfers. To address these issues, I create FastPyG: a performance-engineered fork of PyTorch Geometric, which achieves a 3-6× speedup over comparable PyTorch Geometric codes without impacting model accuracy. The core contribution included in FastPyG is fast_sampler, an efficient and parallel neighborhood sampling implementation in C++. M.Eng. 2022-01-14T14:42:12Z 2022-01-14T14:42:12Z 2021-06 2021-06-17T20:14:27.540Z Thesis https://hdl.handle.net/1721.1/138975 In Copyright - Educational Use Permitted Copyright MIT http://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Stathas, Nickolas Optimizing Graph Neural Network Training on Large Graphs |
| title | Optimizing Graph Neural Network Training on Large Graphs |
| title_full | Optimizing Graph Neural Network Training on Large Graphs |
| title_fullStr | Optimizing Graph Neural Network Training on Large Graphs |
| title_full_unstemmed | Optimizing Graph Neural Network Training on Large Graphs |
| title_short | Optimizing Graph Neural Network Training on Large Graphs |
| title_sort | optimizing graph neural network training on large graphs |
| url | https://hdl.handle.net/1721.1/138975 |
| work_keys_str_mv | AT stathasnickolas optimizinggraphneuralnetworktrainingonlargegraphs |