Visualisation and management tool for supercomputer resource
The machine learning, especially deep learning has gained an astonishing amount of popularity over the last few years. Because of the success of machine learning in many other fields. The interest of applying machine learning to the job scheduling of high performance supercomputer has been raised. T...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Final Year Project (FYP) |
| Language: | English |
| Published: |
2018
|
| Subjects: | |
| Online Access: | http://hdl.handle.net/10356/73967 |
| _version_ | 1826121753502416896 |
|---|---|
| author | Shao, Yiyang |
| author2 | Lee Bu Sung, Francis |
| author_facet | Lee Bu Sung, Francis Shao, Yiyang |
| author_sort | Shao, Yiyang |
| collection | NTU |
| description | The machine learning, especially deep learning has gained an astonishing amount of popularity over the last few years. Because of the success of machine learning in many other fields. The interest of applying machine learning to the job scheduling of high performance supercomputer has been raised. The reason is that current scheduling schemes are mostly heuristic based approach with various strategies such as backfilling and they are relatively fixed and not flexible enough to handle the changing characteristics of large amount of jobs belonging to different types. Therefore, a more dynamic job scheduler is needed. In this report, two of the recent machine learning approaches are examined on real world workload from National Supercomputer Center of Singapore(NSCC). One of them used a nonlinear regression machine learning strategy. The other one made use of the popular deep reinforcement learning technique. The nonlinear regression functions performed fairly well on the real world workload, showing its robustness and generalization ability. The deep reinforcement learning model could not easily adapt the real world workload due to its complicated architecture and difficulty and time consuming in training. Its performance on NSCC real world data is not ideal, either. Therefore, the nonlinear regression method is preferred to be further improved and adapted for production use. |
| first_indexed | 2024-10-01T05:37:20Z |
| format | Final Year Project (FYP) |
| id | ntu-10356/73967 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T05:37:20Z |
| publishDate | 2018 |
| record_format | dspace |
| spelling | ntu-10356/739672023-03-03T20:32:52Z Visualisation and management tool for supercomputer resource Shao, Yiyang Lee Bu Sung, Francis School of Computer Science and Engineering NSCC DRNTU::Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence The machine learning, especially deep learning has gained an astonishing amount of popularity over the last few years. Because of the success of machine learning in many other fields. The interest of applying machine learning to the job scheduling of high performance supercomputer has been raised. The reason is that current scheduling schemes are mostly heuristic based approach with various strategies such as backfilling and they are relatively fixed and not flexible enough to handle the changing characteristics of large amount of jobs belonging to different types. Therefore, a more dynamic job scheduler is needed. In this report, two of the recent machine learning approaches are examined on real world workload from National Supercomputer Center of Singapore(NSCC). One of them used a nonlinear regression machine learning strategy. The other one made use of the popular deep reinforcement learning technique. The nonlinear regression functions performed fairly well on the real world workload, showing its robustness and generalization ability. The deep reinforcement learning model could not easily adapt the real world workload due to its complicated architecture and difficulty and time consuming in training. Its performance on NSCC real world data is not ideal, either. Therefore, the nonlinear regression method is preferred to be further improved and adapted for production use. Bachelor of Engineering (Computer Science) 2018-04-23T02:54:35Z 2018-04-23T02:54:35Z 2018 Final Year Project (FYP) http://hdl.handle.net/10356/73967 en Nanyang Technological University 37 p. application/pdf |
| spellingShingle | DRNTU::Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence Shao, Yiyang Visualisation and management tool for supercomputer resource |
| title | Visualisation and management tool for supercomputer resource |
| title_full | Visualisation and management tool for supercomputer resource |
| title_fullStr | Visualisation and management tool for supercomputer resource |
| title_full_unstemmed | Visualisation and management tool for supercomputer resource |
| title_short | Visualisation and management tool for supercomputer resource |
| title_sort | visualisation and management tool for supercomputer resource |
| topic | DRNTU::Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence |
| url | http://hdl.handle.net/10356/73967 |
| work_keys_str_mv | AT shaoyiyang visualisationandmanagementtoolforsupercomputerresource |