Learning scheduling algorithms for data processing clusters
© 2019 Association for Computing Machinery. Efficiently scheduling data processing jobs on distributed compute clusters requires complex algorithms. Current systems use simple, generalized heuristics and ignore workload characteristics, since developing and tuning a scheduling policy for each worklo...
| Main Authors: | , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Association for Computing Machinery (ACM)
2021
|
| Online Access: | https://hdl.handle.net/1721.1/137428 |
| _version_ | 1826209451878645760 |
|---|---|
| author | Mao, Hongzi Schwarzkopf, Malte Venkatakrishnan, Shaileshh Bojja Meng, Zili Alizadeh, Mohammad |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Mao, Hongzi Schwarzkopf, Malte Venkatakrishnan, Shaileshh Bojja Meng, Zili Alizadeh, Mohammad |
| author_sort | Mao, Hongzi |
| collection | MIT |
| description | © 2019 Association for Computing Machinery. Efficiently scheduling data processing jobs on distributed compute clusters requires complex algorithms. Current systems use simple, generalized heuristics and ignore workload characteristics, since developing and tuning a scheduling policy for each workload is infeasible. In this paper, we showthat modern machine learning techniques can generate highly-efficient policies automatically. Decima uses reinforcement learning (RL) and neural networks to learn workload-specific scheduling algorithms without any human instruction beyond a high-level objective, such as minimizing average job completion time. However, off-the-shelf RL techniques cannot handle the complexity and scale of the scheduling problem. To build Decima, we had to develop new representations for jobs' dependency graphs, design scalable RL models, and invent RL training methods for dealing with continuous stochastic job arrivals. Our prototype integration with Spark on a 25-node cluster shows that Decima improves average job completion time by at least 21% over hand-tuned scheduling heuristics, achieving up to 2× improvement during periods of high cluster load. |
| first_indexed | 2024-09-23T14:22:43Z |
| format | Article |
| id | mit-1721.1/137428 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T14:22:43Z |
| publishDate | 2021 |
| publisher | Association for Computing Machinery (ACM) |
| record_format | dspace |
| spelling | mit-1721.1/1374282023-07-28T17:44:39Z Learning scheduling algorithms for data processing clusters Mao, Hongzi Schwarzkopf, Malte Venkatakrishnan, Shaileshh Bojja Meng, Zili Alizadeh, Mohammad Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory © 2019 Association for Computing Machinery. Efficiently scheduling data processing jobs on distributed compute clusters requires complex algorithms. Current systems use simple, generalized heuristics and ignore workload characteristics, since developing and tuning a scheduling policy for each workload is infeasible. In this paper, we showthat modern machine learning techniques can generate highly-efficient policies automatically. Decima uses reinforcement learning (RL) and neural networks to learn workload-specific scheduling algorithms without any human instruction beyond a high-level objective, such as minimizing average job completion time. However, off-the-shelf RL techniques cannot handle the complexity and scale of the scheduling problem. To build Decima, we had to develop new representations for jobs' dependency graphs, design scalable RL models, and invent RL training methods for dealing with continuous stochastic job arrivals. Our prototype integration with Spark on a 25-node cluster shows that Decima improves average job completion time by at least 21% over hand-tuned scheduling heuristics, achieving up to 2× improvement during periods of high cluster load. 2021-11-05T11:54:21Z 2021-11-05T11:54:21Z 2019-08 2020-11-23T19:02:38Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/137428 Mao, Hongzi, Schwarzkopf, Malte, Venkatakrishnan, Shaileshh Bojja, Meng, Zili and Alizadeh, Mohammad. 2019. "Learning scheduling algorithms for data processing clusters." SIGCOMM 2019 - Proceedings of the 2019 Conference of the ACM Special Interest Group on Data Communication. en 10.1145/3341302.3342080 SIGCOMM 2019 - Proceedings of the 2019 Conference of the ACM Special Interest Group on Data Communication Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf Association for Computing Machinery (ACM) MIT web domain |
| spellingShingle | Mao, Hongzi Schwarzkopf, Malte Venkatakrishnan, Shaileshh Bojja Meng, Zili Alizadeh, Mohammad Learning scheduling algorithms for data processing clusters |
| title | Learning scheduling algorithms for data processing clusters |
| title_full | Learning scheduling algorithms for data processing clusters |
| title_fullStr | Learning scheduling algorithms for data processing clusters |
| title_full_unstemmed | Learning scheduling algorithms for data processing clusters |
| title_short | Learning scheduling algorithms for data processing clusters |
| title_sort | learning scheduling algorithms for data processing clusters |
| url | https://hdl.handle.net/1721.1/137428 |
| work_keys_str_mv | AT maohongzi learningschedulingalgorithmsfordataprocessingclusters AT schwarzkopfmalte learningschedulingalgorithmsfordataprocessingclusters AT venkatakrishnanshaileshhbojja learningschedulingalgorithmsfordataprocessingclusters AT mengzili learningschedulingalgorithmsfordataprocessingclusters AT alizadehmohammad learningschedulingalgorithmsfordataprocessingclusters |