Fair Resource Allocation for Data-Intensive Computing in the Cloud
To address the computing challenge of ’big data’, a number of data-intensive computing frameworks (e.g., MapReduce, Dryad, Storm and Spark) have emerged and become popular. YARN is a de facto resource management platform that enables these frameworks running together in a shared system. However, we...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2016
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/80372 http://hdl.handle.net/10220/40473 |
| _version_ | 1826115913322070016 |
|---|---|
| author | Tang, Shanjiang Lee, Bu-Sung He, Bingsheng |
| author2 | School of Computer Engineering |
| author_facet | School of Computer Engineering Tang, Shanjiang Lee, Bu-Sung He, Bingsheng |
| author_sort | Tang, Shanjiang |
| collection | NTU |
| description | To address the computing challenge of ’big data’, a number of data-intensive computing frameworks (e.g., MapReduce, Dryad, Storm and Spark) have emerged and become popular. YARN is a de facto resource management platform that enables these frameworks running together in a shared system. However, we observe that, in cloud computing environment, the fair resource allocation policy implemented in YARN is not suitable because of its memoryless resource allocation fashion leading to violations of a number of good properties in shared computing systems. This paper attempts to address these problems for YARN. Both singlelevel and hierarchical resource allocations are considered. For single-level resource allocation, we propose a novel fair resource allocation mechanism called Long-Term Resource Fairness (LTRF) for such computing. For hierarchical resource allocation, we propose Hierarchical Long-Term Resource Fairness (H-LTRF) by extending LTRF. We show that both LTRF and H-LTRF can address these fairness problems of current resource allocation policy and are thus suitable for cloud computing. Finally, we have developed LTYARN by implementing LTRF and H-LTRF in YARN, and our experiments show that it leads to a better resource fairness than existing fair schedulers of YARN. |
| first_indexed | 2024-10-01T04:02:53Z |
| format | Journal Article |
| id | ntu-10356/80372 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T04:02:53Z |
| publishDate | 2016 |
| record_format | dspace |
| spelling | ntu-10356/803722020-05-28T07:17:42Z Fair Resource Allocation for Data-Intensive Computing in the Cloud Tang, Shanjiang Lee, Bu-Sung He, Bingsheng School of Computer Engineering Computer Science and Engineering To address the computing challenge of ’big data’, a number of data-intensive computing frameworks (e.g., MapReduce, Dryad, Storm and Spark) have emerged and become popular. YARN is a de facto resource management platform that enables these frameworks running together in a shared system. However, we observe that, in cloud computing environment, the fair resource allocation policy implemented in YARN is not suitable because of its memoryless resource allocation fashion leading to violations of a number of good properties in shared computing systems. This paper attempts to address these problems for YARN. Both singlelevel and hierarchical resource allocations are considered. For single-level resource allocation, we propose a novel fair resource allocation mechanism called Long-Term Resource Fairness (LTRF) for such computing. For hierarchical resource allocation, we propose Hierarchical Long-Term Resource Fairness (H-LTRF) by extending LTRF. We show that both LTRF and H-LTRF can address these fairness problems of current resource allocation policy and are thus suitable for cloud computing. Finally, we have developed LTYARN by implementing LTRF and H-LTRF in YARN, and our experiments show that it leads to a better resource fairness than existing fair schedulers of YARN. Accepted version 2016-05-03T08:34:50Z 2019-12-06T13:48:10Z 2016-05-03T08:34:50Z 2019-12-06T13:48:10Z 2016 2016 Journal Article Tang, S., Lee, B.-S., & He, B. (2016). Fair Resource Allocation for Data-Intensive Computing in the Cloud. IEEE Transactions on Services Computing, in press. 1939-1374 https://hdl.handle.net/10356/80372 http://hdl.handle.net/10220/40473 10.1109/TSC.2016.2531698 191997 en IEEE Transactions on Services Computing © 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. The published version is available at: [http://dx.doi.org/10.1109/TSC.2016.2531698]. 14 p. application/pdf |
| spellingShingle | Computer Science and Engineering Tang, Shanjiang Lee, Bu-Sung He, Bingsheng Fair Resource Allocation for Data-Intensive Computing in the Cloud |
| title | Fair Resource Allocation for Data-Intensive Computing in the Cloud |
| title_full | Fair Resource Allocation for Data-Intensive Computing in the Cloud |
| title_fullStr | Fair Resource Allocation for Data-Intensive Computing in the Cloud |
| title_full_unstemmed | Fair Resource Allocation for Data-Intensive Computing in the Cloud |
| title_short | Fair Resource Allocation for Data-Intensive Computing in the Cloud |
| title_sort | fair resource allocation for data intensive computing in the cloud |
| topic | Computer Science and Engineering |
| url | https://hdl.handle.net/10356/80372 http://hdl.handle.net/10220/40473 |
| work_keys_str_mv | AT tangshanjiang fairresourceallocationfordataintensivecomputinginthecloud AT leebusung fairresourceallocationfordataintensivecomputinginthecloud AT hebingsheng fairresourceallocationfordataintensivecomputinginthecloud |