Multi-stage resource-aware scheduling for data centers with heterogeneous servers
This paper presents a three-stage algorithm for resource-aware scheduling of computational jobs in a large-scale heterogeneous data center. The algorithm aims to allocate job classes to machine configurations to attain an efficient mapping between job resource request profiles and machine resource c...
| Main Authors: | , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Springer US
2018
|
| Online Access: | http://hdl.handle.net/1721.1/116657 https://orcid.org/0000-0002-0422-834X |
| _version_ | 1826206665310994432 |
|---|---|
| author | Padmanabhan, Meghana Li, Heyse Tran, Tony T. Down, Douglas G. Beck, J. Christopher Zhang, Yun |
| author2 | Massachusetts Institute of Technology. Institute for Data, Systems, and Society |
| author_facet | Massachusetts Institute of Technology. Institute for Data, Systems, and Society Padmanabhan, Meghana Li, Heyse Tran, Tony T. Down, Douglas G. Beck, J. Christopher Zhang, Yun |
| author_sort | Padmanabhan, Meghana |
| collection | MIT |
| description | This paper presents a three-stage algorithm for resource-aware scheduling of computational jobs in a large-scale heterogeneous data center. The algorithm aims to allocate job classes to machine configurations to attain an efficient mapping between job resource request profiles and machine resource capacity profiles. The first stage uses a queueing model that treats the system in an aggregated manner with pooled machines and jobs represented as a fluid flow. The latter two stages use combinatorial optimization techniques to solve a shorter-term, more accurate representation of the problem using the first-stage, long-term solution for heuristic guidance. In the second stage, jobs and machines are discretized. A linear programming model is used to obtain a solution to the discrete problem that maximizes the system capacity given a restriction on the job class and machine configuration pairings based on the solution of the first stage. The final stage is a scheduling policy that uses the solution from the second stage to guide the dispatching of arriving jobs to machines. We present experimental results of our algorithm on both Google workload trace data and generated data and show that it outperforms existing schedulers. These results illustrate the importance of considering heterogeneity of both job and machine configuration profiles in making effective scheduling decisions. Keywords: Resource-aware scheduling, Dynamic scheduling, Heterogeneous servers |
| first_indexed | 2024-09-23T13:36:22Z |
| format | Article |
| id | mit-1721.1/116657 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T13:36:22Z |
| publishDate | 2018 |
| publisher | Springer US |
| record_format | dspace |
| spelling | mit-1721.1/1166572022-09-28T15:00:11Z Multi-stage resource-aware scheduling for data centers with heterogeneous servers Padmanabhan, Meghana Li, Heyse Tran, Tony T. Down, Douglas G. Beck, J. Christopher Zhang, Yun Massachusetts Institute of Technology. Institute for Data, Systems, and Society Zhang, Yun This paper presents a three-stage algorithm for resource-aware scheduling of computational jobs in a large-scale heterogeneous data center. The algorithm aims to allocate job classes to machine configurations to attain an efficient mapping between job resource request profiles and machine resource capacity profiles. The first stage uses a queueing model that treats the system in an aggregated manner with pooled machines and jobs represented as a fluid flow. The latter two stages use combinatorial optimization techniques to solve a shorter-term, more accurate representation of the problem using the first-stage, long-term solution for heuristic guidance. In the second stage, jobs and machines are discretized. A linear programming model is used to obtain a solution to the discrete problem that maximizes the system capacity given a restriction on the job class and machine configuration pairings based on the solution of the first stage. The final stage is a scheduling policy that uses the solution from the second stage to guide the dispatching of arriving jobs to machines. We present experimental results of our algorithm on both Google workload trace data and generated data and show that it outperforms existing schedulers. These results illustrate the importance of considering heterogeneity of both job and machine configuration profiles in making effective scheduling decisions. Keywords: Resource-aware scheduling, Dynamic scheduling, Heterogeneous servers Google (Firm) (Research Award) Natural Sciences and Engineering Research Council of Canada 2018-06-27T14:27:35Z 2018-06-27T14:27:35Z 2017-07 2018-03-07T05:22:52Z Article http://purl.org/eprint/type/JournalArticle 1094-6136 1099-1425 http://hdl.handle.net/1721.1/116657 Tran, Tony T., et al. “Multi-Stage Resource-Aware Scheduling for Data Centers with Heterogeneous Servers.” Journal of Scheduling, vol. 21, no. 2, Apr. 2018, pp. 251–67. https://orcid.org/0000-0002-0422-834X en http://dx.doi.org/10.1007/s10951-017-0537-x Journal of Scheduling Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. Springer Science+Business Media, LLC application/pdf Springer US Springer US |
| spellingShingle | Padmanabhan, Meghana Li, Heyse Tran, Tony T. Down, Douglas G. Beck, J. Christopher Zhang, Yun Multi-stage resource-aware scheduling for data centers with heterogeneous servers |
| title | Multi-stage resource-aware scheduling for data centers with heterogeneous servers |
| title_full | Multi-stage resource-aware scheduling for data centers with heterogeneous servers |
| title_fullStr | Multi-stage resource-aware scheduling for data centers with heterogeneous servers |
| title_full_unstemmed | Multi-stage resource-aware scheduling for data centers with heterogeneous servers |
| title_short | Multi-stage resource-aware scheduling for data centers with heterogeneous servers |
| title_sort | multi stage resource aware scheduling for data centers with heterogeneous servers |
| url | http://hdl.handle.net/1721.1/116657 https://orcid.org/0000-0002-0422-834X |
| work_keys_str_mv | AT padmanabhanmeghana multistageresourceawareschedulingfordatacenterswithheterogeneousservers AT liheyse multistageresourceawareschedulingfordatacenterswithheterogeneousservers AT trantonyt multistageresourceawareschedulingfordatacenterswithheterogeneousservers AT downdouglasg multistageresourceawareschedulingfordatacenterswithheterogeneousservers AT beckjchristopher multistageresourceawareschedulingfordatacenterswithheterogeneousservers AT zhangyun multistageresourceawareschedulingfordatacenterswithheterogeneousservers |