JouleMR : towards cost-effective and green-aware data processing frameworks
Interests have been growing in energy management of the cluster effectively in order to reduce the energy consumption as well as the electricity cost. Renewable energy and dynamic pricing schemes in smart grids are two major emerging trends in energy markets. However, current data processing framewo...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2020
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/136965 |
| _version_ | 1826127459585622016 |
|---|---|
| author | Niu, Zhaojie He, Bingsheng Liu, Fangming |
| author2 | Interdisciplinary Graduate School (IGS) |
| author_facet | Interdisciplinary Graduate School (IGS) Niu, Zhaojie He, Bingsheng Liu, Fangming |
| author_sort | Niu, Zhaojie |
| collection | NTU |
| description | Interests have been growing in energy management of the cluster effectively in order to reduce the energy consumption as well as the electricity cost. Renewable energy and dynamic pricing schemes in smart grids are two major emerging trends in energy markets. However, current data processing frameworks are not aware of the efficiency of each joule consumed by the data center workloads in the context of these two major trends. In fact, not all joules are equal in the sense that the amount of work that can be done by a joule can vary significantly in data centers. Ignoring this fact leads to significant energy waste (by 25 percent of the total energy consumption in Hadoop YARN on a Facebook production trace according to our study). In this paper, we propose JouleMR, a cost-effective and green-aware data processing framework. Specifically, we investigate how to exploit such joule efficiency to maximize the benefits of renewable energy as well as dynamic pricing schemes for MapReduce framework. We develop job/task scheduling algorithms with a particular focus on the factors on joule efficiency in the data center, including the energy efficiency of MapReduce workloads, renewable energy supply, dynamic pricing and the battery usage. We further develop a simple yet effective performanceenergy consumption model to guide our scheduling decisions. We have implemented JouleMR on top of Hadoop YARN. The experiments demonstrate the accuracy of our models, and the effectiveness of our cost-effective and green-aware optimizations outperform the state-of-the-art implementations over Hadoop YARN. |
| first_indexed | 2024-10-01T07:09:14Z |
| format | Journal Article |
| id | ntu-10356/136965 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T07:09:14Z |
| publishDate | 2020 |
| record_format | dspace |
| spelling | ntu-10356/1369652020-11-01T04:45:32Z JouleMR : towards cost-effective and green-aware data processing frameworks Niu, Zhaojie He, Bingsheng Liu, Fangming Interdisciplinary Graduate School (IGS) Engineering::Computer science and engineering Data Processing Cost-effective Optimization Interests have been growing in energy management of the cluster effectively in order to reduce the energy consumption as well as the electricity cost. Renewable energy and dynamic pricing schemes in smart grids are two major emerging trends in energy markets. However, current data processing frameworks are not aware of the efficiency of each joule consumed by the data center workloads in the context of these two major trends. In fact, not all joules are equal in the sense that the amount of work that can be done by a joule can vary significantly in data centers. Ignoring this fact leads to significant energy waste (by 25 percent of the total energy consumption in Hadoop YARN on a Facebook production trace according to our study). In this paper, we propose JouleMR, a cost-effective and green-aware data processing framework. Specifically, we investigate how to exploit such joule efficiency to maximize the benefits of renewable energy as well as dynamic pricing schemes for MapReduce framework. We develop job/task scheduling algorithms with a particular focus on the factors on joule efficiency in the data center, including the energy efficiency of MapReduce workloads, renewable energy supply, dynamic pricing and the battery usage. We further develop a simple yet effective performanceenergy consumption model to guide our scheduling decisions. We have implemented JouleMR on top of Hadoop YARN. The experiments demonstrate the accuracy of our models, and the effectiveness of our cost-effective and green-aware optimizations outperform the state-of-the-art implementations over Hadoop YARN. NRF (Natl Research Foundation, S’pore) MOE (Min. of Education, S’pore) Accepted version 2020-02-07T06:09:56Z 2020-02-07T06:09:56Z 2018 Journal Article Niu, Z., He, B., & Liu, F. (2018). JouleMR : towards cost-effective and green-aware data processing frameworks. IEEE Transactions on Big Data, 4(2), 258-271. doi:10.1109/TBDATA.2017.2655037 2332-7790 https://hdl.handle.net/10356/136965 10.1109/TBDATA.2017.2655037 2 4 258 272 en IEEE Transactions on Big Data © 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. The published version is available at: https://doi.org/10.1109/TBDATA.2017.2655037. application/pdf |
| spellingShingle | Engineering::Computer science and engineering Data Processing Cost-effective Optimization Niu, Zhaojie He, Bingsheng Liu, Fangming JouleMR : towards cost-effective and green-aware data processing frameworks |
| title | JouleMR : towards cost-effective and green-aware data processing frameworks |
| title_full | JouleMR : towards cost-effective and green-aware data processing frameworks |
| title_fullStr | JouleMR : towards cost-effective and green-aware data processing frameworks |
| title_full_unstemmed | JouleMR : towards cost-effective and green-aware data processing frameworks |
| title_short | JouleMR : towards cost-effective and green-aware data processing frameworks |
| title_sort | joulemr towards cost effective and green aware data processing frameworks |
| topic | Engineering::Computer science and engineering Data Processing Cost-effective Optimization |
| url | https://hdl.handle.net/10356/136965 |
| work_keys_str_mv | AT niuzhaojie joulemrtowardscosteffectiveandgreenawaredataprocessingframeworks AT hebingsheng joulemrtowardscosteffectiveandgreenawaredataprocessingframeworks AT liufangming joulemrtowardscosteffectiveandgreenawaredataprocessingframeworks |