Optimizing MapReduce Task Scheduling on Virtualized Heterogeneous Environments Using Ant Colony Optimization
Consuming Hadoop MapReduce via virtual infrastructure as a service is becoming common practice as cloud service providers (CSP) offers relevant applications and scalable resources. One of the predominant requirements of cloud users is to improve resource utilization in the virtual cluster during the...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9779225/ |
| _version_ | 1817987890947293184 |
|---|---|
| author | Rathinaraja Jeyaraj Anand Paul |
| author_facet | Rathinaraja Jeyaraj Anand Paul |
| author_sort | Rathinaraja Jeyaraj |
| collection | DOAJ |
| description | Consuming Hadoop MapReduce via virtual infrastructure as a service is becoming common practice as cloud service providers (CSP) offers relevant applications and scalable resources. One of the predominant requirements of cloud users is to improve resource utilization in the virtual cluster during the service period. However, it may not be possible when MapReduce workloads and virtual machines (VM) are highly heterogeneous. Therefore, in this paper, we addressed these heterogeneities and proposed an efficient MapReduce scheduler to improve resource utilization by placing the right combination of the map and reduce tasks in each VM in the virtual cluster. To achieve this, we transformed the MapReduce task scheduling problem into a 2-Dimensional (2D) bin packing model and obtained an optimal schedule using the ant colony optimization (ACO) algorithm. As an added advantage, our proposed ACO based bin packing (ACO-BP) scheduler minimized the makespan for a batch of jobs. To showcase the performance improvement, we compared our proposed scheduler with three existing schedulers that work well in a heterogeneous environment. As expected, results show that ACO-BP significantly outperformed the existing schedulers while dealing with workload and VM level heterogeneities. |
| first_indexed | 2024-04-14T00:27:04Z |
| format | Article |
| id | doaj.art-0ae043a870904829b1b924bb3b06f43f |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-14T00:27:04Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-0ae043a870904829b1b924bb3b06f43f2022-12-22T02:22:40ZengIEEEIEEE Access2169-35362022-01-0110558425585510.1109/ACCESS.2022.31767299779225Optimizing MapReduce Task Scheduling on Virtualized Heterogeneous Environments Using Ant Colony OptimizationRathinaraja Jeyaraj0https://orcid.org/0000-0003-0165-181XAnand Paul1https://orcid.org/0000-0002-0737-2021School of Computer Science and Engineering, Kyungpook National University, Daegu, South KoreaSchool of Computer Science and Engineering, Kyungpook National University, Daegu, South KoreaConsuming Hadoop MapReduce via virtual infrastructure as a service is becoming common practice as cloud service providers (CSP) offers relevant applications and scalable resources. One of the predominant requirements of cloud users is to improve resource utilization in the virtual cluster during the service period. However, it may not be possible when MapReduce workloads and virtual machines (VM) are highly heterogeneous. Therefore, in this paper, we addressed these heterogeneities and proposed an efficient MapReduce scheduler to improve resource utilization by placing the right combination of the map and reduce tasks in each VM in the virtual cluster. To achieve this, we transformed the MapReduce task scheduling problem into a 2-Dimensional (2D) bin packing model and obtained an optimal schedule using the ant colony optimization (ACO) algorithm. As an added advantage, our proposed ACO based bin packing (ACO-BP) scheduler minimized the makespan for a batch of jobs. To showcase the performance improvement, we compared our proposed scheduler with three existing schedulers that work well in a heterogeneous environment. As expected, results show that ACO-BP significantly outperformed the existing schedulers while dealing with workload and VM level heterogeneities.https://ieeexplore.ieee.org/document/9779225/Ant colony optimizationbin packingheterogeneityMapReduceresource utilizationtask scheduling |
| spellingShingle | Rathinaraja Jeyaraj Anand Paul Optimizing MapReduce Task Scheduling on Virtualized Heterogeneous Environments Using Ant Colony Optimization IEEE Access Ant colony optimization bin packing heterogeneity MapReduce resource utilization task scheduling |
| title | Optimizing MapReduce Task Scheduling on Virtualized Heterogeneous Environments Using Ant Colony Optimization |
| title_full | Optimizing MapReduce Task Scheduling on Virtualized Heterogeneous Environments Using Ant Colony Optimization |
| title_fullStr | Optimizing MapReduce Task Scheduling on Virtualized Heterogeneous Environments Using Ant Colony Optimization |
| title_full_unstemmed | Optimizing MapReduce Task Scheduling on Virtualized Heterogeneous Environments Using Ant Colony Optimization |
| title_short | Optimizing MapReduce Task Scheduling on Virtualized Heterogeneous Environments Using Ant Colony Optimization |
| title_sort | optimizing mapreduce task scheduling on virtualized heterogeneous environments using ant colony optimization |
| topic | Ant colony optimization bin packing heterogeneity MapReduce resource utilization task scheduling |
| url | https://ieeexplore.ieee.org/document/9779225/ |
| work_keys_str_mv | AT rathinarajajeyaraj optimizingmapreducetaskschedulingonvirtualizedheterogeneousenvironmentsusingantcolonyoptimization AT anandpaul optimizingmapreducetaskschedulingonvirtualizedheterogeneousenvironmentsusingantcolonyoptimization |