Employing Vertical Elasticity for Efficient Big Data Processing in Container-Based Cloud Environments
Recently, “Big Data” platform technologies have become crucial for distributed processing of diverse unstructured or semi-structured data as the amount of data generated increases rapidly. In order to effectively manage these Big Data, Cloud Computing has been playing an important role by providing...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-07-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/11/13/6200 |
| _version_ | 1797528057855082496 |
|---|---|
| author | Jin-young Choi Minkyoung Cho Jik-Soo Kim |
| author_facet | Jin-young Choi Minkyoung Cho Jik-Soo Kim |
| author_sort | Jin-young Choi |
| collection | DOAJ |
| description | Recently, “Big Data” platform technologies have become crucial for distributed processing of diverse unstructured or semi-structured data as the amount of data generated increases rapidly. In order to effectively manage these Big Data, Cloud Computing has been playing an important role by providing scalable data storage and computing resources for competitive and economical Big Data processing. Accordingly, server virtualization technologies that are the cornerstone of Cloud Computing have attracted a lot of research interests. However, conventional hypervisor-based virtualization can cause performance degradation problems due to its heavily loaded guest operating systems and rigid resource allocations. On the other hand, container-based virtualization technology can provide the same level of service faster with a lightweight capacity by effectively eliminating the guest OS layers. In addition, container-based virtualization enables efficient cloud resource management by dynamically adjusting the allocated computing resources (e.g., CPU and memory) during the runtime through “Vertical Elasticity”. In this paper, we present our practice and experience of employing an adaptive resource utilization scheme for Big Data workloads in container-based cloud environments by leveraging the vertical elasticity of Docker, a representative container-based virtualization technique. We perform extensive experiments running several Big Data workloads on representative Big Data platforms: Apache Hadoop and Spark. During the workload executions, our adaptive resource utilization scheme periodically monitors the resource usage patterns of running containers and dynamically adjusts allocated computing resources that could result in substantial improvements in the overall system throughput. |
| first_indexed | 2024-03-10T09:52:44Z |
| format | Article |
| id | doaj.art-4d4e78c882174d03b21ebc68d31f9775 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-10T09:52:44Z |
| publishDate | 2021-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-4d4e78c882174d03b21ebc68d31f97752023-11-22T02:35:06ZengMDPI AGApplied Sciences2076-34172021-07-011113620010.3390/app11136200Employing Vertical Elasticity for Efficient Big Data Processing in Container-Based Cloud EnvironmentsJin-young Choi0Minkyoung Cho1Jik-Soo Kim2Gabia Inc., Seongnam 13494, KoreaDepartment of Computer Engineering, Myongji University, Yongin 17058, KoreaDepartment of Computer Engineering, Myongji University, Yongin 17058, KoreaRecently, “Big Data” platform technologies have become crucial for distributed processing of diverse unstructured or semi-structured data as the amount of data generated increases rapidly. In order to effectively manage these Big Data, Cloud Computing has been playing an important role by providing scalable data storage and computing resources for competitive and economical Big Data processing. Accordingly, server virtualization technologies that are the cornerstone of Cloud Computing have attracted a lot of research interests. However, conventional hypervisor-based virtualization can cause performance degradation problems due to its heavily loaded guest operating systems and rigid resource allocations. On the other hand, container-based virtualization technology can provide the same level of service faster with a lightweight capacity by effectively eliminating the guest OS layers. In addition, container-based virtualization enables efficient cloud resource management by dynamically adjusting the allocated computing resources (e.g., CPU and memory) during the runtime through “Vertical Elasticity”. In this paper, we present our practice and experience of employing an adaptive resource utilization scheme for Big Data workloads in container-based cloud environments by leveraging the vertical elasticity of Docker, a representative container-based virtualization technique. We perform extensive experiments running several Big Data workloads on representative Big Data platforms: Apache Hadoop and Spark. During the workload executions, our adaptive resource utilization scheme periodically monitors the resource usage patterns of running containers and dynamically adjusts allocated computing resources that could result in substantial improvements in the overall system throughput.https://www.mdpi.com/2076-3417/11/13/6200big datacloud computingcontainerdockerresource managementvirtualization |
| spellingShingle | Jin-young Choi Minkyoung Cho Jik-Soo Kim Employing Vertical Elasticity for Efficient Big Data Processing in Container-Based Cloud Environments Applied Sciences big data cloud computing container docker resource management virtualization |
| title | Employing Vertical Elasticity for Efficient Big Data Processing in Container-Based Cloud Environments |
| title_full | Employing Vertical Elasticity for Efficient Big Data Processing in Container-Based Cloud Environments |
| title_fullStr | Employing Vertical Elasticity for Efficient Big Data Processing in Container-Based Cloud Environments |
| title_full_unstemmed | Employing Vertical Elasticity for Efficient Big Data Processing in Container-Based Cloud Environments |
| title_short | Employing Vertical Elasticity for Efficient Big Data Processing in Container-Based Cloud Environments |
| title_sort | employing vertical elasticity for efficient big data processing in container based cloud environments |
| topic | big data cloud computing container docker resource management virtualization |
| url | https://www.mdpi.com/2076-3417/11/13/6200 |
| work_keys_str_mv | AT jinyoungchoi employingverticalelasticityforefficientbigdataprocessingincontainerbasedcloudenvironments AT minkyoungcho employingverticalelasticityforefficientbigdataprocessingincontainerbasedcloudenvironments AT jiksookim employingverticalelasticityforefficientbigdataprocessingincontainerbasedcloudenvironments |