A map-reduce based framework for heterogeneous processing element cluster environments
In this paper, we present our design of a Processing Element (PE) Aware MapReduce base framework, Pamar. Pamar is designed for supporting distributed computing on clusters where node PE configurations are asymmetric on different nodes. Pamar's main goal is to allow users to seamlessly utilize d...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Conference Paper |
| Language: | English |
| Published: |
2013
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/101493 http://hdl.handle.net/10220/16728 |
| _version_ | 1811693311242534912 |
|---|---|
| author | Tan, Yu Shyang Lee, Bu-Sung He, Bingsheng Campbell, Roy H. |
| author2 | School of Computer Engineering |
| author_facet | School of Computer Engineering Tan, Yu Shyang Lee, Bu-Sung He, Bingsheng Campbell, Roy H. |
| author_sort | Tan, Yu Shyang |
| collection | NTU |
| description | In this paper, we present our design of a Processing Element (PE) Aware MapReduce base framework, Pamar. Pamar is designed for supporting distributed computing on clusters where node PE configurations are asymmetric on different nodes. Pamar's main goal is to allow users to seamlessly utilize different kinds of processing elements (e.g., CPUs or GPUs) collaboratively for large scale data processing. To show proof of concept, we have incorporated our designs into the Hadoop framework and tested it on cluster environments having asymmetric node PE configurations. We demonstrate Pamar's ability to identify PEs available on each node and match-make user jobs with nodes, base on job PE requirements. Pamar allows users to easily parallelize applications across large datasets and at the same time utilizes different PEs for processing different classes of functions efficiently. The experiments show improvement in job queue completion time with Pamar over clusters with asymmetric nodes as compared to clusters with symmetric nodes. |
| first_indexed | 2024-10-01T06:49:40Z |
| format | Conference Paper |
| id | ntu-10356/101493 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T06:49:40Z |
| publishDate | 2013 |
| record_format | dspace |
| spelling | ntu-10356/1014932020-05-28T07:18:08Z A map-reduce based framework for heterogeneous processing element cluster environments Tan, Yu Shyang Lee, Bu-Sung He, Bingsheng Campbell, Roy H. School of Computer Engineering IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (12th : 2012 : Ottawa, Canada) DRNTU::Engineering::Computer science and engineering In this paper, we present our design of a Processing Element (PE) Aware MapReduce base framework, Pamar. Pamar is designed for supporting distributed computing on clusters where node PE configurations are asymmetric on different nodes. Pamar's main goal is to allow users to seamlessly utilize different kinds of processing elements (e.g., CPUs or GPUs) collaboratively for large scale data processing. To show proof of concept, we have incorporated our designs into the Hadoop framework and tested it on cluster environments having asymmetric node PE configurations. We demonstrate Pamar's ability to identify PEs available on each node and match-make user jobs with nodes, base on job PE requirements. Pamar allows users to easily parallelize applications across large datasets and at the same time utilizes different PEs for processing different classes of functions efficiently. The experiments show improvement in job queue completion time with Pamar over clusters with asymmetric nodes as compared to clusters with symmetric nodes. 2013-10-23T07:06:30Z 2019-12-06T20:39:14Z 2013-10-23T07:06:30Z 2019-12-06T20:39:14Z 2012 2012 Conference Paper Tan, Y. S., Lee, B.-S., He, B., & Campbell, R. H. (2012). A Map-Reduce Based Framework for Heterogeneous Processing Element Cluster Environments. 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), 57-64. https://hdl.handle.net/10356/101493 http://hdl.handle.net/10220/16728 10.1109/CCGrid.2012.35 en © 2012 IEEE |
| spellingShingle | DRNTU::Engineering::Computer science and engineering Tan, Yu Shyang Lee, Bu-Sung He, Bingsheng Campbell, Roy H. A map-reduce based framework for heterogeneous processing element cluster environments |
| title | A map-reduce based framework for heterogeneous processing element cluster environments |
| title_full | A map-reduce based framework for heterogeneous processing element cluster environments |
| title_fullStr | A map-reduce based framework for heterogeneous processing element cluster environments |
| title_full_unstemmed | A map-reduce based framework for heterogeneous processing element cluster environments |
| title_short | A map-reduce based framework for heterogeneous processing element cluster environments |
| title_sort | map reduce based framework for heterogeneous processing element cluster environments |
| topic | DRNTU::Engineering::Computer science and engineering |
| url | https://hdl.handle.net/10356/101493 http://hdl.handle.net/10220/16728 |
| work_keys_str_mv | AT tanyushyang amapreducebasedframeworkforheterogeneousprocessingelementclusterenvironments AT leebusung amapreducebasedframeworkforheterogeneousprocessingelementclusterenvironments AT hebingsheng amapreducebasedframeworkforheterogeneousprocessingelementclusterenvironments AT campbellroyh amapreducebasedframeworkforheterogeneousprocessingelementclusterenvironments AT tanyushyang mapreducebasedframeworkforheterogeneousprocessingelementclusterenvironments AT leebusung mapreducebasedframeworkforheterogeneousprocessingelementclusterenvironments AT hebingsheng mapreducebasedframeworkforheterogeneousprocessingelementclusterenvironments AT campbellroyh mapreducebasedframeworkforheterogeneousprocessingelementclusterenvironments |