Resource Profiling and Performance Modeling for Distributed Scientific Computing Environments
Scientific applications often require substantial amount of computing resources for running challenging jobs potentially consisting of many tasks from hundreds of thousands to even millions. As a result, many institutions collaborate to solve large-scale problems by creating virtual organizations (V...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-05-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/12/9/4797 |
| _version_ | 1797505502536531968 |
|---|---|
| author | Md Azam Hossain Soonwook Hwang Jik-Soo Kim |
| author_facet | Md Azam Hossain Soonwook Hwang Jik-Soo Kim |
| author_sort | Md Azam Hossain |
| collection | DOAJ |
| description | Scientific applications often require substantial amount of computing resources for running challenging jobs potentially consisting of many tasks from hundreds of thousands to even millions. As a result, many institutions collaborate to solve large-scale problems by creating virtual organizations (VOs), and integrate hundreds of thousands of geographically distributed heterogeneous computing resources. Over the past decade, VOs have been proven to be a powerful research testbed for accessing massive amount of computing resources shared by several organizations at almost no cost. However, VOs often suffer from providing exact dynamic resource information due to their scale and autonomous resource management policies. Furthermore, shared resources are inconsistent, making it difficult to accurately forecast resource capacity. An effective VO’s resource profiling and modeling system can address these problems by forecasting resource characteristics and availability. This paper presents effective resource profiling and performance prediction models including Adaptive Filter-based Online Linear Regression (AFOLR) and Adaptive Filter-based Moving Average (AFMV) based on the linear difference equation combining past predicted values and recent profiled information, which aim to support large-scale applications in distributed scientific computing environments. We performed quantitative analysis and conducted microbenchmark experiments on a real multinational shared computing platform. Our evaluation results demonstrate that the proposed prediction schemes outperform well-known common approaches in terms of accuracy, and actually can help users in a shared resource environment to run their large-scale applications by effectively forecasting various computing resource capacity and performance. |
| first_indexed | 2024-03-10T04:19:30Z |
| format | Article |
| id | doaj.art-f83a71ce2d694bb09f7115c455afba21 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-10T04:19:30Z |
| publishDate | 2022-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-f83a71ce2d694bb09f7115c455afba212023-11-23T07:54:04ZengMDPI AGApplied Sciences2076-34172022-05-01129479710.3390/app12094797Resource Profiling and Performance Modeling for Distributed Scientific Computing EnvironmentsMd Azam Hossain0Soonwook Hwang1Jik-Soo Kim2Network and Data Analysis Group (NDAG), Department of Computer Science and Engineering, Islamic University and Technology (IUT), Gazipur 1704, BangladeshKorea Institute of Science and Technology Information (KISTI), Daejeon 34141, KoreaDepartment of Computer Engineering, Myongji University, Yongin 17058, KoreaScientific applications often require substantial amount of computing resources for running challenging jobs potentially consisting of many tasks from hundreds of thousands to even millions. As a result, many institutions collaborate to solve large-scale problems by creating virtual organizations (VOs), and integrate hundreds of thousands of geographically distributed heterogeneous computing resources. Over the past decade, VOs have been proven to be a powerful research testbed for accessing massive amount of computing resources shared by several organizations at almost no cost. However, VOs often suffer from providing exact dynamic resource information due to their scale and autonomous resource management policies. Furthermore, shared resources are inconsistent, making it difficult to accurately forecast resource capacity. An effective VO’s resource profiling and modeling system can address these problems by forecasting resource characteristics and availability. This paper presents effective resource profiling and performance prediction models including Adaptive Filter-based Online Linear Regression (AFOLR) and Adaptive Filter-based Moving Average (AFMV) based on the linear difference equation combining past predicted values and recent profiled information, which aim to support large-scale applications in distributed scientific computing environments. We performed quantitative analysis and conducted microbenchmark experiments on a real multinational shared computing platform. Our evaluation results demonstrate that the proposed prediction schemes outperform well-known common approaches in terms of accuracy, and actually can help users in a shared resource environment to run their large-scale applications by effectively forecasting various computing resource capacity and performance.https://www.mdpi.com/2076-3417/12/9/4797distributed scientific computingsupercomputingcluster computingvirtual organizationresource profilingperformance prediction |
| spellingShingle | Md Azam Hossain Soonwook Hwang Jik-Soo Kim Resource Profiling and Performance Modeling for Distributed Scientific Computing Environments Applied Sciences distributed scientific computing supercomputing cluster computing virtual organization resource profiling performance prediction |
| title | Resource Profiling and Performance Modeling for Distributed Scientific Computing Environments |
| title_full | Resource Profiling and Performance Modeling for Distributed Scientific Computing Environments |
| title_fullStr | Resource Profiling and Performance Modeling for Distributed Scientific Computing Environments |
| title_full_unstemmed | Resource Profiling and Performance Modeling for Distributed Scientific Computing Environments |
| title_short | Resource Profiling and Performance Modeling for Distributed Scientific Computing Environments |
| title_sort | resource profiling and performance modeling for distributed scientific computing environments |
| topic | distributed scientific computing supercomputing cluster computing virtual organization resource profiling performance prediction |
| url | https://www.mdpi.com/2076-3417/12/9/4797 |
| work_keys_str_mv | AT mdazamhossain resourceprofilingandperformancemodelingfordistributedscientificcomputingenvironments AT soonwookhwang resourceprofilingandperformancemodelingfordistributedscientificcomputingenvironments AT jiksookim resourceprofilingandperformancemodelingfordistributedscientificcomputingenvironments |