Adaptively Periodic I/O Scheduling for Concurrent HPC Applications
With the convergence of big data and HPC (high-performance computing), various machine learning applications and traditional large-scale simulations with a stochastically iterative I/O periodicity are running concurrently on HPC platforms, which poses more challenges on the scarcely shared I/O resou...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-04-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/11/9/1318 |
| _version_ | 1797505203790938112 |
|---|---|
| author | Benbo Zha Hong Shen |
| author_facet | Benbo Zha Hong Shen |
| author_sort | Benbo Zha |
| collection | DOAJ |
| description | With the convergence of big data and HPC (high-performance computing), various machine learning applications and traditional large-scale simulations with a stochastically iterative I/O periodicity are running concurrently on HPC platforms, which poses more challenges on the scarcely shared I/O resources due to the ever-growing data transfer demand. Currently the existing heuristic online and periodic offline I/O scheduling methods for traditional HPC applications with a fixed I/O periodicity are not suitable for the applications with stochastically iterative I/O periodicities, which are required to schedule the concurrent I/Os from different applications under I/O congestion. In this work, we propose an adaptively periodic I/O scheduling (APIO) method that optimizes the system efficiency and application dilation by taking the stochastically iterative I/O periodicity of the applications into account. We first build a periodic offline scheduling method within a specified duration to capture the iterative nature. After that, APIO adjusts the bandwidth allocation to resist stochasticity based on the actual length of the computing phrase. In the case where the specified duration does not satisfy the actual running requirements, the period length will be extended to adapt to the actual duration. Theoretical analysis and extensive simulations demonstrate the efficiency of our proposed I/O scheduling method over the existing online approach. |
| first_indexed | 2024-03-10T04:15:16Z |
| format | Article |
| id | doaj.art-a6b77a209198454f9564be624ad7f939 |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-10T04:15:16Z |
| publishDate | 2022-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-a6b77a209198454f9564be624ad7f9392023-11-23T08:01:52ZengMDPI AGElectronics2079-92922022-04-01119131810.3390/electronics11091318Adaptively Periodic I/O Scheduling for Concurrent HPC ApplicationsBenbo Zha0Hong Shen1School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, ChinaSchool of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, ChinaWith the convergence of big data and HPC (high-performance computing), various machine learning applications and traditional large-scale simulations with a stochastically iterative I/O periodicity are running concurrently on HPC platforms, which poses more challenges on the scarcely shared I/O resources due to the ever-growing data transfer demand. Currently the existing heuristic online and periodic offline I/O scheduling methods for traditional HPC applications with a fixed I/O periodicity are not suitable for the applications with stochastically iterative I/O periodicities, which are required to schedule the concurrent I/Os from different applications under I/O congestion. In this work, we propose an adaptively periodic I/O scheduling (APIO) method that optimizes the system efficiency and application dilation by taking the stochastically iterative I/O periodicity of the applications into account. We first build a periodic offline scheduling method within a specified duration to capture the iterative nature. After that, APIO adjusts the bandwidth allocation to resist stochasticity based on the actual length of the computing phrase. In the case where the specified duration does not satisfy the actual running requirements, the period length will be extended to adapt to the actual duration. Theoretical analysis and extensive simulations demonstrate the efficiency of our proposed I/O scheduling method over the existing online approach.https://www.mdpi.com/2079-9292/11/9/1318I/O schedulingperiodic I/O schedulingstochastic iterative applicationhigh-performance computing |
| spellingShingle | Benbo Zha Hong Shen Adaptively Periodic I/O Scheduling for Concurrent HPC Applications Electronics I/O scheduling periodic I/O scheduling stochastic iterative application high-performance computing |
| title | Adaptively Periodic I/O Scheduling for Concurrent HPC Applications |
| title_full | Adaptively Periodic I/O Scheduling for Concurrent HPC Applications |
| title_fullStr | Adaptively Periodic I/O Scheduling for Concurrent HPC Applications |
| title_full_unstemmed | Adaptively Periodic I/O Scheduling for Concurrent HPC Applications |
| title_short | Adaptively Periodic I/O Scheduling for Concurrent HPC Applications |
| title_sort | adaptively periodic i o scheduling for concurrent hpc applications |
| topic | I/O scheduling periodic I/O scheduling stochastic iterative application high-performance computing |
| url | https://www.mdpi.com/2079-9292/11/9/1318 |
| work_keys_str_mv | AT benbozha adaptivelyperiodicioschedulingforconcurrenthpcapplications AT hongshen adaptivelyperiodicioschedulingforconcurrenthpcapplications |