Strategies of data layout and cache writing for input-output optimization in high performance scientific computing: Applications to the forward electrocardiographic problem
Input-output (I/O) optimization at the low-level design of data layout on disk drastically impacts the efficiency of high performance computing (HPC) applications. However, such a low-level optimization is in general challenging, especially when using popular scientific file formats designed with an...
| Main Authors: | , , |
|---|---|
| Format: | Journal article |
| Published: |
Public Library of Science
2018
|
| _version_ | 1797079680896991232 |
|---|---|
| author | Cardone-Noott, L Rodriguez, B Bueno Orovio, A |
| author_facet | Cardone-Noott, L Rodriguez, B Bueno Orovio, A |
| author_sort | Cardone-Noott, L |
| collection | OXFORD |
| description | Input-output (I/O) optimization at the low-level design of data layout on disk drastically impacts the efficiency of high performance computing (HPC) applications. However, such a low-level optimization is in general challenging, especially when using popular scientific file formats designed with an emphasis on portability and flexibility. To reconcile these two aspects, we present a novel low-level data layout for HPC applications, fully independent of the number of dimensions in the dataset. The new data layout improves reading and writing efficiency in large HPC applications using many processors, and in particular during parallel post-processing. Furthermore, its combination with a cached write mode, in order to aggregate multiple writes into larger ones, substantially decreased the writing times of the proposed strategy. When applied to our simulation framework for the forward calculation of the human electrocardiogram, the combined strategy resulted in drastic improvements in I/O performance, of up to 40% in writing and 93–98% in reading for post-processing tasks. Given the generality of the proposed strategies and scientific file formats used, our results may represent significant improvements in I/O performance of HPC applications across multiple disciplines, reducing execution and post-processing times and leading to a more efficient use of HPC resource envelopes. |
| first_indexed | 2024-03-07T00:49:13Z |
| format | Journal article |
| id | oxford-uuid:85cad357-6a36-4aa6-be94-f0af92779299 |
| institution | University of Oxford |
| last_indexed | 2024-03-07T00:49:13Z |
| publishDate | 2018 |
| publisher | Public Library of Science |
| record_format | dspace |
| spelling | oxford-uuid:85cad357-6a36-4aa6-be94-f0af927792992022-03-26T21:59:48ZStrategies of data layout and cache writing for input-output optimization in high performance scientific computing: Applications to the forward electrocardiographic problemJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:85cad357-6a36-4aa6-be94-f0af92779299Symplectic Elements at OxfordPublic Library of Science2018Cardone-Noott, LRodriguez, BBueno Orovio, AInput-output (I/O) optimization at the low-level design of data layout on disk drastically impacts the efficiency of high performance computing (HPC) applications. However, such a low-level optimization is in general challenging, especially when using popular scientific file formats designed with an emphasis on portability and flexibility. To reconcile these two aspects, we present a novel low-level data layout for HPC applications, fully independent of the number of dimensions in the dataset. The new data layout improves reading and writing efficiency in large HPC applications using many processors, and in particular during parallel post-processing. Furthermore, its combination with a cached write mode, in order to aggregate multiple writes into larger ones, substantially decreased the writing times of the proposed strategy. When applied to our simulation framework for the forward calculation of the human electrocardiogram, the combined strategy resulted in drastic improvements in I/O performance, of up to 40% in writing and 93–98% in reading for post-processing tasks. Given the generality of the proposed strategies and scientific file formats used, our results may represent significant improvements in I/O performance of HPC applications across multiple disciplines, reducing execution and post-processing times and leading to a more efficient use of HPC resource envelopes. |
| spellingShingle | Cardone-Noott, L Rodriguez, B Bueno Orovio, A Strategies of data layout and cache writing for input-output optimization in high performance scientific computing: Applications to the forward electrocardiographic problem |
| title | Strategies of data layout and cache writing for input-output optimization in high performance scientific computing: Applications to the forward electrocardiographic problem |
| title_full | Strategies of data layout and cache writing for input-output optimization in high performance scientific computing: Applications to the forward electrocardiographic problem |
| title_fullStr | Strategies of data layout and cache writing for input-output optimization in high performance scientific computing: Applications to the forward electrocardiographic problem |
| title_full_unstemmed | Strategies of data layout and cache writing for input-output optimization in high performance scientific computing: Applications to the forward electrocardiographic problem |
| title_short | Strategies of data layout and cache writing for input-output optimization in high performance scientific computing: Applications to the forward electrocardiographic problem |
| title_sort | strategies of data layout and cache writing for input output optimization in high performance scientific computing applications to the forward electrocardiographic problem |
| work_keys_str_mv | AT cardonenoottl strategiesofdatalayoutandcachewritingforinputoutputoptimizationinhighperformancescientificcomputingapplicationstotheforwardelectrocardiographicproblem AT rodriguezb strategiesofdatalayoutandcachewritingforinputoutputoptimizationinhighperformancescientificcomputingapplicationstotheforwardelectrocardiographicproblem AT buenoorovioa strategiesofdatalayoutandcachewritingforinputoutputoptimizationinhighperformancescientificcomputingapplicationstotheforwardelectrocardiographicproblem |