Unsupervised KPIs-Based Clustering of Jobs in HPC Data Centers
Performance analysis is an essential task in high-performance computing (HPC) systems, and it is applied for different purposes, such as anomaly detection, optimal resource allocation, and budget planning. HPC monitoring tasks generate a huge number of key performance indicators (KPIs) to supervise...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-07-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/20/15/4111 |
| _version_ | 1827712473476628480 |
|---|---|
| author | Mohamed S. Halawa Rebeca P. Díaz Redondo Ana Fernández Vilas |
| author_facet | Mohamed S. Halawa Rebeca P. Díaz Redondo Ana Fernández Vilas |
| author_sort | Mohamed S. Halawa |
| collection | DOAJ |
| description | Performance analysis is an essential task in high-performance computing (HPC) systems, and it is applied for different purposes, such as anomaly detection, optimal resource allocation, and budget planning. HPC monitoring tasks generate a huge number of key performance indicators (KPIs) to supervise the status of the jobs running in these systems. KPIs give data about CPU usage, memory usage, network (interface) traffic, or other sensors that monitor the hardware. Analyzing this data, it is possible to obtain insightful information about running jobs, such as their characteristics, performance, and failures. The main contribution in this paper was to identify which metric/s (KPIs) is/are the most appropriate to identify/classify different types of jobs according to their behavior in the HPC system. With this aim, we had applied different clustering techniques (partition and hierarchical clustering algorithms) using a real dataset from the Galician computation center (CESGA). We concluded that (i) those metrics (KPIs) related to the network (interface) traffic monitoring provided the best cohesion and separation to cluster HPC jobs, and (ii) hierarchical clustering algorithms were the most suitable for this task. Our approach was validated using a different real dataset from the same HPC center. |
| first_indexed | 2024-03-10T18:15:18Z |
| format | Article |
| id | doaj.art-b51f4ffb865f46a9890269028537cca0 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T18:15:18Z |
| publishDate | 2020-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-b51f4ffb865f46a9890269028537cca02023-11-20T07:46:10ZengMDPI AGSensors1424-82202020-07-012015411110.3390/s20154111Unsupervised KPIs-Based Clustering of Jobs in HPC Data CentersMohamed S. Halawa0Rebeca P. Díaz Redondo1Ana Fernández Vilas2Business Information Systems Department, Arab Academy for Science Technology and Maritime Transport, Cairo 11799, EgyptInformation & Computing Lab, AtlanTTIC Research Center, Universidade de Vigo, 36310 Vigo, SpainInformation & Computing Lab, AtlanTTIC Research Center, Universidade de Vigo, 36310 Vigo, SpainPerformance analysis is an essential task in high-performance computing (HPC) systems, and it is applied for different purposes, such as anomaly detection, optimal resource allocation, and budget planning. HPC monitoring tasks generate a huge number of key performance indicators (KPIs) to supervise the status of the jobs running in these systems. KPIs give data about CPU usage, memory usage, network (interface) traffic, or other sensors that monitor the hardware. Analyzing this data, it is possible to obtain insightful information about running jobs, such as their characteristics, performance, and failures. The main contribution in this paper was to identify which metric/s (KPIs) is/are the most appropriate to identify/classify different types of jobs according to their behavior in the HPC system. With this aim, we had applied different clustering techniques (partition and hierarchical clustering algorithms) using a real dataset from the Galician computation center (CESGA). We concluded that (i) those metrics (KPIs) related to the network (interface) traffic monitoring provided the best cohesion and separation to cluster HPC jobs, and (ii) hierarchical clustering algorithms were the most suitable for this task. Our approach was validated using a different real dataset from the same HPC center.https://www.mdpi.com/1424-8220/20/15/4111high-performance computingtime series analysisunsupervised learningclustering |
| spellingShingle | Mohamed S. Halawa Rebeca P. Díaz Redondo Ana Fernández Vilas Unsupervised KPIs-Based Clustering of Jobs in HPC Data Centers Sensors high-performance computing time series analysis unsupervised learning clustering |
| title | Unsupervised KPIs-Based Clustering of Jobs in HPC Data Centers |
| title_full | Unsupervised KPIs-Based Clustering of Jobs in HPC Data Centers |
| title_fullStr | Unsupervised KPIs-Based Clustering of Jobs in HPC Data Centers |
| title_full_unstemmed | Unsupervised KPIs-Based Clustering of Jobs in HPC Data Centers |
| title_short | Unsupervised KPIs-Based Clustering of Jobs in HPC Data Centers |
| title_sort | unsupervised kpis based clustering of jobs in hpc data centers |
| topic | high-performance computing time series analysis unsupervised learning clustering |
| url | https://www.mdpi.com/1424-8220/20/15/4111 |
| work_keys_str_mv | AT mohamedshalawa unsupervisedkpisbasedclusteringofjobsinhpcdatacenters AT rebecapdiazredondo unsupervisedkpisbasedclusteringofjobsinhpcdatacenters AT anafernandezvilas unsupervisedkpisbasedclusteringofjobsinhpcdatacenters |