A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings
Artificial Intelligence (AI) is one of the hottest topics in our society, especially when it comes to solving data-analysis problems. Industry are conducting their digital shifts, and AI is becoming a cornerstone technology for making decisions out of the huge amount of (sensors-based) data availabl...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-09-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/21/18/6195 |
| _version_ | 1797517182283808768 |
|---|---|
| author | Paul-Lou Benedick Jérémy Robert Yves Le Traon |
| author_facet | Paul-Lou Benedick Jérémy Robert Yves Le Traon |
| author_sort | Paul-Lou Benedick |
| collection | DOAJ |
| description | Artificial Intelligence (AI) is one of the hottest topics in our society, especially when it comes to solving data-analysis problems. Industry are conducting their digital shifts, and AI is becoming a cornerstone technology for making decisions out of the huge amount of (sensors-based) data available in the production floor. However, such technology may be disappointing when deployed in real conditions. Despite good theoretical performances and high accuracy when trained and tested in isolation, a Machine-Learning (M-L) model may provide degraded performances in real conditions. One reason may be fragility in treating properly unexpected or perturbed data. The objective of the paper is therefore to study the robustness of seven M-L and Deep-Learning (D-L) algorithms, when classifying univariate time-series under perturbations. A systematic approach is proposed for artificially injecting perturbations in the data and for evaluating the robustness of the models. This approach focuses on two perturbations that are likely to happen during data collection. Our experimental study, conducted on twenty sensors’ datasets from the public University of California Riverside (UCR) repository, shows a great disparity of the models’ robustness under data quality degradation. Those results are used to analyse whether the impact of such robustness can be predictable—thanks to decision trees—which would prevent us from testing all perturbations scenarios. Our study shows that building such a predictor is not straightforward and suggests that such a systematic approach needs to be used for evaluating AI models’ robustness. |
| first_indexed | 2024-03-10T07:13:11Z |
| format | Article |
| id | doaj.art-12dab19a4d3145bab1dc6799c7772e80 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T07:13:11Z |
| publishDate | 2021-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-12dab19a4d3145bab1dc6799c7772e802023-11-22T15:13:04ZengMDPI AGSensors1424-82202021-09-012118619510.3390/s21186195A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial SettingsPaul-Lou Benedick0Jérémy Robert1Yves Le Traon2Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg, 6 Rue Richard Coudenhove-Kalergi, L-1359 Luxembourg, LuxembourgCebi Luxembourg S.A, 30 rue J.F. Kennedy, L-7327 Steinsel, LuxembourgInterdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg, 6 Rue Richard Coudenhove-Kalergi, L-1359 Luxembourg, LuxembourgArtificial Intelligence (AI) is one of the hottest topics in our society, especially when it comes to solving data-analysis problems. Industry are conducting their digital shifts, and AI is becoming a cornerstone technology for making decisions out of the huge amount of (sensors-based) data available in the production floor. However, such technology may be disappointing when deployed in real conditions. Despite good theoretical performances and high accuracy when trained and tested in isolation, a Machine-Learning (M-L) model may provide degraded performances in real conditions. One reason may be fragility in treating properly unexpected or perturbed data. The objective of the paper is therefore to study the robustness of seven M-L and Deep-Learning (D-L) algorithms, when classifying univariate time-series under perturbations. A systematic approach is proposed for artificially injecting perturbations in the data and for evaluating the robustness of the models. This approach focuses on two perturbations that are likely to happen during data collection. Our experimental study, conducted on twenty sensors’ datasets from the public University of California Riverside (UCR) repository, shows a great disparity of the models’ robustness under data quality degradation. Those results are used to analyse whether the impact of such robustness can be predictable—thanks to decision trees—which would prevent us from testing all perturbations scenarios. Our study shows that building such a predictor is not straightforward and suggests that such a systematic approach needs to be used for evaluating AI models’ robustness.https://www.mdpi.com/1424-8220/21/18/6195time series classificationartificial intelligence robustnessindustrial internet of thingsadversarial |
| spellingShingle | Paul-Lou Benedick Jérémy Robert Yves Le Traon A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings Sensors time series classification artificial intelligence robustness industrial internet of things adversarial |
| title | A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings |
| title_full | A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings |
| title_fullStr | A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings |
| title_full_unstemmed | A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings |
| title_short | A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings |
| title_sort | systematic approach for evaluating artificial intelligence models in industrial settings |
| topic | time series classification artificial intelligence robustness industrial internet of things adversarial |
| url | https://www.mdpi.com/1424-8220/21/18/6195 |
| work_keys_str_mv | AT paulloubenedick asystematicapproachforevaluatingartificialintelligencemodelsinindustrialsettings AT jeremyrobert asystematicapproachforevaluatingartificialintelligencemodelsinindustrialsettings AT yvesletraon asystematicapproachforevaluatingartificialintelligencemodelsinindustrialsettings AT paulloubenedick systematicapproachforevaluatingartificialintelligencemodelsinindustrialsettings AT jeremyrobert systematicapproachforevaluatingartificialintelligencemodelsinindustrialsettings AT yvesletraon systematicapproachforevaluatingartificialintelligencemodelsinindustrialsettings |