Practical Considerations for Accuracy Evaluation in Sensor-Based Machine Learning and Deep Learning
Accuracy evaluation in machine learning is based on the split of data into a training set and a test set. This critical step is applied to develop machine learning models including models based on sensor data. For sensor-based problems, comparing the accuracy of machine learning models using the tra...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2019-08-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/19/16/3491 |
| _version_ | 1811185040601645056 |
|---|---|
| author | Issam Hammad Kamal El-Sankary |
| author_facet | Issam Hammad Kamal El-Sankary |
| author_sort | Issam Hammad |
| collection | DOAJ |
| description | Accuracy evaluation in machine learning is based on the split of data into a training set and a test set. This critical step is applied to develop machine learning models including models based on sensor data. For sensor-based problems, comparing the accuracy of machine learning models using the train/test split provides only a baseline comparison in ideal situations. Such comparisons won’t consider practical production problems that can impact the inference accuracy such as the sensors’ thermal noise, performance with lower inference quantization, and tolerance to sensor failure. Therefore, this paper proposes a set of practical tests that can be applied when comparing the accuracy of machine learning models for sensor-based problems. First, the impact of the sensors’ thermal noise on the models’ inference accuracy was simulated. Machine learning algorithms have different levels of error resilience to thermal noise, as will be presented. Second, the models’ accuracy using lower inference quantization was compared. Lowering inference quantization leads to lowering the analog-to-digital converter (ADC) resolution which is cost-effective in embedded designs. Moreover, in custom designs, analog-to-digital converters’ (ADCs) effective number of bits (ENOB) is usually lower than the ideal number of bits due to various design factors. Therefore, it is practical to compare models’ accuracy using lower inference quantization. Third, the models’ accuracy tolerance to sensor failure was evaluated and compared. For this study, University of California Irvine (UCI) ‘Daily and Sports Activities’ dataset was used to present these practical tests and their impact on model selection. |
| first_indexed | 2024-04-11T13:22:27Z |
| format | Article |
| id | doaj.art-9385c39cff664aa3ac0870d57dda4b51 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-04-11T13:22:27Z |
| publishDate | 2019-08-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-9385c39cff664aa3ac0870d57dda4b512022-12-22T04:22:09ZengMDPI AGSensors1424-82202019-08-011916349110.3390/s19163491s19163491Practical Considerations for Accuracy Evaluation in Sensor-Based Machine Learning and Deep LearningIssam Hammad0Kamal El-Sankary1Department of Electrical and Computer Engineering, Dalhousie University, Halifax, NS B3H 4R2, CanadaDepartment of Electrical and Computer Engineering, Dalhousie University, Halifax, NS B3H 4R2, CanadaAccuracy evaluation in machine learning is based on the split of data into a training set and a test set. This critical step is applied to develop machine learning models including models based on sensor data. For sensor-based problems, comparing the accuracy of machine learning models using the train/test split provides only a baseline comparison in ideal situations. Such comparisons won’t consider practical production problems that can impact the inference accuracy such as the sensors’ thermal noise, performance with lower inference quantization, and tolerance to sensor failure. Therefore, this paper proposes a set of practical tests that can be applied when comparing the accuracy of machine learning models for sensor-based problems. First, the impact of the sensors’ thermal noise on the models’ inference accuracy was simulated. Machine learning algorithms have different levels of error resilience to thermal noise, as will be presented. Second, the models’ accuracy using lower inference quantization was compared. Lowering inference quantization leads to lowering the analog-to-digital converter (ADC) resolution which is cost-effective in embedded designs. Moreover, in custom designs, analog-to-digital converters’ (ADCs) effective number of bits (ENOB) is usually lower than the ideal number of bits due to various design factors. Therefore, it is practical to compare models’ accuracy using lower inference quantization. Third, the models’ accuracy tolerance to sensor failure was evaluated and compared. For this study, University of California Irvine (UCI) ‘Daily and Sports Activities’ dataset was used to present these practical tests and their impact on model selection.https://www.mdpi.com/1424-8220/19/16/3491ADCdeep learningedge artificial intelligence (AI)ENOBmachine learninglow powerlow quantizationsensor failuresensor fusionthermal noise |
| spellingShingle | Issam Hammad Kamal El-Sankary Practical Considerations for Accuracy Evaluation in Sensor-Based Machine Learning and Deep Learning Sensors ADC deep learning edge artificial intelligence (AI) ENOB machine learning low power low quantization sensor failure sensor fusion thermal noise |
| title | Practical Considerations for Accuracy Evaluation in Sensor-Based Machine Learning and Deep Learning |
| title_full | Practical Considerations for Accuracy Evaluation in Sensor-Based Machine Learning and Deep Learning |
| title_fullStr | Practical Considerations for Accuracy Evaluation in Sensor-Based Machine Learning and Deep Learning |
| title_full_unstemmed | Practical Considerations for Accuracy Evaluation in Sensor-Based Machine Learning and Deep Learning |
| title_short | Practical Considerations for Accuracy Evaluation in Sensor-Based Machine Learning and Deep Learning |
| title_sort | practical considerations for accuracy evaluation in sensor based machine learning and deep learning |
| topic | ADC deep learning edge artificial intelligence (AI) ENOB machine learning low power low quantization sensor failure sensor fusion thermal noise |
| url | https://www.mdpi.com/1424-8220/19/16/3491 |
| work_keys_str_mv | AT issamhammad practicalconsiderationsforaccuracyevaluationinsensorbasedmachinelearninganddeeplearning AT kamalelsankary practicalconsiderationsforaccuracyevaluationinsensorbasedmachinelearninganddeeplearning |