Design methodology for deep out-of-distribution detectors in real-time cyber-physical systems
When machine learning (ML) models are supplied with data outside their training distribution, they are more likely to make inaccurate predictions; in a cyber-physical system (CPS), this could lead to catastrophic system failure. To mitigate this risk, an out-of-distribution (OOD) detector can run in...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Conference Paper |
| Language: | English |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/178682 |
| _version_ | 1811687836487778304 |
|---|---|
| author | Yuhas, Michael Ng, Daniel Jun Xian Easwaran, Arvind |
| author2 | College of Computing and Data Science |
| author_facet | College of Computing and Data Science Yuhas, Michael Ng, Daniel Jun Xian Easwaran, Arvind |
| author_sort | Yuhas, Michael |
| collection | NTU |
| description | When machine learning (ML) models are supplied with data outside their training distribution, they are more likely to make inaccurate predictions; in a cyber-physical system (CPS), this could lead to catastrophic system failure. To mitigate this risk, an out-of-distribution (OOD) detector can run in parallel with an ML model and flag inputs that could lead to undesirable outcomes. Although OOD detectors have been well studied in terms of accuracy, there has been less focus on deployment to resource constrained CPSs. In this study, a design methodology is proposed to tune deep OOD detectors to meet the accuracy and response time requirements of embedded applications. The methodology uses genetic algorithms to optimize the detector's preprocessing pipeline and selects a quantization method that balances robustness and response time. It also identifies several candidate task graphs under the Robot Operating System (ROS) for deployment of the selected design. The methodology is demonstrated on two variational autoencoder based OOD detectors from the literature on two embedded platforms. Insights into the trade-offs that occur during the design process are provided, and it is shown that this design methodology can lead to a drastic reduction in response time in relation to an unoptimized OOD detector while maintaining comparable accuracy. |
| first_indexed | 2024-10-01T05:22:38Z |
| format | Conference Paper |
| id | ntu-10356/178682 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T05:22:38Z |
| publishDate | 2024 |
| record_format | dspace |
| spelling | ntu-10356/1786822024-07-10T03:11:05Z Design methodology for deep out-of-distribution detectors in real-time cyber-physical systems Yuhas, Michael Ng, Daniel Jun Xian Easwaran, Arvind College of Computing and Data Science School of Computer Science and Engineering 2022 IEEE 28th International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA) Computer and Information Science Quantization (signal) Design methodology When machine learning (ML) models are supplied with data outside their training distribution, they are more likely to make inaccurate predictions; in a cyber-physical system (CPS), this could lead to catastrophic system failure. To mitigate this risk, an out-of-distribution (OOD) detector can run in parallel with an ML model and flag inputs that could lead to undesirable outcomes. Although OOD detectors have been well studied in terms of accuracy, there has been less focus on deployment to resource constrained CPSs. In this study, a design methodology is proposed to tune deep OOD detectors to meet the accuracy and response time requirements of embedded applications. The methodology uses genetic algorithms to optimize the detector's preprocessing pipeline and selects a quantization method that balances robustness and response time. It also identifies several candidate task graphs under the Robot Operating System (ROS) for deployment of the selected design. The methodology is demonstrated on two variational autoencoder based OOD detectors from the literature on two embedded platforms. Insights into the trade-offs that occur during the design process are provided, and it is shown that this design methodology can lead to a drastic reduction in response time in relation to an unoptimized OOD detector while maintaining comparable accuracy. Ministry of Education (MOE) Submitted/Accepted version This research was funded in part by MoE, Singapore, Tier-2 grant number MOE2019-T2-2-040. 2024-07-03T07:56:02Z 2024-07-03T07:56:02Z 2022 Conference Paper Yuhas, M., Ng, D. J. X. & Easwaran, A. (2022). Design methodology for deep out-of-distribution detectors in real-time cyber-physical systems. 2022 IEEE 28th International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA), 180-185. https://dx.doi.org/10.1109/RTCSA55878.2022.00025 9781665453448 https://hdl.handle.net/10356/178682 10.1109/RTCSA55878.2022.00025 2-s2.0-85141309374 180 185 en MOE2019-T2-2-040 10.21979/N9/UZY54Q © 2022 IEEE. All rights reserved. This article may be downloaded for personal use only. Any other use requires prior permission of the copyright holder. The Version of Record is available online at http://doi.org/10.1109/RTCSA55878.2022.00025. application/pdf |
| spellingShingle | Computer and Information Science Quantization (signal) Design methodology Yuhas, Michael Ng, Daniel Jun Xian Easwaran, Arvind Design methodology for deep out-of-distribution detectors in real-time cyber-physical systems |
| title | Design methodology for deep out-of-distribution detectors in real-time cyber-physical systems |
| title_full | Design methodology for deep out-of-distribution detectors in real-time cyber-physical systems |
| title_fullStr | Design methodology for deep out-of-distribution detectors in real-time cyber-physical systems |
| title_full_unstemmed | Design methodology for deep out-of-distribution detectors in real-time cyber-physical systems |
| title_short | Design methodology for deep out-of-distribution detectors in real-time cyber-physical systems |
| title_sort | design methodology for deep out of distribution detectors in real time cyber physical systems |
| topic | Computer and Information Science Quantization (signal) Design methodology |
| url | https://hdl.handle.net/10356/178682 |
| work_keys_str_mv | AT yuhasmichael designmethodologyfordeepoutofdistributiondetectorsinrealtimecyberphysicalsystems AT ngdanieljunxian designmethodologyfordeepoutofdistributiondetectorsinrealtimecyberphysicalsystems AT easwaranarvind designmethodologyfordeepoutofdistributiondetectorsinrealtimecyberphysicalsystems |