A Tiny Transformer-Based Anomaly Detection Framework for IoT Solutions
The widespread proliferation of Internet of Things (IoT) devices has pushed for the development of novel transformer-based Anomaly Detection (AD) tools for an accurate monitoring of functionalities in industrial systems. Despite their outstanding performances, transformer models often rely on large...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Open Journal of Signal Processing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10319782/ |
| _version_ | 1797401045956034560 |
|---|---|
| author | Luca Barbieri Mattia Brambilla Mario Stefanutti Ciro Romano Niccolo De Carlo Manuel Roveri |
| author_facet | Luca Barbieri Mattia Brambilla Mario Stefanutti Ciro Romano Niccolo De Carlo Manuel Roveri |
| author_sort | Luca Barbieri |
| collection | DOAJ |
| description | The widespread proliferation of Internet of Things (IoT) devices has pushed for the development of novel transformer-based Anomaly Detection (AD) tools for an accurate monitoring of functionalities in industrial systems. Despite their outstanding performances, transformer models often rely on large Neural Networks (NNs) that are difficult to be executed by IoT devices due to their energy/computing constraints. This paper focuses on introducing tiny transformer-based AD tools to make them viable solutions for on-device AD. Starting from the state-of-the-art Anomaly Transformer (AT) model, which has been shown to provide accurate AD functionalities but it is characterized by high computational and memory demand, we propose a tiny AD framework that finds an optimized configuration of the AT model and uses it for devising a compressed version compatible with resource-constrained IoT systems. A knowledge distillation tool is developed to obtain a highly compressed AT model without degrading the AD performance. The proposed framework is firstly analyzed on four widely-adopted AD datasets and then assessed using data extracted from a real-world monitoring facility. The results show that the tiny AD tool provides a compressed AT model with a staggering 99.93% reduction in the number of trainable parameters compared to the original implementation (from 4.8 million to 3300 or 1400 according to the input dataset), without significantly compromising the accuracy in AD. Moreover, the compressed model substantially outperforms a popular Recurrent Neural Network (RNN)-based AD tool having a similar number of trainable weights as well as a conventional One-Class Support Vector Machine (OCSVM) algorithm. |
| first_indexed | 2024-03-09T02:04:13Z |
| format | Article |
| id | doaj.art-d671c13530944775b8238c63da0b1c02 |
| institution | Directory Open Access Journal |
| issn | 2644-1322 |
| language | English |
| last_indexed | 2024-03-09T02:04:13Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Open Journal of Signal Processing |
| spelling | doaj.art-d671c13530944775b8238c63da0b1c022023-12-08T00:07:59ZengIEEEIEEE Open Journal of Signal Processing2644-13222023-01-01446247810.1109/OJSP.2023.333375610319782A Tiny Transformer-Based Anomaly Detection Framework for IoT SolutionsLuca Barbieri0https://orcid.org/0000-0002-8948-8158Mattia Brambilla1https://orcid.org/0000-0001-5442-6507Mario Stefanutti2https://orcid.org/0009-0007-9281-6208Ciro Romano3https://orcid.org/0009-0007-6026-1797Niccolo De Carlo4https://orcid.org/0009-0000-0080-4357Manuel Roveri5https://orcid.org/0000-0001-7828-7687Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milan, ItalyDipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milan, ItalySensoworks, Rome, ItalySensoworks, Rome, ItalySensoworks, Rome, ItalyDipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milan, ItalyThe widespread proliferation of Internet of Things (IoT) devices has pushed for the development of novel transformer-based Anomaly Detection (AD) tools for an accurate monitoring of functionalities in industrial systems. Despite their outstanding performances, transformer models often rely on large Neural Networks (NNs) that are difficult to be executed by IoT devices due to their energy/computing constraints. This paper focuses on introducing tiny transformer-based AD tools to make them viable solutions for on-device AD. Starting from the state-of-the-art Anomaly Transformer (AT) model, which has been shown to provide accurate AD functionalities but it is characterized by high computational and memory demand, we propose a tiny AD framework that finds an optimized configuration of the AT model and uses it for devising a compressed version compatible with resource-constrained IoT systems. A knowledge distillation tool is developed to obtain a highly compressed AT model without degrading the AD performance. The proposed framework is firstly analyzed on four widely-adopted AD datasets and then assessed using data extracted from a real-world monitoring facility. The results show that the tiny AD tool provides a compressed AT model with a staggering 99.93% reduction in the number of trainable parameters compared to the original implementation (from 4.8 million to 3300 or 1400 according to the input dataset), without significantly compromising the accuracy in AD. Moreover, the compressed model substantially outperforms a popular Recurrent Neural Network (RNN)-based AD tool having a similar number of trainable weights as well as a conventional One-Class Support Vector Machine (OCSVM) algorithm.https://ieeexplore.ieee.org/document/10319782/Anomaly detectionmachine learningself-attentionknowledge distillationInternet of Thingstransformer |
| spellingShingle | Luca Barbieri Mattia Brambilla Mario Stefanutti Ciro Romano Niccolo De Carlo Manuel Roveri A Tiny Transformer-Based Anomaly Detection Framework for IoT Solutions IEEE Open Journal of Signal Processing Anomaly detection machine learning self-attention knowledge distillation Internet of Things transformer |
| title | A Tiny Transformer-Based Anomaly Detection Framework for IoT Solutions |
| title_full | A Tiny Transformer-Based Anomaly Detection Framework for IoT Solutions |
| title_fullStr | A Tiny Transformer-Based Anomaly Detection Framework for IoT Solutions |
| title_full_unstemmed | A Tiny Transformer-Based Anomaly Detection Framework for IoT Solutions |
| title_short | A Tiny Transformer-Based Anomaly Detection Framework for IoT Solutions |
| title_sort | tiny transformer based anomaly detection framework for iot solutions |
| topic | Anomaly detection machine learning self-attention knowledge distillation Internet of Things transformer |
| url | https://ieeexplore.ieee.org/document/10319782/ |
| work_keys_str_mv | AT lucabarbieri atinytransformerbasedanomalydetectionframeworkforiotsolutions AT mattiabrambilla atinytransformerbasedanomalydetectionframeworkforiotsolutions AT mariostefanutti atinytransformerbasedanomalydetectionframeworkforiotsolutions AT ciroromano atinytransformerbasedanomalydetectionframeworkforiotsolutions AT niccolodecarlo atinytransformerbasedanomalydetectionframeworkforiotsolutions AT manuelroveri atinytransformerbasedanomalydetectionframeworkforiotsolutions AT lucabarbieri tinytransformerbasedanomalydetectionframeworkforiotsolutions AT mattiabrambilla tinytransformerbasedanomalydetectionframeworkforiotsolutions AT mariostefanutti tinytransformerbasedanomalydetectionframeworkforiotsolutions AT ciroromano tinytransformerbasedanomalydetectionframeworkforiotsolutions AT niccolodecarlo tinytransformerbasedanomalydetectionframeworkforiotsolutions AT manuelroveri tinytransformerbasedanomalydetectionframeworkforiotsolutions |