Characterizing the Impact of Data-Damaged Models on Generalization Strength in Intrusion Detection
Generalization is a longstanding assumption in articles concerning network intrusion detection through machine learning. Novel techniques are frequently proposed and validated based on the improvement they attain when classifying one or more of the existing datasets. The necessary follow-up question...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-04-01
|
| Series: | Journal of Cybersecurity and Privacy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2624-800X/3/2/8 |
| _version_ | 1797594075181875200 |
|---|---|
| author | Laurens D’hooge Miel Verkerken Tim Wauters Filip De Turck Bruno Volckaert |
| author_facet | Laurens D’hooge Miel Verkerken Tim Wauters Filip De Turck Bruno Volckaert |
| author_sort | Laurens D’hooge |
| collection | DOAJ |
| description | Generalization is a longstanding assumption in articles concerning network intrusion detection through machine learning. Novel techniques are frequently proposed and validated based on the improvement they attain when classifying one or more of the existing datasets. The necessary follow-up question of whether this increased performance in classification is meaningful outside of the dataset(s) is almost never investigated. This lacuna is in part due to the sparse dataset landscape in network intrusion detection and the complexity of creating new data. The introduction of two recent datasets, namely CIC-IDS2017 and CSE-CIC-IDS2018, opened up the possibility of testing generalization capability within similar academic datasets. This work investigates how well models from different algorithmic families, pretrained on CICIDS2017, are able to classify the samples in CSE-CIC-IDS2018 without retraining. Earlier work has shown how robust these models are to data reduction when classifying state-of-the-art datasets. This work experimentally demonstrates that the implicit assumption that strong generalized performance naturally follows from strong performance on a specific dataset is largely erroneous. The supervised machine learning algorithms suffered flat losses in classification performance ranging from 0 to 50% (depending on the attack class under test). For non-network-centric attack classes, this performance regression is most pronounced, but even the less affected models that classify the network-centric attack classes still show defects. Current implementations of intrusion detection systems (IDSs) with supervised machine learning (ML) as a core building block are thus very likely flawed if they have been validated on the academic datasets, without the consideration for their general performance on other academic or real-world datasets. |
| first_indexed | 2024-03-11T02:18:34Z |
| format | Article |
| id | doaj.art-955c9fb9e402420190ce93b89dae044f |
| institution | Directory Open Access Journal |
| issn | 2624-800X |
| language | English |
| last_indexed | 2024-03-11T02:18:34Z |
| publishDate | 2023-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Journal of Cybersecurity and Privacy |
| spelling | doaj.art-955c9fb9e402420190ce93b89dae044f2023-11-18T11:01:58ZengMDPI AGJournal of Cybersecurity and Privacy2624-800X2023-04-013211814410.3390/jcp3020008Characterizing the Impact of Data-Damaged Models on Generalization Strength in Intrusion DetectionLaurens D’hooge0Miel Verkerken1Tim Wauters2Filip De Turck3Bruno Volckaert4IDLab-Imec, Department of Information Technology, Ghent University, 9052 Gent, BelgiumIDLab-Imec, Department of Information Technology, Ghent University, 9052 Gent, BelgiumIDLab-Imec, Department of Information Technology, Ghent University, 9052 Gent, BelgiumIDLab-Imec, Department of Information Technology, Ghent University, 9052 Gent, BelgiumIDLab-Imec, Department of Information Technology, Ghent University, 9052 Gent, BelgiumGeneralization is a longstanding assumption in articles concerning network intrusion detection through machine learning. Novel techniques are frequently proposed and validated based on the improvement they attain when classifying one or more of the existing datasets. The necessary follow-up question of whether this increased performance in classification is meaningful outside of the dataset(s) is almost never investigated. This lacuna is in part due to the sparse dataset landscape in network intrusion detection and the complexity of creating new data. The introduction of two recent datasets, namely CIC-IDS2017 and CSE-CIC-IDS2018, opened up the possibility of testing generalization capability within similar academic datasets. This work investigates how well models from different algorithmic families, pretrained on CICIDS2017, are able to classify the samples in CSE-CIC-IDS2018 without retraining. Earlier work has shown how robust these models are to data reduction when classifying state-of-the-art datasets. This work experimentally demonstrates that the implicit assumption that strong generalized performance naturally follows from strong performance on a specific dataset is largely erroneous. The supervised machine learning algorithms suffered flat losses in classification performance ranging from 0 to 50% (depending on the attack class under test). For non-network-centric attack classes, this performance regression is most pronounced, but even the less affected models that classify the network-centric attack classes still show defects. Current implementations of intrusion detection systems (IDSs) with supervised machine learning (ML) as a core building block are thus very likely flawed if they have been validated on the academic datasets, without the consideration for their general performance on other academic or real-world datasets.https://www.mdpi.com/2624-800X/3/2/8intrusion detectionnetwork securitysupervised machine learninggeneralization strengthCIC-IDS2017CSE-CIC-IDS2018 |
| spellingShingle | Laurens D’hooge Miel Verkerken Tim Wauters Filip De Turck Bruno Volckaert Characterizing the Impact of Data-Damaged Models on Generalization Strength in Intrusion Detection Journal of Cybersecurity and Privacy intrusion detection network security supervised machine learning generalization strength CIC-IDS2017 CSE-CIC-IDS2018 |
| title | Characterizing the Impact of Data-Damaged Models on Generalization Strength in Intrusion Detection |
| title_full | Characterizing the Impact of Data-Damaged Models on Generalization Strength in Intrusion Detection |
| title_fullStr | Characterizing the Impact of Data-Damaged Models on Generalization Strength in Intrusion Detection |
| title_full_unstemmed | Characterizing the Impact of Data-Damaged Models on Generalization Strength in Intrusion Detection |
| title_short | Characterizing the Impact of Data-Damaged Models on Generalization Strength in Intrusion Detection |
| title_sort | characterizing the impact of data damaged models on generalization strength in intrusion detection |
| topic | intrusion detection network security supervised machine learning generalization strength CIC-IDS2017 CSE-CIC-IDS2018 |
| url | https://www.mdpi.com/2624-800X/3/2/8 |
| work_keys_str_mv | AT laurensdhooge characterizingtheimpactofdatadamagedmodelsongeneralizationstrengthinintrusiondetection AT mielverkerken characterizingtheimpactofdatadamagedmodelsongeneralizationstrengthinintrusiondetection AT timwauters characterizingtheimpactofdatadamagedmodelsongeneralizationstrengthinintrusiondetection AT filipdeturck characterizingtheimpactofdatadamagedmodelsongeneralizationstrengthinintrusiondetection AT brunovolckaert characterizingtheimpactofdatadamagedmodelsongeneralizationstrengthinintrusiondetection |