Anomaly detection using intelligent analytics
Anomaly detection is an important issue which has been studied in diverse application domains. In computer networks, the anomalies could indicate possible attacks to the target server. Thus, it is necessary to establish fast detection of network anomalies in the server for the purpose of intrusion d...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | English |
| Published: |
2018
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/82275 http://hdl.handle.net/10220/46633 |
| _version_ | 1811694701575667712 |
|---|---|
| author | Chen, Zhaomin |
| author2 | Lee Bu Sung, Francis |
| author_facet | Lee Bu Sung, Francis Chen, Zhaomin |
| author_sort | Chen, Zhaomin |
| collection | NTU |
| description | Anomaly detection is an important issue which has been studied in diverse application domains. In computer networks, the anomalies could indicate possible attacks to the target server. Thus, it is necessary to establish fast detection of network anomalies in the server for the purpose of intrusion detection in the server side. In addition, in the bottleneck router, it is also necessary to detect and mitigate the Low-rate Denial of Service attacks. In the field of computer vision, anomaly detection is often referred to as outlier detection, which is also very important, especially for the deep learning techniques. Nowadays, deep learning has become a mainstream approach in many research pertaining to computer vision. Deep learning technique requires large datasets. Thus, outlier detection techniques are often applied here to clean the datasets. In this dissertation, the author focuses mainly on developing novel network anomaly detection and image outlier detection techniques.
First, the author proposes Multi-scale Principal Component Analysis (MSPCA) algorithm to detect network anomalies. By combining PCA with wavelet analysis, MSPCA algorithm is supposed to address the normal subspace contamination problem so as to increase the detection accuracy. In addition, the author introduces MMECE threshold and Bayesian-PCA to the conventional-MSPCA algorithm. Thus, this improved-MSPCA algorithm has lower time complexity and fewer parameters to set. Furthermore, the author proposes a novel network anomaly detection system which combines the improved-MSPCA algorithm with sketch data structure. Upon the detection, it can identify the anomalous source IP addresses. As the proposed system detects the anomalies based on flow-level features, it has a better detection performance, compared to other packet-based detection systems.
Second, the author proposes Power Spectrum Entropy (PSE) to detect low-rate denial of service (LDoS) attacks in the bottleneck routers. PSE combines signal processing techniques with information entropy. It first decomposes the traffic signals into the frequency domain. Then the entropy of the power spectrum density is calculated. Due to the periodic property of the LDoS attack signal, its corresponding PSE value is comparatively smaller than that of the normal TCP signal. Therefore, PSE can be applied to detect the LDoS attacks. Furthermore, the author embeds PSE into the Robust RED queuing algorithm to propose the PSE-based Robust RED (PRRED) LDoS mitigation algorithm. PRRED consists of two detection phases: arrival time based detection and PSE-based detection. Arrival time based detection phase is meant to detect the suspicious set of incoming packets. PSE-based detection further tests whether these suspicious packets are LDoS attacks or not, based on the calculation of PSE. Once the packet is determined to be an attack packet, PRRED will drop this packet. These two detection phases help PRRED algorithm detect and mitigate the LDoS attacks efficiently and swiftly.
The author next focuses on outlier detection in the high-dimensional image datasets. Existing outlier detection works are not suitable for the image datasets due to the high dimension. The author combines autoencoder with Adaboost algorithm to address this issue. In this Adaboost-Autoencoder (ADAE) algorithm, multiple weak autoencoders will be optimized based on training data and the corresponding learning weights. ADAE ensembles a sequence of weak autoencoder to fully capture the feature correlations of normal images, so that it can achieve better detection performance compared to the traditional detection techniques. The author also introduces the Sparse Group Lasso (SGL) constraint to the optimization objective of each autoencoder, so that the compact detection model can be obtained. In order to optimize the SGL regularized learning objective, the author combines Adagrad optimization algorithm with Proximal Gradient Descent (Ada-PGD). Furthermore, Evolutionary Multi-Objective (EMO) based optimization framework is proposed here to determine the optimal penalty factors of SGL for each autoencoder. The compact detection model derived proves to be very efficient in outlier detection. |
| first_indexed | 2024-10-01T07:11:46Z |
| format | Thesis |
| id | ntu-10356/82275 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T07:11:46Z |
| publishDate | 2018 |
| record_format | dspace |
| spelling | ntu-10356/822752020-06-22T06:43:05Z Anomaly detection using intelligent analytics Chen, Zhaomin Lee Bu Sung, Francis Yeo Chai Kiat School of Computer Science and Engineering DRNTU::Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence Anomaly detection is an important issue which has been studied in diverse application domains. In computer networks, the anomalies could indicate possible attacks to the target server. Thus, it is necessary to establish fast detection of network anomalies in the server for the purpose of intrusion detection in the server side. In addition, in the bottleneck router, it is also necessary to detect and mitigate the Low-rate Denial of Service attacks. In the field of computer vision, anomaly detection is often referred to as outlier detection, which is also very important, especially for the deep learning techniques. Nowadays, deep learning has become a mainstream approach in many research pertaining to computer vision. Deep learning technique requires large datasets. Thus, outlier detection techniques are often applied here to clean the datasets. In this dissertation, the author focuses mainly on developing novel network anomaly detection and image outlier detection techniques. First, the author proposes Multi-scale Principal Component Analysis (MSPCA) algorithm to detect network anomalies. By combining PCA with wavelet analysis, MSPCA algorithm is supposed to address the normal subspace contamination problem so as to increase the detection accuracy. In addition, the author introduces MMECE threshold and Bayesian-PCA to the conventional-MSPCA algorithm. Thus, this improved-MSPCA algorithm has lower time complexity and fewer parameters to set. Furthermore, the author proposes a novel network anomaly detection system which combines the improved-MSPCA algorithm with sketch data structure. Upon the detection, it can identify the anomalous source IP addresses. As the proposed system detects the anomalies based on flow-level features, it has a better detection performance, compared to other packet-based detection systems. Second, the author proposes Power Spectrum Entropy (PSE) to detect low-rate denial of service (LDoS) attacks in the bottleneck routers. PSE combines signal processing techniques with information entropy. It first decomposes the traffic signals into the frequency domain. Then the entropy of the power spectrum density is calculated. Due to the periodic property of the LDoS attack signal, its corresponding PSE value is comparatively smaller than that of the normal TCP signal. Therefore, PSE can be applied to detect the LDoS attacks. Furthermore, the author embeds PSE into the Robust RED queuing algorithm to propose the PSE-based Robust RED (PRRED) LDoS mitigation algorithm. PRRED consists of two detection phases: arrival time based detection and PSE-based detection. Arrival time based detection phase is meant to detect the suspicious set of incoming packets. PSE-based detection further tests whether these suspicious packets are LDoS attacks or not, based on the calculation of PSE. Once the packet is determined to be an attack packet, PRRED will drop this packet. These two detection phases help PRRED algorithm detect and mitigate the LDoS attacks efficiently and swiftly. The author next focuses on outlier detection in the high-dimensional image datasets. Existing outlier detection works are not suitable for the image datasets due to the high dimension. The author combines autoencoder with Adaboost algorithm to address this issue. In this Adaboost-Autoencoder (ADAE) algorithm, multiple weak autoencoders will be optimized based on training data and the corresponding learning weights. ADAE ensembles a sequence of weak autoencoder to fully capture the feature correlations of normal images, so that it can achieve better detection performance compared to the traditional detection techniques. The author also introduces the Sparse Group Lasso (SGL) constraint to the optimization objective of each autoencoder, so that the compact detection model can be obtained. In order to optimize the SGL regularized learning objective, the author combines Adagrad optimization algorithm with Proximal Gradient Descent (Ada-PGD). Furthermore, Evolutionary Multi-Objective (EMO) based optimization framework is proposed here to determine the optimal penalty factors of SGL for each autoencoder. The compact detection model derived proves to be very efficient in outlier detection. Doctor of Philosophy 2018-11-13T04:29:57Z 2019-12-06T14:52:18Z 2018-11-13T04:29:57Z 2019-12-06T14:52:18Z 2018 Thesis Chen, Z. (2018). Anomaly detection using intelligent analytics. Doctoral thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/82275 http://hdl.handle.net/10220/46633 10.32657/10220/46633 en 163 p. application/pdf |
| spellingShingle | DRNTU::Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence Chen, Zhaomin Anomaly detection using intelligent analytics |
| title | Anomaly detection using intelligent analytics |
| title_full | Anomaly detection using intelligent analytics |
| title_fullStr | Anomaly detection using intelligent analytics |
| title_full_unstemmed | Anomaly detection using intelligent analytics |
| title_short | Anomaly detection using intelligent analytics |
| title_sort | anomaly detection using intelligent analytics |
| topic | DRNTU::Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence |
| url | https://hdl.handle.net/10356/82275 http://hdl.handle.net/10220/46633 |
| work_keys_str_mv | AT chenzhaomin anomalydetectionusingintelligentanalytics |