Positive and unlabeled learning for anomaly detection
Anomaly detection is of great interest to big data applications but still remains a challenging problem for machine learning-based methods. For unsupervised learning, the performance may not be satisfactory due to the lack of label information while for supervised learning, it is difficult to acquir...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | English |
| Published: |
2018
|
| Subjects: | |
| Online Access: | http://hdl.handle.net/10356/75883 |
| _version_ | 1811692647779139584 |
|---|---|
| author | Zhang, Jiaqi |
| author2 | Tan Yap Peng |
| author_facet | Tan Yap Peng Zhang, Jiaqi |
| author_sort | Zhang, Jiaqi |
| collection | NTU |
| description | Anomaly detection is of great interest to big data applications but still remains a challenging problem for machine learning-based methods. For unsupervised learning, the performance may not be satisfactory due to the lack of label information while for supervised learning, it is difficult to acquire labeled anomaly data for training which is usually rare and diversely distributed.
To address the challenge, we propose a hybrid solution by applying Positive and Unlabeled (PU) Learning for anomaly detection problem. As a semi-supervised method, only normal (positive) data and unlabeled data (could be positive or negative) are required by the proposed method for anomaly detection. We start by using a linear model to extract the most reliable negative instances followed by an iterative self-learning process to update the classifier with different speeds based on the estimated positive class prior. Our proposed method is verified on several benchmark datasets and outperforms existing methods under different experiment settings. |
| first_indexed | 2024-10-01T06:39:07Z |
| format | Thesis |
| id | ntu-10356/75883 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T06:39:07Z |
| publishDate | 2018 |
| record_format | dspace |
| spelling | ntu-10356/758832023-07-04T17:15:06Z Positive and unlabeled learning for anomaly detection Zhang, Jiaqi Tan Yap Peng Yuan Junsong School of Electrical and Electronic Engineering DRNTU::Engineering::Electrical and electronic engineering Anomaly detection is of great interest to big data applications but still remains a challenging problem for machine learning-based methods. For unsupervised learning, the performance may not be satisfactory due to the lack of label information while for supervised learning, it is difficult to acquire labeled anomaly data for training which is usually rare and diversely distributed. To address the challenge, we propose a hybrid solution by applying Positive and Unlabeled (PU) Learning for anomaly detection problem. As a semi-supervised method, only normal (positive) data and unlabeled data (could be positive or negative) are required by the proposed method for anomaly detection. We start by using a linear model to extract the most reliable negative instances followed by an iterative self-learning process to update the classifier with different speeds based on the estimated positive class prior. Our proposed method is verified on several benchmark datasets and outperforms existing methods under different experiment settings. Master of Engineering 2018-07-09T01:37:12Z 2018-07-09T01:37:12Z 2018 Thesis Zhang, J. (2018). Positive and unlabeled learning for anomaly detection. Master's thesis, Nanyang Technological University, Singapore. http://hdl.handle.net/10356/75883 10.32657/10356/75883 en 77 p. application/pdf |
| spellingShingle | DRNTU::Engineering::Electrical and electronic engineering Zhang, Jiaqi Positive and unlabeled learning for anomaly detection |
| title | Positive and unlabeled learning for anomaly detection |
| title_full | Positive and unlabeled learning for anomaly detection |
| title_fullStr | Positive and unlabeled learning for anomaly detection |
| title_full_unstemmed | Positive and unlabeled learning for anomaly detection |
| title_short | Positive and unlabeled learning for anomaly detection |
| title_sort | positive and unlabeled learning for anomaly detection |
| topic | DRNTU::Engineering::Electrical and electronic engineering |
| url | http://hdl.handle.net/10356/75883 |
| work_keys_str_mv | AT zhangjiaqi positiveandunlabeledlearningforanomalydetection |