A Novel Unsupervised Video Anomaly Detection Framework Based on Optical Flow Reconstruction and Erased Frame Prediction
Reconstruction-based and prediction-based approaches are widely used for video anomaly detection (VAD) in smart city surveillance applications. However, neither of these approaches can effectively utilize the rich contextual information that exists in videos, which makes it difficult to accurately p...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-05-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/10/4828 |
| _version_ | 1797598402908782592 |
|---|---|
| author | Heqing Huang Bing Zhao Fei Gao Penghui Chen Jun Wang Amir Hussain |
| author_facet | Heqing Huang Bing Zhao Fei Gao Penghui Chen Jun Wang Amir Hussain |
| author_sort | Heqing Huang |
| collection | DOAJ |
| description | Reconstruction-based and prediction-based approaches are widely used for video anomaly detection (VAD) in smart city surveillance applications. However, neither of these approaches can effectively utilize the rich contextual information that exists in videos, which makes it difficult to accurately perceive anomalous activities. In this paper, we exploit the idea of a training model based on the “Cloze Test” strategy in natural language processing (NLP) and introduce a novel unsupervised learning framework to encode both motion and appearance information at an object level. Specifically, to store the normal modes of video activity reconstructions, we first design an optical stream memory network with skip connections. Secondly, we build a space–time cube (STC) for use as the basic processing unit of the model and erase a patch in the STC to form the frame to be reconstructed. This enables a so-called ”incomplete event (IE)” to be completed. On this basis, a conditional autoencoder is utilized to capture the high correspondence between optical flow and STC. The model predicts erased patches in IEs based on the context of the front and back frames. Finally, we employ a generating adversarial network (GAN)-based training method to improve the performance of VAD. By distinguishing the predicted erased optical flow and erased video frame, the anomaly detection results are shown to be more reliable with our proposed method which can help reconstruct the original video in IE. Comparative experiments conducted on the benchmark UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets demonstrate AUROC scores reaching 97.7%, 89.7%, and 75.8%, respectively. |
| first_indexed | 2024-03-11T03:20:43Z |
| format | Article |
| id | doaj.art-bb305c895e6a4f0290d85d024a2bdf77 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-11T03:20:43Z |
| publishDate | 2023-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-bb305c895e6a4f0290d85d024a2bdf772023-11-18T03:13:13ZengMDPI AGSensors1424-82202023-05-012310482810.3390/s23104828A Novel Unsupervised Video Anomaly Detection Framework Based on Optical Flow Reconstruction and Erased Frame PredictionHeqing Huang0Bing Zhao1Fei Gao2Penghui Chen3Jun Wang4Amir Hussain5School of Electronic and Information Engineering, Beihang University, Beijing 100190, ChinaInspur Electronic Information Industry Co., Ltd., Beijing 100085, ChinaSchool of Electronic and Information Engineering, Beihang University, Beijing 100190, ChinaSchool of Electronic and Information Engineering, Beihang University, Beijing 100190, ChinaSchool of Electronic and Information Engineering, Beihang University, Beijing 100190, ChinaCyber and Big Data Research Laboratory, Edinburgh Napier University, Edinburgh EH11 4BN, UKReconstruction-based and prediction-based approaches are widely used for video anomaly detection (VAD) in smart city surveillance applications. However, neither of these approaches can effectively utilize the rich contextual information that exists in videos, which makes it difficult to accurately perceive anomalous activities. In this paper, we exploit the idea of a training model based on the “Cloze Test” strategy in natural language processing (NLP) and introduce a novel unsupervised learning framework to encode both motion and appearance information at an object level. Specifically, to store the normal modes of video activity reconstructions, we first design an optical stream memory network with skip connections. Secondly, we build a space–time cube (STC) for use as the basic processing unit of the model and erase a patch in the STC to form the frame to be reconstructed. This enables a so-called ”incomplete event (IE)” to be completed. On this basis, a conditional autoencoder is utilized to capture the high correspondence between optical flow and STC. The model predicts erased patches in IEs based on the context of the front and back frames. Finally, we employ a generating adversarial network (GAN)-based training method to improve the performance of VAD. By distinguishing the predicted erased optical flow and erased video frame, the anomaly detection results are shown to be more reliable with our proposed method which can help reconstruct the original video in IE. Comparative experiments conducted on the benchmark UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets demonstrate AUROC scores reaching 97.7%, 89.7%, and 75.8%, respectively.https://www.mdpi.com/1424-8220/23/10/4828video anomaly detectionoptical flowincomplete event |
| spellingShingle | Heqing Huang Bing Zhao Fei Gao Penghui Chen Jun Wang Amir Hussain A Novel Unsupervised Video Anomaly Detection Framework Based on Optical Flow Reconstruction and Erased Frame Prediction Sensors video anomaly detection optical flow incomplete event |
| title | A Novel Unsupervised Video Anomaly Detection Framework Based on Optical Flow Reconstruction and Erased Frame Prediction |
| title_full | A Novel Unsupervised Video Anomaly Detection Framework Based on Optical Flow Reconstruction and Erased Frame Prediction |
| title_fullStr | A Novel Unsupervised Video Anomaly Detection Framework Based on Optical Flow Reconstruction and Erased Frame Prediction |
| title_full_unstemmed | A Novel Unsupervised Video Anomaly Detection Framework Based on Optical Flow Reconstruction and Erased Frame Prediction |
| title_short | A Novel Unsupervised Video Anomaly Detection Framework Based on Optical Flow Reconstruction and Erased Frame Prediction |
| title_sort | novel unsupervised video anomaly detection framework based on optical flow reconstruction and erased frame prediction |
| topic | video anomaly detection optical flow incomplete event |
| url | https://www.mdpi.com/1424-8220/23/10/4828 |
| work_keys_str_mv | AT heqinghuang anovelunsupervisedvideoanomalydetectionframeworkbasedonopticalflowreconstructionanderasedframeprediction AT bingzhao anovelunsupervisedvideoanomalydetectionframeworkbasedonopticalflowreconstructionanderasedframeprediction AT feigao anovelunsupervisedvideoanomalydetectionframeworkbasedonopticalflowreconstructionanderasedframeprediction AT penghuichen anovelunsupervisedvideoanomalydetectionframeworkbasedonopticalflowreconstructionanderasedframeprediction AT junwang anovelunsupervisedvideoanomalydetectionframeworkbasedonopticalflowreconstructionanderasedframeprediction AT amirhussain anovelunsupervisedvideoanomalydetectionframeworkbasedonopticalflowreconstructionanderasedframeprediction AT heqinghuang novelunsupervisedvideoanomalydetectionframeworkbasedonopticalflowreconstructionanderasedframeprediction AT bingzhao novelunsupervisedvideoanomalydetectionframeworkbasedonopticalflowreconstructionanderasedframeprediction AT feigao novelunsupervisedvideoanomalydetectionframeworkbasedonopticalflowreconstructionanderasedframeprediction AT penghuichen novelunsupervisedvideoanomalydetectionframeworkbasedonopticalflowreconstructionanderasedframeprediction AT junwang novelunsupervisedvideoanomalydetectionframeworkbasedonopticalflowreconstructionanderasedframeprediction AT amirhussain novelunsupervisedvideoanomalydetectionframeworkbasedonopticalflowreconstructionanderasedframeprediction |