Static Video Summarization Using Video Coding Features with Frame-Level Temporal Subsampling and Deep Learning
There is an abundance of digital video content due to the cloud’s phenomenal growth and security footage; it is therefore essential to summarize these videos in data centers. This paper offers innovative approaches to the problem of key frame extraction for the purpose of video summarization. Our ap...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-05-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/13/10/6065 |
| _version_ | 1797601265889312768 |
|---|---|
| author | Obada Issa Tamer Shanableh |
| author_facet | Obada Issa Tamer Shanableh |
| author_sort | Obada Issa |
| collection | DOAJ |
| description | There is an abundance of digital video content due to the cloud’s phenomenal growth and security footage; it is therefore essential to summarize these videos in data centers. This paper offers innovative approaches to the problem of key frame extraction for the purpose of video summarization. Our approach includes the extraction of feature variables from the bit streams of coded videos, followed by optional stepwise regression for dimensionality reduction. Once the features are extracted and their dimensionality is reduced, we apply innovative frame-level temporal subsampling techniques, followed by training and testing using deep learning architectures. The frame-level temporal subsampling techniques are based on cosine similarity and the PCA projections of feature vectors. We create three different learning architectures by utilizing LSTM networks, 1D-CNN networks, and random forests. The four most popular video summarization datasets, namely, TVSum, SumMe, OVP, and VSUMM, are used to evaluate the accuracy of the proposed solutions. This includes the precision, recall, F-score measures, and computational time. It is shown that the proposed solutions, when trained and tested on all subjective user summaries, achieved F-scores of 0.79, 0.74, 0.88, and 0.81, respectively, for the aforementioned datasets, showing clear improvements over prior studies. |
| first_indexed | 2024-03-11T03:58:43Z |
| format | Article |
| id | doaj.art-7e8414ce5c4d40e787fddb8daae27371 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-11T03:58:43Z |
| publishDate | 2023-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-7e8414ce5c4d40e787fddb8daae273712023-11-18T00:20:00ZengMDPI AGApplied Sciences2076-34172023-05-011310606510.3390/app13106065Static Video Summarization Using Video Coding Features with Frame-Level Temporal Subsampling and Deep LearningObada Issa0Tamer Shanableh1Department of Computer Science and Engineering, American University of Sharjah, Sharjah P.O. Box 26666, United Arab EmiratesDepartment of Computer Science and Engineering, American University of Sharjah, Sharjah P.O. Box 26666, United Arab EmiratesThere is an abundance of digital video content due to the cloud’s phenomenal growth and security footage; it is therefore essential to summarize these videos in data centers. This paper offers innovative approaches to the problem of key frame extraction for the purpose of video summarization. Our approach includes the extraction of feature variables from the bit streams of coded videos, followed by optional stepwise regression for dimensionality reduction. Once the features are extracted and their dimensionality is reduced, we apply innovative frame-level temporal subsampling techniques, followed by training and testing using deep learning architectures. The frame-level temporal subsampling techniques are based on cosine similarity and the PCA projections of feature vectors. We create three different learning architectures by utilizing LSTM networks, 1D-CNN networks, and random forests. The four most popular video summarization datasets, namely, TVSum, SumMe, OVP, and VSUMM, are used to evaluate the accuracy of the proposed solutions. This includes the precision, recall, F-score measures, and computational time. It is shown that the proposed solutions, when trained and tested on all subjective user summaries, achieved F-scores of 0.79, 0.74, 0.88, and 0.81, respectively, for the aforementioned datasets, showing clear improvements over prior studies.https://www.mdpi.com/2076-3417/13/10/6065video summarizationvideo codingtemporal subsamplingconvolution neural networkslong-short term memory |
| spellingShingle | Obada Issa Tamer Shanableh Static Video Summarization Using Video Coding Features with Frame-Level Temporal Subsampling and Deep Learning Applied Sciences video summarization video coding temporal subsampling convolution neural networks long-short term memory |
| title | Static Video Summarization Using Video Coding Features with Frame-Level Temporal Subsampling and Deep Learning |
| title_full | Static Video Summarization Using Video Coding Features with Frame-Level Temporal Subsampling and Deep Learning |
| title_fullStr | Static Video Summarization Using Video Coding Features with Frame-Level Temporal Subsampling and Deep Learning |
| title_full_unstemmed | Static Video Summarization Using Video Coding Features with Frame-Level Temporal Subsampling and Deep Learning |
| title_short | Static Video Summarization Using Video Coding Features with Frame-Level Temporal Subsampling and Deep Learning |
| title_sort | static video summarization using video coding features with frame level temporal subsampling and deep learning |
| topic | video summarization video coding temporal subsampling convolution neural networks long-short term memory |
| url | https://www.mdpi.com/2076-3417/13/10/6065 |
| work_keys_str_mv | AT obadaissa staticvideosummarizationusingvideocodingfeatureswithframeleveltemporalsubsamplinganddeeplearning AT tamershanableh staticvideosummarizationusingvideocodingfeatureswithframeleveltemporalsubsamplinganddeeplearning |