FCN-LectureNet: Extractive Summarization of Whiteboard and Chalkboard Lecture Videos
Recording and sharing of educational or lecture videos has increased in recent years. Within these recordings, we find a large number of math-oriented lectures and tutorials which attract students of all levels. Many of the topics covered by these recordings are better explained using handwritten co...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9494351/ |
| _version_ | 1823939505697062912 |
|---|---|
| author | Kenny Davila Fei Xu Srirangaraj Setlur Venu Govindaraju |
| author_facet | Kenny Davila Fei Xu Srirangaraj Setlur Venu Govindaraju |
| author_sort | Kenny Davila |
| collection | DOAJ |
| description | Recording and sharing of educational or lecture videos has increased in recent years. Within these recordings, we find a large number of math-oriented lectures and tutorials which attract students of all levels. Many of the topics covered by these recordings are better explained using handwritten content on whiteboards or chalkboards. Hence, we find large numbers of lecture videos that feature the instructor writing on a surface. In this work, we propose a novel method for extraction and summarization of the handwritten content found in such videos. Our method is based on a fully convolutional network, FCN-LectureNet, which can extract the handwritten content from the video as binary images. These are further analyzed to identify the unique and stable units of content to produce a spatial-temporal index of handwritten content. A signal which approximates content deletion events is then built using information from the spatial-temporal index. The peaks of this signal are used to create temporal segments of the lecture based on the notion that sub-topics change when large portions of content are deleted. Finally, we use these segments to create an extractive summary of the handwritten content based on key-frames. This will facilitate content-based search and retrieval of these lecture videos. In this work, we also extend the AccessMath dataset to create a novel dataset for benchmarking of lecture video summarization called LectureMath. Our experiments on both datasets show that our novel method can outperform existing methods especially on the larger and more challenging dataset. Our code and data are publicly available. |
| first_indexed | 2024-12-17T02:58:54Z |
| format | Article |
| id | doaj.art-467ff5f35f67428ea023da8600705f9a |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-17T02:58:54Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-467ff5f35f67428ea023da8600705f9a2022-12-21T22:06:09ZengIEEEIEEE Access2169-35362021-01-01910446910448410.1109/ACCESS.2021.30994279494351FCN-LectureNet: Extractive Summarization of Whiteboard and Chalkboard Lecture VideosKenny Davila0https://orcid.org/0000-0001-6308-7113Fei Xu1https://orcid.org/0000-0002-9353-9528Srirangaraj Setlur2Venu Govindaraju3https://orcid.org/0000-0002-5318-7409Facultad de Ingenieria, Universidad Tecnológica Centroamericana, Tegucigalpa, HondurasDepartment of Computer Science and Engineering, University at Buffalo, Buffalo, NY, USADepartment of Computer Science and Engineering, University at Buffalo, Buffalo, NY, USADepartment of Computer Science and Engineering, University at Buffalo, Buffalo, NY, USARecording and sharing of educational or lecture videos has increased in recent years. Within these recordings, we find a large number of math-oriented lectures and tutorials which attract students of all levels. Many of the topics covered by these recordings are better explained using handwritten content on whiteboards or chalkboards. Hence, we find large numbers of lecture videos that feature the instructor writing on a surface. In this work, we propose a novel method for extraction and summarization of the handwritten content found in such videos. Our method is based on a fully convolutional network, FCN-LectureNet, which can extract the handwritten content from the video as binary images. These are further analyzed to identify the unique and stable units of content to produce a spatial-temporal index of handwritten content. A signal which approximates content deletion events is then built using information from the spatial-temporal index. The peaks of this signal are used to create temporal segments of the lecture based on the notion that sub-topics change when large portions of content are deleted. Finally, we use these segments to create an extractive summary of the handwritten content based on key-frames. This will facilitate content-based search and retrieval of these lecture videos. In this work, we also extend the AccessMath dataset to create a novel dataset for benchmarking of lecture video summarization called LectureMath. Our experiments on both datasets show that our novel method can outperform existing methods especially on the larger and more challenging dataset. Our code and data are publicly available.https://ieeexplore.ieee.org/document/9494351/Fully convolutional networkshandwritten text detectionimage binarizationlecture videosvideo summarization |
| spellingShingle | Kenny Davila Fei Xu Srirangaraj Setlur Venu Govindaraju FCN-LectureNet: Extractive Summarization of Whiteboard and Chalkboard Lecture Videos IEEE Access Fully convolutional networks handwritten text detection image binarization lecture videos video summarization |
| title | FCN-LectureNet: Extractive Summarization of Whiteboard and Chalkboard Lecture Videos |
| title_full | FCN-LectureNet: Extractive Summarization of Whiteboard and Chalkboard Lecture Videos |
| title_fullStr | FCN-LectureNet: Extractive Summarization of Whiteboard and Chalkboard Lecture Videos |
| title_full_unstemmed | FCN-LectureNet: Extractive Summarization of Whiteboard and Chalkboard Lecture Videos |
| title_short | FCN-LectureNet: Extractive Summarization of Whiteboard and Chalkboard Lecture Videos |
| title_sort | fcn lecturenet extractive summarization of whiteboard and chalkboard lecture videos |
| topic | Fully convolutional networks handwritten text detection image binarization lecture videos video summarization |
| url | https://ieeexplore.ieee.org/document/9494351/ |
| work_keys_str_mv | AT kennydavila fcnlecturenetextractivesummarizationofwhiteboardandchalkboardlecturevideos AT feixu fcnlecturenetextractivesummarizationofwhiteboardandchalkboardlecturevideos AT srirangarajsetlur fcnlecturenetextractivesummarizationofwhiteboardandchalkboardlecturevideos AT venugovindaraju fcnlecturenetextractivesummarizationofwhiteboardandchalkboardlecturevideos |