Visual understanding and personalization for an optimal recollection experience
The affordability of wearable cameras such as the Narrative Clip and GoPro allows mass-market consumers to continuously record their lives, producing large amounts of unstructured visual data. Moreover, users tend to record with their smartphones more multimedia content than they can possibly share...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | English |
| Published: |
2019
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/82932 http://hdl.handle.net/10220/50437 |
| _version_ | 1811684672435912704 |
|---|---|
| author | Ana, Garcia del Molino |
| author2 | Tan Ah Hwee |
| author_facet | Tan Ah Hwee Ana, Garcia del Molino |
| author_sort | Ana, Garcia del Molino |
| collection | NTU |
| description | The affordability of wearable cameras such as the Narrative Clip and GoPro allows mass-market consumers to continuously record their lives, producing large amounts of unstructured visual data. Moreover, users tend to record with their smartphones more multimedia content than they can possibly share or review. We use each of these devices for different purposes: action cameras for travels and adventures; our smartphones to capture on the spur of the moment; a lifelogging device to record unobtrusively all our daily life activities. As a result, the few important shots end up buried among many repetitive images or uninteresting long segments, requiring hours of manual analysis in order to, say, select highlights in a day or find the most aesthetic pictures.
Tackling challenges in end-to-end consumer video summarization, this thesis contributes to the state of the art in three major aspects: (i) Contextual Event Segmentation, an episodic event segmentation method that is able to detect boundaries between heterogeneous events and ignore local occlusions and brief diversions. CES improves the performance of the baselines by over 16% in F-measure, and is competitive with manual annotations. (ii) Personalized Highlight Detection, a highlight detector that is personalized via its inputs. The experimental results show that using the user history substantially improves the prediction accuracy. PHD outperforms the user-agnostic baselines even with only one single person-specific example. (iii) Active Video Summarization, an interactive approach to video exploration that gathers the user’s preferences while creating a video summary. AVS achieves an excellent compromise between usability and quality. The diverse and uniform nature of AVS summaries makes it alsoa valuable tool for browsing someone else’s visual collection. Additionally, this thesis contributes two large-scale datasets for First Person View video analysis, CSumm and R3, and a large-scale dataset for personalized video highlights, PHD2. |
| first_indexed | 2024-10-01T04:32:21Z |
| format | Thesis |
| id | ntu-10356/82932 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T04:32:21Z |
| publishDate | 2019 |
| record_format | dspace |
| spelling | ntu-10356/829322020-10-28T08:40:50Z Visual understanding and personalization for an optimal recollection experience Ana, Garcia del Molino Tan Ah Hwee School of Computer Science and Engineering A*STAR Centre for Computational Intelligence Lim Joo Hwee Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision Engineering::Computer science and engineering::Computing methodologies::Pattern recognition The affordability of wearable cameras such as the Narrative Clip and GoPro allows mass-market consumers to continuously record their lives, producing large amounts of unstructured visual data. Moreover, users tend to record with their smartphones more multimedia content than they can possibly share or review. We use each of these devices for different purposes: action cameras for travels and adventures; our smartphones to capture on the spur of the moment; a lifelogging device to record unobtrusively all our daily life activities. As a result, the few important shots end up buried among many repetitive images or uninteresting long segments, requiring hours of manual analysis in order to, say, select highlights in a day or find the most aesthetic pictures. Tackling challenges in end-to-end consumer video summarization, this thesis contributes to the state of the art in three major aspects: (i) Contextual Event Segmentation, an episodic event segmentation method that is able to detect boundaries between heterogeneous events and ignore local occlusions and brief diversions. CES improves the performance of the baselines by over 16% in F-measure, and is competitive with manual annotations. (ii) Personalized Highlight Detection, a highlight detector that is personalized via its inputs. The experimental results show that using the user history substantially improves the prediction accuracy. PHD outperforms the user-agnostic baselines even with only one single person-specific example. (iii) Active Video Summarization, an interactive approach to video exploration that gathers the user’s preferences while creating a video summary. AVS achieves an excellent compromise between usability and quality. The diverse and uniform nature of AVS summaries makes it alsoa valuable tool for browsing someone else’s visual collection. Additionally, this thesis contributes two large-scale datasets for First Person View video analysis, CSumm and R3, and a large-scale dataset for personalized video highlights, PHD2. Doctor of Philosophy 2019-11-19T00:52:58Z 2019-12-06T15:08:33Z 2019-11-19T00:52:58Z 2019-12-06T15:08:33Z 2019 Thesis Ana, G. d. M. (2019). Visual understanding and personalization for an optimal recollection experience. Doctoral thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/82932 http://hdl.handle.net/10220/50437 10.32657/10356/82932 en 133 p. application/pdf |
| spellingShingle | Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision Engineering::Computer science and engineering::Computing methodologies::Pattern recognition Ana, Garcia del Molino Visual understanding and personalization for an optimal recollection experience |
| title | Visual understanding and personalization for an optimal recollection experience |
| title_full | Visual understanding and personalization for an optimal recollection experience |
| title_fullStr | Visual understanding and personalization for an optimal recollection experience |
| title_full_unstemmed | Visual understanding and personalization for an optimal recollection experience |
| title_short | Visual understanding and personalization for an optimal recollection experience |
| title_sort | visual understanding and personalization for an optimal recollection experience |
| topic | Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision Engineering::Computer science and engineering::Computing methodologies::Pattern recognition |
| url | https://hdl.handle.net/10356/82932 http://hdl.handle.net/10220/50437 |
| work_keys_str_mv | AT anagarciadelmolino visualunderstandingandpersonalizationforanoptimalrecollectionexperience |