Labelling unlabelled videos from scratch with multi-modal self-supervision
A large part of the current success of deep learning lies in the effectiveness of data -- more precisely: of labeled data. Yet, labelling a dataset with human annotation continues to carry high costs, especially for videos. While in the image domain, recent methods have allowed to generate meaningfu...
| Main Authors: | , , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
NeurIPS
2020
|
| _version_ | 1797057844506263552 |
|---|---|
| author | Asano, YM Patrick, M Rupprecht, C Vedaldi, A |
| author_facet | Asano, YM Patrick, M Rupprecht, C Vedaldi, A |
| author_sort | Asano, YM |
| collection | OXFORD |
| description | A large part of the current success of deep learning lies in the effectiveness of data -- more precisely: of labeled data. Yet, labelling a dataset with human annotation continues to carry high costs, especially for videos. While in the image domain, recent methods have allowed to generate meaningful (pseudo-) labels for unlabelled datasets without supervision, this development is missing for the video domain where learning feature representations is the current focus. In this work, we a) show that unsupervised labelling of a video dataset does not come for free from strong feature encoders and b) propose a novel clustering method that allows pseudo-labelling of a video dataset without any human annotations, by leveraging the natural correspondence between audio and visual modalities. An extensive analysis shows that the resulting clusters have high semantic overlap to ground truth human labels. We further introduce the first benchmarking results on unsupervised labelling of common video datasets.
|
| first_indexed | 2024-03-06T19:42:09Z |
| format | Conference item |
| id | oxford-uuid:210d707b-339e-471f-a776-f73a8a2ea7d7 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-06T19:42:09Z |
| publishDate | 2020 |
| publisher | NeurIPS |
| record_format | dspace |
| spelling | oxford-uuid:210d707b-339e-471f-a776-f73a8a2ea7d72022-03-26T11:31:04ZLabelling unlabelled videos from scratch with multi-modal self-supervisionConference itemhttp://purl.org/coar/resource_type/c_5794uuid:210d707b-339e-471f-a776-f73a8a2ea7d7EnglishSymplectic ElementsNeurIPS2020Asano, YMPatrick, MRupprecht, CVedaldi, AA large part of the current success of deep learning lies in the effectiveness of data -- more precisely: of labeled data. Yet, labelling a dataset with human annotation continues to carry high costs, especially for videos. While in the image domain, recent methods have allowed to generate meaningful (pseudo-) labels for unlabelled datasets without supervision, this development is missing for the video domain where learning feature representations is the current focus. In this work, we a) show that unsupervised labelling of a video dataset does not come for free from strong feature encoders and b) propose a novel clustering method that allows pseudo-labelling of a video dataset without any human annotations, by leveraging the natural correspondence between audio and visual modalities. An extensive analysis shows that the resulting clusters have high semantic overlap to ground truth human labels. We further introduce the first benchmarking results on unsupervised labelling of common video datasets. |
| spellingShingle | Asano, YM Patrick, M Rupprecht, C Vedaldi, A Labelling unlabelled videos from scratch with multi-modal self-supervision |
| title | Labelling unlabelled videos from scratch with multi-modal self-supervision |
| title_full | Labelling unlabelled videos from scratch with multi-modal self-supervision |
| title_fullStr | Labelling unlabelled videos from scratch with multi-modal self-supervision |
| title_full_unstemmed | Labelling unlabelled videos from scratch with multi-modal self-supervision |
| title_short | Labelling unlabelled videos from scratch with multi-modal self-supervision |
| title_sort | labelling unlabelled videos from scratch with multi modal self supervision |
| work_keys_str_mv | AT asanoym labellingunlabelledvideosfromscratchwithmultimodalselfsupervision AT patrickm labellingunlabelledvideosfromscratchwithmultimodalselfsupervision AT rupprechtc labellingunlabelledvideosfromscratchwithmultimodalselfsupervision AT vedaldia labellingunlabelledvideosfromscratchwithmultimodalselfsupervision |