Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100
This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head...
| Main Authors: | , , , , , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2022
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/162036 |
| _version_ | 1811696546820915200 |
|---|---|
| author | Damen, Dima Doughty, Hazel Farinella, Giovanni Maria Furnari, Antonino Kazakos, Evangelos Ma, Jian Moltisanti, Davide Munro, Jonathan Perrett, Toby Price, Will Wray, Michael |
| author2 | School of Computer Science and Engineering |
| author_facet | School of Computer Science and Engineering Damen, Dima Doughty, Hazel Farinella, Giovanni Maria Furnari, Antonino Kazakos, Evangelos Ma, Jian Moltisanti, Davide Munro, Jonathan Perrett, Toby Price, Will Wray, Michael |
| author_sort | Damen, Dima |
| collection | NTU |
| description | This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras. Compared to its previous version (Damen in Scaling egocentric vision: ECCV, 2018), EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments). This collection enables new challenges such as action detection and evaluating the “test of time”—i.e. whether models trained on data collected in 2018 can generalise to new footage collected two years later. The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics. |
| first_indexed | 2024-10-01T07:41:05Z |
| format | Journal Article |
| id | ntu-10356/162036 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T07:41:05Z |
| publishDate | 2022 |
| record_format | dspace |
| spelling | ntu-10356/1620362022-09-30T06:29:55Z Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100 Damen, Dima Doughty, Hazel Farinella, Giovanni Maria Furnari, Antonino Kazakos, Evangelos Ma, Jian Moltisanti, Davide Munro, Jonathan Perrett, Toby Price, Will Wray, Michael School of Computer Science and Engineering Engineering::Computer science and engineering Video Dataset Annotation Quality This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras. Compared to its previous version (Damen in Scaling egocentric vision: ECCV, 2018), EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments). This collection enables new challenges such as action detection and evaluating the “test of time”—i.e. whether models trained on data collected in 2018 can generalise to new footage collected two years later. The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics. Published version Research at Bristol is supported by Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Program (DTP), EPSRC Fellowship UMPIRE (EP/T004991/1). Research at Catania is sponsored by Piano della Ricerca 2016-2018 linea di Intervento 2 of DMI, by MISE - PON I&C 2014-2020, ENIGMA project (CUP: B61B19000520008) and by MIUR AIM - Attrazione e Mobilita Internazionale Linea 1 - AIM1893589 - CUP E64118002540007. 2022-09-30T06:29:54Z 2022-09-30T06:29:54Z 2022 Journal Article Damen, D., Doughty, H., Farinella, G. M., Furnari, A., Kazakos, E., Ma, J., Moltisanti, D., Munro, J., Perrett, T., Price, W. & Wray, M. (2022). Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100. International Journal of Computer Vision, 130(1), 33-55. https://dx.doi.org/10.1007/s11263-021-01531-2 0920-5691 https://hdl.handle.net/10356/162036 10.1007/s11263-021-01531-2 2-s2.0-85117411098 1 130 33 55 en International Journal of Computer Vision © The Author(s) 2021. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecomm ons.org/licenses/by/4.0/. application/pdf |
| spellingShingle | Engineering::Computer science and engineering Video Dataset Annotation Quality Damen, Dima Doughty, Hazel Farinella, Giovanni Maria Furnari, Antonino Kazakos, Evangelos Ma, Jian Moltisanti, Davide Munro, Jonathan Perrett, Toby Price, Will Wray, Michael Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100 |
| title | Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100 |
| title_full | Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100 |
| title_fullStr | Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100 |
| title_full_unstemmed | Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100 |
| title_short | Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100 |
| title_sort | rescaling egocentric vision collection pipeline and challenges for epic kitchens 100 |
| topic | Engineering::Computer science and engineering Video Dataset Annotation Quality |
| url | https://hdl.handle.net/10356/162036 |
| work_keys_str_mv | AT damendima rescalingegocentricvisioncollectionpipelineandchallengesforepickitchens100 AT doughtyhazel rescalingegocentricvisioncollectionpipelineandchallengesforepickitchens100 AT farinellagiovannimaria rescalingegocentricvisioncollectionpipelineandchallengesforepickitchens100 AT furnariantonino rescalingegocentricvisioncollectionpipelineandchallengesforepickitchens100 AT kazakosevangelos rescalingegocentricvisioncollectionpipelineandchallengesforepickitchens100 AT majian rescalingegocentricvisioncollectionpipelineandchallengesforepickitchens100 AT moltisantidavide rescalingegocentricvisioncollectionpipelineandchallengesforepickitchens100 AT munrojonathan rescalingegocentricvisioncollectionpipelineandchallengesforepickitchens100 AT perretttoby rescalingegocentricvisioncollectionpipelineandchallengesforepickitchens100 AT pricewill rescalingegocentricvisioncollectionpipelineandchallengesforepickitchens100 AT wraymichael rescalingegocentricvisioncollectionpipelineandchallengesforepickitchens100 |