The Pascal visual object classes challenge: a retrospective
The Pascal Visual Object Classes (VOC) challenge consists of two components: (i) a publicly available dataset of images together with ground truth annotation and standardised evaluation software; and (ii) an annual competition and workshop. There are five challenges: classification, detection, segme...
| Main Authors: | , , , , , |
|---|---|
| Format: | Journal article |
| Language: | English |
| Published: |
Springer Verlag
2014
|
| _version_ | 1811141047591370752 |
|---|---|
| author | Everingham, M Eslami, SMA Van Gool, L Williams, CKI Winn, J Zisserman, A |
| author_facet | Everingham, M Eslami, SMA Van Gool, L Williams, CKI Winn, J Zisserman, A |
| author_sort | Everingham, M |
| collection | OXFORD |
| description | The Pascal Visual Object Classes (VOC) challenge consists of two components: (i) a publicly available dataset of images together with ground truth annotation and standardised evaluation software; and (ii) an annual competition and workshop. There are five challenges: classification, detection, segmentation, action classification, and person layout. In this paper we provide a review of the challenge from 2008–2012. The paper is intended for two audiences: algorithm designers, researchers who want to see what the state of the art is, as measured by performance on the VOC datasets, along with the limitations and weak points of the current generation of algorithms; and, challenge designers, who want to see what we as organisers have learnt from the process and our recommendations for the organisation of future challenges. To analyse the performance of submitted algorithms on the VOC datasets we introduce a number of novel evaluation methods: a bootstrapping method for determining whether differences in the performance of two algorithms are significant or not; a normalised average precision so that performance can be compared across classes with different proportions of positive instances; a clustering method for visualising the performance across multiple algorithms so that the hard and easy images can be identified; and the use of a joint classifier over the submitted algorithms in order to measure their complementarity and combined performance. We also analyse the community’s progress through time using the methods of Hoiem et al. (Proceedings of European Conference on Computer Vision, 2012) to identify the types of occurring errors. We conclude the paper with an appraisal of the aspects of the challenge that worked well, and those that could be improved in future challenges. |
| first_indexed | 2024-09-25T04:31:40Z |
| format | Journal article |
| id | oxford-uuid:4be61639-601d-49f8-9a68-62ce1f52bb92 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-09-25T04:31:40Z |
| publishDate | 2014 |
| publisher | Springer Verlag |
| record_format | dspace |
| spelling | oxford-uuid:4be61639-601d-49f8-9a68-62ce1f52bb922024-08-28T12:52:50ZThe Pascal visual object classes challenge: a retrospectiveJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:4be61639-601d-49f8-9a68-62ce1f52bb92EnglishSymplectic ElementsSpringer Verlag2014Everingham, MEslami, SMAVan Gool, LWilliams, CKIWinn, JZisserman, AThe Pascal Visual Object Classes (VOC) challenge consists of two components: (i) a publicly available dataset of images together with ground truth annotation and standardised evaluation software; and (ii) an annual competition and workshop. There are five challenges: classification, detection, segmentation, action classification, and person layout. In this paper we provide a review of the challenge from 2008–2012. The paper is intended for two audiences: algorithm designers, researchers who want to see what the state of the art is, as measured by performance on the VOC datasets, along with the limitations and weak points of the current generation of algorithms; and, challenge designers, who want to see what we as organisers have learnt from the process and our recommendations for the organisation of future challenges. To analyse the performance of submitted algorithms on the VOC datasets we introduce a number of novel evaluation methods: a bootstrapping method for determining whether differences in the performance of two algorithms are significant or not; a normalised average precision so that performance can be compared across classes with different proportions of positive instances; a clustering method for visualising the performance across multiple algorithms so that the hard and easy images can be identified; and the use of a joint classifier over the submitted algorithms in order to measure their complementarity and combined performance. We also analyse the community’s progress through time using the methods of Hoiem et al. (Proceedings of European Conference on Computer Vision, 2012) to identify the types of occurring errors. We conclude the paper with an appraisal of the aspects of the challenge that worked well, and those that could be improved in future challenges. |
| spellingShingle | Everingham, M Eslami, SMA Van Gool, L Williams, CKI Winn, J Zisserman, A The Pascal visual object classes challenge: a retrospective |
| title | The Pascal visual object classes challenge: a retrospective |
| title_full | The Pascal visual object classes challenge: a retrospective |
| title_fullStr | The Pascal visual object classes challenge: a retrospective |
| title_full_unstemmed | The Pascal visual object classes challenge: a retrospective |
| title_short | The Pascal visual object classes challenge: a retrospective |
| title_sort | pascal visual object classes challenge a retrospective |
| work_keys_str_mv | AT everinghamm thepascalvisualobjectclasseschallengearetrospective AT eslamisma thepascalvisualobjectclasseschallengearetrospective AT vangooll thepascalvisualobjectclasseschallengearetrospective AT williamscki thepascalvisualobjectclasseschallengearetrospective AT winnj thepascalvisualobjectclasseschallengearetrospective AT zissermana thepascalvisualobjectclasseschallengearetrospective AT everinghamm pascalvisualobjectclasseschallengearetrospective AT eslamisma pascalvisualobjectclasseschallengearetrospective AT vangooll pascalvisualobjectclasseschallengearetrospective AT williamscki pascalvisualobjectclasseschallengearetrospective AT winnj pascalvisualobjectclasseschallengearetrospective AT zissermana pascalvisualobjectclasseschallengearetrospective |