Study of Subjective Data Integrity for Image Quality Data Sets with Consumer Camera Content
We need data sets of images and subjective scores to develop robust no reference (or blind) visual quality metrics for consumer applications. These applications have many uncontrolled variables because the camera creates the original media and the impairment simultaneously. We do not fully understan...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-02-01
|
| Series: | Journal of Imaging |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2313-433X/6/3/7 |
| _version_ | 1818035651179708416 |
|---|---|
| author | Jakub Nawała Margaret H. Pinson Mikołaj Leszczuk Lucjan Janowski |
| author_facet | Jakub Nawała Margaret H. Pinson Mikołaj Leszczuk Lucjan Janowski |
| author_sort | Jakub Nawała |
| collection | DOAJ |
| description | We need data sets of images and subjective scores to develop robust no reference (or blind) visual quality metrics for consumer applications. These applications have many uncontrolled variables because the camera creates the original media and the impairment simultaneously. We do not fully understand how this impacts the integrity of our subjective data. We put forward two new data sets of images from consumer cameras. The first data set, CCRIQ2, uses a strict experiment design, more suitable for camera performance evaluation. The second data set, VIME1, uses a loose experiment design that resembles the behavior of consumer photographers. We gather subjective scores through a subjective experiment with 24 participants using the Absolute Category Rating method. We make these two new data sets available royalty-free on the Consumer Digital Video Library. We also present their integrity analysis (proposing one new approach) and explore the possibility of combining CCRIQ2 with its legacy counterpart. We conclude that the loose experiment design yields unreliable data, despite adhering to international recommendations. This suggests that the classical subjective study design may not be suitable for studies using consumer content. Finally, we show that Hoßfeld−Schatz−Egger <inline-formula> <math display="inline"> <semantics> <mi>α</mi> </semantics> </math> </inline-formula> failed to detect important differences between the two data sets. |
| first_indexed | 2024-12-10T06:58:26Z |
| format | Article |
| id | doaj.art-1179af634e3a45858687765b6b495984 |
| institution | Directory Open Access Journal |
| issn | 2313-433X |
| language | English |
| last_indexed | 2024-12-10T06:58:26Z |
| publishDate | 2020-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Journal of Imaging |
| spelling | doaj.art-1179af634e3a45858687765b6b4959842022-12-22T01:58:22ZengMDPI AGJournal of Imaging2313-433X2020-02-0163710.3390/jimaging6030007jimaging6030007Study of Subjective Data Integrity for Image Quality Data Sets with Consumer Camera ContentJakub Nawała0Margaret H. Pinson1Mikołaj Leszczuk2Lucjan Janowski3AGH University of Science and Technology, MP 30059 Kraków, PolandInstitute for Telecommunication Sciences, National Telecommunications and Information Administration, Boulder, CO 80305, USAAGH University of Science and Technology, MP 30059 Kraków, PolandAGH University of Science and Technology, MP 30059 Kraków, PolandWe need data sets of images and subjective scores to develop robust no reference (or blind) visual quality metrics for consumer applications. These applications have many uncontrolled variables because the camera creates the original media and the impairment simultaneously. We do not fully understand how this impacts the integrity of our subjective data. We put forward two new data sets of images from consumer cameras. The first data set, CCRIQ2, uses a strict experiment design, more suitable for camera performance evaluation. The second data set, VIME1, uses a loose experiment design that resembles the behavior of consumer photographers. We gather subjective scores through a subjective experiment with 24 participants using the Absolute Category Rating method. We make these two new data sets available royalty-free on the Consumer Digital Video Library. We also present their integrity analysis (proposing one new approach) and explore the possibility of combining CCRIQ2 with its legacy counterpart. We conclude that the loose experiment design yields unreliable data, despite adhering to international recommendations. This suggests that the classical subjective study design may not be suitable for studies using consumer content. Finally, we show that Hoßfeld−Schatz−Egger <inline-formula> <math display="inline"> <semantics> <mi>α</mi> </semantics> </math> </inline-formula> failed to detect important differences between the two data sets.https://www.mdpi.com/2313-433X/6/3/7image qualitydata integrityconsumer camerablind quality assessmentevaluationsubjective datano referencenr metricssubjective study |
| spellingShingle | Jakub Nawała Margaret H. Pinson Mikołaj Leszczuk Lucjan Janowski Study of Subjective Data Integrity for Image Quality Data Sets with Consumer Camera Content Journal of Imaging image quality data integrity consumer camera blind quality assessment evaluation subjective data no reference nr metrics subjective study |
| title | Study of Subjective Data Integrity for Image Quality Data Sets with Consumer Camera Content |
| title_full | Study of Subjective Data Integrity for Image Quality Data Sets with Consumer Camera Content |
| title_fullStr | Study of Subjective Data Integrity for Image Quality Data Sets with Consumer Camera Content |
| title_full_unstemmed | Study of Subjective Data Integrity for Image Quality Data Sets with Consumer Camera Content |
| title_short | Study of Subjective Data Integrity for Image Quality Data Sets with Consumer Camera Content |
| title_sort | study of subjective data integrity for image quality data sets with consumer camera content |
| topic | image quality data integrity consumer camera blind quality assessment evaluation subjective data no reference nr metrics subjective study |
| url | https://www.mdpi.com/2313-433X/6/3/7 |
| work_keys_str_mv | AT jakubnawała studyofsubjectivedataintegrityforimagequalitydatasetswithconsumercameracontent AT margarethpinson studyofsubjectivedataintegrityforimagequalitydatasetswithconsumercameracontent AT mikołajleszczuk studyofsubjectivedataintegrityforimagequalitydatasetswithconsumercameracontent AT lucjanjanowski studyofsubjectivedataintegrityforimagequalitydatasetswithconsumercameracontent |