<i>Caveat emptor:</i> On the Need for Baseline Quality Standards in Computer Vision Wood Identification
Computer vision wood identification (CVWID) has focused on laboratory studies reporting consistently high model accuracies with greatly varying input data quality, data hygiene, and wood identification expertise. Employing examples from published literature, we demonstrate that the highly optimistic...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-04-01
|
| Series: | Forests |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1999-4907/13/4/632 |
| _version_ | 1797504767287623680 |
|---|---|
| author | Prabu Ravindran Alex C. Wiedenhoeft |
| author_facet | Prabu Ravindran Alex C. Wiedenhoeft |
| author_sort | Prabu Ravindran |
| collection | DOAJ |
| description | Computer vision wood identification (CVWID) has focused on laboratory studies reporting consistently high model accuracies with greatly varying input data quality, data hygiene, and wood identification expertise. Employing examples from published literature, we demonstrate that the highly optimistic model performance in prior works may be attributed to evaluating the wrong functionality—wood specimen identification rather than the desired wood species or genus identification—using limited datasets with data hygiene practices that violate the requirement of clear separation between training and evaluation data. Given the lack of a rigorous framework for a valid methodology and its objective evaluation, we present a set of minimal baseline quality standards for performing and reporting CVWID research and development that can enable valid, objective, and fair evaluation of current and future developments in this rapidly developing field. To elucidate the quality standards, we present a critical revisitation of a prior CVWID study of North American ring-porous woods and an exemplar study incorporating best practices on a new dataset covering the same set of woods. The proposed baseline quality standards can help translate models with high in silico performance to field-operational CVWID systems and allow stakeholders in research, industry, and government to make informed, evidence-based modality-agnostic decisions. |
| first_indexed | 2024-03-10T04:09:02Z |
| format | Article |
| id | doaj.art-4a1170fd1577498f99f8d05bfa33f661 |
| institution | Directory Open Access Journal |
| issn | 1999-4907 |
| language | English |
| last_indexed | 2024-03-10T04:09:02Z |
| publishDate | 2022-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Forests |
| spelling | doaj.art-4a1170fd1577498f99f8d05bfa33f6612023-11-23T08:15:10ZengMDPI AGForests1999-49072022-04-0113463210.3390/f13040632<i>Caveat emptor:</i> On the Need for Baseline Quality Standards in Computer Vision Wood IdentificationPrabu Ravindran0Alex C. Wiedenhoeft1Department of Botany, University of Wisconsin, Madison, WI 53706, USADepartment of Botany, University of Wisconsin, Madison, WI 53706, USAComputer vision wood identification (CVWID) has focused on laboratory studies reporting consistently high model accuracies with greatly varying input data quality, data hygiene, and wood identification expertise. Employing examples from published literature, we demonstrate that the highly optimistic model performance in prior works may be attributed to evaluating the wrong functionality—wood specimen identification rather than the desired wood species or genus identification—using limited datasets with data hygiene practices that violate the requirement of clear separation between training and evaluation data. Given the lack of a rigorous framework for a valid methodology and its objective evaluation, we present a set of minimal baseline quality standards for performing and reporting CVWID research and development that can enable valid, objective, and fair evaluation of current and future developments in this rapidly developing field. To elucidate the quality standards, we present a critical revisitation of a prior CVWID study of North American ring-porous woods and an exemplar study incorporating best practices on a new dataset covering the same set of woods. The proposed baseline quality standards can help translate models with high in silico performance to field-operational CVWID systems and allow stakeholders in research, industry, and government to make informed, evidence-based modality-agnostic decisions.https://www.mdpi.com/1999-4907/13/4/632wood identificationcomputer visionmachine learningXyloTronbest practices |
| spellingShingle | Prabu Ravindran Alex C. Wiedenhoeft <i>Caveat emptor:</i> On the Need for Baseline Quality Standards in Computer Vision Wood Identification Forests wood identification computer vision machine learning XyloTron best practices |
| title | <i>Caveat emptor:</i> On the Need for Baseline Quality Standards in Computer Vision Wood Identification |
| title_full | <i>Caveat emptor:</i> On the Need for Baseline Quality Standards in Computer Vision Wood Identification |
| title_fullStr | <i>Caveat emptor:</i> On the Need for Baseline Quality Standards in Computer Vision Wood Identification |
| title_full_unstemmed | <i>Caveat emptor:</i> On the Need for Baseline Quality Standards in Computer Vision Wood Identification |
| title_short | <i>Caveat emptor:</i> On the Need for Baseline Quality Standards in Computer Vision Wood Identification |
| title_sort | i caveat emptor i on the need for baseline quality standards in computer vision wood identification |
| topic | wood identification computer vision machine learning XyloTron best practices |
| url | https://www.mdpi.com/1999-4907/13/4/632 |
| work_keys_str_mv | AT praburavindran icaveatemptoriontheneedforbaselinequalitystandardsincomputervisionwoodidentification AT alexcwiedenhoeft icaveatemptoriontheneedforbaselinequalitystandardsincomputervisionwoodidentification |