Thoughts about artifact badging
Reproducibility: the extent to which consistent results are obtained when an experiment is repeated, is important as a means to validate experimental results, promote integrity of research, and accelerate follow up work. Commitment to artifact reviewing and badging seeks to promote reproducibility a...
| Main Authors: | , |
|---|---|
| Format: | Journal article |
| Language: | English |
| Published: |
Association for Computing Machinery
2020
|
| _version_ | 1797059266979299328 |
|---|---|
| author | Zilberman, N Moore, AW |
| author_facet | Zilberman, N Moore, AW |
| author_sort | Zilberman, N |
| collection | OXFORD |
| description | Reproducibility: the extent to which consistent results are obtained when an experiment is repeated, is important as a means to validate experimental results, promote integrity of research, and accelerate follow up work. Commitment to artifact reviewing and badging seeks to promote reproducibility and rank the quality of submitted artifacts.
However, as illustrated in this issue, the current badging scheme, with its focus upon an artifact being reusable, may not identify limitations of architecture, implementation, or evaluation.
<br>
We propose that to improve the insight into artifact reproducibility, the depth and nature of artifact evaluation must move beyond simply considering if an artifact is reusable. Artifact evaluation should consider the methods of that evaluation alongside the varying of inputs to that evaluation. To achieve this, we suggest an extension to the scope of artifact badging, and describe both approaches and best practice arising in other communities. We seek to promote conversation and make a call to action intended to strengthen the scientific method within our domain. |
| first_indexed | 2024-03-06T20:01:45Z |
| format | Journal article |
| id | oxford-uuid:278b4a7f-79e5-439f-bc0e-19569153d1e4 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-06T20:01:45Z |
| publishDate | 2020 |
| publisher | Association for Computing Machinery |
| record_format | dspace |
| spelling | oxford-uuid:278b4a7f-79e5-439f-bc0e-19569153d1e42022-03-26T12:07:33ZThoughts about artifact badgingJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:278b4a7f-79e5-439f-bc0e-19569153d1e4EnglishSymplectic ElementsAssociation for Computing Machinery2020Zilberman, NMoore, AWReproducibility: the extent to which consistent results are obtained when an experiment is repeated, is important as a means to validate experimental results, promote integrity of research, and accelerate follow up work. Commitment to artifact reviewing and badging seeks to promote reproducibility and rank the quality of submitted artifacts. However, as illustrated in this issue, the current badging scheme, with its focus upon an artifact being reusable, may not identify limitations of architecture, implementation, or evaluation. <br> We propose that to improve the insight into artifact reproducibility, the depth and nature of artifact evaluation must move beyond simply considering if an artifact is reusable. Artifact evaluation should consider the methods of that evaluation alongside the varying of inputs to that evaluation. To achieve this, we suggest an extension to the scope of artifact badging, and describe both approaches and best practice arising in other communities. We seek to promote conversation and make a call to action intended to strengthen the scientific method within our domain. |
| spellingShingle | Zilberman, N Moore, AW Thoughts about artifact badging |
| title | Thoughts about artifact badging |
| title_full | Thoughts about artifact badging |
| title_fullStr | Thoughts about artifact badging |
| title_full_unstemmed | Thoughts about artifact badging |
| title_short | Thoughts about artifact badging |
| title_sort | thoughts about artifact badging |
| work_keys_str_mv | AT zilbermann thoughtsaboutartifactbadging AT mooreaw thoughtsaboutartifactbadging |