A Re-Analysis of Repeatability and Reproducibility in the Ames-USDOE-FBI Study
Forensic firearms identification, the determination by a trained firearms examiner as to whether or not bullets or cartridges came from a common weapon, has long been a mainstay in the criminal courts. Reliability of forensic firearms identification has been challenged in the general scientific comm...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Taylor & Francis Group
2022-10-01
|
| Series: | Statistics and Public Policy |
| Subjects: | |
| Online Access: | https://www.tandfonline.com/doi/10.1080/2330443X.2022.2120137 |
| _version_ | 1797985907742081024 |
|---|---|
| author | Alan H. Dorfman Richard Valliant |
| author_facet | Alan H. Dorfman Richard Valliant |
| author_sort | Alan H. Dorfman |
| collection | DOAJ |
| description | Forensic firearms identification, the determination by a trained firearms examiner as to whether or not bullets or cartridges came from a common weapon, has long been a mainstay in the criminal courts. Reliability of forensic firearms identification has been challenged in the general scientific community, and, in response, several studies have been carried out aimed at showing that firearms examination is accurate, that is, has low error rates. Less studied has been the question of consistency, of whether two examinations of the same bullets or cartridge cases come to the same conclusion, carried out by an examiner on separate occasions—intrarater reliability or repeatability—or by two examiners—interrater reliability or reproducibility.One important study, described in a 2020 Report by the Ames Laboratory-USDOE to the Federal Bureau of Investigation, went beyond considerations of accuracy to investigate firearms examination repeatability and reproducibility. The Report’s conclusions were paradoxical. The observed agreement of examiners with themselves or with other examiners appears mediocre. However, the study concluded repeatability and reproducibility are satisfactory, on grounds that the observed agreement exceeds a quantity called the expected agreement. We find that appropriately employing expected agreement as it was intended does not suggest satisfactory repeatability and reproducibility, but the opposite. |
| first_indexed | 2024-04-11T07:25:04Z |
| format | Article |
| id | doaj.art-19a9b0fc90574e6f9de73cf36e6db829 |
| institution | Directory Open Access Journal |
| issn | 2330-443X |
| language | English |
| last_indexed | 2024-04-11T07:25:04Z |
| publishDate | 2022-10-01 |
| publisher | Taylor & Francis Group |
| record_format | Article |
| series | Statistics and Public Policy |
| spelling | doaj.art-19a9b0fc90574e6f9de73cf36e6db8292022-12-22T04:37:06ZengTaylor & Francis GroupStatistics and Public Policy2330-443X2022-10-019117518410.1080/2330443X.2022.2120137A Re-Analysis of Repeatability and Reproducibility in the Ames-USDOE-FBI StudyAlan H. Dorfman0Richard Valliant1National Center for Health Statistics (Retired), Bethesda, Maryland;Research Professor Emeritus, Universities of Michigan & Maryland, Ann Arbor, MichiganForensic firearms identification, the determination by a trained firearms examiner as to whether or not bullets or cartridges came from a common weapon, has long been a mainstay in the criminal courts. Reliability of forensic firearms identification has been challenged in the general scientific community, and, in response, several studies have been carried out aimed at showing that firearms examination is accurate, that is, has low error rates. Less studied has been the question of consistency, of whether two examinations of the same bullets or cartridge cases come to the same conclusion, carried out by an examiner on separate occasions—intrarater reliability or repeatability—or by two examiners—interrater reliability or reproducibility.One important study, described in a 2020 Report by the Ames Laboratory-USDOE to the Federal Bureau of Investigation, went beyond considerations of accuracy to investigate firearms examination repeatability and reproducibility. The Report’s conclusions were paradoxical. The observed agreement of examiners with themselves or with other examiners appears mediocre. However, the study concluded repeatability and reproducibility are satisfactory, on grounds that the observed agreement exceeds a quantity called the expected agreement. We find that appropriately employing expected agreement as it was intended does not suggest satisfactory repeatability and reproducibility, but the opposite.https://www.tandfonline.com/doi/10.1080/2330443X.2022.2120137Expected agreementFirearmsForensic scienceKappa indexObserved agreement |
| spellingShingle | Alan H. Dorfman Richard Valliant A Re-Analysis of Repeatability and Reproducibility in the Ames-USDOE-FBI Study Statistics and Public Policy Expected agreement Firearms Forensic science Kappa index Observed agreement |
| title | A Re-Analysis of Repeatability and Reproducibility in the Ames-USDOE-FBI Study |
| title_full | A Re-Analysis of Repeatability and Reproducibility in the Ames-USDOE-FBI Study |
| title_fullStr | A Re-Analysis of Repeatability and Reproducibility in the Ames-USDOE-FBI Study |
| title_full_unstemmed | A Re-Analysis of Repeatability and Reproducibility in the Ames-USDOE-FBI Study |
| title_short | A Re-Analysis of Repeatability and Reproducibility in the Ames-USDOE-FBI Study |
| title_sort | re analysis of repeatability and reproducibility in the ames usdoe fbi study |
| topic | Expected agreement Firearms Forensic science Kappa index Observed agreement |
| url | https://www.tandfonline.com/doi/10.1080/2330443X.2022.2120137 |
| work_keys_str_mv | AT alanhdorfman areanalysisofrepeatabilityandreproducibilityintheamesusdoefbistudy AT richardvalliant areanalysisofrepeatabilityandreproducibilityintheamesusdoefbistudy AT alanhdorfman reanalysisofrepeatabilityandreproducibilityintheamesusdoefbistudy AT richardvalliant reanalysisofrepeatabilityandreproducibilityintheamesusdoefbistudy |