Development and evaluation of a survey instrument to assess veterinary medical record suitability for multi-center research studies
Here we describe the development and evaluation of a survey instrument to assess the research suitability of veterinary electronic medical records (EMRs) through the conduct of two studies as part of the Dog Aging Project (DAP). In study 1, four reviewers used the instrument to score a total of 218...
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-08-01
|
| Series: | Frontiers in Veterinary Science |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fvets.2022.941036/full |
| _version_ | 1811220409543032832 |
|---|---|
| author | Dora Praczko Amanda K. Tinkle Crystal R. Arkenberg Robyn L. McClelland Kate E. Creevy M. Katherine Tolbert Brian G. Barnett Lucy Chou Jeremy Evans Kellyn E. McNulty Dog Aging Project Consortium Jonathan M. Levine |
| author_facet | Dora Praczko Amanda K. Tinkle Crystal R. Arkenberg Robyn L. McClelland Kate E. Creevy M. Katherine Tolbert Brian G. Barnett Lucy Chou Jeremy Evans Kellyn E. McNulty Dog Aging Project Consortium Jonathan M. Levine |
| author_sort | Dora Praczko |
| collection | DOAJ |
| description | Here we describe the development and evaluation of a survey instrument to assess the research suitability of veterinary electronic medical records (EMRs) through the conduct of two studies as part of the Dog Aging Project (DAP). In study 1, four reviewers used the instrument to score a total of 218 records in an overlapping matrix of pairs to assess inter-rater agreement with respect to appropriate format (qualification), identification match (verification), and record quality. Based upon the moderate inter-rater agreement with respect to verification and the relatively large number of records that were incorrectly rejected the instrument was modified and more specific instructions were provided. In study 2, a modified instrument was again completed by four reviewers to score 100 different EMRs. The survey scores were compared to a gold standard of board-certified specialist review to determine receiver operating curve statistics. The refined survey had substantial inter-rater agreement across most qualification and verification questions. The cut-off value identified had a sensitivity of 95 and 96% (by reviewer 1 and reviewer 2, respectively) and a specificity of 82% and 91% (by reviewer 1 and reviewer 2, respectively) to predict gold standard acceptance or rejection of the record. Using just qualification and verification questions within the instrument (as opposed to full scoring) minimally impacted sensitivity and specificity and resulted in substantial time savings in the review process. |
| first_indexed | 2024-04-12T07:41:09Z |
| format | Article |
| id | doaj.art-a3cc331eb463429aab595ac2eb678b9f |
| institution | Directory Open Access Journal |
| issn | 2297-1769 |
| language | English |
| last_indexed | 2024-04-12T07:41:09Z |
| publishDate | 2022-08-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Veterinary Science |
| spelling | doaj.art-a3cc331eb463429aab595ac2eb678b9f2022-12-22T03:41:48ZengFrontiers Media S.A.Frontiers in Veterinary Science2297-17692022-08-01910.3389/fvets.2022.941036941036Development and evaluation of a survey instrument to assess veterinary medical record suitability for multi-center research studiesDora Praczko0Amanda K. Tinkle1Crystal R. Arkenberg2Robyn L. McClelland3Kate E. Creevy4M. Katherine Tolbert5Brian G. Barnett6Lucy Chou7Jeremy Evans8Kellyn E. McNulty9Dog Aging Project Consortium10Jonathan M. Levine11Department of Small Animal Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Texas A&M University, College Station, TX, United StatesDepartment of Small Animal Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Texas A&M University, College Station, TX, United StatesDepartment of Small Animal Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Texas A&M University, College Station, TX, United StatesDepartment of Biostatistics, University of Washington, Seattle, WA, United StatesDepartment of Small Animal Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Texas A&M University, College Station, TX, United StatesDepartment of Small Animal Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Texas A&M University, College Station, TX, United StatesDepartment of Small Animal Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Texas A&M University, College Station, TX, United StatesDepartment of Small Animal Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Texas A&M University, College Station, TX, United StatesDepartment of Small Animal Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Texas A&M University, College Station, TX, United StatesDepartment of Small Animal Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Texas A&M University, College Station, TX, United StatesDepartment of Small Animal Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Texas A&M University, College Station, TX, United StatesDepartment of Small Animal Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Texas A&M University, College Station, TX, United StatesHere we describe the development and evaluation of a survey instrument to assess the research suitability of veterinary electronic medical records (EMRs) through the conduct of two studies as part of the Dog Aging Project (DAP). In study 1, four reviewers used the instrument to score a total of 218 records in an overlapping matrix of pairs to assess inter-rater agreement with respect to appropriate format (qualification), identification match (verification), and record quality. Based upon the moderate inter-rater agreement with respect to verification and the relatively large number of records that were incorrectly rejected the instrument was modified and more specific instructions were provided. In study 2, a modified instrument was again completed by four reviewers to score 100 different EMRs. The survey scores were compared to a gold standard of board-certified specialist review to determine receiver operating curve statistics. The refined survey had substantial inter-rater agreement across most qualification and verification questions. The cut-off value identified had a sensitivity of 95 and 96% (by reviewer 1 and reviewer 2, respectively) and a specificity of 82% and 91% (by reviewer 1 and reviewer 2, respectively) to predict gold standard acceptance or rejection of the record. Using just qualification and verification questions within the instrument (as opposed to full scoring) minimally impacted sensitivity and specificity and resulted in substantial time savings in the review process.https://www.frontiersin.org/articles/10.3389/fvets.2022.941036/fullelectronic medical recordinter-rater agreementclinical trialverificationpoint score |
| spellingShingle | Dora Praczko Amanda K. Tinkle Crystal R. Arkenberg Robyn L. McClelland Kate E. Creevy M. Katherine Tolbert Brian G. Barnett Lucy Chou Jeremy Evans Kellyn E. McNulty Dog Aging Project Consortium Jonathan M. Levine Development and evaluation of a survey instrument to assess veterinary medical record suitability for multi-center research studies Frontiers in Veterinary Science electronic medical record inter-rater agreement clinical trial verification point score |
| title | Development and evaluation of a survey instrument to assess veterinary medical record suitability for multi-center research studies |
| title_full | Development and evaluation of a survey instrument to assess veterinary medical record suitability for multi-center research studies |
| title_fullStr | Development and evaluation of a survey instrument to assess veterinary medical record suitability for multi-center research studies |
| title_full_unstemmed | Development and evaluation of a survey instrument to assess veterinary medical record suitability for multi-center research studies |
| title_short | Development and evaluation of a survey instrument to assess veterinary medical record suitability for multi-center research studies |
| title_sort | development and evaluation of a survey instrument to assess veterinary medical record suitability for multi center research studies |
| topic | electronic medical record inter-rater agreement clinical trial verification point score |
| url | https://www.frontiersin.org/articles/10.3389/fvets.2022.941036/full |
| work_keys_str_mv | AT dorapraczko developmentandevaluationofasurveyinstrumenttoassessveterinarymedicalrecordsuitabilityformulticenterresearchstudies AT amandaktinkle developmentandevaluationofasurveyinstrumenttoassessveterinarymedicalrecordsuitabilityformulticenterresearchstudies AT crystalrarkenberg developmentandevaluationofasurveyinstrumenttoassessveterinarymedicalrecordsuitabilityformulticenterresearchstudies AT robynlmcclelland developmentandevaluationofasurveyinstrumenttoassessveterinarymedicalrecordsuitabilityformulticenterresearchstudies AT kateecreevy developmentandevaluationofasurveyinstrumenttoassessveterinarymedicalrecordsuitabilityformulticenterresearchstudies AT mkatherinetolbert developmentandevaluationofasurveyinstrumenttoassessveterinarymedicalrecordsuitabilityformulticenterresearchstudies AT briangbarnett developmentandevaluationofasurveyinstrumenttoassessveterinarymedicalrecordsuitabilityformulticenterresearchstudies AT lucychou developmentandevaluationofasurveyinstrumenttoassessveterinarymedicalrecordsuitabilityformulticenterresearchstudies AT jeremyevans developmentandevaluationofasurveyinstrumenttoassessveterinarymedicalrecordsuitabilityformulticenterresearchstudies AT kellynemcnulty developmentandevaluationofasurveyinstrumenttoassessveterinarymedicalrecordsuitabilityformulticenterresearchstudies AT dogagingprojectconsortium developmentandevaluationofasurveyinstrumenttoassessveterinarymedicalrecordsuitabilityformulticenterresearchstudies AT jonathanmlevine developmentandevaluationofasurveyinstrumenttoassessveterinarymedicalrecordsuitabilityformulticenterresearchstudies |