Achieving high inter-rater reliability in establishing data labels: a retrospective chart review study

Achieving high inter-rater reliability in establishing data labels: a retrospective chart review study

Background In medical research, the effectiveness of machine learning algorithms depends heavily on the accuracy of labeled data. This study aimed to assess inter-rater reliability (IRR) in a retrospective electronic medical chart review to create high quality labeled data on comorbidities and adver...

Full description

Bibliographic Details
Main Authors:	Hude Quan, Danielle A Southern, Yuan Xu, Cathy Eastwood, Guosong Wu, Natalie Sapiro, Cheligeer Cheligeer
Format:	Article
Language:	English
Published:	BMJ Publishing Group 2024-04-01
Series:	BMJ Open Quality
Online Access:	https://bmjopenquality.bmj.com/content/13/2/e002722.full

Similar Items

Improving Detection of Hospital Adverse Events Using Machine Learning on Real-World Narrative EMR Data
by: Cheligeer Cheligeer, et al.
Published: (2024-09-01)

Administrative health data validity: Changes over 19 years
by: Jie Pan, et al.
Published: (2024-09-01)

Development of machine learning models for the detection of surgical site infections following total hip and knee arthroplasty: a multicenter cohort study
by: Guosong Wu, et al.
Published: (2023-09-01)

Enhancing Disease Detection in Electronic Medical Records: Integrating Human Expertise and Large Language Models with Application to Diabetes, Hypertension, and Acute Myocardial Infarction
by: Jie Pan, et al.
Published: (2024-09-01)

Health Data Governance for Research Use in Alberta
by: Namneet Sandhu, et al.
Published: (2023-10-01)

Intra‐rater and inter‐rater reliability of robotic arthrometer DYNEELAX®
by: Katja Mihalinec, et al.
Published: (2024-07-01)

Inter-rater and Intra-rater Reliability of Het’s MMT in Pelvic Floor Assessment
by: Dhara Santosh Agnihotri, et al.
Published: (2024-12-01)

Inter-Rater Reliability & Value of the Hawkins Sign
by: Logan Lake, et al.
Published: (2024-12-01)

Intra-Rater and Inter-Rater Reliability of the Kinvent Hand-Held Dynamometer in Young Adults
by: Manuel Barbosa de Almeida, et al.
Published: (2023-08-01)

Inter-rater reliability of the assessment of communication skills in Pharmacology
by: P Ravi Shankar, et al.
Published: (2003-11-01)

THE COMMUNICATION DEALL DEVELOPMENTAL CHECKLIST - INTER RATER RELIABILITY
by: Pratibha Karanth
Published: (2011-05-01)

An assessment of the inter-rater and intra-rater reliability of the modified Gordon pin infection classification system
by: Anirejuoritse Bafor, et al.
Published: (2024-10-01)

Inter-rater and intra-rater reliability of isotonic exercise monitoring device for measuring active knee extension
by: Chonnanid Limsakul, et al.
Published: (2023-01-01)

Inter-rater reliability of vehicle color perception for forensic intelligence
by: Khai, Lee, et al.
Published: (2019)

Comparison of Inter-Rater Reliability Techniques in Performance-Based Assessment
by: Sinem Arslan Mancar, et al.
Published: (2022-06-01)

Intrarater and Inter-rater Reliability of Tibial Plateau Fracture Classifications
by: Fardis Vosoughi, MD, et al.
Published: (2024-12-01)

Grant Peer Review: Improving Inter-Rater Reliability with Training.
by: David N Sattler, et al.
Published: (2015-01-01)

Inter-rater Reliability of Triages Performed by the Electronic Triage System
by: Faramarz Pourasghar, et al.
Published: (2015-10-01)

Inter-rater reliability of nursing home quality indicators in the U.S
by: Roy Jason, et al.
Published: (2003-11-01)

The validity and inter-rater reliability of project assessment in mathematics learning
by: Kusaeri Kusaeri, et al.
Published: (2019-05-01)

Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
by: Kevin A. Hallgren
Published: (2012-02-01)

Ultrasound Imaging Analysis of the Lumbar Multifidus Muscle Echo Intensity: Intra-Rater and Inter-Rater Reliability of a Novice and an Experienced Rater
by: Maryse Fortin, et al.
Published: (2021-05-01)

Reducing variability, increasing reliability: exploring the psychology of intra- and inter-rater reliability
by: Aslett, Helen J.
Published: (2006)

Reliability of ultrasound assessment of the rectus femoris muscle thickness: intra-rater, inter-rater, and inter-day analysis accounting for age and sex
by: Martin Bjørn Stausholm, et al.
Published: (2024-11-01)

Strengths and Barriers to Coding Hospital Chart Information from Health Information Manager Perspectives: A Qualitative Study
by: Chelsea Doktorchik, et al.
Published: (2018-09-01)

Revised Hammersmith Scale for spinal muscular atrophy: Inter and intra-rater reliability and agreement.
by: Danielle Ramsey, et al.
Published: (2022-01-01)

The intra-rater and inter-rater reliability of a method measuring the sagittal curvature of the spine using surface topography
by: Hannink, E, et al.
Published: (2019)

Ultrasound imaging in patients with hip pain and suspected hip osteoarthritis: an inter-rater and intra-rater reliability study
by: Lene Terslev, et al.
Published: (2020-11-01)

Improving inter-rater reliability of the enhancing assessment of common therapeutic factors (ENACT) measure through training of raters
by: Mwamba M. Mwenge, et al.
Published: (2022-09-01)

Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters
by: Ming Li, et al.
Published: (2023-08-01)

Inter-rater reliability in labeling quality and pathological features of retinal OCT scans: A customized annotation software approach.
by: Katherine Du, et al.
Published: (2024-01-01)

Inter-rater reliability of risk of bias tools for non-randomized studies
by: Isabel Kalaycioglu, et al.
Published: (2023-12-01)

Inter-rater reliability of measures to characterize the tobacco retail environment in Mexico
by: Marissa G Hall, et al.
Published: (2015-11-01)

Content validity and inter-rater reliability of the Comprehensive Quality of Life Scale
by: Alicia Boluarte Carbajal, et al.
Published: (2017-07-01)

Inter-rater reliability of an ultrasound protocol to evaluate the anterolateral ligament of the knee
by: Michel Kandel, et al.
Published: (2019-09-01)

Inter-rater reliability of the QuIS as an assessment of the quality of staff-inpatient interactions
by: Ines Mesa-Eguiagaray, et al.
Published: (2016-12-01)

Inter-rater reliability of the extended Composite Quality Score (CQS-2)
by: Steffen Mickenautsch, et al.
Published: (2023-08-01)

Evaluating Inter-Rater Reliability: Transitioning to a Single Rater for Marking Modified Essay Questions in Undergraduate Medical Education
by: Shahid Hassan, et al.
Published: (2024-11-01)

Intersession Intra-Rater and Inter-Rater Reliability of Myotonometer for Upper and Lower Extremity Muscles in Children with Spinal Muscular Atrophy
by: Seval Kutlutürk Yıkılmaz, et al.
Published: (2024-10-01)

Intra-Rater and Inter-Rater Reliability of Pressure Pain Algometry of the Sural and Tibial Nerves in Asymptomatic Elite Youth Footballers
by: Daniel Richards, et al.
Published: (2021-09-01)