A machine learning framework supporting prospective clinical decisions applied to risk prediction in oncology
Abstract We present a general framework for developing a machine learning (ML) tool that supports clinician assessment of patient risk using electronic health record-derived real-world data and apply the framework to a quality improvement use case in an oncology setting to identify patients at risk...
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2022-08-01
|
| Series: | npj Digital Medicine |
| Online Access: | https://doi.org/10.1038/s41746-022-00660-3 |
| _version_ | 1827775516225044480 |
|---|---|
| author | Lorinda Coombs Abigail Orlando Xiaoliang Wang Pooja Shaw Alexander S. Rich Shreyas Lakhtakia Karen Titchener Blythe Adamson Rebecca A. Miksad Kathi Mooney |
| author_facet | Lorinda Coombs Abigail Orlando Xiaoliang Wang Pooja Shaw Alexander S. Rich Shreyas Lakhtakia Karen Titchener Blythe Adamson Rebecca A. Miksad Kathi Mooney |
| author_sort | Lorinda Coombs |
| collection | DOAJ |
| description | Abstract We present a general framework for developing a machine learning (ML) tool that supports clinician assessment of patient risk using electronic health record-derived real-world data and apply the framework to a quality improvement use case in an oncology setting to identify patients at risk for a near-term (60 day) emergency department (ED) visit who could potentially be eligible for a home-based acute care program. Framework steps include defining clinical quality improvement goals, model development and validation, bias assessment, retrospective and prospective validation, and deployment in clinical workflow. In the retrospective analysis for the use case, 8% of patient encounters were associated with a high risk (pre-defined as predicted probability ≥20%) for a near-term ED visit by the patient. Positive predictive value (PPV) and negative predictive value (NPV) for future ED events was 26% and 91%, respectively. Odds ratio (OR) of ED visit (high- vs. low-risk) was 3.5 (95% CI: 3.4–3.5). The model appeared to be calibrated across racial, gender, and ethnic groups. In the prospective analysis, 10% of patients were classified as high risk, 76% of whom were confirmed by clinicians as eligible for home-based acute care. PPV and NPV for future ED events was 22% and 95%, respectively. OR of ED visit (high- vs. low-risk) was 5.4 (95% CI: 2.6–11.0). The proposed framework for an ML-based tool that supports clinician assessment of patient risk is a stepwise development approach; we successfully applied the framework to an ED visit risk prediction use case. |
| first_indexed | 2024-03-11T13:51:35Z |
| format | Article |
| id | doaj.art-cb2b983d85054f37a35dc77815601424 |
| institution | Directory Open Access Journal |
| issn | 2398-6352 |
| language | English |
| last_indexed | 2024-03-11T13:51:35Z |
| publishDate | 2022-08-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | npj Digital Medicine |
| spelling | doaj.art-cb2b983d85054f37a35dc778156014242023-11-02T08:45:06ZengNature Portfolionpj Digital Medicine2398-63522022-08-01511910.1038/s41746-022-00660-3A machine learning framework supporting prospective clinical decisions applied to risk prediction in oncologyLorinda Coombs0Abigail Orlando1Xiaoliang Wang2Pooja Shaw3Alexander S. Rich4Shreyas Lakhtakia5Karen Titchener6Blythe Adamson7Rebecca A. Miksad8Kathi Mooney9Huntsman Cancer Institute, University of UtahFlatiron Health, IncFlatiron Health, IncFlatiron Health, IncFlatiron Health, IncFlatiron Health, IncHuntsman Cancer Institute, University of UtahFlatiron Health, IncFlatiron Health, IncHuntsman Cancer Institute, University of UtahAbstract We present a general framework for developing a machine learning (ML) tool that supports clinician assessment of patient risk using electronic health record-derived real-world data and apply the framework to a quality improvement use case in an oncology setting to identify patients at risk for a near-term (60 day) emergency department (ED) visit who could potentially be eligible for a home-based acute care program. Framework steps include defining clinical quality improvement goals, model development and validation, bias assessment, retrospective and prospective validation, and deployment in clinical workflow. In the retrospective analysis for the use case, 8% of patient encounters were associated with a high risk (pre-defined as predicted probability ≥20%) for a near-term ED visit by the patient. Positive predictive value (PPV) and negative predictive value (NPV) for future ED events was 26% and 91%, respectively. Odds ratio (OR) of ED visit (high- vs. low-risk) was 3.5 (95% CI: 3.4–3.5). The model appeared to be calibrated across racial, gender, and ethnic groups. In the prospective analysis, 10% of patients were classified as high risk, 76% of whom were confirmed by clinicians as eligible for home-based acute care. PPV and NPV for future ED events was 22% and 95%, respectively. OR of ED visit (high- vs. low-risk) was 5.4 (95% CI: 2.6–11.0). The proposed framework for an ML-based tool that supports clinician assessment of patient risk is a stepwise development approach; we successfully applied the framework to an ED visit risk prediction use case.https://doi.org/10.1038/s41746-022-00660-3 |
| spellingShingle | Lorinda Coombs Abigail Orlando Xiaoliang Wang Pooja Shaw Alexander S. Rich Shreyas Lakhtakia Karen Titchener Blythe Adamson Rebecca A. Miksad Kathi Mooney A machine learning framework supporting prospective clinical decisions applied to risk prediction in oncology npj Digital Medicine |
| title | A machine learning framework supporting prospective clinical decisions applied to risk prediction in oncology |
| title_full | A machine learning framework supporting prospective clinical decisions applied to risk prediction in oncology |
| title_fullStr | A machine learning framework supporting prospective clinical decisions applied to risk prediction in oncology |
| title_full_unstemmed | A machine learning framework supporting prospective clinical decisions applied to risk prediction in oncology |
| title_short | A machine learning framework supporting prospective clinical decisions applied to risk prediction in oncology |
| title_sort | machine learning framework supporting prospective clinical decisions applied to risk prediction in oncology |
| url | https://doi.org/10.1038/s41746-022-00660-3 |
| work_keys_str_mv | AT lorindacoombs amachinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT abigailorlando amachinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT xiaoliangwang amachinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT poojashaw amachinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT alexandersrich amachinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT shreyaslakhtakia amachinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT karentitchener amachinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT blytheadamson amachinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT rebeccaamiksad amachinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT kathimooney amachinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT lorindacoombs machinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT abigailorlando machinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT xiaoliangwang machinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT poojashaw machinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT alexandersrich machinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT shreyaslakhtakia machinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT karentitchener machinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT blytheadamson machinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT rebeccaamiksad machinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology AT kathimooney machinelearningframeworksupportingprospectiveclinicaldecisionsappliedtoriskpredictioninoncology |