Attention-based neural networks for clinical prediction modelling on electronic health records
Background Deep learning models have had a lot of success in various fields. However, on structured data they have struggled. Here we apply four state-of-the-art supervised deep learning models using the attention mechanism and compare against logistic regression and XGBoost using d...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
BioMed Central
2023
|
| Online Access: | https://hdl.handle.net/1721.1/153169 |
| _version_ | 1811093172278788096 |
|---|---|
| author | Fridgeirsson, Egill A. Sontag, David Rijnbeek, Peter |
| author2 | Massachusetts Institute of Technology. Institute for Medical Engineering & Science |
| author_facet | Massachusetts Institute of Technology. Institute for Medical Engineering & Science Fridgeirsson, Egill A. Sontag, David Rijnbeek, Peter |
| author_sort | Fridgeirsson, Egill A. |
| collection | MIT |
| description | Background
Deep learning models have had a lot of success in various fields. However, on structured data they have struggled. Here we apply four state-of-the-art supervised deep learning models using the attention mechanism and compare against logistic regression and XGBoost using discrimination, calibration and clinical utility.
Methods
We develop the models using a general practitioners database. We implement a recurrent neural network, a transformer with and without reverse distillation and a graph neural network. We measure discrimination using the area under the receiver operating characteristic curve (AUC) and the area under the precision recall curve (AUPRC). We assess smooth calibration using restricted cubic splines and clinical utility with decision curve analysis.
Results
Our results show that deep learning approaches can improve discrimination up to 2.5% points AUC and 7.4% points AUPRC. However, on average the baselines are competitive. Most models are similarly calibrated as the baselines except for the graph neural network. The transformer using reverse distillation shows the best performance in clinical utility on two out of three prediction problems over most of the prediction thresholds.
Conclusion
In this study, we evaluated various approaches in supervised learning using neural networks and attention. Here we do a rigorous comparison, not only looking at discrimination but also calibration and clinical utility. There is value in using deep learning models on electronic health record data since it can improve discrimination and clinical utility while providing good calibration. However, good baseline methods are still competitive. |
| first_indexed | 2024-09-23T15:40:49Z |
| format | Article |
| id | mit-1721.1/153169 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T15:40:49Z |
| publishDate | 2023 |
| publisher | BioMed Central |
| record_format | dspace |
| spelling | mit-1721.1/1531692024-03-20T19:26:51Z Attention-based neural networks for clinical prediction modelling on electronic health records Fridgeirsson, Egill A. Sontag, David Rijnbeek, Peter Massachusetts Institute of Technology. Institute for Medical Engineering & Science Background Deep learning models have had a lot of success in various fields. However, on structured data they have struggled. Here we apply four state-of-the-art supervised deep learning models using the attention mechanism and compare against logistic regression and XGBoost using discrimination, calibration and clinical utility. Methods We develop the models using a general practitioners database. We implement a recurrent neural network, a transformer with and without reverse distillation and a graph neural network. We measure discrimination using the area under the receiver operating characteristic curve (AUC) and the area under the precision recall curve (AUPRC). We assess smooth calibration using restricted cubic splines and clinical utility with decision curve analysis. Results Our results show that deep learning approaches can improve discrimination up to 2.5% points AUC and 7.4% points AUPRC. However, on average the baselines are competitive. Most models are similarly calibrated as the baselines except for the graph neural network. The transformer using reverse distillation shows the best performance in clinical utility on two out of three prediction problems over most of the prediction thresholds. Conclusion In this study, we evaluated various approaches in supervised learning using neural networks and attention. Here we do a rigorous comparison, not only looking at discrimination but also calibration and clinical utility. There is value in using deep learning models on electronic health record data since it can improve discrimination and clinical utility while providing good calibration. However, good baseline methods are still competitive. 2023-12-14T20:31:12Z 2023-12-14T20:31:12Z 2023-12-07 2023-12-10T04:07:47Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/153169 BMC Medical Research Methodology. 2023 Dec 07;23(1):285 PUBLISHER_CC en https://doi.org/10.1186/s12874-023-02112-2 Creative Commons Attribution https://creativecommons.org/licenses/by/4.0/ The Author(s) application/pdf BioMed Central BioMed Central |
| spellingShingle | Fridgeirsson, Egill A. Sontag, David Rijnbeek, Peter Attention-based neural networks for clinical prediction modelling on electronic health records |
| title | Attention-based neural networks for clinical prediction modelling on electronic health records |
| title_full | Attention-based neural networks for clinical prediction modelling on electronic health records |
| title_fullStr | Attention-based neural networks for clinical prediction modelling on electronic health records |
| title_full_unstemmed | Attention-based neural networks for clinical prediction modelling on electronic health records |
| title_short | Attention-based neural networks for clinical prediction modelling on electronic health records |
| title_sort | attention based neural networks for clinical prediction modelling on electronic health records |
| url | https://hdl.handle.net/1721.1/153169 |
| work_keys_str_mv | AT fridgeirssonegilla attentionbasedneuralnetworksforclinicalpredictionmodellingonelectronichealthrecords AT sontagdavid attentionbasedneuralnetworksforclinicalpredictionmodellingonelectronichealthrecords AT rijnbeekpeter attentionbasedneuralnetworksforclinicalpredictionmodellingonelectronichealthrecords |