Explanation and prediction of clinical data with imbalanced class distribution based on pattern discovery and disentanglement
Abstract Background Statistical data analysis, especially the advanced machine learning (ML) methods, have attracted considerable interest in clinical practices. We are looking for interpretability of the diagnostic/prognostic results that will bring confidence to doctors, patients and their relativ...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2021-01-01
|
| Series: | BMC Medical Informatics and Decision Making |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12911-020-01356-y |
| _version_ | 1818953223002652672 |
|---|---|
| author | Pei-Yuan Zhou Andrew K. C. Wong |
| author_facet | Pei-Yuan Zhou Andrew K. C. Wong |
| author_sort | Pei-Yuan Zhou |
| collection | DOAJ |
| description | Abstract Background Statistical data analysis, especially the advanced machine learning (ML) methods, have attracted considerable interest in clinical practices. We are looking for interpretability of the diagnostic/prognostic results that will bring confidence to doctors, patients and their relatives in therapeutics and clinical practice. When datasets are imbalanced in diagnostic categories, we notice that the ordinary ML methods might produce results overwhelmed by the majority classes diminishing prediction accuracy. Hence, it needs methods that could produce explicit transparent and interpretable results in decision-making, without sacrificing accuracy, even for data with imbalanced groups. Methods In order to interpret the clinical patterns and conduct diagnostic prediction of patients with high accuracy, we develop a novel method, Pattern Discovery and Disentanglement for Clinical Data Analysis (cPDD), which is able to discover patterns (correlated traits/indicants) and use them to classify clinical data even if the class distribution is imbalanced. In the most general setting, a relational dataset is a large table such that each column represents an attribute (trait/indicant), and each row contains a set of attribute values (AVs) of an entity (patient). Compared to the existing pattern discovery approaches, cPDD can discover a small succinct set of statistically significant high-order patterns from clinical data for interpreting and predicting the disease class of the patients even with groups small and rare. Results Experiments on synthetic and thoracic clinical dataset showed that cPDD can 1) discover a smaller set of succinct significant patterns compared to other existing pattern discovery methods; 2) allow the users to interpret succinct sets of patterns coming from uncorrelated sources, even the groups are rare/small; and 3) obtain better performance in prediction compared to other interpretable classification approaches. Conclusions In conclusion, cPDD discovers fewer patterns with greater comprehensive coverage to improve the interpretability of patterns discovered. Experimental results on synthetic data validated that cPDD discovers all patterns implanted in the data, displays them precisely and succinctly with statistical support for interpretation and prediction, a capability which the traditional ML methods lack. The success of cPDD as a novel interpretable method in solving the imbalanced class problem shows its great potential to clinical data analysis for years to come. |
| first_indexed | 2024-12-20T10:02:51Z |
| format | Article |
| id | doaj.art-485ac8fd294340b7ba8237a4a210f538 |
| institution | Directory Open Access Journal |
| issn | 1472-6947 |
| language | English |
| last_indexed | 2024-12-20T10:02:51Z |
| publishDate | 2021-01-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Informatics and Decision Making |
| spelling | doaj.art-485ac8fd294340b7ba8237a4a210f5382022-12-21T19:44:18ZengBMCBMC Medical Informatics and Decision Making1472-69472021-01-0121111510.1186/s12911-020-01356-yExplanation and prediction of clinical data with imbalanced class distribution based on pattern discovery and disentanglementPei-Yuan Zhou0Andrew K. C. Wong1Systems Design Engineering, University of WaterlooSystems Design Engineering, University of WaterlooAbstract Background Statistical data analysis, especially the advanced machine learning (ML) methods, have attracted considerable interest in clinical practices. We are looking for interpretability of the diagnostic/prognostic results that will bring confidence to doctors, patients and their relatives in therapeutics and clinical practice. When datasets are imbalanced in diagnostic categories, we notice that the ordinary ML methods might produce results overwhelmed by the majority classes diminishing prediction accuracy. Hence, it needs methods that could produce explicit transparent and interpretable results in decision-making, without sacrificing accuracy, even for data with imbalanced groups. Methods In order to interpret the clinical patterns and conduct diagnostic prediction of patients with high accuracy, we develop a novel method, Pattern Discovery and Disentanglement for Clinical Data Analysis (cPDD), which is able to discover patterns (correlated traits/indicants) and use them to classify clinical data even if the class distribution is imbalanced. In the most general setting, a relational dataset is a large table such that each column represents an attribute (trait/indicant), and each row contains a set of attribute values (AVs) of an entity (patient). Compared to the existing pattern discovery approaches, cPDD can discover a small succinct set of statistically significant high-order patterns from clinical data for interpreting and predicting the disease class of the patients even with groups small and rare. Results Experiments on synthetic and thoracic clinical dataset showed that cPDD can 1) discover a smaller set of succinct significant patterns compared to other existing pattern discovery methods; 2) allow the users to interpret succinct sets of patterns coming from uncorrelated sources, even the groups are rare/small; and 3) obtain better performance in prediction compared to other interpretable classification approaches. Conclusions In conclusion, cPDD discovers fewer patterns with greater comprehensive coverage to improve the interpretability of patterns discovered. Experimental results on synthetic data validated that cPDD discovers all patterns implanted in the data, displays them precisely and succinctly with statistical support for interpretation and prediction, a capability which the traditional ML methods lack. The success of cPDD as a novel interpretable method in solving the imbalanced class problem shows its great potential to clinical data analysis for years to come.https://doi.org/10.1186/s12911-020-01356-yPattern discoveryDisentanglementClinical decision-makingImbalance classification |
| spellingShingle | Pei-Yuan Zhou Andrew K. C. Wong Explanation and prediction of clinical data with imbalanced class distribution based on pattern discovery and disentanglement BMC Medical Informatics and Decision Making Pattern discovery Disentanglement Clinical decision-making Imbalance classification |
| title | Explanation and prediction of clinical data with imbalanced class distribution based on pattern discovery and disentanglement |
| title_full | Explanation and prediction of clinical data with imbalanced class distribution based on pattern discovery and disentanglement |
| title_fullStr | Explanation and prediction of clinical data with imbalanced class distribution based on pattern discovery and disentanglement |
| title_full_unstemmed | Explanation and prediction of clinical data with imbalanced class distribution based on pattern discovery and disentanglement |
| title_short | Explanation and prediction of clinical data with imbalanced class distribution based on pattern discovery and disentanglement |
| title_sort | explanation and prediction of clinical data with imbalanced class distribution based on pattern discovery and disentanglement |
| topic | Pattern discovery Disentanglement Clinical decision-making Imbalance classification |
| url | https://doi.org/10.1186/s12911-020-01356-y |
| work_keys_str_mv | AT peiyuanzhou explanationandpredictionofclinicaldatawithimbalancedclassdistributionbasedonpatterndiscoveryanddisentanglement AT andrewkcwong explanationandpredictionofclinicaldatawithimbalancedclassdistributionbasedonpatterndiscoveryanddisentanglement |