Improved Evaluation Metrics for Sentence Suggestions in Nursing and Elderly Care Record Applications
This paper presents a new approach called EmbedHDP, which aims to enhance the evaluation models utilized for assessing sentence suggestions in nursing care record applications. The primary objective is to determine the alignment of the proposed evaluation metric with human evaluators who are caregiv...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-01-01
|
| Series: | Healthcare |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-9032/12/3/367 |
| _version_ | 1797318734702968832 |
|---|---|
| author | Defry Hamdhana Haru Kaneko John Noel Victorino Sozo Inoue |
| author_facet | Defry Hamdhana Haru Kaneko John Noel Victorino Sozo Inoue |
| author_sort | Defry Hamdhana |
| collection | DOAJ |
| description | This paper presents a new approach called EmbedHDP, which aims to enhance the evaluation models utilized for assessing sentence suggestions in nursing care record applications. The primary objective is to determine the alignment of the proposed evaluation metric with human evaluators who are caregivers. It is crucial due to the direct relevance of the provided provided to the health or condition of the elderly. The motivation for this proposal arises from challenges observed in previous models. Our analysis examines the mechanisms of current evaluation metrics such as BERTScore, cosine similarity, ROUGE, and BLEU to achieve reliable metrics evaluation. Several limitations were identified. In some cases, BERTScore encountered difficulties in effectively evaluating the nursing care record domain and consistently providing quality assessments of generated sentence suggestions above 60%. Cosine similarity is a widely used method, but it has limitations regarding word order. This can lead to potential misjudgments of semantic differences within similar word sets. Another technique, ROUGE, relies on lexical overlap but tends to ignore semantic accuracy. Additionally, while BLEU is helpful, it may not fully capture semantic coherence in its evaluations. After calculating the correlation coefficient, it was found that EmbedHDP is effective in evaluating nurse care records due to its ability to handle a variety of sentence structures and medical terminology, providing differentiated and contextually relevant assessments. Additionally, this research used a dataset comprising 320 pairs of sentences with correspondingly equivalent lengths. The results revealed that EmbedHDP outperformed other evaluation models, achieving a coefficient score of 61%, followed by cosine similarity, with a score of 59%, and BERTScore, with 58%. This shows the effectiveness of our proposed approach in improving the evaluation of sentence suggestions in nursing care record applications. |
| first_indexed | 2024-03-08T03:56:39Z |
| format | Article |
| id | doaj.art-645b57e362c64d4d82642bc8f6cdd88f |
| institution | Directory Open Access Journal |
| issn | 2227-9032 |
| language | English |
| last_indexed | 2024-03-08T03:56:39Z |
| publishDate | 2024-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Healthcare |
| spelling | doaj.art-645b57e362c64d4d82642bc8f6cdd88f2024-02-09T15:12:45ZengMDPI AGHealthcare2227-90322024-01-0112336710.3390/healthcare12030367Improved Evaluation Metrics for Sentence Suggestions in Nursing and Elderly Care Record ApplicationsDefry Hamdhana0Haru Kaneko1John Noel Victorino2Sozo Inoue3Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, Kitakyushu 808-0196, JapanGraduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, Kitakyushu 808-0196, JapanGraduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, Kitakyushu 808-0196, JapanGraduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, Kitakyushu 808-0196, JapanThis paper presents a new approach called EmbedHDP, which aims to enhance the evaluation models utilized for assessing sentence suggestions in nursing care record applications. The primary objective is to determine the alignment of the proposed evaluation metric with human evaluators who are caregivers. It is crucial due to the direct relevance of the provided provided to the health or condition of the elderly. The motivation for this proposal arises from challenges observed in previous models. Our analysis examines the mechanisms of current evaluation metrics such as BERTScore, cosine similarity, ROUGE, and BLEU to achieve reliable metrics evaluation. Several limitations were identified. In some cases, BERTScore encountered difficulties in effectively evaluating the nursing care record domain and consistently providing quality assessments of generated sentence suggestions above 60%. Cosine similarity is a widely used method, but it has limitations regarding word order. This can lead to potential misjudgments of semantic differences within similar word sets. Another technique, ROUGE, relies on lexical overlap but tends to ignore semantic accuracy. Additionally, while BLEU is helpful, it may not fully capture semantic coherence in its evaluations. After calculating the correlation coefficient, it was found that EmbedHDP is effective in evaluating nurse care records due to its ability to handle a variety of sentence structures and medical terminology, providing differentiated and contextually relevant assessments. Additionally, this research used a dataset comprising 320 pairs of sentences with correspondingly equivalent lengths. The results revealed that EmbedHDP outperformed other evaluation models, achieving a coefficient score of 61%, followed by cosine similarity, with a score of 59%, and BERTScore, with 58%. This shows the effectiveness of our proposed approach in improving the evaluation of sentence suggestions in nursing care record applications.https://www.mdpi.com/2227-9032/12/3/367sentence suggestionnursing care recordevaluation metrics |
| spellingShingle | Defry Hamdhana Haru Kaneko John Noel Victorino Sozo Inoue Improved Evaluation Metrics for Sentence Suggestions in Nursing and Elderly Care Record Applications Healthcare sentence suggestion nursing care record evaluation metrics |
| title | Improved Evaluation Metrics for Sentence Suggestions in Nursing and Elderly Care Record Applications |
| title_full | Improved Evaluation Metrics for Sentence Suggestions in Nursing and Elderly Care Record Applications |
| title_fullStr | Improved Evaluation Metrics for Sentence Suggestions in Nursing and Elderly Care Record Applications |
| title_full_unstemmed | Improved Evaluation Metrics for Sentence Suggestions in Nursing and Elderly Care Record Applications |
| title_short | Improved Evaluation Metrics for Sentence Suggestions in Nursing and Elderly Care Record Applications |
| title_sort | improved evaluation metrics for sentence suggestions in nursing and elderly care record applications |
| topic | sentence suggestion nursing care record evaluation metrics |
| url | https://www.mdpi.com/2227-9032/12/3/367 |
| work_keys_str_mv | AT defryhamdhana improvedevaluationmetricsforsentencesuggestionsinnursingandelderlycarerecordapplications AT harukaneko improvedevaluationmetricsforsentencesuggestionsinnursingandelderlycarerecordapplications AT johnnoelvictorino improvedevaluationmetricsforsentencesuggestionsinnursingandelderlycarerecordapplications AT sozoinoue improvedevaluationmetricsforsentencesuggestionsinnursingandelderlycarerecordapplications |