An Accurate Deep Learning–Based System for Automatic Pill Identification: Model Development and Validation
BackgroundMedication errors account for a large proportion of all medical errors. In most homes, patients take a variety of medications for a long period. However, medication errors frequently occur because patients often throw away the containers of their medications....
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2023-01-01
|
| Series: | Journal of Medical Internet Research |
| Online Access: | https://www.jmir.org/2023/1/e41043 |
| _version_ | 1827857711694348288 |
|---|---|
| author | Junyeong Heo Youjin Kang SangKeun Lee Dong-Hwa Jeong Kang-Min Kim |
| author_facet | Junyeong Heo Youjin Kang SangKeun Lee Dong-Hwa Jeong Kang-Min Kim |
| author_sort | Junyeong Heo |
| collection | DOAJ |
| description |
BackgroundMedication errors account for a large proportion of all medical errors. In most homes, patients take a variety of medications for a long period. However, medication errors frequently occur because patients often throw away the containers of their medications.
ObjectiveWe proposed a deep learning–based system for reducing medication errors by accurately identifying prescription pills. Given the pill images, our system located the pills in the respective pill databases in South Korea and the United States.
MethodsWe organized the system into a pill recognition step and pill retrieval step, and we applied deep learning models to train not only images of the pill but also imprinted characters. In the pill recognition step, there are 3 modules that recognize the 3 features of pills and their imprints separately and correct the recognized imprint to fit the actual data. We adopted image classification and text detection models for the feature and imprint recognition modules, respectively. In the imprint correction module, we introduced a language model for the first time in the pill identification system and proposed a novel coordinate encoding technique for effective correction in the language model. We identified pills using similarity scores of pill characteristics with those in the database.
ResultsWe collected the open pill database from South Korea and the United States in May 2022. We used a total of 24,404 pill images in our experiments. The experimental results show that the predicted top-1 candidates achieve accuracy levels of 85.6% (South Korea) and 74.5% (United States) for the types of pills not trained on 2 different databases (South Korea and the United States). Furthermore, the predicted top-1 candidate accuracy of our system was 78% with consumer-granted images, which was achieved by training only 1 image per pill. The results demonstrate that our system could identify and retrieve new pills without additional model updates. Finally, we confirmed through an ablation study that the language model that we emphasized significantly improves the pill identification ability of the system.
ConclusionsOur study proposes the possibility of reducing medical errors by showing that the introduction of artificial intelligence can identify numerous pills with high precision in real time. Our study suggests that the proposed system can reduce patients’ misuse of medications and help medical staff focus on higher-level tasks by simplifying time-consuming lower-level tasks such as pill identification. |
| first_indexed | 2024-03-12T12:45:33Z |
| format | Article |
| id | doaj.art-8955d7ce362743b5ae1c39bc931b9269 |
| institution | Directory Open Access Journal |
| issn | 1438-8871 |
| language | English |
| last_indexed | 2024-03-12T12:45:33Z |
| publishDate | 2023-01-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | Journal of Medical Internet Research |
| spelling | doaj.art-8955d7ce362743b5ae1c39bc931b92692023-08-28T23:26:33ZengJMIR PublicationsJournal of Medical Internet Research1438-88712023-01-0125e4104310.2196/41043An Accurate Deep Learning–Based System for Automatic Pill Identification: Model Development and ValidationJunyeong Heohttps://orcid.org/0000-0001-5525-9069Youjin Kanghttps://orcid.org/0000-0002-6808-5157SangKeun Leehttps://orcid.org/0000-0002-6249-8217Dong-Hwa Jeonghttps://orcid.org/0000-0003-4896-9681Kang-Min Kimhttps://orcid.org/0000-0003-2335-7072 BackgroundMedication errors account for a large proportion of all medical errors. In most homes, patients take a variety of medications for a long period. However, medication errors frequently occur because patients often throw away the containers of their medications. ObjectiveWe proposed a deep learning–based system for reducing medication errors by accurately identifying prescription pills. Given the pill images, our system located the pills in the respective pill databases in South Korea and the United States. MethodsWe organized the system into a pill recognition step and pill retrieval step, and we applied deep learning models to train not only images of the pill but also imprinted characters. In the pill recognition step, there are 3 modules that recognize the 3 features of pills and their imprints separately and correct the recognized imprint to fit the actual data. We adopted image classification and text detection models for the feature and imprint recognition modules, respectively. In the imprint correction module, we introduced a language model for the first time in the pill identification system and proposed a novel coordinate encoding technique for effective correction in the language model. We identified pills using similarity scores of pill characteristics with those in the database. ResultsWe collected the open pill database from South Korea and the United States in May 2022. We used a total of 24,404 pill images in our experiments. The experimental results show that the predicted top-1 candidates achieve accuracy levels of 85.6% (South Korea) and 74.5% (United States) for the types of pills not trained on 2 different databases (South Korea and the United States). Furthermore, the predicted top-1 candidate accuracy of our system was 78% with consumer-granted images, which was achieved by training only 1 image per pill. The results demonstrate that our system could identify and retrieve new pills without additional model updates. Finally, we confirmed through an ablation study that the language model that we emphasized significantly improves the pill identification ability of the system. ConclusionsOur study proposes the possibility of reducing medical errors by showing that the introduction of artificial intelligence can identify numerous pills with high precision in real time. Our study suggests that the proposed system can reduce patients’ misuse of medications and help medical staff focus on higher-level tasks by simplifying time-consuming lower-level tasks such as pill identification.https://www.jmir.org/2023/1/e41043 |
| spellingShingle | Junyeong Heo Youjin Kang SangKeun Lee Dong-Hwa Jeong Kang-Min Kim An Accurate Deep Learning–Based System for Automatic Pill Identification: Model Development and Validation Journal of Medical Internet Research |
| title | An Accurate Deep Learning–Based System for Automatic Pill Identification: Model Development and Validation |
| title_full | An Accurate Deep Learning–Based System for Automatic Pill Identification: Model Development and Validation |
| title_fullStr | An Accurate Deep Learning–Based System for Automatic Pill Identification: Model Development and Validation |
| title_full_unstemmed | An Accurate Deep Learning–Based System for Automatic Pill Identification: Model Development and Validation |
| title_short | An Accurate Deep Learning–Based System for Automatic Pill Identification: Model Development and Validation |
| title_sort | accurate deep learning based system for automatic pill identification model development and validation |
| url | https://www.jmir.org/2023/1/e41043 |
| work_keys_str_mv | AT junyeongheo anaccuratedeeplearningbasedsystemforautomaticpillidentificationmodeldevelopmentandvalidation AT youjinkang anaccuratedeeplearningbasedsystemforautomaticpillidentificationmodeldevelopmentandvalidation AT sangkeunlee anaccuratedeeplearningbasedsystemforautomaticpillidentificationmodeldevelopmentandvalidation AT donghwajeong anaccuratedeeplearningbasedsystemforautomaticpillidentificationmodeldevelopmentandvalidation AT kangminkim anaccuratedeeplearningbasedsystemforautomaticpillidentificationmodeldevelopmentandvalidation AT junyeongheo accuratedeeplearningbasedsystemforautomaticpillidentificationmodeldevelopmentandvalidation AT youjinkang accuratedeeplearningbasedsystemforautomaticpillidentificationmodeldevelopmentandvalidation AT sangkeunlee accuratedeeplearningbasedsystemforautomaticpillidentificationmodeldevelopmentandvalidation AT donghwajeong accuratedeeplearningbasedsystemforautomaticpillidentificationmodeldevelopmentandvalidation AT kangminkim accuratedeeplearningbasedsystemforautomaticpillidentificationmodeldevelopmentandvalidation |