Multifaceted Natural Language Processing Task-Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation.

Autor:	Kim K; Interdisciplinary Program for Bioengineering, Seoul National University, Seoul, Republic of Korea., Park S; Seoul National University Medical Research Center, Seoul, Republic of Korea., Min J; Interdisciplinary Program for Bioengineering, Seoul National University, Seoul, Republic of Korea., Park S; Institute of Medical and Biological Engineering, Medical Research Center, Seoul National University, Seoul, Republic of Korea., Kim JY; Division of Rheumatology, Department of Internal Medicine, Seoul National University Hospital, Seoul, Republic of Korea., Eun J; Human Computer Interaction and Design Lab, Seoul National University, Seoul, Republic of Korea., Jung K; Human Computer Interaction and Design Lab, Seoul National University, Seoul, Republic of Korea., Park YE; Human Computer Interaction and Design Lab, Seoul National University, Seoul, Republic of Korea., Kim E; Human Computer Interaction and Design Lab, Seoul National University, Seoul, Republic of Korea., Lee EY; Division of Rheumatology, Department of Internal Medicine, Seoul National University Hospital, Seoul, Republic of Korea., Lee J; Human Computer Interaction and Design Lab, Seoul National University, Seoul, Republic of Korea., Choi J; Institute of Medical and Biological Engineering, Medical Research Center, Seoul National University, Seoul, Republic of Korea.; Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, Republic of Korea, 82 2-766-3421.
Jazyk:	angličtina
Zdroj:	JMIR medical informatics [JMIR Med Inform] 2024 Oct 30; Vol. 12, pp. e52897. Date of Electronic Publication: 2024 Oct 30.
DOI:	10.2196/52897
Abstrakt:	Background: The bidirectional encoder representations from transformers (BERT) model has attracted considerable attention in clinical applications, such as patient classification and disease prediction. However, current studies have typically progressed to application development without a thorough assessment of the model's comprehension of clinical context. Furthermore, limited comparative studies have been conducted on BERT models using medical documents from non-English-speaking countries. Therefore, the applicability of BERT models trained on English clinical notes to non-English contexts is yet to be confirmed. To address these gaps in literature, this study focused on identifying the most effective BERT model for non-English clinical notes. Objective: In this study, we evaluated the contextual understanding abilities of various BERT models applied to mixed Korean and English clinical notes. The objective of this study was to identify the BERT model that excels in understanding the context of such documents. Methods: Using data from 164,460 patients in a South Korean tertiary hospital, we pretrained BERT-base, BERT for Biomedical Text Mining (BioBERT), Korean BERT (KoBERT), and Multilingual BERT (M-BERT) to improve their contextual comprehension capabilities and subsequently compared their performances in 7 fine-tuning tasks. Results: The model performance varied based on the task and token usage. First, BERT-base and BioBERT excelled in tasks using classification ([CLS]) token embeddings, such as document classification. BioBERT achieved the highest F1-score of 89.32. Both BERT-base and BioBERT demonstrated their effectiveness in document pattern recognition, even with limited Korean tokens in the dictionary. Second, M-BERT exhibited a superior performance in reading comprehension tasks, achieving an F1-score of 93.77. Better results were obtained when fewer words were replaced with unknown ([UNK]) tokens. Third, M-BERT excelled in the knowledge inference task in which correct disease names were inferred from 63 candidate disease names in a document with disease names replaced with [MASK] tokens. M-BERT achieved the highest hit@10 score of 95.41. Conclusions: This study highlighted the effectiveness of various BERT models in a multilingual clinical domain. The findings can be used as a reference in clinical and language-based applications. (© Kyungmo Kim, Seongkeun Park, Jeongwon Min, Sumin Park, Ju Yeon Kim, Jinsu Eun, Kyuha Jung, Yoobin Elyson Park, Esther Kim, Eun Young Lee, Joonhwan Lee, Jinwook Choi. Originally published in JMIR Medical Informatics (https://medinform.jmir.org).)
Databáze:	MEDLINE
Externí odkaz:	Zobrazit plný text záznamu