Performance of a Chest Radiograph AI Diagnostic Tool for COVID-19: A Prospective Observational Study.

Autor:	Sun J; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Peng L; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Li T; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Adila D; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Zaiman Z; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Melton-Meaux GB; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Ingraham NE; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Murray E; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Boley D; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Switzer S; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Burns JL; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Huang K; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Allen T; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Steenburg SD; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Gichoya JW; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Kummerfeld E; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.)., Tignanelli CJ; Department of Computer Science and Engineering (J.S., L.P., T.L., D.A., D.B.), Institute for Health Informatics (G.B.M.M., E.K., C.J.T.), Department of Surgery (G.B.M.M., C.J.T.), Department of Medicine, Division of Pulmonary and Critical Care (N.E.I.), Department of Medicine (S.S.), and Department of Radiology (T.A.), University of Minnesota, 420 Delaware St SE, Minneapolis, MN 55455; Departments of Computer Science (Z.Z.) and Radiology (J.W.G.), Emory University, Atlanta, Ga; M Health Fairview Informatics, Minneapolis, Minn (E.M.); The School of Medicine (J.L.B., K.H.) and Department of Radiology (S.D.S.), Indiana University, Indianapolis, Ind; and Department of Surgery, North Memorial Health Hospital, Robbinsdale, Minn (C.J.T.).
Jazyk:	angličtina
Zdroj:	Radiology. Artificial intelligence [Radiol Artif Intell] 2022 Jun 01; Vol. 4 (4), pp. e210217. Date of Electronic Publication: 2022 Jun 01 (Print Publication: 2022).
DOI:	10.1148/ryai.210217
Abstrakt:	Purpose: To conduct a prospective observational study across 12 U.S. hospitals to evaluate real-time performance of an interpretable artificial intelligence (AI) model to detect COVID-19 on chest radiographs. Materials and Methods: A total of 95 363 chest radiographs were included in model training, external validation, and real-time validation. The model was deployed as a clinical decision support system, and performance was prospectively evaluated. There were 5335 total real-time predictions and a COVID-19 prevalence of 4.8% (258 of 5335). Model performance was assessed with use of receiver operating characteristic analysis, precision-recall curves, and F1 score. Logistic regression was used to evaluate the association of race and sex with AI model diagnostic accuracy. To compare model accuracy with the performance of board-certified radiologists, a third dataset of 1638 images was read independently by two radiologists. Results: Participants positive for COVID-19 had higher COVID-19 diagnostic scores than participants negative for COVID-19 (median, 0.1 [IQR, 0.0-0.8] vs 0.0 [IQR, 0.0-0.1], respectively; P < .001). Real-time model performance was unchanged over 19 weeks of implementation (area under the receiver operating characteristic curve, 0.70; 95% CI: 0.66, 0.73). Model sensitivity was higher in men than women ( P = .01), whereas model specificity was higher in women ( P = .001). Sensitivity was higher for Asian ( P = .002) and Black ( P = .046) participants compared with White participants. The COVID-19 AI diagnostic system had worse accuracy (63.5% correct) compared with radiologist predictions (radiologist 1 = 67.8% correct, radiologist 2 = 68.6% correct; McNemar P < .001 for both). Conclusion: AI-based tools have not yet reached full diagnostic potential for COVID-19 and underperform compared with radiologist prediction. Keywords: Diagnosis, Classification, Application Domain, Infection, Lung Supplemental material is available for this article. . © RSNA, 2022. Competing Interests: Disclosures of conflicts of interest: J.S. No relevant relationships. L.P. No relevant relationships. T.L. No relevant relationships. D.A. No relevant relationships. Z.Z. No relevant relationships. G.B.M.M. Payments to institution: Fairview, University of Minnesota, NIH, AHRQ; payment or honoraria for lectures, presentations, or speakers’ bureaus from AMIA, ACMI, Washington University Informatics, and NIH; support for attending meetings and/or travel from AMIA, ACMI, Washington University Informatics, and NIH; patent planned, issued, or pending for application 17/302,373; participation on a data safety monitoring board or advisory board not related to this article; leadership or fiduciary role at AMIA and ACMI. N.E.I. No relevant relationships. E.M. No relevant relationships. D.B. No relevant relationships. S.S. No relevant relationships. J.L.B. No relevant relationships. K.H. Internal funding from Indiana University Precision Health Initiative; author received research grants from NIH; author was principal investigator (PI) for a research contract with Merck (not relevant to this manuscript); author is PI for a research contract with Eli Lilly (not relevant to this manuscript). T.A. Speakers’ bureau for Genentech and Boehringer Ingelheim; gives nonbranded lectures on interstitial lung disease unrelated to this manuscript. S.D.S. No relevant relationships. J.W.G. Medical Imaging and Data Resource Center grant from NIH; Nightingale Open Science grant; NSF Future of Work grant; Kheiron Breast AI Validation grant; BDC Data Anonymization grant; Radiology: Artificial Intelligence trainee editorial board lead. E.K. Grant UL1TR002494; this technology submitted for a U.S. patent. C.J.T. Microsoft AI for Health provided GPU resources via a COVID-19 Research Grant (no payments, other support, or any role in the actual conduct of the research was had by Microsoft); technology was submitted for a U.S. patent. (© 2022 by the Radiological Society of North America, Inc.)
Databáze:	MEDLINE
Externí odkaz:	Zobrazit plný text záznamu