Crowdsourcing for error detection in cortical surface delineations

Autor: Lena Maier-Hein, Melanie Ganz, Daniel Kondermann, Gitte M. Knudsen, Jonas Andrulis
Rok vydání: 2016
Předmět:
Quality Control
020205 medical informatics
Computer science
media_common.quotation_subject
Big data
Biomedical Engineering
Datasets as Topic
Neuroimaging
Health Informatics
02 engineering and technology
computer.software_genre
Crowdsourcing
Bottleneck
Automation
03 medical and health sciences
Imaging
Three-Dimensional
0302 clinical medicine
Image Processing
Computer-Assisted
0202 electrical engineering
electronic engineering
information engineering
Humans
Radiology
Nuclear Medicine and imaging
Quality (business)
Sensitivity (control systems)
media_common
Cerebral Cortex
business.industry
General Medicine
Online community
Magnetic Resonance Imaging
Computer Graphics and Computer-Aided Design
Computer Science Applications
Surgery
Computer Vision and Pattern Recognition
Data mining
Error detection and correction
business
Focus (optics)
computer
Algorithms
030217 neurology & neurosurgery
Zdroj: International Journal of Computer Assisted Radiology and Surgery. 12:161-166
ISSN: 1861-6429
1861-6410
DOI: 10.1007/s11548-016-1445-9
Popis: With the recent trend toward big data analysis, neuroimaging datasets have grown substantially in the past years. While larger datasets potentially offer important insights for medical research, one major bottleneck is the requirement for resources of medical experts needed to validate automatic processing results. To address this issue, the goal of this paper was to assess whether anonymous nonexperts from an online community can perform quality control of MR-based cortical surface delineations derived by an automatic algorithm. So-called knowledge workers from an online crowdsourcing platform were asked to annotate errors in automatic cortical surface delineations on 100 central, coronal slices of MR images. On average, annotations for 100 images were obtained in less than an hour. When using expert annotations as reference, the crowd on average achieves a sensitivity of 82 % and a precision of 42 %. Merging multiple annotations per image significantly improves the sensitivity of the crowd (up to 95 %), but leads to a decrease in precision (as low as 22 %). Our experiments show that the detection of errors in automatic cortical surface delineations generated by anonymous untrained workers is feasible. Future work will focus on increasing the sensitivity of our method further, such that the error detection tasks can be handled exclusively by the crowd and expert resources can be focused on error correction.
Databáze: OpenAIRE
Externí odkaz: