Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction
| Autor: | Byron C. Wallace, Ani Nenkova, Oshin Agarwal, Chris Tar, Yinfei Yang |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
FOS: Computer and information sciences
Computer science media_common.quotation_subject 02 engineering and technology Crowdsourcing Machine learning computer.software_genre Domain (software engineering) Task (project management) 03 medical and health sciences 020204 information systems 0202 electrical engineering electronic engineering information engineering Quality (business) Set (psychology) 030304 developmental biology media_common 0303 health sciences Computer Science - Computation and Language business.industry Biomedical information Artificial intelligence Noise (video) Routing (electronic design automation) business computer Computation and Language (cs.CL) |
| Zdroj: | NAACL-HLT (1) |
| DOI: | 10.48550/arxiv.1905.07791 |
| Popis: | Modern NLP systems require high-quality annotated data. In specialized domains, expert annotations may be prohibitively expensive. An alternative is to rely on crowdsourcing to reduce costs at the risk of introducing noise. In this paper we demonstrate that directly modeling instance difficulty can be used to improve model performance, and to route instances to appropriate annotators. Our difficulty prediction model combines two learned representations: a `universal' encoder trained on out-of-domain data, and a task-specific encoder. Experiments on a complex biomedical information extraction task using expert and lay annotators show that: (i) simply excluding from the training data instances predicted to be difficult yields a small boost in performance; (ii) using difficulty scores to weight instances during training provides further, consistent gains; (iii) assigning instances predicted to be difficult to domain experts is an effective strategy for task routing. Our experiments confirm the expectation that for specialized tasks expert annotations are higher quality than crowd labels, and hence preferable to obtain if practical. Moreover, augmenting small amounts of expert data with a larger set of lay annotations leads to further improvements in model performance. Comment: NAACL2019 |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|