Is the crowd better as an assistant or a replacement in ontology engineering? An exploration through the lens of the Gene Ontology
| Autor: | Hua Fan-Minogue, Kimberly Van Auken, Mark A. Musen, Michel Dumontier, Jonathan M. Mortensen, Jacob J. Hughey, Natalie Telis |
|---|---|
| Rok vydání: | 2016 |
| Předmět: |
0301 basic medicine
Computer science Health Informatics Ontology (information science) computer.software_genre Crowdsourcing Article Task (project management) Open Biomedical Ontologies 03 medical and health sciences 0302 clinical medicine Systematized Nomenclature of Medicine Task Performance and Analysis Humans 030212 general & internal medicine Analysis of Variance Internet SNOMED CT Ontology engineering business.industry Computational Biology Data science Computer Science Applications Search Engine Gene Ontology 030104 developmental biology Area Under Curve The Internet business computer Algorithms Software |
| Zdroj: | Journal of Biomedical Informatics, 60, 199-209. Elsevier Science |
| ISSN: | 1532-0464 |
| DOI: | 10.1016/j.jbi.2016.02.005 |
| Popis: | Display Omitted Experts identified 16 errors, mainly related to acids and metals.Definition quality and task difficulty affect crowd performance significantly.The number of Internet search results could help identify "easy" crowd tasks.The crowd fits best as an assistant, helping experts with their verification tasks. Biomedical ontologies contain errors. Crowdsourcing, defined as taking a job traditionally performed by a designated agent and outsourcing it to an undefined large group of people, provides scalable access to humans. Therefore, the crowd has the potential to overcome the limited accuracy and scalability found in current ontology quality assurance approaches. Crowd-based methods have identified errors in SNOMED CT, a large, clinical ontology, with an accuracy similar to that of experts, suggesting that crowdsourcing is indeed a feasible approach for identifying ontology errors. This work uses that same crowd-based methodology, as well as a panel of experts, to verify a subset of the Gene Ontology (200 relationships). Experts identified 16 errors, generally in relationships referencing acids and metals. The crowd performed poorly in identifying those errors, with an area under the receiver operating characteristic curve ranging from 0.44 to 0.73, depending on the methods configuration. However, when the crowd verified what experts considered to be easy relationships with useful definitions, they performed reasonably well. Notably, there are significantly fewer Google search results for Gene Ontology concepts than SNOMED CT concepts. This disparity may account for the difference in performance - fewer search results indicate a more difficult task for the worker. The number of Internet search results could serve as a method to assess which tasks are appropriate for the crowd. These results suggest that the crowd fits better as an expert assistant, helping experts with their verification by completing the easy tasks and allowing experts to focus on the difficult tasks, rather than an expert replacement. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect