Annotation of phenotypes using ontologies:a gold standard for the training and evaluation of natural language processing systems
| Autor: | Prashanti Manda, James P. Balhoff, Hong Cui, Hilmar Lapp, Todd Vision, Paula M. Mabee, Nizar Ibrahim, T. Alexander Dececchi, Wasila M. Dahdul |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2018 |
| Předmět: |
0301 basic medicine
0106 biological sciences Ontology (information science) computer.software_genre Semantics 010603 evolutionary biology 01 natural sciences General Biochemistry Genetics and Molecular Biology 03 medical and health sciences Consistency (database systems) Annotation Library and Information Studies Similarity (psychology) Data Mining Humans Data Curation 030304 developmental biology Natural Language Processing 0303 health sciences Point (typography) Agricultural and Biological Sciences(all) business.industry Biochemistry Genetics and Molecular Biology(all) Principal (computer security) Gold standard (test) Object (computer science) Data Format 030104 developmental biology Networking and Information Technology R&D Gene Ontology Phenotype Original Article Artificial intelligence General Agricultural and Biological Sciences business computer Natural language processing Natural language Information Systems |
| Zdroj: | Dahdul, W, Manda, P, Cui, H, Balhoff, J P, Dececchi, T A, Ibrahim, N, Lapp, H, Vision, T & Mabee, P M 2018, ' Annotation of phenotypes using ontologies : a gold standard for the training and evaluation of natural language processing systems ', Database: The Journal of Biological Databases and Curation, vol. 2018 . https://doi.org/10.1093/database/bay110 Database: The Journal of Biological Databases and Curation |
| Popis: | Natural language descriptions of organismal phenotypes, a principal object of study in biology, are abundant in the biological literature. Expressing these phenotypes as logical statements using ontologies would enable large-scale analysis on phenotypic information from diverse systems. However, considerable human effort is required to make these phenotype descriptions amenable to machine reasoning. Natural language processing tools have been developed to facilitate this task, and the training and evaluation of these tools depend on the availability of high quality, manually annotated gold standard data sets. We describe the development of an expert-curated gold standard data set of annotated phenotypes for evolutionary biology. The gold standard was developed for the curation of complex comparative phenotypes for the Phenoscape project. It was created by consensus among three curators and consists of entity–quality expressions of varying complexity. We use the gold standard to evaluate annotations created by human curators and those generated by the Semantic CharaParser tool. Using four annotation accuracy metrics that can account for any level of relationship between terms from two phenotype annotations, we found that machine–human consistency, or similarity, was significantly lower than inter-curator (human–human) consistency. Surprisingly, allowing curatorsaccess to external information did not significantly increase the similarity of their annotations to the gold standard or have a significant effect on inter-curator consistency. We found that the similarity of machine annotations to the gold standard increased after new relevant ontology terms had been added. Evaluation by the original authors of the character descriptions indicated that the gold standard annotations came closer to representing their intended meaning than did either the curator or machine annotations. These findings point toward ways to better design software to augment human curators and the use of the gold standard corpus will allow training and assessment of new tools to improve phenotype annotation accuracy at scale. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|