Crowdsourcing Without a Crowd: Reliable Online Species Identification Using Bayesian Models to Minimize Crowd Size
| Autor: | Elaine O'mahony, Chris Mellish, Christopher Lambin, Richard Comont, Nirwan Sharma, René van der Wal, Anne-Marie Robinson, Advaith Siddharthan |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2016 |
| Předmět: |
0106 biological sciences
Majority rule Computer science media_common.quotation_subject Bayesian probability 02 engineering and technology Machine learning computer.software_genre Bayesian inference Crowdsourcing 010603 evolutionary biology 01 natural sciences Theoretical Computer Science Task (project management) Artificial Intelligence 0202 electrical engineering electronic engineering information engineering Citizen science Quality (business) media_common business.industry Data science Data quality 020201 artificial intelligence & image processing Artificial intelligence business computer |
| ISSN: | 2157-6912 |
| Popis: | We present an incremental Bayesian model that resolves key issues of crowd size and data quality for consensus labeling. We evaluate our method using data collected from a real-world citizen science program, B ee W atch , which invites members of the public in the United Kingdom to classify (label) photographs of bumblebees as one of 22 possible species. The biological recording domain poses two key and hitherto unaddressed challenges for consensus models of crowdsourcing: (1) the large number of potential species makes classification difficult, and (2) this is compounded by limited crowd availability, stemming from both the inherent difficulty of the task and the lack of relevant skills among the general public. We demonstrate that consensus labels can be reliably found in such circumstances with very small crowd sizes of around three to five users (i.e., through group sourcing). Our incremental Bayesian model, which minimizes crowd size by re-evaluating the quality of the consensus label following each species identification solicited from the crowd, is competitive with a Bayesian approach that uses a larger but fixed crowd size and outperforms majority voting. These results have important ecological applicability: biological recording programs such as B ee W atch can sustain themselves when resources such as taxonomic experts to confirm identifications by photo submitters are scarce (as is typically the case), and feedback can be provided to submitters in a timely fashion. More generally, our model provides benefits to any crowdsourced consensus labeling task where there is a cost (financial or otherwise) associated with soliciting a label. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|