FourEyes
| Autor: | Raymond Fok, Walter S. Lasecki, Jean Y. Song, Juho Kim |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
Computer science
business.industry 05 social sciences Aggregate (data warehouse) 020207 software engineering 02 engineering and technology Image segmentation Machine learning computer.software_genre Crowdsourcing Object (computer science) Image (mathematics) Variety (cybernetics) Human-Computer Interaction Artificial Intelligence 0202 electrical engineering electronic engineering information engineering 0501 psychology and cognitive sciences Segmentation Artificial intelligence Set (psychology) business computer 050107 human factors |
| Zdroj: | ACM Transactions on Interactive Intelligent Systems. 10:1-30 |
| ISSN: | 2160-6463 2160-6455 |
| DOI: | 10.1145/3237188 |
| Popis: | Crowdsourcing is a common means of collecting image segmentation training data for use in a variety of computer vision applications. However, designing accurate crowd-powered image segmentation systems is challenging, because defining object boundaries in an image requires significant fine motor skills and hand-eye coordination, which makes these tasks error-prone. Typically, special segmentation tools are created and then answers from multiple workers are aggregated to generate more accurate results. However, individual tool designs can bias how and where people make mistakes, resulting in shared errors that remain even after aggregation. In this article, we introduce a novel crowdsourcing approach that leverages tool diversity as a means of improving aggregate crowd performance. Our idea is that given a diverse set of tools, answer aggregation done across tools can help improve the collective performance by offsetting systematic biases induced by the individual tools themselves. To demonstrate the effectiveness of the proposed approach, we design four different tools and present FourEyes, a crowd-powered image segmentation system that uses aggregation across different tools. We then conduct a series of studies that evaluate different aggregation conditions and show that using multiple tools can significantly improve aggregate accuracy. Furthermore, we investigate the idea of applying post-processing for multi-tool aggregation in terms of correction mechanism. We introduce a novel region-based method for synthesizing more accurate bounds for image segmentation tasks through averaging surrounding annotations. In addition, we explore the effect of adjusting the threshold parameter of an EM-based aggregation method. Our results suggest that not only the individual tool’s design, but also the correction mechanism, can affect the performance of multi-tool aggregation. This article extends a work presented at ACM IUI 2018 [46] by providing a novel region-based error-correction method and additional in-depth evaluation of the proposed approach. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|