| Popis: |
The goal of this study is to understand the emergence of demographic biases in data annotation tasks. Researchers often ask research assistants or crowdsourced workers to annotate text (e.g. news articles, social media posts), images, and other pieces of data; either for direct analysis or to train machine learning models. Performance across annotators is often evaluated using common inter-rater reliability (IRR) statistics, and when IRR is satisfactory, researchers move on in using the manual annotations for their analyses or training their models. However, anecdotal evidence suggests that individual characteristics of manual annotators (e.g. gender and ideology) may lead to systematic differences in the annotation process. In addition, although coding-training may help substantially reduce such biases for some annotation tasks, they may not work as well in other contexts. |
