Model-agnostic unsupervised detection of bots in a Likert-type questionnaire.

Autor: Ilagan MJ; Department of Psychology, McGill University, 2001 McGill College, 7th Floor, H3A 1G1, Montreal, QC, Canada., Falk CF; Department of Psychology, McGill University, 2001 McGill College, 7th Floor, H3A 1G1, Montreal, QC, Canada. carl.falk@mcgill.ca.
Jazyk: angličtina
Zdroj: Behavior research methods [Behav Res Methods] 2024 Aug; Vol. 56 (5), pp. 5068-5085. Date of Electronic Publication: 2023 Nov 20.
DOI: 10.3758/s13428-023-02246-7
Abstrakt: To detect bots in online survey data, there is a wealth of literature on statistical detection using only responses to Likert-type items. There are two traditions in the literature. One tradition requires labeled data, forgoing strong model assumptions. The other tradition requires a measurement model, forgoing collection of labeled data. In the present article, we consider the problem where neither requirement is available, for an inventory that has the same number of Likert-type categories for all items. We propose a bot detection algorithm that is both model-agnostic and unsupervised. Our proposed algorithm involves a permutation test with leave-one-out calculations of outlier statistics. For each respondent, it outputs a p value for the null hypothesis that the respondent is a bot. Such an algorithm offers nominal sensitivity calibration that is robust to the bot response distribution. In a simulation study, we found our proposed algorithm to improve upon naive alternatives in terms of 95% sensitivity calibration and, in many scenarios, in terms of classification accuracy.
(© 2023. The Psychonomic Society, Inc.)
Databáze: MEDLINE
Externí odkaz: