DiPietro-Hazari Kappa: A Novel Metric for Assessing Labeling Quality via Annotation

Autor:	DiPietro, Daniel M., Hazari, Vivek
Rok vydání:	2022
Předmět:	Computer Science - Machine Learning Mathematics - Statistics Theory
Druh dokumentu:	Working Paper
Popis:	Data is a key component of modern machine learning, but statistics for assessing data label quality remain sparse in literature. Here, we introduce DiPietro-Hazari Kappa, a novel statistical metric for assessing the quality of suggested dataset labels in the context of human annotation. Rooted in the classical Fleiss's Kappa measure of inter-annotator agreement, the DiPietro-Hazari Kappa quantifies the the empirical annotator agreement differential that was attained above random chance. We offer a thorough theoretical examination of Fleiss's Kappa before turning to our derivation of DiPietro-Hazari Kappa. Finally, we conclude with a matrix formulation and set of procedural instructions for easy computational implementation. Comment: 6 pages
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2209.08243 Zobrazit plný text záznamu View this record from Arxiv