DiQAD: A Benchmark Dataset for End-to-End Open-domain Dialogue Assessment

Autor:	Zhao, Yukun, Yan, Lingyong, Sun, Weiwei, Meng, Chong, Wang, Shuaiqiang, Cheng, Zhicong, Ren, Zhaochun, Yin, Dawei
Rok vydání:	2023
Předmět:	Computer Science - Computation and Language
Druh dokumentu:	Working Paper
Popis:	Dialogue assessment plays a critical role in the development of open-domain dialogue systems. Existing work are uncapable of providing an end-to-end and human-epistemic assessment dataset, while they only provide sub-metrics like coherence or the dialogues are conversed between annotators far from real user settings. In this paper, we release a large-scale dialogue quality assessment dataset (DiQAD), for automatically assessing open-domain dialogue quality. Specifically, we (1) establish the assessment criteria based on the dimensions conforming to human judgements on dialogue qualities, and (2) annotate large-scale dialogues that conversed between real users based on these annotation criteria, which contains around 100,000 dialogues. We conduct several experiments and report the performances of the baselines as the benchmark on DiQAD. The dataset is openly accessible at https://github.com/yukunZhao/Dataset_Dialogue_quality_evaluation. Comment: Accepted to Findings of EMNLP 2023
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2310.16319 Zobrazit plný text záznamu View this record from Arxiv