ElicitationGPT: Text Elicitation Mechanisms via Language Models

Autor:	Wu, Yifan, Hartline, Jason
Rok vydání:	2024
Předmět:	Computer Science - Artificial Intelligence Computer Science - Computer Science and Game Theory Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Scoring rules evaluate probabilistic forecasts of an unknown state against the realized state and are a fundamental building block in the incentivized elicitation of information and the training of machine learning models. This paper develops mechanisms for scoring elicited text against ground truth text using domain-knowledge-free queries to a large language model (specifically ChatGPT) and empirically evaluates their alignment with human preferences. The empirical evaluation is conducted on peer reviews from a peer-grading dataset and in comparison to manual instructor scores for the peer reviews.
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2406.09363 Zobrazit plný text záznamu View this record from Arxiv