XferBench: a Data-Driven Benchmark for Emergent Language

Autor:	Boldt, Brendon, Mortensen, David
Rok vydání:	2024
Předmět:	Computer Science - Computation and Language I.2.7 I.6.m
Zdroj:	Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 1475-1489
Druh dokumentu:	Working Paper
Popis:	In this paper, we introduce a benchmark for evaluating the overall quality of emergent languages using data-driven methods. Specifically, we interpret the notion of the "quality" of an emergent language as its similarity to human language within a deep learning framework. We measure this by using the emergent language as pretraining data for a downstream NLP tasks in human language -- the better the downstream performance, the better the emergent language. We implement this benchmark as an easy-to-use Python package that only requires a text file of utterances from the emergent language to be evaluated. Finally, we empirically test the benchmark's validity using human, synthetic, and emergent language baselines. Comment: 15 pages, 5 figures
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2407.03456 Zobrazit plný text záznamu View this record from Arxiv