Rethinking STS and NLI in Large Language Models

Autor:	Wang, Yuxia, Wang, Minghan, Nakov, Preslav
Rok vydání:	2023
Předmět:	Computer Science - Computation and Language
Druh dokumentu:	Working Paper
Popis:	Recent years have seen the rise of large language models (LLMs), where practitioners use task-specific prompts; this was shown to be effective for a variety of tasks. However, when applied to semantic textual similarity (STS) and natural language inference (NLI), the effectiveness of LLMs turns out to be limited by low-resource domain accuracy, model overconfidence, and difficulty to capture the disagreements between human judgements. With this in mind, here we try to rethink STS and NLI in the era of LLMs. We first evaluate the performance of STS and NLI in the clinical/biomedical domain, and then we assess LLMs' predictive confidence and their capability of capturing collective human opinions. We find that these old problems are still to be properly addressed in the era of LLMs. Comment: arXiv admin note: text overlap with arXiv:2212.13138 by other authors
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2309.08969 Zobrazit plný text záznamu View this record from Arxiv