Word Embeddings Revisited: Do LLMs Offer Something New?

Autor:	Freestone, Matthew, Santu, Shubhra Kanti Karmaker
Rok vydání:	2024
Předmět:	Computer Science - Computation and Language I.2.7
Druh dokumentu:	Working Paper
Popis:	Learning meaningful word embeddings is key to training a robust language model. The recent rise of Large Language Models (LLMs) has provided us with many new word/sentence/document embedding models. Although LLMs have shown remarkable advancement in various NLP tasks, it is still unclear whether the performance improvement is merely because of scale or whether underlying embeddings they produce significantly differ from classical encoding models like Sentence-BERT (SBERT) or Universal Sentence Encoder (USE). This paper systematically investigates this issue by comparing classical word embedding techniques against LLM-based word embeddings in terms of their latent vector semantics. Our results show that LLMs tend to cluster semantically related words more tightly than classical models. LLMs also yield higher average accuracy on the Bigger Analogy Test Set (BATS) over classical methods. Finally, some LLMs tend to produce word embeddings similar to SBERT, a relatively lighter classical model. Comment: 7 pages, 4 figures
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2402.11094 Zobrazit plný text záznamu View this record from Arxiv