Self-Adaptive Reconstruction with Contrastive Learning for Unsupervised Sentence Embeddings

Autor:	Liu, Junlong, Shang, Xichen, Feng, Huawen, Zheng, Junhao, Ma, Qianli
Rok vydání:	2024
Předmět:	Computer Science - Computation and Language Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Unsupervised sentence embeddings task aims to convert sentences to semantic vector representations. Most previous works directly use the sentence representations derived from pretrained language models. However, due to the token bias in pretrained language models, the models can not capture the fine-grained semantics in sentences, which leads to poor predictions. To address this issue, we propose a novel Self-Adaptive Reconstruction Contrastive Sentence Embeddings (SARCSE) framework, which reconstructs all tokens in sentences with an AutoEncoder to help the model to preserve more fine-grained semantics during tokens aggregating. In addition, we proposed a self-adaptive reconstruction loss to alleviate the token bias towards frequency. Experimental results show that SARCSE gains significant improvements compared with the strong baseline SimCSE on the 7 STS tasks. Comment: 8 pages, 3 figures
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2402.15153 Zobrazit plný text záznamu View this record from Arxiv